3 rd SUMMER EUROPEAN UNIVERSITY Surgial robotis Montpellier, September 2007 VISUAL SERVOING with appliations in medial robotis Florent Nageotte, Mihel de Mathelin Louis Pasteur Uniersity Strasbourg LSIIT Control, Vision & Robotis Researh Group UMR CNRS 7005 1
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 2
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 3
Visual seroing priniple Control the end-effetor of a robot using a ision sensor (2D sensor). E.g : position the effetor of a robot w.r.t. an objet Similar to registration but the eloity of the robot is generally ontinuously updated Minimize a task funtion dependent of the error between the urrent pose of the robot and the referene pose Creates a irtual link between robot and objet Image proessing, omputer ision and ontrol issues 4
I.1 Bakground and definitions A. Coordinates and pose Coordinates of point P with respet to oordinate frame i : i P Position and orientation of frame i with respet to frame j : Pose j p i Tx T y T z α β γ translation etor rotation angles j j T R i i origin of frame i w.r. to frame j rotation matrix 5
6 B. Coordinate transformations Coordinates of point i P with respet to oordinate frame j : Coordinates of etor i V with respet to frame j : Homogeneous transformation from frame i to frame j : I.1 Bakground I.1 Bakground and and definitions definitions P R T P i i j i j j + V R V i i j j 0 0 1 1 1 0 V H V P H P T R H i i j j i i j j i j i j i j i k k j i j H H H
7 C. Veloity of a rigid objet Veloity srew of frame i with respet to frame j in frame j oordinates: Veloity of point P rigidly attahed to frame i with respet to frame j expressed in frame j oordinates : I.1 Bakground I.1 Bakground and and definitions definitions ( ) ( ) ( ) ( ) i j j i j i i j i j j i j j j i j j j V T P R V P P + + Ω + Ω ) ( ( ) ( ) ( ) i j j i j j z y x z y x i j j V r Ω eloity rotational eloity translatio nal ω ω ω
8 D. Camera projetion model Perspetie projetion : oordinates of point P with respet to the amera frame I.1 Bakground I.1 Bakground and and definitions definitions image plane P Y X Z λ o x y u optial enter amera frame + + u C z y k z x k u u z y z x y x z y x P λ λ λ 0 0 p λ foal length P pixels
9 Intrinsi amera parameters obtained by alibration Mathematial model of a perfet optial system / physial phenomenon (distorsions, et.) Numerial sensor : aliasing, dynamial behaiour (limited bandwidth) Other models apply for other types of «isual» sensors, e.g., C-arm, CTsan, ultrasound probe, I.1 Bakground I.1 Bakground and and definitions definitions D. Camera projetion model u z y x K z y x k u k u z 1 0 0 0 0 1 0 0 λ λ
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 10
I.2 Classifiation I.21 Camera position A. Eye-in-hand onfiguration H t b H e e H e H must be known (alibration) or the referene position must be learned (showing) 11
I.2 Classifiation I.21 Camera position B. External amera onfiguration H e must be measured or b H must be known b H e b H H e H t 12
I.2 Classifiation I.22 Control leel A. Indiret isual seroing Cartesian ontroller p or r Trajetory generator q Joint ontrol Power amplifiers Control law - joint angles q Image proessing T Robot+amera image Suitable for slow isual seroing (T -1 < 50 Hz) Control law is easier to design 13
I.2 Classifiation I.22 Control leel B. Diret isual seroing - Control law τ or q Low leel joint ontrollers q Image proessing T Robot+amera image Suitable for fast isual seroing (T -1 > 50 Hz) Control law design is more omplex (robot dynamis must be taken into aount) 14
I.2 Classifiation I.23 Feedbak ariables A. Position-based isual seroing (3D isual seroing) p d - Control Power amplifiers p Robot+amera 15 Pose estimation Features extration A model of the objet must be known or multiple images should be used Calibration errors may indue large pose estimation errors Control law design is easier Possible loss of target for large errors f T image
I.2 Classifiation I.23 Feedbak ariables B. Image-based isual seroing (2D isual seroing) f d - Control Power amplifiers 16 Features extration Smaller omputational burden Eliminates errors due to alibration More omplex ontrol law Workspae limits an be hit for large errors f T Robot+amera image
I.2 Classifiation I.23 Feedbak ariables C. Hybrid isual seroing (e.g. 2D1/2 isual seroing) f pd p pd - - Hybrid ontrol law Power amplifiers p p f p Partial pose estimation Features extration Smaller errors due to alibration Simplified model of the target Better properties of motion (in the image and in 3D) f T Robot+amera image 17
I.2 Classifiation I.24 Bandwidth of the isual sero-loop A. Slow isual seroing Sampling frequeny < 50Hz Indiret isual seroing Robot transfer funtion model without dynamis Proportional ontrol law (P) B. Fast isual seroing Sampling frequeny > 50 Hz Diret isual seroing Dynamial model of the robot must be taken into aount More adaned ontrol laws : PID, preditie, robust, non-linear, 18
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 19
I.3 Position-based isual seroing I.31 Indiret isual seroing A. Control law p d e p - Control law p or r Cartesian ontroller pˆ p + δp Pose estimation fˆ joint angles q Features extration T Robot+amera image 20 e p generally expressed as translation etor and Look-then-moe strategy (T ery large, asynhronous) : Pseudo-ontinuous strategy : p T J r Control law : p θu p ep r T k J 1 p ep
I.4 Image-based isual seroing I.41 Indiret isual seroing B. Obtaining referene pd Referene position of the end-effetor effetor is gien w.r.t an objet Computation of the orresponding position of the amera w.r.t the objet (eye-in-hand) requires the knowledge of e H Learning of the image in the final position and pose estimation of the amera 21
22 Case of a eye-in-hand system u T e p θ I.3 I.3 Position Position-based based isual isual seroing seroing I.31 Indiret isual I.31 Indiret isual seroing seroing C. Interation matrix Jp Ω Ω + Ω Ω e e e e e e e e e e p V R R EC R R R R R R EC V R R C e 0 ) ( ) (
D. Pose estimation Camera alibration : Tsaï (IEEE Trans. Rob. Aut., 1987), Zhang ( ICCV 99) Pose estimation : Analytial : 3 or 4 points : Tsaï (o-planar target) Numerial Iteratie : I.3 Position-based isual seroing I.31 Indiret isual seroing - DeMenthon (IEEE Trans. PAMI 1992, Int. J. Comp. Vision 1995), - VVS (Marhand, Eurographis 2002), - Minimization of reprojetion error (gradient, Leenberg-Marquardt, et.) 23
I.3 Position-based isual seroing I.31 Indiret isual seroing E. Stability and robustness Stability is not an issue : Look-then-moe strategy is always stable and low speed ision loop: r r p k J T p J T p 1 e p Exponential onergene J p depends on amera position w.r. to end-effetor (eye-in-hand onfiguration) Stability if J J ˆ 1 P P > 0 Measurement error is an issue: an be ery large! δp Main soures of unertainty : Improement: use learning Camera intrinsi parameters of p by showing when possible Camera position w.r. to end-effetor if eye-in-hand onfiguration Camera position w.r. to the robot base and robot kinemati hain if external amera, exept if end-effetor pose is estimated by ision Feature detetion error : δf f ˆ f 24
I.3 Position-based isual seroing I.32 Diret isual seroing A. Control law p d e p or Control - law q τ Low-leel joint ontrollers pˆ p + δp Robot+amera Pose estimation Features extration T image p p R q Control law : 1 T 1 q k J R ( q) J p e p T T p J r J J ( q) 25
I.3 Position-based isual seroing I.32 Diret isual seroing B. Stability and robustness (1) Stability may be an issue: Low speed ision loop : q q p kj T p J R ( q) J 1 R ( q) J T p 1 e p Exponential onergene High speed ision loop : Linearized approah : 1 p J s T p J R ( q) F( s, q) q q ( s) F( s, q) q ( s) Joint-leel eloity feedbak loops hae a linearizing and deoupling effet 1 R T 1 p q J ( q) J p Control law: with using a LPV disrete-time model of the ision loop p omputed 26 This approah works in pratie with 6DOF ision loop!
I.3 Position-based isual seroing I.32 Diret isual seroing B. Stability and robustness (2) LPV disrete-time model p d ontrol algorithm pˆ + p δp p J 1 1 ( ) T R q J p aquisition and image proessing Tz d 1 z 1 p q J T p F(z, q) robot identified model J R (q) q GPC of a 6DOF robot ision loop : J. Gangloff & M. de Mathelin (Adaned Robotis, ol 17, no 10, dé. 2003) 27
I.3 Position-based isual seroing I.32 Diret isual seroing B. Stability and robustness (3) Non linear approah : rigid link robot manipulator model τ M ( q) q + C( q, q ) q + g( q) + fr ( q, q ) Inertia Coriolis, entripetal graity frition -PD ontrol sheme(arimoto): τ g( q) K q K e with - Passiity-based sheme (Slotine & Li): τ M ( q) ζ + C( q, q ) ζ + g( q ) K e ζ q q K p p e q Λ d e q Problems: - asymptoti stability is proen with no frition - qd and q d are generally unknown - joint flexibilities and baklash q e q q q d High omplexity for 6DOF! 28
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 29
I.4 Image-based isual seroing I.41 Indiret isual seroing A. Control law f d e f - Control law p or r Cartesian ontroller q fˆ f + δf Features extration pixel noise T Robot+amera image Look-then-moe strategy : selet or to derease a ost funtion of Pseudo-ontinuous strategy : p f J I r q Control law : r k + e f J I e f 30
B. Obtaining referene d I.4 Image-based isual seroing I.41 Indiret isual seroing f Computation of the perspetie projetion of the objet in the referene position requires a model of the objet requires the knowledge of e H requires an aurate model of amera Learning of the image in the final position 31
I.4 Image-based isual seroing I.41 Indiret isual seroing C. Image jaobian or interation matrix Image Jaobian J I : p x n matrix (p feature etor dimension, n # dof of effetor). Depends on kind of features used. Exemple : ase of a point P seen by a amera attahed to the effetor of a robot u P X C Z C Z C Z O P X C Y C X O Y O 32
33 ( ) z y x z y x o r ω ω ω ( ) ( ) o o V P P + Ω z y x P + + + z y u x y x z u x z y k u u k z z k z u u k z z z y x P ) ) ( ) ( ( ) ( ) ( 0 0 0 0 ω ω λ ω ω λ ω λ ω image oordinates in pixel: (u,) I.4 I.4 Image Image-based based isual isual seroing seroing I.41 Indiret isual I.41 Indiret isual seroing seroing ) ( ) ( ) )( ( ) ( ) ( ) ( 0 ) ( ) ( ) ( ) )( ( ) ( 0 0 0 0 2 0 2 0 0 2 0 2 0 0 0 o u u u u u u r k u u k k u u k k z z k k k k u u k k u u z u u z k u + + λ λ λ λ λ λ λ λ Ω Ω e e I e e e e e o V J u f V R EC R R r 0 ) (
I.4 Image-based isual seroing I.41 Indiret isual seroing J I depends on intrinsi parameters and on pose! Also depends on pose of amera w.r.t effetor J I must be estimated Ĵ I J I must be full rank for ontrolling all degrees of freedom (at least 3 points required) End-effetor with n dofs only n features omponents an be ontrolled in the image. Expressions of image jaobian for lassial geometrial objets : straight lines, spheres, ylinders et. in F. Chaumette, thesis 90 and Espiau, TRA 92 and for any planar objets using moments in Tahri O and Chaumette, F, IEEE TR 2005, 21(6) 34
I.4 Image-based isual seroing I.41 Indiret isual seroing D. Stability and robustness (1) Low speed : f k J I J r + I e f r Exponential onergene Behaes as a gradient desent optimization : may be stuk in loal minima! 2 nd order optimization Malis, E., Improing isionbased ontrol using effiient seond-order minimization tehniques, 2004 The norm of e f dereases, but some omponents may inrease during motion (possible loss of features) If more features than dofs, least square of error minimization no ontrol in the artesian spae better behaiour for small errors 35
I.4 Image-based isual seroing I.41 Indiret isual seroing D. Stability and robustness (2) Main soure of unertainty : Ji f + k J Jˆ e Exponential onergene if ˆ + J I J I > 0 Easily guaranteed : oarse alibration and oarse pose estimation are generally suffiient. No proof of global stability in funtion of k Error in the final position independent of errors on amera / effetor position independent of erros on intrinsi parameters of the amera if referene image has been learned funtion of pixel noise Pixel noise attenuation : pik more features smaller Problem of reahing the workspae limits Hybrid isual seroing Malis, E., 21/2D isual seroing TRA 99 I δf I f 36
I.4 Image-based isual seroing I.42 Diret isual seroing A. Control law f d e f Control - law q or τ Low-leel joint ontrollers fˆ f + δf Features extration pixel noise T Robot+amera image f J r J J ( q ) q I I R Control law : q k J 1 + R ( q) J I e f ˆ 37
I.4 Image-based isual seroing I.42 Diret isual seroing B. Stability and robustness (1) Stability may be an issue: Low speed ision loop : q q f High speed ision loop : Linearized approah : 1 f J s I I J R R 1 R k J J ( q) J ( q) Jˆ ( q) F( s, q) q + I e f q ( s) Exponential onergene if F( s, q) q ( s) ˆ > 0 + J I J I 1 q J R ( q) r Control law: with using a LPV disrete-time model of the ision loop r omputed This approah works in pratie with 6DOF ision loops! 38
I.4 Image-based isual seroing I.42 Diret isual seroing B. Stability and robustness (2) LPV disrete-time model image-based identifiation e p + Ĵ I ontrol algorithm e f + - f d fˆ r δf + f J 1 R ( q) q aquisition and image proessing Tz d 1 z 1 f J I F(z, q) robot identified model J R (q) q GPC of a 6DOF robot ision loop : J. Gangloff & M. de Mathelin (Adaned Robotis, ol 17, no 10, dé. 2003) 39
I.4 Image-based isual seroing I.42 Diret isual seroing B. Stability and robustness (3) High speed ision loop : Non linear approah : rigid link robot manipulator model τ M ( q) q + C( q, q ) q + g( q) + fr ( q, q ) PD ontrol sheme : (R. Kelly, IEEE Trans. Rob. Aut., ol 12, 1996) T R τ g( q) K q J ( q) K Jˆ p + I e f Stability is proed only for a 2 DOF robot with no frition > Preious restritions apply 40
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 41
Case of planar objets Images linked by a homography u p Gp 1 u G 1 I.5 Visual seroing without feature extration g G g g with det(g) 1 Computation of G using SSD minimization Example : ESM (S. Benhimane and E. Malis 2004) Computation of eulidean 1 homography m K GKm Hm 11 21 31 g g g 12 22 32 g g g 13 23 33 42
Control law I SSD + r ESM I.5 Visual seroing without feature extration H Control law r Cartesian ontroller I I T Robot+amera image r λν 0 0 e λ ω e ν ω with e e e ν ω e ν ( H I) m and [ ] T and e ω H H m 1 K p Camera Camera oordinates of a seleted point in the referene image 43
Stability and robustness I.5 Visual seroing without feature extration S. Benhimane and E. Malis, Homography-based 2D isual seroing, IEEE ICRA 2006 No need of interation matrix omputation Proof of loal stability Pratially : good robustness w.r.t intrinsi parameters Large domain of stability Reently extended to non-planar objets 44
Bibliography Hager, G. and S. Huthinson. Speial issue on ision-based ontrol of robot manipulators. IEEE Trans. Rob. Autom., ol 12, no 5, 1996. Corke, P. Visual ontrol of robots. Researh studies Press Ltd., Somerset, UK, 1996. B. Espiau and F. Chaumette, A new approah to isual seroing in robotis, IEEE Transations on robotis and automation, ol 8, no 3, 1992, S. Benhimane and E. Malis, Homography-based 2D isual seroing, IEEE ICRA 2006 45
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 46
Visual seroing in medial robotis Main appliations In laparosopi surgery In ophtalmology For physiologial motion ompensation In US imaging diagnosis and interention M.A.Vitrani and G.Morel, ICRA 2005 : guidane of surgial instruments A. Krupa and F. Chaumette, IROS 2005 : ontrol of US probe 47
Laparosopi surgery Small inisions, long-shaft instruments Endosopi ision system Adantages : Shorter reoery time Lower infetion risk Smaller osts Diffiulties : Tiring gestures Indiret isual feedbak, lak of depth information Inerted motions limited to 4 DOF Poor hapti feedbak 48
Laparosopi surgery Classial roboti ontrol of the endosope AESOP (Computer Motion/Intuitie Surgial) EndoAssist (Armstrong-Healthare) Voie ontrolled Controlled with head motions 49
Laparosopi surgery Vision-based ontrol of the endosope R. Taylor (95), A. Casals (95), G. Wei and G. Hirzinger (97), S. Voros and Ph. Cinquin (2006), et. More ision guided systems than real isual seroing Centering the image on marked instruments Traking of instruments Centering the image on speifi physiologial markers 50
Laparosopi surgery Visual seroing based ontrol of the instrument (Krupa 2003) Goal : surgial instrument guidane Autonomous reoery of the instruments Bring the instrument in the field of iew Centering of the instrument in the image Automati positionning Put the instrument at a desired loation w.r.t to tissues Proide depth informations Motiations : - inrease safety - improe ergonomy - speed up surgial proedures 51
Laparosopi surgery Experimental setup Surgial robot amera optis 52 instrument and Laser pointing troart 1 Laser spots organ troart 2 External amera onfiguration Laser projetion to add information to the sene
Laparosopi surgery Experimental setup laser pointing deie 4 laser spots To be used with standard instruments (4 mm) and standard troarts (10 mm) 3 optial markers (detetion of the instrument) 53
Laparosopi surgery Kinematis in laparosopi surgery - onstraints: 4 DOF W ( ω ω ω ) op x y z z - AESOP: 6 DOF (2 passie joints) Τ ω x ω y q2 q3 2 3 4 θ z ω z d 1 q 1 1 0 q 4 q 5 5 7 6 q 6 kinemati model : W J ( q, d ) q op op 1 q J q ˆ W 1 op(, d1) op 54
Laparosopi surgery Visual features detetion Goal: to obtain the image oordinates of the enters of laser spots and optial markers refleting lights diffused laser pattern 55
Robust detetion Laparosopi surgery Blinking laser spots and optial markers synhronized with frame aquisition Een frames : laser on markers off Odd frames : markers on laser off Seletion by high-pass filtering laser marker ligne N : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 6 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 8 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 10 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Not a feature 56
Robust detetion Laparosopi surgery 57
Laparosopi surgery Control of the position of the spot 2D diret isual seroing sheme (Image-based) z s p ω + - Z ˆ B { B} k s p 1 J ω ω z J op ( q, d1) 1 q Robot ontroller AESOP 58 Xˆ B Yˆ B sp O Q - ontrolled ariable : S ( ) T p up p - ontrol input : ω ( ω ) T x ωy - ontrol law : p k ( p p) Features detetion S S S J ω ω image amera
Estimation of interation matrix Open loop estimation of Hypothesis : standstill amera T T Laparosopi surgery Stability if ω x y x st ω 0 ω 0 Jˆ J ω u ω11 ω x Jˆ Jˆ Jˆ ω11 ω12 ω Jˆ ˆ ω21 Jω22 J ω st ω12 ω 22 y J Jˆ 0 ω 1 ω > Jˆ p T u p ω T y Jˆ Jˆ ω 21 ω x y p T p ω T 59
Laparosopi surgery 60
Laparosopi surgery Autonomous omplex gestures (F. Nageotte 2005) Automati stithing in laparosopi surgery Trajetory planning in the amera frame Trajetory following using the endosopi amera Aoid registration proedure between sensors 61
Laparosopi surgery Autonomous omplex gestures (F. Nageotte 2005) 2D isual seroing : Marked instrument Referene images : projetion of the needle-holder in the endosopi image fine amera alibration required Controlled ariables : marker points + ylinder (10 to 16) Control input : 4 DOFs of end-effetor 62
Laparosopi surgery Autonomous omplex gestures (F. Nageotte 2005) Complex interation matrix with arying size. Hypothesis : small traking error using known referene pose required matrix omputed no pose estimation Predition of feature appearane and disappearane for update 63
Laparosopi surgery Automati suturing 64
Oeriew Part I : Fundamentals of isual seroing Bakground and definitions Seroing arhitetures and lassifiation Position-based isual seroing Image-based isual seroing Visual seroing without feature extration Part II : Medial robotis appliations Laparosopi surgery Internal organ motion traking 65
Physiologial motions ompensation Objeties Filter Stabilized image Visual feedbak Controller 66
Physiologial motions ompensation Breathing motion ompensation (R. Ginhoux 2003) Preditie ontrol law : take into aount periodi disturbane GPC ontrol law with periodi output perturbation model Dynami model of the robot is required : linearized model obtained by pseudo-random exitation Controlled ariable : distane to the organ Control input : first axis of the robot (partiular onfiguration) Identifiation of the interation gain 67
Physiologial motions ompensation In io experiment 68
Physiologial motions ompensation Beating heart motion ompensation (R. Ginhoux 2003) Fast robot Fast amera (500hz) In io experiments 2 types of motion: - slow (~0.25 Hz) - fast (~1.5 Hz) 69
Physiologial motions ompensation Beating heart motion ompensation Heart motion Respiratory omponent Adaptie filtering of 12 harmonis of the heart beat rate 70
Physiologial motions ompensation Beating heart motion ompensation Predition of the perturbation referene orretion GPC ontrol law 3 ontrolled ariables : distane to the organ in the image z, spot position on the organ surfae (x,y) Control input : 3 axes of the robot (q1, q2, q3) Linearized dynami model of the axes of the robot onsidered as dynamially deoupled Hypothesis : interation matrix partly deoupled x 0 y 0 z L 31 L L 12 22 0 L13 q 1 L 23 q 2 0 q 3 Estimated around working position in open loop 71
In io experiment Physiologial motions ompensation 72
Physiologial motions ompensation Breathing motion ompensation with a flexible endosope Flexible endosopes : gastrosopy, olosopy, transluminal operations Hard to manually ontrol, orientation and translation oordination ery omplex Baklashes, dead zones Autonomous following of the organs 73
Physiologial motions ompensation Breathing motion ompensation with a flexible endosope 74
Physiologial motions ompensation Breathing motion ompensation with a flexible endosope Controlled ariables : position of an interest area in the image (u, ) Control input : q 1 and q 2 u L11 L12 q 1 Interation matrix estimated beforehand q Repetitie ontrol law rejetion of breathing disturbane and model errors after some disturbane periods L 21 L 22 2 75
Physiologial motions ompensation Breathing motion ompensation with a flexible endosope 76
Bibliography A. Krupa. Visual seroing in laparosopi surgery. Ph.D. thesis, Uniersity Louis Pasteur, Strasbourg I, june 2003 R. Ginhoux. Compensation des mouements physiologiques en hirurgie robotisée par ommande préditie. Ph.D. Thesis, Strasbourg Uniersity, deember 2003 F. Nageotte. Contributions à la suture assistée par ordinateur en hirurgie mini-inasie. Ph.D. Thesis, Strasbourg Uniersity, noember 2005 A. Krupa, J. Gangloff, C. Doignon, M. de Mathelin, G. Morel, J. Leroy, L. Soler, J. Maresaux. Autonomous 3D positioning of surgial instruments in robotized laparosopi surgery using isual seroing. IEEE Transations on Robotis, ol 19, no 5, pages 842-853, 2003 R. Ginhoux, J. Gangloff, M. de Mathelin, L. Soler, M. Arenas Sanhez and J. Maresaux. Atie Filtering of Physiologial Motion in Robotized Surgery Using Preditie Control. IEEE Transations on Robotis, ol 21, no 1, pages 67-79, 2005 L. Ott, Ph. Zanne, F. Nageotte and M. De Mathelin, Problemati of transluminal operations, Surgetia 2007 F. Nageotte, Ph. Zanne, M. De Mathelin and Ch. Doignon, Visual seroing based traking for endosopi path following, IROS 2006 77
Perspeties New sensors (isual seroing under CT san?) Seroing without features extration on deformable objets High speed isual seroing with high resolution 78