Surgery Simulation. Olivier Clatz. Hervé Delingette. Asclepios Research project INRIA Sophia Antipolis, France

Transcription

1 1 Surgery Simulation Olivier Clatz Hervé Delingette Asclepios Research project INRIA Sophia Antipolis, France

2 Motivations of surgery simulation 2 Increasing complexity of therapy and especially surgery Increasing need for training surgeons and residents Medical malpractice has become socially and economically unacceptable Increasing need for objective evaluation of surgeons (see Cordis Nitanol endovascular carotid stent) Natural extension of surgery planning

3 3 Need for Training Hand-eye Synchronisation Camera being manipulated by an assistant Long instruments going through a fixed point in the abdomen

4 4 Current Training Techniques Mechanical Simulators Animals Procedure Duration (min) Average Duration of an hysterectomy with laparoscopy Cadavers Patients (source, Department of Obstetrics and Gynecology, Helsinki University Central Hospital) Total number of patients Resident Training (source, Ayudamos) Example of a mechanical simulator (source, US Surgical Corporation) Surgery Training on a pig at EITS

5 Goal for simulation : Training versus Rehearsal 5 Training: Modelling a standard patient for teaching classical or rare situations Rehearsal: Modelling a specific patient to plan and rehearse a delicate intervention, and evaluate consequences beforehand

6 Simulator Workflow 6 Position Collision Contact Deformation Force Force

7 7 Different Technical Issues Mesh Reconstruction from Images Soft Tissue Modeling Tissue Cutting Collision Detection Contact Modeling Surface Rendering Haptic Feedback With realtime constraints

8 8 Different Technical Issues Mesh Reconstruction from Images Soft Tissue Modeling Tissue Cutting Collision Detection Contact Modeling Surface Rendering Haptic Feedback

9 Liver Reconstruction 9 Deformation from a reference model reconstructed from the «Visible Human Project»

10 10 Different Technical Issues Mesh Reconstruction from Images Soft Tissue Modeling Tissue Cutting Collision Detection Contact Modeling Surface Rendering Haptic Feedback

11 11 Soft Tissue Characterization Biomechanical behavior of biological tissue is very complex Most biological tissue is composed of several components : Fluids : water or blood Fibrous materials : muscle fiber, neuronal fibers, Membranes : interstitial tissue, Glisson capsule Parenchyma : liver or brain

12 Estimating material parameters 12 Complex for biological tissue : Heterogeneous and anisotropic materials Tissue behavior changes between in-vivo and in-vitro Effect of preconditioning Potential large variability across population Ethics clearance for performing experimental studies

13 13 Soft Tissue Characterization Different possible methods In vitro rheology In vivo rheology Elastometry Solving Inverse problems

14 Soft Tissue Characterization 14 In vitro rheology can be performed in a laboratory. Technique is mature Not realistic for soft tissue (perfusion, )

15 Soft Tissue Characterization 15 In vivo rheology can provide stress/strain relationships at several locations Influence of boundary conditions not well understood Source : Cimit, Boston USA

16 Soft Tissue Characterization 16 Elastometry (MR, Ultrasound) mesure property inside any organ non invasively validation? Only for linear elastic materials Source Echosens, Paris

17 Soft Tissue Characterization 17 Inverse Problems well-suited for surgery simulation (computational approach) requires geometry & BC before and after deformation

18 Soft Tissue Characterization 18 To characterize a tissue, its stressstrain relationship is studied eight h Radius r Stress σ = Strain ε = F πr 2 h*-h h Height h* Force F Radius r* Rest Position Cylindrical Piece of Tissue

19 Linear Elastic Material 19 Simplest Material behaviour Only valid for small deformations (less than 5%) Contrainte Déplacement

20 Biological Tissue 20 Far more complex phenomena arises σ σ σ loading V 0 V 1 V 2 V 3 Hysteresis Unloading ε Visco-elasticity ε Linear Domain Non-Linearity Slope =Young Modulus ε Anisotropy

21 Continuum Mechanics 21 Continuum Mechanics Fluid Mechanics Structural Mechanics Linear Elasticity (Anisotropic, heterogeneous) Hyperelasticity Elasticity Plasticity

22 Material Modeling: Basics of Continuum Mechanics 22 Deformation Function Displacement Function Rest Position Φ X U Ωa φ ( X ) R ( X ) = φ( X ) X 3 Ω Deformed Position X U(X)

23 23 Basics of Continuum Mechanics The local deformation is captured by the deformation gradient : = = X X X X X X X X X X F j i ij φ φ φ φ φ φ φ φ φ φ X F = φ Ω X φ(x) Rest Position Deformed Position F(X) is the local affine transformation that maps the neighborhood of X into the neighborhood of φ(x)

24 Basics of Continuum Mechanics 24 Distance between point may not be preserved Rest Position Ω Deformed Position X+dX X φ(x) φ(x+dx) Distance between deformed points ( ) 2 ( ) ( ) T ds X dx X dx ( T = φ + φ φ φ)dx Cauchy-Green Deformation tensor C 2 T = φ φ Measures the change of metric in the deformed body

25 Basics of Continuum Mechanics 25 Example : Rigid Body motion entails no deformation φ ( X ) = RX + T F ( X ) = φ ( X ) = R T Strain tensor captures the amount of deformation It is defined as the distance between C and the Identity matrix E = 1 2 C = R 1 2 R = Id ( T φ φ Id ) = ( C Id )

26 Strain Tensor 26 Diagonal Terms : ε i Capture the length variation along the 3 axis Off-Diagonal Terms :γ i Capture the shear effect along the 3 axis E = ε γ γ x xy xz γ ε γ y xy yz γ γ ε z xz yz

27 Linearized Strain Tensor 27 Use displacement rather than deformation E φ 1 = U 2 ( X ) = Id + U ( X ) ( T T + U + U U ) Assume small displacements E Lin = 2 1 ( T U + U )

28 Hyperelastic Energy 28 The energy required to deform a body is a function of the invariants of strain tensor E : Trace E = = I 1 Trace E*E= I 2 Determinant of E = I 3 Rest Position Ω Deformed Position ( φ ) (,, ) 1 2 W 3 = w I I I dx Ω Total Elastic Energy

29 Isotropic Energy ( λ, µ ) w (X ) : Lamé coefficients Linear Elasticity : density of elastic energy λ w X ) = Lin µ 2 ( ) 2 2 tr E + tr E ( Lin Hooke s Law 29 Advantage : Quadratic function of displacement λ 2 2 µ w = ( div U ) + µ U rot U 2 2 Drawback : Not invariant with respect to global rotation Extension for anisotropic materials 2

30 Shortcomings of linear elasticity Non valid for «large rotations and displacements» 30

31 St-Venant Kirchoff Elasticity 31 Isotropic Energy λ w ( X ) = 2 µ E ( λ, µ ) : Lamé coefficients Advantage : Generalize linear elasticity Invariant to global rotations Drawback : Poor behavior in compression Quartic function of displacement Extension for anisotropic materials ( ) 2 2 tr E + tr

32 St Venant Kirchoff vs Linear Elasticity 32 Linear St Venant Kirchoff Rest Position Linear St Venant Kirchoff

33 Other Hyperelastic Material Neo-Hookean Model Fung Isotropic Model w ( X ) = µ + I tre f ( ) 3 2 = µ ( ) w( X ) e tre + f I Fung Anisotropic Model Veronda-Westman Mooney-Rivlin : µ tre k 4 w( X ) = e + 2 k 1 2 k2 ( I 1) ( e ) + f ( I ) 1 1 ( γ tre ) 2 e + c tre + f ( ) w( X ) = c I w ( ) 2 ( X ) = c10tre + c01tre f I3

34 Discretisation techniques 34 Four main approaches : Volumetric Mesh Based Surface Mesh Based Meshless Particles

35 Different types of meshes Surface Elements : 35 Triangle 3, 12 nodes and more Quad 4, 8 nodes and more Volume Elements Tetrahedron 4, 10 nodes Prismatic 6, 15 nodes and more Hexahedron 8, 20 nodes and more

36 Structured vs Unstructured meshes Example 1 : Liver meshed with hexahedra 36 3 months work (courtesy of ESI) Example 2: Liver meshed with tetrahedra Automatically generated (10s)

37 Volumetric Mesh Discretization 37 Classical Approaches : Finite Element Method (weak form) Rayleigh Ritz Method (variational form) Finite Volume Method (conservation eq.) Finite Differences Method (strong form) FEM, RRM, FVM are equivalent when using linear elements

38 Rayleigh-Ritz Method Step1 : Choose Finite Element (e.g. linear tetrahedron) Mesh discrediting the domain of computation Hyperelastic Material with its parameters Boundary Conditions 38 Tetrahedron 4 nodes Fixed nodes 2 2 ( tr E ) + tr λ w ( X ) = µ E 2 Young Modulus Poisson Coefficient

39 Rayleigh-Ritz Method Step2 Write the elastic energy required to deform a single element 39 P 1 Q 1 u ( P ) = Q P = U i i i i P 4 WT i 1 P 3 P 2 = jk U t j Q 4 Q 3 [ ] T K jk U k i Q 2 T T ( λ M M + µ M M + ( M M )[ Id ]) Ti [ K jk ] = i k j i j k µ i j k 3x3 36.V( Ti ) u( X ) = tre 4 i= 1 λ ( X ) u( i P i M i λi ( X ) = 6V ( T ) = M i Ui 6V ( T ) i )

40 Rayleigh-Ritz Method 40 Step3 Sum to get the total elastic energy W ( ) ( ) T U w I, I I dx = W = U K U = 1 2, 3 Ω h Write the conservation of energy T i T i W ( U ) F T U + ρ ( X ) ( X g )dx Internal Energy = Ω Nodal Forces Gravity Potential Energy

41 Step3 Rayleigh-Ritz Method Write first variation of the energy : Linear Elasticity KU = R Static case 41 M U&& + C U& + KU = R (t ) Dynamic case HyperElasticity=NonLinear Elasticity K ( U ) = R M U&& + C U& + K = ( U ) R (t ) Static case Dynamic case

42 Surface-Based Methods 42 Only consider the mesh surface under some hypothesis : Linear Elastic Material (sometimes homogeneous) Only interact with organ surface Pros : No need to produce volumetric meshes Much faster than volumetric computation Cons : Only linear material No cutting

43 Dynamic evolution Discrete models = lumped mass particles submitted to forces Newtonian evolution (1 st order differential system): { δp= V.dt δv= M -1 F(P,V).dt Explicit schemes: { Euler: δp= V t.dt δv= M -1 F(P t, V t ).dt Evolution 43 Runge-Kutta: several evaluations to better extrapolate the new state [press92] Unstable for large time-step!! Semi-Implicit schemes: { { Euler: δp= V t+dt.dt δv= M -1 F(P t, V t ).dt P t+dt = 2P t P t-dt.+m -1 F(P t, V t ).dt2 V t+dt = (P t+dt P t.)dt-1

44 Evolution 44 Implicit schemes [terzopoulos87], [baraff98], [desbrun99], [volino01], [hauth01] First-order expansion of the force: F(P t+dt, V t+dt ) F(P t, V t ) + F/ PδP + F/ VδV Euler implicit { δp= V t+dt.dt H = I - M -1 F/ V dt - M -1 F/ P dt 2 with δv= H -1 Y Y = M -1 F(P t, V t ) + M -1 F/ P V t dt 2 Backward differential formulas (BDF) : Use of previous states Unconditionally stable for any time-step But requires the inversion of a large sparse system Choleski decomposition + relaxation Conjugate gradient Speed and accuracy can be improve through preconditioning (alteration of H)

45 Example of Soft Tissue Models 45 Pre-computed Elastic Model Tensor-Mass and Relaxation-based Model Non-Linear Tensor-Mass Model Computational Efficiency Cutting Simulation Large Displacements - - +

46 Precomputed linear elastic model Tetrahedra

47 47 Different Technical Issues Mesh Reconstruction from Images Soft Tissue Modeling Tissue Cutting Collision Detection Contact Modeling Surface Rendering Haptic Feedback

48 Different algorithms for cutting tetrahedral meshes 48 Split of tetrahedra [Bielser, 2000] [Mohr, 2000] [Nienhuys, 2001] + Accurate, realistic - Decrease of Mesh Quality Removing Tetrahedra [Forest, 2002] + Keeps a good mesh quality - Gross cut

49 49 Proposed Technique Remove Tetrahedra Refine Mesh before removing material

50 50 Dynamic Refinement Example of refine-before-cutting

51 Refinement by Edge Split 51 Decomposition of tetrahedra by edge split

52 Topological Singularities 52 Removing a tetrahedron may create a singularity (zero thickness at edge and vertices) (see [forest]) Singularity

53 Non-linear Tensor-Mass Models 53

54 54 Different Technical Issues Mesh Reconstruction from Images Soft Tissue Modeling Tissue Cutting Collision Detection Contact Modeling Surface Rendering Haptic Feedback

55 55 Previous Work A lot of research on Collision Detection Hierarchy of oriented bounding boxes: Gottshalk & al. - Obb-tree: A hierarchical structure for rapid interference detection - SIGGRAPH 96 public domain package RAPID Very efficient, but needs pre-computation

56 56 The Rendering Process Camera = viewing volume + projection Two steps: geometry & rasterization

57 Collision Detection and Rendering analogy 57 a tool collides the organ a part of the organ is inside the tool if we define a camera with a viewing volume that matches the tool geometry, the organ will be in the picture.

58 58 Different Technical Issues Mesh Reconstruction from Images Soft Tissue Modeling Tissue Cutting Collision Detection Contact Modeling Surface Rendering Haptic Feedback

59 Tool-Soft Tissue Interaction 59 Prevent penetration of tool inside the soft tissue Detect intersections Push explicitly mesh vertices outside the tool

60 First Approach [Picinbono, 2001] 60 2 different tools : tip and handle Compute average normal in the neighborhood of the contact Projection of vertices in this plane Handle Tip

61 Collision Processing 61 Contact with the tip of the instrument Projection on the plane defined by the tip of the instrument and the average normal of intersected triangles

62 Collision Processing 62 Contact with the handle of the tool Projection on the plane defined by the tool direction and the averaged normal direction

63 Collision Processing 63 Perform 2 detections simultaneously

64 Possible interactions 64 Slip on the surface

65 Limitations of this approach 65 Same normal vector for all triangles in the same neighborhood Leads to instabilities when handling a complex geometry

66 66 New approach Three steps Prevent vertices to collide with the tool axis Move vertices near the tip of the tool Move vertices outside the volume of the tool

67 Example 67

68 68 Different Technical Issues Mesh Reconstruction from Images Soft Tissue Modeling Tissue Cutting Collision Detection Contact Modeling Surface Rendering Haptic Feedback

69 69 Haptic Feedback Principle Give a realistic sense of contact with the soft tissue Motivation Increase realism Naturally limit the amplitude of hand motion Pitfalls Frequency update of haptics > 500 Hz Mouvement non contraint par le retour d effort Frequency update of deformable models 30 Hz

70 70 First approach [Picinbono, 2001] Force F Computed Forces Estimated Forces Deformable Model Frequency Update 20 Hz Force Computation Position Force Feedback Frequency Update 300 Hz Force Extrapolation Example of extrapolation t Unstable if complex geometry Difficult extrapolation for large hand motions P' P F p t F ( F ) n ( ) = Fn + n n 1 Pn Pn 1 t n t < t n+1

71 Local Model [Mendoza, 2001] [Balaniuk, 1999] [Mark, 1996] 71 Modèle local Deformable Model Position Retour d effort Update Frequency 20 Hz Computation of a local model Update Frequency 300 Hz Force Computation from a local model Smooth Transition from one local model to the next

72 72 Computing the local model Described as a set of planes One model for the tip One model for the handle

73 73 Force Computation Proportional to the penetration of the tool tip in the planes described by the local model F = k.( EndP O P n r ). P

74 Tensor-Mass Models (low resolution) 74 N = 1394 (6342 Tétraèdres)

75 Simulation of surgical gestures 75 Gliding Gripping Cutting (pliers) Cutting (US)

76 SOFA : 76

77 SOFA : 77

78 Thank you 78