Proton Therapy: 3D Reconstructions from 2D Images Ben Herbst Department of Applied Mathematics University of Stellenbosch email: herbst@sun.ac.za Colorado School of Mines 16 September 2005
2 Collaborators Neil Muller Barry Sherlock Karin Hunter Brian O Kennedy Chris Venter Lourenco Magaia Francois Malan Pieter Rautenbach
Overview Proton Therapy and Patient Positioning Stereo Reconstruction 3D Reconstruction from video sequences 3D Reconstruction, single image
4 Patient Positioning Nuclear Accelerator
Physics Protons from hydrogen atoms, separated from water: 3 10 10 Boosted to 20MEV 5 million trips around the accelerator ring (600,000,000rpm) 1 speed of light 2 In live tissue: destroys DNA through ionization Bragg peak implies almost perfect Localization
6 Proton Therapy Bragg Peak Robert R Wilson 1946
First Patients: 1990 20 facilities world wide 3 in USA 1 facility near Stellenbosch Background Information
8 Aligning Proton Beam Many 6000 pound magnets that consume 3000 amperes each Precisely aligned by three-story high 200 ton gantries Aim the beam within a fraction of a millimeter Accelerator pumps a million watts in and out of power plant Deliver 1 5 watt, for 1 3 s Need a cheaper solution
Patient Positioning Position the Patient
Patient Positioning 10 (a) Mask (b) Reflective markers Patient Mask
CT Calibration (c) CT Scanner (d) CT Scanner CT Scanner
Patient Positioning 12 (e) Vault (f) Camera Treatment Vault: In development
Perspective Cameras Need to know: geometry - camera center, relative orientation corresponding points lines in 3D do not intersect
2D Projective Geometry 14 Equation for a line: For line becomes x = ax + by + c = 0 x y 1, l = x T l = 0 x and l are homogeneous vectors: identify all cx, cl all real c 0 x, l P 2 R 2 P 2 : P 2 R 2 : [ x y x y z ] a b c x y 1 [ x/z y/z, ]
Intersection of Lines The intersection of l 1 and l 2 : x = l 1 l 2 The line through x 1 and x 2 : l = x 1 x 2 Example l 1 : x 1 = 0 l 2 : x 2 = 0 Intersection x = 1 0 1 1 0 2 = 0 1 0 Two parallel lines intersect in well-defined point
Projective Transformation 16 H : P 2 P 2 preserves straight lines H is a 3 3 nonsingular, homogeneous matrix (g) Transformed image (h) Corrected image
Perspective Camera transformation Project X in object (world coordinates), to x on CCD (pixel coordinates) where x = PX P = KR [ I C ] P is 3 4, homogeneous, rank 3 P C = 0 ( C camera center in world coordinates)
Camera Calibration 18 (i) Lego Calibration Object (j) Real Calibration Pattern Calibration Object
Triangulation For x = PX and P C = 0 X = P + x + λ C For stereo camera pair, this gives X. Find corresponding points from epipolar lines: x T Fx = 0 F is fundamental matrix: calculated from camera matrices F = e P P +
Lines do not intersect in 3D 20 Minimize d 2 + d 2 subject to x T Fx = 0 Linearize SVD or Lagrangian multipliers
3D Reconstruction from a video sequence
Shackleton (Endurance 1914 1916) 22 Frank Worseley(navigator): On the thirteenth day we were getting nearer to our destination. If we made the tragic mistake of passing it we could never retrace our way on account of the winds and the currents, it therefore became essential that I should get observations. But the morning was foggy, and if you cannot see the horizon it is impossible to measure the altitude of the sun to establish your position. Now, the nearer your eye is to the surface of the sea, the nearer is the horizon. So I adopted the expedient of kneeling on the stones in the bottom of the boat, and by this means succeeded in taking a rough observation. It would have been a bold assumption to say that it was a correct one; but is was the best I could do, and we had to trust it. Two observations are necessary, however, to fix your position, and my troubles were far from over; for at noon, when I wanted to observe our latitude, I found conditions equally difficult. The fog, which before had been on a level with us and therefore did not altogether obscure the sun, had now risen above us and was hovering between the sun and ourselves, so that all I could see was a dim blur. I measured to the centre of this ten times, using the mean of these observations as the sun s altitude.
Conditional Densities 0.25 0.2 p(x z) 0.15 0.1 0.05 0 5 0 z 5 Given noisy measurement z 0, variance σ 2 z 0 of position x: Conditional density: p(x z 0 ) = N (z 0, σ 2 z 0 ) Best estimate of x: ^x(1 0) = z 0 (estimate at t 1 given measurement z 0 at t 0 )
Conditional Densities 24 σ z1 σ z0 z 0 z 1 Given second noisy measurement z 1, variance σ 2 z 1 of x: Conditional density: p(x z 0, z 1 ) = N (x z, σ 2 z)
Conditional Densities σ σ z1 σ z0 z 0 µ z 1 Conditional density: p(x z 0, z 1 ) = N (x z, σ 2 z ) [ σ 2 z 1 x z = σ 2 z 0 + σ 2 z 1 1 = 1 + 1 σ 2 z 0 σ 2 z 1 σ 2 z ] z 0 + [ σ 2 z 0 σ 2 z 0 + σ 2 z 1 ] z 1 σ 2 z smaller than either σ2 z 0 or σ 2 z 1 less accurate estimates can be used to improve estimate
Dynamics 26 dx dt = u + w u: velocity w: noise, variance σ 2 w. Combine best knowledge of position at t 1 with dynamics to get best estimate at t 2 : x(2 1) = x(1 1) + (t 2 t 1 )u. variance σ 2 x(2 1) = σ 2 x(1 1) + (t 2 t 1 )σ 2 w. Disaster, unless measurement z 2 is obtained
Dynamics Given: x(2 1) = x(1 1) + (t 2 t 1 )u, with σ 2 x(2 1) = σ 2 x(1 1) + (t 2 t 1 )σ 2 w new measurement z 2, variance σ 2 z 2 Combine and x(2 2) = x(2 1) + K(t 2 )(z 2 x(2 1)), σ 2 x (2 2) = σ2 x (2 1) K(t 2)σ 2 x (2 1). with σ 2 K(t 2 ) = x(2 1) σ 2 x (2 1) + σ2 z 2 If measurement noise σ z2 is large, then K(t 2 ) is small little confidence in noisy measurement If σ z2 then K(t 2 ) 0 ignores measurement If dynamic system noise σ w is large, so is K(t 2 ) little confidence in dynamic system if σ 2 w then K(t 2) 1 and x(2 2) z 2 all information obtained from measurement
Kalman Filter 28 Dynamics: x k+1 = A k x k + w k Measurement: z k+1 = H k+1 x k+1 + ν k+1 Noise w k and ν k have normal distributions: N (0, Q k ) and N (0, R k ) Dynamic update Given x(k k) and covariance P(k k) at time t k prior to the measurement at t k+1 : (a) Predicted state at t k+1 : Covariance: (b) Predicted measurement: Covariance: x(k + 1 k) = A k x(k k) P(k + 1 k) = A k P(k k)a T k + Q k ẑ(k + 1 k) = H k x(k + 1 k) S(k + 1 k) = H k+1 P(k + 1 k)h T k+1 + R k+1
Kalman Filter Dynamics: x k+1 = A k x k + w k Measurement: z k+1 = H k+1 x k+1 + ν k+1 Noise w k and ν k have normal distributions: N (0, Q k ) and N (0, R k ) Measurement update Given x(k + 1 k) and P(k + 1 k), and measurement z k+1 at t k+1 : ẑ(k + 1 k) and S(k + 1 k) (a) Kalman gain: W(k + 1) = P(k + 1 k)h T k+1s(k + 1 k) 1 (b) State update: x(k + 1 k + 1) = x(k + 1 k) + W(k + 1)(z k+1 ẑ(k + 1 k)) Covariance: P(k + 1 k + 1) = P(k + 1 k) W(k + 1)S(k + 1 k)w(k + 1) T.
Application 30 State variables: Structure: X j, j = 1,..., N (time independent) Motion: Rotation R and translation t Dynamic update: x k+1 = x k + w k Measurement: (Nonlinear) Error covariances: update, measurement Need nonlinear extension of Kalman, EKF, UKF, etc