Course 11. Tracking: Beyond 15 Minutes of Thought

Transcription

1 Course 11 Tracking: Beyond 15 Minutes of Thought

2 Introductions Danette Allen Greg Welch Gary Bishop

3 Why thinking about tracking is so fun It s s a simple problem to state It has a little of everything A little physics A little electronics A little math A little signal processing A little programming

4 Why don t you just Mount TV cameras on the walls? Use GPS? Use MEMS accelerometers? Use carbon nanotubes?

5 The 15 minute effect... Confidence Time

6 Goals To get you to the 15 minute point and beyond... To equip you to evaluate the various offerings and understand the strengths and weaknesses of each.

7 Tracking Technologies (Danette Allen)

8 Many ways to slice! Configuration Outside-in vs. Inside-out Type of measurement Absolute vs. Relative Range vs. angle Physical medium Physical medium Five categories

9 Five (Six) Technologies Inertial Acoustic Magnetic Mechanical Optical Radio (GPS) is the sixth... typically used outdoors not addressed in this course

10 Inertial Tracking Passive Newton s s 2nd Law of Motion F = ma τ = Iα (linear) (rotational) No physical limits on working volume Accelerometers and gyroscopes Derivative measurements

11 Inertial Tracking Accelerometers Measure force exerted on a mass since we cannot measure acceleration directly. Proof-mass and damped spring Displacement proportional to acceleration m Potentiometric and piezoelectric transducers

12 Inertial Tracking Gyroscopes Inertia rigidity in space Precession a comparatively slow gyration of the rotation axis of a spinning body about another line intersecting it so as to describe a cone (Mirriam-Webster) Gimbal deflection (Discovery, 2001)

13 Microgyroscope MEMS Tuning fork Ring-laser and Fiber Doppler effect Beat frequency (Systron Donner, 2001) Axis of rotation

14 Inertial Drift Error accumulation due to integration Poor SNR at low frequencies Inverse square weighting of noise () s ε User a && x && xˆ 1? s xˆ & 1 s xˆ LaPlace Transform s = σ + jω Gravity vector misalignment 1º tilt error over 10 seconds 9m position error Periodic recalibration hybrid systems typical

15 Time [s] to 0.1 [m] Error Effect of Sensor Noise on Inertial-Only Performance (time to 0.1 meters error) Accelerometer dynamic range (3G divided by noise) bits Rate gyro dynamic range (5rad/s divided by noise) bits

16 Acoustic Tracking The Geometry The intersection of 2 spheres is a circle. The intersection of 3 spheres is 2 points. One of the two points easily eliminated Speed of Sound Varies with temperature and pressure ~ 331[m/s] in air at 0º C Ultrasonic 1 ft/ms SLOW!! 40 [khz] typical

17 Acoustic Tracking Methods Time of Flight Measures the time required for a sonic pulse or pattern to travel from a transmitter to a receiver. d [m] = v [m/s] * t [s], v = speed of sound (c) Absolute range measurement Phase Coherence Measures phase difference between transmitted and received sound waves Relative to previous measurement still absolute!!

18 Phase Coherence Equations A cos(ωt - φ) c[m/s] = λ[m] * f[1/s] δ[m] = λ[m] * (φ/2π) Relative Result Fractional wavelength Need previous range estimate No integration!!!

19 Magnetic Tracking Three mutually-orthogonal coils M M H = cosθ H = r θ 2πd 3 2πd 3 Each transmitter coil activated serially AC vs.. DC Three measurements apiece (three receiver coils) Nine-element measurement for 6D position Ferromagnetic interference sinθ

20 Mechanical Tracking Ground-based or body-based Used primarily for motion capture Provide angle and range measurements gears bend sensors Elegant addition of force feedback

21 Optical Tracking Provides angle measurements One 2D point defines a ray Two 2D points define a point for 3D position Additional 2D points required for orientation Speed of light * 10 8 m/s (1 ft/ns) C θ, φ C

22 Active vs. Passive Targets Typical detectors Video and CCD cameras Computer vision techniques CCD cell/pixel Passive targets Reflective materials, high contrast patterns

23 Active vs. Passive Targets Active vs. Passive Targets Typical detectors Typical detectors LEPDs Quad Cells Active targets Active targets LEDs ( ) ( ) ( ) ( ) i i i i i i i i y i i i i i i i i x = = ( ) ( ) = = L x L I I or L x L I I 0 ) sinh ) ( sinh 0 α α x I L I

24 Many ways to slice! Configuration Outside-in vs. Inside-out Type of measurement Absolute vs. Relative next Range vs. angle Physical medium Five categories

25 Source/Sensor Configurations (Gary Bishop)

26 Sensor Configurations Geometric arrangement of sensors and sources impacts: accuracy usability algorithms

27 for example CODA mpx D CCDs are stationary LED targets move Very interesting optics and sensing

28 CODA mpx30 Measures angles in lab coordinate frame Angle determines a plane Intersecting 3 planes determines a point Flatland

29 CODA mpx30 Such outside-looking-in systems measure position very well allow many small moving targets use multiple targets to get orientation trade off accuracy and working volume provide larger volume / more accuracy with more sensors use really simple math

30 HiBall 6 2-D sensors and 6 lenses in dodecahedron 1000 s s of LEDs fixed on the ceiling Calibration gives effectively 26 cameras

31 HiBall Measures angles in user coordinate frame Angles determine a constraint relating position orientation view led location Flatland

32 HiBall Such inside-looking out systems directly measure orientation allow large working volume with accuracy are larger than LED targets and thus harder to use for hands, feet, etc.

33 Arc Second Vulcan Sensors Sources Sources scan planes of light through space Sensors on target detect passing plane

34 Arc Second Vulcan Time of plane passing converts to angle at the source Measures angles in world frame Thus like CODA and other outside- looking-in systems Direction of looking really isn t t the issue but coordinate frame of measurement

35 User and Sensor Uncertainty/Information (Greg Welch)

36 Pose Uncertainty Measurement uncertainty Pose estimates from noisy sensor measurements User pose uncertainty Noisy and temporally-discrete measurements Modeling user motion is difficult [Weber] Modeling pose uncertainty is less difficult

37 Noise-Driven Processes Random noise Velocity uncertainty Position uncertainty User True sensor measurement Random noise Σ Noisy measurement Tracker

38 Random Variables and Signals Map sample space real numbers For example, time to voltage Random Signals For example, electrical signals Continuous random variables Probability over a region of sample space Spatial (statistical) and temporal (spectral) aspects

39 Cumulative Distribution Function (, ] F ( x)= P x X 1. F ( x) 0 as x X 2. F ( x) 1 as x + X 3. F x is a non - decreasing function of x X ( )

40 Probability Density Function f x d ( x)= dx F ( X x ) 1. f x is a non - negative function 2. X ( ) f ( x) dx= 1 X

41 Probability (Continuous) P a, b f x dx x b [ ]= ( ) a x

42 Statistical Moments µ = EX [ m ]= m Continuous: Discrete: x m f ( x) dx x m p x X X x ( )

43 1st Moment or Mean µ = EX [ ]= Continuous: Discrete: xf ( x) dx xp x X X x ( )

44 Central Moments m c = E[ ( X µ ) ]= m Continuous: ( m x µ ) f ( x) dx X Discrete: µ x m ( x ) p ( x) X

45 2nd Central Moment or Variance [( ) 2 µ ] V[ X]= E X 2 2 = EX [ ] µ Mean of square minus square of mean

46 Standard Deviation σ = V[ X]

47 Gaussian/Normal Distribution 1 f ( x)= e X 2πσ 1 ( x 2 ) 2 µ σ 2 X ~ N ( µσ, 2 ) f ( x) X σ µ x +

48 Autocorrelation (Time Domain) ( )= [ () ( + )] R τ E X t Xt τ X R X () τ X 2 X 1 τ 0 τ

49 Spectral Density (Frequency Domain) The Wiener-Khinchine relationship S ( jω)= F R ( τ) X [ ] X = ( ) R τ e dτ X jωτ

50 White Noise Process R X ( τ)= if τ = 0 then else 0 A S ( jω)= A X R X () τ S X () jω τ 0 τ 0 ω

51 Growth in Pose Uncertainty [ ]= 0 V X dt w where w ~ N0 (, q) and white.

52 Control of Pose Uncertainty Measurements pose information

53 Sensor Measurements

54 Break (15 Minutes)

55 Traditional Approaches (Gary Bishop)

56 Traditional Solution Methods Simple problem: Determine pose given sensor readings. Linear algebra taught us about N equations in N unknowns Each equation is a constraint 3 DOF 3 3 constraints & 6 DOF 6 constraints Unfortunately often non-linear constraints often with multiple solutions

57 Range Tracker Intersect 3 spheres Unfortunately, the solution is ugly...

58 Simplify a. put mike 0 at origin b. put mike 1 out X axis 1 unit c. put mike 2 out Y axis 1 unit d. all 3 mikes in Z=0 plane Usual coordinate transform to convert to lab coordinates

59 Simpler Range Equations Note ambiguity

60 Optical with fixed 1D sensors For example, 1D CCD with razor blade casting a shadow Calibrate to determine 3D plane equation from sensor reading (non-trivial) For each sensor reading, write a linear equation relating unknown x,y,z to plane Solve the system of equations for x,y,z

61 Solve...

62 Optical with fixed 2D sensors For example, two video cameras looking at LEDs on the user. Could treat as four 1-D sensors OR Calibrate to get ray equation from u,v Rays won t t intersect! Minimize distance between them

63 Set up equations Ray equations Distance Minimum distance line must be perpendicular to both rays, so...

64 Solve Distance out each ray to closest point Halfway between B is the baseline

65 Stochastic Approaches (Greg Welch)

66 Motivation Take into account Stochastic nature of sensor signals Varying amounts of sensor information Model of user motion Combine sensor/measurement information Combat (otherwise growing) pose uncertainty Fuse information from heterogeneous sensors

67 State-Space Models State-Space Models Begin with difference equation for process y a y a y u k k k n k k n k + + = ,, K Re-write as x y y y y a a a a y y y y k k k k k n n n k k k k n = M L L M O M M u

68 State-Space Models State-Space Models x Ax Gu y Hx k k k k k + = + = 1 A x i G x y y y y a a a a y y y y k k k k k n n n k k k k n = M L L M O M M u

69 Observer Design Problem x = Ax + Gu k k 1 k 1 z = Hx + v k k k Measurement noise Process noise

70 Optimal Estimation c T cost ( a t, b t, t) dt 0 = () () Integral of Absolute Value of Error (IAE) Integral of Square of Error (ISE) cost = a b cost = ( a b) 2

71 The Kalman Filter R.E. Kalman, 1960 Recursive optimal estimator Minimum variance of error Versatile & robust predict correct Estimation Sensor fusion Robotics, navigation, computer vision, economics, Java-Based Learning Tool, books, papers, etc. ACM SIGGRAPH 2001 tutorial (earlier today)

72 PREDICT x k k = Axk 1 P = AP A + Q k 1 transition uncertainty T

73 CORRECT ( ) k k k k x = x + K z Hx P = I actual KH P k ( ) k predicted ( ) T T 1 k k K = P H HP H + R denominator (measurement space)

74 Hybrid Systems and Multi-Sensor Fusion magnetic coil accelerometer accelerometer gyroscope gyroscope gyroscope magnetic coil magnetic coil accelerometer

75 Incremental Estimation A Single Constraint at a Time C = constraints needed for a unique solution M = constraints used per estimate update Over-constrained Unique Solution Under-constrained SCAAT M > C M = C 1 < M < C M = 1

76 Inter-Estimate Summary Multiple Constraints SCAAT Measurements (constraints) Time

77 Benefits of SCAAT Approach Purposefully using minimal constraints Avoid simultaneity assumption Temporal improvements Simplicity and flexibility Online source/sensor autocalibration Can be applied to virtually any tracking system Measurement model for each type of sensor Dynamic model for user motion (possibly trivial)

78 Bishop, 1982

79 Error Sources (Greg Welch)

80 Error in Head Pose Hard to fool mother nature Lifetime of visual experience and expectations Visual-proprioceptive conflicts Virtual-real misregistration What to do? Some amount of error is unavoidable Understand sources and seek to minimize

81 Error Classification Pose estimate life cycle Noisy sensor measurement Estimate Transport Transform Display Two primary classes of error Static (spatial) Delay-induced (temporal)

82 Static Measurement Error Static field distortion Repeatable error in the measurement data Bias that might be corrected via calibration Random noise or jitter Non-repeatable error Random (electrical) noise such as described earlier Often dependent on the current pose

83 Pose-Dependent Noise (Example) Baseline noise ξ 0 and coefficients a, b, and c were determined off line.

84 Delay-Induced Error Measurement validity Good at sample time, then old (aging) Finite, non-zero sample time Old sample misregistration Motion predicition Measure where you are, but want where you will be Later w/ Bishop

85 The Simultaneity Assumption Measure Compute User Motion Time Moderate arm & wrist translation 1/2 [s] 3 [m/s] [ms] 1-10 [cm] Moderate head rotation 1/2 [s] 180 [ /s] [ms] 6-25 [cm] (at arm s length)

86 First-Order Dynamic Error ε dyn,θ ε dyn, x = θ t = x t Instantaneous velocities Tracker + graphics pipeline latency

87 Synchronization Delay A.k.a. phase delay or rendezvous delay

88 Pose Estimate Timeline

89 Total Tracker Latency Estimate the pose Network transport Sample Estimate the sensor(s) cycle Server bufferclient buffer synchronization synchronization synchronization

90 Total Tracker Error Static error Simultaneity Total assumption tracker Remaining latency latency

91 Closing (Error Sources) Did I mention error magnification? Consider the technology Understand its limitations Stay within the envelope Prediction (next)

92 Motion Prediction (Gary Bishop)

93 Motion Prediction End-to-end delay hurts in VR / hurts worse in AR sources time to measure pose delay in communicating pose application response to change graphics update display refresh

94 What to do about delay? 1. Monitor 2. Minimize 3. Mitigate Latency is not only a tracker problem. But mitigation is best handled at the tracker.

95 Can prediction help? Blue no prediction Red w/out inertial Green w/ inertial

96 Can prediction help?

97 Limits to prediction Prediction error grows quadratically with motion bandwidth and prediction interval

98 Prediction ideas Extrapolate past behavior to the future The more history the better Correlations in the users coordinate frame Inertial sensors help Monitor predicted actual for tuning Use image shifting to reduce jitter

99 Conclusions (Gary Bishop)

100 Final Thoughts No silver bullet Tracking anywhere for any purpose is a dream No free lunch, only tradeoffs Energy / Accuracy / Bandwidth / Latency / Noise No end in sight Lots of possibilities for interesting work ReActor not based on any of the principles described here

101 Resources cs.unc.edu/~ /~welch/kalman check out Course 8 notes Dozens of books on KF, here are a few Optimal Estimation with an... by Lewis Introduction to Random Signals... by Brown Kalman Filtering Theory and Practice by Grewal Beginnings of a tracking bibliography

102 Exhibits we re going to check out 3 rd Tech (cool demo) 5DT Ascension (ReActor is a new method) InterSense Measurand MetaMotion / PhoeniX Technologies / Vicon Polhemus

103 End