Imaging In Challenging Weather Conditions

Size: px

Start display at page:

Download "Imaging In Challenging Weather Conditions"

Lucinda Hardy
5 years ago
Views:

1 Imaging In Challenging Weather Conditions Guy Satat Computational Imaging for Self-Driving CVPR 2018

4 Imaging Through Fog == Imaging Through Scattering?

6 Why not RADAR? Visible X rays UV IR Microwave Radio Waves Wavelength Resolution Optical contrast

7 Different Strategies to Drive in Fog Fog Sensor Detects Fog Stop the Car! Fog Sensor Detects Fog Level i Drive with Foggy Algorithm i Fog Invariant Computational Imaging System Drive Normal As if the Fog were Not There

9 Information Carried by Light The plenoptic function: I(r, λ, t, θ, P, n, Φ) Irradiance Position Wavelength Time Angle Polarization Bounce Phase

10 Information Carried by Light The plenoptic function: I(r, λ, t, θ, P, n, Φ) Irradiance Position Wavelength Time Angle Polarization Bounce Phase

11 Scattering Types Sparse Scattering Milky glass Paper Lensless imaging ν ν x, ν, t x, ν, t Volumetric Scattering Tissue Fog x, ν, t x, ν, t

12 Lessons learned from seeing into the body

13 Phase Conjugation

14 Phase Conjugation SLM Phase conjugation Long iterative process Requires Guide Star

15 Diffuse Optical Tomography Constrained Imaging Geometry Photo credit: Wikipedia

16 Descattering with Photon Gating Angle Time Polarization Coherence Not enough photons Doesn t reject all scattered light No computation

17 Constrained Imaging Geometry Continuum of possible densities Patchy (heterogeneous) Moving platform

18 Towards Photography Through Realistic Fog Guy Satat, Matthew Tancik, Ramesh Raskar ICCP 2018

20 Dense, Dynamic, Heterogeneous

21 Key Idea Observation: Photons reflected from fog and those reflected from target obey different statistics Solution: A probabilistic technique to reject the backreflected photons

22 Fog Generator Power Meter SPAD Camera IR flashlight Regular Camera Diffused Pulsed Laser

23 Optical Thickness: OT t = log P 0 P t

24 Pixel wise Model d 5 d 6 d 7 d 8 d 4 T d 3 d2 d 1 k T = 1 c i=1 d i

25 Pixel wise Model τ 5 τ 6 τ 7 τ 8 τ 4 T τ 3 τ2 τ 1 k T = 1 c i=1 d i = k i=1 τ i

26 Pixel wise Model τ 5 τ 6 τ 7 τ 8 τ 4 T τ 3 τ2 τ 1 k T = 1 c i=1 d i = k i=1 τ i

27 Photon Classes Background Signal Dark Count

28 Fog Model τ 8 k T = i=1 τ i τ 4 τ 5 τ 6 τ 7 τ 3 τ2 τ 1 t τ i Exp{μ s } 1/μ s - mean time between scattering events T Gamma μ s, k f T t B = μ s k Γ(k) tk 1 exp μ s t

29 Fog Model T = Στ i τ i Exp{μ s } T Gamma μ s, k f T t B = μ s k Γ(k) tk 1 exp μ s t

30 Signal Model Another Gamma? f T t S = 1 2 exp t μ 2 2πσ σ

31 Measurement Model f T t = P B f T t B + P S f T t S Probability to measure a photon at time t Probability to measure a background photon Gamma distribution Probability to measure a signal photon Normal distribution Encodes the target depth and reflectance

32 Model Estimation f T t = P B f T t B + P S f T t S Fog Signal

33 Model Estimation f T t = P B f T t B + P S f T t S Fog Signal Time Profile Background Signal Target

34 Time Profile Estimation f T t = P B f T t B + P S f T t S Fog Signal KDE KDE (Kernel Density Estimator): Works well with a few sampling points

35 Background Estimation f T t = P B f T t B + P S f T t S Fog Signal Gamma fit

36 Signal Estimation f T t = P B f T t B + P S f T t S Fog Signal

37 Signal Estimation f T t = P B f T t B + P S f T t S Fog Signal

38 Signal Estimation f T t = P B f T t B + P S f T t S Fog Signal P(S), P(B) = argmin P(S),P(B) t P B f T t B + P S f T t S f T t 2

39 Fog Density Target Distance

40 f T t = P B f T t B + P S f T t S Target Recovery P S f T t S = P S 1 2 exp t μ 2 2πσ σ Reflectance Depth

43 Reflectance Recovery Error

44 Depth Recovery Error

46 How Many Photons?

47 Limitations Ignores spatial nature of scattering Impose priors Deblur

48 Limitations Ignores spatial nature of scattering Photon efficiency Current hardware efficiency is ~1: 10 6 Algorithm efficiency could improve

49 Limitations Ignores spatial nature of scattering Photon efficiency Acquisition time New frame every 100μs Currently use constant window of 20K frames 2s Dynamic window based of fog estimate

50 Limitations Ignores spatial nature of scattering Photon efficiency Acquisition time Scale Optical thickness is unitless Larger scenes relaxed requirement for time resolution More dependency on spatial scattering?

51 Object Classification through Scattering Media with Deep Learning Guy Satat, Matthew Tancik, Otkrist Gupta, Barmak Heshmat, Ramesh Raskar Optics Express (2017)

53 We Have to Calibrate

54 Why Deep Learning? Can learn invariants

55 Learning Invariant to Calibration Parameters Randomly sample scene properties Compute Forward Model Train CNN

56 Train on Synthetic Data Test on Lab Measurement

57 No graduate students were harmed in calibrating the system

Summary media.mit.edu/~guysatat guysatat@mit.

58 Summary media.mit.edu/~guysatat Variety of weather conditions Imaging through fog Imaging through scattering Wide range of fog conditions: Dense, dynamic, heterogeneous Calibration free No raster scan Probabilistic Computational Imaging Data Driven Computational Imaging

INTRODUCTION TO MICROWAVE REMOTE SENSING. Dr. A. Bhattacharya

1 INTRODUCTION TO MICROWAVE REMOTE SENSING Dr. A. Bhattacharya Why Microwaves? More difficult than with optical imaging because the technology is more complicated and the image data recorded is more varied.