Machine Learning and Rendering

Transcription

1 East Building, Balloom BC nvidia.com/siggaph2018 Machine Leaning and Rendeing Alex Kelle, Diecto of Reseach

2 Machine Leaning and Rendeing Couse web page at 14:00 Fom Machine Leaning to Gaphics and back Alexande Kelle, NVIDIA 14:40 Robust & Efficient Light Tanspot by Machine Leaning Jaoslav Křivánek, Chales Univesity, Pague 15:15 Deep Leaning fo Light Tanspot Simulation Jan Novàk, Disney Reseach 16:05 Neual Realtime Rendeing in Image Space Anton Kaplanyan, Facebook Reality Labs 16:40 Deep Realtime Rendeing Maco Salvi, NVIDIA 2

3 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw S 2 (x) P Camea 3

4 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw S 2 (x) Z cosq y = V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy P Camea 3

5 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw P Camea S 2 (x) Z cosq y = V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy Z Z = V (x 0,y)d x (x 0 )L i (x 0,w)f (w,x 0 cosq y,w)cosq x 0 V V x 0 y 2 dx0 dy L 3

6 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw P Camea S 2 (x) Z cosq y = V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy Z Z = V (x 0 c,y) lim B (x x 0 ) V V (x)!0 p(x) 2 L i (x 0,w)f (w,x 0 cosq y,w)cosq x 0 x 0 y 2 dx0 dy L 3

7 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw P Camea S 2 (x) = Z cosq y V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy = Z Z lim V (x 0,y) c B(x x 0 ) (x)!0 V V p(x) 2 L i (x 0,w)f (w,x 0 cosq,w)cosq x 0 y x 0 y 2 dx0 dy L 3

8 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw P Camea S 2 (x) Z cosq y = V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy Z Z c = lim B (x h(y,w)) (x)!0 V S 2 (y) p(x) 2 L i (h(y,w),w)f (w,h(y,w),w)cosq y dwdy L 3

9 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw P Camea S 2 (x) Z cosq y = V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy Z Z c = lim B (x h(y,w)) (x)!0 V S 2 (y) p(x) 2 L i (h(y,w),w)f (w,h(y,w),w)cosq y dwdy Z = L i (x,w)f (w,x,w)cosq x dw S 2 (x) L 3

10 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw P Camea S 2 (x) Z cosq y = V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy Z Z c = lim B (x h(y,w)) (x)!0 V S 2 (y) p(x) 2 L i (h(y,w),w)f (w,h(y,w),w)cosq y dwdy Z R B(x) = lim )L i (x 0,w)dx 0! R S 2 (x) (x)!0 B(x) w(x,x0 )dx 0 f (w,x,w)cosq x dw L 3

11 Moden Path Tacing Light tanspot simulation ways to fomulate the adiance L eflected in a suface point x L (x,w ) Z = L i (x,w)f (w,x,w)cosq x dw P Camea S 2 (x) Z cosq y = V (x,y)l i (x,w)f (w,x,w)cosq x V x y 2 dy Z Z c = lim B (x h(y,w)) (x)!0 V S 2 (y) p(x) 2 L i (h(y,w),w)f (w,h(y,w),w)cosq y dwdy Z R B(x) = lim )L i (x 0,w)dx 0 (x)!0 S 2 (x) RB(x) w(x,x0 )dx 0 f (w,x,w)cosq x dw L 3

12 Moden Path Tacing Light tanspot simulation path tacing: Stating paths fom camea and iteating scatteing and ay tacing bad fo small light souces, good fo lage light souces P P P 4

13 Moden Path Tacing Light tanspot simulation path tacing with next event estimation by shadow ays (dashed lines) good fo small light souces, bad fo close light souces P P P P P 4

14 Moden Path Tacing Light tanspot simulation light tacing, i.e. paths stating fom the light souce connected to the camea can captue some caustics, whee path tacing and next event estimation do not wok P P P P P P P P 4

15 Moden Path Tacing Light tanspot simulation all obvious ways to geneate light tanspot paths which ones ae good? P P P P P P P P P 4

16 Moden Path Tacing Light tanspot simulation bidiectional path tacing, optimally combining all techniques by weighting each contibution  l i=0 w l,i = 1 fo path length l 1, l 2 N w 1,1 P +w 1,0 P +w 2,2 P +w 2,1 P +w 2,0 P +w 3,3 P +w 3,2 P +w 3,1 P +w 3,0 P 4

17 Moden Path Tacing Light tanspot simulation bidiectional path tacing, optimally combining all techniques by weighting each contibution  l i=0 w l,i = 1 fo path length l 1, l 2 N w 1,1 P +w 1,0 P +w 2,2 P +w 2,1 P +w 2,0 P +w 3,3 P +w 3,2 P +w 3,1 P +w 3,0 P poblem of insufficient techniques, fo example, if only one wl,i 6= 0 4

18 Moden Path Tacing Numeical intego-appoximation Monte Calo methods Z g(y)= f (y,x)dx [0,1) s 5

19 Moden Path Tacing Numeical intego-appoximation Monte Calo methods Z g(y)= f (y,x)dx 1 [0,1) s n n  f (y,x i ) i=1 unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes 5

20 Moden Path Tacing Numeical intego-appoximation Monte Calo methods Z g(y)= f (y,x)dx 1 [0,1) s n n  f (y,x i ) i=1 unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes 5

21 Moden Path Tacing Numeical intego-appoximation Monte Calo methods Z g(y)= f (y,x)dx 1 [0,1) s n n  f (y,x i ) i=1 unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes 5

22 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes 5

23 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes 5

24 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes quasi-monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) 5

25 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes quasi-monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) much moe unifom coelated samples x i ealized by low-discepancy sequences, which ae pogessive Latin-hypecube samples 5

26 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes quasi-monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) much moe unifom coelated samples x i ealized by low-discepancy sequences, which ae pogessive Latin-hypecube samples 5

27 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes quasi-monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) much moe unifom coelated samples x i ealized by low-discepancy sequences, which ae pogessive Latin-hypecube samples 5

28 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes quasi-monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) much moe unifom coelated samples x i ealized by low-discepancy sequences, which ae pogessive Latin-hypecube samples 5

29 Moden Path Tacing Numeical intego-appoximation Monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) unifom, independent, unpedictable andom samples x i simulated by pseudo-andom numbes quasi-monte Calo methods g(y)= Z [0,1) s f (y,x)dx 1 n n  i=1 f (y,x i ) much moe unifom coelated samples x i ealized by low-discepancy sequences, which ae pogessive Latin-hypecube samples 5

30 Moden Path Tacing Pushbutton paadigm deteministic may impove speed of convegence epoducible and simple to paallelize 6

31 Moden Path Tacing Pushbutton paadigm deteministic may impove speed of convegence epoducible and simple to paallelize unbiased zeo diffeence between expectation and mathematical object not sufficient fo convegence 6

32 Moden Path Tacing Pushbutton paadigm deteministic may impove speed of convegence epoducible and simple to paallelize biased allows fo amelioating the poblem of insufficient techniques can temendously incease efficiency 6

33 Moden Path Tacing Pushbutton paadigm deteministic may impove speed of convegence epoducible and simple to paallelize biased allows fo amelioating the poblem of insufficient techniques can temendously incease efficiency consistent eo vanishes with inceasing set of samples no pesistent atifacts intoduced by algoithm I Quasi-Monte Calo image synthesis in a nutshell I The Iay light tanspot simulation and endeing system 6

34 Moden Path Tacing Pushbutton paadigm deteministic may impove speed of convegence epoducible and simple to paallelize biased allows fo amelioating the poblem of insufficient techniques can temendously incease efficiency consistent eo vanishes with inceasing set of samples no pesistent atifacts intoduced by algoithm I Quasi-Monte Calo image synthesis in a nutshell I The Iay light tanspot simulation and endeing system 6

35 Reconstuction fom noisy input: Massively paallel path space filteing (link)

36 Fom Machine Leaning to Gaphics

37 Machine Leaning Taxonomy unsupevised leaning fom unlabeled data examples: clusteing, auto-encode netwoks 9

38 Machine Leaning Taxonomy unsupevised leaning fom unlabeled data examples: clusteing, auto-encode netwoks semi-supevised leaning by ewads example: einfocement leaning 9

39 Machine Leaning Taxonomy unsupevised leaning fom unlabeled data examples: clusteing, auto-encode netwoks semi-supevised leaning by ewads example: einfocement leaning supevised leaning fom labeled data examples: suppot vecto machines, decision tees, atificial neual netwoks 9

40 Reinfocement Leaning Goal: maximize ewad state tansition yields ewad t+1 (a t s t ) 2 R Agent s t s t+1 t+1 (a t s t ) a t Envionment 10

41 Reinfocement Leaning Goal: maximize ewad state tansition yields ewad t+1 (a t s t ) 2 R Agent s t s t+1 t+1 (a t s t ) a t lean a policy pt to select an action a t 2 A(s t ) given the cuent state s t 2 S Envionment 10

42 Reinfocement Leaning Goal: maximize ewad state tansition yields ewad t+1 (a t s t ) 2 R Agent s t s t+1 t+1 (a t s t ) a t lean a policy pt to select an action a t 2 A(s t ) given the cuent state s t 2 S Envionment maximizing the discounted cumulative ewad V (s t ) Â g k t+1+k (a t+k s t+k ), whee 0 < g < 1 k=0 10

43 Reinfocement Leaning Q-Leaning [Watkins 1989] leans optimal action selection policy fo any given Makov decision pocess Q 0 (s,a) = (1 a) Q(s,a)+a (s,a)+g V (s 0 ) fo a leaning ate a 2 [0,1] 11

44 Reinfocement Leaning Q-Leaning [Watkins 1989] leans optimal action selection policy fo any given Makov decision pocess Q 0 (s,a) = (1 a) Q(s,a)+a (s,a)+g V (s 0 ) fo a leaning ate a 2 [0,1] with the following options fo the discounted cumulative ewad 8 max a 0 2A Q(s 0,a 0 ) conside best action in next state s 0 >< V (s 0 ) >: 11

45 Reinfocement Leaning Q-Leaning [Watkins 1989] leans optimal action selection policy fo any given Makov decision pocess Q 0 (s,a) = (1 a) Q(s,a)+a (s,a)+g V (s 0 ) fo a leaning ate a 2 [0,1] with the following options fo the discounted cumulative ewad 8 max a 0 2A Q(s 0,a 0 ) conside best action in next state s 0 >< V (s 0 ) Â a 0 2A p(s 0,a 0 )Q(s 0,a 0 ) policy weighted aveage ove discete action space >: 11

46 Reinfocement Leaning Q-Leaning [Watkins 1989] leans optimal action selection policy fo any given Makov decision pocess Q 0 (s,a) = (1 a) Q(s,a)+a (s,a)+g V (s 0 ) fo a leaning ate a 2 [0,1] with the following options fo the discounted cumulative ewad 8 max a 0 2A Q(s 0,a 0 ) conside best action in next state s 0 >< V (s 0 ) Â a 0 2A p(s 0,a 0 )Q(s 0,a 0 ) policy weighted aveage ove discete action space >: R A p(s0,a 0 )Q(s 0,a 0 )da 0 policy weighted aveage ove continuous action space 11

47 Reinfocement Leaning Maximize ewad by leaning impotance sampling online adiance integal equation L(x,w) = L e (x,w) + R S 2 +(x) f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i 12

48 Reinfocement Leaning Maximize ewad by leaning impotance sampling online stuctual equivalence of integal equation and Q-leaning L(x,w) = L e (x,w) + R S+(x) 2 f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i Q 0 (s,a) =(1 a)q(s,a)+a (s,a) + g R A p(s 0,a 0 ) Q(s 0,a 0 ) da 0 12

49 Reinfocement Leaning Maximize ewad by leaning impotance sampling online stuctual equivalence of integal equation and Q-leaning L(x,w) = L e (x,w) + R S+(x) 2 f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i Q 0 (s,a) = (1 a)q(s,a)+a (s,a) + g R A p(s 0,a 0 ) Q(s 0,a 0 ) da 0 12

50 Reinfocement Leaning Maximize ewad by leaning impotance sampling online stuctual equivalence of integal equation and Q-leaning L(x,w) = L e (x,w) + R S+(x) 2 f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i Q 0 (s,a) = (1 a)q(s,a)+a (s,a) + g R A p(s 0,a 0 ) Q(s 0,a 0 ) da 0 12

51 Reinfocement Leaning Maximize ewad by leaning impotance sampling online stuctual equivalence of integal equation and Q-leaning L(x,w) = L e (x,w) + R S+(x) 2 f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i Q 0 (s,a) = (1 a)q(s,a)+a (s,a) + g R A p(s 0,a 0 ) Q(s 0,a 0 ) da 0 12

52 Reinfocement Leaning Maximize ewad by leaning impotance sampling online stuctual equivalence of integal equation and Q-leaning L(x,w) = L e (x,w) + R S+(x) 2 f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i Q 0 (s,a) = (1 a)q(s,a)+a (s,a) + g R A p(s 0,a 0 ) Q(s 0,a 0 ) da 0 12

53 Reinfocement Leaning Maximize ewad by leaning impotance sampling online stuctual equivalence of integal equation and Q-leaning L(x,w) = L e (x,w) + R S+(x) 2 f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i Q 0 (s,a) =(1 a)q(s,a)+a (s,a) + g R A p(s 0,a 0 ) Q(s 0,a 0 ) da 0 gaphics example: leaning the incident adiance Z Q 0 (x,w)=(1 a)q(x,w)+a L e (y, w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i 12

54 Reinfocement Leaning Maximize ewad by leaning impotance sampling online stuctual equivalence of integal equation and Q-leaning L(x,w) = L e (x,w) + R S+(x) 2 f s (w i,x,w)cosq i L(h(x,w i ), w i ) dw i Q 0 (s,a) =(1 a)q(s,a)+a (s,a) + g R A p(s 0,a 0 ) Q(s 0,a 0 ) da 0 gaphics example: leaning the incident adiance Z Q 0 (x,w)=(1 a)q(x,w)+a L e (y, w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i to be used as a policy fo selecting an action w in state x to each the next state y := h(x,w) the leaning ate a is the only paamete left I Technical Note: Q-Leaning 12

55 Reinfocement Leaning Online algoithm fo guiding light tanspot paths Function pathtace(camea,scene) thoughput 1 ay setuppimayray(camea) fo i 0 to do y,n intesect(scene, ay) if isenvionment(y) then etun thoughput getradiancefomenvionment(ay,y) else if isaealight(y) etun thoughput getradiancefomaealight(ay,y) w,p w,f s samplebsdf(y,n) thoughput thoughput f s cos(n,w) / p w ay y, w 13

56 Reinfocement Leaning Online algoithm fo guiding light tanspot paths Function pathtace(camea,scene) thoughput 1 ay fo i y,n setuppimayray(camea) 0 to do if i > 0 then intesect(scene, ay) Q 0 (x,w)=(1 if isenvionment(y) then a)q(x,w)+a L e (y, w)+ RS 2+(y) f s(w i,y, w)cosq i Q(y,w i )dw i etun thoughput getradiancefomenvionment(ay,y) else if isaealight(y) w,p w,f s etun thoughput getradiancefomaealight(ay,y) thoughput ay y, w samplescatteingdiectionpopotionaltoq(y) thoughput f s cos(n,w) / p w 13

57 appoximate solution Q stoed on discetized hemisphees acoss scene suface

58 2048 paths taced with BRDF impotance sampling in a scene with challenging visibility

59 Path tacing with online einfocement leaning at the same numbe of paths

60 Metopolis light tanspot at the same numbe of paths

61 Reinfocement Leaning Guiding paths to whee the value Q comes fom shote expected path length damatically educed numbe of paths with zeo contibution vey efficient online leaning by leaning Q fom Q 18

62 Reinfocement Leaning Guiding paths to whee the value Q comes fom shote expected path length damatically educed numbe of paths with zeo contibution vey efficient online leaning by leaning Q fom Q diections fo eseach epesentation of value Q: data stuctues fom games impotance sampling popotional to the integand, i.e. the poduct of policy g p times value Q I On-line leaning of paametic mixtue models fo light tanspot simulation I Poduct impotance sampling fo light tanspot path guiding I Fast poduct impotance sampling of envionment maps I Leaning light tanspot the einfoced way I Pactical path guiding fo efficient light-tanspot simulation 18

63 Fom Gaphics back to Machine Leaning

64 Atificial Neual Netwoks in a Nutshell Supevised leaning of high dimensional function appoximation input laye a0, L 1 hidden layes, and output laye a L a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 20

65 Atificial Neual Netwoks in a Nutshell Supevised leaning of high dimensional function appoximation input laye a0, L 1 hidden layes, and output laye a L a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 n l ectified linea units (ReLU) a l,i =max{0,âw l,j,i a l 1,j } in laye l 20

66 Atificial Neual Netwoks in a Nutshell Supevised leaning of high dimensional function appoximation input laye a0, L 1 hidden layes, and output laye a L a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 n l ectified linea units (ReLU) a l,i =max{0,âw l,j,i a l 1,j } in laye l backpopagating the eo d l 1,i = Â al,j >0 d l,j w l,j,i 20

67 Atificial Neual Netwoks in a Nutshell Supevised leaning of high dimensional function appoximation input laye a0, L 1 hidden layes, and output laye a L a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 n l ectified linea units (ReLU) a l,i =max{0,âw l,j,i a l 1,j } in laye l backpopagating the eo d l 1,i = Â al,j >0 d l,j w l,j,i, update weights w 0 l,j,i = w l,j,i ld l,j a l 1,i if a l,j > 0 20

68 Atificial Neual Netwoks in a Nutshell Supevised leaning of high dimensional function appoximation example achitectues classifie I Multilaye feedfowad netwoks ae univesal appoximatos I Appoximation capabilities of multilaye feedfowad netwoks I Univesal appoximation bounds fo supepositions of a sigmoidal function 21

69 Atificial Neual Netwoks in a Nutshell Supevised leaning of high dimensional function appoximation example achitectues classifie geneato I Multilaye feedfowad netwoks ae univesal appoximatos I Appoximation capabilities of multilaye feedfowad netwoks I Univesal appoximation bounds fo supepositions of a sigmoidal function 21

70 Atificial Neual Netwoks in a Nutshell Supevised leaning of high dimensional function appoximation example achitectues classifie geneato auto-encode I Multilaye feedfowad netwoks ae univesal appoximatos I Appoximation capabilities of multilaye feedfowad netwoks I Univesal appoximation bounds fo supepositions of a sigmoidal function 21

71 Efficient Taining of Atificial Neual Netwoks Using an integal equation fo supevised leaning Q-leaning Q 0 (x,w)=(1 a)q(x,w)+a L e (y, Z w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i 22

72 Efficient Taining of Atificial Neual Netwoks Using an integal equation fo supevised leaning Q-leaning Q 0 (x,w)=(1 a)q(x,w)+a L e (y, Z w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i fo a = 1 yields the esidual, i.e. loss Z Q := Q(x,w) L e (y, w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i 22

73 Efficient Taining of Atificial Neual Netwoks Using an integal equation fo supevised leaning Q-leaning Q 0 (x,w)=(1 a)q(x,w)+a L e (y, Z w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i fo a = 1 yields the esidual, i.e. loss Z Q := Q(x,w) L e (y, w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i supevised leaning algoithm light tanspot paths geneated by a low discepancy sequence fo online taining 22

74 Efficient Taining of Atificial Neual Netwoks Using an integal equation fo supevised leaning Q-leaning Q 0 (x,w)=(1 a)q(x,w)+a L e (y, Z w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i fo a = 1 yields the esidual, i.e. loss Z Q := Q(x,w) L e (y, w)+ f s (w i,y, S+(y) 2 w)cosq i Q(y,w i )dw i supevised leaning algoithm light tanspot paths geneated by a low discepancy sequence fo online taining lean weights of an atificial neual netwok fo Q(x, n) by back-popagating loss of each path I A machine leaning diven sky model I Global illumination with adiance egession Functions I Machine leaning and integal equations I Neual impotance sampling 22

75 Efficient Taining of Atificial Neual Netwoks Leaning fom noisy/sampled labeled data find set of weights q of an atificial neual netwok f to minimize summed loss L using clean tagets y i and data ˆx i distibuted accoding to ˆx p(ˆx y i ) agmin q ÂL(f q (ˆx i ),y i ) i 23

76 Efficient Taining of Atificial Neual Netwoks Leaning fom noisy/sampled labeled data find set of weights q of an atificial neual netwok f to minimize summed loss L using clean tagets y i and data ˆx i distibuted accoding to ˆx p(ˆx y i ) agmin q ÂL(f q (ˆx i ),y i ) i using tagets ŷ i distibuted accoding to ŷ p(ŷ) instead agmin q ÂL(f q (ˆx i ),ŷ i ) i 23

77 Efficient Taining of Atificial Neual Netwoks Leaning fom noisy/sampled labeled data find set of weights q of an atificial neual netwok f to minimize summed loss L using clean tagets y i and data ˆx i distibuted accoding to ˆx p(ˆx y i ) agmin q ÂL(f q (ˆx i ),y i ) i using tagets ŷ i distibuted accoding to ŷ p(ŷ) instead agmin q ÂL(f q (ˆx i ),ŷ i ) i allows fo much faste taining of atificial neual netwoks used in simulations amounts to leaning integation and intego-appoximation I Noise2Noise: Leaning image estoation without clean data 23

78 Example Applications of Atificial Neual Netwoks in Rendeing Leaning fom noisy/sampled labeled data denoising quasi-monte Calo endeed images noisy tagets computed 2000 faste than clean tagets 24

79 Example Applications of Atificial Neual Netwoks in Rendeing Sampling accoding to a distibution given by obseved data geneative advesaial netwok (GAN) I image souce I Tutoial on GANs 25

80 Example Applications of Atificial Neual Netwoks in Rendeing Sampling accoding to a distibution given by obseved data geneative advesaial netwok (GAN) I image souce I Tutoial on GANs 25

81 Example Applications of Atificial Neual Netwoks in Rendeing Sampling accoding to a distibution given by obseved data geneative advesaial netwok (GAN) I image souce I Tutoial on GANs 25

82 Example Applications of Atificial Neual Netwoks in Rendeing Sampling accoding to a distibution given by obseved data geneative advesaial netwok (GAN) update geneato G using qg  m i=1 log(1 D(G(x i ))) I image souce I Tutoial on GANs 25

83 Example Applications of Atificial Neual Netwoks in Rendeing Sampling accoding to a distibution given by obseved data geneative advesaial netwok (GAN) update disciminato D (k times) using qd 1 m  m i=1 [log D(x i )+log(1 D(G(x i )))] update geneato G using qg  m i=1 log(1 D(G(x i ))) I image souce I Tutoial on GANs 25

84 Example Applications of Atificial Neual Netwoks in Rendeing Sampling accoding to a distibution given by obseved data Celebity GAN I Pogessive gowing of GANs fo impoved quality, stability, and vaiation 26

85 Example Applications of Atificial Neual Netwoks in Rendeing Replacing simulations by leaned pedictions fo moe efficiency much faste simulation of paticipating media hieachical stencil of volume densities as input to the neual netwok I Deep scatteing: Rendeing atmospheic clouds with adiance-pedicting neual netwoks I Leaning paticle physics by example: Acceleating science with geneative advesaial netwoks 27

86 Neual Netwoks linea in Time and Space

87 Neual Netwoks linea in Time and Space Complexity the bain about neve cells with to up to 10 4 connections to othes 29

88 Neual Netwoks linea in Time and Space Complexity the bain about neve cells with to up to 10 4 connections to othes atificial neual netwoks numbe of neual units L n = Â n l whee n l is the numbe of neuons in laye l l=1 29

89 Neual Netwoks linea in Time and Space Complexity the bain about neve cells with to up to 10 4 connections to othes atificial neual netwoks numbe of neual units L n = Â n l whee n l is the numbe of neuons in laye l l=1 numbe of weights L n w = Â n l 1 n l l=1 29

90 Neual Netwoks linea in Time and Space Complexity the bain about neve cells with to up to 10 4 connections to othes atificial neual netwoks numbe of neual units L n = Â n l whee n l is the numbe of neuons in laye l l=1 numbe of weights L n w = Â c n l l=1 constain to constant numbe c of weights pe neuon 29

91 Neual Netwoks linea in Time and Space Complexity the bain about neve cells with to up to 10 4 connections to othes atificial neual netwoks numbe of neual units L n = Â n l whee n l is the numbe of neuons in laye l l=1 numbe of weights L n w = Â c n l = c n l=1 constain to constant numbe c of weights pe neuon to each complexity linea in n 29

92 Neual Netwoks linea in Time and Space Sampling popotional to the weights of the tained neual units patition of unit inteval by sums Pk := Â k j=1 w j of nomalized absolute weights 0 w 1 w 2 w m 1 P 0 P 1 P 2 P m 1 P m 30

93 Neual Netwoks linea in Time and Space Sampling popotional to the weights of the tained neual units patition of unit inteval by sums Pk := Â k j=1 w j of nomalized absolute weights 0 w 1 w 2 w m 1 P 0 P 1 P 2 P m 1 P m using a unifom andom vaiable x 2 [0,1) to select input i, P i 1 apple x < P i satisfying Pob({P i 1 apple x < P i })= w i 30

94 Neual Netwoks linea in Time and Space Sampling popotional to the weights of the tained neual units patition of unit inteval by sums Pk := Â k j=1 w j of nomalized absolute weights 0 w 1 w 2 w m 1 P 0 P 1 P 2 P m 1 P m using a unifom andom vaiable x 2 [0,1) to select input i, P i 1 apple x < P i satisfying Pob({P i 1 apple x < P i })= w i in fact deivation of quantization to tenay weights in { 1,0,+1} intege weights esult fom neuons efeenced moe than once elation to dop connect and dop out 30

95 Neual Netwoks linea in Time and Space Sampling popotional to the weights of the tained neual units Test Accuacy 0.6 LeNet on MNIST 0.4 LeNet on CIFAR-10 AlexNet on CIFAR Top-5 Accuacy AlexNet on ILSVRC12 Top-1 Accuacy AlexNet on ILSVRC Pecent of fully connected layes sampled 31

96 Neual Netwoks linea in Time and Space Sampling paths though netwoks complexity bounded by numbe of paths times depth L of netwok 32

97 Neual Netwoks linea in Time and Space Sampling paths though netwoks complexity bounded by numbe of paths times depth L of netwok application afte taining backwads andom walks using sampling popotional to the weights of a neuon compession and quantization by impotance sampling 32

98 Neual Netwoks linea in Time and Space Sampling paths though netwoks complexity bounded by numbe of paths times depth L of netwok application afte taining backwads andom walks using sampling popotional to the weights of a neuon compession and quantization by impotance sampling application befoe taining unifom (bidiectional) andom walks to connect inputs and outputs spase fom scatch 32

99 Neual Netwoks linea in Time and Space Sampling paths though netwoks spase fom scatch a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 33

100 Neual Netwoks linea in Time and Space Sampling paths though netwoks spase fom scatch a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 33

101 Neual Netwoks linea in Time and Space Sampling paths though netwoks spase fom scatch a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 guaanteed connectivity I Monte Calo methods and neual netwoks 33

102 Neual Netwoks linea in Time and Space Sampling paths though netwoks spase fom scatch a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 guaanteed connectivity I Monte Calo methods and neual netwoks 33

103 Neual Netwoks linea in Time and Space Sampling paths though netwoks spase fom scatch a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 guaanteed connectivity I Monte Calo methods and neual netwoks 33

104 Neual Netwoks linea in Time and Space Sampling paths though netwoks spase fom scatch a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 guaanteed connectivity I Monte Calo methods and neual netwoks 33

105 Neual Netwoks linea in Time and Space Sampling paths though netwoks spase fom scatch a 0,0 a L,0 a 1,0 a 2,0 a 0,1 a L,1 a 1,1 a 2,1 a 0,2 a L,2. a 1,2. a 2,2.. a 0,n0 1 a L,nL 1 a 1,n1 1 a 2,n2 1 guaanteed connectivity and coveage I Monte Calo methods and neual netwoks 33

106 Neual Netwoks linea in Time and Space Test accuacy fo 4 laye feedfowad netwok (784/300/300/10) tained spase fom scatch Test Accuacy MNIST Fashion MNIST Numbe of pe pixel paths though netwok 34

107 Fom Machine Leaning to Gaphics and back Summay light tanspot and einfocement leaning descibed by same integal equation lean whee adiance comes fom neual netwoks esults of linea complexity by path tacing tenaization and quantization of tained atificial neual netwoks spase fom scatch taining 35