Application to Intelligent Transportation Systems M. Shawky Heudiasyc Laboratory, JRU with CNRS Université de Technologie de Compiègne France

Transcription

1 Future Challenges in Design Frameworks for Embedded Systems Application to Intelligent Transportation Systems M. Shawky Heudiasyc Laboratory, JRU with CNRS Université de Technologie de Compiègne France

2 Meet the Expectations! Baby Girl ipad_magazine Confusion - YouTube [360p].mp4

3 Dum systems no more tolerated! In the next 10 to 20 years A dum system will be considered dangerous A car without pedestrian detection will no more be tolerated! The same for obstacle detection with automatic breaking So, no more: dum vehicle! Dum design tool! Dum component! Dum compiler!

4 Intelligent Transportation Systems The bigger picture!

5

6

7 satellite DMB GPS Terrestrial DMB Portable Internet cellular Portable Internet cellular Hot - Spot ( Wireless LAN ) RSE RSE - to - RSE RSE 5 GHz Wireless LAN IR DSRC vehicle - to - vehicle ( Wireless LAN or 60 GHz ) portable - to - vehicle RSE RSE 5. 8 GHz DSRC Courtesy of Fiat

8 Bigger picture, cooperate to drive better

9 LOOSE cooperation 9 CVIS Core Technology Interfaces Courtesy of Renault Version

10 Outline What do we mean by cognitive behavior in ITS? Recognition of Driving situation Environment perception Sensors, data fusion, dealing with uncertainty, etc. Overcome uncertainty or Live with uncertainty Distributed uncertainty management Redundancy; multi/many core opportunity

11 Cognitive car? BMW Automatic Driving - YouTube [720p].mp4

12 Lack of tools No programming framework today provides functional blocs for: Pedestrian detection Or even Pedestrian modeling for image processing toolbox Even if you find it No easy way to integrate it in an embedded vehicle architecture Probably not complying to Autosar What about much more complex functions like cognitive functions?

13 Cognitive functions Understanding Reasonning Double checking Downgraded operation

14 EU integrated projects (700 M )

15 Driving situation characterization Véronique Cherfaoui Heudiasyc Lab UTC, France Michèle Rombaut University of Joseph Fourier, Grenoble, France Data fusion for driving situation characterization 15

16 Overtaking sequence Camera Telemeter 16

17 Perception architecture Situation characterization Data fusion Data fusion Signal/image processing Physical sensor Signal/image processing Physical sensor Signal/image processing Physical sensor 17 Data fusion for driving situation characterization

18 For a particular application What objectives to reach? What information to get? Front vehicle following: position and speed of the front vehicle (accuracy: position 20cm, speed 5km/h) Overtaking assistance: existence of a rear left vehicle (no vehicle: 100%, a vehicle 90%) Characterization of the data: accuracy, reliability, frequency, delay 18 DRiiVE : Data reduction and analysis

19 Definition of accuracy Estimation of the difference between the measure m from the sensor and the real unknown value X to measure Ordered and continuous space of definition W X m x W 19 Data fusion for driving situation characterization

20 Example The distance between the experimental vehicle and the front vehicle (target) is 23m more or less 60cm This means : The real value X of the distance is in the interval [22,4m ; 23,6m] 20 Data fusion for driving situation characterization

21 Accuracy modeled by probabilities p(x/m): probability that X = x, if the measure is m Gaussian distribution : mean m, variance s 2 p(x/m) s 21 Data fusion for driving situation characterization m X x

22 Accuracy modeled by fuzzy sets p m (x): possibility that X = x, if the measure is m The membership function m m (x)= p m (x) is defined by an expert m m (x) m X x 22 Data fusion for driving situation characterization

23 Accuracy modeled by evidential theory The space of discernment is the set 2 W of the subsets A i of W m m (A i ) is the evidence that X is in A i if the measure is m 23 Data fusion for driving situation characterization

24 Accuracy modeled by evidential theory m(a i ) A 1 A 2 m(a 1 ) m X A 3 m(a 2 ) m(a 3 ) x 24 Data fusion for driving situation characterization

25

26

27

28

29

30

31 Definition of reliability Estimation of the confidence in an hypothesis H i Discrete and non-ordered space of definition W H 1 H 2 H 4 H 3 H 1 : the target is a car H 2 : the target is a truck H 3 : the target is a motorbike H 4 : the target is a pedestrian 32 Data fusion for driving situation characterization

32 Data processing sequence 1. Temporal data fusion 2. Fusion of redundant data 3. Fusion of complementary data 4. Symbolic characterisation of the situations 33 DRiiVE : Data reduction and analysis

33 1- Temporal data fusion The experimental vehicle (EV) moves in a static environment Other vehicles around the experimental vehicle move too The information, true at time t, becomes false at time t + D t Need to time stamp the data (different delays and frequencies) 34 DRiiVE : Data reduction and analysis

34 Data evolution Use of the model evolution (a priori knowledges) v (t + D t) = g D t + v (t) x (t + D t) = 1/2 g D t 2 + (v (t + D t) - v (t)) D t + x (t) Based on the Kalman filter Target following algorithm line following multi-vehicles following Y t+ D t t X 35 DRiiVE : Data reduction and analysis

35 2- Fusion of redundant data Simultaneous observations of the same object Improve the accuracy Few redondant data because of the lack of sensors Y camera 1 X camera 2 36 DRiiVE : Data reduction and analysis

36 3- Fusion of complementary data Same object, different types of data Different objects Increase the knowledge on environment Y camera (Y,Z) X telemeter (X,Z) 37 DRiiVE : Data reduction and analysis

37 Fusion of complementary data Y X 38 DRiiVE : Data reduction and analysis

38 4- Symbolic characterisation Data interpretation Definition of the symbolic models Use of a priori knowledges DRiiVE : Data reduction and analysis 1 : -0,75m } 2 :+0,80m 3 :... EV on the right lane

39 The numeric/symbolic conversion m(x) m low (m) m middle (m) low middle high m x 40 Data fusion for driving situation characterization

40 Maneuver recognition Temporal sequence of situations Example of maneuver: the overtaking 41 DRiiVE : Data reduction and analysis

41 State : Overtaking Top view Front camera Rear camera 42 DRiiVE : Data reduction and analysis

42 State : approach Top view Front camera Rear camera 43 DRiiVE : Data reduction and analysis

43 State : approach Top view Front camera Rear camera 44 DRiiVE : Data reduction and analysis

44 State : approach Top view Front camera Rear camera 45 DRiiVE : Data reduction and analysis

45 State : approach Top view Front camera Rear camera 46 DRiiVE : Data reduction and analysis

46 State : approach Top view Front camera Rear camera 47 DRiiVE : Data reduction and analysis

47 State : lane change Top view Front camera Rear camera 48 DRiiVE : Data reduction and analysis

48 State : lane change Top view Front camera Rear camera 49 DRiiVE : Data reduction and analysis

49 State : overtake Top view Front camera Rear camera 50 DRiiVE : Data reduction and analysis

50 State : overtake Top view Front camera Rear camera 51 DRiiVE : Data reduction and analysis

51 State : overtake Top view Front camera Rear camera 52 DRiiVE : Data reduction and analysis

52 State : lane change Top view Front camera Rear camera 53 DRiiVE : Data reduction and analysis

53 State : lane change Top view Front camera Rear camera 54 DRiiVE : Data reduction and analysis

54 State : move away Top view Front camera Rear camera 55 DRiiVE : Data reduction and analysis

55 State : move away Top view Front camera Rear camera 56 DRiiVE : Data reduction and analysis

56 State : move away Top view Front camera Rear camera 57 DRiiVE : Data reduction and analysis

57 State : move away Top view Front camera Rear camera 58 DRiiVE : Data reduction and analysis

58 State : Top view Front camera Rear camera 59 DRiiVE : Data reduction and analysis

59 High-level interpretations of driving situations - Conclusions drawn from previous work - Become Intermediate Data for the next step Sophie LORIETTE-ROUGEGREZ Jean-Marc NIGRO Université de technologie de Troyes Laboratoire LM2S 60 TROYES France

60 Raw data and intermediate data Time X Y S Teta Acc Phi Rg Rd Data obtained from the experimental vehicle y EV TV Rd Time Clock ( s) Acc Acceleration of EV relative to TV (meters²/second) Phi Front wheel angle of EV (in degrees) Rd Position of EV against the right road side (meters) Rg Position of EV against the left road side (meters) Teta Angle of the target TV ( degrees) S Speed of EV relative to TV (meters/second) X Position on the x's axis of TV against EV (meters) Y Position on the y's axis of TV against EV (meters) x Data's meaning 61

61 Overtaking maneuver recognition graph Beginning of overtaking Wait for overtaking While EV in front of TV On the same lane Steering wheel to the left Signalling of intent of overtaking EV behind TV, same lane Crossing of the Left discontinuous line 62

62 IDRES Experimentation results End of lane changing to the right Crossing right discontinuous line Beginning of lane changing to the right End of passing Passing End of lane changing to the left Crossing left discontinuous line Beginning of lane changing to the left Overtaking intent Waiting for overtaking Normal overtaking Time 66

63 Modeling overtaking maneuver with a Petri net PN = <P,T,R,M> P: the set of places M:the marking vector T: the set of transitions R: the vector of receptivity p1 t p2 p3 p4 p 1 t t 2 3 t4 5 Initial state Overtaking Final state Left lane change Right lane change LS > 0 SWA > 0 LS small LA small or >0 LS<0 SWA<0 LS small LA small or >0 LS:Lateral Speed, LA:Longitudinal Acceleration, SWA:Steering Wheels Angle 67

64 The belief Petri net P p 1 p 2,, p 3 PN = <P,T,R,M> p, p, p, p, p, p, p, p, p, p, p 2 P p , 3 P 1 The new marking function t 1 m k : 2 P 0,1 t 3 P 2 t 2 A P m k A 1 P 3 68

65 First step * The transitions are sure k R 0,1, 0 T * The initial state, at time k, P 1 m m k k k p, p 0.6, m p, p 1 p, p, p , t 3 t 1 P 2 t 2 P 3 k T R 0,1,0 p, p p, p 1 k T R 0,1,0 p, p p 2 k T R 0,1,0 p, p, p p, p

66 Uncertain knowledge of transitions The frame of discernment: W 0,1 t i t i t i Is false Is true Is false or true R k i,1 k i, 2 R R m k i, 3 k i k i m 0 1 m k i 0,1 R k k k,, R, R The vector of receptivity: i 1 i,2 i,3 i 1 q k R 70

67 P 1 t 1 k R t 3 P 2 t 2 m m k 1 k k 1 k k p p m k p, p p m 0, P 3 m R k k m k 1 k m k 1 k 1 A k k A m m B B 71

68 Increasing complexity and tools increasing cognition 10 increasing complexity increasing complexity 10 code size increasing cognition code size Lack of tools! Currently, no programming tool provides uncertainty management Fuzzy logic and Interval programming tools are insufficient

69 Programming with uncertainty management Example : Speed regulator Usual programming IF (DISTANCE > 30) THEN Programming with uncertainty management IF (bel(distance,30)>80%) AND pl(distance,30) < 20%) OR (.. AND.) OR ( AND.) OR (. AND ) THEN It would better be integrated in a meta language, with automatic source code generation, all included in a single framework

70 Applications complexity versus tools Applications complexity Mathematical support Programming Frameworks

71 Dempster combination laws Belief functions: canonical decompositions and combination rules. Frédéric Pichon, PhD. Thesis, March 2009 Combine veracities from different sources for the same hypothesis Conjunctive sum Disjunctive sum

72 Critical systems Engineering and Uncertainty Overcome uncertainty as used to do?

73 Overcome uncertainty Objective of Engineering today is to OVERCOME uncertainty What about taking into account uncertainty Serious change in system design and programming paradigm Do we really have the choice?

74 Uncertainty: do we really have the choice? Understanding level Intermediate level Sensors level

75 Embedded critical systems Astrium (EADS group) example Interesting concepts But how may I plug this stuff in the docking procedure of my spacecraft to the International Space Station (ISS)? Centimetric, millisecond precision! What are we comparing exactly? Nominal operation System behaves as modeled or exceptional operation System behavior model Which approach is more resilient?

76 Redundancy and uncertainty Multi-track paradigm (example geolocation)

77 Multi track Paradigm & Framework Multi-track approach Selection Result Programmer writes all alternative tracks code Programmer writes selection function code

78 Multi track Paradigm & Framework Multi-track approach Programmer writes one instruction stream Compiler Selection Result

79 Lack of tools, once more! No programming frameworks providing: Programming just the track skeleton Automatic generation of multi-track instances Gathering all results Comparing and deciding Quite a challenge to develop this framework! How would a component look like in this case? What would be the input? Etc.

80 Functional approach: a major Key To capture Need (functional analysis, non-functional constraints allocation) To design Solution (functions allocation to components) To ensure consistency between Need & Solution (unique, consistent functional dataflow allocated) F21 F1 F22 F6 F3 C1 C2 C3

81 How to validate Need understanding Operational Analysis (actors, tasks, roles, missions & goals) Including capture of non-functional constraints Functions traceability & justification Vs Requirements and operational analysis A1 A2 A2 Reqs F1 F2 F3 F5 F4

82 How to validate Solution /1 Perform a multi-viewpoint trade-off Analysis safety & perf & interface & product line & weight & cost & ViewPoints Solution Architecture

83 How to validate Solution /2 Confront Components Architecture Vs Requirements and Need analysis Operational, Functional, non-functional A1 A2 A2 F1 F2 F3 F5 F4 F21 F1 F22 C1 C2 F6 F3 C3 ViewPoints

84 Summary: Steps & Models Operational Analysis F1 A1 A2 A2 F2 F3 F5 F4 Reqs Functional/NF Analysis F1 F21 F22 C1 C2 Logical Architecture Viewpoints trade-off F6 C11 F21 F1 F6 F7 C12 F22 C1 C3 F3 C2 C3 C4 ViewPoints Physical Architecture Viewpoints trade-off Processors ViewPoints Buses

85 Redundancy and multi/many core processors Towards automatic redundancy?

86 Smart Redundancy over multi/many core Programmer writes one instruction stream Compiler Selection Result

87 Automatic Redundancy Framework Improving execution reliability of parallel applications on multi-core architectures,o. Tahan, PhD. Thesis, December 2012

88 Redundancy Framework, Slowdown!

89 Redundancy Framework but overall speedup wrt mono-core

90 Speed-up using smart redundancy

91 Tools for automatic redundancy From the fault tree analysis Automatically generate diagnosis models Check if architecture (hardware/software) are satisfying diagnosis model Automatically generate redundant software code

92 Conclusion Overcome uncertainty or manage uncertainty Exact reasonning/programming Versus Approximate reasonning/ «programming?» Automatic redundancy generation Many core processors opportunity Lack of tools In all your research works, Please: Think Algorithm but prototype a tool!