Pilot Evaluation of a UAS Detect-and-Avoid System s Effectiveness in Remaining Well Clear!

Transcription

1 National Aeronautics and Space Administration Pilot Evaluation of a UAS Detect-and-Avoid System s Effectiveness in Remaining Well Clear! Presenter: Confesor Santiago Co-author: Eric Mueller Affiliation: NASA June 25, 2015

2 Mo#va#on Interoperability? Air Traffic Control Airspace Procedures See and Avoid Requirements? All aircrae are required to see and avoid and remain well clear of other traffic Instrument Flight Rules (IFR) Visual Flight Rules (VFR) Detect and Avoid Safety Unmanned AircraE Systems (UAS) Research Ques?ons: What s the effec#veness of a pilot- in- the- loop detect- and- avoid system? Which parameters of a detect- and- avoid system most improve the pilot s ability to detect and avoid? 2

3 Agenda Background and Previous Work Well Clear Descrip#on Simula#on Environment Procedures Results Experiment 1: Func#on of detect- and- avoid display features and whether the display was stand- alone or integrated within the main traffic display Experiment 2: Follow- on to Experiment 1, but focused on differences between specific display features when the display is integrated, and surveillance detec#on ranges Summary 3

4 Terminology Unmanned AircraE Systems (UAS) = Remotely Piloted AircraE Systems (RPAS) Detect and Avoid (DAA) = Sense and Avoid (SAA) Self- separa#on = Compliance with remaining well clear 4

5 Background RTCA Special Commi\ee 228 (SC- 228) was established to develop minimum opera#onal performance requirements for unmanned aircrae systems SC- 228 consists of a consor#um of government, academia, and industry collaborators Requirements will apply to UAS s new detect- and- avoid systems necessary to integrate UAS within the United States Na#onal Airspace System (NAS) DAA- equipped UAS opera#ng in the NAS will have to meet a certain level of safety Previous DAA system evalua#ons were part of producing a deployable system for specialized applica#ons, so unclear how to generalize the results Other work that could be generalized were not evaluated in the context of an overall safety case in the NAS (focused more on human- systems interac#on) 5

6 Objec#ves To quan#fy the effec#veness of the pilot- automa#on system to remain well clear as a func#on of display and algorithm features that help drive DAA system requirements To collect and analyze pilot performance parameters that will improve the modeling of overall DAA system performance in non- human- in- the- loop simula#ons that help drive interoperability considera#ons 6

7 How did we approach this problem? 7

8 Well Clear Descrip#on Well clear is vague, subjec#ve, but vitally important A well clear separa#on standard is recommended to be, Large enough to: Avoid inducing collision avoidance maneuvers Avoid excessive concern for pilots of proximate piloted aircrae Minimize traffic alert issuances by air traffic control But, small enough to: Prevent the need for large devia#ons that poten#ally disrupt traffic flow and ATC separa#on management plans 8

9 Well Clear Defini#on Pre- dates defini-on accepted by UAS Community in U.S.A., but a source of lineage Criteria Horizontal Separa#on Ver#cal Separa#on Time to Closest Point of Approach (CPA) Threshold <= 0.8 nau#cal miles <= 400 feet <= 40 seconds 0.8 nmi 400 E 40 s The CPA between two aircrae is an es#mate of the future minimum slant range. Time- to- CPA is defined as the difference between the #me at predicted CPA and current #me. 9

10 Human- in- the- loop Simula#ons Two experiments involving 21 professional UAS pilots Mixture of Global Hawk and Predator- B flight experience Evalua#on of candidate DAA display and algorithms used by UAS pilots opera#ng under IFR with the need to detect and avoid other traffic in the airspace Dependent Variables Loss of Well Clear Propor#ons (discussed here) Maneuver Timeliness (in paper) UAS pilot/atc Coordina#on Times (in paper) 10

11 Vigilant Spirit Control Sta#on Control Panel NavAids Ownship Waypoints Flight Plan Vigilant Spirit Control Sta#on (AFRL/RH). Distribu#on A: Approved for public release; distribu#on unlimited. 88ABW Cleared 3/18/2013; 88ABW

12 Air Traffic Simula#on Environment Air Traffic Control Station Pseudo Pilot Station 12

13 Procedures AircraE Predator B- like +/ feet per min. climb/descent 20- degree nominal bank angle Airspace Oakland Center 40/41 (Nor. California) Class A & E with mixed IFR/VFR traffic Manned traffic modeled using real traffic data Primary Tasks Navigate along pre- filed flight plan Comply with clearances and traffic display alerts Secondary Tasks: Communica#ons System health alerts Mission 38- minute mission route Wild Fire Monitoring Al#tude: 9k 14k MSL Speed: 110 KCAS 13

14 Aler#ng Logic Ver?cal Separa?on Well- clear: 400 feet Self Separa#on: 900 feet Preven#ve: 900 feet 0.8nm UAS Well Clear Threshold 40 sec Self Separa?on Threshold Preven?ve Alert Threshold 80/110 sec Time to CPA 1.2nm 2nm Pilot Ac?ons Ac#on No Immediate ac#on only required necessary if ac#on one or required both aircrae suddenly maneuvers May Coordinate Considera#ons No#fy want ATC to with as coordinate soon when ATC as to determining prac#cable determine with a an resolu#on appropriate maneuver (if printed, see Pilot Ac8ons table in backup for clarity) 14

15 Self- separa#on Timeline Time to CPA 110 sec 95 sec 85 sec 55 sec 40 sec Detect Intruders Alert Pilots Gain Situa?on Awareness Pilots Determine Resolu?on Nego?ate Clearance with ATC and uplink maneuver to aircral AircraL Maneuvers Well Clear Threshold 15

16 Experiment 1 Summary and Results Experimental Design Loca#on of DAA Display (Stand- alone or Integrated with ground- control sta#on) Level of DAA Traffic Informa#on and Tools ( Basic or Advanced ) Alert Time (80 or 110 to CPA, embedded) Results No significant difference in remaining well clear given self- separa#on alerts at 80 vs. 110 seconds to CPA Pilots were almost never able to remain well clear when first alerted at 55 seconds to CPA or less Incorpora#on of Advanced DAA informa#on and tools significantly reduced the propor#on of losses of well clear when compared to the Basic configura#on Integra#on of DAA traffic informa#on and tools with the pilot s ground- control sta#on did not significantly improved the pilots ability to remain well clear Other human- systems integra#on research reveals difference in response #me and workload 16

17 Experiment 2 - Design Display Informa#on Level D1: Advanced Display with Informa#on Only Roughly Basic display in Experiment 1 D2: Advanced Display with Informa#on + Vector Planner D3: Advanced Display with Informa#on + Maneuver Recommenda#ons D4: Advanced Display with Informa#on + Vector Planner + Maneuver Recommenda#ons Roughly same as Advanced suite in Experimental 1 Embedded Variable Track Type (manipulated within each scenario; not counterbalanced) 1. Coopera#ve Traffic (ADS- B- equipped) 2. Non- Coopera#ve Traffic (Airborne Radar- equipped) Par#cipants 5 ac#ve California Air Na#onal Guard (163 rd Reconnaissance Squadron) MQ- 1 (Predator)/MQ- 9 (Reaper) pilots 5 ac#ve Beale Air Force Base RQ- 4 (Global Hawk) pilots 17

18 Propor#on of Losses of Well Clear by Display Proportion of Losses" of Well Clear" D1 D2 D3 D4 Take- away: Unclear which Advanced display feature from Experiment 1 affected the reduc#on in losses of well clear 18

19 Experiment 2: Coopera#ve and Non- coopera#ve AircraE Coopera#ve - electronic means of iden#fying aircrae and exchanging of state informa#on (e.g. transponder or ADS- B) Horizontal range: 80 nmi Ver?cal range: +/- 10,000 feet Non- coopera#ve no means of electronically iden#fying aircrae One state- of- the- art method for detec#ng non- coopera#ve aircrae is an airborne radar Range: 6 nmi, Azimuth: +/- 110 degrees, Eleva?on: +/- 20 degrees Surveillance Range Horizontal Field of Regard (Azimuth Angle) Surveillance Range Ver?cal Field of Regard (Eleva?on Angle) 19

20 Losses of Well Clear by Alert Time Coopera?ve Encounters Only Actual Losses of Well Clear Prevented Losses of Well Clear Number of Encounters Popup losses of well clear, i.e. already viola#on when first alerted Only one loss of well clear when alert #me is 110 s 5 0 <= Time to Closest Point of Approach 20

21 Losses of Well Clear by Alert Time Number of Encounters Ineffec#ve with short alert #me Non- coopera?ve Encounters Only Actual Losses of Well Clear Prevented Losses of Well Clear Almost always effec#ve in remaining well clear at > 60s 0 <= Time to Closest Point of Approach (s) More uniformly distributed alerts #mes due to limited detec#on range for non- coopera#ve aircrae Suggests seconds to CPA as prac#cal alert #me to avoid a high propor#on of losses of well clear 21

22 Summary Examined a UAS detect- and- avoid system s effec#veness in remaining well clear with varying display features and loca#on orienta#ons Evaluated a quan#ta#ve well clear separa#on standard and aler#ng logic that has direct lineage to current- day/accepted proposals Experiment 1 Addi#on of tools (Advanced displays) increased effec#veness of pilots ability to remain clear, however integra#ng the display with control sta#on did not Pilots were almost never able to remain well clear when alerted with 15 seconds or less prior to loss of well clear Experiment 2 Maneuver recommenda#ons appear to be the Advanced feature most effec#ve in remaining well clear Although non- coopera#ve aircrae can only be detected at a limited range, most losses of well clear can be prevented given alert of at least 60 seconds to CPA 22

23 Ques#ons? NASA s Research Ground Control Sta#on (Experiment 2) Contact Informa#on Confesor San#ago confesor.san#ago@nasa.gov 23

24 BACKUP 24

25 Future Work Part- Task 5 Newly accepted well clear defini#on by UAS community Sensor noise V&V of drae DAA requirements in non- real- #me- #me simula#ons using improved models from HITLs 25

26 Alert Level Pilot Ac#ons Alert Level Pilot Action Time to CPA Well Clear Recovery Immediate ac#on required No#fy ATC as soon as prac#cable 40 sec Self- separa#on Ac#on required Coordinate with ATC to determine an appropriate maneuver 80/110 sec Self- separa#on buffered Ac#on will be necessary only if one or both aircrae suddenly maneuvers May want to coordinate with ATC about the inten#ons of the other aircrae. 80/110 sec Preven#ve No ac#on necessary Considera#ons when determining a resolu#on maneuver 80/110 sec 26

27 Basic Traffic Informa#on Advanced Traffic Informa#on and Tools Standalone DAA Display 2x2 Design: Integrated DAA Display 27

28 Experiment 1: Highlighted Results 28

29 Losses of Well Clear by Alert Time to CPA Actual Losses of Well Clear Prevented Losses of Well Clear Number of Encounters Number of Encounters <= >85 Time to Closest Point of Approach (s) Time To Closest Point of Approach (s) Pilots were almost never able to remain well clear when first alerted at 55 seconds to CPA or less This helps drive surveillance and aler#ng requirements 29

30 Propor#on of Losses of Well Clear by Display Propor#on of Losses of Well Clear Addi#on of traffic and resolu#on tool (Advanced) significantly reduced the propor#on of LoWCs when compared to the Basic configura#on Integra#on of DAA traffic informa#on and tools with the pilot s main traffic display slightly improved the pilots ability to remain well clear, however differences were not sta#s#cally significant 30

31 Propor#on of LoWC by Display Type in Experiment Proportion 0.40 Propor#on Standalone Integrated 1 Basic Advanced 2 Basic Stdln. Adv. Stdln. Basic Integ. Adv. Integ. Display Type 31

32 Experiment 2 Display Condi#ons D1 D2 Click and Drag Vector Arrow Informa-on- Only Vector Planner Only (No Auto Resolu#on) D3 Auto Resolu#on generated by Autoresolver Auto Resolu#on generated by Autoresolver D4 Click and Drag Vector Arrow Auto Resolu#on Only (No Vector Planner) Vector Planner + Auto Resolu#on 32 32

33 Mean response #me by whether the pilot obtained an ATC clearance before maneuvering Mean Response Time (s) Pilot Acted Without Clearance Pilot Obtained Clearance 5 0 <= Time to Closest Point of Approach (s) Response #me amount of #me from first SS alert to pilot upload of a resolu#on maneuver Overall Mean Response Times Pilot Acted Without ATC Clearance: 10.8 seconds Pilot Obtained ATC Clearance: 18.7 seconds 33

34 Pilot Response Time Modeling Use response #me data to build a pilot response #me model, so there are realis#c responses to SS alerts in non- real- #me- #me simula#ons Sample for a distribu#on, and wait that amount of #me before commanding maneuver Perceive a new Self Separa#on Alert Distributions fit to results High Urgency Evaluate Remaining Time to LoWC (Urgency) Mid Urgency Low Urgency Determine Pilot Response Delay 34