Imagination and Scientific Visualizations. Greg Trafton (NRL) Susan Trickett (NRL) Chris Schunn (University of Pittsburgh)

Transcription

1 Imagination and Scientific Visualizations Greg Trafton (NRL) Susan Trickett (NRL) Chris Schunn (University of Pittsburgh)

2 Creativity in Science Many anecdotes of creativity in scientific discovery seem to depend on imagination: Einstein (chasing a beam of light) Kekulé (imagining atoms combining to form molecues) Maxwell (envisioned lines of magnetic force)

3 Creativity and Imagination Studying creativity and imagintion in real scientists is hard (not to understate the problem) We have been working with a method so that we can study working scientists imagination as they work on real tasks

4 Our focus We focus on scientists and meteorologists who work with complex visualizations Primarily during the data analysis stage (not hypothesis generation, not data collection, not writeup,...) Imagination reliably occur when skilled individuals work with complex visualizations (Schunn et all, 2007; Trafton et al., 2005; Trickett & Trafton, 2007), especially when uncertain or during anomalies (Modeling these in context is too hard right now, at least with the modeling framework I currently use)

5

6 OK, so what the heck IS imagination for us? Examples of real scientists and meteorologists follow

7 Imagination in Astronomy

8 Transcript Utterance I mean, in a perfect, in a perfect world, in a perfect sort of spider diagram If you looked at the velocity contours without any sort of streaming motions, without streaming motions You d probably expect these lines here [gesturing to screen] to go all the way across, you know.

9 Imagination in Meteorology

10 Conceptual Simulation Transcription Utterance You also have a 12 max 14 winds are not supporting that the next chart has it moving down further to the south there is probably a low coming off the coast so I would move it further to the south [points] and that just supports what I said about ours, OK

11 Types of imagination We focus on imagination that occurs spatially: creating a new spatial object, manipulating it, comparing it, playing what-if scenarios with it, etc. For this project, we will divide imagination into three categories: Pure Spatial Transformations Comparison Spatial Transformations Conceptual Simulations

12 Theoretical Framework Spatial Transformations A spatial transformation occurs when a spatial object is transformed from one mental state or location into another mental state or location. They occur in a mental representation that is an analog of physical space. Can occur purely mentally or on top of visualizations.

13 Pure Spatial Transformations Mental Rotation (Shepard & Metzler, 1985) Time Series extrapolation Object Movement Object Animation 2D 3D transformations (St. John et al)

14 Example of a pure spatial transformation Meteorology Utterance But I have a problem saying it s going to be 55º with the flow from this area. The flow is such that you got warm air being pushed up even though, even though you ve got a fall in thickness. I ll make it 60º Notes Anomaly/ Uncertainty ST: Mental animation Verifying his understanding by playing out Resolution

15 Example of a pure spatial transformation Astronomy Utterance Notes The [galaxy] arm is detaching here and sort of flowing away... ST: Mental animation The visualization is static

16 Comparison Spatial Transformations When someone imagines something and then explicitly compares it to another (usually visible) object, that is a comparison spatial transformation Comparison spatial transformations deal with alignment (Gentner), differences, and similarities at a spatial level These types of mental operations are qualitatively different from pure spatial transformations (Kosslyn et al., 1998, Trafton et al., 2005)

17 Example of a comparison spatial transformation Utterance Also one thing I m noting is that the Canadian model is having a problem picking up the two lows It [the isobar] is very small and will circulate around it and that the ENSAP [model] has a low right there, but it has it closed off and this looks exactly like the satellite shot Notes Anomaly/ Uncertainty reading info from vis + ST ST-C reading info from vis ST-C

18 Example of a comparison spatial transformation Utterance stuff out of the plane in all directions the fact that you see such a strong concentration of gas in the ring... Notes ST (image doesn t show this) ST-C Astronomer imagined stuff coming out of the plane and then tries to compare and align it to current visualization

19 Conceptual Simulations (Trickett & Trafton, 2007) Emphasis on hypotheticals / What if scenarios Conceptual simulations are specific steps of operations: Create a new representation (imagine something) Transform that representation (spatially) in a hypothetical manner Explore the result (see what happens)

20

21 Conceptual Simulation example Utterance Notes I mean, in a perfect, in a perfect world, in a perfect sort of spider diagram If you looked at the velocity contours without any sort of streaming motions, without streaming motions You d probably expect these lines here [gesturing to screen] to go all the way across, you know. Setting up a hypothetical imagining how it would be without streaming motions seeing how it plays out

22 Conceptual Simulation example [CFD] Utterance Notes It is conceivably possible that this curve is floating around all over the place and what they re showing is an average Setting up a hypothetical imagining how it would be if the curve was floating [and if that s right] I may actually still get the curve right seeing how it plays out

23 Summary We are going to look at 3 different types of imagination (pure ST, comparison ST, and conceptual simulations) In real scientists and meteorologists

24 Approach We used an in vivo / ex vivo (Dunbar, 1995, 1997) approach where we examined working practioners as they do their work We studied expert scientists, expert meteorologists, and novice meteorologists in vivo (as they did their real-world task) We video taped everyone as they talked aloud, transcribed their utterances, coded anomalies and uncertainty, pure ST, comparison ST, and conceptual simulations

25 Participants Scientists: 4 Experts (10+ years of experience) in a variety of areas (astronomy, CFD, neuroscience, fmri) Expert meteorologists: 5 experts (10+ years experience) predicting what the weather would be (high seas warnings, amount of precipitation, etc.) Novice meteorologists: 10 novices (juniors and seniors in college) predicting what the weather would be All data collection occurred on their own computers with their own tools

26 Data Analysis We focused our analyses around anomalies and uncertainty because creativity is most likely to occur around anomalies (theoretically and empirically) Any time a scientist / meteorologist found an anomaly, we coded pure ST, comparison ST, and conceptual simulations until resolution How did the groups differ?

27 Simple data People spent about minutes analyzing data All participants found anomalies, ranging from 1-10 No huge differences across tasks or domains, though scientists took a bit longer than the novice meteorologists We have strong IRR for all these categories (85%+, kappa ~.8+)

28 Imagination differs Occurrences per anomaly 4 Number of occurrences per anomaly Novices Experts Scientists 0 Pure ST Comparison ST Conceptual Simulations

29 Summary Imagination and creativity is hard, but: Novice Meteorologists deal with anomalies via comparison ST (they compare weather models a lot) [least creative] Expert Meteorologists deal with anomalies via pure ST (moving fronts around) [pretty creative] Expert scientists deal with anomalies via conceptual simulations (doing a lot of what-if scenarios) [highly creative]

30 Creativity and imagination Even though all groups are doing a lot of relatively creative imagination, it s not clear that both groups really need (or want) to be doing it The engineers are trying to get a specific answer to a specific question (e.g., will it rain or snow tomorrow?) The scientists are doing more scientific discovery These differences may be critical to computational (support) systems

31 Computational Tool Building? So far I haven t focused much (any) on the computational tools that scientists and meteorologists use How do computational tools impact the imagination process?

32 Meteorologists Tools and tasks Meteorologists frequently use both a computational system and a mapwall The mapwall does not scaffold any type of imagination; the computer may

33 Experiment 10 Novice forecasters (junior and senior meteorology majors) made 2 ex vivo forecasts Uncertainty manipulation (within, counterbalanced) 1 forecast contained 2 models that agreed with each other [No-Discrepancy condition] 1 forecast contained 2 models that disagreed with each other (amount of rain) [Discrepancy condition] Interface manipulation (between) (computer vs. mapwall) Talk-aloud protocols given

34 Computer System Demo Interface support for ST: animation, comparisons

35 Experimental Mapwall

36 Coding Protocols were coded for: Whether forecasters found the major discrepancy Spatial Transformations Which products they looked at in order, etc. [not discussed here]

37 Predictions Forecasters should have noticed major discrepancy Forecasters should perform more spatial transformations immediately after discrepancy than right before it Mapwall condition should have more spatial transformations than computer

38 Manipulation check No differences in accuracy between conditions: mapwall and computer were accurate about 80% 9 out of 10 forecasters found the major discrepancy

39 Uncertainty and spatial transformations For the 9 forecasters who noticed the major discrepancy, we coded the number of spatial transformations that occurred in the 10 utterances before and the 10 utterances after the discrepancy (Most of these ST were comparison ST, but we collapsed pure and comparison ST for this analysis)

40 Forecasters do more ST after a discrepancy Number of Spatial Transformations (per 10 utterances) Spatial Transformations before and after discrepancy Control Condition Before Discrepancy After Discrepancy

41 Mapwall vs. computer Mapwall condition should have more spatial transformations than computer condition since forecasters can perform interface actions using the computer (people will not do the work if they don t have to) Variously known by the soft constraints hypothesis (Gray et al., 06) or the people are lazy hypothesis

42 Number of Spatial Transformations (per 10 utterances) Fewer ST on computer than mapwall Mapwall Spatial Transformations before and after discrepancy Before Discrepancy After Discrepancy Computer

43 Summary Computational support for ST can reduce the number of ST that novices have to perform For engineering disciplines, this is probably good news

44 Conclusions Imagination is important in all scientific, technical, and engineering domains Different domains and levels of expertise use different types of imagination Novice meteorologists use comparison ST Expert meteorologists use pure ST Expert scientists use conceptual simulations Different computational tools can change the number (and perhaps the type) of imagination that are used, impacting creativity (?)

45 It s a nice story, but... I was very happy with this story, the data, everything. But I do have an anomaly I m currently working on [chance for creativity and imagination]

46 The mapwall vs. the computer No difference in accuracy between conditions (about 80% accurate) N=5 per condition

47 Anomaly alert! We all thought that the computer would be faster or equal to the mapwall All of our participants had lots of experience with both the way we built our computer system (we modeled it on systems they were currently using) and the mapwall Why was the mapwall so much faster than the computer?

48 _OM Time (mins) Upper Air Climo Surface analysis Satellite GFS GFS Vorticity NAM NAM Vorticity GFS Thickness NAM Thickness GFS Relative Humidity NAM Relative Humidity GFS MOS NAM MOS GFS text NAM text Forecast sheet Writes forecast _OH Time (mins) Upper Air Climo Surface analysis Satellite GFS GFS Vorticity NAM NAM Vorticity GFS Thickness NAM Thickness GFS Relative Humidity NAM Relative Humidity GFS MOS NAM MOS GFS text NAM text Forecast sheet Writes forecast Mapwall Computer

49 Avg. time spent looking at each product per session Number of products looked at per session (Only graphical products) Avg. time per product Number of products mapwall computer mapwall computer Condition p = # repeats (multiple product looks) per session (Only graphical products) Condition p = Percentage of repeats per session (Only graphical products) # repeats Percentage repeats mapwall computer mapwall computer Condition p = Condition p =

50 Can we replicate? (Chris data: we were taking bets) Weather Forecasting Deer Population

51 Anomaly summary Significant time savings can be had with a mapwall type presentation of material The exact process-level reasons are currently being investigated, but it is an (easily) replicable finding Perhaps speaks to the problem of ease of access of data and information, which is important for creativity in science

52 Greg Trafton <Google me to get my website>