Normalised mutual information-based ranking of spatio-temporal localisation maps

Transcription

1 Normalised mutual information-based ranking of spatio-temporal localisation maps Nicolas Méger, Christophe Rigotti, Lionel Gueguen, Felicity Lodge, Catherine Pothier, Rémi Andreoli and Mihai Datcu ESA-EUSC-JRC 2012 Image Information Mining Conference, 24/10/12, DLR, Oberpfaffenhofen. p. 1 I m going to present work we have been doing using Normalised Mutual Information to rank Spatio Temporal Localisation maps. This work is a collaboration between several people... 1

2 Outline 1. Problem statement 2. SITS, base of sequences, GFS-patterns, STL-maps 3. Normalised mutual information-based ranking 4. Application to soil erosion monitoring 5. Conclusions 6. References p. 2 First I will describe the objectives of this work. And the techniques we use. Then I will describe the method using Normalised Mutual Information that we have used to rank the results. After this I will show some results of application of the method to a series of satellite images of New Caledonia to monitor soil erosion. Finishing off with some conclusions and some relevant references. 2

3 Outline 1. Problem statement 2. SITS, base of sequences, GFS-patterns, STL-maps 3. Normalised mutual information-based ranking 4. Application to soil erosion monitoring 5. Conclusions 6. References p. 3 First, the problem statement. 3

4 We have a series of satellite images of the same area at different times, and we want to be able to describe this satellite image time series, or SITS, both temporally and spatially, using unsupervised techniques. 4

5 Our requirements are, that we have no identification of objects prior to the mining step. And patterns are searched for at the pixel level, so we are looking for pixel evolutions or sub-evolutions. And we are looking for evolutions or sub-evolutions which cover MANY CONNECTED PIXELS. Some of the problems we face with this approach are: Since All possible evolutions and sub-evolutions must be taken into account, this can result in millions of pixel evolutions which must be mined, so the calculations can be huge. Also We want to allow for different occurrence dates of a single evolution, to handle noisy acquisitions and to look for temporal evolution of the spatial nature of the patterns. As a consequence, clustering methods, which would take into account all acquisition times of a pattern, are not considered. 5

6 Outline 1. Problem statement 2. SITS, base of sequences, GFS-patterns, STL-maps 3. Normalised mutual information-based ranking 4. Application to soil erosion monitoring 5. Conclusions 6. References p. 6 Now onto the mining step 6

7 So, we have a series of images of the same area at different times. For each Pixel, values are quantised and the resulting values represented by Symbols. Here, symbols are denoted by different colours, but could also use ABC... We then have a symbolic pixel evolution for each pixel. Which gives a set of symbolic evolutions like those identified by the data mining community in 1995 as a Base of Sequences. Original applications dealt with web logs and purchase sequences. But once we have out symbolic pixel evolution from the satellite images, we can use similar techniques to extract patterns from the data. 7

8 ... So we extract sequential patterns from the pixel evolutions. These patterns do not have to be consecutive, so there can be other symbols between the ones in the pattern we are looking at. This means that if there are problems with the image, such as clouds, we can jump that image but we can still see the pattern. For each pattern, we measure the Support, which is the number of pixels this evolution occurs in. We define a Minimum Support, Sigma. Then evolutions occurring in at least Sigma pixels are Frequent Sequential Patterns. Since the minimum support, Sigma, is an Anti Monotonic Constraint, it can be used to prune the search space which reduces the computational effort needed to mine the images. Which is very important given the large size of some image series. 8

9 So now we have Frequent Sequential Patterns, but these can be dispersed anywhere over the images, And what we want is to find Localised Groups of Pixels which share the same evolution, So we introduce a Spatial measure, the Average Connectivity, which describes the average number of Nearest Neighbouring pixels, that is the 8 surrounding pixels Which also show this pattern. We define a threshold K which is the Minimum Average Connectivity, Then Frequent Sequential Patterns covering pixels with sufficient local connectivity are then termed Grouped Frequent Sequential Patterns, Or GFS patterns. This constraint is partly anti-monotonic and can be used to further prune the search space. 9

10 Spatio-Temporal Localisation maps (STL-maps) time GFS-pattern A->A->B->C->A->A p. 10 Now we have a mechanism to find Grouped Frequent Sequential Patterns in our series of images, but these patterns can occur at different times in the series. To visualise when the pattern occurs, we create a Spatio Temporal Localisation map, Or STL map. Pixels showing a pattern are coloured using a colour palette, like the one shown on the right. The colour is defined by the last date of the pattern, so the later the pattern ends in the series, the more the colour tends to the violet end of the spectrum. When you compare the image to the colour key, you can clearly see the pattern occurring in different areas at different times And you see a rainbow effect, where the pattern is propagated through space with time. 10

11 Outline 1. Problem statement 2. SITS, base of sequences, GFS-patterns, STL-maps 3. Normalised mutual information-based ranking 4. Application to soil erosion monitoring 5. Conclusions 6. References p. 11 But even constraining the search to find Grouped Frequent Sequential patterns can leave us with a huge number of patterns, so we have been trying to find a way to define how interesting these images might be to the user, and we have had some nice results using Normalised Mutual Information. 11

12 Swap-randomisation p B A t1 t2 t3 t4 2 pixel positions and dates are chosen using random methods p A B t1 t2 t3 t4 Pixel states to be swapped p A B t1 t2 t3 t4 Swap-randomisation is done spatially and temporally Pixel state frequencies are not modified p B A t1 t2 t3 t4 Pixel states once swapped p. 12 So first, what we do is to make randomised symbolic pixel evolutions. We do this using a Swap Randomisation method inspired by work on Swap Randomisation of Boolean Matrices. We have our set of symbolic pixel evolutions. Two events are chosen randomly. We want to maintain pixel state frequencies, so a swap in one pixel is only made if the reverse swap can be made in the other pixel. That is, if the first pixel shows states A then B, then the second pixel must show states B then A. Otherwise the swap is rejected and the process starts again. By maintaining the pixel state frequency, we allow some spatial and temporal randomisation, but the intrinsic structure of the dataset is retained. We then make STL maps of the patterns we found previously, on the new Randomised Data set. 12

13 STL-map ranking through Normalised Mutual Information (NMI) ostl-map original SITS X Y ostl-map randomised SITS So then we have the STL map of the pattern which is the image of the pattern s occurrence on the original SITS, And we have another image of the pattern s occurrence on the randomised SITS. We then want to compare these two images using a similarity measure. Mutual information has been shown to be an effective similarity measure, With a statistical relationship that can be captured by analysing the joint Entropy of two images. Mutual Information measures how much information two variables share So if knowing one of these variables doesn t give us any help in predicting the other, the normalised mutual information would be zero. We calculate the Normalised Mutual Information between these two images using this definition: With the Mutual Information on the top, divided by the Minimum Entropy. The idea is that if the Normalised Mutual Information is high, then the occurrence of the pattern in the original image isn t giving us any more information than the occurrence of the pattern in the randomised image, Therefore the pattern is less interesting. So what we are looking for is a low value for the Normalised Mutual Information. p

14 Outline 1. Problem statement 2. SITS, base of sequences, GFS-patterns, STL-maps 3. Normalised mutual information-based ranking 4. Application to soil erosion monitoring 5. Conclusions 6. References p. 14 Now I am going to show you some results on a series of images of the southern tip of New Caledonia. 14

15 New Caledonia is a small island about 1500km off Queensland 15

16 The ground in New Caledonia is rich in minerals, particularly Nickel, 16

17 Which has a long history of exploitation using open cast mining. This has been going on for many years and has left large devegetated areas which are then subject to erosion and landslides. 17

18 In addition it is very dry in the summer, and the scrub that covers the mountains is prone to fires, and the resulting devegetation makes the land also prone to a natural erosion. The problem then is that, after heavy rains and cyclones, the earth and the waste material from the mining gets washed down into the rivers, 18

19 And into the sea, As you can see from the beaches near the mining sites. Much of the sea surrounding the island is a UNESCO World Heritage site because of the surrounding coral reefs and lagoons... 19

20 So we are trying to develop a method to help monitor changes in the landscape due to erosion. 20

21 Data Data and ground truth provided by Bluecham SAS 16 co-registered Landsat 7 images (513X513) , 30 m, over New Caledonia (Yaté rural district, nickel open cast mining activities, scrub fires, erosion, landsildes, UNESCO protected coral reefs and lagoons) Bands: blue ( nm), green ( nm), red ( nm), near infra-red ( nm) Synthetic bands: NDVI, brightness, redness All bands were quanitised into 3 levels (33 rd et 66 th percentiles) p. 21 We have 16 co-registered Landsat 7 images provided by Bluecham. They have 4 bands, blue, green, red and near infra-red, and we also have the synthetic bands ndvi, brightness and redness. The bands were quantised into 3 levels using the 33rd and 66th percentiles. 21

22 Some samples Clouds... and artefacts p. 22 Here are some examples of problems with the image series. Most of the images were cloudy to some extent. This is a particularly bad one. And sensor artifacts are also present on some images. If I have time at the end I will show you the full series. Not very optimistic when I first saw the images, but in fact we have had some nice results. While we are here, if we have a quick look at the second image, you can see the sea and the coastline, around the coastline you see a coral barrier, and on the land some devegetated areas of bare earth. 22

23 GS-pattern extraction and ranking σ = 6000 k = 5 Number of GFS-patterns: Number of maximal GFS-patterns: 295 NMI ranking stabilise at 40M swaps Lowest NMI: 0,028 Highest NMI: 1 Average NMI: 0,1 Standard deviation of NMI measures: 0,07 p. 23 The easiest results to interpret, and the ones I am going to present, were obtained using the NDVI synthetic band. With the minimum support Sigma at 6000 and the average connectivity K at 5. Using these parameters, we get about 15 thousand patterns Of which 295 were maximal patterns, which are patterns that are not sub-patterns in any larger pattern. 40 million swaps were found to be sufficient to get constant results for the Normalised Mutual Information. 23

24 Best ranked pattern t - Coloured: maquis scrub - Black: sea (south-east), a lake, mining activities, other vegetation (stars along the coast) - NMI = 0,028 - Patterns ranked 2nd and 3rd are similar pattern 2,2,3,2,2,2,3 p. 24 This is an image of the best ranked pattern, which has highlighted the maquis scrub vegetation which covers much of the area. The stars along the coast are a different type of vegetation. Top left is a black area which is a lake, And the other 3 circled black areas lower down are areas of mining activity The 2 nd and 3rd ranked patterns showed very similar images, 24

25 Ranked as the 4th pattern 4th pattern t - 0: contour of the lake («grand lac») - 1 5: mining activities pattern 2, 2, 1, 1, 1, 2 p. 25 The 4th pattern again highlights the lake, this time in contour, and the coastline, And it very effectively shows up the different areas of Current mining activity. These areas have been confirmed to be relevant by experts in New Caledonia and match the Ground Truth provided by Bluecham. 25

26 Ranked as the 4th pattern 4th pattern - (0) a lake showing mud infiltration and droughts phenomena - (1) a scrub nursery used for reforestation (light blue) - (2) the more recent main nickel mining area (gradient blue dark violet violet corresp. expansion over the time) - (3) a new mining area (violet) - (4) tailings storage area (gradient dark violet violet corresp. expansion North-East over time - (5) the nickel ore processing facilities (subareas from light blue to violet) - (6) the staff housing (same colours as area 5) t p. 26 They have been identified as: 0) a lake showing lots of phenomena such as infiltrations of mud and effects of droughts.this is also shown around the coastline, particularly this area in the top right which is a lagoon formed by a coral barrier with inputs from rivers that go through some of the older mining areas. There are records of sedimentation events here which correspond to the timings suggested by these results. Area 1 is a scrub nursery used for reforestation. Area 2 is the more recent nickel mining area, with the gradient blue to violet corresponding to expansion over time Area 3 in violet is a new mining area. Area 4 is a tailings storage area, with the colour gradient corresponding to expansion to the North East over time. Area 5 is the nickel ore processing facilities And area 6 is the staff housing, Which was built at the same time as the processing facilities. 26

27 Ranked as the «worst» pattern Worst (and last) pattern t - NMI = 1 - Made of 16 symbols «1» - Sea, cloud and sensor problems. pattern 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 p. 27 The pattern with the highest NMI, is the one with the symbol 1 for all 16 images. Obviously a full series of a single number cannot be randomised, since a switch has no effect. This image shows the sea, with a few black spaces where there have been images with clouds and some sensor problems. 27

28 Outline 1. Problem statement 2. SITS, base of sequences, GFS-patterns, STL-maps 3. Normalised mutual information-based ranking 4. Application to soil erosion monitoring 5. Conclusions 6. References p. 28 Finally, to sum up the main points of the presentation, 28

29 Conclusions and perspectives Interesting and meaningful patterns are ranked as best ones Useless patterns are ranked as the worst ones Users are automatically guided towards interesting patterns Study non-boolean swap-randomisation properties (convergence, uniformity) Best ranked GFS-pattern clustering Iterative techniques to filter the patterns Radar imagery p. 29 We have presented a method for finding Grouped Frequent Sequential Patterns in a series of images, Along with a method for ranking these patterns by interestingness using Normalised Mutual Information. Our initial tests have been encouraging, with the best ranked patterns, that is those with the lowest NMI, Being confirmed to be meaningful by experts in the field, While the patterns ranked worst do not contain much useful information. This automatic ranking procedure means that users are guided towards the most interesting patterns, And could prove invaluable in cases when there are thousands of patterns are produced. Some areas where we are developing this work further: Are further studying the properties of the swap randomisation within our non-boolean system. We know empirically that our randomisation converges, but whether it converges to a true steady state with a uniform distribution or to some pseudo steady state has not yet been confirmed. We are also clustering the best ranked patterns to group similar evolutions. And we are testing the technique on radar images, which is showing similarly encouraging results. 29

30 Outline 1. Problem statement 2. SITS, base of sequences, GFS-patterns, STL-maps 3. Normalised mutual information-based ranking 4. Application to soil erosion monitoring 5. Conclusions 6. References p

31 Thank you for your attention. 31

32 32

33 33

34 34

35 35

36 36

37 p. 37 Here are a couple more examples of images from the lowest interest ranking. They are not showing up the temporal patterns that we are interested in. NMI and randomisation together tend to discard patterns which are always the same and that end in the same date, In other words, we look for evolving and moving patterns. 37

38 p

39 p. 39 which is clearer in this image, where the pattern is propagated through space with time. 39

40 40

41 41

42 Swap-randomisation Choose a pixel and date at random p B A t1 t2 t3 t4 Choose a second event with the same symbol randomly p A B t1 t2 t3 t4 Check the corresponding dates on the other pixel to see if you can swap p p B A t1 t2 t3 t4 A B t1 t2 t3 t4 p. 42 Choose the first pixel and date randomly. Choose a second event with the same symbol as the first event randomly from a list. This gives you pixel 2 and date 2. Check the symbols for date 2 on pixel 1 and date 1 on pixel 2. If they are the same, you can swap. 42

43 questions p. 43 want test the independence between X and Y (if you say X predicts Y, it's not well understood from that experience). About NMI, I also had to precise that we consider pixels values as labels and not integers/floats (even if we could consider date values), which explains why we use entropies: it's a way of considering more general dependencies (compared to standard correlation measures, 43

44 Monitoring the Haiyuan fault p

45 Data ENVISAT InSAR time series covering the Haiyuan fault (north-eastern boundary of the Tibetan plateau) over the period. Hit by several major earthquakes at the beginning of the 20th century. 24 raw SAR images. p

46 Processing chain SAR image coregistration to a single master and interferogram generation. Joint inversion of residual orbital and atmospheric delays. [Cavalié08] Measurement of interseismic strain across the Haiyuan fault by InSar EPSL Validation of atmospheric corrections using the ERA40 global atmospheric model (ECMWF) and correction of each unwrapped interferogram [Doin09] Corrections of stratified tropospheric delays in SAR interferometry: validation with global atmospheric models - AG Inversion of the interferogram series on each track to obtain the increments of smooth LOS radar phase evolution [Jolivet11] Shallow creep on the Haiyuan Fault, Gansu, China, revealed by SAR Interferometry A 24 image SITS (701x701 px, resolution 80mx80m) of phase increments. Contain ground deformation and atmospheric turbulences. p

47 47

48 1->1->1->1->1->1->1->1->1->1 time Creep revealed. Colors show the ending dates of GFS-pattern occurrences. Not radial w.r.t. to the creeping zone: creep migration. p. 48

49 3->3->3->3->3->3->3->3->3->3 time Creep revealed. Colors show the ending dates of GFS-pattern occurrences. Radial w.r.t. to the creeping zone. p. 49