FuRII - An ENVI toolbox for image classification Part 1: Theory related to fuzzy sets, evidential reasoning and fusion

Transcription

1 FuRII - An ENVI toolbox for image classification Part 1: Theory related to fuzzy sets, evidential reasoning and fusion François Leduc DRDC Valcartier Defence R&D Canada Valcartier Technical Memorandum DRDC Valcartier TM July 2008

2

3 FuRII An ENVI toolbox for image classification Part 1: Theory related to fuzzy sets, evidential reasoning and fusion François Leduc DRDC Valcartier Defence R&D Canada Valcartier Technical Memorandum DRDC Valcartier TM July 2008

4 Author François Leduc Approved by Jean-Marc Garneau Section Head Approved for release by Christian Carrier Chief Scientist Her Majesty the Queen as represented by the Minister of National Defence, 2008 Sa majesté la reine, représentée par le ministre de la Défense nationale, 2008

5 Abstract In a context of ATD/ATR based on image analysis, there are many sources of imprecision that can lead to uncertain conclusions. In this context it is very important to have a tool that correctly quantifies the uncertainty of detection results given by a set of different sources. Fuzzy sets and evidential reasoning are well suited for this purpose, but at this time there exists no tool dedicated to target detection based on these concepts. For this reason it was decided to develop such a tool called FuRII (Fuzzy Reasoning applied to Image Intelligence). This document addresses image classification. It describes the foundations of fuzzy sets and evidential theories and the fusion mechanisms offered by these two approaches. It is also demonstrated how fuzzy sets and evidential theories can be combined by modeling imprecise knowledge with fuzzy sets and by fusing multisource information with the evidence theory. Advantages and disadvantages of both theories and fusion mechanisms are explained in detail and illustrated with numerous examples. This document is the first of a series of three. Document II addresses the development and the implementation of FuRII and Document III addresses the validation of the tool. FuRII was developed in the framework of the IB project as a part of the Sensor Data Processing Group (Spectral and Geospatial Exploitation Section) contribution to this project. FuRII is written in the IDL programming language in order to implement the concepts in ENVI which is a well known tool in the military image and intelligence community. Résumé Dans une situation de détection de cible basée sur l analyse d image, il existe de nombreuses sources d imprécision qui mènent inévitablement à des conclusions incertaines. Dans ce cas, il est important d avoir un outil qui quantifie correctement l incertitude associée à des résultats de détection donnés par un ensemble de sources différentes. Les ensembles flous et le raisonnement évidentiel sont tout à fait adéquats pour répondre à ce besoin. Toutefois, il n existe actuellement aucun outil destiné à la détection de cible et qui intègre ces concepts. C est pour cette raison qu un outil nommé FuRII (Fuzzy Reasoning applied to Image Intelligence) a été développé. Ce mémorandum est consacré à la classification d image. Il décrit les fondements de la théorie des ensembles flous et de la théorie de l évidence ainsi que les différents mécanismes de fusion offerts par ces deux approches. Il est également démontré comment il est possible de combiner les deux approches en modélisant la connaissance imprécise avec des ensembles flous et en fusionnant l information multisource avec la théorie de l évidence. Les avantages et inconvénients des deux approches et des mécanismes de fusion sont détaillés et illustrés par de nombreux exemples. Ce document est le premier d une série de trois. Le second document traite de l implantation de FuRII tandis que le troisième et dernier document traite de la validation du système. FuRII a été développé dans le cadre du projet IB et constitue une partie de la contribution du Groupe Traitement des Données des Capteurs (Section Exploitation Spectrale et Géospatiale) à ce projet. Le langage de programmation IDL a été choisi dans le but d implanter les concepts dans ENVI, un outil bien connu dans la communauté militaire de renseignement. DRDC Valcartier TM i

6 This page intentionally left blank. ii DRDC Valcartier TM

7 Executive summary When image analysts are monitoring a site they can use several types of imagery such as panchromatic (EO), multispectral (MSI) and radar (SAR) data. Each of these sources taken separately may appear incomplete but when combined together they may produce a synergetic process leading to a better understanding of the site. In the context of ATD/ATR multisource information should produce better results than single source information. However, even multisource information, because of imprecisions, leads to uncertain conclusions. Thus, it is important to have a tool that facilitates modeling the imprecision properly, to fuse the multisource information adequately and to quantify the uncertainty meaningfully. Fuzzy sets and Dempster- Shafer (evidential) theories are well suited to fulfill these requirements. Unfortunately, there is no tool dedicated to ATD/ATR and image classification that integrates these concepts. This document describes the theoretical aspects of both theories that are implemented in a tool called FuRII (Fuzzy Reasoning applied to Image Intelligence) which is developed in the IDL programming language and built as an ENVI toolbox. FuRII is developed internally at DRDC Valcartier and is part of the Data Exploitation Group contribution to the IB project. It is also shown how fuzzy set theory can be used to model imprecise knowledge about targets of interest by means of membership functions. The process of fuzzy inference is detailed and it is explained how membership values resulting from it correspond to the similarity between an observation and the targets of interest. The different fuzzy fusion operators are also described. The evidential theory is explained and its different fusion rules are compared between them and with the fuzzy fusion operators. It may happens that image analysts are looking for some targets of interests but has difficulty to define all possible targets they may find. In this situation, the evidence theory allows to work in an open-world context which means that the solution given by a system may be something else. The link between the open-world context, the sources reliability and conflict between sources is explained. Finally, it is shown how both theories can be combined through a combination of fuzzy modeling and evidential fusion. This document is the first of three documents. It describes the theoretical aspects of fuzzy and evidence theories and explains in detail the different ways to model information and the different fusion operators. It also explains how source reliability can be integrated into the fusion process. Numerous examples are given with the explanations. Part II addresses the development and the implementation FuRII while Part III addresses the validation process. Leduc, F FuRII An ENVI toolbox for image classification. Part I. DRDC Valcartier TM Defence R&D Canada Valcartier. DRDC Valcartier TM iii

8 Sommaire Lorsqu un analyste d image surveille un site il peut arriver qu il ait à combiner des données panchromatiques, multispectrales et radar. Chacune de ces sources prises individuellement peut paraître incomplète mais lorsqu elles sont combinées elles peuvent produire une synergie qui mène à une meilleure compréhension de la scène. Dans une situation de détection de cible, l information multisource devrait produire des meilleurs résultats qu une source unique. Toutefois, même l information multisource, parce qu imprécise, produit des résultats incertains. Ainsi, il est important d avoir un outil qui permette de modéliser l imprécision correctement, de fusionner l information multisource adéquatement et de quantifier l incertitude de manière sémantiquement valable. Les ensembles flous et la théorie de Dempster-Shafer (évidentielle) sont tout à fait appropriés pour répondre à ces exigences. Malheureusement, il n existe aucun outil destiné à la détection de cible et à la classification d image qui intègre ces concepts. Or, ce mémorandum décrit les fondements de ces deux théories qui sont implantées dans un outil nommé FuRII (Fuzzy Reasoning applied to Image Intelligence). FuRII est développé dans le langage IDL afin d être intégré au logiciel commercial ENVI. FuRII est développé à l interne à RDDC Valcartier et constitue une partie de la contribution du Groupe Traitement des Données des Capteurs au projet IB. Il est démontré dans ce document comment la logique floue peut être utilisée pour modéliser la connaissance imprécise sur les cibles d intérêt par l entremise de fonctions d appartenance. Le processus d inférence floue est détaillé et on explique comment les valeurs d appartenance résultant de ce processus correspondent à la similarité entre une observation et des cibles d intérêt. Les différents opérateurs de fusion sont aussi décrits. La théorie de l évidence est expliquée et ses différentes règles de fusion sont comparées entre elles et avec les opérateurs de fusion flous. Il peut arriver qu un analyste cherche des cibles quelconques sans pouvoir toutefois toutes les énumérer. Dans ce cas, il peut être avantageux de travailler dans un contexte de «monde ouvert», lequel permet à un système de détection d émettre une conclusion «autre» que l ensemble initial des hypothèses. Ce qui amène à explique le lien entre le contexte de monde ouvert, la fiabilité des sources et le conflit entre celles-ci. Finalement on démontre comment les deux théories peuvent être utilisées de façon conjointe par l intermédiaire d une modélisation floue et d une fusion évidentielle. Ce document est le premier d une série de trois. Il traite des fondements des deux théories impliquées et explique les différentes façons de modéliser l information et les différents mécanismes de fusion. L intégration de la fiabilité des sources est également détaillée et de nombreux exemples accompagnent les explications. Le second document traite de l implantation de FuRII tandis que le troisième et dernier traite de la validation du système. Leduc, F FuRII An ENVI toolbox for image classification. Part I. RDDC Valcartier TM R&D pour la Défense Canada Valcartier. iv DRDC Valcartier TM

9 Table of contents Abstract... i Executive summary...iii Sommaire... iv Table of contents... v List of figures...viii List of tables... x 1. Introduction Definitions Image classification Information Imprecision and uncertainty Sources and sensors Fusion Fuzzy sets theory Fuzzy reasoning Membership values and certainty Fusion operators Conjunctive fusion or t-norms Disjunctive fusion or t-conorms Adaptive fusion Quantified adaptive fusion Sources agreement and possibility theory Fuzzy facts Membership function construction Conclusion on fuzzy sets theory DRDC Valcartier TM v

10 4. Dempster-Shafer theory Theory Description Relation between membership values and basic belief values Dempster fusion rule Decision criteria Other fusion rules Dubois & Prade fusion operator Yager fusion operator Smets fusion operator Associative properties Example of fusion Conclusion on the Dempster-Shafer theory From fuzzy sets to Dempster-Shafer theory Closed-world Bayesian mass functions Open-world Bayesian mass functions Closed-world nested mass functions Open-world nested mass functions Formalism on nested mass functions Open- and closed-world paradigms Conclusion Fusion, conflict and reliability Fusion and conflict Source reliability Capacity to make a clear decision Capacity to take the good decision (Sources performance) Capacity to discriminate classes Ability to take good measurement Trade-off rule Discount rule Managing conflict Conflict as a normalizing factor Conflict in an open-world context vi DRDC Valcartier TM

11 6.3.3 Using reliability coefficient Combining reliability coefficients with the open-world context Conclusion Comparison of fusion operators Conjunctive fusion Zadeh s t-norm Disjunctive fusion Zadeh s t-conorm Adaptive fusion Quantified adaptive fusion Dempster fusion rule with Bayesian mass functions Nested mass functions Building mass functions Dempster fusion rule Dubois and Prade fusion operator Yager fusion operator Smets fusion operator Fusion with reliability Conjunctive fusion Dempster fusion rule Conclusion Conclusion References DRDC Valcartier TM vii

12 List of figures Figure 1. Membership functions or graphical representation of fuzzy sets... 9 Figure 2. Deciduous forest membership function for the reflectance in the near infrared Figure 3. Location of a pixel of interest in the Landsat ETM4 (NIR) band Figure 4. Typical forest and clear cut reflectance in the NIR band Figure 5. Forest and clear cut reflectance modeled with membership functions in the red, near infrared and short-wave infrared Figure 6. Location of a pixel of interest in the Landsat ETM3 (red), ETM4 (NIR) and ETM5 (SWIR 1) bands Figure 7. Possibility distributions intersection (h) as a measure of agreement Figure 8. Sensors agreement for the observation leading to the membership values Table Figure 9. Fuzzy inference with a fuzzy fact Figure 10. Fuzzy inference with a more imprecise fuzzy fact Figure 11. Fuzzy knowledge (red), knowledge complement (green) and fuzzy fact (blue) Figure 12. Three types of membership functions: triangle, trapeze and Gaussian Figure 13. Example of histogram-based membership function Figure 14. S and Z membership functions Figure 15. Example of saturation in a decision process Figure 16. Graphical representation of the Dempster fusion rule Figure 17. Graphical representation of the Zadeh s example Figure 18. Relation between the conflict value and the normalization factor (α) Figure 19. Graphical representation of the Zadeh s example processed with the DP fusion rule35 Figure 20. Graphical representation of the Zadeh s example processed with the Yager fusion operator Figure 21. Possible orders for the fusion process of mass functions provided by three sensors38 viii DRDC Valcartier TM

13 Figure 22. Fuzzy inference considering three classes Figure 23. Fusion of the two mass functions of Table 27. Left: without using reliability. Right: using the discount rule Figure 24. Illustration of the Dempster fusion rule applied to the famous Zadeh s example Figure 25. Fusion of the discounted mass functions of Table Figure 26. Dempster fusion rule applied to S 1 S 2 fusion Figure 27. Fusion of sources S 1 and S 6. Data from Table Figure 28. Illustration of the S 1 S 2 fusion. Data from Table DRDC Valcartier TM ix

14 List of tables Table 1. Example of some t-norms Table 2. Example of conjunctive fusion with the Zadeh s t-norm Table 3. Example of some t-conorms Table 4. Example of disjunctive fusion with the Zadeh s t-conorm Table 5. Example of agreement (h) computing Table 6. Illustration of the adaptive fusion Table 7. Example of the adaptive fusion in the presence of strong conflict Table 8. Illustration of the quantified adaptive fusion Table 9. Illustration of the quantified adaptive fusion Table 10. Membership values assigned to two classes according to four sources Table 11. Mass functions derived from Table 10 membership values Table 12. Fusion of mass functions of Table 11 according to four fusion rules/ Table 13. Illustration of the number of conflict measures to compute AS Action of the number of sensors Table 14. Membership values sorted in a decreasing order Table 15. Mass function for membership values of Table Table 16. Membership values to five classes Table 17. Constructing mass function from membership values of Table Table 18. Final mass function from membership values of Table Table 19. Membership values to five classes. None of them is equal to one Table 20. Constructing mass function from membership values of Table Table 21. Correction of the mass function of Table 20 (A). By normalization (B). By adding the empty set element (C) Table 22. Membership values to five classes according to six sensors x DRDC Valcartier TM

15 Table 23. Membership values to five classes according to four sensors Table 24. Example of reliability (1-H) values computed by membership functions intersections Table 25. Example of the application of the trade-off rule Table 26. Example of the application of the discount rule Table 27. Example of two mass functions adjusted with the discount rule and used in Figure Table 28. Application of the discount rule on membership values Table 29. Discounted mass function Table 30. Fusion result of Figure 25 before and after normalization Table 31. Discounted mass function using different reliability coefficients Table 32. Fusion of the two mass functions of Table Table 33. Membership values to five classes according to six sensors Table 34. Mass functions (containing only singletons) obtained by normalizing membership values of Table Table 35. Membership values (left) and mass functions (right) Table 36. Membership values (left) and mass functions (right) Table 37. Membership values to five classes according to six sensors Table 38. The six mass functions corresponding to membership values of Table Table 39. Fusion of sources S 1 and S Table 40. Result of S 1 S 2 fusion Table 41. Fusion of sensors S 1 and S 2. Data from Table Table 42. Pignistic probabilities after DP fusion of the six sources of Table Table 43. Order of fusion of the six mass functions of Table Table 44. Different orders of fusion of the six mass functions of Table Table 45. Influence of the fusion order on the pignistic probabilities. DP fusion operator DRDC Valcartier TM xi

16 Table 46. Step-by-step fusion of Table 38 mass functions with the Yager operator Table 47. Pignistic probabilities computed from the final mass function (S E ) of Table Table 48. Influence of the fusion order on the pignistic probabilities. Yager fusion operator75 Table 49. Closed-world mass functions for sensors S 1 and S 2 obtained from Table Table 50. Reliability coefficients of the six sources of Table Table 51. Discount rule applied on membership values of Table Table 52.The six closed-world mass functions corresponding to membership values of Table Table 53. Adjusted mass functions Table 52 with the discount rule Table 54. Fusion of the mass functions of Table Table 55. Pignistic probabilities computed from the final mass function obtained by the fusion of the information of Table xii DRDC Valcartier TM

17 1. Introduction Image classification consists in labelling every image object with a real-word attribute. This process implies that a fact, such as a pixel value, is compared with a knowledge base which contains a list of objects of interest and their descriptors. A pixel is thus associated with the object to which it is the most similar. In this process, descriptors modeling is very concerned with imprecision because of several factors such as sensor calibration, atmospheric effects, objects intrinsic variations and similarities between objects. Because reasoning with imprecise knowledge will result in uncertain conclusions, it is important to have appropriate tools to manage adequately and to quantify the uncertainty. In many remote sensing applications such as surveillance and site monitoring, what the human wants is a system that would be able to recognize objects with an estimated level of confidence even if imprecise multisource data is provided to be compared with imprecise knowledge. Fuzzy reasoning and evidence theory are well suited for such a system as they both offer imprecision management capabilities and fusion operators. Imprecision is related to the information content while uncertainty is related to the veracity of the information. Finally, fusion is the aggregation of the information provided by several sources. Dealing with uncertainty is not recent since probability concepts were used in the 17 th century in relation with games of chance. At that time, uncertainty was related to chance thus being described by a frequency definition. Probabilities, being appropriate to evaluate chances, are restrictive (the sum of probabilities must equal one) and not suitable for representing imprecision. During the past 30 years many new approaches to imprecision handling have been proposed. Among them, fuzzy set theory was first proposed in 1965 by Zadeh [1]. The strength of this theory resides in part in its flexibility to model imprecision. With this theory, imprecise knowledge is modeled with membership functions which can be of any shape and constructed from expert knowledge or from experimentation. The difference between a fuzzy set and a probability distribution function is that the fuzzy set makes it possible to measure to what extent an observation corresponds to a characteristic of an object while the probability distribution measures what is the chance of meeting this criterion. Fuzzy set theory is an extension of classical set theory. Its main advantage is that it avoids the false premise trap. With classical set theory, reality is black or white, so if an action is based on a previous conclusion that is false the action will be erroneous. With the fuzzy set theory, an initial conclusion can be partly true in which case, the following action will be undertaken but with a different confidence level. DRDC Valcartier TM

18 The theory of Evidence (also called Dempster-Shafer theory) was first proposed by Dempster in 1968 [2] and formalized in 1976 by Shafer [3]. This theory has probabilistic foundations but has the advantage of assigning basic belief values (or masses) not only to singletons but also to compound elements. Thus a mass can be assigned to the element {A,B}, which corresponds to the basic belief of observing A or B without being able to discriminate between the two. The construction of a probability distribution (a probability density function in the continuous domain) is based on sampling and experimentation. On the other hand, with the evidence theory, there is no unique method for building a mass function, so this leaves a great deal of flexibility in the use of this approach. The evidence theory is also interesting for its capacity of managing total ignorance and conflict during fusion process. Fuzzy set theory is useful for its imprecise knowledge representation and evidence theory is interesting for its conflict integration and its fusion operators. Both theories can be combined in a system by performing fuzzy inference with fuzzy sets and fusion with the evidence theory. In that case, membership values resulting from the fuzzy inference need to be converted into masses of the evidence theory. This transformation can be done in several ways these will be detailed in this document. Fusion is the process of integrating multisource information in order to take advantage of source complementarities, redundancy and concordance. Fusion should be, hopefully, a synergetic process where results should become better, or at least stay the same, when the number of sources increases. This will be true is the conflict between sources is low or if the fusion operator takes this conflict into account. As of today there exists no image classification tool dedicated to ATD/ATR based on the concepts described here. The main objective of this work is to remedy this situation by building such a tool. The tool, called FuRII (Fuzzy Reasonning applied to Image Intelligence), is developed in the framework of the project IB. It is written in the IDL programming language and it is implemented as an ENVI toolbox. Finally, the reason for using fuzzy sets and evidence theory is given by their flexibility and by the limitation of the traditional probability approaches. In probability theory like the Bayesian inference, there are many restrictions such as: 1) probabilities are assigned to singletons only; 2) the sum of probabilities must equal one; 3) the probability of an event plus the probability of its negation must equal one (P(A) + P( A) = 1). Also, with Bayesian reasoning a priori probabilities are based on past events and need to be updated as new knowledge is provided. Finally, if a new hypothesis is added, all probabilities have to be updated. This document describes the theoretical foundations of fuzzy logic and evidence theory and their fusion mechanisms applied to pixel-based image classification. Chapter 2 provides some definitions concerning image processing while Chapter 3 clarifies the difference between imprecision and uncertainty. Chapter 4 explains fuzzy 2 DRDC Valcartier TM

19 sets theory and its fusion operators. It also explains different methods of knowledge modeling. Chapter 5 describes the evidence theory and its different fusion mechanisms. Chapter 6 shows how both theories can be combined for reasoning under uncertainty conditions. Chapter 7 gives many examples of information fusion using the different fusion operators described in this memorandum. Finally, Chapter 8 concludes this document. This work was performed within the framework of the IB project, as part of the Sensor Data Processing Group (Spectral and Geospatial Exploitation Section) contribution to this project. DRDC Valcartier TM

20 2. Definitions This Section aims at defining and differentiating several terms used in the field of image processing and information fusion. Each of these terms, classification, information, imprecision, uncertainty, source, sensor and fusion, is briefly described so that ambiguities are decreased. 2.1 Image classification Classification is the process of labelling image objects with a real-word attribute [4]. Classification methods can be divided into supervised and unsupervised approaches [5]. With the supervised approach, the user selects samples within the image then the algorithm analyzes the whole image in order to associate image primitives with the samples. The idea of supervised classification is to associate primitives with known real-world objects. Note that the sample collection step may be skipped if a knowledge base is used. There are a tremendous number of supervised classification techniques going from statistical approaches (maximum likelihood, minimum distance, parallelepiped, spectral angle) to artificial intelligence approaches (neural networks, Bayesian networks, fuzzy sets, Dempster-Shafer) [6]. With unsupervised classification (also known as clustering), the user only specifies the number of desired objects. The algorithm groups pixels into homogenous clusters according to different criteria and it becomes the role of the user to associate clusters to real-world labels. Among unsupervised classification techniques we find the k- means, the isodata, fuzzy C-means algorithms [5], just to name a few. In image processing, when the objects used as input for a classification are pixels, the method is said to be pixel-based and if the primitives correspond to polygons, the method is said to be region-based or polygon-based. In this latter case, it means that image has been segmented before classification begins. The segmentation process can be done with unsupervised clustering or with methods that integrate spatial criteria such as region-growing and slit-and-merge methods [7]. Classification methods can be used for land cover mapping (where each pixel of an image is labelled with a land use/land cover attribute) or for target detection where each primitive is labelled as target / non-target. 2.2 Information Information, data, observation, fact and knowledge are terms that are often used interchangeably without paying attention to their real meaning. Data can be anything from letters to numbers and can be collected by several means such as surveys or by 4 DRDC Valcartier TM

21 the use of sensors. For example, a value of 28 is a piece of data that becomes information when the concept of reflectance is attached to it. Moreover, a reflectance of 28% becomes a more valuable piece of information if it is attached to a given object (with a known location) and the type of sensor used is known. Mendel [8] splits the knowledge definition into two types: 1) objective knowledge, which is used all the time in engineering problem formulation, and 2) subjective knowledge, which represents linguistic information such as rules, expert information and design requirements. A membership function can be considered as a piece of subjective knowledge as we will see in this document. A membership function allows one to model a typical but imprecise characteristic of an object. In this document, the concept of knowledge often refers to a membership function. Observation and fact can be considered as synonyms. In the case of pixel-based image classification, a fact corresponds to the value of a pixel. A fact is rapidly transformed into information by knowing that a pixel at location X,Y in band N is characterized with a value V. 2.3 Imprecision and uncertainty Imprecision and uncertainty are two terms that need to be distinguished in order to better understand which of these terms influence the other and how it does so. In [9], [10] imprecision is associated with information content, while uncertainty is related to its veracity in the sense of conformity with reality. For example, consider the planning of meetings according to time. The information the next meeting will be in January is imprecise because there is 31 days in this month. Guessing that meeting will be held on January 14 is uncertain because of the 30 other possibilities. Let consider another example of reasoning containing imprecise knowledge. In remote sensing, it is well known that the typical reflectance of a deciduous forest canopy is about 40% in the near infrared [11]. This knowledge regarding deciduous reflectance is by definition imprecise due to the presence of the word about. This lack of precision comes from several sources such as illumination conditions, scale of measurements (outside or in laboratory), sensor precision and calibration and vegetation condition (health, phenomenology). In fact, the typical deciduous reflectance might be located in the reflectance interval 35 45% so the knowledge about deciduous reflectance is not defined with a single value. It was said that imprecision is related to information content but in the previous example it was also mentioned that the knowledge was imprecise. This imprecision of knowledge is caused, in part, by the fact that knowledge is derived from the information. For example, a spectrometer is used to measure the reflectance of an DRDC Valcartier TM

22 object in the near infrared. Ten measures are taken. The typical reflectance of the object can be computed by the average of the ten measures leading to an imprecise knowledge. Moreover, each of the ten individual measures was imprecise because of the instrument calibration. So the imprecision of the information is transferred to the knowledge. In the image classification process, a source of confusion comes from classes overlapping (in the spectral space) and thus not having clear boundaries [12]. If the typical reflectance of deciduous forest is 35% 45% in the near infrared, the conifer typical reflectance is 30% 40 %. So measuring a reflectance of 37% leads to an uncertainty about the nature of the forest; it can be categorized as deciduous or coniferous with different levels of uncertainty. In other words, imprecision is associated with information and knowledge while uncertainty is related to conclusions. 2.4 Sources and sensors It is important to clarify the distinction between a source and a sensor. A sensor is a device that can take some measurements. For example, ETM+ (Enhanced Thematic Mapper Plus) is the sensor aboard the Landsat 7 satellite. The ETM+ sensor splits the signal into six multispectral bands (plus other bands) and each of these bands produces images that can be considered as independent sources. From one image, it is possible to derive supplemental sources of information. For example, from one thermal image, it is possible to separate temperature from emissivity, thus having two sources instead of a unique one. Another example is the extraction of edges or texture computation from imagery. So a multisource data set can be composed of many features such as visible and infrared data, digital elevation model, extracted edges, texture imagery, meteorological data, etc. A source of information is anything that may help in analyzing a scene. 2.5 Fusion A scene can be observed with several types of sensors like EO, SAR and thermal. The interpretation of images can also be combined with land cover maps, meteorological data, elevation models and so on. So, using different combinations of data sources will lead to different interpretation results. There are many definitions to fusion. For example in [13] we can read: Information fusion or data fusion is the process of acquisition, filtering, correlation and integration of relevant information from various sources, like sensors, databases, knowledge bases and humans, into one representational format that is appropriate for deriving decisions regarding the interpretation of the information, system goals (like recognition, tracking or situation assessment), sensor management, or system control. 6 DRDC Valcartier TM

23 The JDL [14] had another long definition which has been modified in a concise way as the process of combining data to refine state estimates and predictions [15]. In [16], a distinction is made between multisensor integration and multisensor fusion. The integration is the synergistic use of multisource information to assist in a decision-making process by a system while fusion can occurs during any stage of the integration process. Fusion can take place at any level such as signal, pixel, feature and symbolic representations. The authors in [16] also specify that these definitions are not standard in the literature. Finally, fusion is considered in this document as the process of combining multisource information in order to gain synergy and to increase the reliability of a conclusion. DRDC Valcartier TM

24 3. Fuzzy sets theory This chapter addresses the theoretical foundations of fuzzy set theory applied to image processing. Fuzzy reasoning will first be described in detail, then several fusion operators and their operation mechanisms will be explained. We will also see how knowledge can be modeled with membership functions and how to perform fuzzy inference with fuzzy facts. The link between source consensus and the notion of source reliability will also be introduced. For each of the fusion operators, real-world examples are presented. As was mentioned in the introduction, the traditional probabilistic approach to uncertain reasoning is limited by several restrictions. Fuzzy sets overcome these restrictions on several fronts. First, the sum of the membership values of several hypotheses does not have to sum to unity. This is a great advantage of fuzzy sets, as when new hypotheses are considered all other membership values (μ) do not have to be updated. Also, the membership value of one hypothesis does not have to be the opposite of its negation, so we can have μ(target) = 0.75 and μ(non-target) = Fuzzy reasoning Fuzzy set theory was introduced in 1965 by Zadeh [1] for representing imprecision of knowledge. Fuzzy set theory is an extension of classical set theory [17]. In classical theory, we can define a universal set U, containing all possible reflectance (real) values ranging between 0 and 100%: U = [ 0 ; 100 ] From this universal set U, we can define three subsets corresponding to low (L), medium (M) and high (H) reflectance with values ranging between 0 and 40%, 30 and 70% and between 60 and 100%: L = [ 0 ; 40 ] M = [ 30 ; 70 ] H = [ 60 ; 100 ] From the classical set theory point of view, a reflectance value of 39% corresponds to low reflectance category to the same level as a value of 10% does because classical sets are crisp sets. On the opposite, fuzzy set theory incorporates degrees of membership assigning to which extent a value is part of a set. The sets L, M and H sets can be fuzzified by assigning membership values (numbers between parentheses): L = [ 0(1) ; 10(1) 20(0.75) ; 30(0.5) ; 40(0.25) ; 45(0) ] M = [ 30(0) ; 40(0,5) ; 50(1) ; 60(0,5) ; 70(0) ] H = [ 55(0) ; 60(0,25) ; 70(0,5) ; 80(0,75) ; 90(1) ; 100(1) ] With these fuzzy sets, we can now conclude that a reflectance value of 50% surely corresponds to medium reflectance and that a reflectance of 40% more likely corresponds to medium reflectance than low reflectance. 8 DRDC Valcartier TM

25 Membership functions and fuzzy sets are equivalent because membership functions assign to each object a grade of membership ranging between zero and one [1],[12]. In other words, a membership value measures to what extent an object belongs to a set. Figure 1 shows a graphical representation of low, medium and high membership functions. Membership functions are also called possibility distributions, which refer to the possibility theory proposed by Zadeh in 1978 [18], [19]. Traditionally, with the fuzzy sets theory, one can infer to what extent an observation corresponds to a characteristic modeled with a membership function. With the membership functions of Figure 1, we can infer that a measured reflectance of 20% corresponds to low reflectance with a membership value of The possibility theory allows evaluating the possibility of observing a reflectance of 20% knowing that the reflectance is low. In [18] Zadeh explains the similarities and differences between fuzzy sets and possibility theories. From the fuzzy concepts young defined by ages (in years) ranging between 0 and 100, one can infer how John is young knowing that he is 28 years old. Possibility theory can be seen as a fuzzy restriction and allows evaluating the possibility of John being 28 years old knowing that he is young. 1 membership value 0,8 0,6 Low 0,4 medium High 0, reflectance (%) Figure 1. Membership functions or graphical representation of fuzzy sets Concerning the reflectance feature, instead of defining membership functions for three categories such as low, medium and high reflectance, it would be advantageous to define functions corresponding to real-world objects. For example, if the deciduous forest reflectance is about 46% in the near infrared band, its membership function could be something like the one in Figure 2. With such a membership function, the closer the measured reflectance is to 46%, the higher is the membership value enforcing the similarity or correspondence between the observation and the object deciduous forest. DRDC Valcartier TM

26 membership value 1 0,8 0,6 0,4 0,2 0 Deciduous forest reflectance (near infrared) Figure 2. Deciduous forest membership function for the reflectance in the near infrared Now, suppose that the typical reflectance of a forest cover observed with the Landsat ETM4 (near infrared) band is about 28% and that reflectance of a clear cut is about 24%. If a pixel of interest (+ sign, Figure 3) is characterized with a reflectance value of 23.0%, two membership values μ are obtained by fuzzy inference (Figure 4): μ forest = 0.62 μ clear cut = 0.85 Figure 3. Location of a pixel of interest in the Landsat ETM4 (NIR) band If a pixel within an image is characterized by a reflectance value of 23.0%, two membership values μ are obtained by fuzzy inference (Figure 4): μ forest = 0.62 μ clear cut = 0.85 So with a value of 23%, the similarity between the pixel and the clear cut class is greater than its similarity with the conifers. Fuzzy sets like those of Figure 4 can be 10 DRDC Valcartier TM

27 obtained from external knowledge such a literature review or by selecting samples on images. The membership value to one class resulting from fuzzy inference can be considered as a certainty value. This relation is explained next. Figure 4. Typical forest and clear cut reflectance in the NIR band Membership values and certainty Fuzzy inference is the process of comparing a fact (i.e. a pixel value) to a membership function. While membership functions are used to represent imprecise knowledge, membership values resulting from fuzzy inference correspond to the uncertainty of the pixel value corresponding to the modelled object. Another way to see membership values as certainty measurements if through rule-based reasoning. According to Orchard [20], the certainty of a conclusion (CF C ) when reasoning with a fuzzy rulebased system is computed by: CF C = CF R * CF F * S Eq. 1 where CF R is the certainty factor of the rule, CF F is the certainty factor of the fact and where S is the similarity or the result of the fuzzy inference. If CF R and CF F are set to one, then CF C entirely relies on the fuzzy inference value S. Analogies between fuzzy inference and rule-based systems are also presented in [21] and [8]. Figure 4 showed an example of fuzzy inference when using only one source (red band). It will be demonstrated next how fusion operators are necessary in a multisource context. The notion of the membership function shape will be discussed in Section 3.4 DRDC Valcartier TM

28 3.2 Fusion operators Consider the two objects, conifers and clear cut (Figure 4) of Section 3.1. Adding two sensors in the red and the short-wave IR (SWIR) we obtain four more membership functions (Figure 5). If an observation is r = [4.5 ; 23.0 ; 16.6] corresponding to a reflectance of 4.5% in red, 23% in NIR and 16.6% in SWIR, the membership values are: μ forest red = 0 μ forest NIR = 0.62 μ forest SWIR = 0.45 μ clear cut red = 0 μ clear cut NIR = 0.85 μ clear cut SWIR = 0 The appearance of the pixel in imagery is presented in Figure 6. Figure 5. Forest and clear cut reflectance modeled with membership functions in the red, near infrared and short-wave infrared 12 DRDC Valcartier TM

29 Figure 6. Location of a pixel of interest in the Landsat ETM3 (red), ETM4 (NIR) and ETM5 (SWIR 1) bands To what object (or class of object) is the observation corresponding? The answer will depend on the selection of the fusion operator. If all the sources are known as reliable, it is suggested to use the minimum membership value which is called the conjunctive fusion operator. The minimum membership value corresponds to the highest consensus or agreement between the sources. If one of the sources is not reliable, but we do not know which one, it is suggested to use the maximum membership value. This is the disjunctive fusion operator. Other fusion operators were proposed as compromises such as the adaptive and quantified adaptive operators. These fusion operators are described in detail in the subsequent sections. The concept of source reliability will be detailed in Chapter Conjunctive fusion or t-norms If all sources are reliable it means that we can trust each of the conclusions given by them. By selecting the minimum membership value, for each class, and then by preserving the maximum of those minima, we select the highest consensus between the sources. The conjunctive fusion is also known as triangular norm or t-norm. Several authors [22], [23] have proposed different types of t-norms (Table 1), but here we will work with the Zadeh s t-norm because it is simple to calculate and because it was proposed by the father of fuzzy set theory. T-norms have the properties of being commutative and associative, which means that no matter in which order the sources are combined the achieved result will be the same. DRDC Valcartier TM

30 Table 1. Example of some t-norms t-norm Symbol Origin min(a,b) T Z Zadeh a * b T P Probabilistic max(0, a + b-1) T L Lukasiewicz - min(a,b) for a = 1 or b = 1-0, otherwise T D drastic Conjunctive fusion, with the Zadeh t-norm, consists in keeping the minimum membership value for each class or hypothesis [19]. The final decision is then based on the maximum of these minima. An example of conjunctive fusion is given in Table 2. Conjunctive fusion is considered to be severe, and one primary effect of this is that many pixels can remain unclassified or unlabelled (as it is the case in this example). Table 2. Example of conjunctive fusion with the Zadeh s t-norm Sensor Hypotheses Conifers Clear cut Red 0 0 Near infrared SWIR max: min: Unclassified Conjunctive fusion corresponds to the AND logical operator. In a rule of the type IF condition 1 is true AND IF condition 2 is true THEN conclusion there are two premises which both need to be true for the conclusion to be activated. In analogy to the previous example (Figure 5, Table 2), the forest hypothesis can be expressed in a rule like IF reflectance in red is close to 3% AND IF reflectance in NIR is close to 28% THEN it is forest. According to the membership values of Table 2, the NIR premise is not true. Because one of the two premises is not true, the membership value to forest class is zero. Considering M classes and N sources (S), the conjunctive fusion operator can be mathematically expressed, for a given class m, by: μ C ( x) = min { μc S ( x)} Eq. 2 m n= 1,..., N m A object X is then labelled as class C m0 if and only if : μ C x) = max { μ ( )} Eq. 3 ( C 1,... m S m= M x m 0 n n 14 DRDC Valcartier TM

31 3.2.2 Disjunctive fusion or t-conorms If one or more of the sources is unreliable, it means that some sources are giving untrustable results. By selecting the maximum membership value, for each class, and then by preserving the maximum of those maxima, we are selecting the disjunctive fusion which corresponds to the logical OR. The disjunctive fusion is also known as triangular conorm, or t-conorm and is used to measure the union of two fuzzy sets [22], [23]. Several authors have proposed different types of t-conorms (Table 3), but here we will work with the Zadeh s t- conorm. T-conorms have also the properties of being commutative and associative. Disjunctive fusion, with the Zadeh s t-conorm, consists in keeping the maximum membership value for each class or hypothesis [19]. An example of disjunctive fusion is given in Table 4. Table 3. Example of some t-conorms t-norm Symbol Origin max(a,b) T Z Zadeh a+b a*b T P Probabilistic min(1, a + b) T L Lukasiewicz - max(a,b) for a *b = 0-1, otherwise T D drastic Table 4. Example of disjunctive fusion with the Zadeh s t-conorm Sensor Hypotheses Forest Clear cut Red 0 0 Near infrared SWIR max: max: (clear cut) The disjunctive fusion operator is considered to be permissive. For example, with the Zadeh t-conorm, the maximum membership value for each class is used. In fact, disjunctive fusion corresponds to the OR logical operator. In a rule of the type IF condition 1 is true OR IF condition 2 is true THEN conclusion there are two premises and only one of these premises needs to be verified for the conclusion to be activated. Analyzing the forest with the example of Table 4 is equivalent to using the rule IF reflectance in red is close to 3% OR IF reflectance in NIR is close to 20% THEN it is forest. However it is important to note that this rule does not respect the multispectral definition of the class forest. For example, the first premise could be DRDC Valcartier TM

32 verified but not the second. It could be that the reflectance in NIR is 40%. It that case, it could correspond to another object but the forest class would still be activated. This explains why the disjunctive fusion operator is considered as permissive. Generally, conjunctive fusion is used if all information sources are reliable and disjunctive fusion is used when some sources are unreliable without knowing which one-s is-are. We will see in Section 6.2 how reliability can be measured. Considering M classes and N sources (S), the disjunctive fusion operator can be mathematically expressed, for a given class m, by: μ C ( x) = max { μ ( x)} m C 1,..., m S n= N n Eq. 4 An object X is then labelled as class C m0 if and only if: μ ( C x) = max { μ ( x)} m 0 C 1,... m S m= M n Eq. 5 Another important point to note is that conjunctive and disjunctive fusion operators are associative and commutative, which means that the order in which sensors are fused does not influence the results (commutative property) and if some sensors are fused together (associative property) before the final fusion, the results will not change. Conjunctive and disjunctive fusion operators can also be used in complex rules of the form (IF condition 1 OR condition 2) AND (IF condition 3 OR condition 4) THEN conclusion. Faced with the drastic differences between the severe conjunctive and the permissive disjunctive operators, some authors [24] have proposed compromise fusion operators such as adaptive and quantified adaptive fusion. These operators select disjunctive, conjunctive or in-between fusion, depending on source agreement Adaptive fusion The adaptive fusion operator was proposed by Dubois and Prade [24] in order to use the advantages of conjunctive and disjunctive fusion while avoiding their negative aspects. The adaptive fusion operator is named adaptive because its behaviour varies depending on the consensus or agreement (h) between sources. Recall that it was mentioned previously that the conjunctive fusion operator corresponds to the highest level of consensus between sources. In other words, source agreement is computed the same way as the conjunctive fusion (shown in Table 5). The adaptive fusion operator (π ad ) is expressed by: π conj π = max,min(, ) ad conf π disj Eq. 6 h 16 DRDC Valcartier TM

33 where π conj and π disj are conjunctive and disjunctive fusion operators and h quantifies sensor agreement. Finally, π conj /h corresponds to the normalized conjunctive fusion operator and conf is the conflict between sources (conf = 1 - h). With this operator, the more conflict increases, the more the disjunctive fusion gains in importance. With a total conflict (h=0), the normalized conjunctive fusion (π conj /h) is not defined, so the disjunctive fusion is used. Table 5. Example of agreement (h) computing Sensors Classes C 1 C 2 C 3 C 4 C 5 S 1 m.v. m.v. m.v. m.v. m.v. S 2 m.v. m.v. m.v. m.v. m.v. S 3 m.v. m.v. m.v. m.v. m.v. S 4 m.v. m.v. m.v. m.v. m.v. Min min min min min h = max m.v. : membership value The adaptive fusion operator is presented with the same example from Table 2 and Table 4 with the ETM7 band (SWIR 2) added. First, the agreement (h) and the conjunctive fusion (A) are computed. Second, the normalized conjunctive fusion (B) and the disjunctive fusion (C) are computed. Third, the conflict (D) in computed. Fourth, the minimum between C and D is calculated (E) and finally, the result (F) is given by the maximum between B and E. In this example, the agreement is null so the conflict is total. Because the normalized conjunctive fusion is not defined, the disjunctive part of the rule is used so the class clear cut is chosen with a membership value of Table 6. Illustration of the adaptive fusion Sources Classes conifers clear cut ETM3 0 0 ETM ETM ETM A π conj (min) 0 0 h = 0 B π conj / h not defined n.d. C π disj (max) 0, D Conf (1-h) 1 1 E min(c,d) F max(e,b ) DRDC Valcartier TM

34 Equation (6) of this fusion operator shows that when the consensus (h) is null, the normalized conjunctive fusion (π conj /h) is not defined. In this case of total conflict, disjunctive fusion is used. When the agreement is total (conf = 0) then the normalized conjunctive fusion is used because min(conf, π disj ) gives a zero result and which gives full weight to the π conj /h operator. Because h is equal to one, the normalized conjunctive fusion is equal to the conjunctive fusion. The inconvenience of the adaptive fusion operator resides in the results when agreement is low. In this case, the normalization gives a high membership value to the winning class even if all sources do not give high membership to this class. Table 7 shows an example of adaptive fusion in presence of strong conflict. The fusion leads the class 2 conclusion with a membership value of 1 even if all sensors do not gives a membership value higher than 0.5. In this case, the membership value of 1 can hardly be considered as a certainty factor. Table 7. Example of the adaptive fusion in the presence of strong conflict Sensors C 1 C 2 Classes C 3 C 4 C 5 S S S S S S h A π conj (min) B π conj / h C π disj (max) D 1 - h E min(c,d) F max(e,b ) We can consider the maximum of the minima as a measure of consensus. This is a compromise between global consensus measurement (once for the whole image) and local consensus measurement for each class. Recall that an array of membership values such as that presented in Table 6 is obtained after fuzzy inference at the pixel level. This fuzzy inference is performed by comparing a pixel value to membership functions. Global consensus measurement would be taken by comparing fuzzy sets of the knowledge base. This would fix the fusion operator once for the whole image by either selecting disjunctive or conjunctive fusion, depending on the consensus. We would then lose the adaptive aspect of the fusion operator for the processing of the whole image. At the other extreme, consensus could be computed locally for each class and for each pixel. For the membership values of Table 6, five consensus values would be computed and for each class, the appropriate fusion operator would be used. This has the major disadvantage of reinforcing the least plausible classes and reducing weighting for the most plausible classes [25]. 18 DRDC Valcartier TM

35 Finally, even if the consensus is computed locally (at every pixel) but only once for all classes, the adaptive fusion operator comes to select either normalized conjunctive fusion or disjunctive fusion. The behaviour of this adaptive operator results in a decreasing number of unclassified pixels because where the conjunctive fusion operator is too severe to classify a pixel, the disjunctive operator allows one to assign a label. Another important aspect of the adaptive fusion operator is the loss of the associative property. This limitation can be avoided by fusing all pieces of information at the same time as in Table 6 or Table Quantified adaptive fusion Quantified adaptive fusion considers the number of reliable sources [24] evaluated from optimistic and pessimistic points of view. The optimistic evaluation of the number of reliable sources (n) is given by the class that is the most supported (1) while the pessimistic evaluation of the number of reliable sources (m) is given by the class that is characterized by the highest core( 2 ). Table 8 gives an example of membership values for five classes according to six sensors. Two classes (C 3 and C 5 ) have five membership values higher than 0 so n = 5. Class 5 has two membership values equal to 1, so m = 2. The mathematical expression of the quantified adaptive fusion is given by: conj = max π ( n) conj π adq, min(1 h( n), π ( m ) ( ) ) Eq. 7 h n B A The application of this rule almost always results in the use of the normalized conjunctive conj fusion (part A of the rule) because for at least one class there will be a value of π ( n) / h( n) equal to one. The exception for this happens if all membership values are null in which case the pixel remains unclassified. conj The different variable values of equation 7 according to Table 8 are as follow: π ( m) = 1, conj h ( n) = 0,2 and π 0, 2. ( n) = In the example of Table 8, the winning class, once membership values are sorted, is C 3 because it has the highest membership value among the two classes being supported by the highest number of sources. This is similar to working with an OWA (Ordered Weighted Averaging) fusion operator as proposed by Yager [26], [27], [28]. An OWA fusion operator is the weighted average of n membership values where n is context-specific. 1 The support, for one class, is given by the number of membership values greater than 0. 2 The core, for one class, is given by the number of membership values equal to 1. DRDC Valcartier TM

36 Sorted values Table 8. Illustration of the quantified adaptive fusion Sensors Classes C 1 C 2 C 3 C 4 C 5 S S S S S S B: A: max(a.b) Finally, the quantified adaptive fusion operator can be simplified and implemented as: conj π adq = max( π ( n) ) Eq. 8 which equates to conjunctive fusion between the classes that are most supported. Table 9 shows the sorted membership values of Table 6 fused with the quantified adaptive operator of equation 8. The class conifers wins with the membership value of Table 9. Illustration of the quantified adaptive fusion Classes conifers clear cut sorted membership values The whole quantified adaptive operator (equation 8) applied to the same example of conj conj Table 9 produced the following result: m = 0, π ( m) = 0, h ( n) = 0, 11 and π ( n) = 0, 11 so class conifers wins with the normalized membership value of 1. Among the fusion operators described here, the quantified adaptive operator is the one that generally gives the best results as it considers the class being observed by the greatest number of sensors. Adaptive and quantified adaptive fusion operators, contrary to conjunctive and disjunctive fusion, are not associative. This means that if 20 DRDC Valcartier TM

37 sources are fused two by two, the order of fusion will influence the results. This problem of non-associativity is avoided is sources are fused all at once as in the examples shown here (Table 6 and Table 8). During the description of the different fusion operators some terms, such as source agreement and reliability, were introduced. We will next explain in more detail their interrelation and their link to the possibility theory Sources agreement and possibility theory The adaptive and quantified adaptive fusion operators are often presented in the literature in the framework of the possibility theory. But as mentioned previously, in Section 3.1, a possibility value (π) is technically the same as a membership value (μ) [18], [29]. If we have a membership function such as the one of Figure 2 and we observe a pixel value of 40%, the membership value resulting from the fuzzy inference will be This membership value (μ) reflects the similarity between the observation and the hypothesis deciduous tree. In terms of possibility theory, the value of 0.42 corresponds to the possibility (π), knowing that the hypothesis deciduous tree is true, to observe the reflectance value of 40%. The main difference between a membership function and a possibility distribution is in the graphical representation. In membership functions such as those of Figure 5 the x-axis contains the possible values with which the different objects are modeled. On the opposite, in possibility distributions, the x-axis contains the set of hypothesis. The place where two possibility distributions intersect corresponds to where they give the same possibility value to the same hypothesis. It actually corresponds to the agreement or consensus (h). This reasoning is not the same with membership functions such as the ones in Figure 2 because the two functions correspond to different objects. In fact, the more two membership functions intersect the more there is confusion between the two objects. DRDC Valcartier TM

38 1 0,8 π 1 π 2 Possibility 0,6 0,4 0,2 h 0 set of hypotheses (Ω) Figure 7. Possibility distributions intersection (h) as a measure of agreement Computing the sensors agreement (h) the same way as the conjunctive fusion operator comes from the possibility theory [30], [31] in which h is equal to the possibility distributions intersection. In order to make the link between the conjunctive operator and the possibility distributions intersection, the membership values should be represented as in Figure 8 which represents the values of Table 7 with the x-axis corresponding to the highest of the minimum intersections. Figure 8. Sensors agreement for the observation leading to the membership values Table 7 When using the adaptive fusion operator, the computing of the agreement as the maximum of the minimum membership values implies the disadvantage of either selecting the conjunctive or the disjunctive fusion. If h > 0 then normalized conjunctive fusion is used. Otherwise disjunctive fusion is used. 22 DRDC Valcartier TM

39 A few times the term source reliability was mentioned but without measuring it. Actually, in [25] there is a relation between agreement and reliability. If agreement is total, sources must be reliable coincidental agreement is practically impossible with large data sets. On the other hand, if sources are in total conflict, there must be some unreliable sources. For the moment, leave this concept as is. Reliability will be described in detail in Section Fuzzy facts This section is presented as a reference on fuzzy reasoning with fuzzy facts. In the actual implementation of FuRII this topic is not implemented. The examples of Figure 4 and Figure 5 consider fuzzy knowledge, expressed with a membership function and a crisp fact, expressed as a single pixel value (a scalar). But the fact could also contain imprecision. For example, a pixel value could be converted into true reflectance after atmospheric corrections. The pixel reflectance would contain some imprecision due, in part, to the atmospheric model and this imprecision could be expressed by a fuzzy set such as in Figure 9. In this case, one simple way to compute fuzzy inference is to keep the maximum value of the fuzzy sets intersection [32]. In this case, according to the fuzzy fact reflectance is about 9.5% (Figure 9), the membership value to the class clear cut would be 0.6. This way of computing inference is fast and shows that with higher imprecision of the fuzzy fact, the result of the fuzzy inference can be higher. Figure 10 shows the fuzzy inference with a fact that contains more imprecision and which leads to a higher membership value of Membership value 1 0,9 clear cut 0,8 0,7 fuzzy fact 0,6 0,5 0,4 0,3 0,2 0, Reflectance - red (%) Figure 9. Fuzzy inference with a fuzzy fact DRDC Valcartier TM

40 Membership value 1 clear cut 0,9 0,8 fuzzy fact 0,7 0,6 0,5 0,4 0,3 0,2 0, Reflectance - red (%) Figure 10. Fuzzy inference with a more imprecise fuzzy fact In [20] and [33] a fuzzy inference method that takes into account the possibility of a piece of knowledge and the possibility of its contradiction is described. This method is based on the use of two measures that are possibility (P) and necessity (N). In the example of Figure 4, the computed membership value (also called similarity) to the class forest is of 0.6. The possibility is an optimistic evaluation of the similarity while the necessity is a pessimistic evaluation. So the true similarity value (S) is located somewhere between P and N: P(x) S(x) N(x) Orchard [20] describes a method that considers the intersection between the complement of the knowledge and the fuzzy fact. Figure 11 revisits the example of Figure 9 and also includes the representation of the complement of the knowledge concerning the clear cut knowledge. 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0 Membership value p(k F) p(k' F) Reflectance - red (%) knowledge (k) know. Comp. (k') fuzzy fact (F) Figure 11. Fuzzy knowledge (red), knowledge complement (green) and fuzzy fact (blue) The similarity (S) between a fuzzy fact (F) and a fuzzy knowledge (K) is given by: S = P(K F) if N(K F) > 0,5 24 DRDC Valcartier TM

41 S = [N(K F) + 0.5] * P(K F) otherwise Eq. 9 where P(K F) is the possibility, result of the fuzzy inference between the fuzzy fact (F) and the fuzzy knowledge (K) given by: P(K F) = max[min(μ K, μ F )] Eq. 10 P(K F) corresponds to the maximum of the intersections. The necessity N(K F) is given by: N(K F) = 1 - P(K F) Eq. 11 where P(K F) is the maximum of the intersections between F and the complement of knowledge (K ): P(K F) = max[min(μ K, μ F )] Eq. 12 In other words, the necessity of an event is equal to the impossibility of its contrary. This method of computing the certainty or the similarity between fuzzy fact and fuzzy knowledge makes it possible to take into account the quantity of imprecision in both facts and knowledge. In the example of Figure 11, the possibility of the clear cut, P(K F), and the possibility of its contrary, P(K F) are both equal to 0.6. The necessity N(K F) is 0.4 because: N(K F) = 1 - P(K F) = = 0.4 Given that necessity is lower than 0,5, the similarity is obtained by: S = [N(K F) + 0.5] * P(K F) = ( ) * 0.6 = 0.54 For Figure 10, the similarity is: S = [N(K F) + 0.5] * P(K F) = ( ) * 0.75 = The consideration of fuzzy facts in fuzzy inference is of particular interest when analyzing an image at the polygonal level. Instead of processing every pixel individually, a region, or neighbourhood, can be analyzed by its histogram or its mean value and its variance. In the case of a pixel-based classification, the imprecision attached to each pixel needs to be modeled in order to use such a fuzzy inference method. 3.4 Membership function construction Triangular membership functions such as those of Figure 4 and Figure 5 correspond to the simplest model for representing an expression such as To what extent the reflectance value r corresponds to the class n?. The closer that the observed value is to the central value, the greater the membership value for that fuzzy set. Such a rule can also be represented with Gaussian and trapezoidal functions (Figure 12). The most accessible source for building membership functions is the data itself which is used for the classification. By identifying samples on imagery, statistics can be computed and statistical distribution parameters and histograms can be used to compute membership functions [25], [34]. The mean value can determine the center DRDC Valcartier TM

42 of a membership function and the steepness of the curves can be guided by the standard deviation. Trapezoidal functions can be used to give more importance to central values. 1 0,8 membership value 0,6 0,4 0,2 0 feature value Figure 12. Three types of membership functions: triangle, trapeze and Gaussian The membership functions of Figure 12 assume that data is normally distributed. But take a look at the data distribution of Figure 13 (histogram data in red). As this data is not normally distributed, the Gaussian function does not reflect the data distribution. In this case, it would be more suitable to use a histogram-based membership function. This methodology has the advantage of modeling any type of distributions but lacks in data generalization. However, the use of histogram-based membership functions is safer than assuming a normal distribution. 26 DRDC Valcartier TM

43 Figure 13. Example of histogram-based membership function There are other shapes such as the S ( ) and Z ( ) membership functions (Figure 14). They can be for modeling rule such as IF a value is greater than x THEN or IF a value is lower than x THEN. These types of membership functions should be used carefully in order to avoid saturation in the decision process. Figure 15 shows an example where four classes are modeled with S membership functions. Given an observation x=37, all four classes have a membership value of one. Figure 14. S and Z membership functions DRDC Valcartier TM