The family of hierarchical classes models: A state-of-the-art overview. Iven Van Mechelen, Eva Ceulemans, Iwin Leenen, and Jan Schepers

Transcription

1 The family of hierarchical classes models: state-of-the-art overview Iven Van Mechelen, Eva Ceulemans, Iwin Leenen, and Jan Schepers University of Leuven

2 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

3 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

4 . Data I I I N-way N-mode data array... N X with entries x i i... i N binary, rating-valued, real-valued array-conditionality Note: We leave aside: () structurally incomplete data () multiblock data

5 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

6 . Models. Context N-way Tucker model: x a g e N N N n i i... i =... Π i p p p... p + i i... i p = p = p = n= N n n N N n In n with denoting the n th component matrix G and denoting the core array

7 x a g e N N N n i i... i =... Π i p p p... p + i i... i p = p = p = n= N n n N N options:. identity matrix for,,, N- component matrices Tucker N- N-way Tucker N- model Tucker G n. superdiagonal or superidentity N-way CNDECOM/RFC

8 models studied in research group: () primary constraint: n binary for,,, N component matrices overlapping clustering for n th mode () operator: Σ or Max (Min) x a g e N N N n i i... i =... Π i p p p... p + i i... i p = p = p = n= N n n N N x a g e... Max Max... Max N N n i i i = i p p p... p i i... i p = p = p = Π + n= N n n N N N unifying model: Van Mechelen & Schepers (submitted)

9 focus of present talk: n () binary for all component matrices overlapping clustering for all modes () operator: Max G (3) takes values in +

10 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

11 . Models. Hierarchical classes models.. Basic model two-way two-mode ( ) x = Max Max a a g + e i i i p i p p p i i p = p = reconstructed data xˆi i

12 ( ) xˆ = Max Max a a g i i i p i p p p p = p = three cases: G. binary (/) is identity matrix G xˆ = Max a a ( ) i i i p i p p= ( ) i i = i p i p p= xˆ a a (Boolean sum) G. rating-valued (i.e., values in {,,, V} ) G 3. real-valued (i.e., values in ) +

13 . Models. Hierarchical classes models.. Justification of the Max operator Three (interrelated reasons):. representation by component matrices of quasi-orders that can be naturally defined on each of the modes. substantive interpretation of decomposition rule 3. decision rule for overlapping biclusters (triclusters etc.)

14 . Models. Hierarchical classes models.. Justification of the Max operator Three (interrelated reasons):. representation by component matrices of quasi-orders that can be naturally defined on each of the modes. substantive interpretation of decomposition rule 3. decision rule for overlapping biclusters (triclusters etc.)

15 binary case (G is /) illustrative example: 7 6 feature by object reconstructed data O O O 3 O 4 O 5 O 6 F F ˆ = X F 3 F 4 F 5 F 6 F 7 implication relation among features e.g., F F 5, F 3 F 4, F 5 F 7

16 similar implication relation among objects quasi-order: reflexive, transitive implies partial order on resulting equivalence classes graphical representation: (quasi-) Hasse diagram F 7 F 5, F 6 F F F 3, F 4 HIERRCHICL CLSSES!!!

17 representation of quasi-orders by component matrices: quasi-order on {F,, F 7 } quasi-order on {F,, F 7 } as implied by ˆX = as implied by O O O 3 O 4 O 5 O 6 3 F F F F ˆ = X F 3 F 4 F 3 F 4 = F 5 F 5 F 6 F 6 F 7 F 7

18 representation of quasi-orders by component matrices: quasi-order on {F,, F 7 } quasi order-on {F,, F 7 } as implied by ˆX = as implied by 3 F F 7 F F 5, F 6 F F F 3, F 4 F 3 F 4 F 5 F 6 F 7 =

19 representation of quasi-orders by component matrices: quasi-order on {O,, O 6 } quasi-order on {O,, O 6 } as implied by ˆX = as implied by O O O 3 O 4 O 5 O 6 33 F O F O ˆ = X F 3 F 4 O 3 O 4 = F 5 5 F 6 6 F 7

20 representation of quasi-orders by component matrices: quasi-order on {O,, O 6 } quasi-order on {O,, O 6 } as implied by ˆX = as implied by 33 O O O 5 O 6 O O, O 3 O 4 O 3 O 4 O 5 O 6 =

21 Note: (quasi) Hasse diagram as implied by F 7 F 5, F 6 F F F 3, F 4

22 Note: (quasi) Hasse diagram as implied by O 5 O 6 O O, O 3 O 4

23 Note: (quasi) Hasse diagrams as implied by and F 7 F 5, F 6 F F F 3, F 4 O 5 O 6 O O, O 3 O 4

24 Note: F 7 F 5, F 6 F F F 3, F 4 O O, O 3 O 4 O 5 O 6

25 Note: comprehensive graphical representation of HICLS model!!! F 7 F 5, F 6 F F F 3, F 4 O O, O 3 O 4 O 5 O 6

26 rating- or positively real-valued case illustrative example: 7 6 feature by object reconstructed data O O O 3 O 4 O 5 O 6 F 4 4 F ˆ = X F 3 F 4 F F F generalized implication relation among features e.g., F F 5, F 3 F 4, F 5 F 7

27 generalized implication relation is again a quasi-order similar quasi-order on objects quasi-orders can be graphically represented by (quasi-) Hasse diagrams e.g., F 7 F 5, F 6 F F F 3, F 4 quasi-orders are to be represented by component matrices

28 representation of quasi-orders by component matrices: quasi-order on {F,, F 7 } quasi-order on {F,, F 7 } as implied by Xˆ = as implied by O O O 3 O 4 O 5 O 6 3 F 4 4 F F F ˆ = X F 3 F 4 F 3 F 4 = F F 5 F F 6 F F 7

29 Note: G F O 4 F O 5 F 3 O 3 3 F 4 O 4 F 5 O 5 F 6 O 6 F 7

30 Note: comprehensive graphical representation of HICLS model F 7 F 5, F 6 F F F 3, F O O, O 3 O 4 O 5 O 6

31 Note: Max-operator can be shown to be only operator that allows representations of quasi-orders!!!

32 . Models. Hierarchical classes models.. Justification of the Max operator Three (interrelated reasons):. representation by component matrices of quasi-orders that can be naturally defined on each of the modes. substantive interpretation of decomposition rule 3. decision rule for overlapping biclusters (triclusters etc.)

33 consider binary (/) case assume (reconstructed) data pertain to person by problem failure/success: x ˆ = : i i x ˆ = : i i person i fails for problem i person i succeeds for problem i underlying mechanism: -(latent) solution strategies - person i may master strategy p or not: a = or - strategy p may be suitable for solving problem i or not: i p a = or i p

34 xˆ = Max a a ( ) i i i p i p p= ( ) xˆ = a a i i i p i p p= ( ) xˆ = iff p: a = and a = i i i p i p Note: person i succeeds iff there is at least one strategy p that for problem i person i masters and that is suitable for solving problem i existential quantifier disjunctive model

35 . Models. Hierarchical classes models.. Justification of the Max operator Three (interrelated reasons):. representation by component matrices of quasi-orders that can be naturally defined on each of the modes. substantive interpretation of decomposition rule 3. decision rule for overlapping biclusters (triclusters etc.)

36 consider positively real-valued (reconstructed) object by variable data consider the following two models: ( ) xˆ a a g = i i i p i p p p p = p = ( ) xˆ = Max Max a a g i i i p i p p p p = p =

37 ( ) xˆ a a g = i i i p i p p p p = p = ( ) xˆ = Max Max a a g i i i p i p p p p = p = O V O V 3 3 O 3 V 3 G O 4 V 4 O 5 V 5 O 6 V 6 V 7

38 O O ( ) xˆ a a g = ( xˆ ) i i = Max Max a i p a i p g p p p = p= i i i p i p p p p = p = O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

39 ( ) xˆ a a g = i i i p i p p p p = p = ( ) xˆ = Max Max a a g i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

40 ( ) xˆ a a g = i i i p i p p p p = p = ( ) xˆ = Max Max a a g i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

41 ( ) xˆ a a g = i i i p i p p p p = p = ( ) xˆ = Max Max a a g i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

42 ( ) xˆ a a g = i i i p i p p p p = p = ( ) xˆ = Max Max a a g i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

43 ( ) xˆ a a g = i i i p i p p p p = p = ( ) xˆ = Max Max a a g i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

44 ( ) xˆ a a g = ( xˆ ) i i = Max Max a i p a i p g p p p = p= i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O? O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

45 ( ) xˆ a a g = ( xˆ ) i i = Max Max a i p a i p g p p p = p= i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

46 ( ) xˆ a a g = ( xˆ ) i i = Max Max a i p a i p g p p p = p= i i i p i p p p p = p = V V V 3 V 4 V 5 V 6 V 7 O O O 3 O 4 O 5 O 6 O O O 3 O 4 O 5 O 6 G 3 V V V 3 V 4 V 5 V 6 V 7

47 Note: taxonomic overview of biclustering: see Van Mechelen, Bock, & De Boeck (4)

48 . Models. Hierarchical classes models..3 The HICLS family key features:. operator = Max x a g e... Max Max... Max N N n i i i = i p p p... p i i... i p = p = p = Π + n= N n n N N N. n all / 3. entries of G 4. representation of quasi-orders by + n

49 unconstrained models: G / N = G =I (disjunctive) hierarchical classes model (De Boeck and Rosenberg, 988) Notes: * dual conjunctive model with Min-operator: Van Mechelen, De Boeck & Rosenberg, 995) * stochastic extension: Bayesian HICLS (Leenen, Van Mechelen, Gelman & De Knop, submitted)

50 N=3 * G superidentity (CNDECOM/RFC case): INDCLS (Leenen, Van Mechelen, De Boeck & Rosenberg, 999) * G general: - Tucker3 HICLS (Ceulemans, Van Mechelen & Leenen, 3) - Tucker HICLS (Ceulemans & Van Mechelen, 4) G rating-valued (N =) HICLS-R (Van Mechelen, Lombardi & Ceulemans, conditionally accepted) Note: option to put limit on number of distinct values in G ( optimal coarsening )

51 entries G + real-valued HICLS (Schepers & Van Mechelen, submitted) constrained models taxonomy of constraints (Ceulemans, Van Mechelen & Kuppens, 4), based on:. locus: component matrices core. nature: value (e.g., values from previous study) vs. structure (e.g., Guttman scale, consecutive ones, decomposability, lower bound on number of ones, etc.) 3. extent: full vs. partial 4. use of external information: no vs. yes Note: varying amount of required algorithmic adaptations

52 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

53 3. Research topics 3. Models mathematical properties (e.g., rank) relevant structures: Boolean algebra and tropical semiring ( +,Max, ) Notes:. differences between these structures and common algebras over e.g., row, column, and decomposition (Schein) rank may differ e.g., only invertible Boolean matrices are permutation matrices. even with Σ-operator rank issues considerably change if one or n more of the are constrained to be / ( -/)

54 identifiability / uniqueness: - only invertible Boolean matrices are permutation matrices less identifiability problems than for common N-way Tucker models in general only permutational freedom rather than full rotational freedom - remaining amount of nonuniqueness pertains to particular decompositions - part of remaining nonuniqueness removed by requirement of representation of quasi-order - at this moment only sufficient condition for uniqueness in / case (Van Mechelen, De Boeck & Rosenberg, 995; Ceulemans & Van Mechelen, 3) HICLS/INDCLS: presence of all component patterns that contain single Tucker3 HICLS: idem + no core slice is subset of union of other core slices

55 model interrelations (Ceulemans & Van Mechelen, 5): Tucker () Tucker (B) Tucker (C) Tucker (B) Tucker (C) Tucker (BC) Tucker3 (BC) RFC (BC)

56 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

57 3. Research topics 3. Estimation L = xˆ x i i... in i i... i i, i,..., i N ( ˆ ) i i i i i i L = x x... N... i, i,..., i N N N Note: in pure / case L = L evaluation criteria for algorithms: - primary: goodness of fit;?global optimum - secondary: goodness of recovery

58 types of algorithms under study: - alternating least L / L - simulated annealing (Ceulemans, Van Mechelen & Leenen, submitted) - MCMC (Metropolis) (Leenen, Van Mechelen, Gelman, & De Knop, submitted)

59 algorithmic issues - parametrization of solution space (and partitioning of parameter vector, if applicable) (Schepers, Van Mechelen & Ceulemans, in press) - starts * number * nature: rational from data / rational from data + noise / random from data / purely random (Ceulemans, Van Mechelen & Leenen, submitted) - iterative process - choice metaparameters in simulated annealing and MCMC - greedy vs. branch and bound in conditional updating of alternating least L p algorithms (Leenen & Van Mechelen, )

60 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

61 3. Research topics 3. Model selection and model checking issues - model type - rank - correct representation quasi-orders - assumptions about error process types of methods under study - scree-test based approaches (including extended convex hull methods (Leenen & Van Mechelen, ; Ceulemans & Van Mechelen, 5; see also Ceulemans & Kiers, in press) - pseudo IC approach (Ceulemans & Van Mechelen, 5) - Bayesian approaches with posterior predictive checks (Leenen, Van Mechelen, Gelman & De Knop, submitted)

62 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

63 3. Research topics 3.4 Quantification of uncertainty issues: - telling apart strong and weak parts in obtained representations; confidence intervals - not only single parameters; also aspects derived from multiple parameters (e.g., classes, hierarchical relations) - confidence intervals in / case are tricky! types of methods under study - nonparametric Bootstrap approach - Bayesian approach based on simulated posterior (Leenen, Van Mechelen, Gelman & De Knop, submitted)

64 Overview of the talk. data. models. context. hierarchical classes models.. basic model.. justification of the Max operator..3 the HICLS family 3. research topics 3. models 3. estimation 3.3 model selection and model checking 3.4 quantification of uncertainty 4. references

65 4. References Ceulemans, E., & Kiers, H..L. (in press). Selecting among three-mode principal component models of different types and complexities: numerical convex hull based method. British Journal of Mathematical and Statistical sychology. Ceulemans, E., & Van Mechelen, I. (3). Uniqueness of N-way N- mode hierarchical classes models. Journal of Mathematical sychology, 47, Ceulemans, E., Van Mechelen, I., & Leenen, I. (3). Tucker3 hierarchical classes analysis. sychometrika, 68, Ceulemans, E., & Van Mechelen, I. (4). Tucker hierarchical classes analysis. sychometrika, 69, Ceulemans, E., & Van Mechelen, I. (5). Hierarchical classes models for three-way three-mode binary data: Interrelations and model selection. sychometrika, 7,

66 Ceulemans, E., Van Mechelen, I., & Leenen, I. (submitted). The local minima problem in hierarchical classes analysis: n evaluation of a simulated annealing algorithm and various multistart procedures. De Boeck,., & Rosenberg, S. (988). Hierarchical classes: Model and data analysis. sychometrika, 53, Leenen, I., Van Mechelen, I., De Boeck,., & Rosenberg, S. (999). INDCLS: Individual differences hierarchical classes analysis. sychometrika, 64, 9 4. Leenen, I., Van Mechelen, I., & De Boeck,. (999). generic disjunctive/conjunctive decomposition model for n-ary relations. Journal of Mathematical sychology, 43, -. Leenen, I., & Van Mechelen, I. (). n evaluation of two algorithms for hierarchical classes analysis. Journal of Classification, 8, Leenen, I., Van Mechelen, I., Gelman,., & De Knop, S. (submitted). Bayesian hierarchical classes analysis. Schepers, J., & Van Mechelen, I. (submitted). The real-valued model of hierarchical classes.

67 Schepers, J., Van Mechelen, I., & Ceulemans, E. (in press). Three-mode partitioning. Computational Statistics and Data nalysis. Van Mechelen, I., De Boeck,., & Rosenberg, S. (995). The conjunctive model of hierarchical classes. sychometrika, 6, Van Mechelen, I., Bock, H.-H., & De Boeck,. (4). Two-mode clustering methods: structured overview. Statistical Methods in Medical Research, 3, Van Mechelen, I., Lombardi, L., & Ceulemans, E. (conditionally accepted). Hierarchical classes modeling of rating data. sychometrika. Van Mechelen, I., & Schepers, J. (submitted). unifying model involving a categorical and/or dimensional reduction for multimode data.