Supplementary Material Accompanying Geometry Driven Semantic Labeling of Indoor Scenes

Transcription

1 Supplementary Material Accompanying Geometry Driven Semantic Labeling of Indoor Scenes Salman H. Khan 1, Mohammed Bennamoun 1, Ferdous Sohel 1 and Roberto Togneri 2 School of CSSE 1, School of EECE 2 The University of Western Australia, 35 Stirling Highway, Crawley, WA 6009, Australia 1 Disintegration of Higher Order Energies In this appendix, we will show how the higher order energy potentials can be minimized using graph cuts. Since, graph cuts can efficiently minimize submodular functions, we will transform our higher order energy function (Eq. 9 to a submodular second order energy function. For the case of both αβ-swap and α- expansion move making algorithms, we will explain this transformation and the process of optimal moves computation 1. All of the previously defined notations are used in the same context and all of the newly introduced symbols are defined in this section. The function that accounts for the number of disagreeing nodes in a clique is defined as: n l (y c = i c w l i 1 yi=l The function 1 yi=l is a zero-one indicator function that returns a unit value when y i = l. We suppose here that weights are symmetric for all labels l L i.e., wi l = w i. Further, for our implementation we set w i = 1 i c. This setting satisfies the required constraints for these parameters, i.e., wi l 0 and wi l = #c l L. We define a summation function that adds the weights for a subset s of c, i c W (s = i s w l i = #s l L. 1.1 Disintegration of Higher Order Energies to Second Order Sub-Modular Energies for Swap Moves Suppose, in a clique c, the locations of the active nodes is represented by a set of indices c a. The nodes which remain inactive during the move making process 1 The development of this section is similar to [13]. We also used the same notation - wherever possible - to allow the reader to easily sort out differences and commonalities.

2 2 S. H. Khan, M. Bennamoun, F. Sohel and R. Togneri are termed as passive nodes. Their locations are denoted by c a = {c \ c i c a }. The corresponding set of available moves to the swap move making algorithm are encoded in the form of a vector t ca. For the sake of simple demonstration, let us focus on the two class labeling problem i.e., l {0, 1}. The induced labeling is the combination of the old labeling for the inactive nodes and the new labeling for the active nodes i.e., yc n = y c a T αβ (yc a, t ca. If yc n denotes the new labeling induced by move t ca and yc denotes the old labeling, we can define the energy of move for an αβ swap as: ψ m c (t ca = ψ c (y n c = ψ c (y c a T αβ (y c a, t ca = min {λ max (λ max λ l exp( W (c n l(y ca T αβ(y ca,t ca l L Q } l = min l L {λ max (λ max λ α exp( W (c nm 0 (t ca, λ max (λ max λ β exp( W (c c a+n m 0 (t ca Q β }, where, W (c a = n m 0 (t ca + n m 1 (t ca. The minimization operation in the above equation can be replaced by defining a piecewise function: λ max (λ max λ α exp( W (c nm 0 (t ca if n m 0 (t ca > ϱ αβ ( W (c W (c ca Q β ψc m log( λmax λα λ (t ca = max λ β λ max (λ max λ β exp( W (c c a+n m 0 (t ca Q β if n m 0 (t ca < ϱ αβ ( W (c W (c ca Q β log( λmax λα λ max λ β where, ϱ αβ = QαQ β +Q β. The function n m l (t c a is defined as: n m l (t ca = i c a w i δ l (t i. From Theorem 1 in [1], the energy defined above can be transformed to the submodular quadratic pseudo-boolean function with two binary meta variables. In this form the αβ-swap algorithm can be used for minimizing the energy function. 1.2 Disintegration of Higher Order Energies to Second Order Sub-Modular Energies for Expansion Moves Suppose, in a clique c, the location of the nodes with label l is represented by a set of indices c l. The current labeling solution is denoted by y c. If the dominant label is denoted by d L in the current labeling y c is, s.t W (c d > W (c Q d where d α,

3 Geometry Driven Semantic Labeling of Indoor Scenes 3 there must be one dominant label: Q a + Q b < W (c a b L, ψ m c (t c = ψ c (T α (y c, t c = min {λ max (λ max λ α exp( l L λ max (λ max λ d exp( w it i i c, W (c i c w it i Q d }. The minimization operator in the above function can be replaced by a piecewise function: λ max (λ max λ α exp( nm 0 (t c if n m 0 (t c > ϱ αd ( W (c ψc m log( λmax λα λ (t c, t cd = max λ d λ max (λ max λ d exp( W (c nm 0 (t c d Q d if n m 0 (t c < ϱ αd ( W (c log( λmax λα λ max λ d where, ϱ αd = QαQ d +Q d and function n m l (t c is defined as: n m l (t c = i c w i δ l (t i. From Theorem 2 in [1], the energy defined above can be transformed to the submodular quadratic pseudo-boolean function with two binary meta variables. In this form the α-expansion algorithm can be used for minimizing the energy function. 2 Plane Detection Algorithm Algorithm 1 shows our region growing algorithm for depth based segmentation. Note that, point normals are found by local plane fitting and Singular Value Decomposition (SVD. In step 2, k sm is the smoothing kernel and N sm is the smooth version of local normal image. In step 23, we also check if n r idx already been replaced by another region or not. If n r idx is found to be previously replaced, we use the updated value. In the following section, we list some example results of our algorithm. 2.1 More Example Results

4 4 S. H. Khan, M. Bennamoun, F. Sohel and R. Togneri Algorithm 1 Region Growing Algorithm for Depth Based Segmentation Input: Point cloud = {P}, Depth map = {D}, RGB image = {I}, Edge matching threshold e th, Normalized boundary matching threshold b th Output: Labeled planar regions = {R} 1: Calculate point normals: {N} F normal (D 2: Remove inconsistencies by low-pass filtering: {N sm} N k sm 3: Cluster 3D points with similar normal orientations: {N clu } F k means (N sm 4: Initialize: R N clu 5: Line segment detector: {L} F LSD(I 6: Diffused line map: {L sm} L k sm 7: Identify planar regions with missing depth values: {M} F holes (N clu, D 8: Find adjacency relationships for each cluster in N clu : A clu 9: Identify all unique neighbors of clusters in M: U nb 10: From U nb, separate correct and faulty clusters into N cor and N inc respectively 11: Initialize available cluster list: L avl N cor 12: Initialize label propagation list: L prp 13: while list L avl is not empty do 14: Randomly draw a cluster from available N cor: r idx 15: Identify r idx neighbors (N r idx with faulty depth values using A clu and M 16: for each neighbor n r idx in N r idx do 17: Find mutual boundary (b m of r idx and n r idx 18: Calculate edge strength at b m using L sm: e str 19: Calculate normalized boundary matching cost: b str = b m/ Area of n r idx 20: if e str < e th b str > b th then add add 21: n r idx N cor, n r idx L avl rem rem 22: r idx L avl, n r idx N inc 23: Update L prp with r idx and n r idx rem 24: r idx L avl 25: for any leftover clusters in N inc do 26: Randomly draw a cluster from available N inc: r idx 27: Execute similar steps (from line 15 to 24 for r idx 28: Update R according to L prp 29: return {R}

5 Geometry Driven Semantic Labeling of Indoor Scenes 5 Fig. 1. Comparison of our algorithm (last row with [2] (middle row is shown. Note that the maroon color in middle row shows non-planar regions (but not in the last row where all regions are approximated by planes. The last row shows detected planes averaged over super-pixels.

6 Bibliography [1] Kohli, P., Torr, P.H., et al.: Robust higher order potentials for enforcing label consistency. IJCV 82(3, (2009 [2] Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: ECCV, pp Springer (2012