Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications

Transcription

1 Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications Chong-Yung Chi Institute of Communications Engineering & Department of Electrical Engineering National Tsing Hua University, Taiwan Web: Invited tutorial talk at ITCOM Summer School, NTHU, Taiwan, 2017/8/1-4 Acknowledgment: My post doctor, Dr. Chia-Hsiang Lin for preparing slides of Part II. My Ph.D. student, Yao-Rong Syu for some slides preparation. 1 / 122

2 Outline 1 Part I: Fundamentals of Convex Optimization 2 Part II: Application in Hyperspectral Image Analysis: (Big Data Analysis and Machine Learning) 3 Part III: Application in Wireless Communications (5G Systems) Subsection I: Outage Constrained Robust Transmit Optimization for Multiuser MISO Downlinks Subsection II: Outage Constrained Robust Hybrid Coordinated Beamforming for Massive MIMO Enabled Heterogeneous Cellular Networks 2 / 122

3 Optimization problem Optimization problem: minimize subject to f(x) x C (1) where f(x) is the objective function to be minimized and C is the feasible set from which we try to find an optimal solution. Let x = arg min f(x) (optimal solution or global minimizer) (2) x C Challenges in applications: Local optima; large problem size; decision variable x involving real and/or complex vectors, matrices; feasible set C involving generalized inequalities, etc. Computational complexity: NP-hard; polynomial-time solvable. Performance analysis: Performance insights, properties, perspectives, proofs (e.g., identifiability and convergence), limits and bounds. 3 / 122

4 Convex sets and convex functions-1 Affine (convex) combination: Provided that C is a nonempty set, x = θ 1x θ Kx K, x i C i (3) is called an affine (a convex) combination of x 1,..., x K (K vectors or points of a set) if θi = 1, θi R (θi R+), K Z+. K i=1 Affine (convex) set: C is an affine (a convex) set if C is closed under the operation of affine (convex) combination; an affine set is constructed by lines; a convex set is constructed by line segments. Conic set: If θx C for any θ 0 and any x C, then C is a cone; a cone is constructed by rays starting from the origin; the linear combination (3) is called a conic combination if θ i 0 i; C is a convex cone if C is closed under the operation of conic combination. 4 / 122

5 Convex sets and convex functions-2 y Left plot: 1 +y 2 / B, implying that B is not a convex set; right plot: f(x) 2 is a convex function (by (4)). 5 / 122

6 Convex Set Examples Left plot: An ellipsoid centered at x c with semiaxes λ 1, λ 2, and an Euclidean ball centered at x c with radius r > max{ λ 1, λ 2} in R 2 ; right plot: Second-order cone C = { (x, t) R n+1 x 2 t } in R 3. 6 / 122

7 Convex Set Examples Left plot: convex cone formed by C = {x 1, x 2} via conic combinations, i.e., the smallest conic set that contains C, called the conic hull of C; right plot: conic hull of another nonconvex set C (star). 7 / 122

8 Convex sets and convex functions-3 Let A = {a 1,..., a N } R M. Affine hull of A (the smallest affine set containing A) is defined as { N N } aff A = x = θ ia i θ i = 1, θ i R i i=1 i=1 = { x = Cα + d α R p} (affine set representation) where C R M p is of full column rank, d aff A, and affdim A = p min{n 1, M}. A is affinely independent (A.I.) with affdim A = N 1 if the set {a a i a A, a a i} is linearly independent for any i; moreover, aff A = {x b T x = h} H(b, h) (when M = N) is a hyperplane, where (b, h) can be determined from A (closed-form expressions available). 8 / 122

9 Convex sets and convex functions-4 Figure 1: An illustration in R 3, where conv{a 1, a 2, a 3 } is a simplex defined by the shaded triangle, and conv{a 1, a 2, a 3, a 4 } is a simplex (and also a simplest simplex) defined by the tetrahedron with the four extreme points {a 1, a 2, a 3, a 4 }. 9 / 122

10 Convex sets and convex functions-5 Let A = {a 1,..., a N } R M. Convex hull of A (the smallest convex set containing A) is defined as conv A = { x = N N θ ia i i=1 i=1 } θ i = 1, θ i R + i aff A conv A is called a simplex (a polytope with N vertices) if A is A.I. When A is A.I. and M = N 1, conv A is called a simplest simplex of N vertices (i.e., a 1,..., a N ); aff (A \ {a i}) = H(b i, h i), i I N = {1,..., N} is a hyperplane, where the N boundary hyperplane parameters {(b i, h i), i = 1,..., N} can be uniquely determined from A (closed-form expressions available) and vice versa; this geometry fact plays an essential role in hyperspectral unmixing (a cutting edge research in remote sensing) to be introduced in Part II. 10 / 122

11 Convex sets and convex functions-6 Convex function: f is convex if dom f (the domain of f) is a convex set, and for all x, y dom f, f(θx + (1 θ)y) θf(x) + (1 θ)f(y), 0 θ 1. (4) f is concave if f is convex. Some Examples of Convex Functions An affine function f(x) = a T x + b is both convex and concave on R n. f(x) = x T Px + 2q T x + r, where P S n, q R n and r R is convex if and only if P S n +. Every norm on R n (e.g., p for p 1) is convex. Linear function f(x) = Tr(AX) (where Tr(V) denotes the trace of a square matrix V) is both convex and concave on R n n. f(x) = log det(x) is convex on S n / 122

12 Convex sets and convex functions-2 y Left plot: 1 +y 2 / B, implying that B is not a convex set; right plot: f(x) 2 is a convex function (by (4)). 12 / 122

13 Ways Proving Convexity of a Function First-order Condition Suppose that f is differentiable. f is a convex function if and only if dom f is a convex set and f(y) f(x) + f(x) T (y x) x, y dom f. (5) Second-order Condition Suppose that f is twice differentiable. f is a convex function if and only if dom f is a convex set and 2 f(x) 0 (positive-semidefinite), x dom f. (6) Epigraph The epigraph of a function f : R n R is defined as epi f = {(x, t) x dom f, f(x) t} R n+1. (7) A function f is convex if and only if epi f is a convex set. 13 / 122

14 First-order Condition and Epigraph Left plot: first-order condition for a convex function f for the one-dimensional case: f(b) f(a) + f (a)(b a), for all a, b dom f; right plot: the epigraph of a convex function f : R R. 14 / 122

15 Convex Functions (Cont.) Convexity Preserving Operations Intersection, i Ci of convex sets Ci is a convex set; Nonnegative weighted sum, i θifi (where θi 0) of convex functions f i is a convex function. Image, h(c) = {h(x) x C)}, of a convex set C via affine mapping h(x) Ax + b, is a convex set; Composition f(h(x)) of a convex function f with affine mapping h is a convex function; Image, p(c), of a convex set C via perspective mapping p(x, t) x/t is a convex set; Perspective, g(x, t) = tf(x/t) (where t > 0) of a convex function f is a convex function. 15 / 122

16 Perspective Mapping & Perspective of a Function Left plot: pinhole camera interpretation of the perspective mapping p(x, t) = x/t, t > 0; right plot: epigraph of the perspective g(x, t) = tf(x/t), t > 0 of f(x) = x 2, where each ray is associated with g(at, t) = a 2 t for a different value of a. 16 / 122

17 Convex optimization problem Convex problem: (CVXP) p = min x C f(x) is a convex problem if the objective function f( ) is a convex function and C is a convex set (called the feasible set) in standard form as follows: C = {x D f i(x) 0, h j(x) = 0, i = 1,..., m, j = 1,..., p}, where f i(x) is convex for all i and h j(x) is affine for all j and { m } { p } D = dom f dom f i dom h i is called the problem domain. i=1 Advantages: Global optimality: x can be obtained by closed-form solution, analytically (algorithm), or numerically by convex solvers (e.g., CVX and SeDuMi). Computational complexity: Polynomial-time solvable. Performance analysis: KKT conditions are the backbone for analysis. i=1 17 / 122

18 Global optimality and solution An optimality criterion: Any suboptimal solution to (CVXP) is globally optimal. Assume that f is differentiable. Then a point x C is optimal if and only if f(x ) T (x x ) 0, x C (8) 18 / 122

19 Global optimality and solution Because the optimality criterion (8) can only solve limited convex problems, a complementary approach for solving problem (CVXP) is founded on the duality theory. Dual problem: L(x, λ, ν) f(x) + m i=1 λifi(x) + p i=1 νihi(x) (Lagrangian) g(λ, ν) = inf x D L(x, λ, ν)> (dual function) d = max {g(λ, ν) λ 0, ν R p } (dual problem) p = min {f(x) x C)} (primal problem (CVXP)) (9) where λ = (λ 1,..., λ m) and ν = (ν 1,..., ν p) are dual variables. Problem (CVXP) and its dual can be solved simultaneously by solving the so-called KKT conditions. 19 / 122

20 Global optimality and solution KKT conditions: Suppose that f, f 1,..., f m, h 1,..., h p are differentiable and x is primal optimal and (λ, ν ) is dual optimal to problem (CVXP). Under strong duality, i.e., p = d = L(x, λ, ν ) (which holds true under Slater s condition: a strictly feasible point exists, i.e., relint C ), the KKT conditions for solving x and (λ, ν ) are as follows: xl(x, λ, ν ) = 0, f i(x ) 0, i = 1,..., m, h i(x ) = 0, i = 1,..., p, λ i 0, i = 1,..., m, λ i f i(x ) = 0, i = 1,..., m. (complementary slackness) The above KKT conditions (10) and the optimality criterion (8) are equivalent under Slater s condition. (10a) (10b) (10c) (10d) (10e) 20 / 122

21 Strong Duality Lagrangian L(x, λ), dual function g(λ), and primal-dual optimal solution (x, λ ) = (1, 1) of the convex problem min{f 0(x) = x 2 (x 2) 2 1} with strong duality. Note that f 0(x ) = g(λ ) = L(x, λ ) = / 122

22 Standard Convex Optimization Problems Linear Programming (LP) - Inequality Form min c T x s.t. Gx h, Ax = b, ( stands for componentwise inequality) (11) where c R n, A R p n, b R p, G R m n, h R m, and x R n is the unknown vector variable. LP- Standard Form min c T x (12) s.t. x 0, Ax = b, where c R n, A R p n, b R p, and x R n is the unknown vector variable. 22 / 122

23 Standard Convex Optimization Problems (Cont.) Quadratic Programming (QP): Convex if and only if P 0 (i.e., P is positive semidefinite) min 1 2 xt Px + q T x + r (13) s.t. Ax = b, Gx h, where P S n, G R m n, and A R p n. Quadratically constrained QP (QCQP): Convex if and only if P i 0, i min 1 2 xt P 0x + q T 0 x + r 0 (14) s.t. 1 2 xt P ix + q T i x + r i 0, i = 1,..., m, Ax = b, where P i S n, i = 0, 1,..., m, and A R p n. 23 / 122

24 Standard Convex Optimization Problems (Cont.) Second-order cone programming (SOCP) min c T x s.t. A ix + b i 2 f T i x + d i, i = 1,..., m, Fx = g, (15) where A i R n i n, b i R n i, f i R n, d i R, F R p n, g R p, c R n, and x R n is the vector variable. Semidefinite programming (SDP) - Standard Form min Tr(CX) (16) s.t. X 0, Tr(A ix) = b i, i = 1,..., p, with variable X S n, where A i S n, C S n, and b i R. 24 / 122

25 Alternating direction method of multiplers (ADMM) Consider the following convex optimization problem: min x R n,z R m f1(x) + f2(z) s.t. x S 1, z S 2 z = Ax where f 1 : R n R and f 2 : R m R are convex functions, A is an m n matrix, and S 1 R n and S 2 R m are nonempty convex sets. (17) The considered dual problem of (17) is given by max min {f c } 1(x)+f 2(z)+ Ax z 2 ν R m x S 1,z S νt (Ax z), (18) where c is a penalty parameter, and ν is the dual variable associated with the equality constraint in (17). 25 / 122

26 ADMM (Conti) Inner minimization (convex problems): z(q + 1) = arg min {f 2(z) ν(q) T z + c Ax(q) z } 2, (19a) z S x(q + 1) = arg min {f 1(x) + ν(q) T Ax + c Ax z(q + 1) } 2. (19b) x S ADMM Algorithm 1: Set q = 0, choose c > 0. 2: Initialize ν(q) and x(q). 3: repeat 4: Solve (19a) and (19b) for z(q + 1) and x(q + 1) by two distributed equipments including the information exchange of z(q + 1) and x(q + 1) between them; 5: ν(q+1) = ν(q) + c (Ax(q + 1) z(q + 1)); 6: q := q + 1; 7: until the predefined stopping criterion is satisfied. When S 1 is bounded or A T A is invertible, ADMM is guaranteed to converge and the obtained {x(q), z(q)} is an optimal solution of problem (17). 26 / 122

27 Nonconvex problem Reformulation into a convex problem: Equivalent representations (e.g. epigraph representations); function transformation; change of variables, etc. Stationary-point solutions: Suppose that C is closed and convex but f is nonconvex. A point x is a stationary point of problem (1) if f (x ; v) lim inf λ 0 f(x + λv) f(x ) λ f(x ) T (x x ) 0 0 x + v C (20) x C (when f is differentiable) where f (x ; v) is the directional derivative of f at a point x in direction v. Block successive upper bound minimization (BSUM) [Razaviyayn 13] guarantees a stationary-point solution under some convergence conditions. KKT points (i.e., solutions of KKT conditions) are also stationary points provided that the Slater s condition is satisfied. [Razaviyayn 13] M. Razaviyayn, M. Hong, and Z.-Q. Luo, A unified convergence analysis of block successive minimization methods for nonsmooth optimization, SIAM J. Optimiz., vol. 23, no. 2, pp , / 122

28 Stationary points and BSUM Illustration of stationary points of a nonconvex function and BSUM for one-dimensional case. 28 / 122

29 Nonconvex problem Approximate solutions when f is a convex function but C is a nonconvex set: Convex restriction to C: Successive convex approximation (SCA) where C i C is convex for all i. Then f(x i+1) = x i = arg min x C i f(x) C i+1 (21) min f(x) f(x i ), C i+1 i+1 j=1c j C x C i+1 (22) After convergence, a stationary-point solution may be obtained depending on the problem under consideration and the choice of the convex set C i. Convex relaxation to C (e.g., semidefinite relaxation (SDR)): C = {X S n + rank(x) = 1} C relaxed to conv C = S n + (SDR); C = { 3, 1, +1, +3} C relaxed to conv C = [ 3, 3] (23) The obtained X or x may not be feasible to problem (1); for SDR, a good approximate solution can be obtained from X via Gaussian randomization. 29 / 122

30 Successive Convex Approximation (SCA) 30 / 122

31 Algorithm development Foundamental theory and tools: Calculus, linear algebra, matrix analysis and computations, convex sets, convex functions, convex problems (e.g., geometric program (GP), LP, QP, SOCP, SDP), duality, interior-point method; CVX and SeDuMi. 31 / 122

32 Outline 1 Part I: Fundamentals of Convex Optimization 2 Part II: Application in Hyperspectral Image Analysis: (Big Data Analysis and Machine Learning) 3 Part III: Application in Wireless Communications (5G Systems) Subsection I: Outage Constrained Robust Transmit Optimization for Multiuser MISO Downlinks Subsection II: Outage Constrained Robust Hybrid Coordinated Beamforming for Massive MIMO Enabled Heterogeneous Cellular Networks 32 / 122

33 Introduction to hyperspectral unmixing (HU) A hyperspectral sensor records electromagnetic fingerprints (scattering patterns) of substances (materials) in a scene, known as spectral signatures, over hundreds of spectral bands from visible to short-wave infrared wavelength region. Issue: mixed pixel spectra in the hyperspectral image/data (due to limited spatial resolution of the hypersepctral sensor) must be decomposed for identifying the underlying materials (i.e., spectral unmixing) [Keshava 02]. Airborne Sensors. Courtesy: [Keshava 02] N. Keshava et al., Spectral unmixing, IEEE Signal Process. Mag., vol. 19, no. 1, pp , Jan / 122

34 Introduction to hyperspectral unmixing (HU) Airborne Sensors. Courtesy: Hyperspectral unmixing (HU) is a signal processing procedure of extracting hidden spectral signatures of substances (called endmembers) and the corresponding proportions (called abundances) (i.e., distribution map of substances) in a scene, from the hyperspectral observations [Keshava 02]. Applications: Terrain classification, mineral identification and quantification, agricultural monitoring, military surveillance, space object identification, etc. [Keshava 02] N. Keshava et al., Spectral unmixing, IEEE Signal Process. Mag., vol. 19, no. 1, pp , Jan / 122

35 Illustration of HU [Ambikapathi 11] Red pixel: a mixed pixel (land+vegetation+water) Blue pixel: a pure pixel (only water) [Ambikapathi 11] A. Ambikapathi et al., Chance constrained robust minimum volume enclosing simplex algorithm for hyperspectral unmixing, IEEE Trans. Geoscience and Remote Sensing, vol. 49, no. 11, pp , Nov / 122

36 Signal model and assumptions Linear Mixing Model The pixel vector x[n] of M spectral bands can be represented as a linear combination of N endmember signatures {a 1,..., a N }, i.e., { As[n] = N i=1 si[n]ai RM, n I L {1,..., L}. x[n] = s j[n]a j (a pure pixel) if s i[n] = 0 i j (24) A = [a 1,..., a N ] R M N (spectral signature matrix or mixing matrix). s[n] [s 1[n],..., s N [n]] T R N is the nth abundance vector; s i = {s i[n] n I L} is called source i or abundance map i. Standard Assumptions (Non-statistical) [Keshava 02] (A1) s[n] 0 N and N i=1 si[n] = 1, n IL. (A2) A = [a 1,..., a N ] 0 M N has full column rank and min{l, M} N. 36 / 122

37 Craig s HU criterion Observation: X {x[1],..., x[l]} conv{a 1,..., a N } T a R M (an N-vertex simplex due to (A1) and (A2)); conv X = T a if a i X i. Craig s belief: The vertices of the minimum-volume data-enclosing simplex T a yield good estimate â i [Craig 94] even without any pure pixels, a i in X. Figure 2: Visually, vol(t a) < vol(t i ), i I 2 (the dots are data points x[n]). [Craig 94] M. D. Craig, Minimum-volume transforms for remotely sensed data, IEEE Trans. Geosci. Remote Sens., vol. 32, no. 3, pp , May / 122

38 Craig s HU criterion (Cont.) Craig s criterion [Craig 94] (an NP-hard problem): {â 1,..., â N } arg min vol(conv{b 1,..., b N }) b i R M i s.t. X conv{b 1,..., b N } (25) In [Lin 15], we theoretically proved that as long as the data uniform purity level γ is above a threshold (a mild condition), i.e., γ max{r T e B(r) conv{s[1],..., s[l]}> 1/ N 1 where T e conv{e 1,..., e N } R N (unit simplex) and B(r) {x R N x r}, Craig s criterion can perfectly identify the ground-truth endmembers {a 1,..., a N } (i.e., the true simplex T a). Can we devise a super-efficient HU algorithm using Craig s criterion? [Craig 94] M. D. Craig, Minimum-volume transforms for remotely sensed data, IEEE Trans. Geosci. Remote Sens., vol. 32, no. 3, pp , May [Lin 15] C.-H. Lin et al., Identifiability of the simplex volume minimization criterion for blind hyperspectral unmixing: The no pure-pixel case, IEEE Trans. Geosci. Remote Sens., vol. 53, no.10, pp , Oct / 122

39 Dimension reduction and problem formulation Dimension-reduced (DR) representation for noise suppression and computational complexity reduction: Due to affdim T a = N 1, x[n] = C x[n] + d X T a aff T a R M (affine set fitting) (26) x[n] = C (x[n] d) N = s i[n]α i T α = conv {α 1,..., α N } aff T α R N 1 (27) i=1 C [q 1(UU T ),..., q N 1(UU T )] (a semiunitary matrix), where q i( ) is the ith principal eigenvector, and U [x[1] d,..., x[l] d] R M L (mean removed data matrix); d 1 L L n=1 x[n] (mean of the data set X); T α is an N-vertex simplest simplex; X = { x[1],..., x[l]} T α R N 1 (DR data set) (28) α i C (a i d) R N 1 (DR endmembers). (29) 39 / 122

40 Illustration of Dimension Reduction Dimension reduction illustration using affine set fitting for N = 3, where the geometric center d of the data cloud X in the M-dimensional space maps to the origin in the (N 1)-dimensional space. 40 / 122

41 Dimension reduction and problem formulation Problem (25) can be reformulated in the DR space as: { α 1,..., α N } arg min β i R N 1 i { vol(conv{β 1,..., β N }) = det(b) } (N 1)! s.t. X conv{β 1,..., β N } where B = [β 1 β N,..., β N 1 β N ] R (N 1) (N 1). Endmember estimates in the original space R M : Existing Challenges â i = C α i + d, i I N (cf. (29)). Pure pixel assumption (PPA) enables various simple and fast blind HU algorithmic schemes (for finding the purest pixels in the data set X or the DR data set X ), but it is often seriously infringed. Without requiring the PPA, Craig s blind HU criterion identifies the N-vertex minimum-volume data-enclosing simplex T a R M, but suffering from heavy simplex volume computations. (30) 41 / 122

42 New Philosophy: Hyperplane-based CSI Algorithm Geometry Fact 1 [Lin 16]: s i[n] = 0 if and only if x[n] H i, where H i aff({α 1,..., α N } \ {α i}) R N 1 is the boundary hyperplane of the DR endmembers simplest simplex T α R N 1. Observation The abundance maps often show large sparseness [Cichocki 09], and many pixels lying on or close to the boundary hyperplanes of the Craig s simplex. Geometry Fact 2: A simplest simplex of N vertices can be defined by its N boundary hyperplanes H i (each containing N 1 vertices) and vice versa. 42 / 122

43 Real data experiments The good performance of HyperCSI in the experiment also implies that the requirement of sufficient (i.e., N (N 1) = 72) active pixels lying close to the hyperplanes of the actual endmembers simplex, has been met for the considered hyperspectral scene. 43 / 122

44 Super-fast HU algorithm: HyperCSI HyperCSI Algorithm: A novel practical HU algorithm HyperCSI algorithm tries to estimate the N boundary hyperplanes of the N-vertex simplest simplex T α R N 1 (in DR space), without any simplex volume computations. Each hyperplane is constructed by N 1 A.I. data pixels that are identified via simple linear algebraic computations. With better performance than state-of-the-art algorithms, HyperCSI is at least computationally efficient as PPA based HU algorithms. [Lin 16] C.-H. Lin et al., A fast hyperplane-based minimum-volume enclosing simplex algorithm for blind hyperspectral unmixing, IEEE Trans. Signal Processing, vol. 64, no.8, pp , Apr / 122

45 Block diagram of HyperCSI algorithm An algorithm with parallel processing structure for estimation of normal vectors (b i), inner product parameters (h i), and abundance maps (s i), where the PPA based successive projection algorithm (SPA) is employed to obtain initial estimates α 1,..., α N. 45 / 122

46 Graphical illustration of HyperCSI Why R (i) k should be disjoint? Consider {p (1) 2, q} identified by (33). Why not b 1? The purest pixel α 3 may not be close to H 1 = aff{α 2, α 3}, leading to nontrivial orientation difference between b 1 and b 1. However, {p (1) 1, p(1) 2 } identified by (34) are very close to H1, so the orientations of b 1 and b 1 are almost the same. 46 / 122

47 HyperCSI Algorithm Pseudocode for the HyperCSI Algorithm 1 Given Hyperspectral data {x[1],..., x[l]}, number of endmembers N, and η = Obtain DR dataset X = { x[1],..., x[l]} by (27). (DR processing) 3 Obtain purest pixels { α 1,..., α N } by SPA. (preprocessing) 4 Obtain b i = v i( α 1,..., α N ) by (31), and then obtain A.I. set {p (i) 1,..., p(i) N 1 } by (34) i IN. (A.I. w.p.1) 5 Obtain b i and ĥi by (35) and (36), respectively, i IN. 6 Obtain c by (37), and set c = c /η. (further denoising) 7 Obtain α i by (38) and then obtain â i by (39) i I N.(identifiable w.p.1) 8 Calculate ŝ[n] = [ŝ 1[n],..., ŝ N [n]] T by (41) for all n I L. O(N 2 L) 9 Output The endmember estimates {â 1,..., â N } and abundance maps {ŝ 1,..., ŝ N }. 47 / 122

48 Hyperplane representation of Simplest Simplex Since T α is a simplest simplex in the DR space R N 1, enabling the following handy parameterization for H i: H i aff(t α \ {α i}) = {x R N 1 b T i x = h i} H(b i, h i), where b i R N 1 and h i R respectively denote the outward-pointing normal vector of H i and the inner product constant of H i. Proposition 1 [Lin 16] The endmembers α i can then be reconstructed by the hyperplanes H i: α i = B 1 i h i, i I N, where B i [b 1,..., b i 1, b i+1,..., b N ] T ; h i [h 1,..., h i 1, h i+1,..., h N ] T. Problem (30) can be decoupled into N parallel subproblems of estimating (b i, h i), i I N. Next, let us focus on how to estimate (b i, h i), i I N. 48 / 122

49 Normal vector b i estimation The normal vector b i can be reconstructed by N 1 A.I. points on H i. Proposition 2 [Lin 16] Given A.I. {p (i) 1,..., p(i) N 1 } Hi, we have for arbitrary k I N 1, where b i = v i(p (i) 1,..., p(i) i 1, 0N 1, p(i) i,..., p (i) ( ) I N 1 P(P T P) 1 P T p (i) k, P Q p (i) k 1T N 2 R (N 1) (N 2), N 1 Q [p (i) 1,..., p(i) k 1, p(i) k+1,..., p(i) N 1 ] R(N 1) (N 2). ), (31) Motivated by Proposition 2, we aim at extracting N 1 A.I. data pixels p (i) k from the DR dataset X that are closest to H i. 49 / 122

50 Normal vector b i Estimation (Cont.) Proposition 3 [Lin 16] Observing that all the points p X lie on the same side of H i, i.e., b T i p h i, p X, and dist(p, H i) = h i b T i p / b i, we see that the point p X closest to H i is exactly the one with maximum of b T i p, because b i is outward-pointing. Let H i aff({ α 1,..., α N } \ { α i}) H( b i, h i), where α i are the purest pixels extracted by the successive projection algorithm [Arora 12]. By Proposition 2 and Proposition 3, a naive way for estimating p (i) k {p (i) 1,..., p(i) N 1 } arg max { b T i (p p N 1) p k X }. (32) = The identified p (i) k can be quite close to each other. is [Arora 12] S. Arora et al., A practical algorithm for topic modeling with provable guarantees, arxiv preprint arxiv: , / 122

51 Normal vector b i Estimation (Cont.) By Proposition 2 and Proposition 3, a naive way for estimating p (i) k {p (i) 1,..., p(i) N 1 } arg max { b T i (p p N 1) p k X }. (33) = The identified p (i) k can be quite close to each other. is [Arora 12] S. Arora et al., A practical algorithm for topic modeling with provable guarantees, arxiv preprint arxiv: , / 122

52 Normal vector b i Estimation (Cont.) Accordingly, p (i) k where R (i) k p (i) k for H i are sifted by arg max{ b T i p p X R (i) k }, k IN 1 (34) are disjoint regions (norm balls with the same radius) defined by { R (i) k B( α k, r), if k < i, B( α k+1, r), if k i, B( α k, r) {x R N x α k < r}, r (1/2) min{ α i α j 1 i < j N} > 0. Normal vector estimates: b i = v i(p (i) 1,..., p(i) i 1, 0N 1, p(i) i,..., p (i) N 1 ) i. (35) 52 / 122

53 Inner product constant h i Estimation Craig s simplex must tightly enclose X : ĥ i = max { b T i p p X }. (36) Considering the simplex volume expansion caused by noise, the estimated hyperplanes need to be properly shifted closer to the origin, or equivalently ĥ i should be scaled down as ĥi/c for some c 1 in general. Non-data dependent choice of the parameter c As A 0 M N, Proposition 1 indicates c c (empirically, c = c /η and η = 0.9), where c min {c 1 C ( B c i ĥ i) + c d 0 M, i I N } 1 = max{1, max{ v ij/d j i I N, j I M }}, (37) 1 in which v ij is the jth entry of C ( B i ĥ i) and dj is the jth entry of d. Endmember estimates: 1 α i = B i ĥ i/c RN 1, (38) â i = C α i + d 0 M, i I N. (39) 53 / 122

54 Abundance vector s[n] estimation Lemma 3 [Lin 16] (Closed-form Expression of s[n]) Assume that (A1) and (A2) hold true. Then, s[n] = [s 1[n],..., s N [n]] T the following closed-form expression: has s i[n] = hi bt i x[n] h i b T i αi, i I N, n I L. (40) ŝ i[n] = max { ĥi b } T i x[n] ĥ i b, 0, i I N, n I L. (41) T i αi Theorem 1 (Computational Complexity Analysis) [Lin 16] The computational complexity of HyperCSI is O(NL) with parallel processing and O(NL 2 ) without parallel processing. 54 / 122

55 Statistical analysis (A3) s[1],..., s[l] are independent and identically distributed (i.i.d.) with a continuous probability density function (p.d.f.) whose support is the unit simplex T e conv{e 1,..., e N } R N (e.g., Dirichlet distribution). Theorem 2 (Affine Independence Analysis) [Lin 16] Under (A1)-(A3), the pixels { p (i) 1,..., p(i) N 1 } extracted by (34) are A.I. with probability 1 (w.p.1), i I N. Theorem 3 (Identifiability of HyperCSI Algorithm) [Lin 16] Under (A1)-(A3), the noiseless assumption and L, the simplex identified by the HyperCSI algorithm with c = 1 is exactly the Craig s simplex and the true endmembers simplex conv{α 1,..., α N } in the DR space w.p / 122

56 Numerical simulaitons HyperCSI is compared with five state-of-the-art algorithms [Ma 14]: 1 Minimum-volume simplex analysis (MVSA) algorithm; 2 Interior-point method based MVSA (ipmvsa) algorithm [Li 15]; 3 Minimum-volume enclosing simplex (MVES) algorithm [Chan 09]; 4 Simplex identification via split augmented Lagrangian (SISAL) algorithm; 5 Minimum-volume constrained non-negative matrix factorization (MVC-NMF) algorithm [Miao 07]. [Ma 14] W.-K. Ma et al., A signal processing perspective on hyperspectral unmixing, IEEE Signal Process. Mag., vol. 31, no. 1, pp , Jan [Li 15] J. Li et al., Minimum volume simplex analysis: A fast algorithm for linear hyperspectral unmixing, IEEE Trans. Geosci. Remote Sens., vol. 53, no. 9, pp , Apr [Chan 09] T.-H. Chan, et al., A convex analysis-based minimum-volume enclosing simplex algorithm for hyperspectral unmixing, IEEE Trans. Signal Processing, vol. 57, no. 11, pp , Nov (Citations: 245 by Google Scholar) [Miao 07] L. Miao et al., Endmember extraction from highly mixed data using minimum volume constrained nonnegative matrix factorization, IEEE Trans. Geosci. Remote Sens., vol. 45, no. 3, pp , / 122

57 Numerical simulaitons (con t) Data generation: N = 6 endmembers with M = 224 spectral bands are randomly selected from the US Geological Survey (USGS) library to generate L = 10, 000 noiseless synthetic data, and then added by Gaussian noise. Performance measures: 1 Computation time T ; 2 Root-mean-square (RMS) spectral angle error φ en (between a i and â i): φ en = min 1 N [ ( )] a T 2 arccos i â πi, π Π N N a i â πi i=1 where Π N is the set of all the permutations of {1,..., N}. 3 RMS angle error φ ab (between s i and ŝ i): φ ab = min 1 N [ ( )] s T 2 arccos i ŝ πi. π Π N N s i ŝ πi i=1 57 / 122

58 Numerical simulaitons (con t) The asymptotic analysis in Theorem 3 (noiseless and c = 1) is illustrated below: HyperCSI performs well even with a moderately finite number of pixels L, i.e., several tens of thousands, which is typical in HU. The curve for larger N converges at a slower rate as we need more pixels (i.e., N(N 1)) lying close to the boundary hyperplanes for larger N. 58 / 122

59 Numerical simulaitons (con t) Two sets of sparsely, non-i.i.d. and non-dirichlet distributed maps are used to generate two synthetic datasets [Iordache 12], denoted by SYN1 and SYN2, for performance evaluation, where SYN1 contains L = 10, 000 pixels and SYN2 contains L = 16, 900 pixels, resp. [Iordache 12]. [Iordache 12] M.-D. Iordache et al., Total variation spatial regularization for sparse hyperspectral unmixing, IEEE Trans. Geosci. Remote Sens., vol. 50, no. 11, pp , / 122

60 Numerical simulaitons (con t) Each simulation result for different SNRs, is obtained by averaging over 100 realizations. 60 / 122

61 Real data experiments Real hyperspectral imaging data experiments: Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) taken over the Cuprite mining site, Nevada, in 1997 [AVIRIS 97]. The number of sources for this dataset is estimated to be N = 9 using an information-theoretic minimum description length (MDL) criterion [Lin 16-2]. The proposed HyperCSI algorithm, along with the following two benchmark algorithms (for analyzing the hyperspectral imaging data), are used to process the AVIRIS data: 1 MVC-NMF algorithm [Miao 07] (based on Craig s criterion); 2 VCA algorithm [Nascimento 05] (based on pure-pixel assumption). [AVIRIS 97] AVIRIS Free Standard Data Products. [Online]. Available: [Lin 16-2] C.-H. Lin et al., Detection of sources in non-negative blind source separation by minimum description length criterion, submitted to IEEE Trans. Neural Networks and Learning Systems (acceptable subject to minor revision). [Nascimento 05] J. Nascimento et al., Vertex component analysis: A fast algorithm to unmix hyperspectral data, IEEE Trans. Geosci. Remote Sens., vol. 43, no. 4, pp , Apr / 122

62 Real data experiments (con t) Endmembers extracted by HyperCSI algorithm show better resemblance to their counterparts in library. For instance, the endmember of Alunite extracted by HyperCSI shows much clearer absorption feature than MVC-NMF and VCA, in the bands approximately from 2.3 to 2.5 µm. 62 / 122

63 Real data experiments (con t) The average RMS spectral angle error φ between endmember estimates and their corresponding library spectra, is used for quantitative comparison: As the pure pixels may not be present in the selected subscene, the two Craig criterion based algorithms outperform VCA as expected. In terms of the computation time T, in spite of parallel processing not yet applied, HyperCSI is much faster than the other two algorithms. 63 / 122

64 Real data experiments (con t) The good performance of HyperCSI in the experiment also implies that the requirement of sufficient (i.e., N (N 1) = 72) active pixels lying close to the hyperplanes of the actual endmembers simplex, has been met for the considered hyperspectral scene. 64 / 122

65 Summary Based on the hyperplane representation for a simplest simplex, the presented HyperCSI algorithm has the following remarkable characteristics: 1 Craig s simplex is reconstructed from N(N 1) pixels (regardless of the data length L), without involving any simplex volume computations. 2 It is reproducible (without involving random initialization and tuning of regularization parameters) and not data-dependent, regardless of the existence of pure pixels. 3 Its superior performance over state-of-the-art methods has been demonstrated by analysis, simulations and real data experiments. 4 It only involves simple linear algebraic computations, with a complexity O(NL) with or O(N 2 L) without parallel implementation, thereby sustaining its practical applicability. 65 / 122

66 Outline 1 Part I: Fundamentals of Convex Optimization 2 Part II: Application in Hyperspectral Image Analysis: (Big Data Analysis and Machine Learning) 3 Part III: Application in Wireless Communications (5G Systems) Subsection I: Outage Constrained Robust Transmit Optimization for Multiuser MISO Downlinks Subsection II: Outage Constrained Robust Hybrid Coordinated Beamforming for Massive MIMO Enabled Heterogeneous Cellular Networks 66 / 122

67 Outline 1 Part I: Fundamentals of Convex Optimization 2 Part II: Application in Hyperspectral Image Analysis: (Big Data Analysis and Machine Learning) 3 Part III: Application in Wireless Communications (5G Systems) Subsection I: Outage Constrained Robust Transmit Optimization for Multiuser MISO Downlinks Subsection II: Outage Constrained Robust Hybrid Coordinated Beamforming for Massive MIMO Enabled Heterogeneous Cellular Networks 67 / 122

68 Outline (Subsection I) 1 System Model and Problem Statement 2 Convex Restriction Methods to the Rate Outage Constrained Problem: Bernstein-type Inequality 3 Simulation Results 4 Conclusions 68 / 122

69 Outline (Subsection I) 1 System Model and Problem Statement 2 Convex Restriction Methods to the Rate Outage Constrained Problem: Bernstein-Type Inequality 3 Simulation Results 4 Conclusions 69 / 122

70 1. System Model and Problem Statement Multiuser multiple-input single-output (MISO) downlink: A practical scenario in wireless communications where one base station (BS) equipped with N t antennas sends independent messages to K single-antenna users. 70 / 122

71 1. System Model and Problem Statement Transmit signal from BS: x(t) = K x i(t). (42) i=1 x i(t) C N t : information signal for user i; x i(t) CN (0, S i) with S i 0 denoting the signal covariance matrix. Assuming that rank(s i) = d, x i(t) can be expressed as [Vu07] x i(t) = d λk (S i)w k s ik (t). (43) k=1 λ k (S i): the kth largest eigenvalue of S i; s ik (t) CN (0, 1): kth independent data stream for user i; w k C N t : orthonormal eigenvectors of S i. When d = 1, the transmit strategy for x i(t) reduces to transmit beamforming. [Vu07] M. Vu and A. Paulraj, MIMO wireless linear precoding, IEEE Signal Process. Magazine, vol. 24, no. 5, pp , Sep / 122

72 1. System Model and Problem Statement Received signal of user i: y i(t) = h H i x(t) + n i(t). (44) h i C N t : the channel of user i; n i(t) CN (0, σ 2 i ): additive noise at the user i. Achievable rate of user i (in bits/sec/hz), assuming single-user detection with perfect h i at the receiver i [Telatar99]: ) R i({s k } K h k=1; h i) = log 2 (1 H i S ih i + K k i hh i S, i = 1,..., K, kh i + σi 2 }{{} SINR (45) where SINR denotes the signal-to-interference-plus-noise ratio associated with user i. [Telatar99] E. Telatar, Capacity of multi-antenna Gaussian channels, Bell Labs Tech. J., vol. 10, no. 6, pp , Nov./Dec / 122

73 1. System Model and Problem Statement Rate Constrained Problem under Perfect Channel State Information (CSI) With the given CSI h 1,..., h K that are known to the BS, min S 1,...,S K H N t K Tr(S i) i=1 (46a) s.t. R i({s k } K k=1; h i) r i, i = 1,..., K, (46b) S 1,..., S K 0, where each r i 0 is the required information rate (target rate) for user i. (46c) Problem (46) can be reformulated as a convex semidefinite program (SDP), which is polynomial-time solvable [Bengtsson01] [Gershman10]. [Bengtsson01] M. Bengtsson and B. Ottersten, Handbook of Antennas in Wireless Communications, L. C. Godara, Ed., CRC Press, Aug [Gershman10] A. B. Gershman and N. D. Sidiropoulos and S. Shahbazpanahi and M. Bengtsson and B. Ottersten, Convex optimization-based beamforming, IEEE Signal Process. Mag., vol. 27, no. 3, pp , May / 122

74 1. System Model and Problem Statement Unfortunately, the BS cannot acquire perfect CSI h i (used in the conventional formulation in (46)) due to imperfect channel estimation and limited feedback [Love08]. CSI error model: h i = h i + e i, i = 1,..., K, (47) where h i C N t is the presumed channel at the BS, and e i C N t is the channel error vector. Gaussian channel error model [Marco05] [Shenouda08] (suitable for imperfect channel estimation at BS): for some known error covariance C i 0. e i CN (0, C i) (48) [Love08] D. J. Love, R. Heath, V. K. N. Lau, D. Gesbert, B. Rao, and M. Andrews, An overview of limited feedback in wireless communication systems, IEEE J. Sel. Areas Commun., vol. 26, no. 8, pp , Oct [Marco05] D. Marco and D. L. Neuhoff, The validity of the additive noise model for uniform scalar quantizers, IEEE Trans. Inform. Theory, vol. 51, no. 5, pp , May [Shenouda08] M. B. Shenouda and T. N. Davidson, Probabilistically-constrained approaches to the design of the multiple antenna downlink, in Proc. 42nd Asilomar Conference 2008, Pacific Grove, October 26-29, 2008, pp / 122

75 Effect of CSI errors on QoS requirement (information rate) Rate outage probabilities: ρ i Prob{R i r i}, i = 1,..., K. Consider a simulation example for N t = K = 3, target rates r i = r, ρ i ρ = 0.1 for all i, with the common target SINR value γ 2 r 1 = 11 db (cf. (45)). 500 sets of { h i} K i=1 are randomly generated with h i CN (0, I Nt ); For each { h k } K k=1, obtain {Ŝk} K k=1 by solving (46) (as if { h k } K k=1 were perfect), and using 4 robust designs, probabilistic SOCP [Shenouda08] and Methods I-III (to be presented later) with C i = 0.002I Nt ; generate 10, 000 Gaussian CSI errors e i CN (0, C i), i = 1,..., K, and then obtain the actual ˆρ i = N i/10000, where N i denotes the no. of error realizations for which R i({ŝk} K k=1; h i + e i) r, i = 1,..., K. [Shenouda08] M. B. Shenouda and T. N. Davidson, Probabilistically-constrained approaches to the design of the multiple antenna downlink, in Proc. 42nd Asilomar Conference 2008, Pacific Grove, October 26-29, 2008, pp / 122

76 Effect of CSI errors on QoS requirement (information rate) Histograms of actual satisfaction probabilities {1 ˆρ i } K i=1 over the 500 channel realizations: 76 / 122

77 1. System Model and Problem Statement Rate Outage Constrained Problem Given rate requirements r 1,..., r K > 0 and maximum tolerable outage probabilities ρ 1,..., ρ K (0, 1], min S 1,...,S K H N t K Tr(S i) i=1 s.t. Prob {R i({s k } K k=1; h } i + e i) r i ρ i, i=1,..., K, S 1,..., S K 0, (49a) (49b) (49c) where R i({s k } K k=1, h i + e i) is defined in (45). Problem (49) is hard to solve since rate outage probabilities in (49b) have no closed-form expressions and are unlikely to be efficiently computable in general. 77 / 122

78 Outline (Subsection I) 1 System Model and Problem Statement 2 Convex Restriction Methods to the Rate Outage Constrained Problem: Bernstein-Type Inequality 3 Simulation Results 4 Conclusions 78 / 122

79 2. A Restriction Approach for Problem (49) The rate outage constraints in (49b) can be expressed as Prob{e H i Q ie i + 2Re{e H i r i} + s i < 0} ρ i, i = 1,..., K, (50) where for notational simplicity, e i CN (0, I Nt ) (originally denoting channel error), and Q i = C 1/2 i 1 S i S k C 1/2 i, r i = C 1/2 1 i S i S k hi, γ i γ i k i k i (51a) s i = h H i 1 S i S k hi σi 2, γ i k i (51b) in which γ i = 2 r i 1 (target SINR for user i (cf. (45))) corresponding to the rate requirement r i = log 2 (1 + γ i). 79 / 122

80 2. A Restriction Approach for Problem (49) Challenge 1 Let e CN (0, I n) and (Q, r, s) H n C n R be an arbitrary 3-tuple of (deterministic) variables. Find an efficiently computable convex function f : H n C n R R such that { } Prob e H Qe + 2Re{e H r} + s < 0 f(q, r, s). (52) Once a function f satisfying (52) is found, f(q, r, s) ρ (53) = Prob{e H Qe + 2Re{e H r} + s < 0} ρ. (54) Constraint (53) gives a convex restriction of constraint (54), because the associated Q i, r i, s i defined by (51) are affine in S 1,..., S K, implying that both f and (53) are convex with respect to decision variable S i. 80 / 122

81 2. Convex Restriction Methods (Bernstein-Type Inequality) Alternative expression for the outage probability constraint in (54): Prob{e H Qe + 2Re{e H r} + s 0} 1 ρ. Lemma 2 (Bernstein-Type Inequality) [Bechar09] Let e CN (0, I n), and let Q H n and r C n be given. Then, for any η > 0, { } Prob e H Qe + 2Re{e H r} Υ(η) 1 e η, (55) where Υ : R ++ R is defined by Υ(η) = Tr(Q) 2η Q 2 F + 2 r 2 2 ηλ+ (Q), and λ + (Q) = max{λ max( Q), 0}. [Bechar09] I. Bechar, A Bernstein-type inequality for stochastic processes of quadratic forms of Gaussian variables, 2009, preprint, available on / 122

82 2. Convex Restriction Methods (Bernstein-Type Inequality) A sufficient condition for achieving the outage constraint (54): f(q, r, s) = e Υ 1 ( s) ρ, or equivalently Υ(ln(1/ρ)) s, i.e., Tr(Q) 2 ln(1/ρ) Q 2 F + 2 r ln(ρ) λ+ (Q) + s 0. (56) 82 / 122

83 2. Convex Restriction Methods (Bernstein-Type Inequality) Method II (Bernstein-Type Inequality) A convex restriction approximation of problem (49): min S i H N t,x i,y i R, i=1,...,k K Tr(S i) i=1 (57a) s.t. Tr(Q i) 2 ln(1/ρ i) x i+ln(ρ i) y i+s i 0, i, (57b) [ ] vec(qi) x i, i = 1,..., K, (57c) 2ri 2 y ii Nt + Q i 0, i = 1,..., K, y 1,..., y K 0, S 1,..., S K 0, (57d) (57e) where Q i, r i and s i are defined in (51), i = 1,..., K. 83 / 122

84 Outline (Subsection I) 1 System Model and Problem Statement 2 Convex Restriction Methods to the Rate Outage Constrained Problem: Bernstein-Type Inequality 3 Simulation Results 4 Conclusions 84 / 122

85 3. Simulation Results Simulation Setting (spatially i.i.d. Gaussian CSI errors): N t = K = 3; Users noise powers: σ 2 1 = = σ 2 K = 0.1; Preset outage probabilities: ρ 1 = = ρ K = 0.1; SINR requirements: γ 1 = = γ K γ (recall that γ i = 2 r i 1); Spatially i.i.d. Gaussian CSI errors C 1 = = C K = 0.002I Nt ; In each simulation, 500 sets of the presumed channels { h i} K i=1 are randomly and independently generated with h i CN (0, I Nt ); Performance comparisons with probabilistic SOCP [Shenouda08]. [Shenouda08] M. B. Shenouda and T. N. Davidson, Probabilistically-constrained approaches to the design of the multiple antenna downlink, in Proc. 42nd Asilomar Conference 2008, Pacific Grove, October 26-29, 2008, pp / 122

86 3. Simulation Results Feasibility rate of the various methods, where Method II is Bernstein inequality based method. Feasibility rate no. of feasible channels no. of tested channels 100%; The lower the feasibility rate, the more conservative (implying higher transmit power in general) for the method under test. 86 / 122

87 3. Simulation Results Transmit power performance of the various methods. 87 / 122

88 3. Simulation Results In the simulation, a rank-1 solution for (Ŝ1,..., ŜK) is obtained if λ max(ŝi) Tr(Ŝi) for all i. Ratio of rank-one solution no. of realizations yielding a rank-one solution no. of realizations yielding a feasible solution. Table 1: Ratios of rank-one solutions. ρ 0.1 γ (db) Method I 464/ / / /292 Method II 489/ / / /363 Method III 488/ / / /251 ρ 0.01 γ (db) Method I 450/ / / /225 Method II 477/ / / /322 Method III 473/ / / / / 122

89 Outline (Subsection I) 1 System Model and Problem Statement 2 Convex Restriction Methods to the Rate Outage Constrained Problem: Bernstein-Type Inequality 3 Simulation Results 4 Conclusions 89 / 122

90 4. Conclusions We considered the multiuser MISO downlink scenario with Gaussian CSI errors and studied a rate outage constrained optimization problem. Bernstein-type inequality based method for efficiently computable convex restriction of the probabilistic constraints using analytic tools from probability theory was presented [Wang 14]. Simulation results demonstrated that Bernstein-type inequality based method significantly improve upon the existing state-of-the-art method [Shenouda08] in terms of both computational complexity and solution accuracy. [Wang 14] K.-Y. Wang, A. M.-C. So, T.-H. Chang, W.-K. Ma, and Chong-Yung Chi, Outage constrained robust transmit optimization for multiuser MISO downlinks: Tractable approximations by conic optimization, IEEE Trans. Signal Processing, vol. 62, no. 21, pp , Nov (Citations: 114 by Google Scholar) 90 / 122

91 Outline 1 Part I: Fundamentals of Convex Optimization 2 Part II: Application in Hyperspectral Image Analysis: (Big Data Analysis and Machine Learning) 3 Part III: Application in Wireless Communications (5G Systems) Subsection I: Outage Constrained Robust Transmit Optimization for Multiuser MISO Downlinks Subsection II: Outage Constrained Robust Hybrid Coordinated Beamforming for Massive MIMO Enabled Heterogeneous Cellular Networks 91 / 122

92 Outline (Subsection II) 1 System Model and Problem Formulation 2 Proposed Outage Constrained HyCoBF Design 1 Analog Beamforming Algorithm 2 Digital Robust CoBF Design 3 Simulation Results 4 Conclusions 92 / 122

93 Outline (Subsection II) 1 System Model and Problem Formulation 2 Proposed Outage Constrained HyCoBF Design 1 Analog Beamforming Algorithm 2 Digital Robust CoBF Design 3 Simulation Results 4 Conclusions 93 / 122

94 1. System Model and Problem Formulation Massive MIMO enabled two-tier heterogeneous network (HetNet): A macrocell base station (MBS) equipped with large-scale N MBS antennas, and a femtocell base station (FBS) equipped with N FBS antennas, serve K single-antenna macrocell user equipments (MUEs) and J single-antenna femtocell user equipments (FUEs), respectively. h F F 1 FUE j h MF j h MM1 MUE 1 FUE 1 h h MMK F Mk h F F J FBS J MBS j=1 ujsf FUE J j CN FBS MUE k MUE K (transmitted signal by FBS) K k=1 Femto Cell VwksMk CN MBS (transmitted signal by MBS) Macro Cell 94 / 122

95 1. System Model and Problem Formulation Hybrid coordinated beamforming (HyCoBF) structure at MBS for the HetNet, with N RF radio frequency (RF) chains satisfying N MBS N RF K, and N RF N MBS analog phase shifters [Molisch 16]. s M1 RF Chain 1 Ant 1 Digital Beamformer w 1,..., w K RF Switch Λ(b) s MK RF Chain N RF Analog Beamformer V Ant N MBS [Molisch 16] A. F. Molisch, V. V. Ratnam, S. Han, Z. Li, S. Nguyen, S. Li, and K. Haneda, Hybrid beamforming for massive MIMO-A survey, Sep / 122

96 1. System Model and Problem Formulation Let F C N MBS N MBS be the N MBS-point discrete Fourier transform (DFT) matrix (codebook), and collecting all the B {b {1,..., N MBS} N RF b i b j, i j}, (58) ( NMBS ) different combinations. N RF The selected analog beamforming matrix V, consisting of N RF distinct columns of F, and the RF switch matrix Λ(b) can be expressed as Λ(b) = [e(b 1),..., e(b NRF )] R N MBS N RF, b B, V = FΛ(b) C N MBS N RF, (59) where e(b i) denotes the b i-th unit column vector. 96 / 122

97 1. System Model and Problem Formulation Received signal of MUE k: y Mk = h H MMkVw k s Mk + + K l=1,l k h H MMkVw l s Ml J h H F Mku js F j + n Mk. (60) j=1 s Mk, s F j : transmit signals with unit power intended for MUE k and FUE j, respectively; n Mk CN (0, σ 2 Mk) (AWGN); h MMk C N MBS, h F Mk C N FBS : channels from MBS and FBS to MUE k, respectively; w k C N RF, u j C N FBS : baseband beamforming vectors for MUE k and FUE j, respectively. Signal-to-interference-plus-noise ratio (SINR) of MUE k: SINR Mk = K l=1,l k h H MMkVw k 2. (61) h H MMk Vw l 2 + J h H F Mk uj 2 + σmk 2 j=1 97 / 122

98 1. System Model and Problem Formulation Received signal of FUE j: y F j = h H F F ju js F j + + J m=1,m j h H F F ju ms F m K h H MF jvw k s Mk + n F j. (62) k=1 h MF j C N MBS, h F F j C N FBS : channels from MBS and FBS to FUE j, respectively; n F j CN (0, σ 2 F j). SINR of FUE j: SINR F j = J m=1,m j h H F F j uj 2. (63) h H F F j um 2 + K h H MF j Vw k 2 + σf 2 j k=1 98 / 122

99 1. System Model and Problem Formulation Imperfect CSI model [Wang 14]: h MMk = ĥmmk + e MMk, h MF j = ĥmf j + emf j, (64a) h F F j = ĥf F j + ef F j, h F Mk = ĥf Mk + e F Mk. (64b) ĥ MMk, ĥmf j CN MBS, and ĥf F j, ĥf Mk C N FBS : channel estimates that are known to MBS and FBS; e MMk, e MF j C N MBS, e F F j, e F Mk C N FBS : Gaussian CSI errors e MMk CN (0, C MMk ), e MF j CN (0, C MF j ), (65a) e F F j CN (0, C F F j ), e F Mk CN (0, C F Mk ), (65b) where C MMk 0, C MF j 0 and C F F j 0, C F Mk 0. [Wang 14] K.-Y. Wang, A. M.-C. So, T.-H. Chang, W.-K. Ma, and Chong-Yung Chi, Outage constrained robust transmit optimization for multiuser MISO downlinks: Tractable approximations by conic optimization, IEEE Trans. Signal Processing, vol. 62, no. 21, pp , November / 122

100 1. System Model and Problem Formulation Outage Constrained Robust HyCoBF Problem Given γ Mk and γ F j for the target SINRs of MUEs and FUEs, respectively, and the associated outage probabilities ρ Mk and ρ F j, min Vw k 2 + u j 2 (66a) V,{w k },{u j } k I K j I J s.t. Pr(SINR Mk γ Mk ) 1 ρ Mk, k I K = {1,..., K} (66b) Pr(SINR F j γ F j) 1 ρ F j, j I J = {1,..., J} V {FΛ(b) b B} (cf. (2)). (66c) (66d) Solving the robust HyCoBF design problem (66) is hard, due to nonconvex probabilistic constraints (66b), (66c) [Nemirovski 06] and analog beam selection constraint (66d). [Nemirovski 06] A. Nemirovski and A. Shapiro, Convex approximations of chance constrained programs, SIAM J. Optim., vol. 17, no. 4, pp , / 122

101 Outline (Subsection II) 1 System Model and Problem Formulation 2 Proposed Outage Constrained HyCoBF Design 1 Analog Beamforming Algorithm 2 Digital Robust CoBF Design 3 Simulation Results 4 Conclusions 101 / 122

102 2.1. Analog Beamforming Algorithm A new beam selection criterion named power ratio maximization (PRM), by maximizing the ratio of the total channel power of MUEs to the total intercell interference channel power from MBS to FUEs: V = FΛ(b ) (cf. (59)), (67) { } b = arg max J (b) ĤH MM FΛ(b) 2 F, (PRM criterion) b B MF FΛ(b) 2 F (68) where B was defined in (58), ĤMM [ĥ MM1,..., ĥ MMK ] and Ĥ MF [ĥ MF 1,..., ĥ MF J]. A low-complexity beam selection algorithm for solving (68) is proposed, motivated by the idea of single most likely replacement (SMLR) detector used in seismic deconvolution for detecting a Bernoulli-Gaussian signal with nonzero magnitudes [Mendel 83]. [Mendel 83] J. M. Mendel, Optimal seismic deconvolution: An estimated-based approach, Oval Road, London, UK: Academic Press, Inc., / 122

103 2.1. Analog Beamforming Algorithm Low complexity and guaranteed convergence: Computing J (b) by (68) N RF (N MBS N RF) times per iteration. Beam Selection Algorithm-PRM 1 Given ĤMM, ĤMF, IN MBS = {1,..., NMBS}; 2 Initialize b = (b 1,..., b NRF ) B, and S = I NMBS \ {b 1,..., b NRF }; 3 Compute J (b) by (68); 4 repeat 5 Obtain j l = arg max j S J (b l (j)), l = 1,..., N RF, where b l (j) (b 1,..., b l 1, j, b l+1,..., b NRF ); 6 Obtain l = arg max{j (b l (j l )), l = 1,..., N RF } ; 7 If J (b l (j l )) > J (b), update b := b l (j l ), S := I NMBS \ {b 1,..., b NRF }, J (b) := J (b l (j l )); 8 until J (b l (j l )) J (b). 9 Output selected analog beamforming matrix V = FΛ(b). 103 / 122

104 Outline (Subsection II) 1 System Model and Problem Formulation 2 Proposed Outage Constrained HyCoBF Design 1 Analog Beamforming Algorithm 2 Digital Robust CoBF Design 3 Simulation Results 4 Conclusions 104 / 122

105 2.2. Conservative Approximate CoBF Solution Let W k = w k w H k and U j = u ju H j, and define B k = γ 1 MkV W k (V ) H K l k V W l (V ) H ; D = J j=1 Uj; F j = γ 1 F j U j J l j U l; G = K (69a) k=1 V W k (V ) H ; (69b) Q MMk = C 1/2 MMk B kc 1/2 MMk ; Q F Mk = C 1/2 F Mk DC1/2 F Mk ; Q F F j = C 1/2 F F j FjC1/2 F F j ; QMF j = C 1/2 MF j GC1/2 MF j ; r MMk = C 1/2 MMk B kĥmmk; r F Mk = C 1/2 F Mk Dĥ F Mk ; r F F j = C 1/2 F F j FjĥF F j; rmf j = C 1/2 MF jgĥ MF j ; (69f) c Mk = ĥh MMkB k ĥ MMk + ĥh F MkDĥF Mk σ 2 Mk; c F j = ĥh F F jf j ĥ F F j + ĥh MF jgĥmf j σ2 F j. (69c) (69d) (69e) (69g) (69h) 105 / 122

106 2.2. Conservative Approximate CoBF Solution Outage Constrained Digital CoBF: Conservative Approximation Applying semidefinite relaxation (SDR) (i.e., replacing w k wk H by W k 0 and u ju H j by U j 0) to problem (66) yields min Tr (V W k (V ) H) + Tr (U j) {W k },{U j } k I K j I J { (70a) s.t. Pr δ1 H Q MMk δ 1 + δ2 H Q F Mk δ 2 + 2Re{δ1 H r MMk } + } 2Re{δ2 H r F Mk } + c Mk 0 1 ρ Mk, k I K, (70b) { Pr δ1 H Q MF jδ 1 + δ2 H Q F F jδ 2 + 2Re{δ1 H r MF j} + } 2Re{δ2 H r F F j} + c F j 0 1 ρ F j, j I J, (70c) W k 0, k I K, U j 0, j I J. (70d) where δ 1 CN (0, I NMBS ), δ 2 CN (0, I NFBS ). 106 / 122

107 2.2. Conservative Approximate CoBF Solution Define g 1(δ 1, Q MMk, r MMk ) = δ H 1 Q MMk δ 1 + 2Re{r H MMkδ 1}, (71) g 2(δ 2, Q F Mk, r F Mk ) = δ H 2 Q F Mk δ 2 + 2Re{r H F Mkδ 2}. (72) Alternative expression for the outage probability constraint in (70b): { } Pr g 1(δ 1, Q MMk, r MMk ) + g 2(δ 2, Q F Mk, r F Mk ) + c Mk 0 1 ρ Mk. (73) (70c) can also be equivalently re-expressed as the same form as (73). An extension form of the Bernstein-type inequality [Bechar 09] is presented in Lemma 1 below for finding a conservative convex approximation to (73). [Bechar 09] I. Bechar, A Bernstein-type inequality for stochastic processes of quadratic forms of Gaussian variables, Apr / 122

108 2.2. Conservative Approximate CoBF Solution Lemma 1: Extension Form of the Bernstein-type Inequality Given δ 1 CN (0, I NMBS ), δ 2 CN (0, I NFBS ), Q MMk H N MBS, Q F Mk H N FBS, r F Mk C N FBS Then, the following inequality holds: { Pr g 1(δ 1, Q MMk, r MMk ) + g 2(δ 2, Q F Mk, r F Mk ) } Υ(ln(1/ρ Mk ) Q MMk, r MMk, Q F Mk, r F Mk ) 1 ρ Mk, k I K, (74) where Υ : R ++ R is defined as Υ(ln(1/ρ Mk ) Q MMk, r MMk, Q F Mk, r F Mk ) = Tr(Q MMk ) + Tr(Q F Mk ) ln(1/ρ Mk ) λ + (Q MMk, Q F Mk ) α Mk Q MMk 2 F + 2 r MMk 2 + Q F Mk 2 F + 2 r F Mk 2, (75) in which λ + (Q MMk, Q F Mk ) max{λ max( Q MMk ), λ max( Q F Mk ), 0} and α Mk = 2 ln(1/ρ Mk ). By Lemma 1, Υ(ln(1/ρ Mk ) Q MMk, r MMk, Q F Mk, r F Mk ) + c Mk 0 is an approximate deterministic conservative convex constraint to (70b). 108 / 122

109 2.2. Conservative Approximate CoBF Solution Robust Digital CoBF Problem (Centralized Solution) By Lemma 1, problem (70) can be approximated as a convex semidefinite programming (SDP) [Luo 10]: ( ) min Tr V W k (V ) H ) + Tr (U j) (76a) {W k },{U j }, t M,t k I F K j I J s.t. ({W k }, {U j}, t M ) C M, (76b) ({W k }, {U j}, t F ) C F, W k 0, k I K, U j 0, j I J, (76c) (76d) C M and C F are approximate conservative convex constraint sets to (70b) and (70c) (cf. (78) and (79) in Appendix A), respectively; t M R 3K, t F R 3J are auxiliary variables. [Luo 10] Z.-Q. Luo, W.-K. Ma, A. M.-C. So, Y. Ye, and S. Zhang, Semidefinite relaxation of quadratic optimization problems, IEEE Signal Process. Mag., vol. 27, pp , May / 122

110 Outline (Subsection II) 1 System Model and Problem Formulation 2 Proposed Outage Constrained HyCoBF Design 1 Analog Beamforming Algorithm 2 Digital Robust CoBF Design 3 Simulation Results 4 Conclusions 110 / 122

111 3. Simulation Results Simulation Setting: Users AWGN powers: σ 2 k = σ 2 j = σ 2, k I K, j I J; Target SINRs: γ Mk = γ F j = γ, k, j; SINR outage probabilities: ρ Mk = ρ F j = ρ; CSI error covariance matrices: C MMk = C MF j = ε 2 I NMBS, C F F j = C F Mk = ε 2 I NFBS ; The performance evaluations were performed using CVX for the proposed HyCoBF design. The digital CoBF of the proposed robust HyCoBF design reduces to full digital (FD) CoBF as N MBS = N RF. 111 / 122

112 3. Simulation Results (Power Performance) Total transmit power (dbm) N MBS = 16, K = 4, N FBS = J = 2, N RF = {4, 8, 16}; ρ = 0.1, ε 2 = {0.001, 0.002} Non-robust FD Robust FD ( = 0.002) Proposed HyCoBF RF =4 Proposed HyCoBF RF =8 Proposed HyCoBF RF =16 Robust FD ( = 0.001) Proposed HyCoBF RF =4 Proposed HyCoBF RF =8 Proposed HyCoBF RF = γ(db) ε 2 ε / 122

113 3. Simulation Results (Power Performance) N MBS = 64, K = 4, N FBS {2, 4}, N RF = {4, 8, 16}, J = 2; ρ = 0.1, ε 2 = Total transmit power (dbm) Non-robust FD (FBS =2) Proposed HyCoBF RF = 4 10 Proposed HyCoBF RF = 8 Proposed HyCoBF RF = 16 Non-robust FD (FBS =4) 5 Proposed HyCoBF RF = 4 Proposed HyCoBF RF = 8 Proposed HyCoBF RF = γ(db) 113 / 122

114 3. Simulation Results (Satisfaction Probability) N MBS = 16, N RF = K = 4; N FBS = J = 2; ρ = 0.1, γ = 9 db and ε 2 = Counts over different channel realizations Counts over different channel realizations NonRobust FD Actual SINR satisfaction probability (Mean = ) HyCoBF-PRM Actual SINR satisfaction probability (Mean = ) Counts over different channel realizations Counts over different channel realizations Robust FD Actual SINR satisfaction probability (Mean = ) HyCoBF-MM Actual SINR satisfaction probability ( Mean = ) 114 / 122

115 3. Simulation Results (Rank-one Solution) In simulation, a rank-one solution for W k and U j is obtained if the following conditions hold: λ max(w k) Tr(W k) , λmax(u j ) Tr(U j ) , k I K, j I J. (77) Counts of rank-one solutions and all the feasible solutions for each simulation case for ε 2 {0.01, 0.002} and γ {1, 5, 9, 13} db. ε γ (db) Robust FD (447, 447) (396, 396) (286, 286) (102, 102) HyCoBF-PRM (418, 418) (250, 250) (63, 63) (35, 35) ε γ (db) Robust FD (490, 490) (481, 481) (460, 460) (401, 401) HyCoBF-PRM (413, 413) (402, 402) (336, 336) (191, 191) 115 / 122

116 Outline (Subsection II) 1 System Model and Problem Formulation 2 Proposed Outage Constrained HyCoBF Design 1 Analog Beamforming Algorithm 2 Digital Robust CoBF Design 3 Simulation Results 4 Conclusions 116 / 122

117 4. Conclusions We have introduced an outage probability constrained robust HyCoBF design for a massive MIMO enabled HetNet in the presence of Gaussian CSI errors, by solving a nonconvex total transmit power minimization problem. The robust HyCoBF design consists of: The analog beamforming at MBS is a newly devised low-complexity beam selection scheme by maximizing the ratio of MUEs channel power to FUEs interference channel power, given a DFT matrix codebook. The digital beamforming at MBS and FBS is a conservative approximate CoBF solution (by SDR technique and an extended Bernstein-type inequality). 117 / 122

118 4. Conclusions Simulation results have demonstrated that the proposed robust HyCoBF design algorithm can yield promising performance and, most importantly, it can achieve comparable performance to the FD beamforming scheme with much fewer RF chains. Recently, a distributed implementation for the CoBF solution using ADMM in the proposed HyCoBF has been finished [Xu 17]. [Xu 17] G.-X. Xu, C.-H. Lin, W.-G. Ma, S.-Z. Chen, and Chong-Yung Chi, Outage constrained robust hybrid coordinated beamforming for massive MIMO enabled heterogeneous cellular networks, accepted as a regular paper by IEEE Access. 118 / 122

119 Appdendix A The convex constraint by using Lemma 1 for (70b) can be expressed in the following form: C M = { ({Wk }, {U j}, t M ) Tr(Q MMk ) + Tr(Q F Mk ) + ln(ρ Mk )y Mk + c Mk α Mk [xmmk, x F Mk ] T 0, k IK [ ] vec(qmmk ) x MMk, k I K 2rMMk [ ] vec(qf Mk ) x F Mk, k I K 2rF Mk y Mk I NMBS + Q MMk 0, } y Mk I NFBS + Q F Mk 0, y Mk 0, k I K (78) where t M = [x MM1,..., x MMK, x F M1,..., x F MK, y M1,..., y MK] T R 3K are the introduced auxiliary variables. 119 / 122

120 Appdendix A Similarly, (70c) can be expressed in the following form: { C ({Wk ) F = }, {U j}, t F Tr(Q F F j) + Tr(Q MF j) + ln(ρ F j)y F j + c F j [xf β F j F j, x MF j] T 0, j IJ [ ] vec(qf F j) x F F j, j I J 2rF F j [ ] vec(qmf j) x MF j, j I J 2rMF j y F ji NMBS + Q MF j 0, y F ji NFBS + Q F F j 0, y F j 0, j I J } (79) where t F = [x F F 1,..., x F F J, x MF 1,..., x MF J, y F 1,..., y F J] T R 3J collects all the auxiliary variables in the derivation of (79). 120 / 122

121 Conclusion: A new book Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications Chong-Yung Chi, Wei-Chiang Li, Chia-Hsiang Lin (Publisher: CRC Press, 2017, 432 pages, ISBN: ) Motivation: Most of mathematical books are hard to read for engineering students and professionals due to lack of enough fundamental details and tangible linkage between mathematical theory and applications. The book is written in a causally sequential fashion; namely, one can review/peruse the related materials introduced in early chapters/sections again, to overpass hurdles in reading. Covers convex optimization from fundamentals to advanced applications, while holding a strong link from theory to applications. Provides comprehensive proofs and perspective interpretations, many insightful figures, examples and remarks to illuminate core convex optimization theory. 121 / 122

122 Book features Illustrates, by cutting-edge applications, how to apply the convex optimization theory, like a guided journey/exploration rather than pure mathematics. Has been used for a 2-week short course under the book title at 8 major universities (Shandong Univ, Tsinghua Univ, Tianjin Univ, BJTU, Xiamen Univ., UESTC, SYSU, BUPT) in Mainland China more than 12 times since early Thank you for your attention! Acknowledgment: Financial support by NTHU; my students, visiting students and scholars, short-course participants for almost uncountable questions, interactions and comments over the last 7 years. 122 / 122