Techinical Proofs for Nonlinear Learning using Local Coordinate Coding

Size: px
Start display at page:

Download "Techinical Proofs for Nonlinear Learning using Local Coordinate Coding"

Transcription

1 Techinical Proofs for Nonlinear Learning using Local Coordinate Coding 1 Notations and Main Results Denition 1.1 (Lipschitz Smoothness) A function f(x) on R d is (α, β, p)-lipschitz smooth with respect to a norm if f(x ) f(x) α x x, and where we assume α, β > 0 and p (0, 1]. f(x ) f(x) f(x) (x x) β x x 1+p, Denition 1.2 (Coordinate Coding) A coordinate coding is a pair (γ, C), where C R d is a set of anchor points, and γ is a map of x R d to γ v (x)] R C such that v γ v(x) = 1. It induces the following physical approximation of x in R d : γ(x) = γ v (x)v. Moreover, for all x R d, we dene the coding norm as ( ) 1/2 x γ = γ v (x) 2. Proposition 1.1 The map x γ v(x)v is invariant under any shift of the origin for representing data points in R d if and only if v γ v(x) = 1. Lemma 1.1 (Linearization) Let (γ, C) be an arbitrary coordinate coding on R d. (α, β, p)-lipschitz smooth function. We have for all x R d : f(x) γ v (x)f(v) α x γ(x) + β γ v (x) v γ(x) 1+p. Let f be an Denition 1.3 (Localization Measure) Given α, β, p, and coding (γ, C), we dene Q α,β,p (γ, C) = E x α x γ(x) + β ] γ v (x) v γ(x) 1+p. 1

2 Denition 1.4 (Manifold) A subset M R d is called a p-smooth (p > 0) manifold with intrinsic dimensionality m = m(m) if there exists a constant c p (M) such that given any x M, there exists m vectors v 1 (x),..., v m (x) R d so that x M: m inf γ R m x x γ j v j (x) c p(m) x x 1+p. Denition 1.5 (Covering Number) Given any subset M R d, and ɛ > 0. The covering number, denoted as N (ɛ, M), is the smallest cardinality of an ɛ-cover C M. That is, sup x M inf x v ɛ. Theorem 1.1 (Manifold Coding) If the data points x lie on a compact p-smooth manifold M, and the norm is dened as x = (x Ax) 1/2 for some positive denite matrix A. Then given any ɛ > 0, there exist anchor points C M and coding γ such that C (1 + m(m))n (ɛ, M), Q α,β,p (γ, C) αc p (M) + (1 + m p m)β] ɛ 1+p. Moreover, for all x M, we have x 2 γ 1 + (1 + m) 2. Given a local-coordinate coding scheme (γ, C), we approximate each f(x) F a α,β,p by f(x) f γ,c (ŵ, x) = ŵ v γ v (x), where we estimate the coecients using ridge regression as: n ŵ v ] = arg min φ (f γ,c (w, x i ), y i ) + λ ] (w v g(v)) 2, (1) w v] i=1 Theorem 1.2 (Generalization Bound) Suppose φ(p, y) is Lipschitz: φ 1 (p, y) B. Consider coordinate coding (γ, C), and the estimation method (1) with random training examples S n = {(x 1, y 1 ),..., (x n, y n )}. Then the expected generalization error satises the inequality: E Sn E x,y φ(f γ,c (ŵ, x), y) inf f F α,β,p E x,y φ (f(x), y) + λ (f(v) g(v)) 2 ] + B2 2λn E x x 2 γ + BQ α,β,p (γ, C). Theorem 1.3 (Consistency) Suppose the data lie on a compact manifold M R d, and the norm is the Euclidean norm in R d. If loss function φ(p, y) is Lipschitz. As n, we choose α, β, α/n, β/n 0 (α, β depends on n), and p = 0. Then it is possible to nd coding (γ, C) using unlabeled data such that C /n 0 and Q α,β,p (γ, C) 0. If we pick λn, and λ C 0. Then the local coordinate coding method (1) with g(v) 0 is consistent as n : lim E S n n E x,y φ(f(ŵ, x), y) = inf E x,yφ (f(x), y). f:m R 2

3 2 Proofs 2.1 Proof of Proposition 1.1 Consider a change of the R d origin by u R d, which shifts any point x R d to x + u, and points v C to v + u. The shift-invariance requirement implies that after the change, we map x + u to γ v(x)v + u, which should equal γ v(x)(v + u). This is equivalent to u = γ v(x)u, which holds if and only if γ v(x) = Proof of Lemma 1.1 For simplicity, let γ v = γ v (x) and x = γ(x) = γ vv. We have f(x) γ v f(v) f(x) f(x ) + γ v (f(v) f(x )) = f(x) f(x ) + γ v (f(v) f(x ) f(x ) (v x )) This implies the bound. 2.3 Proof of Theorem 1.1 f(x) f(x ) + α x x 2 + β γ v (f(v) f(x ) f(x ) (v x )) γ v x v 1+p. Let m = m(m). Given any ɛ > 0, consider an ɛ-cover C of M with C N (ɛ, M). Given each u C, dene C u = {v 1 (u),..., v d (u)}, where v j (u) are dened in Denition 1.4. Dene the anchor points as C = u C {u + v j (u) : j = 1,..., m} C. It follows that C (1 + m)n (ɛ, M). In the following, we only need to prove the existence of a coding γ on M that satises the requirement of the theorem. Without loss of generality, we assume that v j (u) = ɛ for each u and j, and given u, {v j (u) : j = 1,..., m} are orthogonal with respect to A: vj (u)av k(u) = 0 when j k. For each x M, let u x C be the closest point to x in C. We have x u x ɛ by the denition of C. Now, Denition 1.4 implies that there exists γ j (x) (j = 1,..., m) such that m x u x γ j(x)v j (u x ) c p(m)ɛ 1+p. The optimal choice is the A-projection of x u x to the subspace spanned by {v j (u x ) : j = 1,..., m}. The orthogonality condition thus implies that m γ j(x) 2 v j (u x ) 2 x u x 2 ɛ 2. 3

4 Therefore which implies that for all x: m γ j(x) 2 1, m γ j(x) m. We can now dene the coordinate coding of x M as γ j v = u x + v j (u x ) γ v (x) = 1 m γ j v = u x 0 otherwise. This implies the following bounds: and x γ(x) c p (M)ɛ 1+p 1 j = 1 m γ j v γ(x) 1+p = γ ux (x) γ(x) u x + where we have used v u x = ɛ, and γ(x) u x x u x ɛ. 2.4 Proof of Theorem 1.2 m γ j(x) (v u x ) (γ(x) u x ) 1+p (1 + m m)ɛ 1+p γ j(x) (ɛ + ɛ) 1+p (3) =1 + m p m]ɛ 1+p. (4) Consider n + 1 samples S = {(x 1, y 1 ),..., (x, y )}. We shall introduce the following notation: 1 w v ] = arg min φ (f γ,c (w, x i ), y i ) + λ ] wv 2. (5) w v] n i=1 Let k be an integer randomly drawn from {1,..., n + 1}. Let ŵ v (k) ] be the solution of ŵ v (k) ] = arg min 1 φ (f γ,c (w, x i ), y i ) + λ w 2 v, w v] n i=1,...,;i k with the k-th example left-out. We have the following stability lemma from 1], which can be stated as follows using our terminology: (2) 4

5 Lemma 2.1 The following inequality holds f γ,c (ŵ (k), x k ) f γ,c ( w, x k ) x k 2 γ 2λn φ 1(f γ,c ( w, x k ), y k ). By using Lemma 2.1, we obtain for all α > 0: φ(f γ,c ( w, x k ), y k ) φ(f γ,c (ŵ (k), x k ), y k ) =φ(f γ,c ( w, x k ), y k ) φ(f γ,c (ŵ (k), x k ), y k ) φ 1(f γ,c (ŵ (k), x k ), y k )(f γ,c ( w, x k ) f γ,c (ŵ (k), x k )) + φ 1(f γ,c (ŵ (k), x k ), y k )(f γ,c ( w, x k ) f γ,c (ŵ (k), x k )) φ 1(f γ,c (ŵ (k), x k ), y k )(f γ,c ( w, x k ) f γ,c (ŵ (k), x k )) φ 1(f γ,c (ŵ (k), x k ), y k ) 2 x k 2 γ/(2λn) B 2 x k 2 γ/(2λn). In the above derivation, the rst inequality uses the convexity of φ(f, y) with respect to f, which implies that φ(f 1, y) φ(f 2, y) φ 1 (f 2, y)(f 1 f 2 ) 0. The second inequality uses Lemma 2.1, and the third inequality uses the assumption of the loss function. Now by summing over k, and consider any xed f F α,β,p, we obtain: φ(f γ,c (ŵ (k), x k ), y k ) n n φ(f γ,c ( w, x k ), y k ) + B ] 2λn x k 2 γ ( ) 1 φ γ v (x k )f(v), y k + λ ] f(v) 2 n 1 (f(x k ), y k ) + BQ(x k )] + λ n φ ] f(v) 2 + B2 2λn + B2 2λn x k 2 γ x k 2 γ, where Q(x) = α x γ(x) + β γ v(x) v γ(x) 1+p. In the above derivation, the second inequality follows from the denition of w as the minimizer of (5). The third inequality follows from Lemma 1.1. Now by taking expectation with respect to S, we obtain (n + 1)E S φ(f γ,c (ŵ (), x ), y ) n + 1 n n E x,yφ (f(x), y) + n + 1 n BQ α,β,p(γ, C) + λ ] f(v) 2 This implies the desired bound. 2.5 Proof of Theorem B2 (n + 1) E x x 2 2λn γ. Note that any measurable function f : M R can be approximated by F α,β,p with α, β and p = 0. Therefore we only need to show lim E S n n E x,y φ(f γ,c (ŵ, x), y) = lim inf E x,y φ (f(x), y). n f F α,β,p 5

6 Theorem 1.1 implies that it is possible to pick (γ, C) such that C /n 0 and Q α,β,p (γ, C) 0. Moreover, x γ is bounded. Given any f F α,β,0 and any n independent xed A > 0; if we let f A (x) = max(min(f(x), A), A), then it is clear that f A (x) F α,α+β,0. Therefore Theorem 1.2 implies that as n, E Sn E x,y φ(f γ,c (ŵ, x), y) E x,y φ (f A (x), y) + o(1). Since A is arbitrary, we let A to obtain the desired result. References 1] Tong Zhang. Leave-one-out bounds for kernel methods. Neural Computation, 15: ,

Nonlinear Learning using Local Coordinate Coding

Nonlinear Learning using Local Coordinate Coding Nonlinear Learning using Local Coordinate Coding Kai Yu NEC Laboratories America kyu@sv.nec-labs.com Tong Zhang Rutgers University tzhang@stat.rutgers.edu Yihong Gong NEC Laboratories America ygong@sv.nec-labs.com

More information

Linear Regression and Its Applications

Linear Regression and Its Applications Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start

More information

Improved Local Coordinate Coding using Local Tangents

Improved Local Coordinate Coding using Local Tangents Improved Local Coordinate Coding using Local Tangents Kai Yu NEC Laboratories America, 10081 N. Wolfe Road, Cupertino, CA 95129 Tong Zhang Rutgers University, 110 Frelinghuysen Road, Piscataway, NJ 08854

More information

Richard DiSalvo. Dr. Elmer. Mathematical Foundations of Economics. Fall/Spring,

Richard DiSalvo. Dr. Elmer. Mathematical Foundations of Economics. Fall/Spring, The Finite Dimensional Normed Linear Space Theorem Richard DiSalvo Dr. Elmer Mathematical Foundations of Economics Fall/Spring, 20-202 The claim that follows, which I have called the nite-dimensional normed

More information

STATISTICAL BEHAVIOR AND CONSISTENCY OF CLASSIFICATION METHODS BASED ON CONVEX RISK MINIMIZATION

STATISTICAL BEHAVIOR AND CONSISTENCY OF CLASSIFICATION METHODS BASED ON CONVEX RISK MINIMIZATION STATISTICAL BEHAVIOR AND CONSISTENCY OF CLASSIFICATION METHODS BASED ON CONVEX RISK MINIMIZATION Tong Zhang The Annals of Statistics, 2004 Outline Motivation Approximation error under convex risk minimization

More information

1 Solutions to selected problems

1 Solutions to selected problems 1 Solutions to selected problems 1. Let A B R n. Show that int A int B but in general bd A bd B. Solution. Let x int A. Then there is ɛ > 0 such that B ɛ (x) A B. This shows x int B. If A = [0, 1] and

More information

' '-'in.-i 1 'iritt in \ rrivfi pr' 1 p. ru

' '-'in.-i 1 'iritt in \ rrivfi pr' 1 p. ru V X X Y Y 7 VY Y Y F # < F V 6 7»< V q q $ $» q & V 7» Q F Y Q 6 Q Y F & Q &» & V V» Y V Y [ & Y V» & VV & F > V } & F Q \ Q \» Y / 7 F F V 7 7 x» > QX < #» > X >» < F & V F» > > # < q V 6 & Y Y q < &

More information

Problem Set 6: Solutions Math 201A: Fall a n x n,

Problem Set 6: Solutions Math 201A: Fall a n x n, Problem Set 6: Solutions Math 201A: Fall 2016 Problem 1. Is (x n ) n=0 a Schauder basis of C([0, 1])? No. If f(x) = a n x n, n=0 where the series converges uniformly on [0, 1], then f has a power series

More information

l(y j ) = 0 for all y j (1)

l(y j ) = 0 for all y j (1) Problem 1. The closed linear span of a subset {y j } of a normed vector space is defined as the intersection of all closed subspaces containing all y j and thus the smallest such subspace. 1 Show that

More information

Mathematics 530. Practice Problems. n + 1 }

Mathematics 530. Practice Problems. n + 1 } Department of Mathematical Sciences University of Delaware Prof. T. Angell October 19, 2015 Mathematics 530 Practice Problems 1. Recall that an indifference relation on a partially ordered set is defined

More information

Midterm 1. Every element of the set of functions is continuous

Midterm 1. Every element of the set of functions is continuous Econ 200 Mathematics for Economists Midterm Question.- Consider the set of functions F C(0, ) dened by { } F = f C(0, ) f(x) = ax b, a A R and b B R That is, F is a subset of the set of continuous functions

More information

CS229T/STATS231: Statistical Learning Theory. Lecturer: Tengyu Ma Lecture 11 Scribe: Jongho Kim, Jamie Kang October 29th, 2018

CS229T/STATS231: Statistical Learning Theory. Lecturer: Tengyu Ma Lecture 11 Scribe: Jongho Kim, Jamie Kang October 29th, 2018 CS229T/STATS231: Statistical Learning Theory Lecturer: Tengyu Ma Lecture 11 Scribe: Jongho Kim, Jamie Kang October 29th, 2018 1 Overview This lecture mainly covers Recall the statistical theory of GANs

More information

EC 521 MATHEMATICAL METHODS FOR ECONOMICS. Lecture 1: Preliminaries

EC 521 MATHEMATICAL METHODS FOR ECONOMICS. Lecture 1: Preliminaries EC 521 MATHEMATICAL METHODS FOR ECONOMICS Lecture 1: Preliminaries Murat YILMAZ Boğaziçi University In this lecture we provide some basic facts from both Linear Algebra and Real Analysis, which are going

More information

Approximation by ridge functions with weights in a specified set

Approximation by ridge functions with weights in a specified set Approximation by ridge functions with weights in a specified set David Stewart & Palle Jorgensen University of Iowa Mathematics September 11, 2018 Extreme Learning Machines Extreme Learning Machines are

More information

Second Order Elliptic PDE

Second Order Elliptic PDE Second Order Elliptic PDE T. Muthukumar tmk@iitk.ac.in December 16, 2014 Contents 1 A Quick Introduction to PDE 1 2 Classification of Second Order PDE 3 3 Linear Second Order Elliptic Operators 4 4 Periodic

More information

Linear Analysis Lecture 5

Linear Analysis Lecture 5 Linear Analysis Lecture 5 Inner Products and V Let dim V < with inner product,. Choose a basis B and let v, w V have coordinates in F n given by x 1. x n and y 1. y n, respectively. Let A F n n be the

More information

Linear Algebra Massoud Malek

Linear Algebra Massoud Malek CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

More information

5 Banach Algebras. 5.1 Invertibility and the Spectrum. Robert Oeckl FA NOTES 5 19/05/2010 1

5 Banach Algebras. 5.1 Invertibility and the Spectrum. Robert Oeckl FA NOTES 5 19/05/2010 1 Robert Oeckl FA NOTES 5 19/05/2010 1 5 Banach Algebras 5.1 Invertibility and the Spectrum Suppose X is a Banach space. Then we are often interested in (continuous) operators on this space, i.e, elements

More information

UCLA Chemical Engineering. Process & Control Systems Engineering Laboratory

UCLA Chemical Engineering. Process & Control Systems Engineering Laboratory Constrained Innite-Time Nonlinear Quadratic Optimal Control V. Manousiouthakis D. Chmielewski Chemical Engineering Department UCLA 1998 AIChE Annual Meeting Outline Unconstrained Innite-Time Nonlinear

More information

Semi-strongly asymptotically non-expansive mappings and their applications on xed point theory

Semi-strongly asymptotically non-expansive mappings and their applications on xed point theory Hacettepe Journal of Mathematics and Statistics Volume 46 (4) (2017), 613 620 Semi-strongly asymptotically non-expansive mappings and their applications on xed point theory Chris Lennard and Veysel Nezir

More information

12 CHAPTER 1. PRELIMINARIES Lemma 1.3 (Cauchy-Schwarz inequality) Let (; ) be an inner product in < n. Then for all x; y 2 < n we have j(x; y)j (x; x)

12 CHAPTER 1. PRELIMINARIES Lemma 1.3 (Cauchy-Schwarz inequality) Let (; ) be an inner product in < n. Then for all x; y 2 < n we have j(x; y)j (x; x) 1.4. INNER PRODUCTS,VECTOR NORMS, AND MATRIX NORMS 11 The estimate ^ is unbiased, but E(^ 2 ) = n?1 n 2 and is thus biased. An unbiased estimate is ^ 2 = 1 (x i? ^) 2 : n? 1 In x?? we show that the linear

More information

Generalization theory

Generalization theory Generalization theory Daniel Hsu Columbia TRIPODS Bootcamp 1 Motivation 2 Support vector machines X = R d, Y = { 1, +1}. Return solution ŵ R d to following optimization problem: λ min w R d 2 w 2 2 + 1

More information

satisfying ( i ; j ) = ij Here ij = if i = j and 0 otherwise The idea to use lattices is the following Suppose we are given a lattice L and a point ~x

satisfying ( i ; j ) = ij Here ij = if i = j and 0 otherwise The idea to use lattices is the following Suppose we are given a lattice L and a point ~x Dual Vectors and Lower Bounds for the Nearest Lattice Point Problem Johan Hastad* MIT Abstract: We prove that given a point ~z outside a given lattice L then there is a dual vector which gives a fairly

More information

Projection Theorem 1

Projection Theorem 1 Projection Theorem 1 Cauchy-Schwarz Inequality Lemma. (Cauchy-Schwarz Inequality) For all x, y in an inner product space, [ xy, ] x y. Equality holds if and only if x y or y θ. Proof. If y θ, the inequality

More information

j=1 [We will show that the triangle inequality holds for each p-norm in Chapter 3 Section 6.] The 1-norm is A F = tr(a H A).

j=1 [We will show that the triangle inequality holds for each p-norm in Chapter 3 Section 6.] The 1-norm is A F = tr(a H A). Math 344 Lecture #19 3.5 Normed Linear Spaces Definition 3.5.1. A seminorm on a vector space V over F is a map : V R that for all x, y V and for all α F satisfies (i) x 0 (positivity), (ii) αx = α x (scale

More information

A SYNOPSIS OF HILBERT SPACE THEORY

A SYNOPSIS OF HILBERT SPACE THEORY A SYNOPSIS OF HILBERT SPACE THEORY Below is a summary of Hilbert space theory that you find in more detail in any book on functional analysis, like the one Akhiezer and Glazman, the one by Kreiszig or

More information

Convex Analysis and Optimization Chapter 2 Solutions

Convex Analysis and Optimization Chapter 2 Solutions Convex Analysis and Optimization Chapter 2 Solutions Dimitri P. Bertsekas with Angelia Nedić and Asuman E. Ozdaglar Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com

More information

INVERSE FUNCTION THEOREM and SURFACES IN R n

INVERSE FUNCTION THEOREM and SURFACES IN R n INVERSE FUNCTION THEOREM and SURFACES IN R n Let f C k (U; R n ), with U R n open. Assume df(a) GL(R n ), where a U. The Inverse Function Theorem says there is an open neighborhood V U of a in R n so that

More information

quantitative information on the error caused by using the solution of the linear problem to describe the response of the elastic material on a corner

quantitative information on the error caused by using the solution of the linear problem to describe the response of the elastic material on a corner Quantitative Justication of Linearization in Nonlinear Hencky Material Problems 1 Weimin Han and Hong-ci Huang 3 Abstract. The classical linear elasticity theory is based on the assumption that the size

More information

Converse Lyapunov Functions for Inclusions 2 Basic denitions Given a set A, A stands for the closure of A, A stands for the interior set of A, coa sta

Converse Lyapunov Functions for Inclusions 2 Basic denitions Given a set A, A stands for the closure of A, A stands for the interior set of A, coa sta A smooth Lyapunov function from a class-kl estimate involving two positive semidenite functions Andrew R. Teel y ECE Dept. University of California Santa Barbara, CA 93106 teel@ece.ucsb.edu Laurent Praly

More information

Plan of Class 4. Radial Basis Functions with moving centers. Projection Pursuit Regression and ridge. Principal Component Analysis: basic ideas

Plan of Class 4. Radial Basis Functions with moving centers. Projection Pursuit Regression and ridge. Principal Component Analysis: basic ideas Plan of Class 4 Radial Basis Functions with moving centers Multilayer Perceptrons Projection Pursuit Regression and ridge functions approximation Principal Component Analysis: basic ideas Radial Basis

More information

Introduction to Empirical Processes and Semiparametric Inference Lecture 13: Entropy Calculations

Introduction to Empirical Processes and Semiparametric Inference Lecture 13: Entropy Calculations Introduction to Empirical Processes and Semiparametric Inference Lecture 13: Entropy Calculations Michael R. Kosorok, Ph.D. Professor and Chair of Biostatistics Professor of Statistics and Operations Research

More information

2 Garrett: `A Good Spectral Theorem' 1. von Neumann algebras, density theorem The commutant of a subring S of a ring R is S 0 = fr 2 R : rs = sr; 8s 2

2 Garrett: `A Good Spectral Theorem' 1. von Neumann algebras, density theorem The commutant of a subring S of a ring R is S 0 = fr 2 R : rs = sr; 8s 2 1 A Good Spectral Theorem c1996, Paul Garrett, garrett@math.umn.edu version February 12, 1996 1 Measurable Hilbert bundles Measurable Banach bundles Direct integrals of Hilbert spaces Trivializing Hilbert

More information

Nonlinear Control. Nonlinear Control Lecture # 8 Time Varying and Perturbed Systems

Nonlinear Control. Nonlinear Control Lecture # 8 Time Varying and Perturbed Systems Nonlinear Control Lecture # 8 Time Varying and Perturbed Systems Time-varying Systems ẋ = f(t,x) f(t,x) is piecewise continuous in t and locally Lipschitz in x for all t 0 and all x D, (0 D). The origin

More information

Optimality Conditions for Nonsmooth Convex Optimization

Optimality Conditions for Nonsmooth Convex Optimization Optimality Conditions for Nonsmooth Convex Optimization Sangkyun Lee Oct 22, 2014 Let us consider a convex function f : R n R, where R is the extended real field, R := R {, + }, which is proper (f never

More information

Functional Analysis. Franck Sueur Metric spaces Definitions Completeness Compactness Separability...

Functional Analysis. Franck Sueur Metric spaces Definitions Completeness Compactness Separability... Functional Analysis Franck Sueur 2018-2019 Contents 1 Metric spaces 1 1.1 Definitions........................................ 1 1.2 Completeness...................................... 3 1.3 Compactness......................................

More information

ON TRIVIAL GRADIENT YOUNG MEASURES BAISHENG YAN Abstract. We give a condition on a closed set K of real nm matrices which ensures that any W 1 p -grad

ON TRIVIAL GRADIENT YOUNG MEASURES BAISHENG YAN Abstract. We give a condition on a closed set K of real nm matrices which ensures that any W 1 p -grad ON TRIVIAL GRAIENT YOUNG MEASURES BAISHENG YAN Abstract. We give a condition on a closed set K of real nm matrices which ensures that any W 1 p -gradient Young measure supported on K must be trivial the

More information

Basic Theorems about Independence, Spanning, Basis, and Dimension

Basic Theorems about Independence, Spanning, Basis, and Dimension Basic Theorems about Independence, Spanning, Basis, and Dimension Theorem 1 If v 1 ; : : : ; v m span V; then every set of vectors in V with more than m vectors must be linearly dependent. Proof. Let w

More information

1. Subspaces A subset M of Hilbert space H is a subspace of it is closed under the operation of forming linear combinations;i.e.,

1. Subspaces A subset M of Hilbert space H is a subspace of it is closed under the operation of forming linear combinations;i.e., Abstract Hilbert Space Results We have learned a little about the Hilbert spaces L U and and we have at least defined H 1 U and the scale of Hilbert spaces H p U. Now we are going to develop additional

More information

DIFFERENTIAL OPERATORS ON DOMAINS WITH CONICAL POINTS: PRECISE UNIFORM REGULARITY ESTIMATES

DIFFERENTIAL OPERATORS ON DOMAINS WITH CONICAL POINTS: PRECISE UNIFORM REGULARITY ESTIMATES Dedicated to Monique Dauge on her 60 th birthday DIFFERENTIAL OPERATORS ON DOMAINS WITH CONICAL POINTS: PRECISE UNIFORM REGULARITY ESTIMATES CONSTANTIN BÄCU TÄ, HENGGUANG LI and VICTOR NISTOR Communicated

More information

Variational Formulations

Variational Formulations Chapter 2 Variational Formulations In this chapter we will derive a variational (or weak) formulation of the elliptic boundary value problem (1.4). We will discuss all fundamental theoretical results that

More information

UNBOUNDED OPERATORS ON HILBERT SPACES. Let X and Y be normed linear spaces, and suppose A : X Y is a linear map.

UNBOUNDED OPERATORS ON HILBERT SPACES. Let X and Y be normed linear spaces, and suppose A : X Y is a linear map. UNBOUNDED OPERATORS ON HILBERT SPACES EFTON PARK Let X and Y be normed linear spaces, and suppose A : X Y is a linear map. Define { } Ax A op = sup x : x 0 = { Ax : x 1} = { Ax : x = 1} If A

More information

A NOTE ON LINEAR FUNCTIONAL NORMS

A NOTE ON LINEAR FUNCTIONAL NORMS A NOTE ON LINEAR FUNCTIONAL NORMS YIFEI PAN AND MEI WANG Abstract. For a vector u in a normed linear space, Hahn-Banach Theorem provides the existence of a linear functional f, f(u) = u such that f = 1.

More information

Locally convex spaces, the hyperplane separation theorem, and the Krein-Milman theorem

Locally convex spaces, the hyperplane separation theorem, and the Krein-Milman theorem 56 Chapter 7 Locally convex spaces, the hyperplane separation theorem, and the Krein-Milman theorem Recall that C(X) is not a normed linear space when X is not compact. On the other hand we could use semi

More information

Applied Analysis (APPM 5440): Final exam 1:30pm 4:00pm, Dec. 14, Closed books.

Applied Analysis (APPM 5440): Final exam 1:30pm 4:00pm, Dec. 14, Closed books. Applied Analysis APPM 44: Final exam 1:3pm 4:pm, Dec. 14, 29. Closed books. Problem 1: 2p Set I = [, 1]. Prove that there is a continuous function u on I such that 1 ux 1 x sin ut 2 dt = cosx, x I. Define

More information

Problem Set 5: Solutions Math 201A: Fall 2016

Problem Set 5: Solutions Math 201A: Fall 2016 Problem Set 5: s Math 21A: Fall 216 Problem 1. Define f : [1, ) [1, ) by f(x) = x + 1/x. Show that f(x) f(y) < x y for all x, y [1, ) with x y, but f has no fixed point. Why doesn t this example contradict

More information

From Fractional Brownian Motion to Multifractional Brownian Motion

From Fractional Brownian Motion to Multifractional Brownian Motion From Fractional Brownian Motion to Multifractional Brownian Motion Antoine Ayache USTL (Lille) Antoine.Ayache@math.univ-lille1.fr Cassino December 2010 A.Ayache (USTL) From FBM to MBM Cassino December

More information

6. Duals of L p spaces

6. Duals of L p spaces 6 Duals of L p spaces This section deals with the problem if identifying the duals of L p spaces, p [1, ) There are essentially two cases of this problem: (i) p = 1; (ii) 1 < p < The major difference between

More information

i=1 β i,i.e. = β 1 x β x β 1 1 xβ d

i=1 β i,i.e. = β 1 x β x β 1 1 xβ d 66 2. Every family of seminorms on a vector space containing a norm induces ahausdorff locally convex topology. 3. Given an open subset Ω of R d with the euclidean topology, the space C(Ω) of real valued

More information

Boosting with Early Stopping: Convergence and Consistency

Boosting with Early Stopping: Convergence and Consistency Boosting with Early Stopping: Convergence and Consistency Tong Zhang Bin Yu Abstract Boosting is one of the most significant advances in machine learning for classification and regression. In its original

More information

Chapter 1. Optimality Conditions: Unconstrained Optimization. 1.1 Differentiable Problems

Chapter 1. Optimality Conditions: Unconstrained Optimization. 1.1 Differentiable Problems Chapter 1 Optimality Conditions: Unconstrained Optimization 1.1 Differentiable Problems Consider the problem of minimizing the function f : R n R where f is twice continuously differentiable on R n : P

More information

CHARACTERISTIC POINTS, RECTIFIABILITY AND PERIMETER MEASURE ON STRATIFIED GROUPS

CHARACTERISTIC POINTS, RECTIFIABILITY AND PERIMETER MEASURE ON STRATIFIED GROUPS CHARACTERISTIC POINTS, RECTIFIABILITY AND PERIMETER MEASURE ON STRATIFIED GROUPS VALENTINO MAGNANI Abstract. We establish an explicit formula between the perimeter measure of an open set E with C 1 boundary

More information

Vector Spaces ปร ภ ม เวกเตอร

Vector Spaces ปร ภ ม เวกเตอร Vector Spaces ปร ภ ม เวกเตอร 5.1 Real Vector Spaces ปร ภ ม เวกเตอร ของจ านวนจร ง Vector Space Axioms (1/2) Let V be an arbitrary nonempty set of objects on which two operations are defined, addition and

More information

MARKOV CHAINS: STATIONARY DISTRIBUTIONS AND FUNCTIONS ON STATE SPACES. Contents

MARKOV CHAINS: STATIONARY DISTRIBUTIONS AND FUNCTIONS ON STATE SPACES. Contents MARKOV CHAINS: STATIONARY DISTRIBUTIONS AND FUNCTIONS ON STATE SPACES JAMES READY Abstract. In this paper, we rst introduce the concepts of Markov Chains and their stationary distributions. We then discuss

More information

Linear Algebra (part 1) : Vector Spaces (by Evan Dummit, 2017, v. 1.07) 1.1 The Formal Denition of a Vector Space

Linear Algebra (part 1) : Vector Spaces (by Evan Dummit, 2017, v. 1.07) 1.1 The Formal Denition of a Vector Space Linear Algebra (part 1) : Vector Spaces (by Evan Dummit, 2017, v. 1.07) Contents 1 Vector Spaces 1 1.1 The Formal Denition of a Vector Space.................................. 1 1.2 Subspaces...................................................

More information

Computation Of Asymptotic Distribution. For Semiparametric GMM Estimators. Hidehiko Ichimura. Graduate School of Public Policy

Computation Of Asymptotic Distribution. For Semiparametric GMM Estimators. Hidehiko Ichimura. Graduate School of Public Policy Computation Of Asymptotic Distribution For Semiparametric GMM Estimators Hidehiko Ichimura Graduate School of Public Policy and Graduate School of Economics University of Tokyo A Conference in honor of

More information

r( = f 2 L 2 (1.2) iku)! 0 as r!1: (1.3) It was shown in book [7] that if f is assumed to be the restriction of a function in C

r(  = f 2 L 2 (1.2) iku)! 0 as r!1: (1.3) It was shown in book [7] that if f is assumed to be the restriction of a function in C Inverse Obstacle Problem: Local Uniqueness for Rougher Obstacles and the Identication of A Ball Changmei Liu Department of Mathematics University of North Carolina Chapel Hill, NC 27599 December, 1995

More information

Lecture 2: Review of Prerequisites. Table of contents

Lecture 2: Review of Prerequisites. Table of contents Math 348 Fall 217 Lecture 2: Review of Prerequisites Disclaimer. As we have a textbook, this lecture note is for guidance and supplement only. It should not be relied on when preparing for exams. In this

More information

Honours Analysis III

Honours Analysis III Honours Analysis III Math 354 Prof. Dmitry Jacobson Notes Taken By: R. Gibson Fall 2010 1 Contents 1 Overview 3 1.1 p-adic Distance............................................ 4 2 Introduction 5 2.1 Normed

More information

Reproducing Kernel Hilbert Spaces Class 03, 15 February 2006 Andrea Caponnetto

Reproducing Kernel Hilbert Spaces Class 03, 15 February 2006 Andrea Caponnetto Reproducing Kernel Hilbert Spaces 9.520 Class 03, 15 February 2006 Andrea Caponnetto About this class Goal To introduce a particularly useful family of hypothesis spaces called Reproducing Kernel Hilbert

More information

Measure and Integration: Solutions of CW2

Measure and Integration: Solutions of CW2 Measure and Integration: s of CW2 Fall 206 [G. Holzegel] December 9, 206 Problem of Sheet 5 a) Left (f n ) and (g n ) be sequences of integrable functions with f n (x) f (x) and g n (x) g (x) for almost

More information

Importance Sampling for Minibatches

Importance Sampling for Minibatches Importance Sampling for Minibatches Dominik Csiba School of Mathematics University of Edinburgh 07.09.2016, Birmingham Dominik Csiba (University of Edinburgh) Importance Sampling for Minibatches 07.09.2016,

More information

In English, this means that if we travel on a straight line between any two points in C, then we never leave C.

In English, this means that if we travel on a straight line between any two points in C, then we never leave C. Convex sets In this section, we will be introduced to some of the mathematical fundamentals of convex sets. In order to motivate some of the definitions, we will look at the closest point problem from

More information

A Proof of the EOQ Formula Using Quasi-Variational. Inequalities. March 19, Abstract

A Proof of the EOQ Formula Using Quasi-Variational. Inequalities. March 19, Abstract A Proof of the EOQ Formula Using Quasi-Variational Inequalities Dir Beyer y and Suresh P. Sethi z March, 8 Abstract In this paper, we use quasi-variational inequalities to provide a rigorous proof of the

More information

Pointwise convergence rate for nonlinear conservation. Eitan Tadmor and Tao Tang

Pointwise convergence rate for nonlinear conservation. Eitan Tadmor and Tao Tang Pointwise convergence rate for nonlinear conservation laws Eitan Tadmor and Tao Tang Abstract. We introduce a new method to obtain pointwise error estimates for vanishing viscosity and nite dierence approximations

More information

Some Results on b-orthogonality in 2-Normed Linear Spaces

Some Results on b-orthogonality in 2-Normed Linear Spaces Int. Journal of Math. Analysis, Vol. 1, 2007, no. 14, 681-687 Some Results on b-orthogonality in 2-Normed Linear Spaces H. Mazaheri and S. Golestani Nezhad Department of Mathematics Yazd University, Yazd,

More information

Algorithms for Nonsmooth Optimization

Algorithms for Nonsmooth Optimization Algorithms for Nonsmooth Optimization Frank E. Curtis, Lehigh University presented at Center for Optimization and Statistical Learning, Northwestern University 2 March 2018 Algorithms for Nonsmooth Optimization

More information

MA651 Topology. Lecture 9. Compactness 2.

MA651 Topology. Lecture 9. Compactness 2. MA651 Topology. Lecture 9. Compactness 2. This text is based on the following books: Topology by James Dugundgji Fundamental concepts of topology by Peter O Neil Elements of Mathematics: General Topology

More information

Lecture # 3 Orthogonal Matrices and Matrix Norms. We repeat the definition an orthogonal set and orthornormal set.

Lecture # 3 Orthogonal Matrices and Matrix Norms. We repeat the definition an orthogonal set and orthornormal set. Lecture # 3 Orthogonal Matrices and Matrix Norms We repeat the definition an orthogonal set and orthornormal set. Definition A set of k vectors {u, u 2,..., u k }, where each u i R n, is said to be an

More information

Elements of Convex Optimization Theory

Elements of Convex Optimization Theory Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a self-contained overview of elements of convex optimization

More information

Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version

Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com

More information

Transductive Experiment Design

Transductive Experiment Design Appearing in NIPS 2005 workshop Foundations of Active Learning, Whistler, Canada, December, 2005. Transductive Experiment Design Kai Yu, Jinbo Bi, Volker Tresp Siemens AG 81739 Munich, Germany Abstract

More information

New concepts: Span of a vector set, matrix column space (range) Linearly dependent set of vectors Matrix null space

New concepts: Span of a vector set, matrix column space (range) Linearly dependent set of vectors Matrix null space Lesson 6: Linear independence, matrix column space and null space New concepts: Span of a vector set, matrix column space (range) Linearly dependent set of vectors Matrix null space Two linear systems:

More information

Math 209B Homework 2

Math 209B Homework 2 Math 29B Homework 2 Edward Burkard Note: All vector spaces are over the field F = R or C 4.6. Two Compactness Theorems. 4. Point Set Topology Exercise 6 The product of countably many sequentally compact

More information

University of Missouri. In Partial Fulllment LINDSEY M. WOODLAND MAY 2015

University of Missouri. In Partial Fulllment LINDSEY M. WOODLAND MAY 2015 Frames and applications: Distribution of frame coecients, integer frames and phase retrieval A Dissertation presented to the Faculty of the Graduate School University of Missouri In Partial Fulllment of

More information

Lecture 11 Hyperbolicity.

Lecture 11 Hyperbolicity. Lecture 11 Hyperbolicity. 1 C 0 linearization near a hyperbolic point 2 invariant manifolds Hyperbolic linear maps. Let E be a Banach space. A linear map A : E E is called hyperbolic if we can find closed

More information

An introduction to some aspects of functional analysis

An introduction to some aspects of functional analysis An introduction to some aspects of functional analysis Stephen Semmes Rice University Abstract These informal notes deal with some very basic objects in functional analysis, including norms and seminorms

More information

Topics in Mathematical Economics. Atsushi Kajii Kyoto University

Topics in Mathematical Economics. Atsushi Kajii Kyoto University Topics in Mathematical Economics Atsushi Kajii Kyoto University 26 June 2018 2 Contents 1 Preliminary Mathematics 5 1.1 Topology.................................. 5 1.2 Linear Algebra..............................

More information

Class 2 & 3 Overfitting & Regularization

Class 2 & 3 Overfitting & Regularization Class 2 & 3 Overfitting & Regularization Carlo Ciliberto Department of Computer Science, UCL October 18, 2017 Last Class The goal of Statistical Learning Theory is to find a good estimator f n : X Y, approximating

More information

ON THE REGULARITY OF SAMPLE PATHS OF SUB-ELLIPTIC DIFFUSIONS ON MANIFOLDS

ON THE REGULARITY OF SAMPLE PATHS OF SUB-ELLIPTIC DIFFUSIONS ON MANIFOLDS Bendikov, A. and Saloff-Coste, L. Osaka J. Math. 4 (5), 677 7 ON THE REGULARITY OF SAMPLE PATHS OF SUB-ELLIPTIC DIFFUSIONS ON MANIFOLDS ALEXANDER BENDIKOV and LAURENT SALOFF-COSTE (Received March 4, 4)

More information

Conditional Value-at-Risk (CVaR) Norm: Stochastic Case

Conditional Value-at-Risk (CVaR) Norm: Stochastic Case Conditional Value-at-Risk (CVaR) Norm: Stochastic Case Alexander Mafusalov, Stan Uryasev RESEARCH REPORT 03-5 Risk Management and Financial Engineering Lab Department of Industrial and Systems Engineering

More information

On the Intrinsic Differentiability Theorem of Gromov-Schoen

On the Intrinsic Differentiability Theorem of Gromov-Schoen On the Intrinsic Differentiability Theorem of Gromov-Schoen Georgios Daskalopoulos Brown University daskal@math.brown.edu Chikako Mese 2 Johns Hopkins University cmese@math.jhu.edu Abstract In this note,

More information

Preparatory Material for the European Intensive Program in Bydgoszcz 2011 Analytical and computer assisted methods in mathematical models

Preparatory Material for the European Intensive Program in Bydgoszcz 2011 Analytical and computer assisted methods in mathematical models Preparatory Material for the European Intensive Program in Bydgoszcz 2011 Analytical and computer assisted methods in mathematical models September 4{18 Basics on the Lebesgue integral and the divergence

More information

Viscosity Iterative Approximating the Common Fixed Points of Non-expansive Semigroups in Banach Spaces

Viscosity Iterative Approximating the Common Fixed Points of Non-expansive Semigroups in Banach Spaces Viscosity Iterative Approximating the Common Fixed Points of Non-expansive Semigroups in Banach Spaces YUAN-HENG WANG Zhejiang Normal University Department of Mathematics Yingbing Road 688, 321004 Jinhua

More information

1 Cheeger differentiation

1 Cheeger differentiation 1 Cheeger differentiation after J. Cheeger [1] and S. Keith [3] A summary written by John Mackay Abstract We construct a measurable differentiable structure on any metric measure space that is doubling

More information

arxiv: v1 [math.oc] 21 Mar 2015

arxiv: v1 [math.oc] 21 Mar 2015 Convex KKM maps, monotone operators and Minty variational inequalities arxiv:1503.06363v1 [math.oc] 21 Mar 2015 Marc Lassonde Université des Antilles, 97159 Pointe à Pitre, France E-mail: marc.lassonde@univ-ag.fr

More information

Fractional Roman Domination

Fractional Roman Domination Chapter 6 Fractional Roman Domination It is important to discuss minimality of Roman domination functions before we get into the details of fractional version of Roman domination. Minimality of domination

More information

i. v = 0 if and only if v 0. iii. v + w v + w. (This is the Triangle Inequality.)

i. v = 0 if and only if v 0. iii. v + w v + w. (This is the Triangle Inequality.) Definition 5.5.1. A (real) normed vector space is a real vector space V, equipped with a function called a norm, denoted by, provided that for all v and w in V and for all α R the real number v 0, and

More information

Machine Learning. Support Vector Machines. Fabio Vandin November 20, 2017

Machine Learning. Support Vector Machines. Fabio Vandin November 20, 2017 Machine Learning Support Vector Machines Fabio Vandin November 20, 2017 1 Classification and Margin Consider a classification problem with two classes: instance set X = R d label set Y = { 1, 1}. Training

More information

NORMS ON SPACE OF MATRICES

NORMS ON SPACE OF MATRICES NORMS ON SPACE OF MATRICES. Operator Norms on Space of linear maps Let A be an n n real matrix and x 0 be a vector in R n. We would like to use the Picard iteration method to solve for the following system

More information

Lebesgue Measure on R n

Lebesgue Measure on R n 8 CHAPTER 2 Lebesgue Measure on R n Our goal is to construct a notion of the volume, or Lebesgue measure, of rather general subsets of R n that reduces to the usual volume of elementary geometrical sets

More information

FUNCTIONAL ANALYSIS-NORMED SPACE

FUNCTIONAL ANALYSIS-NORMED SPACE MAT641- MSC Mathematics, MNIT Jaipur FUNCTIONAL ANALYSIS-NORMED SPACE DR. RITU AGARWAL MALAVIYA NATIONAL INSTITUTE OF TECHNOLOGY JAIPUR 1. Normed space Norm generalizes the concept of length in an arbitrary

More information

Sobolev Spaces. Chapter 10

Sobolev Spaces. Chapter 10 Chapter 1 Sobolev Spaces We now define spaces H 1,p (R n ), known as Sobolev spaces. For u to belong to H 1,p (R n ), we require that u L p (R n ) and that u have weak derivatives of first order in L p

More information

Vector Space Basics. 1 Abstract Vector Spaces. 1. (commutativity of vector addition) u + v = v + u. 2. (associativity of vector addition)

Vector Space Basics. 1 Abstract Vector Spaces. 1. (commutativity of vector addition) u + v = v + u. 2. (associativity of vector addition) Vector Space Basics (Remark: these notes are highly formal and may be a useful reference to some students however I am also posting Ray Heitmann's notes to Canvas for students interested in a direct computational

More information

Pavel Dvurechensky Alexander Gasnikov Alexander Tiurin. July 26, 2017

Pavel Dvurechensky Alexander Gasnikov Alexander Tiurin. July 26, 2017 Randomized Similar Triangles Method: A Unifying Framework for Accelerated Randomized Optimization Methods Coordinate Descent, Directional Search, Derivative-Free Method) Pavel Dvurechensky Alexander Gasnikov

More information

Chapter 2: Preliminaries and elements of convex analysis

Chapter 2: Preliminaries and elements of convex analysis Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt-14-15.shtml Academic year 2014-15

More information

LECTURE 16: LIE GROUPS AND THEIR LIE ALGEBRAS. 1. Lie groups

LECTURE 16: LIE GROUPS AND THEIR LIE ALGEBRAS. 1. Lie groups LECTURE 16: LIE GROUPS AND THEIR LIE ALGEBRAS 1. Lie groups A Lie group is a special smooth manifold on which there is a group structure, and moreover, the two structures are compatible. Lie groups are

More information

Value and Policy Iteration

Value and Policy Iteration Chapter 7 Value and Policy Iteration 1 For infinite horizon problems, we need to replace our basic computational tool, the DP algorithm, which we used to compute the optimal cost and policy for finite

More information

Kernel Methods. Jean-Philippe Vert Last update: Jan Jean-Philippe Vert (Mines ParisTech) 1 / 444

Kernel Methods. Jean-Philippe Vert Last update: Jan Jean-Philippe Vert (Mines ParisTech) 1 / 444 Kernel Methods Jean-Philippe Vert Jean-Philippe.Vert@mines.org Last update: Jan 2015 Jean-Philippe Vert (Mines ParisTech) 1 / 444 What we know how to solve Jean-Philippe Vert (Mines ParisTech) 2 / 444

More information

Introduction to Statistical Learning Theory

Introduction to Statistical Learning Theory Introduction to Statistical Learning Theory Definition Reminder: We are given m samples {(x i, y i )} m i=1 Dm and a hypothesis space H and we wish to return h H minimizing L D (h) = E[l(h(x), y)]. Problem

More information