Introduction of Expectation-Maximization Algorithm, Cross-Entropy Method and Genetic Algorithm
|
|
- Kristian James
- 6 years ago
- Views:
Transcription
1 Itroductio of Expectatio-Maximizatio Algorithm, Cross-Etropy Method ad Geetic Algorithm Wireless Iformatio Trasmissio System Lab. Istitute of Commuicatios Egieerig Natioal Su Yat-se Uiversity 2012/07/23 顏愷緯
2 Outlie Itroductio of Expectatio-Maximizatio Algorithm Itroductio of Geetic Algorithm Itroductio of Cross-Etropy Method 2
3 Itroductio of Expectatio-Maximum Algorithm Wireless Iformatio Trasmissio System Lab. Istitute of Commuicatios Egieerig Natioal Su Yat-se Uiversity 2010/08/03 顏愷緯
4 Expectatio-Maximizatio Algorithm Expectatio-maximizatio (EM) algorithm is foud ad give its ame i 1970s. EM algorithm is a algorithm for fidig maximum likelihood (ML) estimates of parameters i probabilistic models, where the model depeds o uobserved latet variables. It is a efficiet ad quite useful algorithm for fidig the ML estimatio of parameters whe the data model is easy to preset.
5 Expectatio-Maximizatio Algorithm EM alterates betwee performig the followig two steps: expectatio (E) step, which computes a expectatio of the likelihood by icludig the latet variables as if they were observed. maximizatio (M) step, which computes the maximum likelihood estimates of the parameters by maximizig the expected likelihood foud o the E step. The parameters foud o the M step are the used to begi aother E step, ad the process is repeated.
6 Expectatio-Maximizatio Algorithm Assume that X is the observable radom vector, z is the missig radom vector. I maximum likelihood (ML) estimatio for θ, we wat to fid θ such that p X is a maximum. The log likelihood fuctio is kow as L p X l. The EM algorithm is a iterative process for maximizig L.
7 Expectatio-Maximizatio Algorithm We would like to fid a ew θ such that where L L, meas the value of the th iteratio. The probability p X ca be writte i terms of the hidde variables z as X X z z p p, p. z Usig Jese s iequality, it was show that, l v l v. i i i i i1 i1
8 Expectatio-Maximizatio Algorithm The the equatio L L ca be expressed as L L p p p z l X z, z l X p zx, l p, p l p X z z X z p, zx X z, pz pzx, p l p, l p z X z p X z, pz pzx, p z X, l l p X z, pz pzx, X p pz X, l p, l p z X X z z z z p p X z, p z zx, l. p, p z X X X
9 For coveiece, we defie the followig equatio Sice L, it ca be said that is bouded by the likelihood fuctio L ad that the value of the fuctios ad X z, pz p L p zx, l L., p p z z X X L Expectatio-Maximizatio Algorithm are equal at curret estimate because that X z, pz z X, p X p L pzx, l L. z p
10 Therefore ay θ which icreases also icrease the. I order to achieve the greatest possible icrease i the value of L, the EM algorithm calls for selectig θ such that is maximized. Formally, we have Expectatio-Maximizatio Algorithm 1 arg max X z, pz L p arg max, L p zx l., p p z z X X
11 Expectatio-Maximizatio Algorithm Drop the term L which is costat with respect to θ. The we ca obtai z X X z z 1 arg max p, l p, p z arg max p z X, l p X, z z arg max E l p,., X z zx
12 Expectatio-Maximizatio Algorithm Thus, the EM algorithm cosists of iteratig the: E-step: Determie the coditioal expectatio 1 Xz Q E l p, zx, M-step: Maximize this expressio with respect to θ. We ca cosider the set of observatios X as beig icomplete i relatio to the complete set {X, z}.
13 Expectatio-Maximizatio Algorithm 1 If the likelihood fuctio is bouded, ad if Q is cotiuous, the log-likelihood fuctio l p X coverges to a statioary poit (a local maximum). Whe the likelihood fuctio has several maxima, the EM algorithm is ot guarateed to coverge to the global maximum. EM is particularly useful whe maximum likelihood estimatio of a complete data model is easy. If Closed form estimators exist, the M step is ofte trivial.
14 Expectatio-Maximizatio Algorithm
15 Expectatio-Maximizatio Algorithm (Example) We trasmit the complex symbol s t ad we receive y tm o receive ateas. The received sigal ca be expressed as y h s, t 1,2,, T; m 1,2,, M. tm m t where h m is the complex-valued fadig coefficiet betwee the trasmitter ad the mth receiver. The fadig coefficiets are assumed to be idepedet ad CN(0, 1) distributed. Let T be the legth durig which chael state iformatio (CSI) remais costat.
16 Expectatio-Maximizatio Algorithm (Example) If oe block of T symbols are set, the received sigal blocks by M receive ateas ca be expressed by a matrix Y of M T. It ca be writte as Y hs N H The oise is assumed to be white, i.e., E Nt N, 1 t Σ t 2 1t2 MM where Σ is the oise covariace matrix.
17 Expectatio-Maximizatio Algorithm (Example) We defie a complete data, X = (Y,h), for the parameter that we wat to estimate. E-step: The E-step of EM algorithm requires the calculatio of where k Q s s E f [ ] [ k l X s Y, s ]. Q s s [ k ] M-step: M-step is to fid the s that maximize the Q fuctio as s arg max Q s s. [ k1] [ k] s
18 Itroductio of Cross-Etropy Method Wireless Iformatio Trasmissio System Lab. Istitute of Commuicatios Egieerig Natioal Su Yat-se Uiversity 2010/08/03 顏愷緯
19 Cross-Etropy Method The Cross-Etropy Method was origially developed as a simulatio method for the estimatio of rare evet probabilities: Estimate P S X X : radom vector/process takig values i some set χ. S: real-valued fuctio o χ. It was soo realized that the CE Method could also be used as a optimizatio method: Determie max S x X 19
20 Cross-Etropy Method May real-world optimizatio problems ca be formulated mathematically as either (or both) Locate some elemet i a set χ such that where S is a objective fuctio or performace measure defied o χ. x * * S x S x for all x * * Fid S x, the globally maximal value of the fuctio. The CE method ivolves a iterative procedure where each iteratio ca be broke ito two phase: Geerate a radom, data sample accordig to specified mechaism. Update the parameters of the radom mechaism based o the data to produce a better sample i the ext iteratio. 20
21 Geeral CE Algorithm: 1) Iitialize the parameters Cross-Etropy Method 2) Geerate the radom sample from the desity fuctio which is updated i previous iteratio ad calculate the fitess value. 3) Update the probability desity fuctio of the radom sample based o the acquired iformatio to produce a better solutio i the ext iteratio. 4) Retur to Step 2 util some stoppig criterio is met. 21
22 Cross-Etropy Method (Example 1) Cosider a biary vector y y1, y2,, y. Suppose that we do t kow which compoets of y are 0 ad which are 1. We have a oracle which for each biary iput vector returs the performace or respose, j j1 S x x y represetig the umber of matches betwee the elemets of x ad y. j, x x1, x2,, x 22
23 Cross-Etropy Method (Example 1) 23
24 Cross-Etropy Method (Example 1) 1) Start with iteratio umber t 1 ad probability 2) Draw a sample X,, 1 XN of Beroulli vectors with success probability vector. p t1 3) Calculate the performaces S X i for all i, ad order them from the smallest to biggest. Let t be 1 sample quatile of the performaces: t S. 1 N p0 0.5,,
25 Cross-Etropy Method (Example 1) 4) Use the same sample to calculate pt pt,1, pt,2,, pt, via p t, j N I j1 N j 1 S Xi t Xij 1 I S X i I t, j = 1,,, where X X, X, X, ad icrease t by 1. i i1 i2 i 5) If the stoppig criterio is met, the stop; otherwise set t = t + 1 ad reiterate from step 2 ad step 3. 25
26 Cross-Etropy Method (Example 1) As a example cosider the case = 10, N = 50 ad ρ = 0.1 where y 1,1,1,1,1,0,0,0,0,0. 26
27 Cross-Etropy Method (Example 2) 27
28 Cross-Etropy Method (Example 2) S = ilie( exp(-(x - 2).^2)) * exp(-(x + 2).^2) ); mu = -10; sigma = 10; rho = 0.1; N = 100; eps = 1E-3; t = 0; while sigma > eps t = t + 1; x = mu + sigma * rad(n,1); SX = S(x); % compute the performace mu = mea(sortsx((1 - rho) * N:N,1)); sigma = std(sortsx((1 - rho) * N:N,1)); ed 28
29 Cross-Etropy Method (Example 2) 29
30 Cross-Etropy Method Some strategies to improve the performace of CE method: Elite reservatio. Smoothig factor. The umber of ρ. 30
31 Itroductio of Geetic Algorithm Wireless Iformatio Trasmissio System Lab. Istitute of Commuicatios Egieerig Natioal Su Yat-se Uiversity 2010/08/03 顏愷緯
32 Geetic Algorithm Geetic Algorithm (GA), first itroduced by Joh Hollad i the early seveties, is the powerful stochastic algorithm based o the priciples of atural selectio ad atural geetics. They are used as optimizatio methods ad have show their effectiveess i various problems. GAs evolves solutios i a iterative maer by applyig geetic operators to a pool of cadidate solutios. To solve a problem, a GA maitais a populatio of idividuals ad probabilistically modifies the populatio by some geetic operators such as selectio, crossover ad mutatio, with the itet of seekig a suboptimal solutio to the problem. 32
33 Geetic Algorithm Start 1 Iitial populatio Geetic Operators (mutatio ad crossover) Compute the fitess fuctio Termial criteria No 2 Yes 2 Geetic Operators (selectio) ed 1 33
34 Geetic Algorithm I GA, each idividual i a populatio is usually coded as a fixedlegth biary strig. The legth of the strig depeds o the domai of the parameters ad the required precisio. For example, if the domai of the parameter x is [-2,5] ad the precisio requiremet is six places after the decimal poit, the the domai [-2,5] should be divided ito 7,000,000 equal size rages. That implies the legth of the strig requires to be 23, for the reaso that =2 22 < <2 23 =
35 Geetic Algorithm The iitial process is quite simple. We create a populatio of idividuals, where idividual i a populatio is a biary strig with a fixed-legth, ad every bit of the biary strig is iitialized radomly. I each geeratio for which the GA is ru, each idividual i the populatio is evaluated agaist the ukow eviromet. The fitess values are associated with the values of objective fuctio. 35
36 Geetic Algorithm To perform geetic operators, oe must select idividuals i the populatio to be operated o.the selectio strategy is chiefly based o the fitess level of the idividuals actually preseted i the populatio. There are may differet selectio strategies based o fitess, such as Roulette Wheel Selectio, Elitism Strategy ad Touramet Selectio. 36
37 Geetic Algorithm Roulette Wheel Selectio: Step1. evaluate the fitess values for every idividual, s k, k = 1,2,, N. Step2. evaluate the total fitess values of all the idividuals, T. Step3. calculate ratio of fitess of each idividual. x x x x x x x x x x 1 x 2 x 2 x 3 x 4 x 5 x 6 x 7 x 7 37
38 Geetic Algorithm The crossover operator starts with two selected idividuals ad the the crossover poit is selected radomly accordig to a give probability of crossover. The secod geetic operator, mutatio, itroduces radom chages i structures i the populatio, ad it may occasioally have beeficial results: escapig from a local optimum. I biary form GA, mutatio is just to egate every bit of the strigs, i.e., chages a 1 to 0 ad vice versa, with give probability of mutatio. 38
39 Geetic Algorithm Oe-Poit Crossover Two-Poit Crossover
40 Geetic Algorithm Biary form Real-umber form
41 Global Optimizatio Toolbox [x,fval,exitflag,output,populatio,scores] = ga(@fitessfc,vars,a,b,aeq,beq,lb,ub,olco,optios) It fids a local miimum x to fitessfc, subject to the liear iequalities A x b as well as the liear equalities Aeq x = beq. fitessfc accepts iput x ad returs a scalar fuctio value evaluated at x. If the problem has m liear iequality costraits ad variables, the A is a matrix of size m-by- variables. b is a vector of legth m. 41
42 Global Optimizatio Toolbox UB ad LB defie a set of lower ad upper bouds o the desig variables, x, so that a solutio is foud i the rage LB x UB. Use empty matrices for LB ad UB if o bouds exist. optios = gaoptimset('param1',value1,'param2',value2,...) creates a structure called optios ad sets the value of param1 to value1, param2 to value2, ad so o 42
43 Global Optimizatio Toolbox fitessfc vars Aieq Bieq Aeq Beq lb ub olco optios Fitess fuctio Number of desig variables A matrix for liear iequality costraits b vector for liear iequality costraits A matrix for liear equality costraits b vector for liear equality costraits Lower boud o x Upper boud o x Noliear costrait fuctio Optios structure created usig gaoptimset or the Optimizatio Tool 43
44 Global Optimizatio Toolbox Optio Descriptio Values CreatioFc Hadle to the fuctio that creates the oliearfeasible CrossoverFractio The fractio of the populatio at the ext geeratio, ot icludig Positive scalar {0.8} elite childre, that is created by the crossover fuctio MutatioFc Hadle to the fuctio that produces EliteCout Positive iteger specifyig how may idividuals i the curret Positive iteger {2} geeratio are guarateed to survive to the ext geeratio. Not used i gamultiobj. Geeratios Positive iteger specifyig the maximum umber of iteratios Positive iteger {100} before the algorithm halts IitialPealty Iitial value of pealty parameter Positive scalar {10} IitialPopulatio Iitial populatio used to seed the geetic algorithm; ca be partial Matrix {[]} PopulatioSize Size of the populatio Positive iteger {20} PopulatioType Strig describig the data type of the populatio 'bitstrig' 'custom' {'doublevec tor'} to 'bitstrig' or 'custom'. StallGeLimit Positive iteger. The algorithm stops if there is o improvemet i Positive iteger {50} the objective fuctio for StallGeLimit cosecutive geeratios. TolCo Positive scalar. TolCo is used to determie the feasibility with Positive scalar {1e-6} respect to oliear costraits. TolFu Positive scalar. The algorithm rus util the cumulative chage i the fitess fuctio value over StallGeLimit is less tha TolFu. Positive scalar {1e-6}
45 Global Optimizatio Toolbox (Example 1) 45
46 Global Optimizatio Toolbox (Example 1) fuctio [ f ] = objectfuc_coti(x) f = -( x(1)*si(4*pi*x(1)) + x(2)*si(20*pi*x(2))); Lb = [-3 ; 4.1]; Ub = [ 12.1 ; 5.8]; optios = gaoptimset(@ga); optios = gaoptimset(optios,... 'MutatioFc',{@mutatioadaptfeasible,0.001},... 'CrossoverFractio',0.8,... 'PopIitRage', [-3 4.1; ],... 'PopulatioSize',100,... 'Geeratio',1000,... [x fitval] = ga(@objectfuc_coti,2,[],[],[],[],lb,ub,[],optios); 46
47 Global Optimizatio Toolbox (Example 2) Example y = [ ] 47
48 Global Optimizatio Toolbox (Example 2) fuctio [ f ] = objectuc_dis(x) y = [ ] for ii = 1 :10 Sx(ii) = abs(x(ii)-y(ii)); ed f = sum(sx); optios = gaoptimset(@ga); optios = gaoptimset(optios,... 'PopulatioSize',50,... 'Geeratio',3,... 'PopulatioType','bitstrig'); [x,fval,exitflag,output,populatio,scores] = ga(@objectuc_dis,10,[],[],[],[],[],[],[],optios); 48
49 Assigmet Wireless Iformatio Trasmissio System Lab. Istitute of Commuicatios Egieerig Natioal Su Yat-se Uiversity 2012/07/23 顏愷緯
50 DSI Method Sequece 50
51 DSI Method Simulatio coditio BPSK Total subcarriers = 64 Data subcarriers = 56 DSI subcarriers = 8 Oversample = 4 51
52 DSI Method GA ad CE Method Populatio size =10 Maximum iteratio =6 Elitist=1 Mutatio probability = 0.8 Crossover = 0.3 Fitess fuctio: 2 max x 0 1 PAPR= LN 2 Ex 52
53 CCDF DSI Method 10 0 DSI simlatio with GA ad CE method Origial GA CE PAPR 0 53
54 DSI Method Referece: H.-G. Ryu, J.-E. Lee, J.-S. Park, Dummy Sequece Isertio (DSI) for PAPR Reductio i the OFDM Commuicatio System, IEEE Tras. o Cosumer Electroics, Vol. 50, o 1, Feb 2004, pp
Expectation-Maximization Algorithm.
Expectatio-Maximizatio Algorithm. Petr Pošík Czech Techical Uiversity i Prague Faculty of Electrical Egieerig Dept. of Cyberetics MLE 2 Likelihood.........................................................................................................
More informationThe Expectation-Maximization (EM) Algorithm
The Expectatio-Maximizatio (EM) Algorithm Readig Assigmets T. Mitchell, Machie Learig, McGraw-Hill, 997 (sectio 6.2, hard copy). S. Gog et al. Dyamic Visio: From Images to Face Recogitio, Imperial College
More informationMarkov Decision Processes
Markov Decisio Processes Defiitios; Statioary policies; Value improvemet algorithm, Policy improvemet algorithm, ad liear programmig for discouted cost ad average cost criteria. Markov Decisio Processes
More informationInformation Theory and Coding
Sol. Iformatio Theory ad Codig. The capacity of a bad-limited additive white Gaussia (AWGN) chael is give by C = Wlog 2 ( + σ 2 W ) bits per secod(bps), where W is the chael badwidth, is the average power
More informationResampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.
Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator
More informationLinear Programming and the Simplex Method
Liear Programmig ad the Simplex ethod Abstract This article is a itroductio to Liear Programmig ad usig Simplex method for solvig LP problems i primal form. What is Liear Programmig? Liear Programmig is
More informationClustering. CM226: Machine Learning for Bioinformatics. Fall Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar.
Clusterig CM226: Machie Learig for Bioiformatics. Fall 216 Sriram Sakararama Ackowledgmets: Fei Sha, Ameet Talwalkar Clusterig 1 / 42 Admiistratio HW 1 due o Moday. Email/post o CCLE if you have questios.
More informationCSE 527, Additional notes on MLE & EM
CSE 57 Lecture Notes: MLE & EM CSE 57, Additioal otes o MLE & EM Based o earlier otes by C. Grat & M. Narasimha Itroductio Last lecture we bega a examiatio of model based clusterig. This lecture will be
More informationOutline. CSCI-567: Machine Learning (Spring 2019) Outline. Prof. Victor Adamchik. Mar. 26, 2019
Outlie CSCI-567: Machie Learig Sprig 209 Gaussia mixture models Prof. Victor Adamchik 2 Desity estimatio U of Souther Califoria Mar. 26, 209 3 Naive Bayes Revisited March 26, 209 / 57 March 26, 209 2 /
More informationInformation Theory Tutorial Communication over Channels with memory. Chi Zhang Department of Electrical Engineering University of Notre Dame
Iformatio Theory Tutorial Commuicatio over Chaels with memory Chi Zhag Departmet of Electrical Egieerig Uiversity of Notre Dame Abstract A geeral capacity formula C = sup I(; Y ), which is correct for
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationElement sampling: Part 2
Chapter 4 Elemet samplig: Part 2 4.1 Itroductio We ow cosider uequal probability samplig desigs which is very popular i practice. I the uequal probability samplig, we ca improve the efficiecy of the resultig
More informationECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015
ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],
More informationw (1) ˆx w (1) x (1) /ρ and w (2) ˆx w (2) x (2) /ρ.
2 5. Weighted umber of late jobs 5.1. Release dates ad due dates: maximimizig the weight of o-time jobs Oce we add release dates, miimizig the umber of late jobs becomes a sigificatly harder problem. For
More informationOFDM Precoder for Minimizing BER Upper Bound of MLD under Imperfect CSI
MIMO-OFDM OFDM Precoder for Miimizig BER Upper Boud of MLD uder Imperfect CSI MCRG Joit Semiar Jue the th 008 Previously preseted at ICC 008 Beijig o May the st 008 Boosar Pitakdumrogkija Kazuhiko Fukawa
More informationAlgorithms for Clustering
CR2: Statistical Learig & Applicatios Algorithms for Clusterig Lecturer: J. Salmo Scribe: A. Alcolei Settig: give a data set X R p where is the umber of observatio ad p is the umber of features, we wat
More informationCEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering
CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationSequences, Mathematical Induction, and Recursion. CSE 2353 Discrete Computational Structures Spring 2018
CSE 353 Discrete Computatioal Structures Sprig 08 Sequeces, Mathematical Iductio, ad Recursio (Chapter 5, Epp) Note: some course slides adopted from publisher-provided material Overview May mathematical
More informationREGRESSION (Physics 1210 Notes, Partial Modified Appendix A)
REGRESSION (Physics 0 Notes, Partial Modified Appedix A) HOW TO PERFORM A LINEAR REGRESSION Cosider the followig data poits ad their graph (Table I ad Figure ): X Y 0 3 5 3 7 4 9 5 Table : Example Data
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationLecture 27. Capacity of additive Gaussian noise channel and the sphere packing bound
Lecture 7 Ageda for the lecture Gaussia chael with average power costraits Capacity of additive Gaussia oise chael ad the sphere packig boud 7. Additive Gaussia oise chael Up to this poit, we have bee
More informationProblem Set 4 Due Oct, 12
EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios
More informationHOMEWORK 2 SOLUTIONS
HOMEWORK SOLUTIONS CSE 55 RANDOMIZED AND APPROXIMATION ALGORITHMS 1. Questio 1. a) The larger the value of k is, the smaller the expected umber of days util we get all the coupos we eed. I fact if = k
More informationOPTIMAL ALGORITHMS -- SUPPLEMENTAL NOTES
OPTIMAL ALGORITHMS -- SUPPLEMENTAL NOTES Peter M. Maurer Why Hashig is θ(). As i biary search, hashig assumes that keys are stored i a array which is idexed by a iteger. However, hashig attempts to bypass
More informationMATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4
MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.
More informationRecurrence Relations
Recurrece Relatios Aalysis of recursive algorithms, such as: it factorial (it ) { if (==0) retur ; else retur ( * factorial(-)); } Let t be the umber of multiplicatios eeded to calculate factorial(). The
More informationEcon 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara
Poit Estimator Eco 325 Notes o Poit Estimator ad Cofidece Iterval 1 By Hiro Kasahara Parameter, Estimator, ad Estimate The ormal probability desity fuctio is fully characterized by two costats: populatio
More information1.010 Uncertainty in Engineering Fall 2008
MIT OpeCourseWare http://ocw.mit.edu.00 Ucertaity i Egieerig Fall 2008 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu.terms. .00 - Brief Notes # 9 Poit ad Iterval
More informationStatistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample.
Statistical Iferece (Chapter 10) Statistical iferece = lear about a populatio based o the iformatio provided by a sample. Populatio: The set of all values of a radom variable X of iterest. Characterized
More informationInteger Linear Programming
Iteger Liear Programmig Itroductio Iteger L P problem (P) Mi = s. t. a = b i =,, m = i i 0, iteger =,, c Eemple Mi z = 5 s. t. + 0 0, 0, iteger F(P) = feasible domai of P Itroductio Iteger L P problem
More informationStudy on Coal Consumption Curve Fitting of the Thermal Power Based on Genetic Algorithm
Joural of ad Eergy Egieerig, 05, 3, 43-437 Published Olie April 05 i SciRes. http://www.scirp.org/joural/jpee http://dx.doi.org/0.436/jpee.05.34058 Study o Coal Cosumptio Curve Fittig of the Thermal Based
More informationIP Reference guide for integer programming formulations.
IP Referece guide for iteger programmig formulatios. by James B. Orli for 15.053 ad 15.058 This documet is iteded as a compact (or relatively compact) guide to the formulatio of iteger programs. For more
More informationOn comparison of different approaches to the stability radius calculation. Olga Karelkina
O compariso of differet approaches to the stability radius calculatio Olga Karelkia Uiversity of Turku 2011 Outlie Prelimiaries Problem statemet Exact method for calculatio stability radius proposed by
More informationChapter 11 Output Analysis for a Single Model. Banks, Carson, Nelson & Nicol Discrete-Event System Simulation
Chapter Output Aalysis for a Sigle Model Baks, Carso, Nelso & Nicol Discrete-Evet System Simulatio Error Estimatio If {,, } are ot statistically idepedet, the S / is a biased estimator of the true variace.
More information( ) = p and P( i = b) = q.
MATH 540 Radom Walks Part 1 A radom walk X is special stochastic process that measures the height (or value) of a particle that radomly moves upward or dowward certai fixed amouts o each uit icremet of
More informationThe Maximum-Likelihood Decoding Performance of Error-Correcting Codes
The Maximum-Lielihood Decodig Performace of Error-Correctig Codes Hery D. Pfister ECE Departmet Texas A&M Uiversity August 27th, 2007 (rev. 0) November 2st, 203 (rev. ) Performace of Codes. Notatio X,
More informationTime-Domain Representations of LTI Systems
2.1 Itroductio Objectives: 1. Impulse resposes of LTI systems 2. Liear costat-coefficiets differetial or differece equatios of LTI systems 3. Bloc diagram represetatios of LTI systems 4. State-variable
More informationTEACHER CERTIFICATION STUDY GUIDE
COMPETENCY 1. ALGEBRA SKILL 1.1 1.1a. ALGEBRAIC STRUCTURES Kow why the real ad complex umbers are each a field, ad that particular rigs are ot fields (e.g., itegers, polyomial rigs, matrix rigs) Algebra
More informationMath 312 Lecture Notes One Dimensional Maps
Math 312 Lecture Notes Oe Dimesioal Maps Warre Weckesser Departmet of Mathematics Colgate Uiversity 21-23 February 25 A Example We begi with the simplest model of populatio growth. Suppose, for example,
More informationLinear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d
Liear regressio Daiel Hsu (COMS 477) Maximum likelihood estimatio Oe of the simplest liear regressio models is the followig: (X, Y ),..., (X, Y ), (X, Y ) are iid radom pairs takig values i R d R, ad Y
More informationSECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES
SECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES Read Sectio 1.5 (pages 5 9) Overview I Sectio 1.5 we lear to work with summatio otatio ad formulas. We will also itroduce a brief overview of sequeces,
More informationFirst Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise
First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I-1 Part I - 203A A radom variable X is distributed with the margial desity: >
More informationScheduling under Uncertainty using MILP Sensitivity Analysis
Schedulig uder Ucertaity usig MILP Sesitivity Aalysis M. Ierapetritou ad Zheya Jia Departmet of Chemical & Biochemical Egieerig Rutgers, the State Uiversity of New Jersey Piscataway, NJ Abstract The aim
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationCS284A: Representations and Algorithms in Molecular Biology
CS284A: Represetatios ad Algorithms i Molecular Biology Scribe Notes o Lectures 3 & 4: Motif Discovery via Eumeratio & Motif Represetatio Usig Positio Weight Matrix Joshua Gervi Based o presetatios by
More informationSequences. Notation. Convergence of a Sequence
Sequeces A sequece is essetially just a list. Defiitio (Sequece of Real Numbers). A sequece of real umbers is a fuctio Z (, ) R for some real umber. Do t let the descriptio of the domai cofuse you; it
More informationLecture 2: Monte Carlo Simulation
STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?
More informationNUMERICAL METHODS FOR SOLVING EQUATIONS
Mathematics Revisio Guides Numerical Methods for Solvig Equatios Page 1 of 11 M.K. HOME TUITION Mathematics Revisio Guides Level: GCSE Higher Tier NUMERICAL METHODS FOR SOLVING EQUATIONS Versio:. Date:
More informationCS / MCS 401 Homework 3 grader solutions
CS / MCS 401 Homework 3 grader solutios assigmet due July 6, 016 writte by Jāis Lazovskis maximum poits: 33 Some questios from CLRS. Questios marked with a asterisk were ot graded. 1 Use the defiitio of
More informationDETERMINATION OF MECHANICAL PROPERTIES OF A NON- UNIFORM BEAM USING THE MEASUREMENT OF THE EXCITED LONGITUDINAL ELASTIC VIBRATIONS.
ICSV4 Cairs Australia 9- July 7 DTRMINATION OF MCHANICAL PROPRTIS OF A NON- UNIFORM BAM USING TH MASURMNT OF TH XCITD LONGITUDINAL LASTIC VIBRATIONS Pavel Aokhi ad Vladimir Gordo Departmet of the mathematics
More informationOptimization Methods MIT 2.098/6.255/ Final exam
Optimizatio Methods MIT 2.098/6.255/15.093 Fial exam Date Give: December 19th, 2006 P1. [30 pts] Classify the followig statemets as true or false. All aswers must be well-justified, either through a short
More informationRandom Variables, Sampling and Estimation
Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig
More informationSTAT 350 Handout 19 Sampling Distribution, Central Limit Theorem (6.6)
STAT 350 Hadout 9 Samplig Distributio, Cetral Limit Theorem (6.6) A radom sample is a sequece of radom variables X, X 2,, X that are idepedet ad idetically distributed. o This property is ofte abbreviated
More informationStatistical Properties of OLS estimators
1 Statistical Properties of OLS estimators Liear Model: Y i = β 0 + β 1 X i + u i OLS estimators: β 0 = Y β 1X β 1 = Best Liear Ubiased Estimator (BLUE) Liear Estimator: β 0 ad β 1 are liear fuctio of
More informationNCSS Statistical Software. Tolerance Intervals
Chapter 585 Itroductio This procedure calculates oe-, ad two-, sided tolerace itervals based o either a distributio-free (oparametric) method or a method based o a ormality assumptio (parametric). A two-sided
More informationDirection: This test is worth 250 points. You are required to complete this test within 50 minutes.
Term Test October 3, 003 Name Math 56 Studet Number Directio: This test is worth 50 poits. You are required to complete this test withi 50 miutes. I order to receive full credit, aswer each problem completely
More informationG. R. Pasha Department of Statistics Bahauddin Zakariya University Multan, Pakistan
Deviatio of the Variaces of Classical Estimators ad Negative Iteger Momet Estimator from Miimum Variace Boud with Referece to Maxwell Distributio G. R. Pasha Departmet of Statistics Bahauddi Zakariya Uiversity
More informationAn Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More informationPosted-Price, Sealed-Bid Auctions
Posted-Price, Sealed-Bid Auctios Professors Greewald ad Oyakawa 207-02-08 We itroduce the posted-price, sealed-bid auctio. This auctio format itroduces the idea of approximatios. We describe how well this
More informationChapter 6 Principles of Data Reduction
Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a
More information1 Hash tables. 1.1 Implementation
Lecture 8 Hash Tables, Uiversal Hash Fuctios, Balls ad Bis Scribes: Luke Johsto, Moses Charikar, G. Valiat Date: Oct 18, 2017 Adapted From Virgiia Williams lecture otes 1 Hash tables A hash table is a
More informationCHAPTER 10 INFINITE SEQUENCES AND SERIES
CHAPTER 10 INFINITE SEQUENCES AND SERIES 10.1 Sequeces 10.2 Ifiite Series 10.3 The Itegral Tests 10.4 Compariso Tests 10.5 The Ratio ad Root Tests 10.6 Alteratig Series: Absolute ad Coditioal Covergece
More informationDepartment of Mathematics
Departmet of Mathematics Ma 3/103 KC Border Itroductio to Probability ad Statistics Witer 2017 Lecture 19: Estimatio II Relevat textbook passages: Larse Marx [1]: Sectios 5.2 5.7 19.1 The method of momets
More informationSequences A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece 1, 1, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet
More informationMachine Learning for Data Science (CS 4786)
Machie Learig for Data Sciece CS 4786) Lecture & 3: Pricipal Compoet Aalysis The text i black outlies high level ideas. The text i blue provides simple mathematical details to derive or get to the algorithm
More informationSubstantiation of the Water Filling Theorem Using Lagrange Multipliers
J.A. Crawford U048 Water Fillig o Multiple Gaussia Chaels.doc Substatiatio of the Water Fillig heorem Usig Lagrage Multipliers Shao s capacity theorem states that i the case of parallel statistically idepedet
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationAnalysis of Algorithms. Introduction. Contents
Itroductio The focus of this module is mathematical aspects of algorithms. Our mai focus is aalysis of algorithms, which meas evaluatig efficiecy of algorithms by aalytical ad mathematical methods. We
More informationECE 564/645 - Digital Communication Systems (Spring 2014) Final Exam Friday, May 2nd, 8:00-10:00am, Marston 220
ECE 564/645 - Digital Commuicatio Systems (Sprig 014) Fial Exam Friday, May d, 8:00-10:00am, Marsto 0 Overview The exam cosists of four (or five) problems for 100 (or 10) poits. The poits for each part
More informationComputational Intelligence Winter Term 2018/19
Computatioal Itelligece Witer Term 28/9 Prof. Dr. Güter Rudolph Lehrstuhl für Algorithm Egieerig (LS ) Fakultät für Iformatik TU Dortmud Pla for Today Lecture Evolutioary Algorithms (EA) Optimizatio Basics
More informationsubject to A 1 x + A 2 y b x j 0, j = 1,,n 1 y j = 0 or 1, j = 1,,n 2
Additioal Brach ad Boud Algorithms 0-1 Mixed-Iteger Liear Programmig The brach ad boud algorithm described i the previous sectios ca be used to solve virtually all optimizatio problems cotaiig iteger variables,
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationKurskod: TAMS11 Provkod: TENB 21 March 2015, 14:00-18:00. English Version (no Swedish Version)
Kurskod: TAMS Provkod: TENB 2 March 205, 4:00-8:00 Examier: Xiagfeg Yag (Tel: 070 2234765). Please aswer i ENGLISH if you ca. a. You are allowed to use: a calculator; formel -och tabellsamlig i matematisk
More informationRun-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE
Geeral e Image Coder Structure Motio Video (s 1,s 2,t) or (s 1,s 2 ) Natural Image Samplig A form of data compressio; usually lossless, but ca be lossy Redudacy Removal Lossless compressio: predictive
More informationModeling and Estimation of a Bivariate Pareto Distribution using the Principle of Maximum Entropy
Sri Laka Joural of Applied Statistics, Vol (5-3) Modelig ad Estimatio of a Bivariate Pareto Distributio usig the Priciple of Maximum Etropy Jagathath Krisha K.M. * Ecoomics Research Divisio, CSIR-Cetral
More informationOutput Analysis and Run-Length Control
IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%
More informationFACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures
FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals
More informationThe Sampling Distribution of the Maximum. Likelihood Estimators for the Parameters of. Beta-Binomial Distribution
Iteratioal Mathematical Forum, Vol. 8, 2013, o. 26, 1263-1277 HIKARI Ltd, www.m-hikari.com http://d.doi.org/10.12988/imf.2013.3475 The Samplig Distributio of the Maimum Likelihood Estimators for the Parameters
More informationProperties and Hypothesis Testing
Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.
More informationOptimization Methods: Linear Programming Applications Assignment Problem 1. Module 4 Lecture Notes 3. Assignment Problem
Optimizatio Methods: Liear Programmig Applicatios Assigmet Problem Itroductio Module 4 Lecture Notes 3 Assigmet Problem I the previous lecture, we discussed about oe of the bech mark problems called trasportatio
More informationThe Growth of Functions. Theoretical Supplement
The Growth of Fuctios Theoretical Supplemet The Triagle Iequality The triagle iequality is a algebraic tool that is ofte useful i maipulatig absolute values of fuctios. The triagle iequality says that
More informationA sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece,, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet as
More informationOn forward improvement iteration for stopping problems
O forward improvemet iteratio for stoppig problems Mathematical Istitute, Uiversity of Kiel, Ludewig-Mey-Str. 4, D-24098 Kiel, Germay irle@math.ui-iel.de Albrecht Irle Abstract. We cosider the optimal
More informationInduction: Solutions
Writig Proofs Misha Lavrov Iductio: Solutios Wester PA ARML Practice March 6, 206. Prove that a 2 2 chessboard with ay oe square removed ca always be covered by shaped tiles. Solutio : We iduct o. For
More informationSequential Monte Carlo Methods - A Review. Arnaud Doucet. Engineering Department, Cambridge University, UK
Sequetial Mote Carlo Methods - A Review Araud Doucet Egieerig Departmet, Cambridge Uiversity, UK http://www-sigproc.eg.cam.ac.uk/ ad2/araud doucet.html ad2@eg.cam.ac.uk Istitut Heri Poicaré - Paris - 2
More informationSimilarity Solutions to Unsteady Pseudoplastic. Flow Near a Moving Wall
Iteratioal Mathematical Forum, Vol. 9, 04, o. 3, 465-475 HIKARI Ltd, www.m-hikari.com http://dx.doi.org/0.988/imf.04.48 Similarity Solutios to Usteady Pseudoplastic Flow Near a Movig Wall W. Robi Egieerig
More information62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +
62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of
More informationCS321. Numerical Analysis and Computing
CS Numerical Aalysis ad Computig Lecture Locatig Roots o Equatios Proessor Ju Zhag Departmet o Computer Sciece Uiversity o Ketucky Leigto KY 456-6 September 8 5 What is the Root May physical system ca
More informationChapter 12 EM algorithms The Expectation-Maximization (EM) algorithm is a maximum likelihood method for models that have hidden variables eg. Gaussian
Chapter 2 EM algorithms The Expectatio-Maximizatio (EM) algorithm is a maximum likelihood method for models that have hidde variables eg. Gaussia Mixture Models (GMMs), Liear Dyamic Systems (LDSs) ad Hidde
More informationOptimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
More informationAPPENDIX: STUDY CASES A SURVEY OF NONPARAMETRIC TESTS FOR THE STATISTICAL ANALYSIS OF EVOLUTIONARY COMPUTATION EXPERIMENTS
A survey of oparametric tests for the statistical aalysis of evolutioary computatio experimets. Appedix 1 APPENDIX: STUDY CASES A SURVEY OF NONPARAMETRIC TESTS FOR THE STATISTICAL ANALYSIS OF EVOLUTIONARY
More informationExpectation and Variance of a random variable
Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio
More informationInequalities for Entropies of Sets of Subsets of Random Variables
Iequalities for Etropies of Sets of Subsets of Radom Variables Chao Tia AT&T Labs-Research Florham Par, NJ 0792, USA. tia@research.att.com Abstract Ha s iequality o the etropy rates of subsets of radom
More informationClustering: Mixture Models
Clusterig: Mixture Models Machie Learig 10-601B Seyoug Kim May of these slides are derived from Tom Mitchell, Ziv- Bar Joseph, ad Eric Xig. Thaks! Problem with K- meas Hard Assigmet of Samples ito Three
More informationDefinitions and Theorems. where x are the decision variables. c, b, and a are constant coefficients.
Defiitios ad Theorems Remember the scalar form of the liear programmig problem, Miimize, Subject to, f(x) = c i x i a 1i x i = b 1 a mi x i = b m x i 0 i = 1,2,, where x are the decisio variables. c, b,
More informationChapter 7 z-transform
Chapter 7 -Trasform Itroductio Trasform Uilateral Trasform Properties Uilateral Trasform Iversio of Uilateral Trasform Determiig the Frequecy Respose from Poles ad Zeros Itroductio Role i Discrete-Time
More informationWe are mainly going to be concerned with power series in x, such as. (x)} converges - that is, lims N n
Review of Power Series, Power Series Solutios A power series i x - a is a ifiite series of the form c (x a) =c +c (x a)+(x a) +... We also call this a power series cetered at a. Ex. (x+) is cetered at
More informationMath 155 (Lecture 3)
Math 55 (Lecture 3) September 8, I this lecture, we ll cosider the aswer to oe of the most basic coutig problems i combiatorics Questio How may ways are there to choose a -elemet subset of the set {,,,
More informationRandomized Algorithms I, Spring 2018, Department of Computer Science, University of Helsinki Homework 1: Solutions (Discussed January 25, 2018)
Radomized Algorithms I, Sprig 08, Departmet of Computer Sciece, Uiversity of Helsiki Homework : Solutios Discussed Jauary 5, 08). Exercise.: Cosider the followig balls-ad-bi game. We start with oe black
More information