Basics of Information Theory: Markku Juntti. Basic concepts and tools 1 Introduction 2 Entropy, relative entropy and mutual information
|
|
|
- David Skinner
- 5 years ago
- Views:
Transcription
1 : Markku Jutt Overvew Te basc cocepts o ormato teory lke etropy mutual ormato ad EP are eeralzed or cotuous-valued radom varables by troduc deretal etropy ource Te materal s maly based o Capter 9 o te course book [] Telecomm Laboratory Course Overvew Basc cocepts ad tools Itroducto Etropy relatve etropy ad mutual ormato 3 symptotc equpartto t property 4 Etropy rates o a stocastc process ource cod or data compresso 5 Data compresso Cael capacty 8 Cael capacty 9 Deretal etropy 0 Te Gaussa cael Oter applcatos Mamum etropy ad spectral estmato 3 Rate dstorto teory 4 Network ormato teory Telecomm Laboratory Outle o te Lecture Revew o te last lecture Itroducto Detos Te EP or cotuous RV s Relato o deretal etropy ad dscrete etropy Jot ad codtoal deretal etropy Relatve etropy ad mutual ormato Propertes ummary Revew o Last Lecture Te udametal questos bed ormato teory: Wat s te ultmate data compresso rate? swer: etropy Wat s te ultmate data trasmsso rate? swer: cael capacty commucates wt B: pyscal acts o duce desred pyscal state B ad B sould aree o wat was set Te mamum umber o dstusable sals cael capacty = data rate lmt or relable commucato B Telecomm Laboratory 3 Telecomm Laboratory 4
2 Te Cael Cod Teorem ll rates below cael capacty C are acevable For all rate R < C: a sequece o ( R ) codes so tat 0 as (Weak) coverse: ay sequece o acevable ( R ) codes: R C tro coverse: or R < C P () e 0 ad R > C P () e epoetally Capacty wt eedback: supremum o acevable rates wt eedback codes C FB : C FB C ma I ; Y p Feedback does ot crease te capacty ource-cael Cod Teorem a source-cael code wt P () e 0 H(V)<C were te probablty o error s ˆ Pe PrV V pv p y v y v y v were Vˆ s te estmate o V s true (o - zero) () s a dcator ucto: 0 s alse (zero) Coverse: I H(V)>C probablty o error s bouded away rom zero ad relable trasmsso s ot possble Telecomm Laboratory 5 Telecomm Laboratory 6 ummary Itroducto Iormato cael capacty = operatoal cael capacty: C ma I; Y p ll rates below te capacty (R<C) are acevable: 0 as No rate above te capacty (R>C) s acevable Feedback does ot crease te capacty Feedback ca elp acev te capacty source wt etropy rate below te capacty ca be trasmtted relably source wt etropy rate above te capacty caot be trasmtted relably Dscrete-valued radom varables ave bee cosdered so ar Eteso or cotuous RV s eeded deretal etropy tratorward eteso o most oter cocepts partcular te asymptotc equpartto property ome mportat dereces tou Deretal etropy ca be eatve Dscrete etropy s always postve Mamum deretal etropy uder a covarace costrat by ormal (Gaussa) dstrbuto Dscrete etropy wt o costrat mamzed by uorm dstrbuto Telecomm Laboratory 7 Telecomm Laboratory 8
3 Detos Let be a RV wt cumulatve dstrbuto ucto (CDF) F() = Pr( ) I F() s cotuous te radom varable s sad to be cotuous Let df ' F I s ormalzed so tat d d () s called probablty desty ucto (PDF) or Te set o suc values o were () > 0 s called te support set (or smply support) o (or ucto ()) Deretal Etropy Deretal etropy ( te teral ests): d were s te support set o te radom varable () = E[()] Depeds o te PDF oly as does te etropy te dscrete case eeralzato o te etropy H p p marked also as () = () Telecomm Laboratory 9 Telecomm Laboratory 0 Eample #: Uorm Dstrbuto 0 a a /a 0 oterwse a a d a a a 0 d Note #: I a = (a) = 0 () = 0 Note #: I a < (a) < 0 Deretal etropy ca be eatve derece compared to te etropy However () = a = a s te volume o te support set wc always o-eatve Eample #: Normal Dstrbuto () Let te RV ollow ormal (e Gaussa) dstrbuto: ~ () () were deotes te PDF o real Gaussa dstrbuto wt mea zero ad varace : e Due to te epoet ucto t s smplest to calculate te deretal etropy ats Te deretal etropy bts ca be oud va cae o bases Telecomm Laboratory Telecomm Laboratory
4 Eample #: Normal Dstrbuto () l d l d E E l l l e l e ats e e l e bts e bts e e bts Te EP or Cotuous RV s Remd te asymptotc equpartto property or dscrete RV s: p H probablty Most o te probablty s cotaed te typcal set H : p : so tat p H p Pr or lare eou 3 H 3 Te same ca be sow or cotuous RV s Telecomm Laboratory 3 Telecomm Laboratory 4 EP ad Typcal et For a sequece o IID RV s: E probablty Proo: Follows drectly rom te weak law o lare umbers Typcal set: were Volume o set: Vol d d d R : Telecomm Laboratory 5 Propertes o te Typcal et Pr( () ) > or sucetly lare Vol( () ) (()+) or all 3 Vol( () ( ) (() ) ) or sucetly lare Proo: ee te tetbook [ ect 9] Te typcal set s te set wt te smallest volume satsy probablty bl ( ) to rst order o epoet Proo: mlar to te dscrete case Te smallest set tat cotas most o te probablty as appromately volume o Te correspod sde let s () = (sde let o m set wt most prob) Telecomm Laboratory 6
5 Relato o Deretal Etropy ad Dscrete Etropy Cosder a RV wt PDF () Let us dvde te rae o to bs o wdt ssume () s cotuous wt te bs By te mea value teorem: () : d Quatze: = <(+) Pr p d Telecomm Laboratory 7 Dervato Te etropy o te quatzed RV : H p p d d d I ()() ) s Rema terable d 0 Telecomm Laboratory 8 Teorem: Te Relato betwee te Deretal ad Dscrete Etropes I te PDF () s Rema terable te H 0 Te etropy o a -bt quatzato o a cotuous RV s appromately () + Eamples: I ~ U[0] ad = te =0adH( )= bts suce to descrbe to bt accuracy I ~ U[0/8] ad = te = 3 ad H( ) = Te rst 3 bts to te rt o decmal pot must be 0 3 bts suce to descrbe to bt accuracy I eeral () + s te averae umber o bts Te deretal etropy o a dscrete RV ca be cosdered to be result te volume o support set be zero Telecomm Laboratory 9 Jot ad Codtoal Deretal Etropy Jot deretal etropy: d d d Codtoal deretal etropy: Y y y d dy ( Y ) ( ) ( Y ) Telecomm Laboratory 0
6 Jot Deretal Etropy o a Multvarate Gaussa Radom Vector Te Gaussa PDF: ep T μ μ were = det() deotes te determat o te covarace matr Te deretal etropy o real-valued Gaussa RV s wt mea vector ad covarace matr : e Proo: ee te tetbook [ ect 94] Telecomm Laboratory Relatve Etropy ad Mutual Iormato Relatve etropy or ullback-lebler dstace: D Mutual ormato: y I ; Y y d dy y D y y Y Y Y Relato to dscrete mutual ormato: I ; Y H H Y Y I ;Y Telecomm Laboratory Propertes No-eatvty: D 0 D 0 almost everywere Proo by Jese s equalty: D 0 Furter propertes: I ; Y 0 I ; Y 0 y y Y Y y y Telecomm Laboratory 3 Propertes () Ca rule or deretal etropy: Idepedece boud: Hadamard equalty: For ~N(0): Traslatos ad rotatos: c a a Telecomm Laboratory 4
7 Te Gaussa PDF Mamzes Deretal Etropy Multvarate ormal (Gaussa) dstrbuto mamzes te deretal etropy over all dstrbutos wt te same covarace Let te real -dmesoal radom vector ave zero mea ad covarace = E(( T ) or j = E( ) j j Te e wt equalty ad oly ~N(0) Proo Let () be ay PDF satsy te covarace costrat jd j j Let be te PDF o N(0) or Now T ep ep 0 D * Telecomm Laboratory 5 Telecomm Laboratory 6 tep * te Proo T d e d T e T e d e e e d d ummary Eteso o ormato teoretc cocepts to cotuous RV s Deretal etropy: d symptotc equpartto property ad te typcal set: E probablty : Telecomm Laboratory 7 Telecomm Laboratory 8
8 ummary () Relatve etropy: D Mutual ormato: y I ; Y y d dy y D y y Y Y Y Multvarate ormal (Gaussa) dstrbuto mamzes te deretal etropy over all dstrbutos wt te same covarace: e Telecomm Laboratory 9
3. Basic Concepts: Consequences and Properties
: 3. Basc Cocepts: Cosequeces ad Propertes Markku Jutt Overvew More advaced cosequeces ad propertes of the basc cocepts troduced the prevous lecture are derved. Source The materal s maly based o Sectos.6.8
Differential Entropy 吳家麟教授
Deretl Etropy 吳家麟教授 Deto Let be rdom vrble wt cumultve dstrbuto ucto I F s cotuous te r.v. s sd to be cotuous. Let = F we te dervtve s deed. I te s clled te pd or. Te set were > 0 s clled te support set
. The set of these sums. be a partition of [ ab, ]. Consider the sum f( x) f( x 1)
![. The set of these sums. be a partition of [ ab, ]. Consider the sum f( x) f( x 1)](/thumbs/85/92239142.jpg ". The set of these sums. be a partition of [ ab, ]. Consider the sum f( x) f( x 1)")
Chapter 7 Fuctos o Bouded Varato. Subject: Real Aalyss Level: M.Sc. Source: Syed Gul Shah (Charma, Departmet o Mathematcs, US Sargodha Collected & Composed by: Atq ur Rehma (atq@mathcty.org, http://www.mathcty.org
Basics of Information Theory: Markku Juntti. Basic concepts and tools 1 Introduction 2 Entropy, relative entropy and mutual information
: Maru Jutt Overvew he propertes of adlmted Gaussa chaels are further studed, parallel Gaussa chaels ad Gaussa chaels wth feedac are solved. Source he materal s maly ased o Sectos.4.6 of the course oo
Lecture 3 Probability review (cont d)
STATS 00: Itroducto to Statstcal Iferece Autum 06 Lecture 3 Probablty revew (cot d) 3. Jot dstrbutos If radom varables X,..., X k are depedet, the ther dstrbuto may be specfed by specfyg the dvdual dstrbuto
Entropies & Information Theory
Etropes & Iformato Theory LECTURE II Nlajaa Datta Uversty of Cambrdge,U.K. See lecture otes o: http://www.q.damtp.cam.ac.uk/ode/223 quatum system States (of a physcal system): Hlbert space (fte-dmesoal)
Module 7. Lecture 7: Statistical parameter estimation
Lecture 7: Statstcal parameter estmato Parameter Estmato Methods of Parameter Estmato 1) Method of Matchg Pots ) Method of Momets 3) Mamum Lkelhood method Populato Parameter Sample Parameter Ubased estmato
Chapter 5 Properties of a Random Sample
Lecture 6 o BST 63: Statstcal Theory I Ku Zhag, /0/008 Revew for the prevous lecture Cocepts: t-dstrbuto, F-dstrbuto Theorems: Dstrbutos of sample mea ad sample varace, relatoshp betwee sample mea ad sample
ρ < 1 be five real numbers. The
Lecture o BST 63: Statstcal Theory I Ku Zhag, /0/006 Revew for the prevous lecture Deftos: covarace, correlato Examples: How to calculate covarace ad correlato Theorems: propertes of correlato ad covarace
5. Data Compression. Review of Last Lecture. Outline of the Lecture. Course Overview. Basics of Information Theory: Markku Juntti
: Markku Jutt Overvew The deas of lossless data copresso ad source codg are troduced ad copresso lts are derved. Source The ateral s aly based o Sectos 5. 5.5 of the course book []. Teleco. Laboratory
( ) = ( ) ( ) Chapter 13 Asymptotic Theory and Stochastic Regressors. Stochastic regressors model
Chapter 3 Asmptotc Theor ad Stochastc Regressors The ature of eplaator varable s assumed to be o-stochastc or fed repeated samples a regresso aalss Such a assumpto s approprate for those epermets whch
CHAPTER VI Statistical Analysis of Experimental Data
Chapter VI Statstcal Aalyss of Expermetal Data CHAPTER VI Statstcal Aalyss of Expermetal Data Measuremets do ot lead to a uque value. Ths s a result of the multtude of errors (maly radom errors) that ca
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS Postpoed exam: ECON430 Statstcs Date of exam: Jauary 0, 0 Tme for exam: 09:00 a.m. :00 oo The problem set covers 5 pages Resources allowed: All wrtte ad prted
Continuous Random Variables: Conditioning, Expectation and Independence
Cotuous Radom Varables: Codtog, xectato ad Ideedece Berl Che Deartmet o Comuter cece & Iormato geerg atoal Tawa ormal Uverst Reerece: - D.. Bertsekas, J.. Tstskls, Itroducto to robablt, ectos 3.4-3.5 Codtog
= 2. Statistic - function that doesn't depend on any of the known parameters; examples:
of Samplg Theory amples - uemploymet househol cosumpto survey Raom sample - set of rv's... ; 's have ot strbuto [ ] f f s vector of parameters e.g. Statstc - fucto that oes't epe o ay of the ow parameters;
Chapter 4 Multiple Random Variables
Revew for the prevous lecture: Theorems ad Examples: How to obta the pmf (pdf) of U = g (, Y) ad V = g (, Y) Chapter 4 Multple Radom Varables Chapter 44 Herarchcal Models ad Mxture Dstrbutos Examples:
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS
UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS Exam: ECON430 Statstcs Date of exam: Frday, December 8, 07 Grades are gve: Jauary 4, 08 Tme for exam: 0900 am 00 oo The problem set covers 5 pages Resources allowed:
Functions of Random Variables
Fuctos of Radom Varables Chapter Fve Fuctos of Radom Varables 5. Itroducto A geeral egeerg aalyss model s show Fg. 5.. The model output (respose) cotas the performaces of a system or product, such as weght,
X ε ) = 0, or equivalently, lim
Revew for the prevous lecture Cocepts: order statstcs Theorems: Dstrbutos of order statstcs Examples: How to get the dstrbuto of order statstcs Chapter 5 Propertes of a Radom Sample Secto 55 Covergece
Summary of the lecture in Biostatistics
Summary of the lecture Bostatstcs Probablty Desty Fucto For a cotuos radom varable, a probablty desty fucto s a fucto such that: 0 dx a b) b a dx A probablty desty fucto provdes a smple descrpto of the
Introduction to Numerical Differentiation and Interpolation March 10, !=1 1!=1 2!=2 3!=6 4!=24 5!= 120
Itroducto to Numercal Deretato ad Iterpolato Marc, Itroducto to Numercal Deretato ad Iterpolato Larr Caretto Mecacal Egeerg 9 Numercal Aalss o Egeerg stems Marc, Itroducto Iterpolato s te use o a dscrete
Special Instructions / Useful Data
JAM 6 Set of all real umbers P A..d. B, p Posso Specal Istructos / Useful Data x,, :,,, x x Probablty of a evet A Idepedetly ad detcally dstrbuted Bomal dstrbuto wth parameters ad p Posso dstrbuto wth
Discrete Mathematics and Probability Theory Fall 2016 Seshia and Walrand DIS 10b
CS 70 Dscrete Mathematcs ad Probablty Theory Fall 206 Sesha ad Walrad DIS 0b. Wll I Get My Package? Seaky delvery guy of some compay s out delverg packages to customers. Not oly does he had a radom package
Unsupervised Learning and Other Neural Networks
CSE 53 Soft Computg NOT PART OF THE FINAL Usupervsed Learg ad Other Neural Networs Itroducto Mture Destes ad Idetfablty ML Estmates Applcato to Normal Mtures Other Neural Networs Itroducto Prevously, all
Simulation Output Analysis
Smulato Output Aalyss Summary Examples Parameter Estmato Sample Mea ad Varace Pot ad Iterval Estmato ermatg ad o-ermatg Smulato Mea Square Errors Example: Sgle Server Queueg System x(t) S 4 S 4 S 3 S 5
Chapter 14 Logistic Regression Models
Chapter 4 Logstc Regresso Models I the lear regresso model X β + ε, there are two types of varables explaatory varables X, X,, X k ad study varable y These varables ca be measured o a cotuous scale as
4 Round-Off and Truncation Errors
HK Km Slgtly moded 3//9, /8/6 Frstly wrtte at Marc 5 4 Roud-O ad Trucato Errors Errors Roud-o Errors Trucato Errors Total Numercal Errors Bluders, Model Errors, ad Data Ucertaty Recallg, dv dt Δv v t Δt
LECTURE 8: Topics in Chaos Ricker Equation. Period doubling bifurcation. Period doubling cascade. A Quadratic Equation Ricker Equation 1.0. x x 4 0.
LECTURE 8: Topcs Chaos Rcker Equato (t ) = (t ) ep( (t )) Perod doulg urcato Perod doulg cascade 9....... A Quadratc Equato Rcker Equato (t ) = (t ) ( (t ) ). (t ) = (t ) ep( (t )) 6. 9 9. The perod doulg
Lecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model
Lecture 7. Cofdece Itervals ad Hypothess Tests the Smple CLR Model I lecture 6 we troduced the Classcal Lear Regresso (CLR) model that s the radom expermet of whch the data Y,,, K, are the outcomes. The
Idea is to sample from a different distribution that picks points in important regions of the sample space. Want ( ) ( ) ( ) E f X = f x g x dx
Importace Samplg Used for a umber of purposes: Varace reducto Allows for dffcult dstrbutos to be sampled from. Sestvty aalyss Reusg samples to reduce computatoal burde. Idea s to sample from a dfferet
Outline. Point Pattern Analysis Part I. Revisit IRP/CSR
Pot Patter Aalyss Part I Outle Revst IRP/CSR, frst- ad secod order effects What s pot patter aalyss (PPA)? Desty-based pot patter measures Dstace-based pot patter measures Revst IRP/CSR Equal probablty:
Lecture Notes Types of economic variables
Lecture Notes 3 1. Types of ecoomc varables () Cotuous varable takes o a cotuum the sample space, such as all pots o a le or all real umbers Example: GDP, Polluto cocetrato, etc. () Dscrete varables fte
ENGI 4421 Joint Probability Distributions Page Joint Probability Distributions [Navidi sections 2.5 and 2.6; Devore sections
ENGI 441 Jot Probablty Dstrbutos Page 7-01 Jot Probablty Dstrbutos [Navd sectos.5 ad.6; Devore sectos 5.1-5.] The jot probablty mass fucto of two dscrete radom quattes, s, P ad p x y x y The margal probablty
Asymptotic Formulas Composite Numbers II
Iteratoal Matematcal Forum, Vol. 8, 203, o. 34, 65-662 HIKARI Ltd, www.m-kar.com ttp://d.do.org/0.2988/mf.203.3854 Asymptotc Formulas Composte Numbers II Rafael Jakmczuk Dvsó Matemátca, Uversdad Nacoal
Random Variate Generation ENM 307 SIMULATION. Anadolu Üniversitesi, Endüstri Mühendisliği Bölümü. Yrd. Doç. Dr. Gürkan ÖZTÜRK.
adom Varate Geerato ENM 307 SIMULATION Aadolu Üverstes, Edüstr Mühedslğ Bölümü Yrd. Doç. Dr. Gürka ÖZTÜK 0 adom Varate Geerato adom varate geerato s about procedures for samplg from a varety of wdely-used
ESE 523 Information Theory
ESE 53 Iformato Theory Joseph A. O Sullva Samuel C. Sachs Professor Electrcal ad Systems Egeerg Washgto Uversty Urbauer Hall 0E Gree Hall 34-935-473 jao@wustl.edu 09// J. A. O'Sullva, ESE 53, Lecture -6
Chapter 4 Multiple Random Variables
Revew o BST 63: Statstcal Theory I Ku Zhag, /0/008 Revew for Chapter 4-5 Notes: Although all deftos ad theorems troduced our lectures ad ths ote are mportat ad you should be famlar wth, but I put those
Lecture 3. Sampling, sampling distributions, and parameter estimation
Lecture 3 Samplg, samplg dstrbutos, ad parameter estmato Samplg Defto Populato s defed as the collecto of all the possble observatos of terest. The collecto of observatos we take from the populato s called
Chapter 3 Sampling For Proportions and Percentages
Chapter 3 Samplg For Proportos ad Percetages I may stuatos, the characterstc uder study o whch the observatos are collected are qualtatve ature For example, the resposes of customers may marketg surveys
Multivariate Transformation of Variables and Maximum Likelihood Estimation
Marquette Uversty Multvarate Trasformato of Varables ad Maxmum Lkelhood Estmato Dael B. Rowe, Ph.D. Assocate Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Copyrght 03 by Marquette Uversty
Some Applications of the Resampling Methods in Computational Physics
Iteratoal Joural of Mathematcs Treds ad Techoloy Volume 6 February 04 Some Applcatos of the Resampl Methods Computatoal Physcs Sotraq Marko #, Lorec Ekoom * # Physcs Departmet, Uversty of Korca, Albaa,
Lecture 4 Sep 9, 2015
CS 388R: Radomzed Algorthms Fall 205 Prof. Erc Prce Lecture 4 Sep 9, 205 Scrbe: Xagru Huag & Chad Voegele Overvew I prevous lectures, we troduced some basc probablty, the Cheroff boud, the coupo collector
First Law of Thermodynamics
Cocept o Iteral Eergy, U Iteral eergy s the sum o the ketc ad potetal eerges o the partcles that make up the system. Frst Law o Thermodyamcs Chapter Coservato o Eergy At molecular level, cotrbutors to
Generative classification models
CS 75 Mache Learg Lecture Geeratve classfcato models Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Data: D { d, d,.., d} d, Classfcato represets a dscrete class value Goal: lear f : X Y Bar classfcato
Lecture 02: Bounding tail distributions of a random variable
CSCI-B609: A Theorst s Toolkt, Fall 206 Aug 25 Lecture 02: Boudg tal dstrbutos of a radom varable Lecturer: Yua Zhou Scrbe: Yua Xe & Yua Zhou Let us cosder the ubased co flps aga. I.e. let the outcome
Chain Rules for Entropy
Cha Rules for Etroy The etroy of a collecto of radom varables s the sum of codtoal etroes. Theorem: Let be radom varables havg the mass robablty x x.x. The...... The roof s obtaed by reeatg the alcato
BASICS ON DISTRIBUTIONS
BASICS ON DISTRIBUTIONS Hstograms Cosder a epermet whch dfferet outcomes are possble (e. Dce tossg). The probablty of all the outcomes ca be represeted a hstogram Dstrbutos Probabltes are descrbed wth
CS 2750 Machine Learning Lecture 5. Density estimation. Density estimation
CS 750 Mache Learg Lecture 5 esty estmato Mlos Hausrecht mlos@tt.edu 539 Seott Square esty estmato esty estmato: s a usuervsed learg roblem Goal: Lear a model that rereset the relatos amog attrbutes the
Econometrics. 3) Statistical properties of the OLS estimator
30C0000 Ecoometrcs 3) Statstcal propertes of the OLS estmator Tmo Kuosmae Professor, Ph.D. http://omepre.et/dex.php/tmokuosmae Today s topcs Whch assumptos are eeded for OLS to work? Statstcal propertes
6. Nonparametric techniques
6. Noparametrc techques Motvato Problem: how to decde o a sutable model (e.g. whch type of Gaussa) Idea: just use the orgal data (lazy learg) 2 Idea 1: each data pot represets a pece of probablty P(x)
Econometric Methods. Review of Estimation
Ecoometrc Methods Revew of Estmato Estmatg the populato mea Radom samplg Pot ad terval estmators Lear estmators Ubased estmators Lear Ubased Estmators (LUEs) Effcecy (mmum varace) ad Best Lear Ubased Estmators
CODING & MODULATION Prof. Ing. Anton Čižmár, PhD.
CODING & MODULATION Prof. Ig. Ato Čžmár, PhD. also from Dgtal Commucatos 4th Ed., J. G. Proaks, McGraw-Hll It. Ed. 00 CONTENT. PROBABILITY. STOCHASTIC PROCESSES Probablty ad Stochastc Processes The theory
X X X E[ ] E X E X. is the ()m n where the ( i,)th. j element is the mean of the ( i,)th., then
![X X X E[ ] E X E X. is the ()m n where the ( i,)th. j element is the mean of the ( i,)th., then](/thumbs/83/87696552.jpg "X X X E[ ] E X E X. is the ()m n where the ( i,)th. j element is the mean of the ( i,)th., then")
Secto 5 Vectors of Radom Varables Whe workg wth several radom varables,,..., to arrage them vector form x, t s ofte coveet We ca the make use of matrx algebra to help us orgaze ad mapulate large umbers
Parameter, Statistic and Random Samples
Parameter, Statstc ad Radom Samples A parameter s a umber that descrbes the populato. It s a fxed umber, but practce we do ot kow ts value. A statstc s a fucto of the sample data,.e., t s a quatty whose
Mathematics HL and Further mathematics HL Formula booklet
Dploma Programme Mathematcs HL ad Further mathematcs HL Formula booklet For use durg the course ad the eamatos Frst eamatos 04 Mathematcal Iteratoal Baccalaureate studes SL: Formula Orgazato booklet 0
Point Estimation: definition of estimators
Pot Estmato: defto of estmators Pot estmator: ay fucto W (X,..., X ) of a data sample. The exercse of pot estmato s to use partcular fuctos of the data order to estmate certa ukow populato parameters.
Lecture Note to Rice Chapter 8
ECON 430 HG revsed Nov 06 Lecture Note to Rce Chapter 8 Radom matrces Let Y, =,,, m, =,,, be radom varables (r.v. s). The matrx Y Y Y Y Y Y Y Y Y Y = m m m s called a radom matrx ( wth a ot m-dmesoal dstrbuto,
CS 1675 Introduction to Machine Learning Lecture 12 Support vector machines
CS 675 Itroducto to Mache Learg Lecture Support vector maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Mdterm eam October 9, 7 I-class eam Closed book Stud materal: Lecture otes Correspodg chapters
Random Variables and Probability Distributions
Radom Varables ad Probablty Dstrbutos * If X : S R s a dscrete radom varable wth rage {x, x, x 3,. } the r = P (X = xr ) = * Let X : S R be a dscrete radom varable wth rage {x, x, x 3,.}.If x r P(X = x
Continuous Distributions
7//3 Cotuous Dstrbutos Radom Varables of the Cotuous Type Desty Curve Percet Desty fucto, f (x) A smooth curve that ft the dstrbuto 3 4 5 6 7 8 9 Test scores Desty Curve Percet Probablty Desty Fucto, f
Introduction to local (nonparametric) density estimation. methods
Itroducto to local (oparametrc) desty estmato methods A slecture by Yu Lu for ECE 66 Sprg 014 1. Itroducto Ths slecture troduces two local desty estmato methods whch are Parze desty estmato ad k-earest
9 U-STATISTICS. Eh =(m!) 1 Eh(X (1),..., X (m ) ) i.i.d
9 U-STATISTICS Suppose,,..., are P P..d. wth CDF F. Our goal s to estmate the expectato t (P)=Eh(,,..., m ). Note that ths expectato requres more tha oe cotrast to E, E, or Eh( ). Oe example s E or P((,
The expected value of a sum of random variables,, is the sum of the expected values:
Sums of Radom Varables xpected Values ad Varaces of Sums ad Averages of Radom Varables The expected value of a sum of radom varables, say S, s the sum of the expected values: ( ) ( ) S Ths s always true
Supplementary Material for Limits on Sparse Support Recovery via Linear Sketching with Random Expander Matrices
Joata Scarlett ad Volka Cever Supplemetary Materal for Lmts o Sparse Support Recovery va Lear Sketcg wt Radom Expader Matrces (AISTATS 26, Joata Scarlett ad Volka Cever) Note tat all ctatos ere are to
22 Nonparametric Methods.
22 oparametrc Methods. I parametrc models oe assumes apror that the dstrbutos have a specfc form wth oe or more ukow parameters ad oe tres to fd the best or atleast reasoably effcet procedures that aswer
Midterm Exam 1, section 1 (Solution) Thursday, February hour, 15 minutes
coometrcs, CON Sa Fracsco State Uversty Mchael Bar Sprg 5 Mdterm am, secto Soluto Thursday, February 6 hour, 5 mutes Name: Istructos. Ths s closed book, closed otes eam.. No calculators of ay kd are allowed..
Lecture 7: Linear and quadratic classifiers
Lecture 7: Lear ad quadratc classfers Bayes classfers for ormally dstrbuted classes Case : Σ σ I Case : Σ Σ (Σ daoal Case : Σ Σ (Σ o-daoal Case 4: Σ σ I Case 5: Σ Σ j eeral case Lear ad quadratc classfers:
STK4011 and STK9011 Autumn 2016
STK4 ad STK9 Autum 6 Pot estmato Covers (most of the followg materal from chapter 7: Secto 7.: pages 3-3 Secto 7..: pages 3-33 Secto 7..: pages 35-3 Secto 7..3: pages 34-35 Secto 7.3.: pages 33-33 Secto
Reliability evaluation of distribution network based on improved non. sequential Monte Carlo method
3rd Iteratoal Coferece o Mecatrocs, Robotcs ad Automato (ICMRA 205) Relablty evaluato of dstrbuto etwork based o mproved o sequetal Mote Carlo metod Je Zu, a, Cao L, b, Aog Tag, c Scool of Automato, Wua
Introduction to Probability
Itroducto to Probablty Nader H Bshouty Departmet of Computer Scece Techo 32000 Israel e-mal: bshouty@cstechoacl 1 Combatorcs 11 Smple Rules I Combatorcs The rule of sum says that the umber of ways to choose
best estimate (mean) for X uncertainty or error in the measurement (systematic, random or statistical) best
Error Aalyss Preamble Wheever a measuremet s made, the result followg from that measuremet s always subject to ucertaty The ucertaty ca be reduced by makg several measuremets of the same quatty or by mprovg
Binary classification: Support Vector Machines
CS 57 Itroducto to AI Lecture 6 Bar classfcato: Support Vector Maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 57 Itro to AI Supervsed learg Data: D { D, D,.., D} a set of eamples D, (,,,,,
Class 13,14 June 17, 19, 2015
Class 3,4 Jue 7, 9, 05 Pla for Class3,4:. Samplg dstrbuto of sample mea. The Cetral Lmt Theorem (CLT). Cofdece terval for ukow mea.. Samplg Dstrbuto for Sample mea. Methods used are based o CLT ( Cetral
Numerical Differentiation
College o Egeerg ad Computer Scece Mecacal Egeerg Departmet Numercal Aalyss Notes November 4, 7 Istructor: Larry Caretto Numercal Deretato Itroducto Tese otes provde a basc troducto to umercal deretato
Dimensionality Reduction and Learning
CMSC 35900 (Sprg 009) Large Scale Learg Lecture: 3 Dmesoalty Reducto ad Learg Istructors: Sham Kakade ad Greg Shakharovch L Supervsed Methods ad Dmesoalty Reducto The theme of these two lectures s that
Random Variables. ECE 313 Probability with Engineering Applications Lecture 8 Professor Ravi K. Iyer University of Illinois
Radom Varables ECE 313 Probablty wth Egeerg Alcatos Lecture 8 Professor Rav K. Iyer Uversty of Illos Iyer - Lecture 8 ECE 313 Fall 013 Today s Tocs Revew o Radom Varables Cumulatve Dstrbuto Fucto (CDF
DISCRIMINATING BETWEEN WEIBULL AND LOG-LOGISTIC DISTRIBUTIONS
ISSN: 39-8753 Iteratoal Joural o Iovatve Research Scece geerg ad Techology ISO 397: 007 Certed Orgazato ol Issue 8 August 03 DISCRIMINATING BTN IBU AND OG-OGISTIC DISTRIBUTIONS lsayed A lsherpey Sahar
Lecture 9: Tolerant Testing
Lecture 9: Tolerat Testg Dael Kae Scrbe: Sakeerth Rao Aprl 4, 07 Abstract I ths lecture we prove a quas lear lower boud o the umber of samples eeded to do tolerat testg for L dstace. Tolerat Testg We have
Bayesian belief networks
Lecture 14 ayesa belef etworks los Hauskrecht mlos@cs.ptt.edu 5329 Seott Square Desty estmato Data: D { D1 D2.. D} D x a vector of attrbute values ttrbutes: modeled by radom varables { 1 2 d} wth: otuous
Analysis of System Performance IN2072 Chapter 5 Analysis of Non Markov Systems
Char for Network Archtectures ad Servces Prof. Carle Departmet of Computer Scece U Müche Aalyss of System Performace IN2072 Chapter 5 Aalyss of No Markov Systems Dr. Alexader Kle Prof. Dr.-Ig. Georg Carle
Entropy, Relative Entropy and Mutual Information
Etro Relatve Etro ad Mutual Iformato rof. Ja-Lg Wu Deartmet of Comuter Scece ad Iformato Egeerg Natoal Tawa Uverst Defto: The Etro of a dscrete radom varable s defed b : base : 0 0 0 as bts 0 : addg terms
Introduction to Matrices and Matrix Approach to Simple Linear Regression
Itroducto to Matrces ad Matrx Approach to Smple Lear Regresso Matrces Defto: A matrx s a rectagular array of umbers or symbolc elemets I may applcatos, the rows of a matrx wll represet dvduals cases (people,
Naïve Bayes MIT Course Notes Cynthia Rudin
Thaks to Şeyda Ertek Credt: Ng, Mtchell Naïve Bayes MIT 5.097 Course Notes Cytha Rud The Naïve Bayes algorthm comes from a geeratve model. There s a mportat dstcto betwee geeratve ad dscrmatve models.
STK3100 and STK4100 Autumn 2017
SK3 ad SK4 Autum 7 Geeralzed lear models Part III Covers the followg materal from chaters 4 ad 5: Sectos 4..5, 4.3.5, 4.3.6, 4.4., 4.4., ad 4.4.3 Sectos 5.., 5.., ad 5.5. Ørulf Borga Deartmet of Mathematcs
Overview. Basic concepts of Bayesian learning. Most probable model given data Coin tosses Linear regression Logistic regression
Overvew Basc cocepts of Bayesa learg Most probable model gve data Co tosses Lear regresso Logstc regresso Bayesa predctos Co tosses Lear regresso 30 Recap: regresso problems Iput to learg problem: trag
Section 2 Notes. Elizabeth Stone and Charles Wang. January 15, Expectation and Conditional Expectation of a Random Variable.
Secto Notes Elzabeth Stoe ad Charles Wag Jauar 5, 9 Jot, Margal, ad Codtoal Probablt Useful Rules/Propertes. P ( x) P P ( x; ) or R f (x; ) d. P ( xj ) P (x; ) P ( ) 3. P ( x; ) P ( xj ) P ( ) 4. Baes
Bayes (Naïve or not) Classifiers: Generative Approach
Logstc regresso Bayes (Naïve or ot) Classfers: Geeratve Approach What do we mea by Geeratve approach: Lear p(y), p(x y) ad the apply bayes rule to compute p(y x) for makg predctos Ths s essetally makg
INTERNATIONAL BACCALAUREATE ORGANIZATION GROUP 5 MATHEMATICS FORMULAE AND STATISTICAL TABLES
INTERNATIONAL BACCALAUREATE ORGANIZATION GROUP 5 MATHEMATICS ORMULAE AND STATISTICAL TABLES To be used the teachg ad eamato of: Mathematcs HL Mathematcal Methods SL Mathematcal Studes SL urther Mathematcs
Normalized Exponential Tilting: Pricing and Measuring Multivariate Risks
Normalzed Expoetal Tltg: Prcg ad easurg ultvarate Rsks Shau Wag Departmet o Rsk aagemet ad Isurace Georga State Uversty Abstract: Ths paper dscusses expoetal tltg o the probablty desty ucto o a uderlyg
Extreme Value Theory: An Introduction
(correcto d Extreme Value Theory: A Itroducto by Laures de Haa ad Aa Ferrera Wth ths webpage the authors ted to form the readers of errors or mstakes foud the book after publcato. We also gve extesos for
7. Joint Distributions
7. Jot Dstrbutos Chrs Pech ad Mehra Saham Ma 2017 Ofte ou wll work o problems where there are several radom varables (ofte teractg wth oe aother. We are gog to start to formall look at how those teractos
Simple Linear Regression
Statstcal Methods I (EST 75) Page 139 Smple Lear Regresso Smple regresso applcatos are used to ft a model descrbg a lear relatoshp betwee two varables. The aspects of least squares regresso ad correlato
Randomness and uncertainty play an important
CHAPTER 3 Probablty, Statstcs, ad Iformato Theory Radomess ad ucertaty play a mportat role scece ad egeerg. Most spoke laguage processg problems ca be characterzed a probablstc framework. Probablty theory
D. VQ WITH 1ST-ORDER LOSSLESS CODING
VARIABLE-RATE VQ (AKA VQ WITH ENTROPY CODING) Varable-Rate VQ = Quatzato + Lossless Varable-Legth Bary Codg A rage of optos -- from smple to complex A. Uform scalar quatzato wth varable-legth codg, oe
Feature Selection: Part 2. 1 Greedy Algorithms (continued from the last lecture)
CSE 546: Mache Learg Lecture 6 Feature Selecto: Part 2 Istructor: Sham Kakade Greedy Algorthms (cotued from the last lecture) There are varety of greedy algorthms ad umerous amg covetos for these algorthms.
STA 105-M BASIC STATISTICS (This is a multiple choice paper.)
DCDM BUSINESS SCHOOL September Mock Eamatos STA 0-M BASIC STATISTICS (Ths s a multple choce paper.) Tme: hours 0 mutes INSTRUCTIONS TO CANDIDATES Do ot ope ths questo paper utl you have bee told to do
Multiple Linear Regression Analysis
LINEA EGESSION ANALYSIS MODULE III Lecture - 4 Multple Lear egresso Aalyss Dr. Shalabh Departmet of Mathematcs ad Statstcs Ida Isttute of Techology Kapur Cofdece terval estmato The cofdece tervals multple
Using Intraday Statistics for the Estimation of the Return Variance
Joural o Face ad Ivestmet Aalyss, vol., o., 03, -3 ISSN: -0998 (prt verso, -0996(ole Scepress Ltd, 03 Us Itraday Statstcs or the Estmato o the Retur Varace Erhard Reschehoer Abstract Ths paper proposes
1 Review and Overview
CS9T/STATS3: Statstcal Learg Teory Lecturer: Tegyu Ma Lecture #7 Scrbe: Bra Zag October 5, 08 Revew ad Overvew We wll frst gve a bref revew of wat as bee covered so far I te frst few lectures, we stated
ECON 5360 Class Notes GMM
ECON 560 Class Notes GMM Geeralzed Method of Momets (GMM) I beg by outlg the classcal method of momets techque (Fsher, 95) ad the proceed to geeralzed method of momets (Hase, 98).. radtoal Method of Momets