Part II Image recognition

Transcription

1 Image ad speech recogo Lecure oes Włodzmerz Kasprza Proec s co-faced by Europea Uo wh Europea Socal Fud. Paer recogo. Paer rasformao 3. Paer classfcao 4. Image pre-processg CONTENT Par I Paer recogo Par II Image recogo 5. Boudary-based mage segmeao 6. Rego-based mage segmeao 7. Model-based obec recogo W. Kasprza: EIASR

2 CONTENT Par III Speech recogo 8. Speech sgal pre-processg 9. Acousc feaure deeco. Phoec speech model. Word recogo W. Kasprza: EIASR 3 GENERAL REFERENCES. W. Kasprza: Rozpozawae obrazów sygałów mowy. WPW Warszawa 9.. R. Duda P. Har D. Sor: Paer Classfcao. d edo Joh Wley & Sos New Yor. 3. H. Nema: Klassfao vo Muser. d edo Sprger I. Pas: Dgal Image Processg Algorhms ad Applcaos. Prece Hall New Yor ec.. 5. D. Paulus J. Horegger: Appled Paer Recogo. A Praccal Iroduco o Image ad Speech Processg C++. 3d edo Veweg Brauschweg. 6. J. Beesy M.M. Sodh Y. Huag eds: Hadboo of Speech Processg. Sprger Berl Hedelberg 8. W. Kasprza: IASR 4

3 EIASR Paer recogo Lecure Proec s co-faced by Europea Uo wh Europea Socal Fud 5. Iroduco The GOAL of paer recogo: o aalyse o recogze o udersad ad descrbe he coe of gve paer dgal mage speech sgal ec.. Eample The dfferece bewee compuer graphcs mage syhess ad mage aalyss processes: W. Kasprza: EIASR. Paer recogo 6

4 Paer Paers ses of mul-varable fucos ha epress some physcal ey obec sysem ec. Doma of paers: f r Ω { f f r r r } U f r L Eample f rm... Some paers a bmap form from he doma of characers: W. Kasprza: EIASR. Paer recogo 7 Paer recogo approaches Compley of paers paer recogo approach:.smple paers emplae machg classfcao.comple paers a sequece or se of smple paers paer-sequece or -srucure model-o-daa machg 3.absrac paers e.g. a dyamc scee couous speech paer udersadg. -D mage ad speech recogo : paers represeed by speech or -D mage sgals deals wh smple ad comple paer recogo. W. Kasprza: EIASR. Paer recogo 8

5 Templae machg Templae machg a ave paer recogo approach: o mae a po-by-po mach of he paer wh he model. Eample. Characer recogo. Templae. Paer sze ormalzao: W. Kasprza: EIASR. Paer recogo 9 Paer classfcao Paer classfcao a smple paer recogo process: umerc feaure vecor deeco or al symbolc descrpo due o sgal segmeao; classfcao o decde abou he class of feaure vecor or symbolc descrpo. W. Kasprza: EIASR. Paer recogo

6 Comple paer recogo Comple paer machg: paer model - geerc descrpos erms of smple paer-sequece or -srucure; model-o-daa machg. Eample. W. Kasprza: EIASR. Paer recogo Eample Eample. Machg speech recogo: W. Kasprza: EIASR. Paer recogo

7 Eample Eample. Machg -D ad 3-D obec recogo model s a srucure of smple paers: W. Kasprza: EIASR. Paer recogo 3 Paer udersadg Absrac paer recogo paer udersadg: W. Kasprza: EIASR. Paer recogo 4

8 Sgal -D mage A sgal s a fuco of how oe varable s relaed o aoher varable e.g. a s. Sgal varables: Amplude - he fuco value; may represe volage lgh esy soud pressure ec. Tme he ypcal fuco parameer sgal me doma; however oher parameers are used specfc applcaos. -D Image a sgal whch he parameer me aes he form of a -D space e.g. a I y; sgal spaal doma. W. Kasprza: EIASR. Paer recogo 5 Image sca Scag: coverg a -D mage o a -D sgal Le sca Aeral sca Hlber sca W. Kasprza: EIASR. Paer recogo 6

9 . -D mage aalyss -D mage aalyss sysems coa dffere aalyss process a dffere daa absraco levels:.sgal level: mage preprocessg flerg - mage resorao mage ehaceme. Eample: source separao from mures Fe umber of mures Recosruced sources W. Kasprza: EIASR. Paer recogo 7 -D mage aalyss. Icoc level: mage compresso mage regsrao ormalzao classfcao of ere mage. Eample: mage ormalzao w.r.. paer oreao: a Ipu mage b bacgroud suppresso c foregroud border mage d roaed ormalzed mage W. Kasprza: EIASR. Paer recogo 8

10 -D mage aalyss 3 3. Segmeao level: boudary deeco rego deeco eure classfcao deph or moo deeco. Eample. Deeco of characers lcese plae regos: a Moo mas b c wo regos of eres ROI d regos correspodg o characers. W. Kasprza: EIASR. Paer recogo 9 -D mage aalyss 4 4. Obec level: model-based -D ad 3-D obec recogo. Eample. Palm recogo: a -D palm model b 3-D palm model c coour deeco d dscree feaure deeco e feaure-o-model machg W. Kasprza: EIASR. Paer recogo

11 3. Speech recogo Typcal problems seps a speech recogo sysem: W. Kasprza: EIASR. Paer recogo Speech recogo Problems seps co.: W. Kasprza: EIASR. Paer recogo

12 Smple speech classfcao Somemes a smple speech recogo sysem wll do for eample: For he corol of a devce whch eeds several commads oly If selecg a phoe umber by voce. The srucure of a smple word recogo sysem:. Sgal acquso ad VAD.. Specrogram aalyss wh fed frame umber e.g. 4 frames 5 samples.4 pos. 3. Appromao of he specrogram mage reducg he mage resoluo o 64 pos I he learg sage: modellg words from he dcoary by correspodg average low-resoluo specrogram mages. 5. Word recogo va drec mage classfcao. W. Kasprza: EIASR. Paer recogo 3 Specrogram mage classfer Eample. Four low-resoluo specrogram spoe words: a oec ad b OK. mages of he a The codg of specrogram values: Blac - he hghes amplude Blue mddle rage Yellow low rage whe ear zero rage. b W. Kasprza: EIASR. Paer recogo 4

13 Specrogram mage classfer 3 Eample. Specrograms of he spoe umbers polsh. Sgfca dffereces vsble. W. Kasprza: EIASR. Paer recogo 5 4. Probably heory The purpose of applyg probably heory paer recogo: o model he uceray of sesor daa ad of processg resuls for eample by meas of ose he mehods of error correco ormalsao fler requre he sascs of error sources for he udgeme of paer classfcao ad modelbased recogo resuls sochasc dsace measures are useful. W. Kasprza: EIASR. Paer recogo 6

14 Dscree sochasc varable A o-egave probably desy fuco pdf of a dscree sochasc varable X: PX p X. Probably of eves specfed by a erval of values: P A X B p X The cumulaed desy F X of a sochasc varable X : F P X p z X X A B z Mea ad varace of a pdf: µ X p σ X dom X X dom X [ µ X p ] W. Kasprza: EIASR. Paer recogo 7 Eample The dsrbuo of pel values e.g. eses colors a mage ca be modelled as a sochasc dscree varable. The ormalzed hsogram of a mage represes relave frequeces of pel values gve mage: a a mage ad b s esy hsogram W. Kasprza: EIASR. Paer recogo 8

15 Couous sochasc varable The probably value for a gve eac realzao of a couous sochasc varable X PXp X s fesmally small. I s meagful o cosder ervals of values: B P A X B p d The cumulaed desy: The pdf of a Gaussa ormally dsrbued varable: µ σ p X µ σ e σ π Mea ad sadard devao µ ad σ are he oly wo parameers of hs dsrbuo. F f z dz X A X W. Kasprza: EIASR. Paer recogo 9 Couous vs. dscree varable Sochasc dscree varables represe measurable observable ees.e. we deal wh a fe se of observaos he sochasc feaures of he observed varable usually chage wh ew observao. The probably heory by couous varables models he hdde sochasc process. Is feaures may be fed me. Momes he feaures of a pdf: he -h absolue mome of p : m p z p z dz he -h ceral mome of p : µ p m p z m p p z dz W. Kasprza: EIASR. Paer recogo 3

16 A vecor varable Le X [ X X X ] T - a vecor of sochasc varables. The mea vecor: The covarace mar: where T X E X E X E E }] { }... { } { [ } { X σ σ K 3 where Eample. The -dmesoal Gaussa dsrbuo: where de - deerma of a mar. W. Kasprza: EIASR. Paer recogo }} { } { { X E X X E X E X σ Σ σ σ σ σ K M O M de de de ] [ ] [ Σ Σ Σ Σ π π π X T e p µ µ Bayes rule Cha rule for sochasc varables: ]... [ 3 3 p p p p X X X X X X X X X X X p K K K 3 For sochasc depede varables holds: The Bayes rule relaes codoal probably o o dsrbuo: Hece: W. Kasprza: EIASR. Paer recogo ]... [ X p p Π X p p p X X X X X p p p p X X X X X X

17 Eropy Dualy of formao coe ad probably: low formao of eve X : hgh probably of eve. hgh formao of eve X : low probably of eve. Iformao coe of some realzao of varable X: I log P a Eropy of X he mea formao coe of varable X: H X P log P X a W. Kasprza: EIASR. Paer recogo Samplg ad dgalzao Sgal acquso ad aalogue-o-dgal A/D sgal coverso: D samplg me or space ad amplude dgalzao W. Kasprza: EIASR. Paer recogo 34

18 Samplg Samplg heorem: If he samplg frequecy of sgal f s a leas wo mes hgher ha he mamum frequecy compoe ϕ -B B of hs sgal he he full recosruco of he orgal aalogue sgal from s sample se s possble f f +/- +/...; where he samplg perod s / B. Dualy of sgal represeao he me space f ad frequecy doma Fω. W. Kasprza: EIASR. Paer recogo 35 Dgalzao of amplude A sequece of real-valued aalogue samples: { f }. The correspodg dgal-valued samples: { h } The dgalsao error ose : f - h The qualy of he dgalsao - he sgal o ose rao: E{ f } SNR log E{ } W. Kasprza: EIASR. Paer recogo 36

19 PCM codg PCM codg pulse code modulao : opmal dgalzao amplude ervals ad quazao levels. 37 Error measure: Soluo: ervals a ad levels b W. Kasprza: EIASR. Paer recogo + L a a df f p b f ε L df f p df f p f b L b b a a a a a B a a df f p b f b ε a p b a a p b a a ε Opmzao Opmzao heory heory elemes elemes LSE leas square error D case: o esablsh a lear relao bewee wo quaes y ad : y a + b or a + b - y gve N observaos: a + b - y ε a + b - y ε 38 a + b - y ε a N + b - y N ε N. Goal fuco: U a b ε + ε ε N ε ε Mmzao of Ua b:. Soluo: W. Kasprza: EIASR. Paer recogo a U b U y y N N b a

20 Lear programmg Lear programmg LP or lear opmzao: opmzao of a lear obecve fuco subec o lear equaly ad lear equaly cosras. The soluo s wh a cove polyhedro a se defed as he erseco of fely may half spaces. Lear programmg problems caocal form: Mamze he obecve fuco T ˆ arg ma c A b Subec o cosras where. LP MATLAB Opmzao Toolbo: BINTPROG LINPROG W. Kasprza: EIASR. Paer recogo 39 Quadrac programmg I Quadrac Programmg QP he goal obecve fuco s quadrac ad cosras are lear: T T ˆ arg m f H + subec o R b d... m... p where H s a srcly posve-defe mar of dmeso a vecor of legh a c vecors of legh ad b d scalars. I MATLAB - he quadprog roue. a c T T W. Kasprza: EIASR. Paer recogo 4

21 Quadrac programmg Smple: oly equaly cosras appear he QP problem: A b. If he umber of equaly cosras s he same as he umber of depede varables m cosras uquely deerme he soluo o he QP problem: A - b If he umber of equaly cosras s smaller ha he umber of depede varables m< use he elmao mehod for QP. Soluo of he QP. E.g. he acve se mehod: I cosecuve eraos of such algorhm QP ass wh equaly cosras are solved.e. oly acve cosras curre soluo po are ae o accou. The soluo o curre equaly QP gves he dreco whch he po s chaged. The creme s a mmum value ha chages oe prevously acve cosra o a acve oe. W. Kasprza: EIASR. Paer recogo 4 6. Esmao heory Esmao of a sochasc sysem The observao z of he sysem sae s s dsored by ose w: z h s w 3.. N A esmao of sae s s eeded as a drec soluo s o avalable. If he added ose s of zero-mea he we have a basfree esmaor. Sochasc varable esmaor s s a sochasc varable:.the ML esmaor whou pror formao..the MAP esmaor MAP eplores he pror formao. Deermsc varable esmaor:.the LSE esmaor..the MMSE esmaor. W. Kasprza: EIASR. Paer recogo 4

22 ML MAP The ML mamum lelhood esmaor The codoal observao dsrbuo s avalable The ML esmae: s p z z... z s s ML s s arg ma p z z... z s The MAP mamum poseror esmaor A o-uform pdf ps s avalable. The MAP esmae - o mamse a weghed combao of observaos ad pror ps: MAP s arg ma p z z... z s p s s Whe ps s uform o pror formao fla dsrbuo fuco he MAP smplfes o ML. W. Kasprza: EIASR. Paer recogo 43 LSE MMSE The LSE leas square error esmae: s LS arg m s [ z h s] For a Gaussa ose wh zero mea value he LSE esmae s equvale o he ML esmae. The MMSE mmum mea square error esmaor The MMSE esmae s he epeced value of sae w.r.. gve measuremes: MMSE s arg m E[ sˆ s z z... z ] arg m sˆ sˆ [ z h sˆ] For a o-sochasc varable z he MMSE esmae s equvale o he LSE ad s value s he curre real value of he sae varable. p z W. Kasprza: EIASR. Paer recogo 44

23 EIASR Paer rasformao Lecure Proec s co-faced by Europea Uo wh Europea Socal Fud 45. Classfer desg A smple paer classfcao problem: o represe he paer by a umerc feaure vecor c R d ad o assg o a approprae dscree class label κ : d R { L K} ζ c c c κ The desg of a smple paer classfer depeds o:.he qualy of he feaure se ha s avalable for rag learg e.g. mamse he cocerao of feaures from he same class ad / or mamse he separao bewee feaures from dffere classes.he epeced classfcao error due o classfer ype W. Kasprza: EIASR. Paer rasformao 46

24 Feaure space Space rasforms le Fas Fourer Trasform or Dscree Cose Trasform are orhogoal ad fed rasformaos - se depedely of he learg se. We revew basc lear rasforms of he paer space le PCA LDA ICA ha deped o he daa rag se ad whch opmze some crera se o hs daa. The ave bass of paer space: Is here aoher bass whch s a lear combao of he orgal bass ha bes epresses our daa? W. Kasprza: EIASR. Paer rasformao 47 Space rasformao crera Le: K - umber of classes; N - umber of rag samples vecors of sze f a rag sample of he -h class; c feaure for he -h rag sample of he -h class. We are seeg lear rasformaos of he paer space: c Φ f ha are opmal wh respec o some goal fucos crera. Goal fucos: Average squared dsace bewee wo feaures PCA prcpal compoe aalyss: N N T s c c c c N W. Kasprza: EIASR. Paer rasformao 48

25 Trasformao crera s ad s 3 Average squared dsace bewee feaures from wo dffere classes: K N Nl T s c cl c cl K K N N l l 3 Average squared dsace bewee wo feaures from he K N N same class: T s3 c c c c K N A popular form s he combao of s ad s 3 whch gves a crero called Fsher s formao appled lear dscrmae aalyss - LDA. W. Kasprza: EIASR. Paer rasformao 49 Trasformao crera s ad s 3 The opmal rasformao mar w.r.. s l - Φ l coas egevecors ϕ ν l of some symmerc mar Q l. I holds: Q l ϕ ν l λ ν l ϕ ν l where λ l ν ν... are egevalues of Q l. To mamze he crera s or s : compue egevecors ϕ ν or ϕ ν relaed o larges egevalues λ ν or λ ν ν... ; of he mar Q or Q respecvely. To mmze s 3 : compue he egevecors relaed o he smalles egevalues of some mar Q 3. W. Kasprza: EIASR. Paer rasformao Φ l l ϕ l ϕ M l ϕ T T T 5

26 Daa covarace marces Symmerc marces Q l l3 requred by he crera s l : ω f f N f f N N N T T m R mm R Q 5 W. Kasprza: EIASR. Paer rasformao N N N N T T l T l K l K f f N f f N K K K ω + m R m m m m R Q K T K 3 m m R Q Proof dea Show ha every crero s l ca be epressed as: The sum of posve values aas s mamum f all s compoes aa her mama. T l s Q 5 3 Use repeaedly he Hoellg hess: Le Q be some symmerc ad posvely-defed mar ad a vecor of approprae sze. Defe: a T Q. The a s of mamum possble value f s a egevecor of mar Q relaed o he larges egevalue of Q. W. Kasprza: EIASR. Paer rasformao

27 Cocluso The crera are epressed as sums of egevalues of approproaely defed daa covarace marces: s l T l ϕν Q ϕν ν ν λ l ν If some umber N of learly depede vecor samples are avalable [ f... N ] every oe of sze he he mar Q of sze s of ra ad here es dffere posve- ad real-valued egevalues λ > λ > > λ ad correspodg egevecors. W. Kasprza: EIASR. Paer rasformao 53. PCA The rasformao opmzg s s Prcpal Compoe Aalyss also called Karhue-Loeve rasform: Ge he zero-mea covarace mar Q of samples { }. Oba s egevecors by he egevecor decomposo or SVD algorhm: T T Q λ ϕ ϕ UΛU r r r where U represes he mar of egevecors ϕ r of Q ad Λ s he dagoal mar of posve egevaluesλ r whle s he ra of he mar Q. 3 Selec he feaure space dmeso m. The feaure vecor c m correspodg o sample s: c m Λ m / U m. r W. Kasprza: EIASR. Paer rasformao 54

28 PCA PCA: a roao of a orhogoal coordae sysem ha orders he as o capure larges varaces of daa samples. If s he pu vecor sze he some m < prcpal compoes represe eresg srucure whle hose wh lower varaces represe ose. W. Kasprza: EIASR. Paer rasformao 55 Iverse-PCA rasform Le: dmesoc m m; dmeso. I PCA usually m < eve mos ofe m <<. Hece feaure eraco by PCA s paer ecodg or compresso. Iverse PCA decodg decompresso Use o-ormalzed feaure coeffces: c~ T r λrϕr As he mar U PCA s a orhogoal mar s verse s a raspose. Thus he verse rasformao o PCA s: m T ˆ U c~ c~ ϕ r ad s recosruced by lear combao of egevecors ad feaures vecor coeffces. W. Kasprza: EIASR. Paer rasformao r r 56

29 Eample Illusrao of PCA-based mage compresso-decompresso. Blocs of sze 88 are compressed ecoded ad decompressed decoded usg 6 prcpal compoes hus 64 m6. W. Kasprza: EIASR. Paer rasformao 57 Eample Resuls of bloc-based compresso-decompresso by PCA ad verse PCA wh -5 prcpal compoes. W. Kasprza: EIASR. Paer rasformao 58

30 3Lear dscrmae aalyss LDA deermes a opmal separao he sese of s ad s 3 of feaures for samples from dffere classes. The wo-classes case Le N samples {... N} are gve R from wo classes Ω Ω. We loo for a lear rasformao: R R: y w T The mass cere of rego Ω : m N Ω I he oupu rasformed space: ~ T T m w w m y N y Y N Ω T m ~ m ~ w m m W. Kasprza: EIASR. Paer rasformao 59 Two-classes LDA The rasformed wh-class varably: Toal wh-class varably: ~ ~ s + s Defo ~ y m~ y s. The lear Fsher s dscrmae fuco for classes s yw ot where w arg ma J w ad Jw s he goal fuco called Fsher s formao: Fsher s formao he pu space: where S o m~ m~ J w ~ s + ~ s W. Kasprza: EIASR. Paer rasformao w T B m m m m S S + S W S m m Ω J w T Y w w T S S T B W w w 6

31 Two-classes LDA Geeral soluo o w o s: S W S Bw o λw o Here he epresso S B w s a vecor alog he dreco of m - m ad he scalg of w s o mpora. Thus: w o S W m m Geeral LDA more ha wo classes > The wh-class varably mar: S W S where N - umber of samples from class.... The bewee-class varably mar: SB where m - he mass cere of all he samples. N W. Kasprza: EIASR. Paer rasformao 6 m m T N m mm m T Geeral LDA Defo The lear Fsher s dscrmae fuco s: y W o where W o arg ma J W W T W S J W T W S W W ad B. I LDA we apply he W o ha duces he mamum rao bewee he deermas of he bewee-class varably mar ad he wh-class varably mar of samples. Soluo The rows of mar W o are egevecors of he mar S W - S B : S W S B W o ΛW o W W. Kasprza: EIASR. Paer rasformao 6

32 4. ICA Isaaeous-me verse problem: m uow source sgals e.g. [s... s m ]; possbly osy bu dffere lear mures... of he sources > m. he mg coeffces are some uow cosas. m A s + s a + W. Kasprza: EIASR. Paer rasformao 63 The verse problem Soluo of he verse problem :. Idepede compoe aalyss. Proeco pursu 3. Facor aalyss 4. Prcpal compoe aalyss W. Kasprza: EIASR. Paer rasformao 64

33 Facor aalyss PCA Facor aalyss FA assumes ha he sources called facors are ucorrelaed of Gaussa dsrbuo ad of T u varace: E{ s s } I The ose elemes are ucorrelaed wh each oher ad T wh he facors: Q E{ } The he covarace mar of he observao s: T E{ T } R A A + Q Assumg Q s ow or ca be esmaed we aemp o T solve A from: A A R Q T Whou ose he problem smplfes o a PCA: A A R where every dreco space s of u varace. W. Kasprza: EIASR. Paer rasformao 65 ICA for feaure eraco Applcao of ICA depede compoe aalyss: here we are eresed o code a sgal mage bloc by s feaure vecor ha cosss of mg coeffces whle mg some se of depede compoes sadard mage blocs: a a a m s s s m Assumpos ICA:.The sources are sochascally depede w.r.. fucos f g ha capure hgher order sgal sascs.e. for y y holds E{ f y g y} E{ f y} E{ g y}. A mos oe of he sources s of Gaussa dsrbuo. W. Kasprza: EIASR. Paer rasformao 66

34 Eample Eample [Hyvare] Two depede sources: Joed dsrbuo s s : s p s 3 f s 3 oherwse Mg mar Mure dsrbuo 3 s A W. Kasprza: EIASR. Paer rasformao 67 Eample Wheg: a lear rasform so ha he ew mures are ucorrelaed ad have varace oe. PCA: fd orhogoal ma drecos o separao. ICA: source separao by geeralzed depedecy. W. Kasprza: EIASR. Paer rasformao 68

35 FasICA FasICA Hyvare Karhue Oa s a bach mehod for ICA. I requres ha he mures are ceered ad wheed frs. Ceerg: ceered E{ } m Wheg: E{ T wheed } R Λ / UΛU U The he mehod eravely updaes he wegh mar W vecor-by-vecor whle mamzg he No-Gaussay of he proeco w T. T T W. Kasprza: EIASR. Paer rasformao 69 FasICA A. Ialze a ozero mar: W[w w w ] B. Ierae FOR oupus p DO + T T Wegh updae: w p E { g w p } E { g ' w p } w p where g s a olear fuco g s frs dervao. 3 Normalze o u legh: A Gram-Schmd orhogoal: 5 IF W has o ye coverged THEN e erao of -5. W. Kasprza: EIASR. Paer rasformao w p w w w p + p + p w w p p p w w p p w T p w w T 7

36 5. Bld source separao BSS mehods solve he verse problem le ICA. Bu here we focus o o-le recosruco of sources raher ha dealg wh feaure eraco where depede compoes are esablshed a off-le mode. Applcao: solvg he cocal pary Cchoc Amar W. Kasprza: EIASR. Paer rasformao 7 Demg BSS Demg BSS: a m separag mar W s updaed so ha he m-vecor y W becomes a esmae y s of he orgal depede sources up o scalg ad permuao deermacy. If source scalg S ad permuao P are resolved he: W A S P I W. Kasprza: EIASR. Paer rasformao 7

37 BSS The Kullbac Lebler dvergece measures he depedecy amog oupu sgals muual formao: p y D W p ylog dy K p y D W H y; W + Hy W - he average muual formao of he oupus y. The Hy -s are eropes of margal dsrbuos her sum s cosa oly he muual formao of oupus s mpora. The BSS opmzao rule: arg m D W W. Kasprza: EIASR. Paer rasformao W K H y 73 BSS The adapve separao rule geeraed accordg o he aural grade: s: T W + W + η { I f [ y ] g[ y ]} W fy [fy... fy ] T ad gy T [gy... gy ] are vecors of o-lear acvao fucos whch appromae hgher order momes of he sgals. If he sources have egave uross values.e. hey are sub Gaussa sgals we choose: fy y 3 gy y. For super Gaussa sources wh posve uross we choose: fy ahα y gy y. W. Kasprza: EIASR. Paer rasformao 74

38 Eample Soud separao: oe uow source oe mure ad oe separaed sgal are show: W. Kasprza: EIASR. Paer rasformao 75 Eample Four uow sources: Four mures wh added ose: Separaed sgals: W. Kasprza: EIASR. Paer rasformao 76

39 Measurg he separao qualy A If he source sgals are ow For every par oupu Y ; source S wh ampludes scaled o he erval <- > compue her SNR[] he sgal o ose rao as: SNR[ ] log S / MSE[ ] ; m where S - he me-average of -d power of source.e. he average eergy MSE[ ] - he error of appromag S by Y : N MSE[ ] [ S where N - umber of Y ] S Y N samples. A mar P[ p ] s creaed wh p SNR[ ] The error de: EI P ~ p + p m m ~ Every row of P s scaled: P Norm P such ha ma ~ a Every colum s scaled: P NormColP such ha ma W. Kasprza: EIASR. Paer rasformao a 77 Measurg he qualy B If he mg mar A s ow The error de for he se of separaed sources s gve as: EI P + ~ p p m m The eres p -s of mar P P W A are ormalzed alog rows... for he frs epresso or colums m P such ha: ma ~ or ma p Ideal case of perfec separao: P becomes a permuao mar. Oly oe eleme each row ad colum equals o uy ad all he oher elemes are zero. The he mmum EI. p W. Kasprza: EIASR. Paer rasformao 78

40 Measurg he qualy 3 C If boh he sources ad mg mar are uow The ormalsed muual correlao coeffces are compued E{ y y } r y y for every par of oupu sgals y ad y gvg he mar: P[r ]. The error de for he se of separaed sources s compued as: EI P ~ r m W. Kasprza: EIASR. Paer rasformao 79 EIASR Paer classfcao Lecure 3 Proec s co-faced by Europea Uo wh Europea Socal Fud 8

41 . Numerc classfers Opmzao crera for classfers. Mmum compuaoal compley lear dscrmae classfers. Mmum of msclassfcao probably Bayes classfer 3. Ably auomacally o lear a decso rule mullayer percepro 4. Ably o geeralse owledge for o avalable samples Suppor Vecor Mache 5. Mmzao of geeral rs??? W. Kasprza: EIASR 3. Paer classfcao 8 Decso heory approach A. Mae assumpos abou he aalysed doma for eample lear he pror pdf-s. B. Defe he coss ad rs of msclassfcao decso. C. Defe a decso rule whch mmses hs rs. Sochasc classfer - he aalysed doma ad decso rule erms of sochasc varables. Le us observe: Each decso duces dvdual coss. Afer may decsos he average epeced coss ca be esmaed he rs of msclassfcao. The calculao of average values requres he avalably of full sasc formao pdf-s. W. Kasprza: EIASR 3. Paer classfcao 8

42 Rs mmzao A Assumpos abou he aalysed doma: Pdf of observao c gve class p c. Pror probables { p...k } of paers from classes. B Calculao of rs V c - a paer c from class.. Coss of msclassfcao: {r r... ;K};. Oba he probably of paer from class : p c p 3. The rs of paer c : V c: p p c r. C Decso rule. Mmse he rs V. If a bary cos fuco s used: r r for ad... K. he we eed o mamze p c.. Selec class wh hghes p c : arg ma p c. W. Kasprza: EIASR 3. Paer classfcao 83. Poeal fuco classfer Poeal fucos as class dsrbuos feaure space; All fucos from he same paramerc famly of fucos. Lear poeal fuco famly: T + d c a { a ϕ c a R c c c L c ϕ T } L m Lear poeal fuco-based classfer wo class-case: Ge poeals: Decso rule: d d c a c a d c d c a - d c a If d c he selec he class Ω else class Ω a a + c + a c a c W. Kasprza: EIASR 3. Paer classfcao 84

43 Eample Bary classfer for a -D feaure vecor he lear decso rule correspods o wo half-plaes separaed by he le: d c d c. W. Kasprza: EIASR 3. Paer classfcao 85 Lear classfer desg Lear classfer for K classes: ς c arg ma d c a arg ma a Learg:. C: mar of N samples where N samples are from class.. For every class Ω defe he equao sysem: D C a where D [d d... d N] ad d c a f c Ω or - f c Ω. 3. Solve a lear opmsao problem by leas square error approach defed ; hs gves a sysem: X B a whch ca be solved drecly: a B T B - B T X. + a c W. Kasprza: EIASR 3. Paer classfcao 86

44 3. Bayes classfer Sochasc dsrbuos are eeded: -pror class pdf-s pω K -codoal pdf-s pc Ω. Decso rule: ς c arg ma p Ωλ c arg ma p Ωλ p c Ωλ. λ Learg he probably dsrbuos:.pω are esmaed as relave frequeces of classes he learg sample se;.pc Ω : o-paramerc e.g. hsograms; or paramerc - assume a desy fuco famly ad esmae he parameers from he learg samples. λ W. Kasprza: EIASR 3. Paer classfcao 87 Learg a Bayes classfer No-paramerc pdf Paramerc desy ML mamum lelhood esmaor: for Ω specfy parameer seθ whch mamzes he probably of gve observaos ma P c θ θ log Θ c C Solve: θ c C W. Kasprza: EIASR 3. Paer classfcao P c θ θ θ 88

45 Mmum-dsace classfer Ths s a specfc form of he Bayes classfer for Gaussa pdf-s f uses he Eucldea mercs. Selec he class of mmum dsace: dω c l pω c. The specal case I he Bayes codo: p Ω p c Ω > p Ω p c Ω s equvale o: T T l p Ω c µ c µ > l p Ω c µ c µ Wh I ad he mamum lelhood rule pω pω smplfes o: c µ < c µ Ths s he rule of a geomerc mmum-dsace classsfer ha uses he Eucldea mercs. W. Kasprza: EIASR 3. Paer classfcao 89 Learg. 4. The -NN classfer A se of feaure vecors: c C { c c... c } where each sample belogs o class ζ c. Learg a eares eghbour-classfer meas smply o sore all samples. The se of pos he feaure space for whch he eares eghbour amog he learg se s equal o c s called a Voroo cell of c. Decso rule ζ c: selec he mos freque class of he eares eghbours of c. W. Kasprza: EIASR 3. Paer classfcao 9

46 5. Suppor Vecor Mache SVM s a dscrmae-le bary classfer cosdered whe proper class dsrbuos over he feaure space are dffcul o be esablshed lmed se of daa samples. SVM learg or desg: o fd a opmal hyper-plae a lear SVM or hyper-surface for o-lear SVM ha separaes he wo areas feaure space represeg wo classes. The classfcao wh SVM cosss of may bary decsos bewee wo classes or class groups deoed as + -. The Vap Chervoes dmeso VC h s a measure for he se of separag fucos. I case of classes h deermes he mamum amou of paers ha ca be separaed o all possble ways.e. hs umber of separaos s h. W. Kasprza: EIASR 3. Paer classfcao 9 VC dmeso Thess. The Vap-Chervoes dmeso of he se of oreed hyper-plaes he R space s h +. Eample.Three pos o a plae -D pos ca be separaed by a oreed le o all possble wo sub-ses he umber of such paros s 3 8. Ths s o loger rue for 4 pos; hece for -D pos he VC dmeso s h3. VC dmeso gves he umber of supporg vecors he umber of samples eeded o defe he separag hyperplae. W. Kasprza: EIASR 3. Paer classfcao 9

47 SVM for a learly separable se Eample. The separag hyper-plae H*. There could be oher separag plaes e.g. H whch are o opmal. The opmzao crero: o mamze he dsace of he eares suppor vecors or o mmze a. W. Kasprza: EIASR 3. Paer classfcao 93 SVM opmzao problem Defo. The SVM for a learly separable se of samples from classes s deermed by a hyper-plae H * : d T + a a ~ c c a ha sasfes he goal: m f a m a + + a~ R a~ R ad N addoal cosras... N: Ths opmzao problem s called cove quadrac programmg wh equaly-based lear cosras. The decso rule s: T y c a + a c ω Ω ζ c Ω f f d d a ~ a ~ c c < W. Kasprza: EIASR 3. Paer classfcao 94

48 Suppor vecors The cosras are cluded o he goal fuco usg posve-valued Lagrage mulplers: ϑ ϑ... ϑ N T. A modfed goal he Lagrage fuco: N T L a ~ ϑ a ϑ y c a + a Suppor vecors are he oly samples eeded for he hyper-plae; her addoal cosras are acve ad hey have o-zero Lagrage mulplers ϑ >. N T H* : da ~ c ϑ y c c The hyperplae s a weghed scalar produc of he suppor vecors. W. Kasprza: EIASR 3. Paer classfcao + a 95 Lear SVM uder ose Lear separao uder osy codo: addg a pealy compoe he SVM s goal fuco - coverg wrogly separaed samples. Eample. Error dsaces ε 5 ad ε 6 for samples of class + ad ε 7 ad ε 8 samples of class -. W. Kasprza: EIASR 3. Paer classfcao 96

49 The dual problem The modfed goal fuco: N m f a m a + C ~ ~ ε + + a R a R T subec o y c a a ε c ω ε + C s a cosa corollg he fluece of he osy compoe oo he goal fuco. ~ The dual problem: ma L D ϑ ma[f L a ϑ] ~ wh where subec o L Q D ϑ y ϑ C N y ϑ c T ϑ c ; N N N ; ϑ ϑ y ϑ y W. Kasprza: EIASR 3. Paer classfcao 97 y T y [...] ϑ T ϑ c T a T c ϑ Qϑ + T ϑ Solvg he SVM problem Boh he prmary ad he dual form of he SVM opmzao problem are saces of he geeral quadrac programmg problem for eample see he quadprog fuco MATLAB. There ess radoal mehods le Newo mehod or acve se mehod for solvg he quadrac programmg problem. Bu wh may hudreds or housads rag samples radoal mehods cao be drecly appled. A effce learg algorhm ha solves he dual problem for SVM s Sequeal mmal opmzao SMO J.C.Pla. W. Kasprza: EIASR 3. Paer classfcao 98

50 Kerel SVM Lear separaors a hgh-dmesoal feaure space Fc by replacg c ad c H* equao wh F c ad F c Eample: N T H* : da c ϑ y F c F c + a ~ a The rue decso boudary s: +. b Afer mappg o a hree-dmesoal feaure space:. a b W. Kasprza: EIASR 3. Paer classfcao MLP McCulloch-P s model of a sgle euro: Ipus: Ipu fuco: y Acvao fuco: z θy Oupu: z w w where w - bas wegh W. Kasprza: EIASR 3. Paer classfcao

51 Sgle feed-forward layer I ANN arfcal eural ewors he euros are orgased o layers. I a feed-forward percepro ewor he pu sgals are se from he pu o he oupu va ecaory ls. y W z θ y W. Kasprza: EIASR 3. Paer classfcao Acvao fucos Sgle sep fuco: z f y > y where y - a fed hreshold. θ y Sgmod fuco: + ep βy Eample. Acvao fucos: lef sep fuco rgh sgmod fuco. W. Kasprza: EIASR 3. Paer classfcao

52 Sgmod fuco The dervave of a sgmod fuco z θ y ep s: dz + y z z dy Eample. Sgmod fuco wh dffere parameer β. W. Kasprza: EIASR 3. Paer classfcao 3 Mul-layer percepro MLP: some umber a leas wo of feed-forward layers arraged a cascade. The rasformao fuco of each l l l l layer l...: z θ W z w where z. Eample. A MLP wh 3 layers hdde layers. W. Kasprza: EIASR 3. Paer classfcao 4

53 Supervsed learg of MLP The Wdrof-Hoff rule dela rule he wegh of comg l bewee -h euro ad -h pu s sregheed proporo o he dfferece bewee requred ad real acvao: w µ s z where s s he requred acvao of he -h euro ad µ - a learg coeffce. The error bac-propagao rule MLP s a eeso of he dela rule. A updae of weghs layer l l L L- : T l l l W µ d z z where d l s he correco vecor for he l-h layer. A sgle learg erao sars form he las layer l L ad proceeds bac-o-bac layer-by-layer ul l. W. Kasprza: EIASR 3. Paer classfcao 5 Bacpropagao learg The correco values are se as follows: For las layer he requred oupu vecor s s avavlable. Hece L L z d s z.* y For hdde layers he correco vecor s proeced bac. Hece for l L- : l l+ T l+ z d W d.* y.* - a eleme-by-eleme mulplcao of wo vecors. For a sgmod acvao fuco he correcos are: d d L l s z W l + L.* L L [ z ].* z l + l l.*[ z ].* z W. Kasprza: EIASR 3. Paer classfcao T d 6

54 7. Paer cluserg Usupervsed learg procedures use ulabeled samples. -meas cluserg. I cluser ceers: µ.. FOR each observao c DO assocae wh he closes cluser ceer ha s assg ξc where dµ c m dµ c for some dsace fuco d e.g. Eucldea dsace. 3. FOR each cluser wh assged observaos DO esmae he mea of he observaos assged wh cluser as: c ξ c 4. REPEAT seps ad 3 gve umber of mes. µ W. Kasprza: EIASR 3. Paer classfcao 7 EM cluserg The epecao-mamzao EM algorhm s a geeral esmao echque dealg wh mssg daa. If appled for cluserg he EM algorhm gves a mamumlelhood esmae of a Gaussa mure: K α N c Λ p c θ µ where N. deoes a Gaussa pdf ad θ - he parameer se ha eeds o be esmaed: θ { α µ Λ K}. E-sep: FOR each sample c ad FOR each class DO: M-sep: FOR each class DO: c P T c c P µ Λ P P W. Kasprza: EIASR 3. Paer classfcao P p c ξ c θ α p c ξ c θ α α P P 8

55 8. Esemble learg May classfers or epers are geeraed ad combed o solve a parcular classfcao or decso problem:. Boosg combg wea bary classfers o a srog oe. Each erao of boosg creaes hree wea classfers: he frs classfer C s raed wh a radom subse of he rag daa. C s raed o a se oly half of whch s correcly classfed by C ad he oher half s msclassfed. C3 s raed wh saces o whch C ad C dsagree. The 3 classfers are combed hrough a hree-way maory voe. AdaBoos.M adapve Boosg eeso for mul-class classfcao mos popular esemble classfer Freud & Shapre 995. Mure of epers Jacobs & Jorda W. Kasprza: EIASR 3. Paer classfcao 9 AdaBoos AdaBoos s a algorhm for cosrucg a srog classfer as a lear combao of T wea classfers: T F C[ f ] C w h where h s he classfcao hypohess of sample gve by he -h wea classfer w are weghs ad C s he rule of he srog classfer. The se of wea classfers H {h} s poeally fe. Trag daa samples are draw from a dsrbuo ha s eravely updaed such ha subseque classfers focus o creasgly dffcul saces. Prevously msclassfed saces are more lely o appear he e boosrap sample. The classfers are combed hrough weghed maory vog. W. Kasprza: EIASR 3. Paer classfcao

56 AdaBoos.M INPUT: classes: Ω{Ω Ω Ω L }; rag samples: S { } wh labels y Ω ; umber of wea classfers: T. TRAINING INIT dsrbuo: D... FOR T DO. Selec curre rag subse S accordg o D.. WeaLear: ra o S ad reur he wea classfer h : XΩ wh smalles error: ε I h y D IF ε >/ THEN STOP ε 3. Calculae ormalzed error: β ε 4. Updae dsrbuo D : D β or D D + f h y D + f h y Z Z W. Kasprza: EIASR 3. Paer classfcao AdaBoos.M Remar: Z s a ormalzao coeffce ha maes D + a proper dsrbuo. CLASSIFICATION by a weghed maory vog: Gve a ew ulabeled sample s:. Oba oal voe receved by each class v.e. he wea weghs are: T I h s Ω log/ β log / β. Rule C: selec he class Ω whch receves he hghes oal voe v... L W. Kasprza: EIASR 3. Paer classfcao

57 Mure of Epers Several epers classfers are leared. The -h eper produces s oupu: where W s a wegh mar ad f. - a fed oleary. W o f 3 The oupus of epers are combed hrough a geeralzed lear rule: The weghs of hs combao are deermed by a gag ewor: W. Kasprza: EIASR 3. Paer classfcao N g o v o y v v T N e e g ξ ξ ξ ; Mure of Epers The classfcao sep he mure of epers approach ca be epressed sochasc erms as he mamzao of poseror probably: where Ψ s he se of all parameers all eper- ad gag weghs. Ψ N P g P W y v y 4 where Ψ s he se of all parameers all eper- ad gag weghs. The parameers are ypcally raed usg he epecao mamzao EM algorhm. Le he rag se s gve as: { y T}. I he E-sep he s-h epoch for all he rag daa followg poseror probables are compued: W. Kasprza: EIASR 3. Paer classfcao N P g P g p N s s s s... W y v W y v

58 Mure of Epers 3 I he M-sep followg mamzao problems are solved: W V s+ s+ arg ma arg ma T W V T p log P y N p log g v W where V s he se of all he parameers he gag ewor. W. Kasprza: EIASR 3. Paer classfcao 5 EIASR Image pre-processg processg Lecure 4 Proec s co-faced by Europea Uo wh Europea Socal Fud 6

59 . Scee acquso Perspecve proeco A smple phole model of he camera: p - p -y p reversed mage po P c y c c scee po camera coordaes f focal legh. y p p s s y f z c c c f y z c s s y : scalg of scee-o-mage us W. Kasprza: EIASR 4. Image pre-processg 7 Perspecve vs parallel proeco Parallel proeco f f eds o fy : lm f c p s f f + zc s c f y c y p lm sy f f + zc s y y c Perspecve proeco vs. parallel proeco W. Kasprza: EIASR 4. Image pre-processg 8

60 Vecor operaos 3-D vecors: Ier produc of wo vecors: 3 3 v v v v u u u u 3 u u u + + u 9 Cross produc of wo vecors W. Kasprza: EIASR 4. Image pre-processg 3 3 v u v u v u T + + > < v u v u 3 u u u + + u cos v u v u > < θ 3 3 ˆ v u v u v u + + v U v u ˆ 3 3 u u u u u u U Camera Camera parameers parameers A rasformao of scee po P oo pel p depeds o camera parameers of wo ypes:.ersc parameers roao R raslao T.Irsc parameers proeco K F camera-o-mage K s Ersc camera rasformao Ersc camera rasformao Le P w [ w y w z w ] be he world coordaes ad P c [ c y c z c ] he camera coordaes of some scee po P. The: Irsc camera rasformao W. Kasprza: EIASR 4. Image pre-processg T RP P + w c c c c c f c z y f f y z P K y o s o s s y y p y y s p p θ K

61 Homogeeous coordaes A 3-D po P 4-D homogeeous coordaes P h : Traslae a po by vecor [ y z ] T : Scalg of coordaes by [ s s y s z ] T :. ] [ ] [ Z Y X P Z Y X P T T h z y T Roae he coordae sysem: - aroud as Z by agle θ - aroud as X by agle α - aroud as Y by agle β W. Kasprza: EIASR 4. Image pre-processg z y s s s S cos s s cos α α α α R α cos s s cos β β β β R β cos s s cos θ θ θ θ R θ Homogeeous coordaes Perspecve rasformao of homogeeous coordaes: Reverse rasformao s o uque! Le us observe:. / f Ψ f / z f c c c p Le us observe: The rasformao from camera o pel us The operaos of pel scalg sewg ad shf of opcal ceer ca be represeed by a sgle mar for operao o homogeeous coordaes. W. Kasprza: EIASR 4. Image pre-processg / / / f z y f z f f z z y orm orm f y c c c c c c c c f p p h P h Ψ p. y y s o s o s s θ Ψ

62 . Camera calbrao Eample. Ersc parameers R T: - α : roao of aes aroud global as OZ; - β : roao of aes aroud global as OY; - θ : roao of aes aroud global as OX; - G : raslao of global sysem o camera fure sysem; - C : raslao of camera fure sysem o camera sysem; Irsc parameers ψ f ψ s : f : he camera s focal legh s s y : he mage-o-pel u scalg o o y : he camera org o mage org shf. W. Kasprza: EIASR 4. Image pre-processg 3 Eample The rasformao of a po P w gve world coordaes o a pel p : p Ψ Ψ T R R R T P A P s f C θ β α G Remar: hs eample we have defed a raslao of he coordae sysem whle above equao a po s shfed. A rasformao of a po s a dual operao o he rasformao of a coordae sysem. p y p p A P Auo-calbrao of he camera z p 3 The goal s o esmae he combed mar A p4 for he scee-o-mage rasformao based o some umber of pel-o-scee po correspodeces. W. Kasprza: EIASR 4. Image pre-processg w w 4 w

63 Auo-calbrao Mar A cosss of parameers bu oly 8 are depede he 3-d row learly depeds o he 4-h row by f. p a a a3 a4 X y p a a a a Y p 3 4 A Pw z p a a a a Z p4 a4 a4 a43 a44 As p p 4 ad p y p 4 we ge a sysem of 3 lear equaos: p 4 a X + a Y + a 3 Z + a 4 y p 4 a X + a Y + a 3 Z + a 4 p 4 a 4 X + a 4 Y + a 43 Z + a 44 y ad XYZ represe measurable daa whle p 4 s o ow. I eeds o be elmaed from he sysem. W. Kasprza: EIASR 4. Image pre-processg 5 Auo-calbrao Afer elmao of p 4 from he s ad d row we ge equaos wh uows: a X + a Y + a 3 Z - a 4 X - a 4 Y - a 43 Z - a 44 + a 4 a X + a Y + a 3 Z - a 4 y X a 4 y Y - a 43 y Z - a 44 y + a 4 Auo-calbrao algorhm. Deec m 6 mage pos { p y...m} whch are proecos of ow 3-D pos world coordaes - P X Y Z...m.. For every par of pos p P mae wo equaos of above form oal M equaos wh. uows a - a Solve he equao sysem M by a LSE approach. W. Kasprza: EIASR 4. Image pre-processg 6

64 EIASR Image pre-processg processg Lecure 4 Proec s co-faced by Europea Uo wh Europea Socal Fud 7. Scee acquso Seco spped. Camera calbrao Seco spped W. Kasprza: EIASR 4. Image pre-processg 8

65 3. Color spaces Color spaces ha are based o psychologcal observaos o huma percepo: HLS Hue Lghess Saurao HSV Hue Value Saurao. Techcal sysems represe color as a mure of hree basc vecors he color space: A ypcal color sysem for compuer moors - hree basc colors: red gree blue he RGB mage where RGB are coeffces from he erval [ ]. A addve posve-oreed sysem - "whe" s he sum of s compoes. Typcal color prgs o prers - he CMY Cya Magea Yellow sysem. A addve "egave" model - "blac" s he sum of compoes: [C M Y] [ ] - [R G B]. W. Kasprza: EIASR 4. Image pre-processg 9 Color spaces Typcal aalog elevso broadcas - YIQ Y - lumace I Q - chromac compoes: red ad red-blue. A specfc sesvy of he huma eye o he gree color allows o represe gree mosly by Y. Y R I G Q B I dgal meda - YUV or Y C b C r where C b C r - dsaces of he color from "gray" color alog blue ad red aes. Y C C b r R.5 G.83 B Y.99 U.47 V R.436 G. B Noe: RGB are coeffces from [ ]. Y also aes values from [ ] bu C b C r or U V ae values from [-.5.5]. W. Kasprza: EIASR 4. Image pre-processg 3

66 Colour calbrao Ideal colours blue gree red yellow magea whe dar specfc s colours dar s ad lgh s ec. are defed he YUV space. They are appled o mae a colour calbrao of he mage. W. Kasprza: EIASR 4. Image pre-processg Colour plae U-V for md-value Y.5 of or 8 of 55 3 Y-based color ormalzao I heory a Y-based ormalzao of he color compoes U Y s o requred. Bu pracce hs ca help o lm he varably of dar ad lgh versos of he same color. Assume 8 bs per color per pel represeao.e. values from [ 55]. Yp ' Y ' 8; α for Yp > 8 p Yp U p' U p 8 κ p + 8; 56 Yp κ p β for Yp < 8 Vp' Vp 8 κ p + 8; Yp ' for Yp 8 E.g. for s color: α.35 ad β/ α W. Kasprza: EIASR. Paer recogo 3

67 Eample The deeco of s colour he mage: W. Kasprza: EIASR 4. Image pre-processg Image ehaceme Low coras mages ca be ehaced by hsogram srechg or hsogram equalzao. Srechg of hsogram H.Fd a mh A ad b mah A such ha A% of pels have lower value ha a ad A% of pels have hgher value ha b e.g. A.5..Trasform he pel value w o a ew dgal value he erval [... D m ] as: f w < a w a f w Dm f a w b b a Dm f w > b W. Kasprza: EIASR 4. Image pre-processg 34

68 Hsogram equalzao The purpose of hsogram equalzao s o mae every pel value early equally probable: w a f w roud[ Dm H ] PNum where H s he hsogram value for pel value PNum oal pel umber [... D m ] s he erval of dgal levels of varable w he orgal mage roud[] specfes he roudg operao o eares dgal level. Remar: perfecly fla hsograms are seldom obaed whe worg wh dgal varables. Equalzao usually leads o a reduco of pel value levels he rasformed mage f compared o he orgal oe. W. Kasprza: EIASR 4. Image pre-processg 35 Eample Image ad s hsogram before ad afer hsogram equalzao: W. Kasprza: EIASR 4. Image pre-processg 36

69 5. Image barzao Problem: how o se he hreshold for mage barzao o separae he foregroud obec pel from he bacgroud? Soluo dea: he mage hsogram s appromaed by a weghed sum of wo ormal Gaussa dsrbuos. The border bewee dsrbuos deermes he hreshold value: p l αn l m σ + α N l m σ W. Kasprza: EIASR 4. Image pre-processg 37 The Osu mehod. FOR all possble hresholds θ ; oba he bewee-class varace as: σ θ P θ P θ m θ m θ B θ h l l L where P θ θ. h l l. Selec θ such ha σθ B s of mamum value. There ess a fe umber of hreshold posos oly. Hece he above algorhm always ermaes. W. Kasprza: EIASR 4. Image pre-processg 38

70 6. Paer ormalzao A mage obec ca appear of dffere sze ad oreao ad everheless we eed o recogze. Eample: dffere leers W bu he same class of leer. Eample: ormalzao of mage sze W. Kasprza: EIASR 4. Image pre-processg 39 Paer ormalzao Paer rasformao mage space raslao scalg roao mrrorg - fucos of he geomerc momes. A ormalzao sep: paer f y paer h' y' global momes m pq µ pq m pq µ pq p q p q y f y µ ' y ' h ' y ' y y Sep : shf o mass cere c y c m m c yc ' c y' y yc m m ad ormalze he amplude f y h ' y' m µ ad µ µ W. Kasprza: EIASR 4. Image pre-processg 4

71 Paer ormalzao Sep : sze ormalzao scalg of aes r m + m y ' y' r r h ' y' r f y µ + µ Sep 3: roao - o ormalze he oreao of he paer. If m m he mmze ad ge: S α [ cosα y sα] y f c y c aα m m m W. Kasprza: EIASR 4. Image pre-processg 4 Sep 3 co.: Paer ormalzao 3 Amog 4 possble agles selec α such ha afer roao holds: µ < µ ad µ >. ' cosα sα y ' s α cosα y h ' y' f y µ µ < µ ad µ > Sep 4: mrrorg wh respec o he Y as. Selec β {+ -} such ha afer he rasformao ' β y' h ' y' f y he resulg paer s mome s µ >. y W. Kasprza: EIASR 4. Image pre-processg 4

72 7. Image flers Basc mage flers are used o suppress: he hgh frequeces he mage.e. smoohg he mage or he low frequeces.e. deecg edges he mage. Spaal fler: o covolve he pu mage f wh some fler fuco h called erel: g h f Frequecy fler: rasform he mage o he frequecy doma mulply he resul wh he frequecy fler fuco ad 3 re-rasform he resul o he spaal doma. Dscree covoluo s a shf ad mulply operao: for a square erel wh sze M+ M+ he dscree covoluo s: g M / M / m M / M / W. Kasprza: EIASR 4. Image pre-processg h m f m 43 Basc mage flers: Basc mage flers.mea fler - ose reduco NR usg mea of eghbourhood.meda fler - NR usg meda of eghbourhood 3.Gaussa smoohg - NR wh a Gaussa smoohg erel 4.Varous grade-based edge deeco 5.Laplace fler - secod dervao-based edge deeco Mea fler: o replace each pel value a mage wh he mea average value of s eghbours cludg self. The erel: W. Kasprza: EIASR 4. Image pre-processg 44

73 Meda fler No-lear fler: he addo operao dscree covoluo s replaced by some o-lear operaor: g O [ h m f m ] m Meda fler s a o-lear fler. I replaces he pel value wh he meda of eghbour values. The meda s calculaed by frs sorg all he pel values from he local eghbourhood accordg o umercal order ad he replacg he ceral pel by he mddle value. Eample: W. Kasprza: EIASR 4. Image pre-processg 45 Gaussa smoohg Gaussa smoohg operaor uses a erel ha appromaes a Gaussa fuco. -D soropc crcularly symmerc Gaussa: G y + y ep πσ σ Dscree Gaussa erel: assume zero a dsaces more ha hree sadard devaos from he mea ad rucae he erel. Eample: dscree appromao of Gaussa fuco wh σ.. The -D soropc Gaussa s separable o ad y compoes apply a -D covoluo o rows ad colums wh erel: W. Kasprza: EIASR 4. Image pre-processg 46

74 Edge mage deeco A pre-processg sep edge deeco: a smoohg operao order o remove ose spy-le varaos from he mage. Eample: desred edge lef real edge rgh. Basc ypes of edge mage deecors:. dscree mage fuco grades. covolve mage wh erels 3. usg paramerc edge models 4. med approaches. W. Kasprza: EIASR 4. Image pre-processg 47 Dscree mage grades The grade of a -D couous fuco: f y f y f y f f y y y y Dscree dffereces case of a -D mage fˆ ha represes he couous fuco f y: ˆ fˆ + fˆ ˆ fˆ + fˆ f As a resul of edge deeco wo oupu mages are compued:.he magude sregh s or he "absolue" sregh s for compuaoal smplcy s f + f y s ' f + f y ad. he dreco of he ormal vecor o edge: r arca f / f y f y W. Kasprza: EIASR 4. Image pre-processg 48

75 Rober s cross Robers cross : eample of a dscree grade-based edge operaor. Two dscree grades alog 45 o ad 35 o : f - f + + y f + - f +. Remar: hey are equvale o covoluo erels: Edge sregh: s + y or s ' + y Edge oreao correco by -3Π/4 eeded: Robers y arca Characerscs of he Robers Cross operaor. he same mage forma as pu mage e.g. mamum sregh: 55 for 8-b mages smple mplemeao wo dfferece values per pel oly sesve o ose. r 3 π 4 W. Kasprza: EIASR 4. Image pre-processg 49 Dffereces of ceral eleme Two dffereces ha appromae he wo grades of mage fucos alog he ma aes: ˆ fˆ + fˆ ˆ fˆ + fˆ f f y Remar: equvale covoluo erels are: Characerscs: The same pu-oupu forma e.g. mamum sregh 55 for bye/pel smple mplemeao sll sesve o ose. W. Kasprza: EIASR 4. Image pre-processg 5

76 Covoluo-based edge deecor Sobel operaor: G G y Prew operaor: Characerscs: Sobel operaor: dffere pu-oupu mage e.g. mamum sregh: 4 for 8-b pu mage smple mplemeao good resuls. Prew operaor: dffere pu-oupu mage e.g. mamum sregh: for 8-b pu mage smple mplemeao resuls are almos as good as for he Sobel operaor. W. Kasprza: EIASR 4. Image pre-processg 5 Dscree drecos A lmed umber of edge oreaos s deeced. Ths allows for a effce mplemeao of oreao deeco sep. For eample 6 dscree mage drecos are deeced by comparg wo edge grades for each pel. W. Kasprza: EIASR 4. Image pre-processg 5

77 Laplace operaor Laplace operaor he sum of secod-order dervaves For a couous -D fuco he Laplace operaor s defed as: f f f y + f + f yy y The dscree Laplace operaor uses a sgle mas bu s depede from ay dreco oreao formao s los. I gves a posve" aswer.e. a zero value for boh real edges ad homogeeous regos he mage - ca be used a combao wh some oher edge operaor. Basc dscree Laplace operaor: f 4 f - f - f + - f - - f + The covoluo erel: W. Kasprza: EIASR 4. Image pre-processg 53 LOG fler Usg a secod dervave maes he Laplaca hghly sesve o ose use Gaussa smoohg as a preprocessg sep. Laplaca of he Gaussa LOG fler Due o leary: L y L y G y* I y G y* I y W. Kasprza: EIASR 4. Image pre-processg Dervao of he LOG fler D case: G ep σ π σ - frs dervave: G' ep 3 σ π σ - secod dervave: G' ' ep 3 σ π σ σ σ 54

78 Illusrao of he LOG fler: LOG fler Dffere Laplace covoluo erels: a Basc Laplaca bc LOG fler W. Kasprza: EIASR 4. Image pre-processg 55 Color mage edge operaor A edge operaor defed for a moochromac mage ca be eeded o hadle color mage. For eample le be a RGB mage fuco f RGB. Le pars of colour pels be gve: f r g b f r g b - from he local eghbourhood of curre pel. I case of he Sobel color operaor here wll be gve hree pars for he "vercal" mas ad 3 pars for he "horzoal" mas. The dfferece for par f f ca be aleravely compued as: D f ; f { r - r + g - g + b - b } / D f f r - r + g - g + b - b D 3 f f ma { r - r g - g b - b } D 4 f f ω r r - r + ω g g - g + ω b b - b W. Kasprza: EIASR 4. Image pre-processg 56

79 8. Edge hg Threshold-based edge elmao Ths smple edge hg mehod s a edge elmao operaor wh a mmum hreshold parameer θ. The hreshold s eher fed or se adapvely e.g.θ γs ma where γ. s P f s P > θ s h P oherwse No-mamum edge elmao I depeds o a chec he local eghborhood of gve pel P : IF s P s N OR r P r N T AND s P s N THEN s P h L R s P; OR r P r N L ELSE s P h R T ; W. Kasprza: EIASR 4. Image pre-processg 57 Edge modfcao Edge hg A local eghborhood-based modfcao of edge a pel P: f P s he sroges edge eleme he se: P N L N R he: s P sp + α sn L + sn R ; f P s he weaes edge eleme he above se he: s P sp - α sp; f oe eghbour of P deoed by P + s a sroger edge ad aoher eghbour of P deoed by P s a weaer edge eleme he: s P sp - α sp + + α sp -. Several eraos over he whole mage may be ecessary. W. Kasprza: EIASR 4. Image pre-processg 58

80 Edge hg 3 Edge elmao wh hyseress hreshold Ths edge hg mehod wors wh wo edge sregh hresholds: he upperθ H ad he lowerθ L. I he frs ru hese edge pels are dvdually mared as good ha have hgher sreghs ha he upper hreshold. I he e ru hese good pels are red o be eeded alog a poeal coour le boh drecos posve ad egave le dreco. For a sgle eeso he eghbor pel eed o have hgher sregh ha he lower hreshold. Remar: Now he eghbors are o compeg wh each oher ad hey are searched alog he epeced coour le o across. W. Kasprza: EIASR 4. Image pre-processg Cay operaor Ths a mul-sage edge mage deeco ad hg procedure. I eeds 3 parameers: he varace σ for he Gaussa mas edge sregh hresholds θ H θ L where θ H >θ L for hyseress- based hg. Ipu : a grey-scale mage. Oupu : a hed edge mage. Seps of he Cay operaor.image smoohg by covoluo wh a Gaussa erel..edge mage deeco by a dscree-grade operaor or 33 eghbourhood. 3.Edge hg by he o-mamum elmao operaor. 4.Edge elmao wh a hyseress hreshold. W. Kasprza: EIASR 4. Image pre-processg 6

81 EIASR Boudary-based mage segmeao Lecure 5 Proec s co-faced by Europea Uo wh Europea Socal Fud 6. Le deeco Ma approaches o le deeco mages:. A 3-sep le segme deeco Edge mage deeco flerg Edge cha followg segmeao To appromae edge chas by sragh le segmes symbolc descrpo.. A -sep le or coour deeco Edge mage deeco Deeco of sragh les crcles ad coours by usg a Hough rasform. Acve coour mehod W. Kasprza: EIASR 5. Boudary-based mage segmeao 6

82 . Edge cha followg Prcple: searchg for eeso of curre edge pel PP -cur s successor edge NcP cur. Two eghbour edge elemes ca be led f he edge magude sregh ad dreco dffereces are below cera hresholds ad her magudes are relavely large: sp sn T rp rn mod π T sp > T sn > T Deoe he 3 eares pel alog he dreco rp as: N N N 3. The successor of P -cur : a edge eleme N whose sregh ad oreao are mos smlar o P -cur. by W. Kasprza: EIASR 5. Boudary-based mage segmeao 63 Successor caddaes Three caddaes for a successor of pel P: Closg a gap for edge pel P drecos lef o ad rgh 45 o : W. Kasprza: EIASR 5. Boudary-based mage segmeao 64

83 The hyseress hreshold mehod The edge sregh hreshold T s ow spl o wo: - he upper hreshold T ad - he lower hreshold T β T < β < The sregh of he sar edge pel has o be hgher ha he upper hreshold T : sp sar > T The sregh of ay e edge eleme he cha eeds o be hgher ha T. Eample. Edge cha labels: W. Kasprza: EIASR 5. Boudary-based mage segmeao 65 The hyseress hreshold REPEAT Search for a edge pel P sar wh o segme label.e. free wh sregh sp sar > T o upper hreshold. P sar s ow called curre pel P -cur ad he segme label s SegNum. FORWARD_SEARCH P -cur IF cha SegNum s o closed THEN Aga se he curre pel P -cur o be he sar pel P sar. Perform BACKWARD_SEARCHP -cur UNTIL all pels are vsed. Eample: W. Kasprza: EIASR 5. Boudary-based mage segmeao 66

84 Edge cha mage Ipu mage 67 W. Kasprza: EIASR 5. Boudary-based mage segmeao 3. Le segme 3. Le segme f f The equao of a le deermed by wo pos y y : The dsace d of po y from hs le: > b a c by a y y y y y y 68 W. Kasprza: EIASR 5. Boudary-based mage segmeao b a c by a d + +

85 Edge cha appromao Procedure APPROXIMATE CI P PN Add he le l deermed by sar po P ad ed po PN of he cha CI o he se of le segmes LC. For every po P cha CI deerme he dsace s o le l. IF FOR ALL P CI: s θ THEN RETURN END. Selec a po P wh mamum dsace s. Remove he le l from se LC. Decompose he le CI o pars: CI P... P ; CI P... P N. Call APPROXIMATECI P P. Call APPROXIMATECI P P N. W. Kasprza: EIASR 5. Boudary-based mage segmeao 69 Appromao by arcs. Try o appromae wo cosecuve le segmes by a arc. E.g. sar wh arc ACB.. Measure he appromao error E.g. ea arg ma g f eb arg ma g f AND sg g ea ma g f f sg g f g THEN Appromae ACB R 3; 567; W. Kasprza: EIASR 5. Boudary-based mage segmeao ea ea ea eb eb f f eb eb < γr 7

86 Appromao by arcs AppromaeACB R W. Kasprza: EIASR 5. Boudary-based mage segmeao 7 Appromao by arcs 3 3. Try o eed a esg arc by cludg he e le segme E.g. arc ACB s checed o be eeded o ACBD. The a eeso o ACBDF fals bu here s a secod arc DFE. 4. Try o appromae closed arc chas by a crcle or ellpses. W. Kasprza: EIASR 5. Boudary-based mage segmeao 7

87 4. Hough rasform HT A polar represeao of a sragh le defed by a edge eleme e cosss of wo parameers: d α where d cos α + y s α Remar: α 9 o + β ad α re s he edge s dreco. A mage le correspods o a po he Hough space d; α. W. Kasprza: EIASR 5. Boudary-based mage segmeao 73 Hough rasform for les Edge mage E [ e ]... M;... N; s of sze M N ad represes O NUM dscree edge drecos. The Hough accumulaor Hd;α s a dgal represeao of he Hough space: π π M + N d M + N α Hough Trasform for le deeco FOR every edge pel e coordaes y oreao α DO: - Compue d cos α + y s α ; - Appromae d α by eares dscree values d + α + ad crease Hd + α +.e. Hd + α + Hd + α + +. FOR every d α DO: - IF Hd α > Threshold THEN correspodg le s deeced. W. Kasprza: EIASR 5. Boudary-based mage segmeao 74

88 HT for crcle ceer deeco A crcle - c + y - y c r has 3 parameers. A wo-sep crcle recogo approach Sep : deec crcle ceres by approprae Hough rasform; Sep : deec edge chas aroud hose ceres. Hough Trasform for crcle ceer deeco Le a le perpedcular o edge eleme e be: g : y a + b Les correspodg o crcle border pos cross he po C c ; y c - he cere po of crcle. I Hough space Hab hese les wll correspod o collear pos a b. W. Kasprza: EIASR 5. Boudary-based mage segmeao 75 Crcle ceer deeco A le g ha passes hrough c y c sasfes: y a - c + y c a + y c - a c. Ths le correspods o he po Hough space: a ; y c - a c. The relao of a se of pos Hough space s appromaely a sragh le: b - a c + y c. For N po observaos: he LSE soluo s: b b M b N a a L an y C N a ab yc N a a a a b c c W. Kasprza: EIASR 5. Boudary-based mage segmeao 76

89 HT for crcle deeco Now le us apply he Hough rasform o deec all 3 parameers of a crcle equao - c + y - y c r oe sep. Hough rasform for crcle deeco FOR every edge pel e coordaes y oreao α DO: - he uow cere coordaes C y C of a crcle wh uow radus r d passg hrough po y sasfy: C + r d cos α y y C + r d s α ; - FOR every dscree radus r d : r d r ma DO: - compue C y C from he above equaos ad - crease he Hough accumulaor H C y C r d by oe; FOR every C y C r d DO: - IF H C y C r d > Threshold THEN correspodg crcle s deeced. W. Kasprza: EIASR 5. Boudary-based mage segmeao 77 Coour deeco by GHT A geeralsed Hough rasform GHT for coour deeco Parameers of he Hough space: C C ; y C : locao of he ceer mass s - scale α - coour oreao agle. Model learg desg For every par of allowed dscree values s d α g of scale ad oreao creae a able Rs d α g [ rϕ ϕ ] where for each edge B wh edge dreco ϕ he par: [ rϕ B - C ϕ] s sored. Eample: The model coour op drawg ad he caddae coour boom W. Kasprza: EIASR 5. Boudary-based mage segmeao 78

90 5. Acve coour The acve coour or sae s defed as a eergy mmzg polygo - he sae's eergy depeds o s shape ad locao wh he mage. A sae s alzed roughly o represe he obec s boudary. The sae s moo rule s he eravely appled o chage he sae s locao ad shape o sasfy he fal codo: F eral + F eeral Feeral E eeral Eample. Two fal saes esablshed from commo al sae whle usg more ouer or less er eeral force. W. Kasprza: EIASR 5. Boudary-based mage segmeao 79 Edge-based eeral eergy: or eeral eeral Sae s forces E y I y σ E y G y I y The eral eergy s resposble for elascy ad sffess ryg o shore ad o smooh he coour: E.g. E E elasc sffess E E + E eral K K elasc sffess p p p p + p+ W. Kasprza: EIASR 5. Boudary-based mage segmeao 8

91 Moo updae rule Sae dyamcs dscree moo rule : a each erao every corol po p y s moved by a vecor proporoal o he force acg o : + αf + β F + γ F For eample: I parcular: elascx sffessx eeralx y y + αf + β F + γ F elascy sffessy eeraly α.8 β. γ F K + elascx F K y y + y y elascy F K + + sffessx F K y + y y y + y K3 FeeralX I + y I y K3 FeeralY I y + I y sffessy W. Kasprza: EIASR 5. Boudary-based mage segmeao 8 Eample Two dffere fal coours obaed by varyg he weghs of he eral force. Ier coour: α.8 β.5 γ.6 Ouer coour: α.8 β. γ.6 W. Kasprza: EIASR 5. Boudary-based mage segmeao 8

92 6. Po deecor The Harrs-Sephes operaor Deec average mage grades I I y he local eghborhood of mage po y represe hem as a covarace mar: A y A po feaure s deeced whe boh egevalues of mar A are of hgh ad smlar value. W. Kasprza: EIASR 5. Boudary-based mage segmeao 83 EIASR Rego-based mage segmeao Lecure 6 Proec s co-faced by Europea Uo wh Europea Socal Fud 84

93 . Homogeeous rego A rego R s a coeced mage area whch s homogeeous wh respec o some parameer vecor of parameers e.g. esy colour eure ad gve predcae HR. Eamples of homogeey predcae. f ma R m R < θ R H R oherwse where: mar ma p R fp mr m p R fp. where: or f f p mea R < θ R H p R oherwse F R θ R θ A θ A θ B F R ma θ R σ f p mea R ; where p R W. Kasprza: EIASR 6. Rego-based mage segmeao Rego growg ad mergg A smple approach o rego deeco s o sar from some pels called seeds s represeg dsc mage regos R ad o grow hem ul hey cover he ere mage. The al seed deeco: we mae a hsogram of he ere mage; mage pels whose mage value correspods o hsogram peas are seleced o be seeds. Rego growg: a each sage ad for each rego R chec f here are ulabeled eghbour pel of each pel of he rego border ad f he homogeey crero for he elarged rego wll sll be vald. If s rue he elarge he rego by hs pel. W. Kasprza: EIASR 6. Rego-based mage segmeao 86

94 Spl-ad-merge Rego mergg: merge adace regos ha have smlar sascal properes. For eample he wo regos R R are allowed for merge f her arhmec meas are smlar: Predcae 3 mea R mea R < θ R R Spl-ad-merge quadree srucure I: quadree a some md-level SPLIT: chec wheher every leaf ode s homogeeous MERGE: chec wheher 4 relaed leafs ca be merged Image W. Kasprza: EIASR 6. Rego-based mage segmeao 87 Rego deeco algorhm Combed rego deeco algorhm M θ θ A θr : Ial mage essellao o square regos of sze M M. REPEAT wh he quadree represeao DO MERGE sep wh a fed hreshold θ SPLIT sep wh a fed hreshold θ UNTIL furher spl or merge s o possble predcae. Rego mergg/growg wh a fed hreshold θ A pred. 3 Rego mergg wh a adapve hreshold θr pred. W. Kasprza: EIASR 6. Rego-based mage segmeao 88

95 . Teure Teure - a measure of mage regulary coarseess ad smoohess. Eamples of eures: grass wood waer surface ec. W. Kasprza: EIASR 6. Rego-based mage segmeao 89 Teure classfcao Teure recogo s a classfcao problem. Learg: fd approprae feaures ra a classfer Classfcao: deec feaures of curre eure; classfy he feaures erms of leared classes. Teure feaures: sascal e.g. varace sewess; specral e.g. usg he auocorrelao fuco or Fourer rasform; srucural e.g. usg dscree feaures le: colour moo umber of edgs. W. Kasprza: EIASR 6. Rego-based mage segmeao 9

96 Hsogram-based eure feaures Le f L be he values of he mage fuco ad pf he ormalzed hsogram of a mage rego..mea µ µ f p.varace 3.Sewess 4.Kuross 5.Eropy µ µ L f L σ f µ p f L 3 3 f µ p f 3 σ L µ f µ p f H f L p f log p f W. Kasprza: EIASR 6. Rego-based mage segmeao 9 Hsograms of pars Hsograms of sums ad dffereces Le f ; f +µ; +ν be wo pel he mage separaed by he dsplaceme vecor: V µ; ν T. Ther sum ad dfferece are: s ; f ; + f + µ ; + v ; d ; f ; f + µ ; + v Hsograms of sums ad dffereces H s lv ad H d mv for dsplaceme V : H s l V # f ; such ha s ; l H m V # f such ha d d ; ; m They coa formao abou he spaal orgazao of pel values. E.g. a coarse eure resuls a cocerao of hsograms aroud: l mea ad m. Repea for several dsplacemes e.g.: V µ; ν T { - -} W. Kasprza: EIASR 6. Rego-based mage segmeao 9

97 Feaures of sums ad dffereces Afer ormalsao: h H s l s l l... L h d m H l l ypcal feaures are:. Mea s m m Hd m L +... L H L L s d l m L+. Coras L L c 3 l h s l c4 m h d m l m L + 3. Varace L L c 5 s c6 m c h d m l m L+ 4. Eropy L L c 7 hs l log hs l c8 hd m log hd m l ml+ W. Kasprza: EIASR 6. Rego-based mage segmeao 93 m d Co-occurrece mar The co-occurrece mar s defed as: Gd; r [g d; r] where a eleme g d; r specfes he umber of pel pars f f whch are separaed by dsace d alog he dreco r. Eample For a 5 5 mage ad L 4 values he co-occurrece mar has L 4 4 elemes f G; E.g. g f f f f f f f f W. Kasprza: EIASR 6. Rego-based mage segmeao 94

98 Feaures of co-occurece mar d r Afer ormalsao: g d r... L g d r l Typcal feaures:. Global secod-order mome: c g. Eleme dfferece-value mome of order relaed o L coras: g l l c l 3. Normalzed correlao of border dsrbuos of eleme values: g mm c3 σ σ 4. Eropy: c4 g log g g W. Kasprza: EIASR 6. Rego-based mage segmeao 95 Fler bas Covolug a mage bloc wh a se of erels yelds a represeao of bloc s eure a dffere space. There s a srog respose whe he eure loos smlar o he fler erel ad a wea respose whe does. Eample. The colleco of erels may coss of a seres of spos ad bars - a dffere scales. Spo flers respod srogly o small o-oreed pos. Bar flers are oreed ad ed o respod o edges. W. Kasprza: EIASR 6. Rego-based mage segmeao 96

99 Gabor fler bas Gabor flers - Fourer Trasform elemes mulpled by Gaussas. A Gabor erel respods srogly f locaed over mage blocs wh eure havg a parcular spaal frequecy ad oreao. Gabor flers come pars: symmerc ad a-symmerc. A symmerc erel: + y Gsymmerc y cos + y yep{ } σ A a-symmerc erel: + y Gasymmerc y s + y yep{ σ Where y are he spaal frequeces. } W. Kasprza: EIASR 6. Rego-based mage segmeao 97 Eample Eample. Gabor fler erels where md-grey values represe a zero darer values represe egave umbers ad lgher values represe posve umbers. The op row shows hree asymmerc erels ad he boom row - hree symmerc erels; horzoal dreco. W. Kasprza: EIASR 6. Rego-based mage segmeao 98

100 Gabor flers MPEG-7 MPEG-7 descrpors he frequecy doma: polar coordaes he frequecy doma s sampled o 3 propery chaels he phase s wdh s uformly of 3 degree ad he magude wdhs are powers of : 3 θ 3 o r r K 5 s r ω ω s K 4 ω s o 4 W. Kasprza: EIASR 6. Rego-based mage segmeao 99 Gabor flers MPEG-7 co. The Gauss fuco chael s r: The eergy e G P s r ω θ ω ωs ep σ s of he -h chael: e log + g g θ θ r ep ρ σ θ r K3 8 [ GPs r ω θ F ω θ ] ω + θ + The sadard devao of eergy q he -h chael: q log + σ K3 σ 8 [ GPs r ω θ F ω θ g / N ] ω + θ + Where Fωθ D Fourer rasform o he frequecy space represeed by polar coordaes. W. Kasprza: EIASR 6. Rego-based mage segmeao

101 3. -D shape feaures Nomal umbers. Dsaces from wo base les lower ad upper measured a seleced pos. The umbers of crossg pos of he coour ad a gve se of les. Hsograms. Proecg he paer oo s coordae sysem aes resuls wo -D dsrbuos hsograms. W. Kasprza: EIASR 6. Rego-based mage segmeao Compacess feaures Geomery feaures Area A boudary legh L Compacess coeffce: γ A e.g. γ.6 for a crcle 6 for a square L 4πA Normalzed compacess: γ N L e.g. γ N for crcles ad γ N for comple shapes. W. Kasprza: EIASR 6. Rego-based mage segmeao

102 Mome-based feaures Global ad ceral geomerc momes of he -D shape: pq p q y p q m p y µ y y p y y All ceral momes are raslao-vara. pq The followg mome-based fucos {φ φ 7 } are raslao- ad roao-vara: φ µ + φ µ µ + µ φ µ W. Kasprza: EIASR 6. Rego-based mage segmeao 3 y 4 3 µ 3 3µ + 3µ µ 3 φ 4 µ 3 + µ + µ + µ 3 5 µ 3 3µ µ 3 + µ [ µ 3 + µ 3 µ + µ 3 ]+ 3µ µ 3 µ + µ 3 [ 3 µ 3 + µ µ + µ 3 ] 6 µ µ µ 3 + µ µ + µ µ µ 3 + µ µ µ 3 7 3µ µ 3 µ 3 + µ [ µ 3 + µ 3 µ + µ 3 ]+ µ 3 3µ µ + µ 3 [ 3 µ 3 + µ µ + µ 3 ] φ + [ ] φ + φ c c Mome-based feaures The mome-fucos {φ φ 6 } are refleco- mrrorg- vara as well. I case of φ 7 s magude s refleco vara bu s sg chages uder refleco. The mome-fucos {φ φ 7 } ca be made scale-vara afer a ormalzao by: r µ + µ φ φ3 φ4 φ ' φ 4 3' φ 6 4' 6 r r r φ5 φ6 φ7 φ 5 ' φ 6' φ 8 7' r r r W. Kasprza: EIASR 6. Rego-based mage segmeao 4

103 Eample Dffere shapes dsgushed by usg mome-based feaures. W. Kasprza: EIASR 6. Rego-based mage segmeao 5 Coour feaures The coour polygo ha appromaes a D shape s mached o a -D sgal. For eample he amplude values correspod o segme agle chages sampled a cummulave legh segme ed pos. The feaures correspod o DFT Fourer coeffces of he D sgal. W. Kasprza: EIASR 6. Rego-based mage segmeao 6

104 Dscree Dscree Fourer Fourer Trasform Trasform Le be: [... M- ] a dgal D sgal. The Dscree Fourer Trasform DFT of order M s: M M e F π M... I mar form: D F M 4 M M M M M M M m m m m m m m m m F F F F M L M M M M M L L L M m e M π W. Kasprza: EIASR 6. Rego-based mage segmeao 7 -D DFT D DFT For D mages he DFT s eeded o a D DFT: The D DFT ca be facorzed o D DFT: fy Fv Fuv M N y y N v M u uv e e y f F π π 8 fy Fv Fuv rows colums Ivarace of DFT: If a obec s shfed he mage doma he he Fourer doma he amplude of Fourer coeffces remas uchaged varace w.r.. o mage doma shf. W. Kasprza: EIASR 6. Rego-based mage segmeao

105 The Fourer-Mell rasform Roao ad scalg performed mage doma resul a shf he LogPolar doma of Fourer coeffces: - Orgal - Scaled - Scaled ad roaed - LogPolar polar θ F A e l F l A + θ θ s F s A e α θ + e s F s A e α l e s F l s + l A θ + α α W. Kasprza: EIASR 6. Rego-based mage segmeao 9 The Fourer-Mell rasform A secod Fourer rasform appled o he LogPolar rasformed Fourer coeffces leads o he varacy of ew coeffce s ampludes w.r.. scalg ad roao performed he mage doma. DFT[l F DFT[l e α ] [ B s F e ρ ] [ B ] e... M ρ + σ ]... M W. Kasprza: EIASR 6. Rego-based mage segmeao

106 EIASR Model-based obec recogo Lecure 7 Proec s co-faced by Europea Uo wh Europea Socal Fud. A sequece of paers Bayesa approach Opmum paer sequece search o fd he mamum value of a produc of pror probably dsrbuos: p Ω p C Ω p Ω C α p Ω p C Ω pc wh wo sochasc processes: N Ω Ω Ω... Ω - sequece of classes N C c c... c - sequece of observed feaures Le be he umber of classes ad - he legh of he feaure vecor. The geeral he Bayes classfer would requre: N pror probably deses N codoal probably deses of N - dmesoal sochasc vecors. Smplfcao a frs-order Marov process. W. Kasprza: EIASR 7. Model-based obec recogo

107 Frs-order Marov process Two plausble smplfcaos:. Idvdual observaos are sochascally depede N p C Ω p c Ω Ω Isead of N umber of N-dmesoal dsrbuos oly - dmesoal pdf-s are eeded.. The classfcao of observao depeds oly o s drec predecessor frs-order Marov process.e. 3 N N p Ω p Ω p Ω Ω p Ω Ω L p Ω Ω Isead of N pror probables s suffce o defe raso probables PΩ Ω ad probables PΩ. W. Kasprza: EIASR 7. Model-based obec recogo 3 Dyamc Programmg Dyamc Programmg DP s a opmzao approach for deecg a bes sequece of decsos assumg a frs-order Marov process. I operaes dually o a cos measure raher ha o probably. I s a smplfcao of he Bayesa approach as oly oe sochasc process s cosdered he decso sequece. Basc assumpo DP: every decso regardg a subsequece s made a opmal way.e. o bacracg. Ths pus codos oo he cos fuco: moooc cos s o decreasg w.r.. decso seps separable o prevous coss ad ew added coss. W. Kasprza: EIASR 7. Model-based obec recogo 4

108 Model problem graph The problem graph DP represes:.the raso coss dual o pω Ω -.A sgle observao s acceped gve ode of he problem graph ps Ω ad ps Ω for Eample: The decso space The soluo of DP search - a pah he decso space wh lowes cos. Decso space - a -D array deed by he decso sep ad a ode of he problem graph. W. Kasprza: EIASR 7. Model-based obec recogo 5 Decso coss A sgle decso: how o ee a search pah by a raso sep from curre o e ode. Cos fuco The pror cos fuco o observao oly raso coss: R m { R- + r } The poseror cos fuco a observao sequece s gve : R m { R- + r + rs } R-; - coss of sae -; ; r - raso coss of arc. rs coss of accepg he -h observao S ode of he problem graph. W. Kasprza: EIASR 7. Model-based obec recogo 6

109 Decso space Eample co. I he pror case o observao DP search fds he pah whch s of lowes possble cos 6 whereas oher complee pahs bewee ad 6 are of cos: 3 ad 4. I he poseror case f observao s #BEATA# he soluo s he same whle oher pahs are of fe coss. If observao s #B# he soluo s: W. Kasprza: EIASR 7. Model-based obec recogo 7 DP-based obec recogo Eample. Usg DP search provde a machg of he symbol cha observed sequece a d z e e wh he model sequece d z eń. Soluo. The odes he problem graph should represe possble decsos: accep gve leer replace a leer ser a leer delee a leer. All decsos rasos o e ode ecep of he delee a leer -odes move o e leer he observao sequece ad accep or o. The problem graph should allow a pah of decsos: ser a accep d accep z delee accep e replace ń by ser e. W. Kasprza: EIASR 7. Model-based obec recogo 8

110 . A srucure of paers Desg problems:.mae mage segmeao ge daa A;.Se he obec model M 3.Desg he machg mehod for segme o obec pars 4.Desg he udgeme rule for θ hese correspodeces. W. Kasprza: EIASR 7. Model-based obec recogo 9 Opmum obec recogo Obec deeco deerme he mage srucure correspodg o a obec gve by model Ω or M. Obec localsao space deerme he locao R space 3D or plae D 6 parameers 3 raslaos ad 3 roaos 3-D or 3 parameers raslaos ad roao. Opmum recogo sraeges. Sysemac search chec all saces: {M R } arg ma {M R; } θ[im R; A];. Bayes classfer: wh ow prors: pm R pa M R ; he Bayes classfcao rule s: {M R } arg ma {M R; } pm R ; A. where pm R ; A ~ pa M R ; pm R ; W. Kasprza: EIASR 7. Model-based obec recogo

111 Heursc sraeges The sysemac search ad Bayes approach o obec recogo are boh dffcul o mpleme due o a huge compley. The hypohesze-ad-es sraegy REPEAT Hypohess geerao Se al hypoheses obec saces by machg model pars wh segmes: I P M ; O A. For every I esmae s parameers: a' R ; Hypohess verfcao FOR every hypohess I DO Geerae a full ls of segme o-pars maches O A. IF he umber of maches for pars of model M s suffcely large THEN he sace I M O ess AND esmae s locao parameers a R ; AND elmae from A he segmes O. UNTIL more ha W% of segmes are sll avalable A. W. Kasprza: EIASR 7. Model-based obec recogo A* graph search Algorhm A* formed graph search allows a effce search. Nodes are seleced for epaso based o he cos fuco: f g + h where g real coss of he pah from sar ode o ode h epeced cos of remag pah from o a ermal ode. Due o h we call he search o be formed as allows a goaldrve seleco of he e ode. The heurscs h s admssble f s a opmsc esmao of real remag coss h* for every ode.e. : h h* I pracce h should be admssble ad cosse: for each pah from o should hold: h c + h where c s he real raso cos bewee ode ad. W. Kasprza: EIASR 7. Model-based obec recogo

112 A* graph search b W. Kasprza: EIASR 7. Model-based obec recogo 3 3. Geerc obec model A sace of a obec class s dsgushed by a parameer vecor also called arbues e.g. wh parameers for sze localsao shape. We call hs a geerc obec model. The obec sace space s hgher ha fed-sze obec case. Such obec recogo problem wll be epressed here erms of cremeal sae esmao: he uow sae parameers are esmaed from a ow observao sequece whle sasfyg some crero; orgag from esmao heory: ML mamum lelhood MAP mamum poseror LSE leas square error MMSE mmum mea square error. W. Kasprza: EIASR 7. Model-based obec recogo 4

113 Proeco codos for o-road 3-D vehcle obecs Eample A smple geerc vehcle model Obec sae: s s Localsao s Shape s Localsao z φ T s Shape Wdh Hegh Hegh Legh Legh T W. Kasprza: EIASR 7. Model-based obec recogo 5 Eample The model-o-mage proeco hs: Measureme z: mage segmeao ad fdg he bes mach bewee proeced model edges ad deeced le segmes z h s + w z cosss of le segmes: m { Z Z } z... my φ l W. Kasprza: EIASR 7. Model-based obec recogo Z 6

114 Ierave MAP esmao To fd a MAP esmae of he uow sae s gve observaos would requre a very comple pror probably dsrbuo pz z... z s ad a eesve search a hgh-dmesoal space. We solve appromaely by a grade desce erave mehod: sar wh al esmae of he sae ad covarace mar P. I every erao a ew esmae sˆ ad covarace mar P are geeraed ha are beer ha all prevous esmaes. The pror pdf-s: p s ˆ s ~ N ˆ s P p z s ~ N h s R MAP esmae: o mamze he poseror: p s z sˆ ˆ p z s p s s ~ ep Γ s C.e. o mmze: T T Γ s [ z h s] R W. Kasprza: EIASR 7. Model-based obec recogo ŝ [ z h s] + s sˆ P s sˆ 7 Ierave MAP esmao The obecve: Ierave MAP INIT: s P. REPEAT FOR N Γ s H s R [ h s z] + P ˆ s s wh H s h s. Perform mage segmeao gves he measureme z.. Esmae: s s - µ P Hs R - [ hs z ] P - µ P where µ < > - s a small forgeg coeffce. 3. Se for e me po smple predco: s + s P + P. W. Kasprza: EIASR 7. Model-based obec recogo 8

115 EIASR Speech sgal pre-processg processg Lecure 8 Proec s co-faced by Europea Uo wh Europea Socal Fud 9. Dgal audo Pulse code modulao PCM - he commo ype of dgal audo recordg: - Samplg rae s 44 samples ae every secod - Amplude ges covered o a 6-b eger: K. - Two chaels of sereo daa. A geeral mulmeda fle forma s RIFF Resource Ierchage Fle Forma where a fle cosss of daa chus defed by a predecessor ASCII ame of 4 byes. WAV s a sace of RIFF. W. Kasprza: EIASR 8. Speech sgal pre-processg 3

116 WAV W. Kasprza: EIASR 8. Speech sgal pre-processg 3. Boe cha Huma audory sysem. Audory erves 3. Ier ear cochlea 4. Eardrum 5. Ouer ear 6. Eusacha ube The cochlea s acouscally coupled o he eardrum by a seres of y boes a spral of ssue flled wh lqud ad housads of y hars. The hars o he ousde of he spral are loger ha he hars o he sde. The loger hars resoae wh lower frequecy souds ad he shorer hars wh hgher frequeces. W. Kasprza: EIASR 8. Speech sgal pre-processg 3

117 . Fourer Trasform The Fourer rasform FT famly:. Aperodc-Couous sgal Fourer Trasform A sgal eeds o boh posve ad egave fy whou repeag a perodc paer e.g. Gaussa curve.. Perodc-Couous Fourer Seres E.g. se waves square waves ad ay waveform ha repeas self a regular paer from egave o posve fy. 3. Aperodc-Dscree Dscree-me Fourer Trasform Sgals are defed a dscree pos ad do o repea hemselves a perodc fasho. 4. Perodc-Dscree Dscree Fourer Trasform Dscree sgals ha repea hemselves a perodc fasho. W. Kasprza: EIASR 8. Speech sgal pre-processg 33 Fourer Seres Fourer seres: he FT for perodc sgals couous me. I rasforms he sgal o a equvale summao of se ad cose waves wh frequeces ha are mulples of a base frequecy f /T where T s he perod of : a + a cos π f + b s π f The Euler formula I comple-umber form for sgals wh perod π : + πf c e π π a b c π a + b Le he sgal be some π perodc fuco. The: π π a cosπ f d b f d π sπ c π π π π π f oherwse e πf d W. Kasprza: EIASR 8. Speech sgal pre-processg 34

118 Dscree Fourer Trasform DFT For dscree-me sgals we eed o cosder a fe se of se ad cose waves oly as all dgal recordgs have a fe legh. The umber of DFT oupu frequeces s he same as he umber of pu sgal samples he me doma. We ca preed ha he fuco s perodc ad ha he perod T s he same as he me of recordg M samples. The base frequecy f for he M-sample DFT s: f s f where f s s he samplg rae. M T Defo. Le [... M- ] be comple-valued samples of he dscree sgal. The Dscree Fourer Trasform of order M s: M π M F e W. Kasprza: EIASR 8. Speech sgal pre-processg... M 35 DFT mar form: F F F M FM M DFT F D M m m m e M Comple- or real-valued pu m m 4 L L L m m M M I geeral he DFT covers a sampled comple-valued fuco of me o a sampled comple-valued fuco of frequecy. Usually we operae o real-valued pu fucos.e. all he magary pars of he pu sgal are assumed o be zero. W. Kasprza: EIASR 8. Speech sgal pre-processg m M m M M M M L M π M m M M 36

119 Iverse DFT The DFT mar D M s a symmerc mar. The verse mar ess whch s a comple cougae of D M : D * M D * M M M D L m m L m 4 m m L m M M M M M M M M m m L m M M m e M π Iverse DFT: M D * M F W. Kasprza: EIASR 8. Speech sgal pre-processg 37 The Nyqus frequecy The Nyqus frequecy The frequecy de M/ s a specal case: correspods o he Nyqus frequecy whch s always half he samplg rae ay dgal PCM recordg. For eample he Nyqus frequecy a ypcal CD audo recordg s: 44 Hz/ 5 Hz. The Nyqus frequecy s he hghes frequecy ha a PCM dgal audo recordg ca reproduce. Sgals above he Nyqus frequecy are flered ou before he sgals are dgally sampled order o avod alasg problems. W. Kasprza: EIASR 8. Speech sgal pre-processg 38

120 Fas Fourer Trasform FFT FFT Cooley ad Tuey proposed he FFT algorhm 965 The basc dea of he Fas Fourer Trasform s ha he DFT ca be covered o a se of DFT-s of sze. FFT assumes ha he umber of dscree samples s a power of wo.e. M. The DFT requres OM operaos whereas FFT s of compley OM log M. Two equvale FFT algorhms are dsgushed:. FFT wh decmao he me doma. FFT wh decmao he frequecy doma. W. Kasprza: EIASR 8. Speech sgal pre-processg 39 FFT wh decmao frequecy Processg flow for M8: W. Kasprza: EIASR 8. Speech sgal pre-processg 4

121 FFT FOR... - DO for every erao FOR u... - DO for every group FOR every par a b v... of group u DO X [+] [a] X [] [a] + m û X [] [b]; X [+] [b] X [] [a] - m û X [] [b]; FOR s... M- DO Reverse he b order of bary dgs: wh s s... s z s... s [ ] F s X z W. Kasprza: EIASR 8. Speech sgal pre-processg 4 3. Sgal flerg A dsored measureme of a source sgal s modeled as: f s g + where: f measured sgal s he deal o-dsored source g a uow dsoro covolued wh he source a uow addve ose. Frs we cocerae o addve ose reduco whe f s + Hgh-frequecy ose ca be cacelled by applyg a low-pass fler wh approprae cu-off frequecy. Gaussa ose ca be reduced by applyg a Gaussa smoohg fler. Srucured ose has o be defed frs. For eample by he mehod of specral subraco. W. Kasprza: EIASR 8. Speech sgal pre-processg 4

122 Nose reduco Nose reduco by specral subraco f he ose characerscs s uow bu ca be measured. Ieravely esmae he Fourer coeffces of he ose sgal frames whch obvously clude oly he ose sgal: Nˆ γ Nˆ + γ F m m m where he parameer γ. for frames whou speech ad γ. for frames wh speech. Subrac he esmaed ose eergy from he oal sgal eergy frames where we deec speech: F Nˆ f F Nˆ > F Sˆ m α m : m α m β m β Fm : oherwse where he parameersα.9 β.5. m W. Kasprza: EIASR 8. Speech sgal pre-processg 43 Pre-emphass fler The goal of "pre-emphass" s o sreghe he hgher frequeces s performed he me doma: The magude par of he frequecy characerscs: f ' f - ϕ f - where ϕ <.9. >. Eample: A specrogram before ad afer pre-emphass: W. Kasprza: EIASR 8. Speech sgal pre-processg 44

123 Auo-correlao of sgal For a wdow of sze N we compue ormalzed auo-correlao of sgal samples wh s shfed samples by. Sarg wh m-h sample he -shfed ormalzed auo-correlao m+ N s: r m f + m [ f ] [ f + ] f The mamum of ormalzed auo-correlao values always appears for ad s equal o. For a voced speech par local mama appear every samples: r r + ; e.g. for some 3 ec. because may srog harmoc frequeces es hs par. W. Kasprza: EIASR 8. Speech sgal pre-processg 45 Base frequecy By ormalzed muual correlao we shall mea a correlao facor bewee wo cosecuve sgal frames. Le he frames of sgal samples are gve vecor form: f m m [ f m... f ] m f m m + [ f m... f m + ] The ormalzed muual correlao of hese wo vecors s: f m m f m m + ρm f m m f m m + Sar wh ad compue he correlao facor for dffere values of. I a voced par of he speech he mamum of ormalzed muual correlao wll be a ha correspods o he base perod ad base frequecy F of he speaer. W. Kasprza: EIASR 8. Speech sgal pre-processg 46

124 EIASR Acousc feaure deeco Lecure 9 Proec s co-faced by Europea Uo wh Europea Socal Fud 47. Speech feaures Frame-based speech sgal feaures. Mel-frequecy cepsral coeffces MFCC eeded by her frs dervaves me. or. Speech feaures based o Lear Predcve Codg LPC e.g. LPCC lear predcve cepsral coeffces Speech sgal frames Feaures W. Kasprza: EIASR 9. Acousc feaure deeco 48

125 Cepsrum The cepsrum of a sgal [] s he resul of a homomorphc rasformao: cepsrum F - log F where F s he dscree-me Fourer Trasform DFT for MFCC or he Z rasform for LPCC.. Noe: specrum spec rum ceps rum cepsrum MFCC: LPCC: W. Kasprza: EIASR 9. Acousc feaure deeco 49 Cepsrum Why are cepsrum feaures useful for speech recogo? The cepsrum feaures characerzg he mpulse respose of he vocal rac are locaed ear he zero feaure ; whereas he pu mpulse compoes correspodg o he lary-modulaed oscllaos ha are o useful for speech recogo are locaed a hgher values of loger cepsrum me where he cepsrum feaures acheve a mamum value; The useful feaures ca be separaed from he ohers by selecg a approprae umber of hem sarg from or by addoal mulplcao called lferg. The speech mpulse respose ca also be separaed from he acquso chael s mcrophoe respose by usg ceered cepsrum feaures. W. Kasprza: EIASR 9. Acousc feaure deeco 5

126 . MFCC. Shor-me Fourer Trasform STFT A wdowed DFT for every frame τ of he pu sgal: M π M F τ [ τ + ] e w [ ]... M- Wdow fucos w[]. Recagular wdow. Tragle wdow 3. Hammg wdow τ ec. π cos wτ [ ] M for {... M oherwse } W. Kasprza: EIASR 9. Acousc feaure deeco 5 Wdowg eample Eample. [] s he sum of wo sus fucos uformly sampled from o π by 8 samples: [] sπ /5 + sπ / Sgle frame recagular wdow appled...7. Sgle frame Hammg wdow appled W. Kasprza: EIASR 9. Acousc feaure deeco 5

127 Wdowg eample Eample co. Magude of Fourer coeffces: Wh recagular wdow. Wh Hammg wdow W. Kasprza: EIASR 9. Acousc feaure deeco 53 Cocluso: Wdow fucos f w Mag[DFTw] Mag [STFTf w] W. Kasprza: EIASR 9. Acousc feaure deeco 54

128 Specrogram Power of Fourer coeffces squared magude M π M FC τ F τ [ τ + ] e w [ ] τ... M- W. Kasprza: EIASR 9. Acousc feaure deeco 55 Mel frequecy scale No-lear respose of he huma ear o he frequecy compoes he audo specrum: dffereces frequeces a he low ed < Hz are easer deecable ha dffereces of he same magude he hgh ed of he audble specrum. Approach: a o-lear frequecy aalyss performed by he huma ear - he hgher he frequecy he lower s resoluo MEL scale emprcal resul: f mel f 595 log + 7[ Hz] W. Kasprza: EIASR 9. Acousc feaure deeco 56

129 MFC Mel frequecy coeffces MFC Tragular flers are locaed uformly he Mel frequecy scale: M MFC l τ [ D l FC τ] l... L The MFC value assocaed wh each b correspods o a weghed average of he power specral values he parcular frequecy rage specfed by he shape of he fler. W. Kasprza: EIASR 9. Acousc feaure deeco 57 MFCC The Mel-frequecy cepsrum coeffces are compued by he homomorphc rasformao MFCCh FT - {log MFC{FT{h}}} for h w The las sep s he verse Fourer Trasform of logarhmc Mel frequecy coeffces: L l + π MFCC τ [log MFC l τ cos ]... K L Ceered MFCC MFCC ceered l τ MFCC τ mea{ MFCC τ τ...]... K W. Kasprza: EIASR 9. Acousc feaure deeco 58

130 Dela feaures Eergy feaure Addoal feaure - he oal eergy of sgal a sgle frame: E τ log Grades of feaures me dela feaures A schemac vew of specrograms for dffere phoeme ypes: sgle vowels lef dphhogs mddle plosves rgh. M A lear regresso 5 cosecuve frames s appled o fd dela coeffces d of MFCCs ad eergy feaure c : c τ + + c τ + c τ c τ τ d W. Kasprza: EIASR 9. Acousc feaure deeco 59 Feaure se Eergy MFCC Dela eergy Dela MFCC Geeral feaures per frame Toal eergy mea ad varace orm. ma. auo- correlao low-bad rao W. Kasprza: EIASR 9. Acousc feaure deeco 6

131 3. LPC ad LPCC The Z rasform s a dscree-me sgal rasform whch s dual o he Laplace rasform of couous-me sgals ha meas a probg of sgal by susods ad decayg epoeals: X z [ ] z ad z s a comple umber: z r e -ω r e -σ. The syhess model of huma speech z-doma cosss of: a ecao source Ez o he pu a lear fler wh rasmace Hz he speech sgal Xz o s oupu; he sgals ad he fler are represeed he comple-valued doma z. by her rasforms W. Kasprza: EIASR 9. Acousc feaure deeco 6 Speech syhess model Le us deoe by Hz he rasmace of he fler he z rasform of s frequecy respose h[]. There are obvous relaos he z doma: X z H z E z E z A z X z W. Kasprza: EIASR 9. Acousc feaure deeco 6

132 LPC A dgal IIR fler s characerzed by a recursve equao: ha gves he -h oupu sample o base of curre ad prevous pu samples ad prevous oupu samples. L +L ] [ ] [ ] [ ] [ ] [ ] [ a a e b b e e b m p 63 Afer algebrac rasformaos we ge a equvale descrpo he z-doma: The Auo-Regressve AR model assumes ha he umeraor s : W. Kasprza: EIASR 9. Acousc feaure deeco e b a ] [ ] [ ] [ m p z a z b z z X E m p z a z b z H m a z z H LPC Thus he AR model he -h oupu sample s esmaed oly o m prevous oupu samples ad curre pu sample as: I shor: + m a e +L + + ] [ ] [ ] [ ] [ a a e 64 Ideally for voced pars he vocal rac s cyclcally fed by a Drac dela mpulse. The: e e for shor-me frames. Thus he -h speech sample a frame s esmaed as a lear combao of he prevous m samples: W. Kasprza: EIASR 9. Acousc feaure deeco + a e m a ˆ

133 Auo-correlao mehod for LPC The as s o compue he parameers { a... m } for every sgal frame. By he LSE approach we have for gve frame: where are sample dces gve frame. ˆ ε a a ε 65 where are sample dces gve frame. We ge m equaos wh m uows: By roducg he frs m+ auo-correlao coeffces: he equao sysem aes he form: W. Kasprza: EIASR 9. Acousc feaure deeco...m a L + M r m r r a m... Compug LPC parameers m m m m m m m r r r a a a r r r r r r r r r r r r M M L M M L L 3 66 The mar R s a Toeplz mar s symmerc wh equal dagoal elemes. Due hs Toeplz propery a effce algorhm s avalable for compug a whou compug he verse mar R -. Alerave mehod The Levso-Durb algorhm s a erave mehod for he compuao of LPC parameers. W. Kasprza: EIASR 9. Acousc feaure deeco Ra r

134 The Levso-Durb algorhm E represes he predco error K he refleco coeffces bewee cosecuve pars of he acousc ube a he fal predco coeffces INIT: E r FOR o m : K r α r E FOR o m : FOR o - : K α α E K E a M α α K α W. Kasprza: EIASR 9. Acousc feaure deeco 67 Basc parameers LPC-based feaures The predco parameers ca self be cosdered as a feaure vecor usually he error value s also cosdered: c a for... m ; where a ; ad c m+ ε. The umber of predco parameers s: m. Appromaely for he samplg frequecy f s [Hz]: m f s +[ 4]. Specral feaures smoohed sgal specrum By rasformg he parameer vecor o frequecy doma we ge a smooh specrum of he sgal frame. The requred resoluo frequecy doma s acheved by paddg he parameer vecor wh zeros o ge a vecor wh M elemes: A M DFT [ a a... a m...] W. Kasprza: EIASR 9. Acousc feaure deeco 68

135 LPCC LPCC: cepsral LPC Recall he speech syhess fler fuco s rasformed o he z- doma as : H z m a z The polyomal he deomaor par ca be reorgazed gvg a all-pole rasmace fuco: H z m m a z p z The use he l-fuco ad apply he verse Z rasform. m c z [ ] {l[ H z ]} z { p z } W. Kasprza: EIASR 9. Acousc feaure deeco 69 LPCC A drec erave mehod for compug he LPCC feaures Isead of performg he parcular seps of he cepsrum rasformao of LPC coeffces here ess a erave mehod for a drec compuao of LPCC feaures from he LPC coeffces. For m where m s he order of LPC : For > m : c[ ] a c[ ] a ;... m c [ ] c[ ] a ; > m W. Kasprza: EIASR 9. Acousc feaure deeco 7

136 EIASR Phoec speech model Lecure Proec s co-faced by Europea Uo wh Europea Socal Fud 7. Phoec caegores Phoe arculaed soud - he smalles eleme speech. A phoe s a speech soud cosdered as a physcal eve whou regard o s place he soud sysem of a laguage Webser dcoary. A phoe may have dffere realsaos deermed by: oaly durao ad oao. The IPA I. Phoec Assocao has defed some phoes whle a leas 4 are requred for a rough rascrpo. Phoeme a caegory of smlar phoes. Eve hough o wo speech souds are decal all of he phoes classfed o oe phoeme caegory are smlar eough o have he same meag. W. Kasprza: EIASR. Phoec speech model 7

137 Groups of phoemes Prmary groups: vowels cosoas - a basc coras ca a phoeme serve as a syllable ucleus or o? The mode of phoao:. Sooras buzzes - characerzed maly by vocg - he repeve opeg ad closg of he vocal cords.. Frcaves hsses - geerally o-vocg souds. 3. Plosves pops - eplosve souds cludg affrcaes. 4. Slece - phrase marer breahg m-sleces ad closures before a plosve. W. Kasprza: EIASR. Phoec speech model 73 Vowels. Moophhogs phoemes - a sgle vowel qualy. sressed vowels: /A/ /E/ /@/ /:/ /o/ /U/ e.g. he words faher be ba bee above boo; reduced vowels: /^/ /&/ /I/ /u/; e.g. he words bus above b boo r-coloured vowels E: log /3r/ shor /&r/ as brd ad buer.. Dphhogs 6 - a clear chage from sar o ed: /e/ e.g. lae /ai/ e.g. bye />/ e.g. boy /U/ e.g. few /au/ e.g. loud /ou/ boa. W. Kasprza: EIASR. Phoec speech model 74

138 Cosoas 3. Appromas 4 Semvowels smlar o vowels bu more obsacles he vocal rac ha for he vowels: lquds: he /l/ e.g. "le" ad he rerofle /9r/ e.g. "red"; gldes : he // e.g. "yes" ad he /w/ e.g. "wo". 4. Nasals 3 - The arflow s bloced compleely he oral rac bu a lowerg of he velum allows a wea flow hrough he ose: /m/ as "me" // as "ew" /g/ as "sg". 5. Frcaves 9 - Wea or srog frco oses whe he arculaors are close ogeher o cause urbulece he arflow: voceless /f/ /T/ /s/. /S/ /h/ as : fe "hg" sg assure hope ; voced /v/ /D/ /z/ /Z/ as : voce ha resg vso. W. Kasprza: EIASR. Phoec speech model 75 Cosoas 6. Plosves 6 - Burss or eplosve souds produced by complee closure of he vocal rac followed by a rapd release of he closure: uvoced /p h / / h / / h / as ca ad voced: /b/ /d/ /g/. 7. Affrcaes - Plosves released wh frcao: he /S/ soud le "church" he /dz/ le "udge". The IPA Ieraoal Phoec Alphabe c/o Deparme of Lguscs Uversy of Vcora Vcora Brsh Columba Caada s recogsed as he eraoal sadard for he rascrpo of phoemes all laguages. W. Kasprza: EIASR. Phoec speech model 76

139 Arculao of vowels Arculao of vowels ad // accordg o: level of mouh opeg ad ogue locao. W. Kasprza: EIASR. Phoec speech model 77 Arculao of cosoas Classfcao of cosoas accordg o phoec caegores ad arculao area: W. Kasprza: EIASR. Phoec speech model 78

140 . Specral cues Formas eergy coceraos frequecy bads F-F5 Basc frequecy F he pch Usually: 8- Hz ma 5-35 Hz wome F ca vary from 3 Hz o Hz The lower s he closer he ogue s o he roof of he mouh. /:/ has he lowes F: 3 Hz; /A/ has he hghes F 95 Hz. F ca vary from 85 Hz o 5 Hz The F value s proporoal o he fro or bac poso of he ogue p. I addo lp roudg causes a lower F ha he case wh urouded lps. /:/ has a F of Hz he hghes F of ay vowel; /u/ has a F of 85 Hz - he ogue p s far bac ad he lps are rouded. W. Kasprza: EIASR. Phoec speech model 79 Formas Dsrbuo of ma formas vowels: W. Kasprza: EIASR. Phoec speech model 8

141 Typcal specrograms. Moophhog vowels a srog vocg wh sable formas. Dphhog vowels A srog vocg bu movg formas 3. Appromas Voced bu have less vsble formas W. Kasprza: EIASR. Phoec speech model 8 4. Nasals Typcal specrograms low eergy ad characersc "zero" rego : 5. Frcaves have a hgh-frequecy Gaussa rego : 6. Plosves A eplosve burs of acousc eergy followg a shor perod of slece : 7. Affrcaes A plosve followed by a frcave: W. Kasprza: EIASR. Phoec speech model 8

142 3. Sub-phoemes Proucao dffereces co-arculao effecs: phoes have a grea fluece o eghbour phoes. Tr-phoe model: o spl each phoe o oe wo or hree pars depedg o he ypcal durao of ha phoe as well as how much ha phoe wll be flueced by surroudg phoes. Eample. Tr-phoes ha represe he phoeme /E/ "yes":.$fro<e /E/ he coe of a precedg fro vowel.<e> /E/ he mddle wh o coeual effecs 3.E>$frc /E/ he coe of a followg frcave. I geeral /E/ ca be decomposed o 7 r-phoes: 8 for possble lef coe 8 for possble rgh coe ad coe-depede cere phoe. W. Kasprza: EIASR. Phoec speech model 83 Coe caegores.fro - fro vowels ad smlar appromas..md - ceral vowels ad smlar appromas. 3.Bac - bac vowels ad smlar appromas. 4.Sl - slece. 5.Nasal - asals. 6.Rero - rerofle approma ad rerofle r-coloured vowels. 7.Frc - frcaves ad affrcaes. 8.Oher - plosves ad remag appromas. W. Kasprza: EIASR. Phoec speech model 84

143 Coe caegores W. Kasprza: EIASR. Phoec speech model 85 EIASR Word ad seece recogo Lecure Proec s co-faced by Europea Uo wh Europea Socal Fud 86

144 . Speech varably. Acousc-phoec varably covers dffere acces proucaos pches volumes ad so o.. Temporal varably covers dffere speag raes. 3. Syacc varably of seeces A useful smplfcao s o rea hem depedely. The emporal varably s easer o hadle - a effce algorhm s ow as Dyamc Tme Warpg. Acousc-phoec varably s more dffcul o model covers acousc feaure schemes ad phoec modelg. Syacc varably deals wh dffere word sequeces represeg he same seece meag. W. Kasprza: EIASR. Word ad seece recogo 87 Dyamc Tme Warpg Referece paers: Y ;... ; Y l - prooypes X - pu paer. Decso rule s a eares eghbour rule: l arg m D X Y l The dsace DX; Y l bewee he pu ad a referece paer are defed despe dffere durao of paers. The dsace DX; Y for a sequece of segme pars s defed as he sum of local dsaces d d ; y bewee correspodg segmes alog he soluo pah: D m{ D D D l } + d ; ; ; > > oherwse W. Kasprza: EIASR. Word ad seece recogo 88

145 DTW - Eample Machg a 3-segme pu paer 3 wh a 4- eleme model paer y y y 3 y 4. W. Kasprza: EIASR. Word ad seece recogo 89 Word recogo. Speech recogo I word recogo he queso s: wha s he mos lely word represeed by gve acousc sequece? A word cosss of a sequece of phoemes. From he Bayes rule we ge: W arg word ma P Pword sgal Pword sgalα Psgal word Pword Two pror sochasc models are requred:.he acousc-phoec model Psgal word ad.he laguage model Pword. I parcular: he sgal s represeed by feaure vecors: Pword c c...c α Pc c...c word Pword W. Kasprza: EIASR. Word ad seece recogo 9

146 Seece recogo I seece recogo he queso s: wha s he mos lely word seece represeed by gve sequece of words? A seece coss of words gve our leco. S arg sequece ma P Pseece words Seece s a meagful sequece of words sgal s a sequece of observed words. From he Bayes rule we ge: Pseece words α Pwords seece Pseece Two pror sochasc models are eeded:.he applcao model Pwords seece ad.he laguage model Pseece. W. Kasprza: EIASR. Word ad seece recogo 9 Hdde Marow Model A HMM s defed as a 5-uple: HMM S Y Π A B where S {S S... S N } - se of saes Y - se of oupu symbols Π - he sar sae probably vecor A [a ]- he sae raso probably mar B [b ]- he oupu probably mar. W. Kasprza: EIASR. Word ad seece recogo 9

147 HMM word recogo Oupu probably model. Dscree scalar - a sgle oupu symbol s observed oe sae: P O Y S b ;... N;... M ;... T. Dscree vecor a vecor of symbols per sae s observed: 3. Sem-couous dsrbuo a mure of Gaussa dsrbuos per class dsrbuos 4. Couous mure of dffere dsrbuos per sae ad M M class: P O c S b p c Ω b Ν c; µ Σ M P O c S b P c M P O c S b p c Ω b Ν c; µ M Σ W. Kasprza: EIASR. Word ad seece recogo 93 HMM word recogo Srucure: a lef-o-rgh HMM Acos: INSer SUBsue DELee W. Kasprza: EIASR. Word ad seece recogo 94

148 Verb search Verb search - he goal s o fd he bes pah of saes he HMM w.r.. he observed sequece: S... S T argmap S T S... S c c Ths s doe eravely from o T whle eedg every paral pah S S...S - he HMM model a bes way wh respec o he observed sequece of feaure vecors c c...c accordg o he dyamc programmg prcple: LS I pracce a logarhmc scale ca be used: T Lc P S LS S c c Lc P c S ma P S S P S LS clc S lp S LS S c c Lc lp c S + ma lp S S + lp S LS clc S T W. Kasprza: EIASR. Word ad seece recogo 95 Verb search The search space Verb search cosse wh he dyamc programmmg search - s a -D space deed by me observed sequece ad HMM sae dces. Recall ha a dscree HMM: a P S S ; b P Y S W. Kasprza: EIASR. Word ad seece recogo 96

149 Verb search 3 W. Kasprza: EIASR. Word ad seece recogo 97 Baum-Welch rag The parameers of a dscree HMM model λ Π A B ca be esmaed by he Baum-Welch rag a EM-le approach ha uses he forward-bacward algorhm o oba curre forward ad bacward messages. Gve a dscree HMM model ad a observao sequece OO O T he Baum-Welch rag maes a MLesmao of he parameers λ. Ρ HMM λ log P Ο λ log P Ο q λ q S T where q a sae sequece of legh T correspodg o observao O. The goal fuco: ˆ λ arg ma λ Ρ HMM W. Kasprza: EIASR. Word ad seece recogo λ 98

150 Forward-bacward algorhm Le N be he umber of saes. The forward erm - he probably of geerag a paral sequece ad edg up sae wh de a me : λ s q P Ο Ο L α 99 The bacward erm - he probably of geerag he remader of he sequece from sae wh de a me : Thus he probably of vsg he sae wh de a frame for a complee observao sequece O s he produc of above probables: W. Kasprza: EIASR. Word ad seece recogo λ T s q P Ο Ο + L β λ Ο s q P β α N a O b α α O b a N + + β β Baum-Welch algorhm REPEAT Wh curre λ {π a b...n;...m} ad T; ge by he forward-bacward algorhm. The E-sep The epeced probably of raso s s a me gve observao sequece OO O s: β α 3 me gve observao sequece OO O T s: γ - he epeced probably ha he raso from sae o sae appears a ay me from o T: W. Kasprza: EIASR. Word ad seece recogo λ Ο s q s q P + ξ Ο N b a P s q s q P β α β α λ Ο λ Ο ξ γ

151 Baum-Welch γ - he epeced probably ha sae s vsed a ay me from o T: The epeced probably ha sae ems he symbol class Y : Ο Y Y : γ ξ γ ξ 3 The epeced probably of vsg sae S a parcular me s: 3 The M-sep I hs sep we ca re-esmae he HMM parameer: vecor π marces A ad B by ag smple raos bewee above erms. W. Kasprza: EIASR. Word ad seece recogo Ο Y : N N S q P ξ β α β α γ λ Ο Baum-Welch 3 The updae rules for he parameers of a dscree HMM model: N N ˆ ξ β α β α γ π + Ο + T T b a β α ξ 3 Where χa Y f he formula braces s sasfed or χ. oherwse. W. Kasprza: EIASR. Word ad seece recogo + + Ο T T b a a ˆ β α β α γ ξ γ γ Ο Ο T T T T Y Y Y b ˆ β α χ β α γ χ γ γ γ

152 3. Seece recogo Two recogo seps a may-word recogo s performed frs usg a -layer HMM model ad alerave word hypoheses are geeraed; he a seece recogo s performed wh a addoal layer of HMM ha sees for a syaccally proper ad semacally meagful word sequece. W. Kasprza: EIASR. Word ad seece recogo 33 Word lace The word recogo sage geeraes - - or 3-syllablebased parally alerave word hypoheses. Eample syllable W. Kasprza: EIASR. Word ad seece recogo 34

153 N-grams The laguage model gves pror probably of gve word sequece Pwords. For words s... s we have: I pracce eeds o be of lmed legh. s s s P s s s P s s P s P s s P L L 35 The bgram model he probably of e word depeds oly o he drec predecessor word: The dsrbuo Ps s - ca be leared by a coug he relave frequecy of he pars of words a large learg se. N-gram f he N-h word depeds o N- predecessor words. W. Kasprza: EIASR. Word ad seece recogo... s s P s s s P s s P s s P s s P s P s s P... L Smoohg a N-gram Kaz smoohg for a hree-gram model: < > * r w w C w w w C d r w w C w w w C w w w P r α 36 - Dscou rae - Good Turg esmae - Number of samples: r Cw - w W. Kasprza: EIASR. Word ad seece recogo r w w P w w α } { for * r r r d r K r r r r r * + +

154 Toe passg search Toe passg search a radoal ame for geerao of hypohec word sequeces speech recogo ha cludes he call of Verb search for dvdual word recogo. Here we perform frs he mulple word recogo process whch geeraes he Word lace by repeve calls o Verb search for e - - or 3-syllable se of frames. The a search for prospecve pahs he Word lace s separaely performed. Orgally oes have mared he curre eds of pah for pah eeso wh o ceral search ree maageme. Here: A uform-cos search The N-gram model s appled for eveual pah prug All full pahs are sored for seece recogo. W. Kasprza: EIASR. Word ad seece recogo 37 HMM for seece recogo A HMM for seece modellg represeg a sochasc sya ad a meag semacs:. predefed word meags semac caegores. syacc word roles assged o caegores Ad. Every seece s combed from pars coag aomc semac formao. For eample a aomc par may be: a queso form whe where a wha me a me perod a egh a.m. aferoo a eveg desao Warszawa ec. Ad. Syacc roles a seece: subec predcae obec; A sya caegory: ou verb couco adecve umber ec. W. Kasprza: EIASR. Word ad seece recogo 38

155 Eample: HMM of seece W. Kasprza: EIASR. Word ad seece recogo 39 Eample co. The word dcoary coas 38 words base form le: from o ra hour mue day whe Moday oday ec. eeded by he ames of ra desaos. Some words may have dffere grammar forms bu hs s already hadled he word recogo sage - covered o he base form. The saes of HMM represe followg meag caegores: queso arbue deparure form day day-me ra arbue ra from o deparure cy desao cy ed of seece. Every sae ca em several words wh specfc probably. The o-zero sae rasos allow o accep dffere sya sequeces havg he same meag. W. Kasprza: EIASR. Word ad seece recogo 3