Statistical significance p-value. (also important for combining or comparing motifs)

Transcription

1 Statistical significance p-value (als imprtant fr cmbining r cmparing mtifs)

2 This image cannt currently be displayed. Wrd cunting statistics Statistical measures fr significance: O E Z-scre: Z = Var p-value: prb( O k, N) p-value is difficult t calculate, cmmn mistake in many publicatins assumes Var = E

3 Bernulli text (i.i.d. case, E=N/a w ) Var( W ) = Nσ (a:alphabet size; w: wrd size, N: text size) 2 = N a w (2K WW (1/ a) 1 2w a 1 ) w where Crrelatin plynmial K AB ( t) = c c1t c w 1 t w 1 is related t the crrelatin (Cnway s leading number ) AB ( c0, c1,..., c 1) = w Var( AA) Var( AT ) = Var( AAA) Var( ATG) = Ignring the verlapping wrd paradx leads t 200% mistake! Var( RR) Var( RY ) = 5 1 Fr 2-letter alphabet, the verlapping wrd paradx leads t 500% mistake!

4 Bernulli text (general, E=NP W ) Var( W ) = Nσ 2 = NP W (2C WW (1) 1 (2w 1) P W ) where Crrelatin plynmial C AB ω ( z) = P( ω) z i. i. d. K AB ( z / a) ω H AB P W = w i= 1 p w i H AB crrelatin set: (H WW aut-crrelatin set) A ω B Example: W=GCTGGTGG (the Chi mtif in E. cli). The crrelatin set is H W,W ={ε, CTGGTGG}. C WW (z)=1+p CTGGTGG z 7.

5 Exact p-value fr wrd cunt Exact Var: M00 (i.i.d.): Pevzner, Brdwsky & Mirnv 1989; Mk: Kleffe & Bradvsky Extend t a set f IUPAC wrds and duble-strands: Sinha When E=NP W <<1, Z-scres are nt accurate Exact wrd cunt distributin: Guibase & Odlysk Exact waiting time (First ccurrence, between tw successive ccurrences) distributin: M0: Blm 1982; M1: Rbin & Daudin 1999) Duality relatin: Wrd cunts and waiting times (Reinert, Schbath and Waterman, 2000).

6 Shrt sequences The p-value is the prbability that r sequences ut f n cntain W: pval( W ) n r P O O r n r = ( ( 1) (1 ( 1)) P Sig = lg 10 [ P( O r)* D] (van Helden et al. 1998) D is the number f pssible wrd cunt depending n a single r n bth strands. Mtif O Binm LD Ex TGATGA GATGAT ATGATG GATGAG Upstream 800bp f YGR022C

7 Staden89 (i.i.d mdel) If V takes values in 0,1, T with prbabilities g 0 g 1 g T, the generating functin is G(x)=g 0 +g 1 x+g 2 x 2 + +g T x T. Prb(V=n) = g n. If V 1 V 2 V m, are independent, the generating functin f the sum F(x) = G 1 (x)g 2 (x) G m (x). The generating functin f an integer PWM {0 w bj T}: F(x) = Π G j (x) where the generating functin fr clumn j, G j (x) = Σ f b x^w bj

8 Wrked example (Staden) A C G T There are nly three scre pssible fr j=1, namely 9,1 and 0, with prbabilities f A, f G and f C +f G respectively: g 0 = f C +f G, g 1 =f G, g 9 =f A. G 1 (x) = (f C +f T ) x 0 + f G x 1 + f A x 9 G 2 (x) = (f C +f T ) x 0 + f A x 1 + f G x 9 G 3 (x) = (f A +f G +f T ) x 1 + f C x 7. Frm first tw clumns, prbabilities Scre = 0 : f C f C + f C f T + f T f C + f T f T Scre = 1 : f C f A + f T f A + f G f C + f G f T Scre = 2 : f G f A Scre = 9 : f A f C + f A f T + f C f G + f T f G Scre =10: f A f A + f G f G Scre =18: f A f G Stepping thrugh the Algrithm (plynmial multiplicatin) fr the first tw clumns: i j i+j plya(i) plyb(j) plyc(i+j) Sum f C +f T f C +f T (f C +f T )(f C +f T ) S f C +f T f A (f C +f T )f A f C +f T f G (f C +f T )f G f G f C +f T (f C +f T )f A +f G (f C +f T ) S f G f A f G f A S f G f G f G f G f A f C +f T (f C +f T )f A +f G (f C +f T ) S f A f A f G f G + f A f A S f A f G f A f G S18

9 Expected number (Staden) AND : Suppse 3 mtifs A, B, and C with prbabilities P A, P B and P C, and ranges R A, R B and R C. E A =P A R A, E(ABC)=P A P B P C R A R B R C =Π P m R m. OR : E = Σ P m R m. NOT : E = ( 1 P m ) ^ R m. Genral patterns use apprpriate cmbinatins. (Simulatin: A=33%,C=17%,G=17%,T=33%, L=100,000) Mtif Class Mtif Scre Expected Observed Exact match AAAATTTT Exact match ATATATAT Exact match CGCG Exact match GGCC Percentage match AAAATTTT Percentage match ATATATAT Match t given sequence using Matrix, cutff TGACACGT Matrix, cutff Heatshck Matrix, cutff Sp Matrix, cutff TGGCA

10 Other frmulatins Claverie1994. The expected best matching scre fr lng sequence fllws the extreme value distributin. Huang et al 2004 extensin t lcal Markv mdel. Hertzberg et al 2005 algrithm extensin f Staden t nn-i.i.d using Branch and Bund. CONSENSUS (Hertz&Strm1999): I seq =ΣΣ f bj ln (f bj / f b0 ), largedeviatin apprximatin f multinmial distributin. MEME (Bailey&Gribskv1998): Mixture, MLM. Apprximate the distributin f the prduct f mtif scre p-values by the distributin f the prduct f the p-values, as tw independent, unifrm variables. Gibbs sampler (Liu, Neuwald&Lawrence1999) Bayesian, Maximum A Psteriri (MAP); r Frequentist null hypthesis Mnte Carl simulatin.

11 Cmparisn & assessment

12 Mtif Discvery Assessment Tmpa et al (2005)

13 Crrelatin cefficient (ncc) fr all pairs f tls

14 Classificatin & Discriminant mtifs

15 DWE: Discriminate Wrd Enumeratr (Sumazin et al. Biinfrmatics, 2004) Fregrund: n 1 ut f N 1 Backgrund: n 2 ut N = n n N N n N n N p One can use degenerate/cnsensus wrds (TATART) r spaced wrds (CCGNNCGG) Fisher s exact test = / / lg N n N n L

16 DME: Discriminate Matrix Enumeratr (Smith et al. PNAS, 2005) Mtif-Centric Detectin: DME enumerates mtifs t detect ccurrences in a predefined discrete space, and then uses lcal search t refine tp mtifs Benefits f DME s mtif-centric apprach: Guaranteed accuracy (bunded errr) Imprved mtif evaluatin speed Natural integratin f backgrund sets fr discriminant mtif discvery prblems

17 Enumerative apprach Test mtifs that unifrmly cver space f matrices: Strategy f enumerating matrices frm a predefined set des nt restrict the type f bjective functin we can use. A predefined set f matrices means mre can be tested since less time is required t generate them. By using specific sets f clumn types we can define a set f matrices s that at least ne matrix frm the set shuld be clse t the ptimal matrix.

18 Matrix discretizatin Idea: Select a set f clumn types, and generate all pssible matrices built frm clumns in the set. Example set f clumn types: Example matrices built frm thse clumns:

19 The DME algrithm Enumerate matrices (with minimum infrmatin cntent) built frm clumns in the selected set f clumn types. Scre each candidate accrding t the lglikelihd rati (bjective functin). Run lcal imprvement (Grid-refinement) n the best candidate. It is designed t ptimize ur bjective functin starting frm a specific matrix. Output the resulting mtif. Erase ccurrences f that mtif frm the sequence data. Repeat the prcedure t identify next discriminant mtif.

20 Regressin

21 Regressin Apprach t Mtif Discvery (MARSMtif) (Das et al. PNAS, 2004) Assumptins: Crrect parameters: binding affinities & prtein (TF) cncentratins E g = f(b 1,B 2, ) Binding affinities are unknwn Use mtif cunt r weight matrix mdel scres instead E g f (C 1,C 2, ) r E g f (W 1,W 2, )

22 REDUCE * Regulatry element detectin using crrelatin with expressin (ther example: Regressr by Liu et al. 2003) Multivariate Linear Regressin lg(e g /E gc ) = C + Σ μ F μ n μ g n μ g = n f mtif μ in the regulatry regin f gene g y N bsvns (y i,n i ) Fit Eqn: Y = a + b n Minimize: Σ i (y i Y i ) 2 n Several knwn mtifs crrectly predicted Explains nly ~10% f the data (yeast cell cycle; nise 50%) Cperative interactins nt accunted fr Nn-linearities in the data * Bussemaker et. al. Nat. Gen. (2001)

23 MARS: Multivariate Adaptive Regressin Splines (J. Friedman, Annals f Stat. 1991) Multivariate extensin f ne dimensinal splines (x-ξ) + Linear splines are made f piecewise linear functins (ξ-x) + ξ ξ knt x x Basis functins [h 1 (x)]: (x-ξ) + = max(0,x-ξ) = x-ξ, x > ξ = 0, therwise (ξ-x) + = max(0, ξ-x) = 0, x > ξ = ξ-x, therwise lg(e g /E gc ) = β 0 + Σ m=1m β m h m ({n μg })

24 MARS: Mdel building strategy Start frm a cnstant term Stepwise Frward Additin: add basis functins and their prducts Keep terms that minimize residual sum f squares (RSS) Backward Eliminatin: Prune the mdel t avid ver fitting Delete term(s) frm the mdel that causes least increase in RSS Minimize the Generalized Crss Validatin scre t btain the final mdel N GCV ( λ) = = g 1 [ lg( E / E ) f ({ n })] µ g gc λ ( 1 M ( λ) / N ) 2 g 2 M(λ) = effective n f parameters = r + ck r = # linearly independent basis functins K = # f knts; c = cnstant btained by CV

25 Extensive simulatins F g = A 0 + Σ μ A μ n g μ + Σ μ<α B μα n g μ n g α + s*є(g), F g = A 0 + s*є(g), (fregrund genes) (backgrund genes) є ~ N(0,1) {A}, {B} fixed fr each run 0 n μ g 3 (fg) #fg = 1000 n μ g = 0 (bg) #bg = 4000 Effects t nnlinear, int, nise, backgrund; Rbustness. Mars with interactins cnsistently had much better accuracy than the linear r nn-interacting case Crrect predictin except fr the nise Resistant t ver-fitting (when setting int=3 r reducing data) Initial test using 77 experiments,~100 REDUCE mtifs, reductin in variance ~10% -> ~30% in average (3 fld imprvement! The best case ~ 51%, clse t maximum pssible)

26 Set f candidate mtifs Check fr assciatin w/ expressin (Klmgrv Smirnv Test) Pairs f mtifs frm tp 100 mtifs Check fr assciatin w/ expressin (Klmgrv Smirnv Test) Tp 100 significant mtifs Tp 200 significant mtif pairs Run MARS with int=1 (n interactins) Significant mtifs MARSMtif FLOW CHART Run MARS with int > 1 (WITH interactins) Significant mtif and mtif pairs Run MARS with int > 1 (WITH interactins) Significant mtif and mtif pairs

27 Summary f MARSMtif results REDUCE~10% Algrithm Data Set Mtif Discvery methd Average Reductin in Variance %Cases that have imprvement with interactins ver int=1 Average imprvement fr cases in previus clumn ver int=1 MARS MARS Mtif cunts frm Pilpel et. al. Gibbs Sampling 24.7% 77.8% (14/18) 58.7% MARS Mtif scres frm Pilpel et. al. Gibbs Sampling 23.2% 88.9% (16/18) 68.4% MARS MARS MARS 5-7 mer nucletides Mtif cunts frm Kellis et. al. cunts f 5-7 mers clustered using mtifs frm Kellis et. al. Wrd Cunt 39.4% 27.8% (5/18) 64.4% Crss species cnservatin Wrd cunt + Crss species cnservatin 18.0% 77.8% (14/18) 54.0% (median) 31.7% 72.2% (13/18) 68.8% Identify which interacting pairs are imprtant at what time r under what experimental cnditins!

28 %Reductin in Variance fr MCM1-SFF (Linear) %Reductin in Variance Time %Reductin in Variance fr MCM1-SFF (MARS) %Reductin in Variance Time

29 Large-scale systematic identificatin f tissue-specific cis-regulatry mdules Tissue-specific expressin data Knwn TF lcalizatin data DME FG(+), FG( ) DME FG(+), FG( ) Tissue-specific mtif and mtif pairs MARSMtif Regressin n expressin data Ptential functinal tissue-specific mtif and mtif pairs