Comparative Analysis of Bradley-Terry and Thurstone-Mosteller Paired Comparison Models for Image Quality Assessment

Transcription

1 Coparatve Analyss of Bradley-Terry and Thurstone-Mosteller Pared Coparson Models for Iage Qualty Assessent John C. Handley Xerox Corporaton Dgtal Iagng Technology Center 8 Phllps Road, MS 85E Webster, NY 458 USA Jhandley@crt.xerox.co Abstract In age qualty assessent, preference for varous age processng algorths or treatents s often deterned usng pared coparsons. In ths experental desgn, pars of ages processed by dfferent algorths or treatents are presented to a udge. The preferred treatent s selected and a tally s kept of the nuber of tes each treatent s preferred to another. It s possble to estate an nterval scale for treatents n a hypothetcal psychologcal space usng ths ethod. There are two donate pared coparson statstcal odels: Thurstone-Mosteller Case V (TM) (correspondng to Thurstone s Law of Coparatve Judgent, Case V) and Bradley-Terry (BT). Although TM s used alost exclusvely n the agng lterature, the BT forulaton s ore atheatcally developed. Owng to ts parsony, t provdes tractable axu-lkelhood estators for scales, sultaneous confdence ntervals and hypothess tests for odel ft, unforty, and dfferences aong populatons of udges. In practce, TM and BT yeld nearly dentcal scale estates for coplete data. In soe experental desgns, any treatents are copared. Owng to the large nuber of possble treatent pars, not every coparson s ade, leadng to an ncoplete atrx of preference counts. Unlke TM, BT odel apples drectly to ncoplete data under ld restrctons We copare and crtque TM and BT odels. Statstcal analyses, any not avalable under TM, are deonstrated. An arguent s ade that BT offers overwhelng advantages to the agng county and should be used nstead of TM. Introducton Ths paper copares two well-known pared coparson odels: the Thurstone-Mosteller (TM) odel (by whch we ean Thurstone s Law of Coparatve Judgent, Case V) and the Bradley-Terry (BT) odel. (Mosteller s nae s ncluded n TM due to hs work on the statstcal analyss of Thurstone s odel). We argue here that BT odel should be used n place of TM because presently the forer s ore developed atheatcally than the latter. In partcular, easy forulas exst for axu lkelhood estates (le) of scale paraeters. The asyptotc theory of le s yelds estators for confdence regons and test statstcs based on lkelhood ratos for hypothess testng. TM s prvleged wthn the agng county ostensbly owng to ts orgns n psychophyscs. Yet t s unversally acknowledged that TM and BT yeld slar scale estates. The theory (and software) for generalzed lnear odels can produce le s yet BT, wth ts roots n experental desgn and consuer choce odelng, offers nuercally easer statstcal procedures. We present no new research although we do show an alternatve analyss to prevously publshed data. Our ntent s to provde the agng county wth a general context for pared coparsons, copare and contrast the two odels, and deonstrate the advantages of BT. The Lnear Model TM and BT odels are both lnear odels of pared coparsons. In such odels, probabltes of preference can be apped to scales. Forally (followng Davd, 988 [4]), let V and V represent erts of obects A and A,

2 respectvely. In a psychophyscs settng, the V ght represent sensaton agntudes on a scale. We represent the observed ert of obect A by rando varable X owng to observaton-to-observaton varaton. A lnear odel takes the for PX ( > X) π = HV ( V) () where H s a onotonc, ncreasng functon such that H ( ) =, H ( + ) =, and H ( x) = H( x). There are obvously an nfnte nuber of choces for functon H, the two of concern here are the Thurstone-Mosteller odel where H s the noral cuulatve dstrbuton functon wth zero ean and the Bradley-Terry odel where H( x) = [ + tanh( x/) ] () The task s to produce estates v of V, =,,. If the functon H has addtonal paraeters, we need to estate those as well. Assue wthout loss of generalty V = and defne δ = V V. Estaton proceeds = by tallyng α, the nuber of tes obect A s preferred to obect A after n coparsons. A saple estate of π s p = α / n. We defne H ( d ) = p and copute ert or scale estates v by d = v v,,, =,,. It can be shown that a least squares estate of V s v = d (3) Ths estate holds regardless of H and s the usual ethod for Thurstone s Case V odel. Assue that each par s observed a fxed (but possbly unequal) nuber of tes. That s, the sus n are fxed and the talles α are bnoal rando varables: n α n α P( α ) = ( ),,,, n π π α = (4) α Owng to ndependence, the lkelhood functon s L( α) = P( α ) < n α = HV ( V) HV ( V) < α [ ] n α (5) where α = [ ], the atrx of preference counts. α The Thurstone-Mosteller Model The ost general Thurstonan odel on stul posts a ultvarate dstrbuton on ( X,, X ). In pared coparsons, one observes ncoplete rankngs where stul are presented two at a te. Par-wse choce probabltes take the for PX ( > X) = πσ ( + σ σ) / ( µ µ )/( σ + σ σ ) exp( / ) y For a scalng nterpretaton, eans are consdered ordered along a contnuu n a psychologcal space. As dscussed elsewhere (e.g., Engledru [5] or Torgerson [9]), the full-blown Thurstone odel has too any paraeters (eans, varances, and covarances), so splfyng assuptons are appled. Perhaps the ost-used odel n pared coparsons n Thurstone s Case V, where X 's are assued ndependent and dentcally dstrbuted save for locaton paraeters µ, =,, ( µ = v, =,, n the lnear odel dscusson): PX X y dy (7) ( > ) = exp( /) π ( µ µ ) In ths case, one usually coputes least squares estates µ usng Eq. 3. Inferences regardng ˆ µ ˆ = ( ˆ µ,, ˆ µ ) are dffcult to obtan owng to ts unknown (asyptotc) dstrbuton. A lkelhood functon based on coparsons atrxα s L( α ; µ ) = P( α ) < = Φ Φ < α dy n α n α [ ( µ µ )] [ ( µ µ ) ] The log of ths lkelhood functon can be optzed nuercally. The Bradley-Terry Model One can rewrte Eq. as log( π /( π )) = V V, (9) (6) (8)

3 That s, the scale or ert dfferences obey a logstc odel (nstead of a probt odel n the Thurstonan case). Ths odel can be splfed to - paraeters by π π =,, π + π () where F and π = so that Eq. 9 takes the for = logπ logπ = V V. Ths s the Bradley-Terry odel of pared coparsons. One can wrte the odel n a for slar to Eq. 7: z (logf log F ) P( X X ) sech ( y / ) dy, () 4 and V = logπ provde scale paraeters. Owng to Eq., the lkelhood functon, Eq. 5, has a sple for n ters of π = ( π,, π ) and can be solved teratvely: p = a n ( p + p ) () where a = α, the total nuber of coparsons < preferrng A. A suffcent condton for a axu lkelhood s that each partton of the obects nto two nonepty subsets such that soe obect n the second set has been preferred to at least once to soe obect n the frst set [6]. Davd (988) ponts out that f ths condton s volated, t eans one of two thngs: ) there exsts subsets S and T of obects such that no obect n S s copared to obect n T; or, ) there exsts subsets S and T such that every coparson of obects between the favors obects n S [4]. These condtons can often be detected by nspectng the coparsons atrx α. BT (essentally Eq. ) can be developed nto a general dstance odel on ranked data. Mallows [7] nvoked the socalled Babngton-Sth transtvty odel (whch allows only pared coparsons that produce a coplete rankng on obects) on BT to produce the Mallows θ odel. Ths s dscussed n Marden [8]. The reander of ths secton follows Bradley []. In addton to MLE for scale paraeters, BT also provdes a eans to test whether the data are statstcally dfferent fro unfor. To test the hypothess H : π = = π = / (3) aganst the alternatve H : π π for soe,,,, =,, (4) a use the test statstc T = Nlog B U B = n log( p + p ) a log p < (5) whch s dstrbuted approxately ch-squared wth t- degrees of freedo (df) for large n under H. Soetes we wsh to test whether there are dfferences aong groups of responses. In the exaple below, we test whether there s a dfference between experts and nonexperts. Let each of g groups have ts own set of paraeters ndexed the followng way: u π, =,,, u =,, g. To test H : π u = π, =,,, u =,, g (6) versus the alternatve H : π u π for soe and u, (7) a use the the test statstc g TG = B B u u = (8) where B s coputed as above usng data pooled over groups and B s coputed for each group. Under H for u large nu ths test statstc has an approxate ch-squared dstrbuton wth (g-)(t-) degrees of freedo. Bradley also provdes a confdence regon for the vector paraeter π = ( π,, π ). Approxate ( α )% confdence ntervals for the locaton paraeters of nterest are ( log p z σ ˆ / N/ p,log p z σˆ / N/ p α/ α/ ) + (9) =,,, where N = n s the total nuber of < coparsons, the p are the le s, σ s the th dagonal ˆ eleent of the ( + ) by ( + ) atrx ˆ ˆ Λ Σ= ' where Λ= ˆ [ ˆ ], λ ()

4 ˆ λ = p n /[ N( p + p ) ], =,, p ˆ λ = n /[ N( p + p ) ],,, =,,. () -.5 Bradley-Terry scale for "preference" data wth 95% CI - Wth the ad of a atrx nverson routne, these statstcs are easly coded nto C. -.5 Analyss Exaple We analyze a data set usng BT odel to deonstrate ts advantages over TM. The experent s dscussed n detal n []. Four gaut-appng algorths were evaluated n two ways. In the frst part, subects chose the better rendton fro a par of prnts. In the second, subects chose the better reproducton of reference prnts. Tables and contan the coparson data. Table. Coparsons atrx for preference experent Table. Coparsons atrx for reproducton experent Each of eghteen udges vewed fve ages and each prnt was an age/algorth cobnaton. Judges were parttoned nto two classes based on experence: experts () and non-experts (7). Fro the data, we wsh to establsh for each task, whether preferences exst, and f so, a estate a preference scale. Further, we wsh to access whether dfferences exst between experts and non-experts. Preference Data Usng procedures suarzed above we perfor a hypothess test to deterne whether the data are statstcally sgnfcant fro unfor: T = 74. wth 3 df. The 95% chsquare cutoff s 7.8, so we conclude the data are U nonunfor. The estated scale: ( log( p ), =,, 4 ) s (., -.39, -.53, -.86)..5-3 Algorth Fgure. for preference data. The data can be grouped nto coparsons ade by expert and nonexperts. For expert data, the estated scale s: (.5, -.43, -.58, -.8) and the test statstcs for unforty t T U = 5. wth 3 df, whch s sgnfcant at 95%. For nonexpert data, the estated scale s: (.7,-.34, -.46, -.94) and T U = 4.7 wth 3 df, also sgnfcant at 95%. The scales for experts and nonexperts appear to be slar. We can do a hypothess test to copare these two populatons for preference data. The test statstc for unforty of these two groups s T =.75 wth 3 df, whch G s not sgnfcant at 95% and therefore we conclude there s no statstcal dfference n the preferences of these two populatons. Estated scales and confdence ntervals are shown n Fgures through Bradley-Terry scale for expert "preference" data wth 95% CI -3 Algorth Fgure. for expert preference data.

5 Bradley-Terry scale for nonexpert "preference" data wth 95% CI Bradley-Terry scale for expert "reproducton" data wth 95% CI Algorth.5 Algorth Fgure 3. for nonexpert preference data In suary, algorth 4 s preferred by experts and preferred weakly by nonexperts for the experent n whch subects were asked whch rendton they preferred. Reproducton Data The estated scale for the entre reproducton data set s (-.54, -.57, -.5, -.48). The test statstc T U = 5.7 wth 3 df, N = 54, whch sgnfcant at 95%. We therefore conclude that the data s statstcally dfferent fro a pure rando saple fro a unfor dstrbuton and that the data show a preference structure. For the expert responses aong the reproducton data, the estated scale s: (-.5, -.64, -.9, -.67). The test for unforty: T U = 9.3 wth 3 df, sgnfcant at 95%, fro whch we conclude that the data for experts show a preference structure. For nonexperts, the estated scale s: (-.6, -.5, -.7, -.) and T = 3.76 wth 3 df, whch s not U sgnfcant at 95%. We conclude that the data are not statstcally dfferent fro unfor (there s a 8.8% chance we would have gotten ths test statstc value were the data fro a unfor dstrbuton) Bradley-Terry scale for "reproducton" data wth 95% CI Algorth Fgure 4. s for reproducton data Fgure 5. for expert reproducton data. Bradley-Terry scale for nonexpert "reproducton" data wth 95% CI.5 Algorth Fgure 6. for nonexpert reproducton data. To copare experts and nonexperts, the test for unforty of these two groups: T = 7.4 wth 3 df, whch s G not sgnfcant at 95% (but t s sgnfcant at 94%; that s, there s a 6% probablty that ths test statstc value would be obtaned under unforty). Thus algorth 3 s preferred by experts for the reproducton experent n whch vewers were asked to udge whch algorth produced a closer atch to an orgnal. Nonexpert udgents are not statstcally dfferent fro unforly rando preferences. Suary Owng to ts splcty, BT s uch ore developed analytcally than TM (Case V). Many statstcal procedures are avalable and easly pleented. We have deonstrated a few: le s for scale paraeters wth confdence ntervals (and regons), hypothess tests for unforty, and hypothess tests for preference agreeents aong groups. In the an, both odels can be cast nto the fraework of generalzed lnear odels and nuerc technques used to perfor slar analyses [3]. Should one wsh to odel nteractons between pars of stul and dsperson varatons, alternatves to TM are avalable []. In the odern settng, we are no longer restrcted to leastsquares solutons to TM odels. We can explore any

6 general odels usng odern statstcal theory and software. But for the bulk of our work (Thurstone s Case V), BT provdes powerful analyses easly pleented n a few tens of lnes of C-code. References. R. Balasubraanan et al., Gaut appng to preserve spatal lunance varatons, Proc. 8th Color Iagng Conference, pp. 8 ().. R. A. Bradley, Pared coparsons: Soe basc procedures and exaples, Handbook of Statstcs, Vol. 4, P. R. Krshnaah and P. K. Sen, eds., Elsever Scence Publshers, pp (984) 3. D. E. Crtchlow and M. A. Flgner, Pared coparson, trple coparson, and rankng experents as generalzed lnear odels, and ther pleentaton n GLIM, Psychoetrka, 56(3), pp (99). 4. H. A. Davd, The Method of Pared Coparsons (nd ed.), New York, Oxford Unversty Press (988). 5. P. G. Engledru, Psychoetrc Scalng: A Toolkt for Iagng Systes Developent, Wnchester, Ioteck Press (). 6. L. R. Ford, Jr., Soluton of a rankng proble fro bnary coparsons, Aercan Matheatcal Monthly, 64, pp. 8-33, (957). 7. C. L. Mallows, Non-null rankng odels: I, Boetrka, 44, pp. 4-3 (957). 8. J. I. Marden, Analyzng and Modelng Rank Data, New York, Chapan & Hall (995) 9. W. S. Torgerson, Theory and Methods of Scalng, New York, Wley (958).. M. Zhou and C. Cu, New atheatcal odel for the law of coparatve udgent, Proc. IS&T's 6th Internatonal Congress on Dgtal Prntng Technologes,Vancouver, BC, Canada, pp ().