Available online at ScienceDirect. Procedia Economics and Finance 17 ( 2014 ) 39 46

Transcription

1 Avalabl onln at ScncDrct Procda Economcs and Fnanc 7 ( 204 ) Innovaton and Socty 203 Confrnc, IES 203 A masur of ordnal concordanc for th valuaton of Unvrsty courss Donata arasn a, Pro Quatto a and Enrco Ramont a a Dartmnt of Economcs, anagmnt and Statstcs, Statstcal Scton, Unvrsty of lan-bcocca, Pazza Atno uovo, lan 2026, Italy Abstract In ths ar w aly th s statstc, amd to masur th ntr-ratr agrmnt btwn obsrvrs n cas of ordnal varabls, to th valuaton of th qualty of Unvrsty courss. Th objctv s to masur th ntr-ratr agrmnt btwn studnts, along wth thr satsfacton, n ordr to vrfy th consstncy of judgmnts xrssd by ndndnt obsrvrs. s s a modfcaton of a rvously roosd ndx, whch avods th roblm of aradoxs of Cohn s and Flss kaa statstcs. W rsnt th s ndx from both a dscrtv and an nfrntal ont of vw. In artcular, as far as statstcal nfrnc s concrnd, w show that s s a basd stmator of th ntr-ratr agrmnt n th oulaton and, undr th null hyothss of ntr-ratr agrmnt by chanc, s s asymtotcally normally dstrbutd. 204 Th Authors. Publshd by Elsvr B.V. Ths s an on accss artcl undr th CC BY-C-D lcns 204 Th Authors. Publshd by Elsvr B.V. (htt://cratvcommons.org/lcnss/by-nc-nd/3.0/). Slcton and r-rvw undr rsonsblty of th Organzng Commtt of IES 203. Slcton and r-rvw undr rsonsblty of th Organzng Commtt of IES 203. Kywords: ntr-ratr agrmnt, ordnal masurs, satsfacton. Introducton As for th accrdtaton of acadmc courss, Italy has bn adatng to Euroan standards. Th AVUR (atonal Agncy for th Evaluaton of th Unvrsty Systm and Rsarch) has bn algnng to numrous othr agncs cratd mmdatly aftr th Brgn agrmnts of 2005 (Brgn Communqué htt:// Th accrdtaton rocss has had ts offcal launch n th acadmc yar 203-4, accordng to th crtra, ndcators and bnchmarks rovdd by th sam AVUR (htt:// wth artcular attnton to qustonnars dsgnd to dtct th satsfacton of undrgraduat and graduat studnts on crtan ascts of tachng. Qustonnars ar th dal nstrumnt to dal wth concordanc; n fact, n ths contxt, th xamnrs ar studnts, th objcts to b xamnd ar th tms of a qustonnar that studnts ar rqustd to rat, and th catgors ar th ossbl mods of rsons. Th objctv of ths ar s to aly a masur of concordanc, along wth a satsfacton ndx, n th framwork of th valuaton of Unvrsty courss. As a mattr of fact, w am to masur th concordanc n th judgmnts gvn by th studnts: f ths concordanc s wak, thr s a dsrson of judgmnts; n th cas th concordanc s strong,.., th studnts gnrally agr, th answrs ar n th sam drcton, thr towards satsfacton or towards dssatsfacton. Hnc, a masur of satsfacton can b assocatd wth an arorat ndx of concordanc; ths can b usful n ordr to vrfy th consstncy of th judgmnts xrssd by ndndnt obsrvrs. Corrsondng author. Tl.: E-mal addrss: donata.marasn@unmb.t Th Authors. Publshd by Elsvr B.V. Ths s an on accss artcl undr th CC BY-C-D lcns (htt://cratvcommons.org/lcnss/by-nc-nd/3.0/). Slcton and r-rvw undr rsonsblty of th Organzng Commtt of IES 203. do:0.06/s (4)

2 40 Donata arasn t al. / Procda Economcs and Fnanc 7 ( 204 ) As a masur of concordanc, w wll ntroduc a wghtd vrson of th s statstc (Quatto, 2004; Falotco & Quatto, 200), whch has bn ut forth as an altrnatv to th kaa statstc, known as Flss kaa (Flss, 97; Flss t al., 2003) that had bn roosd as an arorat vrson of Cohn s kaa statstc (960), n th cas of multl obsrvrs. In artcular, n Scton 2, w ll dscrb th s statstc (arasn t al., 204), whch s a sutabl gnralzaton of s. Frst, s can dal wth both nomnal and ordnal varabls; scond, s ks nto account th ossblty of a dffrnt numbr o f obsrvrs for ach row of a datast. In Scton 3, s has bn studd n trms of nfrnc, as far as both stmaton and hyothss tstng s concrnd. Ths s a mthodologcal scton, whch rsnts a dscusson of orgnal rsults wth rsct to statstcal nfrnc. Scton 4 dals wth an alcaton on th data of an Italan Unvrsty rgardng studnts assssmnt of two Bachlors, durng th acadmc yar Th last aragrah s ddcatd to a brf concluson. 2. Th s statstc It s worth notng that w consdr th framwork of th valuaton of Unvrsty courss. Thrfor th s statstc wll b tratd n ths artcular ara, whr obsrvrs ar studnts who rat tms of qustonnars on a rdtrmnd scal. Undr ths rms, suos that th gnrc qustonnar admnstrd to studnts conssts of tms, for whch catgors dntfd wth th frst ntgrs ar ossbl, accordng to th followng Tabl : Tabl. studnts hav bn askd to valuat th qualty of acadmc courss on an -dmnsonal scal. j Total whr n rrsnts th numbr of studnts who hav gvn answr j to tm (j =,, ; ; ). A ar of studnts s dfnd as concordant f thy rovd th sam answr to a crtan tm. Lt s consdr tm : th ossbl concordant ars ar, among ths ar concordant on catgory j. Lt s dfn ( =,,; j =,, ), such that: n 2 n ( n ) () n n ( n ) 2 dntfs th robablty that a ar of studnts s concordant on catgory j. Consdr ars of catgors, t fo llo ws that ars of th knd (j,j) ar assocatd wth ars of studnts, whl (j,k) ars wth j<k (k =,, ) ar assocatd wth ars. In artcu lar, snc th catgors j and k ar ndndnt, to (j,k) ar assocatd ars of studnts, so that th robablty that two studnts ar concordant on th ar (j,k) s gvn by: 2nnk k (2) n ( n ) It follows that vry ossbl ars of studnts ar assocatd wth vry ossbl ars of catgors. Suos now to assocat to () and (2) wghts and varyng btwn 0 and (Abrara t al. 999). Snc wghtng s usd n ordr to xrss n mathmatcal trms varous lvls of agrmnt among dffrnt ars of catgors, t sms rasonabl to dfn, (j=,,), so that, for =,,: and n( n ) 2 w nn k j j k j kw (3) n ( n ) j j k j

3 Donata arasn t al. / Procda Economcs and Fnanc 7 ( 204 ) (4) rrsnts th man robablty. Th (4) should b comard wth th smlar man robablty that thr would b undr th rsnc of chanc,.., n cas th choc of on of th rsonss was scord by ach studnt on a random bass, and not as a consqunc of a rsonal valuaton. Followng Quatto (2004), th robablty that a studnt randomly rsonds j to tm can b concvd as /, and t follows: 2 2 w j k j (5) as t wll b shown n th nxt aragrah wth Equaton (3). Th s statstcs (s arasn t al., 204), whch masurs th strngth of concordanc, s obtand comarng wth n th followng trms: s (6) Th (6) s th wghtd vrson of th s statstc whr, fxd, th wghts W dfn th wghts n th ordnal cas as: w ar qual to 0. w r j k j k, r,2 (7) whch gnralzs th absolut rror wgths (r = ) and th squar rror wghts n (7) wth r = 2 (Cohn, 968; Lght, 97; Flss, 97). In th rsnt ar w ll consdr r = ; n ths cas, Equaton (4) and (5) ar xrssd by: and 2 j k nn (8) k n ( n ) j k j j k j k j (9) whr and n (8) and (9) ar, rsctvly, gvn by: and so that: s s th s statstc roosd by Quatto (2004). It s worth notng that th wghts ar hghr for adjacnt than for dstant catgors; as a mattr of fact th ar has wght qual to 0. Brry & lk (988) roosd a lst of dsrabl rorts for a masur of concordanc. To ths rorts, w can add th followng two rorts: () n th cas of ordnal varabls, th mn mu m of th masur should b achvd whn th frquncs ar ntrly concntratd n th frst and n th last catgors (ma xma l ordnal dsrson); () ths masur should

4 42 Donata arasn t al. / Procda Economcs and Fnanc 7 ( 204 ) achv ts ma xmu m n th cas all th rsonss ar concntratd n on catgory. It s ntrstng to not that, n th nomnal cas, th mnmum should b achvd n cas of qu-dstrbuton across catgors (maxmal nomnal dsrson). Usng th Lagrang mu lt lrs t can b shown that th s ndx satsfs both rorts () and () (for dtals, s arasn t al., 204). In fact, n th frst cas: 2 n 2 n and, n th scond cas,. Hnc, s s dfnd n th ntrval: 3 2 n 2 2 n, (0) 3. Th s statstc from an nfrntal ont of vw Th s statstc can b ntrrtd as an stmat of an unknown masur of concordanc n th oulaton of studnts, for nstanc, studnts nrolld n a crtan Bachlor. Such masur can b concvd consdrng th obsrvd concordanc,..,, whr can b bult accordng to th (3), kng nto account th ral sz of th oulaton; from th concordanc du to chanc, bult accordng to (5), has to b subtractd, obtanng a form of th ty (6). In artcular, dfnng wth and th numbr of studnts n th targt oulaton who hav rsondd to tm and th numbr of studnts who hav rsondd j to tm, t follows: ( ) 2 A A j j k A ( A ) P j w AAk () To calculat th xctd valu of s, t can b obsrvd that th -th row of Tabl can b concvd as a multnomal tral wth aramtr vctor ( =,,), wth. Undr th mu ltno mal hyothss th xctd valu of Equaton (3) s gvn by (s arasn t al., 204): E w j k k Th rvous xctd valu can b obtand through straghtforward algbra, consdrng th scond-ordr momnts of th random varabls (rrsntng th numbr of studnts who hav ratd j to tm ) and from th mxd momnts of th varabls and (whr th lattr ndcats rsons k to tm ). It fo llo ws that: E w j k k (2) Lt s now calculat (2) undr th ffct of chanc: ths stuaton can b ntrrtd by a multnomal modl wth aramtr vctor. It mmdatly follows fro m (2): 2 w 2 2 E 0 w (3) j k j k j n agrmnt wth (5), whch rrsnts concordanc du to chanc. It follows that th xctd valu of s s gvn by:

5 Donata arasn t al. / Procda Economcs and Fnanc 7 ( 204 ) E Es j k w k (4) whch, comard wth th (), s a basd stmat of. It s worth notng that th bas assocatd wth (4) dcrass whn th numbr of studnts n th targt oulaton who hav rsondd j to tm ncrass (s Equaton ()). As far as tstng of statstcal hyothss s concrnd, th hyothss of acton of chanc to b tstd s gvn by: H : 0 (5) 0 whch, kng nto account (3) and (4), can b xrssd by: s 0 H (6) : E 0 0 Lt s now dntfy th dstrbuton of th s statstc undr H. W roos two mthods, for th lattr of whch t s rqurd 0 that th numbr of tms s larg nough. Th frst mthod uss ont Carlo smulatons: multnomal random varabls ar smulatd for B tms, and th s statstc s calculatd, so that th B valus s,, s lad to s, whos dstrbuton law s th mrcal dstrbuton functon. Th scond B mthod studs th convrgnc of s to a normal dstrbuton functon wth zro man. Th roof of th convrgnc to th ormal random varabl s a bt laborous and th xrsson of th varanc s vry comlcatd (s arasn t al., 204). To tst th hyothss n (5), t should b notd that, f (6) taks valus n a nghborhood of 0, th concordanc can b consdrd du to chanc; f t s gratr than 0 and taks valus clos to, w can assum th xstnc of agrmnt; f t taks smallr valus than 0 and nar th lowr bound of (0), w can assum dscrancy n dcsons btwn studnts. Thrfor, ths control can b ralzd wth a on-sdd tst, ladng to th acctanc or rjcton of th ffct of concordanc by chanc. For a fxd and an obsrvd valu of, th -valu can b calculatd: (7) rjctng th null hyothss n (5) f. As rvously statd, th null dstrbuton of, dfnd n Equaton (7), can b calculatd by ont Carlo smulatons, or by asymtotc aroxmaton. Undr a gnral hyothss, th dstrbuton of s can b dntfd usng th rcntl bootstra mthod,.., by rsamlng B tms wth rlacmnt th valus (3), and calculatng ach tm th (6). Th combnaton of ths nw B valus lads to th bootstra dstrbuton of th s statstc, and thrfor, at th lvl, w can dntfy th two rcntls of ordr and, thus dtrmnng a confdnc ntrval for. orovr, also t-bootstra confdnc ntrvals can b calculatd; w consdrd ths tchnqu snc t s scond ordr accurat (s Shao & Tu, 995,. 46). In artcular, th ntrval can also b concvd as a bdrctonal tst of sgnfcanc for th hyothss (5). Lt s now furthr commnt th choc of consdrng th multnomal modl. It can b obsrvd that an altrnatv modl would hav bn that of consdrng a multvarat hyrgomtrc random varabl. As a mattr of fact, th studnts rrsnt a saml wthout rlacmnt from th targt oulaton. Howvr, undr ths altrnatv hyothss, t s stll th cas that th stmator gvn by (4) s basd. In fact: (8) whch s dffrnt fro m n Equaton (). From Equaton (8), t follows: (9) To vrfy (8), n th cas of th multvarat hyrgomtrc dstrbutons wth aramtr vctor for, w hav

6 44 Donata arasn t al. / Procda Economcs and Fnanc 7 ( 204 ) and so that n agrmnt wth Equaton (8). In artcular, undr th null hyothss w obtan th xctd agrmnt as n (9). From on hand, t can b asly notd that Equaton (9) dnds from th sz of th oulaton of studnts who hav rsondd to ach tm and ths condton would lmt th calculaton of concordanc du to chanc n th saml. From th othr hand, th multnomal modl aroxmats th multl hyrgmtrc modl for larg. 4. An alcaton W consdrd th valuaton qustonnars, comosd of 9 tms, comld by th studnts of two Unvrsty courss of an Italan Unvrsty. For th uross of ths alcaton, w hav only consdrd th quston "Ar you satsfd of th tachng qualty of ths cours?", whch has bn takn nto account for ach of th courss that hav bn valuatd by studnts. W obsrv that n many alcatons nvolvng th valuaton qustonnars of studnts, th rvous quston s takn nto consdraton, as a quston of synthss for th valuaton of th ntr acadmc cours. Th ossbl ordnal rsons catgors ar four (Dcddly no: Dcsamnt no, or no than ys: Pù no ch sì, or ys than no: Pu sì ch no, Dcddly ys: Dcsamnt sì ), xrssd nto th frst four ntgrs of a Lkrt scal. Th Bachlor durng th acadmc yar was comosd by 23 courss, 0 of whch ar shard wth Bachlor 2, whch nstad s comosd by 24 courss. Shard courss btwn Bachlor and 2 ar fundamntal and basc courss; thrfor thy ar attndd by a largr numbr of studnts than thos that charactrz ach of th dgr rograms. Wth rsct to Tabl. for Bachlor w hav = 23 and n = 059, whl n Bachlor 2 w hav = 24 and n = 579; mo rovr, = 4 for bo th Bachlors. As far as tm s concrnd, w ntroduc a classcal satsfacton ndx (s arasn & Quatto, 20), rrsntng a wghtd man of th valus of th scal, and takng valus on th ntrval : SI j f, wth j n f. n Th satsfacton ndx SI s obtand as a man of th satsfacton ndx calculatd for ach tm (wghtd by th rlatv frquncy of ratrs). Tabl 2 shows th valus of SI; s, th bootstra confdnc ntrvals at lvl wth th rcntl mthod, th t- bootstra confdnc ntrvals, th -valu at lvl, calculatd usng th ont Carlo mthod, and th asymtotc - valu. For both th calculaton of th confdnc ntrvals and th dntfcaton of th -valu wth th ont Carlo mthod, t has bn fxd B = 0,000, whch concds wth th numbr of smulatons n th frst cas and wth th numbr of rsamlngs

7 Donata arasn t al. / Procda Economcs and Fnanc 7 ( 204 ) n th scond cas. Tabl 2. Satsfacton ndx (SI), s, Bootstra confdnc ntrvals, ont Carlo (C) and asymtotc -valus, n th cas of th Bachlors and 2. 95% Prcntl bootstra CI 95% t-bootstra CI SI s nf su nf su C Asym Bachlor <.005 <.005 Bachlor <.005 <.005 Bootstra ntrvals obtand wth th two mthods lad to smlar valus, wth largr ntrval for th t-bootstra than for th rcntl bootstra. An xamnaton of th bootstra ntrvals shows that th null hyothss of th acton of chanc can b rjctd, snc ths ntrval dos not nclud th valu of 0; a smlar concluson can b drawn from an xamnaton of th last two columns that show th rsults of a undrctonal tst, whch ndcats that th hyothss of acton of chanc can b rjctd, and ths lads to acct th rsnc of concordanc. In both courss th concordanc s hgh, whch nforcs a satsfacton ndx that can b adotd to synthsz satsfacton. W dnd our analyss by calculatng th sam aramtrs on all th courss shard btwn Bachlor and 2, rgardng athmatcs, Comutr Scnc and Statstcs. Furthrmor, w consdrd thos courss rmarly rlatd to Economc Statstcs and Economcs for Bachlor ; Dmograhy and Bostatstcs for Bachlor 2, obtanng th rsults shown n Tabl 3. Tabl 3. Satsfacton ndx (SI), s, bootstra confdnc ntrvals, ont Carlo (C) and asymtotc -valus, n th cas of dgr courss n Bachlor and 2, accordng to th courss and lssons shard btwn th two Bachlors and th scfc courss of ach Bachlor (Economc Statstcs and Economcs for Bachlor ; Dmograhy and Bostatstcs for Bachlor 2). 95% Prcntl bootstra CI 95% t-bootstra CI SI s nf su nf su C Asym Bachlor Shard courss <.005 <.005 Scfc courss <.005 <.005 Bachlor 2 Shard courss <.005 <.005 Scfc courss <.005 <.005 It should b notd that th grou of shard tachngs has th sam masur of concordanc, snc th qustonnars hav not bn dffrntatd btwn Bachlor and 2. Th stuaton s vry smlar to th rvous on: n all cass th concordanc s a bt hghr for courss n Bachlor 2 than for Bachlor, whch rnforcs th ntnsty of satsfacton xrssd by studnts for cours unts. Howvr, t should b notd that th numbr of studnts s consdrably largr n Bachlor than n Bachlor 2, whch may crat a gratr varablty of judgmnts, rsultng n a lo wr agrmnt. In both cass, th hyothss of acton of chanc s rjctd both wth th bootstra ntrvals and wth th undrctonal -valus (s th last two columns n Tabl 3). In th contxt of th Unvrsty valuaton, somtms th us of statstcal modls could b hard to b undrstood by nonstatstcans (.g., th managmnt or th ad mn stratv staff). Consquntly, n our o non, n ths framwork, adotng a sngl valu xrssng a satsfacton masur can b mor aalng than a dtald outut of statstcal modllng. From an ald ont of vw, s can b ntutvly hlful n ordr to undrstand how much th judgmnts gvn by th studnts ar dsrsd and how much thy ar du to chanc. 5. Conclusons Th concordanc n studnts rsonss to qustonnars s an mortant tool to b assocatd wth dffrnt masurs that could b usd to synthtz th assssmnt (.g., satsfacton ndxs). Thrfor, w hav roosd to us th s statstc, whch has a mathmatcal structur smlar to that of othr masurs of agrmnt roosd n th ltratur, whr th obsrvd concordanc s normalzd wth rsct to that du to chanc. Startng from th s statstc, whch, comard to th most known statstcs roosd n th ltratur, consdrs a dffrnt xctd agrmnt undr th acton of chanc, th s statstc has bn bult (arasn t al., 204): s gnralzs th s statstc snc t rfrs to a numbr of dffrnt xamnrs,.., studnts, for ach tm. orovr t taks nto account th ordnal scal, wghtng n a dffrnt way th dffrnt ars of catgors: th wght s gratr for ars of nghborng catgors that ndcat a hghr concordanc than n ars ndcatng dstant judgmnts. orovr, th ar (,) s not kt nto account. As far as statstcal nfrnc s concrnd, t has bn shown that s s a basd stmat of th analogous masur of unknown concordanc xstng n th oulaton, and that t has a dstrbuton that convrgs to that of a ormal random varabl. Th asymtotc dstrbuton allows th vrfcaton of th hyothss of rsonss du to chanc. Ths fact s vry mortant n th contxt of th assssmnt of studnts. Indd, t s crdbl that som studnts rsond randomly n many qustonnars, as thy ar askd to coml a larg numbr of tms and thy ar not xosd to th consquncs of thr judgmnts and n ths contxt th acton of chanc has bn ntrrtd undr an qu-dstrbuton modl. In th alcaton, w hav consdrd th valuaton qustonnars comld by th studnts from two Bachlors of an Italan Unvrsty; w

8 46 Donata arasn t al. / Procda Economcs and Fnanc 7 ( 204 ) xamnd th quston: Ar you satsfd of th tachng qualty of ths cours?. Our rsults ont to an ntr-ratr agrmnt whch can b ntrrtd as larg nough; n th cass w v xamnd s has ndcatd that th ntr-ratr agrmnt cannot b du to chanc. A futur drcton of ths ar wll consdr how th s ndx works wth fw ratrs, and how dos ths ndx dnd from th numbr of ratrs. orovr, w shall consdr dffrnt modls of acton of chanc. Rfrncs Abrara, V., Prz d Vargas A., 999. Gnralzaton of th Kaa coffcnt for ordnal catgorcal data, mu lt l obsrvs and ncomlt dsgn. Qustò 23, Brry, K.J., lk, P.W A gnralzaton of Cohn s kaa agrmnt masur to ntrval masurmnt and multl ratrs. Educatonal and Psychologcal asurmnt 48, Cohn J., 960. A coffcnt of agrmnt for nomnal scals. Educatonal and Psychologcal asurmnt 20, Cohn J.A., 968. Wghtd Kaa: nomnal scal agrmnt wth rovson for scald dsagrmnt or artal crdt. Psychologcal Bulltn 70, Falotco R., Quatto P., 200. On avodng aradoxs n assssng ntr-ratr agrmnt. Italan Journal of Ald Statstcs 22, Flss, J.L., 97. asurng nomnal scal agrmnt among many ratrs. Psychologcal Bulltn 76, Flss, J.L., Lvn, B., Pak,.C., Statstcal thods for Rats and Proortons. John Wly & Sons, Hobokn. Lhmann, E.L., 999. Elmnts of larg-saml thory. w Yo rk: Srngr. Lght, R.J., 97. asurs of rsons agrmnt for qualtatv data: som gnralzatons and altrnatvs. Psychologcal Bulltn 76, arasn D., Quatto P., 20. Dscrtv analyss of studnts ratng. Journal of Ald and Quanttatv thods, 8, arasn, D., Quatto, P., Ramont, E., 204. Assssng ordnal ntr-ratr agrmnt through wghtd ndxs. Statstcal thods n dcal Rsarch, acctd for ublcaton. Quatto, P., Un tst d concordanza tra ù samnator. Statstca LXIV, 4-5. Shao, J., Tu, D., 995. Th Jackknf and Bootstra. w York: Srngr.