AI ML The Fut ur e of Assessment How wi l l AI, Aut omat i on, and Mac hi ne Lear ni ng Change How We Devel op and Del i ver Assessment s? Nat han Thomps on, PhD I nt er nat i onal Conf er ence on Educat i onal Measur ement, Eval uat i on, and Assessment
AI & Aut omat i on A 2016 r epor t f r om Del oi t t e est i mat ed t hat 40% of l egal wor k wi l l soon be r epl aced by AI I t i s not a quest i on of whet her aut omat i on wi l l happen i n t he assessment i ndust r y, but wher e?
What will AI al l ow us t o aut omat e? We l l be abl e t o aut omat e ever yt hi ng t hat we can descr i be. - St ephen Wol f r am...well, there is a l ot about t est devel opment and psychomet r i cs t hat we can descr i be!
Def i ni t i on AI : us i ng comput er s t o do t hi ngs t hat nor mal l y woul d expect a human ML : devel opi ng mat hemat i cal model s to trai n systems Aut omat i on: Usi ng computers t o r epl ace humans or make t hem mor e ef f i ci ent, but of t en j us t r ul e- bas ed
El evat e your pr of essi on Item writers Teacher s Essay mar ker s Test devel oper s Pr ogr am manager s Psychomet r i ci ans
Make a car not a horsel ess car r i age!
Make a car not a horsel ess carri age!
About ASC: AI & Mac hi ne Lear ni ng Founded 1979 out of t he Psychomet r i cs pr ogr am at t he Uni ver si t y of Mi nnesot a Sof t war e t hat makes i t easi er f or any org t o devel op qual i t y assessment s We ar e wor ki ng di l i gent l y t o expl or e t he oppor t uni t i es pr esent ed by AI / ML!!
I TEM BANKI NG: LEARN FROM YOUR BANK Pi l ot st udy on usi ng machi ne l ear ni ng t o anal yze i t em banks f ound we can pr edi ct 56% of i t em qual i t y j ust by anal yzi ng whi ch wor ds ar e used i n st em Words on difficult items Words on easy items Term t Term t been 3.886 happy -2.980 that 3.841 free -2.807 their 2.692 going -2.720 mother 2.669 bus -2.363
Cr eat e Mat r i x Prune Matrix Tr ai n Model 1. Pull item texts from FastTest 1. R e mo v e mi s s i n g d a t a 1. Recode as UTF 1. Cr eat e Document - T e r m Mat r i x whi c h maps wh a t wo r ds a r e u s e d o n whi c h i t ems 1. Remove al l wor ds f r o m ma t r i x u s e d l es s t han 5 t i mes 1. Reduces t he mat r i x f r om t hous ands of wor ds t o hundr eds 1. Cr eat e a machi ne l ear ni ng model t o pr edi ct i t em difficulty or item qual i t y based on f r equent l y used t er ms 1. Can t i p of f aut hor s on wor ds t hat make i t ems t oo eas y or har d Machine learning on item banks; system will learn what makes a good item and provide feedback to authors
I TEM BANKI NG Tr ack your wor kf l ow t o l ear n of bot t l enecks and best per f or mer s!
I t em Aut hor i ng: GUIDELINES St r ai ght f or war d but still under ut i l i zed: Gui de aut hor s on i t em wr i t i ng r ul es I mpl ement s i t em r evi ew at t he t i me of wr i t i ng
Item Banki ng: REI NFORCED L E ARNI NG What i f we asked t eacher s t o gi ve i t ems & t est s a t humbs up/ down?
AUTOMATED I TEM GENERATI ON APPROACH 1 APPROACH 2 St af f def i nes i t em skel et ons wi t h sever al var i abl es ( A year ol d man pr esent s wi t h chest pai n af t er... ) 6x ef f i ci ency Feed a t ext book t o an AI al gor i t hm and i t pr ovi des dr af t i t ems back t o you St i l l i n i t s i nf ancy but super exci t i ng SEE IN ACTION
AUTOMATED TEST ASSEMBLY Test Templ at es: f or ce al l t est s t o f ol l ow bl uepr i nt s t o ensur e cont ent validity Aut omat ed t est assembl y: Bui l d test forms di r ect l y t o bl uepr i nt s at cl i ck of a but t on
AUTOMATED TEST ASSEMBLY Cur r ent t ool avai l abl e f or downl oad Exper t s can use Li near Pr ogr ammi ng packages
TEST PUBLI SHI NG Eval uat e pool of i t ems t o aut omat i cal l y det er mi ne appr opr i at e CAT al gor i t hms and publ i s h a def ensi bl e CAT You s houl d not have t o wr i t e c ode t o publ i s h a CAT! Or do IRT!!!
MORE TEST PUBLI SHI NG St andar d set t i ng: Aut omat e how Angof f r at i ngs ar e gat her ed and t hen col l at ed i nt o a r epor t
MORE TEST PUBLI SHI NG Last year s wi nner of t he I nnovat i on Lab at t he ATP conf er ence: AI t hat wat ches how st udent s sol ve t ech- enhanced i t ems and t hen uses i t f or f ut ur e scor i ng. Si mi l ar t o AES. met acog. com
TEST DELI VERY Li near on t he f l y t est i ng ( LOFT) : Al l exami nees get 50 i t ems t o same bl uepr i nt s, but di f f er ent i t ems, f or bet t er secur i t y Comput er i zed adapt i ve t est i ng ( CAT) : The t est per sonal i zes i t sel f t o ever y exami nee. Per f ect mat ch f or per sonal i sed l ear ni ng. Mu l t i - St age Test i ng: Mel ds CAT, LOFT, and l i near t est s - per f ect f or l anguage assessment These ar e AI and aut omat i on based on mac hi ne l ear ni ng pr i nc i pl es ( I RT)!
REPORTI NG AND ANAL YT I CS Psychomet r i c Analytics: test and i t e m p e r f o r ma n c e r epor t s r epl ace a ps yc homet r i c i an wi t h an al gor i t hm
REPORTI NG AND ANAL YT I CS Au t o ma t i c a l l y mo v e psychometri c stati sti cs bac k t o i t em banker Tr ack al l hi st or i cal stats to fl ag possi bl e dr i f t
REPORTI NG AND ANALYTI CS: L OCAT I ONS / P R OGR AMS Ca n we c onnec t a busi ness i nt el l i gence engi ne t o make i t easy f or st akehol der s t o do t hi s however t hey l i ke?
REPORTI NG AND ANALYTI CS: Psychomet r i c For ens i c s Perform psychomet r i c f or ens i cs t o spotl i ght i ssues of test security and val i di t y threats Can we do t hi s i n r eal t i me?
AI & ESSAY I TEMS Aut omat ed essay scor i ng - pr ovi des a f r ee second or t hi r d r at er t hat s been shown t o be as accur at e as humans Two appr oaches: base i t on l anguage t heor y, or on pur e dat a sci ence And yes, t hey exi st f or Ar abi c! Al so: what can t he FACETS appr oach t el l you about r at er s, pr ompt s, and r ubr i cs?
AI & STUDENT FEEDBACK Cogni t i ve di agnost i c model s: pr ovi de det ai l ed f eedback on i ndi vi dual ski l l s bei ng t est ed Theor y- dr i ven machi ne l ear ni ng appr oach
MORE! Thi s i sn t even t ouchi ng on AI / ML i n t he use of t est scor es Adapt i ve l ear ni ng Recommender syst ems Predicting job performance and other workforce out comes What can per sonal i t y aspect s pr edi ct about l oan def aul t s? Rent er saf et y?
THANK YOU nthompson@assess.com