J O U R N A L Hw T Start A n Objective Evaluatin O f Yur Training Prgram DONALD L. KIRKPATRICK, Ph.D. Assistant Prfessr, Industrial Management Institute University f Wiscnsin Mst training m e n agree that it is imprtant t evaluate training prgrams. T h e y als feel that the evaluatin shuld be dne by bjective means. Hwever, the typical training man uses evaluatin sheets r cmment sheets as the sle measure f the effectiveness f his prgrams. H e realizes h e shuld d mre, b u t he just desn't k n w hw t begin an bjective evaluatin. step. Typically, the training directr asks the trainees t fill ut evaluatin sheets at the end f the prgram. Questins that are asked mst frequently are: Accrding t R a y m n d Katzell, a well k n w n authrity in this field, the evaluatin f a training prgram falls int a hierarchy f steps that can be briefly stated as fllws: 4. W h a t did yu learn that yu can use n the jb? Step O n e : T determine hw the trainees feel abut the prgram. Step T w : T determine hw m u c h the trainees learn in the frm f increased knwledge and understanding.. H w d yu rate the prgram? Excellent Gd Very Gd Fair Pr. W h a t subject did yu like best? 3. W h a t subject did yu like least? 5. W h a t subjects w u l d yu like t have discussed at f u t u r e prgrams? Usually the trainees are nt asked t sign their n a m e fr fear they will nt give an hnest reactin. Step Fur: T determine the effects f these behaviral changes n bjective criteria such as prductin, turnver, absenteeism, and waste. T h i s kind f subjective evaluatin is imprtant. It gives a gd indicatin f hw the trainees reacted t the prgram. If they react favrably, the trainer can justifiably pat himself n the back and say, "I gave them a prgram they liked. But he can't rightfully claim that the training prgram accmplished the bjective, unless his bjective was t give t h e m a prgram they liked. I n climbing this ladder f evaluatin, mst trainers have cmpleted the first T h e immediate bjective f any training curse can be stated in terms f the Step T h r e e : T measure the changes in the n-the-jb behavir f the trainees.
Mav-June. 95 desired knwledge and understanding that the prgram is trying t impart t the trainees. It is this stage f evaluatin that shuld be u n d e r t a k e n as the secnd step. It is m u c h mre difficult than step ne and, therefre, is nt undertaken by m a n y trainers. test can be used. Because the cnstructin f a test invlves such factrs as the chice f items, the wrding f items, the n u m b e r and type f pssible respnse, and the sequence f items, it is far better t use an available inventry if it cvers mst f the curse cntent. Amng the pssible methds fr determining w h e t h e r increased knwledge and understanding have taken place, the hest ne seems t be the "befre and alter" paper and pencil test. If the scres n the psttest are significantly higher than n the pretest, the curse can be deemed effective. H a v i n g selected r cnstructed a test, the trainer shuld cnsider sme "D's" fr administering it: I N determining the effectiveness f the training, it is imprtant t nte that the paper and pencil test r inventry must cver the principles and facts that aie discussed in the curse. If the trainer can find a test that cvers this material, he can use it. If he cannt find a suitable ne, he must cnstruct his wn inventry. Sme f the inventries that aje available are: Hw Supervise? by ] e and Remmers; Supervisry Inventry by Wesley Osterberg; and the Supervisry Inventry On I hi man Rec, tins cnstructed by this writer.. H a v e the trainee sign bth the pretest and psttest. T h e n, the increased knwledge and understanding can be cmputed fr each individual.. Give the pretest at the start f the first class and the psttest at the clse f the last sessin. T h i s will minimize the i n f l u e n c e f factrs apart frm the training curse. 3. In instructing the trainees befre they take the pretest: a. Tell them it is a befre and after prcedure. b. Explain the purpse f the test. c. Encurage t h e m t answer truthfully by assuring them that their scres will have n effect n their pay r status in the cmpany. S(> far, then, it has been stated that a befie and after test can be used t ctcimine w h e t h e r r nt increased knwledge and u n d e r s t a n d i n g have ta a n place. Als, that the inventry s!uld cver the curse cntent. In rder t determine w h e t h e r r nt an test is suitable, a trainer must examine his curse utline and list the principles and facts he is trying t teach. e. Encurage them t take their time in taking the test. T h i s will h e l p t mtivate them t read each item carefully. cmparisn f test items with these jeetives will reveal w h e t h e r r nt the In analyzing the test results, there are tw kinds f evaluatins t be made: 0 d. Tell them t answer every questin even if they have t guess. ( T h i s will be taken int accunt in the statistical analysis f scres.)
J O U R N A L 0. W a s the entire curse effective as shwn by gains frm pretest t psttest scres fr all trainees?. W h a t specific facts and principles were learned as shwn by changes frm pretest t psttest fr each item?.. 3. 4. 5.. 7. 8. 9. 0. Overall Effectiveness f the Curse In cnsidering the first questin, the ttal scre f crrect respnses fr each individual is determined fr his pretest a n d psttest. T h e s e scres are cmpared t determine his gain f r m pretest t psttest. T h e m e a n gain ( M g ) is then cmputed as illustrated in T a b l e. TABLE Cmputatin f Mean Gain Psttest Scre Pretest Scre 93 84 94 8 95 94 88 8 90 9 87 78 0 Trainee J. Andersn W. Brwn K. Dalberg M. Fultn G. Gage V. Grenfell B. Hward J. Lewis R. Masn S. Stanley Gain 9 4 8 88 Mean The next step in determining whether r nt the changes in scres f r m pretest t psttest are significant enugh t prve the prgram effective is t calculate the Standard Deviatin () f the Mg. This can be dne by using the fllwing frmula: / N VNEX! (EX) where: N the number f individuals being measured X the gains EX the sum f the squares f the gains (EX) the square f the sums f the gains Using the figures frm TABLE we find the fllwing: X X 9 4 3 8 9 4 3 4 34 94 N 0 EX 94 (EX) () /0 5.3 400 \/ 0 X 94 The next step is the cmputatin f the estimated standard by the fllwing frmula: 5.3 Sm Sm ViVN The " t " scre can then be determined by the f r m u l a : 8.0 Mg t t.8 Sir. 400 4.4.8
May-June, 95 Reference t a "t" table such as can be f u n d in M c N e m a r ' s Psychlgical Statistics reveals that tbis "t" f 4.4 cr- respnds t a prbability ( P ) f less than.0. T h i s means that fr ur example described in T A B L E I, the mean gain f 8 is s large that it can be attributed t chance less than ne time m ne h u n d r e d. Accrding t accepted practice, if the P is.05 r less, the gain is significant and the curse can be termed effective. It fllws that the less the P, the mre effective the curse. J herefre, the example given in tbis article prves that the curse was very effective. Facts and Principles That Were Learned Of equal imprtance in evaluating the curse is the determinatin f w h i c h specific facts and principles were learned by the trainees. In ther wrds, were there a f e w facts that were learned by mst f the trainees r did d i f f e r e n t trainees learn different things? T h i s O kind f evaluatin will reveal the effectiveness f the instructrs in setting acrss specific pints. In rder t measure this, the pretest scres f each item must be cmpared t the psttest scres fr that item. T h e significance f change can be determ i n e d by the relatively simple chi square frmula:* «frmula applies where there are nly tw pssible respnses t each item, an Agree" and a "Disagree." The writer's "Supervisry Inventry n Human delatins" is an example f this kind f a test. (A D)' X A+D mi which: X chi square A changes f r m "Agree" n pretest t "Disagree" n psttest D changes f r m "Disagree" n pretest t "Agree" n psttest n case A -)- D ttals less than 0, the fllwing frmula shuld be used: [(A-D)-n» A + D Prbability^ 0 & square table t h a t als can be fund in McNemar reveals the example, Item n the inventry states: ' > c s t way t train a new wrker is t have him watch a gd man n the jb." A tabulatin f respnses t this item reveals that 0 individuals changed their respnses frm "Agree" n the pretest t Disagree" n the psttest. O n e individual changed his respnse f r m "Disagree n the pretest t "Agree" n the psttest. T h e rest f the individuals respnded with the same answer n bth tests. T h e questin is, did the respnses n this item change significantly e n u g h t prve that the instructr gt acrss his pint? Substitutin in the frmula reveals: X - (A D) (0 l ) A + D 0 + 7.4
J O U R N AL Reference t the chi square table shws that this change has a prbability f less than.0. T h e r e f r e, the trainer was successful in teaching the principle that there is a better way f training a new wrker than t have him watch a gd wrker n the jb. A similar analysis f each item will reveal which facts and principles were learned by the trainees. Summary T r a i n i n g men agree that it is advis able t evaluate training curses as bjectively as pssible. Typically, their evaluatin cnsists f subjective cmm e n t sheets that are cmpleted by trainees at the end f the curse. Prviding that these are prperly administered, these evaluatin sheets give a valid measure f trainee reactin t the prgram. Hwever, they d nt give any evidence f benefits derived. T h e first step in bjectively evaluating the effectiveness f a training curse is t determine w h e t h e r r nt the desired facts and principles were learned by the trainees. T h i s can be d n e by:. U s i n g a suitable paper. a n d pencil test.. T e s t i n g the trainees befre and after the prgram. 3. D e t e r m i n i n g the verall effectiveness f the curse by cmparing pretest and psttest scres fr each trainee. 4. D e t e r m i n i n g w h i c h specific facts and principles were learned by analyzing the changes n each test item frm pretest t psttest. T h e purpse f this article is t suggest a specific t e c h n i q u e fr b e g i n n i n g an bjective evaluatin f a training prgram. F u r t h e r effrts shuld be undertaken by every training m a n t fllw u p this kind f an evaluatin by attempting t measure trainee change in behavir that ccurs as a result f participatin in the prgram. POSITION OPEN EDUCATION ADMINISTRATOR Cllege degree - writing ability - prvide cmprehensive prgrams in emplyee educatin, Training, cmmunicatins, and develpment f r all levels. Develp rientatin methds and ecnmic educatinal prgrams. Give technical assistance in salesman training. Initiate research fr lng and shrt r a n g e develpment prgrams. Apprximate age preferred 35-40. Salary cmmensurate with experience. Send resumes t R. C. McCleary, Persnnel Manager, H. J. Heinz Cmpany, Pittsburgh, Pennsylvania. POSITION OPEN Assistant t the Training Directr in a Texas Gulf Cast chemical plant f a multiplant crpratin. Applicants shuld have industrial experience, an interest in a career in Training and Industrial Relatins, and teaching experience. Cllege graduate, under thirty, with curse wrk in Industrial Educatin and Psychlgy, preferred. Bx 3, Jurnal f the American Sciety f Training Directrs, 00 University Avenue, Madisn 5, Wiscnsin. POSITION WANTED Training Directr with prven ability t write, develp executive and management develpment prgrams t meet specific needs. Develped prgrams in Time Study, Decisin Making, Cmpany Ecnmics, SQC, Methds, Cst Cntrl, Plicies & Prcedures. Used techniques f "Brainstrming," Incident Prcess, Rle-Playing, etc. Wrked with and "sld" tp grups regarding traininghave mfg. exper. & persnnel administratin ability. Age 37. Seek challenge. P r e f e r N.Y. r Calif. Bx 4, Jurnal f the American Sciety f Training Directrs, 00 University Avenue, Madisn 5, Wiscnsin.