How T o Start A n Objective Evaluation O f Your Training Program

Size: px

Start display at page:

Download "How T o Start A n Objective Evaluation O f Your Training Program"

Primrose Richards
5 years ago
Views:

1 J O U R N A L Hw T Start A n Objective Evaluatin O f Yur Training Prgram DONALD L. KIRKPATRICK, Ph.D. Assistant Prfessr, Industrial Management Institute University f Wiscnsin Mst training m e n agree that it is imprtant t evaluate training prgrams. T h e y als feel that the evaluatin shuld be dne by bjective means. Hwever, the typical training man uses evaluatin sheets r cmment sheets as the sle measure f the effectiveness f his prgrams. H e realizes h e shuld d mre, b u t he just desn't k n w hw t begin an bjective evaluatin. step. Typically, the training directr asks the trainees t fill ut evaluatin sheets at the end f the prgram. Questins that are asked mst frequently are: Accrding t R a y m n d Katzell, a well k n w n authrity in this field, the evaluatin f a training prgram falls int a hierarchy f steps that can be briefly stated as fllws: 4. W h a t did yu learn that yu can use n the jb? Step O n e : T determine hw the trainees feel abut the prgram. Step T w : T determine hw m u c h the trainees learn in the frm f increased knwledge and understanding.. H w d yu rate the prgram? Excellent Gd Very Gd Fair Pr. W h a t subject did yu like best? 3. W h a t subject did yu like least? 5. W h a t subjects w u l d yu like t have discussed at f u t u r e prgrams? Usually the trainees are nt asked t sign their n a m e fr fear they will nt give an hnest reactin. Step Fur: T determine the effects f these behaviral changes n bjective criteria such as prductin, turnver, absenteeism, and waste. T h i s kind f subjective evaluatin is imprtant. It gives a gd indicatin f hw the trainees reacted t the prgram. If they react favrably, the trainer can justifiably pat himself n the back and say, "I gave them a prgram they liked. But he can't rightfully claim that the training prgram accmplished the bjective, unless his bjective was t give t h e m a prgram they liked. I n climbing this ladder f evaluatin, mst trainers have cmpleted the first T h e immediate bjective f any training curse can be stated in terms f the Step T h r e e : T measure the changes in the n-the-jb behavir f the trainees.

2 Mav-June. 95 desired knwledge and understanding that the prgram is trying t impart t the trainees. It is this stage f evaluatin that shuld be u n d e r t a k e n as the secnd step. It is m u c h mre difficult than step ne and, therefre, is nt undertaken by m a n y trainers. test can be used. Because the cnstructin f a test invlves such factrs as the chice f items, the wrding f items, the n u m b e r and type f pssible respnse, and the sequence f items, it is far better t use an available inventry if it cvers mst f the curse cntent. Amng the pssible methds fr determining w h e t h e r increased knwledge and understanding have taken place, the hest ne seems t be the "befre and alter" paper and pencil test. If the scres n the psttest are significantly higher than n the pretest, the curse can be deemed effective. H a v i n g selected r cnstructed a test, the trainer shuld cnsider sme "D's" fr administering it: I N determining the effectiveness f the training, it is imprtant t nte that the paper and pencil test r inventry must cver the principles and facts that aie discussed in the curse. If the trainer can find a test that cvers this material, he can use it. If he cannt find a suitable ne, he must cnstruct his wn inventry. Sme f the inventries that aje available are: Hw Supervise? by ] e and Remmers; Supervisry Inventry by Wesley Osterberg; and the Supervisry Inventry On I hi man Rec, tins cnstructed by this writer.. H a v e the trainee sign bth the pretest and psttest. T h e n, the increased knwledge and understanding can be cmputed fr each individual.. Give the pretest at the start f the first class and the psttest at the clse f the last sessin. T h i s will minimize the i n f l u e n c e f factrs apart frm the training curse. 3. In instructing the trainees befre they take the pretest: a. Tell them it is a befre and after prcedure. b. Explain the purpse f the test. c. Encurage t h e m t answer truthfully by assuring them that their scres will have n effect n their pay r status in the cmpany. S(> far, then, it has been stated that a befie and after test can be used t ctcimine w h e t h e r r nt increased knwledge and u n d e r s t a n d i n g have ta a n place. Als, that the inventry s!uld cver the curse cntent. In rder t determine w h e t h e r r nt an test is suitable, a trainer must examine his curse utline and list the principles and facts he is trying t teach. e. Encurage them t take their time in taking the test. T h i s will h e l p t mtivate them t read each item carefully. cmparisn f test items with these jeetives will reveal w h e t h e r r nt the In analyzing the test results, there are tw kinds f evaluatins t be made: 0 d. Tell them t answer every questin even if they have t guess. ( T h i s will be taken int accunt in the statistical analysis f scres.)

3 J O U R N A L 0. W a s the entire curse effective as shwn by gains frm pretest t psttest scres fr all trainees?. W h a t specific facts and principles were learned as shwn by changes frm pretest t psttest fr each item? Overall Effectiveness f the Curse In cnsidering the first questin, the ttal scre f crrect respnses fr each individual is determined fr his pretest a n d psttest. T h e s e scres are cmpared t determine his gain f r m pretest t psttest. T h e m e a n gain ( M g ) is then cmputed as illustrated in T a b l e. TABLE Cmputatin f Mean Gain Psttest Scre Pretest Scre Trainee J. Andersn W. Brwn K. Dalberg M. Fultn G. Gage V. Grenfell B. Hward J. Lewis R. Masn S. Stanley Gain Mean The next step in determining whether r nt the changes in scres f r m pretest t psttest are significant enugh t prve the prgram effective is t calculate the Standard Deviatin () f the Mg. This can be dne by using the fllwing frmula: / N VNEX! (EX) where: N the number f individuals being measured X the gains EX the sum f the squares f the gains (EX) the square f the sums f the gains Using the figures frm TABLE we find the fllwing: X X N 0 EX 94 (EX) () / \/ 0 X 94 The next step is the cmputatin f the estimated standard by the fllwing frmula: 5.3 Sm Sm ViVN The " t " scre can then be determined by the f r m u l a : 8.0 Mg t t.8 Sir

4 May-June, 95 Reference t a "t" table such as can be f u n d in M c N e m a r ' s Psychlgical Statistics reveals that tbis "t" f 4.4 cr- respnds t a prbability ( P ) f less than.0. T h i s means that fr ur example described in T A B L E I, the mean gain f 8 is s large that it can be attributed t chance less than ne time m ne h u n d r e d. Accrding t accepted practice, if the P is.05 r less, the gain is significant and the curse can be termed effective. It fllws that the less the P, the mre effective the curse. J herefre, the example given in tbis article prves that the curse was very effective. Facts and Principles That Were Learned Of equal imprtance in evaluating the curse is the determinatin f w h i c h specific facts and principles were learned by the trainees. In ther wrds, were there a f e w facts that were learned by mst f the trainees r did d i f f e r e n t trainees learn different things? T h i s O kind f evaluatin will reveal the effectiveness f the instructrs in setting acrss specific pints. In rder t measure this, the pretest scres f each item must be cmpared t the psttest scres fr that item. T h e significance f change can be determ i n e d by the relatively simple chi square frmula:* «frmula applies where there are nly tw pssible respnses t each item, an Agree" and a "Disagree." The writer's "Supervisry Inventry n Human delatins" is an example f this kind f a test. (A D)' X A+D mi which: X chi square A changes f r m "Agree" n pretest t "Disagree" n psttest D changes f r m "Disagree" n pretest t "Agree" n psttest n case A -)- D ttals less than 0, the fllwing frmula shuld be used: [(A-D)-n» A + D Prbability^ 0 & square table t h a t als can be fund in McNemar reveals the example, Item n the inventry states: ' > c s t way t train a new wrker is t have him watch a gd man n the jb." A tabulatin f respnses t this item reveals that 0 individuals changed their respnses frm "Agree" n the pretest t Disagree" n the psttest. O n e individual changed his respnse f r m "Disagree n the pretest t "Agree" n the psttest. T h e rest f the individuals respnded with the same answer n bth tests. T h e questin is, did the respnses n this item change significantly e n u g h t prve that the instructr gt acrss his pint? Substitutin in the frmula reveals: X - (A D) (0 l ) A + D

5 J O U R N AL Reference t the chi square table shws that this change has a prbability f less than.0. T h e r e f r e, the trainer was successful in teaching the principle that there is a better way f training a new wrker than t have him watch a gd wrker n the jb. A similar analysis f each item will reveal which facts and principles were learned by the trainees. Summary T r a i n i n g men agree that it is advis able t evaluate training curses as bjectively as pssible. Typically, their evaluatin cnsists f subjective cmm e n t sheets that are cmpleted by trainees at the end f the curse. Prviding that these are prperly administered, these evaluatin sheets give a valid measure f trainee reactin t the prgram. Hwever, they d nt give any evidence f benefits derived. T h e first step in bjectively evaluating the effectiveness f a training curse is t determine w h e t h e r r nt the desired facts and principles were learned by the trainees. T h i s can be d n e by:. U s i n g a suitable paper. a n d pencil test.. T e s t i n g the trainees befre and after the prgram. 3. D e t e r m i n i n g the verall effectiveness f the curse by cmparing pretest and psttest scres fr each trainee. 4. D e t e r m i n i n g w h i c h specific facts and principles were learned by analyzing the changes n each test item frm pretest t psttest. T h e purpse f this article is t suggest a specific t e c h n i q u e fr b e g i n n i n g an bjective evaluatin f a training prgram. F u r t h e r effrts shuld be undertaken by every training m a n t fllw u p this kind f an evaluatin by attempting t measure trainee change in behavir that ccurs as a result f participatin in the prgram. POSITION OPEN EDUCATION ADMINISTRATOR Cllege degree - writing ability - prvide cmprehensive prgrams in emplyee educatin, Training, cmmunicatins, and develpment f r all levels. Develp rientatin methds and ecnmic educatinal prgrams. Give technical assistance in salesman training. Initiate research fr lng and shrt r a n g e develpment prgrams. Apprximate age preferred Salary cmmensurate with experience. Send resumes t R. C. McCleary, Persnnel Manager, H. J. Heinz Cmpany, Pittsburgh, Pennsylvania. POSITION OPEN Assistant t the Training Directr in a Texas Gulf Cast chemical plant f a multiplant crpratin. Applicants shuld have industrial experience, an interest in a career in Training and Industrial Relatins, and teaching experience. Cllege graduate, under thirty, with curse wrk in Industrial Educatin and Psychlgy, preferred. Bx 3, Jurnal f the American Sciety f Training Directrs, 00 University Avenue, Madisn 5, Wiscnsin. POSITION WANTED Training Directr with prven ability t write, develp executive and management develpment prgrams t meet specific needs. Develped prgrams in Time Study, Decisin Making, Cmpany Ecnmics, SQC, Methds, Cst Cntrl, Plicies & Prcedures. Used techniques f "Brainstrming," Incident Prcess, Rle-Playing, etc. Wrked with and "sld" tp grups regarding traininghave mfg. exper. & persnnel administratin ability. Age 37. Seek challenge. P r e f e r N.Y. r Calif. Bx 4, Jurnal f the American Sciety f Training Directrs, 00 University Avenue, Madisn 5, Wiscnsin.

Lab 1 The Scientific Method

Lab 1 The Scientific Method INTRODUCTION The fllwing labratry exercise is designed t give yu, the student, an pprtunity t explre unknwn systems, r universes, and hypthesize pssible rules which may gvern the behavir within them. Scientific