Component Analysis of a Behavioral-Supervisory System for Masters- and Doctoral-Level Research

Transcription

1 Western Michigan University ScholarWorks at WMU Dissertations Graduate College Component Analysis of a Behavioral-Supervisory System for Masters- and Doctoral-Level Research Michael J. Dillon Western Michigan University Follow this and additional works at: Part of the Psychology Commons Recommended Citation Dillon, Michael J., "Component Analysis of a Behavioral-Supervisory System for Masters- and Doctoral-Level Research" (1981). Dissertations This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact maira.bundza@wmich.edu.

2 COMPONENT ANALYSIS OF A BEHAVIORAL-SUPERVISORY SYSTEM FOR MASTERS- AND DOCTORAL-LEVEL RESEARCH by Michael J. Dillon A D isse rta tio n Submitted to the F aculty of The Graduate College in p a r tia l fu lfillm e n t of the requirem ents fo r the Degree of Doctor of Philosophy Department of Psychology Western Michigan U niversity Kalamazoo, Michigan A pril 1981

3 COMPONENT ANALYSIS OF A BEHAVIORAL-SUPERVISORY SYSTEM FOR MASTERS- AND DOCTORAL-LEVEL RESEARCH M ichael J. D illo n, Ph.D. W estern Michigan U n iv ersity, 1981 This p ro je c t d e sc rib e s a stru c tu re d approach to th e supervision of M.A. th eses and Ph.D. d is s e r ta tio n s. The main components of th is supervisory system a re : weekly (a) s p e c ific a tio n of ta sk s and p e rfo r mance standards, (b) m eetings w ith a supervisor (e ith e r fa c u lty or do cto ral stu d e n t), (c) d e a d lin e s, (d) feedback, and (e) added incentiv e s in the form of a p o in t system to be included by the stu d e n t s fa c u lty advisor in any requested l e t t e r s of recommendation. In seven experim ents, the s tu d e n ts research performance was h ig h e st when a l l components of the supervisory system were p re se n t; i t d e te rio ra te d as each major component was removed. Compared to more tr a d itio n a l supervisory approaches, th is system produced a g re a te r percentage of graduates in a s h o rte r le n g th of tim e, and w ith rs se a rc h -p ro je c ts rated as comparable in q u a lity. The amount of fa c u lty supervision time was about 5 hours p er week (with a to ta l of 12 su p erv ise e s). The p a rtic ip a n ts in th i s system highly valued th e ir involvement.

4 ACKNOWLEDGEMENTS With ap o lo g ies to Fred Skinner and Tom G ilb e rt, my p ro fessio n al knowledge and s k i l l s have been most in flu en ced by Jack Michael (in th e area of behavior a n aly sis) and Norm P eterson (in the area of performance en g in eerin g ). With apologies to no one e lse though, I e sp e c ia lly thank Dick M alott fo r h is su p erv isio n in the area of "d ealin g w ith i t " th a t is, accom plishing im portant, long-range personal and o rg a n iz atio n al goals. This research r e f l e c t s the con trib u tio n and value of D ick 's approach to improving human perform ance. In additio n, I thank A rt Snapper, Cheryl Poche, and A rt Falk fo r th e ir comments on an e a r lie r d r a f t of th is p ro je c t, and Brad Huitema fo r suggestions on the s t a t i s t i c a l a n a ly sis. M ichael J. D illon i i

5 INFORMATION TO USERS This was produced from a copy of a document sent to us for microfilming. While the most advanced technological means to photograph and reproduce this document have been used, the quality is heavily dependent upon the quality of the material submitted. The following explanation of techniques is provided to help you understand markings or notations which may appear on this reproduction. 1. The sign or target for pages apparently lacking from the document photographed is Missing Page(s). If it was possible to obtain the missing page(s) or section, they are spliced into the film along with adjacent pages. This may have necessitated cutting through an image and duplicating adjacent pages to assure you of complete continuity. 2. When an image on the film is obliterated with a round black mark it is an indication that the film inspector noticed either blurred copy because of movement during exposure, or duplicate copy. Unless we meant to delete copyrighted materials that should not have been filmed, you will find a good image of the page in the adjacent frame. 3. When a map, drawing or chart, etc., is part of the material being photographed the photographer has followed a definite method in sectioning the material. It is customary to begin filming at the upper left hand comer of a large sheet and to continue from left to right in equal sections with small overlaps. If necessary, sectioning is continued again beginning below the first row and continuing on until complete. 4. For any illustrations that cannot be reproduced satisfactorily by xerography, photographic prints can be purchased at additional cost and tipped into your xerographic copy. Requests can be made to our Dissertations Customer Services Department. 5. Some pages in any document may have indistinct print. In all cases we have filmed the best available copy. University Micrrifilms International 300 N. ZEEB ROAD, ANN ARBOR. Ml BEDFORD ROW, LONDON WC1R 4EJ, ENGLAND

6 D illon,m ichael Jo s eph COMPONENT ANALYSIS OF A BEHAVIORAL-SUPERVISORY SYSTEM FOR MASTERS- AND DOCTORAL-LEVEL RESEARCH Western Michigan University PH.D University Microfilms International 300 N. Zeeb Road, Ann Arbor, MI 48106

7 PLEASE NOTE: In all cases this material has been filmed in the best possible way from the available copy. Problems encountered with this docum ent have been identified here with a check mark V. 1. Glossy photographs or p ag es 2. Colored illustrations, paper or print 3. Photographs with dark background 4. Illustrations are poor copy 5. Pages with black marks, not original copy 6. Print show s through a s there is text on both sides of page_; 7. Indistinct, broken or small print on several pages ^ 8. Print exceeds margin requirem ents 9. Tightly bound copy with print lost in spine 10. Computer printout pages with indistinct print 11. Page(s) lacking when material received, and not available from school or author. 12. Page(s) seem to be missing in numbering only as text follows. 13. Two pages num bered. Text follows. 14. Curling and wrinkled p a g e s 15. Other. University Microfilms International

8 TAB LE OF CONTENTS Acknowledgements.... H L is t of Tables and Figures... iv Chapter I INTRODUCTION... 1 I I METHOD... 5 Subjects... 5 Supervisors... 5 Procedures of the Behavioral-System... 6 Experimental Procedures Observation and R e l i a b i l i t y I I I RESULTS Comparisons W ithin th e Behavioral-System Comparison Between the Behavioral-System and the S ta n d a rd -S y s te m IV DISCUSSION W ith in -S y s te m Between S y s t e m s C ost-e ffectiveness Social V alidation Future Research B ibliography i i i -

9 LIST OF TABLES AND FIGURES Table I. Table I I. Table I I I. Table IV. Figure 1. Figure 2. Figure 3. Required weekly research ta s k s... B ehavioral supervisory system components In each of four experiment co n d itio n s... Experim ental m anipulation of fe a tu re s of the b eh av io ral supervisory system... Comparisons between b eh av io ral- and standard-system m asters stu d en ts... Median percentage of required p o in ts completed fo r each experim ental c o n d itio n. Data were c o lle cte d over fo u r y ears w ith 34 M.A. and 5 Ph.D. s tu d e n ts... Percentage of req u ired p o in ts completed fo r fo u r experim ents. H orizontal s o lid lin e s in d ic a te the median p ercentage completed in each phase. These graphs p o rtray Experiments 1, 3, 5, and 7 describ ed in Table I I I... Latency in months from entrance to graduate program to s t a r t of th ree re sea rc h ta sk s. In each comparison, the P o s itiv e and Negative P oints co n d itio n has 7 stu d e n ts, In stru c tio n s has 8 (both groups under one fa c u lty a d v is o r), Standard (employed 20 hours or le ss per week) has 21 stu d en ts (under 10 d iff e r e n t a d v is o r s ), Standard (employed more than 20 hours per week) has 16 (under 6 ad v iso rs, 4 of whom overlap both Standard groups)... iv

10 CHAPTER I Introduction Educators and ad m in istrato rs have been concerned w ith the success r a te s, a t t r i t i o n r a te s, and length of time in th e program of graduate stu d en ts (B erelson, 1960; H eiss, 1970; N atio n al Science Board, 1969; Spurr, 1970; W ilson, 1965). G enerally, these a u th o rs agree th a t not enough m a tric u la ted stu d en ts graduate, th a t th o se who do graduate take a long tim e, and th a t too many students drop from th e ir program w ithout a degree. The a u th o rs a lso suggest a number of improvements in graduate tra in in g procedures and recommend th e ir im plem entation and ev aluation. A frequent and strong suggestion is th a t the graduate s tu d e n t's major advisor should provide stru ctu red supervision and guidance in the form of re g u la r c o n su lta tio n meetings (Benet, 1977; B erelson, 1960, p. 235; H eiss, 1970, p. 284; Spurr, 1970, pp ; Seeman, 1973; S p riestersb ach & Henry, 1978). According to H eiss (1970, p. 222), "nearly 80% of the stu d en ts reported th a t a d v iso rs fa ile d to schedule reg u lar m eetings fo r the purpose of checking th e c a n d id a te 's progress or n eed s." Seeman (1973, p. 905) recommended a research -su p erv iso ry approach in which " stru c tu re and a c c o u n ta b ility a re c le a rly defined and (fa c u lty ad v iso r) involvement is h ig h." Benet (1977, p. 384) suggested re g u la r m eetings between the student and advisor to discuss the follow ing issu e s defining a research problem, implementing and w ritin g the re se a rc h, coursework and o ther req u irem en ts, and 1

11 p ro fessio n al and p ersonal growth and development. ( I t should be noted th a t the above re p o rts a re not experim ental s tu d ie s; th e a u th o rs' recommendations are based on d e sc rip tiv e s t a t i s t i c s of stu d en t a t t r i tio n and success r a te s, and the a u th o rs' su b jectiv e im p ressio n s.) Previous research in the area of improving graduate tra in in g has emphasized the v a lid a tio n of s e le c tio n -fa c to rs, ra th e r than the evalua tio n of a c tu a l tra in in g procedures. These few stu d ie s have been lim ited to : 1) fu rth e r documentation of the a t t r i t i o n problem in docto r a l programs (Ph.D. stu d en ts in Psychology, Economics and Business programs have shown a t t r i t i o n ra te s of 35-60%); and 2) s t a t i s t i c a l treatm ents in which stu d en ts and program c h a r a c te r is tic s were c o rre la te d w ith eventual stu d en t success in a tta in in g the Ph.D. degree. (No c h a r a c te r is tic s have been c o n siste n tly a sso ciated w ith a tta in in g th a t degree.) (Decker, 1973; Knox, 1970; Lunneborg & Lunneborg, 1973; Pogrow, 1978; W right, 1964) To the b est of th e ex p erim en ter's knowledge, th e re ap p aren tly has been no experim ental ev alu atio n of the graduate tra in in g procedures recommended in the above re p o rts. The p resent research attem pted to address th is issu e through a s e rie s of stu d ie s. The major recommendatio n s on tra in in g seemed to be these involvement of th e fa c u lty adv is o r, reg u lar c o n su lta tio n m eetings, and stru c tu re d re sea rc h - supervision. T herefore, the experim enter designed a supervisory system with these fiv e components: weekly (a) s p e c ific a tio n of re sea rc h -ta sk s and performance stan d ard s, (b) m eetings w ith a superv iso r (e ith e r fa c u lty or d o c to ra l-s tu d e n t), (c) d ead lin es, (d) feedback, and (e) in c e n tiv e s.

12 In the p re sen t view of research production, th e s tu d e n ts ' research time and e f f o r t a re considered in p u ts (or c o sts) to th e supervisory system; re c e ip t of a degree is considered the output (or value) of the system (Decker, 1973; G ilb e rt, 1978, Chapter 1 ). Thus, th e supervisory system would produce th e g re a te s t b e n e fit when more of th e e n ro lle d - students graduate and when they do so in le s s tim e. (Assuming, of \ course, th a t o th er requirem ents of the system and departm ent have a lso been met fo r in sta n c e, the q u a lity of the s tu d e n t's re sea rc h is accep t a b le, the co st of the supervision time is low, o th e r coursework has been completed, e t c. ). A prelim inary ev alu atio n of the components of th e supervisory system has shown th a t p o s itiv e and n eg ativ e p o in ts toward a l e t t e r of recommendation combined w ith weekly feedback a re e f f e c tiv e in c o n tro l lin g the research behavior of M.A. stu d en ts (D illo n, Kent & M alott, 1980); and a second study has shown th a t the p o in ts alone a re e ffe c tiv e in c o n tro llin g the re sea rc h behavior of B.A. stu d en ts (Gant, D illon & M alott, 1980). The p resent study evaluated the sep arate e f f e c ts of th e p o s itiv e p o in ts, the n eg ativ e p o in ts, the feedback, and th e ta s k -d e s c rip tio n s and performance g u id e lin e s on re se a rc h -ta sk com pletion of M.A. and a lso Ph.D. stu d en ts in Psychology. Measures of system -perform ance and p a rtic ip a n t support were then compared between M.A. stu d en ts in th is beh av io ral-su p erv iso ry system and M.A. stu d en ts under o th e r, more tr a d itio n a l supervisory systems in the departm ent. The perform ance measures included la te n cy to s t a r t research a c t i v i t i e s, q u a lity

13 4 r a tin g s, success and a t t r i t i o n r a te s, and le n g th of time to graduate; th e measure of p a rtic ip a n t support was th e ir e v alu atio n of the q u an tity and q u a lity of supervision received by th e stu d en t.

14 CHAPTER II Method S ubjects Across four y ears of research, the su b je c ts were 34 M.A. and 5 Ph.D. stu d en ts under one fa c u lty ad v iso r in the Psychology Department a t Western Michigan U n iv ersity. A ll were e n ro lle d in the Applied Behavior A nalysis program, and a l l received th e s is or d is s e rta tio n c re d it fo r p a rtic ip a tin g in the a d v is o r 's research -su p erv iso ry system. Each student then provided informed co n sen t, by which he or she agreed to p a rtic ip a te in th is research involving m anipulation of components of the system. The d ep artm ent's Human S ubjects Committee reviewed th is research p rio r to implem entation. In a d d itio n, the experim enter examined the American Psychological A ssociation g u id elin es (APA, 1973, Section 5) on assu rin g su b jects the freedom to p a r tic ip a te in research w ithout coercion. A nalysis of the g u id e lin e s in d icated th a t th is re search did safeguard the s u b je c t-rig h ts. Supervisors The fiv e Ph.D. stu d en ts also supervised th e M.A. stu d en ts on th e ir th e s is re sea rc h, and the fa c u lty a d v iso r, in tu rn, supervised each of these Ph.D. stu d en ts. A ll Ph.D. stu d en ts received practicum c re d it fo r p a rtic ip a tin g as su p erv iso rs. The re sea rc h in te r e s t of the M.A. student u su ally determined which Ph.D. stu d en t would serve as the 5

15 su p erv iso r. Throughout th ese s tu d ie s, the Ph.D. stu d en ts supervised 3-4 M.A. stu d en ts; the ad v iso r supervised a maximum of four Ph.D. stu d en ts a t any one tim e. As a Ph.D. stu d en t, the experim enter was one of the su p erv iso rs, but h is data were not included as a su p erv isee. Procedures of the Behavioral-System Dependent V ariab les. The M.A. and Ph.D. stu d en ts receiv ed the follow ing task d e sc rip tio n s (abbreviated here fo r c l a r i t y ) : 1. Individual m eeting attendance: students met weekly fo r oneh a lf hour on an in d iv id u al b a s is w ith th e ir su p erv iso r, e ith e r the Ph.D. student supervisor or fa c u lty su p erv iso r. 2. Review a r t i c l e : the stu d en ts were req u ired to read two a r t i c l e s per week and to w rite a minimum of 100 words on a l i t e r a t u r e - review form fo r each a r t i c l e. Students read a r t i c l e s re le v a n t to th e ir research to p ic; and they discussed th e a r t i c l e in the. m eeting, though w ith no p o in ts con tin g en t. 3. Data p re sen ta tio n : stu d en ts presented new data each week, i f they were implementing th e ir research. 4. Log: th is ta sk was o p tio n a l. The 125-word log l i s t e d id eas, procedures and procedure changes, and re le v an t statem ents th a t came from the stu d e n t s research m eetings and from o th er courses and fa c u lty members. 5. Hours: stu d en ts summed the to ta l number of hours worked on research a c t i v i t i e s fo r the week. Students who were e n ro lle d fo r th e s is or d is s e rta tio n c re d it worked a minimum of 12 hours p er week;

16 students n o t c u rre n tly en ro lled fo r c re d it s p e c ifie d a minimum time requirem ent, u su ally between 6-12 hours. 6. Output graphs: students p lo tte d t h e ir hours worked per week on research a c t i v i t i e s and th e ir personal and group performance from the weekly feedback form. 7. W riting: th is task involved the form al, m anuscript w rite-up, e ith e r 1000 new words or rew ritin g s. 8. E d itin g : stu d en ts ed ited th e ir own w ritin g based on th ree requirem ents desiged to improve re a d a b ility. 9. Research proposal: the stu d en ts provided a 200-word statem ent of the g eneral problem, the s e ttin g th a t they worked in, and sp ecific research recommendations. 10. N on-recurring ta sk s: the supervisor used th is c a tc h a ll c a te gory to sp ecify ta sk s th a t occurred only once (fo r example, the candidacy or diploma a p p lic a tio n, preparing a sp e c ia l graph). 11. Large-group meeting attendance: stu d en ts met weekly fo r two hours w ith a l l o th er M.A. and Ph.D. stu d en ts and the fa c u lty advisor. Once a sem ester, each student presented h is or her research proposal or r e s u lts, follow ed by a round-robin d iscu ssio n. The su p erv iso rs checked a l l of the above ta sk s in the individual meeting, except fo r large-group attendance. In a d d itio n to these ta sk s, each M.A. stu d en t a lso met in d iv id u ally w ith the fa c u lty advisor every th ree weeks, b ut w ith no p o in ts contingent. Students worked on those task s re le v a n t to th e ir le v el of progress in com pleting t h e ir th e s is or d is s e rta tio n re sea rc h. Those who ju s t f

17 entered the program c o n s titu te d th e G enerating Group, as they were generating research q u estio n s fo r p o ssib le im plem entation. Students w ith an accepted research q u estio n were in th e Implementing Group, conducting th e ir research ; stu d en ts w ith accepted r e s u lts were in the W riting Group, preparing the m anuscript. See Table I fo r the ta sk re quirem ents fo r each group. Point and Feedback Systems. Where the experim ental co n d itio n s allow ed, the supervisors assigned two kinds of p o in ts to the s tu d e n t's performance: p o sitiv e and n e g ativ e. The student earned one p o s itiv e po in t (+3 fo r the w ritin g and e d itin g task s) i f he or she met the c rite rio n on the req u ired research ta sk, and th e student earned one negative p o in t (-3 fo r w ritin g and e d itin g ) i f he or she did not meet the c r ite r io n. The p o in t requirem ent fo r com pletion of the w ritin g and e d itin g task s was h ig h er so th a t i t more n early re fle c te d the r e la tiv e amount of behavior involved in com pleting these ta sk s. The fa c u lty advisor used the percentage of p o s itiv e and n eg ativ e p o in ts earned by the stu d en t in any requested l e t t e r s of recommendation. The experim enter c a lc u la te d th e percentage of p o in ts completed fo r each.student by d iv id in g the number of p o s itiv e p o in ts earned in a week by the to ta l number of req u ired p o in ts fo r th a t week (x 100). I t was p o ssib le fo r the student to do ex tra ta sk s each week, thus earning ex tra p o s itiv e p o in ts and in creasin g h is or her percentage completed. The percentage not completed was fig u red by d iv id in g the number of negative p o in ts by th e number of required p o in ts (x 100). However, if the student missed a deadline and earned a n egative p o in t,

18 Table I. Required weekly research tasks.

19 10 Table I Required Weekly Research Tasks Group Tasks Generating Implementing Writing Individual meeting attendance3 X X X a Review article X X Data presentation X Log b Hours X X X Output graphs X X X Writing3 Editing3 X X Research proposal0 X Non-recurring3 X X X Large-group attendance X X X atasks for which extra positive points were possible. ^Optional task. (» Required once, unless topic c h a n g e d.

20 11 then h is or her percentage not completed always showed a n egative p o in t, re g a rd le ss of the number of ex tra p o in ts earned. A computer program (D illon & van Haaren, in p ress) managed the weekly feedback procedures, providing the stu d en t w ith h is or her p o in t percentages fo r th e week and cum ulative t o t a l s on th e student and h is or her group (e ith e r M.A. or Ph.D.). Students could postpone deadlines i f they had the consent of th e ir supervisor 24 hours in advance of the d ead lin e, " i f unavoidable c i r cumstances occurred". Experim ental Procedures Comparisons w ithin the Behavioral-System. At a large-group meeting b efo re the s t a r t of each experim ent, new and old stu d en ts received w ritte n handouts d escrib in g the dependent v a ria b le s, general procedures, experim ental procedures (the type of design or len g th of phases were n o t described only d e sc rip tio n s of the experim ental c o n d itio n s), and meeting schedules. The experim enter then gave a p r e te s t to the stu d en ts over the requirem ents of the supervisory system. The experim ents s ta rte d a f te r the p a rtic ip a n ts viewed th e ir sco res and the c o rre ct answers fo r the q u estio n s (range of means 81-89% c o rre c t). Also a t the s t a r t of each study, the experim enter to ld the students th a t the fa c u lty ad v iso r considered acceptable perform ance to be the completion of 90% (or h igher) of th e ir req u ired p o in ts. Table I I shows the experim ental co n d itio n s manipulated during th e research and the supervisory system components in e ffe c t. Table I I I summarizes four years of re se a rc h, showing seven

21 Table I I. B ehavioral supervisory system components in each of four experim ental co n d itio n s. t

22 13 Table I I B ehavioral Supervisory System Components in Each o f Four Experim ental C onditions Experim ental C onditions Components Pos & Neg P o in ts P o s itiv e P oints Feedback In s tru c tio n s - Only S p e c ific a tio n X X X X Meetings X X X X lin e s X X X X Feedback X X X Consequences P o sitiv e Points X X Negative P oints X

23 Table I I I. Experim ental m anipulation of fe a tu re s of the b eh avioral supervisory system.

24 15 Table I I I Experim ental M anipulation of F e a tu re s o f the B ehavioral Supervisory System Experiment Design3 Phase Length in Weeks N Groupk E xperim ental H isto ry Median % Completed fo r Phase 1 BAB T o tal Naive C BAB T o tal 14 from #1; 1 n aiv e BA'B T o tal Naive BB' G Naive B'B W, I 4 from #3; 6 from #4 6 BB* W, I 3 from //3; 4 from # AB' G Naive a In which B P o s itiv e and Negative P o in ts B* P o s itiv e P oints A ' Feedback. A In stru etio n s-o n ly ^ In which T o ta l Groups G, W and I. c Experim ent #2 also appears in D illon, Kent & M alo tt, 1980.

25 16 experim ents involving the four experim ental c o n d itio n s and the 39 subj e c t s. For in stan ce, Experiment 1 used a re v e rsa l design w ith three phases (P o sitiv e and Negative P o in ts In s tru c tio n s Only P o sitiv e and Negative P o in ts); each phase la s te d fo u r weeks. S ubjects were 15 students w ith no experience in th is supervisory system, and they represented the th ree le v e ls of research progress and a l l stu d en ts. Experiments 1-4 are l i s t e d as they occurred chronologic a lly ; Experiment 7 ran concurrently w ith Experiments 5 and 6. (The l a s t column showing median performance in each phase is described in the R esu lts s e c tio n.) At the end of each experim ent, p a rtic ip a n ts anonymously ra te d the ta sk s and procedures of the system on a fiv e -p o in t sc a le, in d ic a tin g the value in helping them complete th e ir re search and a lso the p le a sa n t n ess. (For example, L ite ra tu re Review task : Valuable A B C D E W orthless; Rewarding A B C D E A versive.) This measure of p a rtic ip a n t support is a recommended type of s o c ia l v a lid ity procedure (Kazdin, 1977). Comparisons Between the B ehavioral- and Standard-System. Several comparisons were made between the M.A. stu d en ts in the B ehavioral- System and other Applied Behavior A nalysis M.A. stu d en ts under other fa c u lty ad v iso rs in the departm ent, grouping a l l of these stu d en ts in to a 'S tandard' Supervisory System. On a q u e stio n n a ire, stu d en ts in re quired graduate c la s s e s indicated when they were adm itted to the graduate program and under which fa c u lty a d v iso r, when ( i f a t a l l ) they had s ta rte d the various research ta s k s, and th e ir ra tin g of the

26 q u an tity and q u a lity of research su p erv isio n. This method of d is tr ib u tio n did not provide a random sampling of a l l Applied Behavior A nalysis students a v a ila b le from a l l supervisory systems p resen t in the d ep artment, but a good c ro ss-se c tio n of stu d en ts (and th e re fo re, supervisory systems) was c o lle c te d. The Behavioral-System stu d en ts a lso f i l l e d out the q u estio n n aire. To measure the q u a lity of th e s is re search, o ra ls committee members ra te d fe a tu re s of the o ra ls document including a p p ro p ria te data analys i s, experim ental c o n tro l shown, g e n e r a liz a b ility of r e s u l t s, and overa l l q u a lity of th e w rite-u p and o ra ls defense. A fiv e -p o in t scale was used (Poor A B C D E O utstanding). The ra tin g was done p riv a te ly ; n e ith e r the student nor the o th er committee members viewed th e r a tin g s. Although c o lle c te d, th e committee c h a ir s data were not included in the comparisons. In a second measure of q u a lity, the experim enter t a l l i e d the number of a r t i c l e s subm itted to m ajor, re fe re ed jo u rn a ls in the f ie ld by graduates of the B ehavioral- and Standard-System s. The experim enter asked the o ra ls committee c h a ir (the s tu d e n t's major ad v iso r) fo r the inform ation on submission and e d ito ra l d e cisio n. To measure the percentage of stu d en ts graduated and th e ir d u ratio n in the program, the experim enter compared o f f i c i a l department adm ittance records w ith u n iv e rs ity graduation reco rd s. The experim enter monitored Standard-System stu d en ts 4 to 5 y ears (from adm ittance in e ith e r 1975 or 1976 u n t i l 1980, when th is research was term in ated ), and Behavioral-System stu d en ts 2-4 y ears (from adm ittance in u n til 1980).

27 18 Observation and R e lia b ility The su p erv iso rs served as the primary o b serv ers. The tra in in g of the Ph.D. stu d en t observers involved reading th e w ritte n handouts on the supervisory system 's procedures and requirem ents, taking the p r e te s t (range of means 90-93%), and viewing the answ ers. The experim enter served as the secondary r e l i a b i l i t y observer. On an unannounced b a s is a cro ss the experim ents, he atten d ed in d iv id u al and larg e group m eetings to assess the work (or presence) of the M.A. or Ph.D. stu d en t. In te ra c tio n between the secondary observer and the p a rtic ip a n ts in the in d iv id u al meeting was b r ie f (two to four minutes) and sp e c ific d iscu ssio n was avoided. The o th e r su p erv iso rs acted as the secondary observer fo r M.A. students whom th e experim enter supervised. The experim enter used a r e l i a b i l i t y c a lc u la tio n (B ijou, Peterson & A ult, 1968) in which th e r e l i a b i l i t y percentage equalled the number of agreements divided by the to ta l number of agreem ents and d isag reements (x 100). The number of agreements and disagreem ents was calcu lated follow ing a comparison of the secondary o b s e rv e r's r e l i a b i l i t y sheet w ith the s u p e rv iso r's data sheet. The o v e ra ll r e l i a b i l i t y percentage on th e dependent v a ria b le s fo r the seven experim ents was 90%, w ith a range from 88% to 96%. The experim enter sampled su p erv iso rs on each re se a rc h ta sk in each phase of the experim ents, checking a median of the ta s k occurrences (range across ta sk s of 16-31%). R e lia b ility p ercentages on the stu d en ts the experim enter supervised showed no d ifferen ce when compared to the percentages of the o th e r supervisors.

28 CHAPTER III R esu lts Comparisons Within the Behavioral-System G enerally, the s tu d e n ts perform ance was h ig h est when a l l compon e n ts of the system were p re sen t; i t d e te rio ra te d as each major compon en t was removed. The M.A. and Ph.D. stu d en ts completed over 90% of t h e ir tasks when p o in ts were in e f f e c t [though negative p o in ts (range %) added nothing to the perform ance measured under p o sitiv e p o in ts (89-98%)]. The performance decreased to 76% during the one phase where points were removed and only feedback was p re sen t. I t decreased even fu rth e r to 66% when the feedback was also removed, leaving only the in stru c tio n s (range 64-73%). (See Figure 1.) There was a s ig n ific a n t d ifferen ce among th e p o in t, the feedback-only, and the in stru c tio n s-o n ly conditions (the experim enter used a median te s t : 2 = 6.92, df = 2, p =.0 5 ); th e re was a lso no overlap in the ranges among the three co n d itio n s. The la s t column in Table I I I shows the median percentage of p o in ts completed per phase fo r each of th e experim ents. In Figure 2, re p re se n ta tiv e data are shown of the fo u r experim ental comparisons: P o sitiv e and Negative P oints w ith In s tru c tio n s, P o sitiv e and N egative P o in ts w ith Feedback, P o sitiv e and N egative P o in ts w ith P o sitiv e P o in ts, and P o sitiv e Points w ith In s tru c tio n s. (No experim ental comparisons were made of the Feedback condition w ith the P o sitiv e P o in ts co n d itio n 19

29 Figure 1. Median percentage of req u ired p o in ts completed fo r each experim ental co n d itio n. Data were c o lle c te d over four years from 34 M.A. and 5 Ph.D. stu d en ts. The P o sitiv e and N egative P o in ts co n d itio n co n tain s 9 phase medians, P o sitiv e P o in ts has 4, Feedback has 1, In stru ctio n s-o n ly has 4.

30 LU o <_} 100 POS & NEG POSITIVE INSTRUCTIONS- POINTS POINTS FEEDBACK ONLY CO I CO «I LU CO o «=c O I 50 LU QC CJ OS U. O LU O-

31 Figure 2. Percentage of required p o in ts completed fo r four experiments. H orizontal so lid lin e s in d ic a te the median percentage completed in each phase. These graphs p o rtray Experiments 1, 3, 5, and 7 described in Table I I I.

32 23 POS I NEG POINTS INSTRUCS POS 1 NEG POINTS POSITIVE POINTS POS t NEC POINTS A PERCENT OF POINTS COMPLETED exp n POS & NEG POINTS FEEDBACK \ POS I NEG POINTS EXP #5 INSTRUCTIONS POSITIVE POINTS!;7 ^ ^ x EXP #3 EXP # A 8 T" 12 1 R 12 W EEKS G O

33 24 or Instructions-O nly c o n d itio n.) Not only the average of the ta s k s, but a lso the in d iv id u al ta sk s showed high completion percentages when p o in ts were used. Removal of the points and removal of th e feedback produced decreases in p e rfo r mance for some b u t n o t a l l ta sk s. For in stan ce, in d iv id u al and group meeting attendance continued a t high percentages of com pletion; data presentation a lso remained h ig h, i f the task had been s ta r te d p r io r to the removal of the p o in ts an d /o r feedback; but l i t e r a t u r e review, hours worked, w ritin g and e d itin g a l l f e l l d ram atically. (The remaining four task s were d i f f i c u l t to c ate g o riz e lo g, research proposal and non-recurring ta sk s because they were not required o ften enough, and output graphs because they took only 2-3 minutes per week to complete and thus were not g e n erally s e n s itiv e to the experim ental manipulatio n s. ) There was no d iffe re n c e between the performance of the Ph.D. students as supervisees and the performance of th e M.A. su p erv isees. The- Ph.D. students showed sim ila r high performance under the complete system and a sim ila r d e te r io ra tio n as components were removed. The so cial v a lid ity a n a ly sis showed th a t the M.A. and Ph.D. s tu dents generally supported a l l fe a tu re s of the supervisory system, except fo r the use of n e g a tiv e p o in ts. Students ra te d highly th e in d i vidual and group m eeting, p o s itiv e p o in ts toward recommendation, weekly feedback on perform ance, and use of deadlines. C o n siste n tly, the most p o sitiv e ly ra te d fe a tu re was the one-on-one meeting w ith the su p erv iso r. (When th is task was made o p tio n al in the In s tru c t ions-only co n d itio n

34 in Experiment 7, a l l e ig h t students had requested and were holding h a lf - 25 hour weekly m eetings w ithin six weeks in to the sem ester.) The most poorly rated fe a tu re was the negative p o in t procedure; stu d en ts f r e quently sta te d th a t i t s use made the e n tir e system unenjoyable. When th e ir performance dropped to low le v e ls, fiv e M.A. stu d en ts (represented 13% of the adm ittees over the four y ears) requested sp e c ia l, a d d itio n a l beh av io ral c o n tractin g procedures fo r com pleting th e ir research ta sk s. These students supplemented lo s s of money, r e cord albums and/or the use of d aily dead lin es in order to complete th e ir work. Four of the fiv e stu d en ts requested these sp ec ia l procedures during the w ritin g of th e ir m anuscripts. (None of the co n tract o r s ' data a re included in the R esults sectio n d a ta.) Comparison Between the Behavioral-System and the Standard-System Students in the Behavioral-System s ta r te d work on th e s is a c t i v i t i e s sooner a f te r beginning the M aster's Program than Standard- Systera stu d en ts; much more so i f p o in ts were contingent (see Figure 3). Under the B ehavioral-system, a g re a te r percentage of M.A. adm ittees graduated (81% compared to 57% fo r o th er departm ental stu d e n ts), and they did so in le s s time (a median of 20 months compared to 28 months, a reduction of two sem esters). The M.A. scudents in th is system rated more highly the q u an tity and q u a lity of re search - supervision than M.A. stu d en ts under the Standard-System (see Table IV).

35 Figure 3. Latency in months from en tran ce to graduate program to s t a r t of th ree research ta s k s. In each comparison, the P o sitiv e and Negative P o in ts co n d itio n has 7 stu d en ts, In s tru c tio n s has 8 (both groups under one fa c u lty advisor) Standard (employed 20 hours or le s s per week) has 21 stu dents (under 10 d if f e r e n t a d v is o rs ), Standard (employed more than 20 hours per week) has 16 (under 6 ad v iso rs, 4 of whom overlap both Standard g ro u p s). t

36 100' KEY: 50 LITERATURE REVIEW POS t NEG POINTS O INSTRUCTIONS-ONLY M STANDARD ( HRS) STANDARD HRS) PERCENT OF STUDENTS [ It I I I ll fll 12 * 1 6 NOT STARTED DATA ' COLLECTION WRITING 1 k ] l ill > 12 * 1 6 NOT STARTED 0 - H I! i " i " i A 12 * 1 6 II. NOT STARTED LATENCY TO START IN MONTHS

37 28 Table IV. Comparisons between B ehavioral- and Standard-System M asters stu d en ts.

38 29 Table IV Comparisons Between B ehavioral- and Standard- System M asters Students Supervisory Systems Comparisons Behavioral Standard Evaluation by Students: 1) Quantity of su p erv isio n 91% Q Q uality of su p erv isio n 91% N (stu d en ts) 11 54% 70% 16 Median months in program a t time of ev alu atio n Range 8 mos mos. 16 mos mos. Evaluation by fa c u lty : a 2) O verall q u a lity of research 61% a O verall q u a lity of w rite-u p 70% O verall q u a lity of s tu d e n t's o ra ls defense3 74% N (stu d en ts) 12 N (advisors) 1 60% 85% 75% 13 6 From U niversity reco rd s: 3) Students graduated 81% N (stu d en ts e n ro lle d ) 20 57% 50 Percentage of r a t e r s c ir c lin g 'A' or 'B '; these a re th e l e t t e r s a t the p o s itiv e end of th e ra tin g scale.

39 30 TABLE IV (Continued) Supervisory Systems Comparisons Behavioral Standard Median months to earn degree 20 mos. 28 mos. Range mos mos Students s t i l l in program 19% 20% Median months a t time of m onitoring records 24 mos. 44 mos. Range mos mos Students who have dropped from program 0% 23% t

40 31 Two measures of q u a lity showed th a t research from the B ehavioral- and Standard-Systems was comparable. F i r s t, fa c u lty ad v iso rs evaluated Behavioral-System research as g en erally comparable to th a t produced by the other M.A. g raduates, though the prim ary research su p erv iso rs in th is system were d o cto ral stu d en ts (Table IV). But they did rank quality of the w rite-up higher fo r the Standard-System ; however, th is may r e f le c t more on the somewhat id io sy n c ra tic s ty le the fa c u lty advisor recommends fo r the Behavioral-System stu d en ts than on the supervisory stru c tu re i t s e l f. Second. 21% (3 of 14) of the Behavioral-System graduates published th e ir th e s is re sea rc h, while 17% (7 of 41) of the Standard-Sy stem graduates did so. However, one of the fa c u lty ad v iso rs in the Standard-System accounted fo r 45% of the subm issions and 71% of the acceptances. Several comments on the above data are re le v a n t. Concerning the survey data of Figure 3, i t is p o ssib le th a t the longer laten cy to s t a r t the research task s showed by Standard-System stu d en ts re s u lte d from the fa c t th a t a l l Behavioral-System stu d en ts worked 20 hours or le s s per week in paid a s s is ta n ts h ip or o u tsid e work, w hile some of the Standard-System stu d en ts were employed more than 20 hours per week. Therefore, the experim enter computed la te n c ie s fo r Standard-System students employed the comparable 20 hours or le s s per week and separa te ly fo r those employed more than 20 hours per week. However, as Figure 3 shows, th ere was no ap p reciab le d ifferen c e between the two types of Standard-System stu d en ts in th e ir laten cy to s t a r t the ta sk s. I t is a lso p o ssib le th a t in te rp re ta tio n of these survey data on

41 32 laten cy (Figure 3) was based on an a r t i f i c i a l d iffe re n c e between the groups of stu d en ts from th e B ehavioral- and Standard-System s. As a r e s u lt of th e stu d en ts having entered the M aster's Program in d iff e r e n t academic y e a rs, they rep resen ted d iff e r e n t len g th s of time in the program when surveyed. P o ssib ly, the Standard-System stu d en ts were surveyed sooner a f te r beginning the M aster's Program than the B ehavioral- System stu d en ts. However, the b ia s is ju s t the o p p o site; B ehavioral- System stu d en ts were surveyed an average of 8 months a f te r m a tric u la tio n and the Standard-System stu d en ts an average of 16 months. Latency d ata were n o t c o lle cte d fo r the P o s itiv e P o in ts and Feedback c o n d itio n s, because a s u f f ic ie n tly larg e group of B ehavioral- System stu d en ts was not involved a t the s t a r t of t h e ir M.A. programs. In the th re e comparisons of Table IV, again Behavioral-System stu d en ts were employed 20 hours or le ss per week; and Standard-System stu d en ts were a lso employed 20 hours or le ss per week in the comparison of q u a n tity and q u a lity of su p erv isio n, but the rem aining two comparisons have an u ncontrolled mix of stu d en ts w ith p a rt-tim e and fu ll-tim e employment. For the second comparison of Table IV, re se a rc h from stu d en ts' M.A. o ra ls was reviewed fo r two y e ars. In th is tim e p e rio d, 100% (12 of 12) of the Behavioral-System graduates were ra te d ; however, only 43% (13 of 30) of the Standard-System graduates were ra te d, fo r v ario u s u n co n tro llab le reasons. In the th ir d comparison, the experim enter removed from the a n a ly sis fo u r of the 20 stu d en ts e n ro lled in the Behavioral-System during the th re e years these data were c o lle c te d. These were one student who

42 was employed f u ll-tim e, tak in g 48 months to g rad u ate, and th re e students who switched to Standard-System ad v iso rs. Removed from the 50 stu d en ts e n ro lle d in th e Standard-System over th e two y e a rs were th re e stu d en ts who switched to o th er M.A. programs in th e departm ent a f te r enrollm ent.

43 CHAPTER IV D iscussion W ithin-system There was a c le a r separation of e ffe c ts among the components of the Behavioral-System. In other words, acro ss th e seven experim ents, the percentage of p o in ts completed was remarkably c o n sis te n t fo r a given experim ental co n d itio n, while no overlap occurred in th e ranges of phase medians among the P o in t, Feedback, and In s tru c tio n co n d itio n s. This was tru e in s p ite of the fa c t th at the same stu d en ts were not presen t throughout a l l of the experim ental m anipulations. And, th is was a lso tru e re g a rd le ss of whether the s tu d e n t's i n i t i a l exposure was to the P oint or In s tru c tio n co n d itio n s, whether th e sequence of conditions the student received was negative p o in ts to p o s itiv e, n eg ativ e to feedback, n egative to in s tru c tio n, in s tru c tio n to p o s itiv e, or p o s itiv e to n eg ativ e, and whether th e phase lengths were 3, 10, or 19 weeks in dura tio n. Thus, re p lic a tio n across a v a rie ty of co n d itio n s and su b jects suggests co n sid erab le r e l i a b i l i t y and g e n e ra lity. This i s what Sidman (1960, Chapter 4) c a ll s "system atic re p lic a tio n ". Based on the p re sen t fin d in g s, one might expect 85-90% of adm ittees to the B ehavioral-system to graduate 75% to work a t high r a te s of completion and to enjoy th e ir experience under th e system as described, and 10-15% to request ex tra procedures; th ese stu d en ts would average 20 months in th e M aster's Program, com pleting a c re d it 34

44 35 hour degree requirem ent. For reasons unknown, th e rem aining 10-15% would not fin is h under th is system. (Across the four y e a rs, four M.A. and one Ph.D. stu d en t dropped from the B ehavioral-system ; the Ph.D. student was th e only adm ittee to drop from the program, however. The extent of th e ir o b je c tio n s, i f any, to the requirem ents of the system were u n c le a r.) Between Systems Compared to Standard-System stu d e n ts, Behavioral-System students s t a r t th e sis a c t i v i t i e s sooner a f te r adm ittance; a g re a te r percentage of these stu d en ts graduate; and they do so in le s s tim e. However, the two groups of stu d en ts were d i f f i c u l t to make eq u iv alen t because the stu d en ts were under d if f e r e n t fa c u lty a d v iso rs, and a ls o, the students adm itted to the B ehavioral-system were a non-random sample of the stu dents adm itted to the Applied Behavior A nalysis program. C o st-e ffectiv en ess The cost of implementing the B ehavioral-system under o th er fa c u lty ad v iso rs would be low. For example, consider the advisor to have 9 M.A. and 3 Ph.D. stu d e n ts. L i t t l e development c o sts need occur, since the various components have already been prepared;'*' day-to-day management of the system could be done by an M.A. stu d e n t, re q u irin g 2 hours D escription of the research ta s k s, g eneral procedures, the computer-feedback program (D illon & van Haaren, in p re ss) and output forms a re a v a ila b le from Dr. Richard W. M alo tt, Department of Psychology, Western Michigan U n iv e rsity, Kalamazoo, Michigan

45 36 p er week. The facu lty ad v iso r would provide 5 hours of supervision per week one 30 minute, in d iv id u a l m eeting w ith each of the th ree d o cto ral stu d en ts and w ith th re e of th e M.A. stu d en ts (held tri-w e e k ly ), p lu s 2 hours of group su p erv isio n. The d o cto ral stu d en ts would provide 4*2 hours per week of in d iv id u a l su p erv isio n. This 9*2 hours of supervision time could manage 108 hours of graduate re s e a rc h -a c tiv ity, w ith most stu d en ts com pleting 90% or more of th e ir required p o in ts. (A weekly output of 108 hours is re p re se n ta tiv e of w o rk -to tals recorded by students in th is re se a rc h.) In some graduate programs, a fa c u lty ad v iso r might supervise only M.A. students and no Ph.D. stu d en ts. By using weekly in d iv id u al meetings of 15 minutes with each M.A. s tu d e n t, the Behavioral-System should continue to produce high le v e ls of perform ance, even w ithout Ph.D.- student su p erv iso rs. Thus, the fa c u lty ad v iso r would provide 4*2 hours per week of supervision (in clu d in g b o th in d iv id u al and group time) fo r 9 M.A. stu d en ts. The Behavioral-System would provide o th e r b e n e fits to the advisee in ad d itio n to research su p erv isio n. For example, the in d iv id u al meeting in th is experiment a lso increased fa c u lty involvement w ith the stu dent in o ther im portant a re a s. P e rio d ic a lly, th e student and advisor discussed the s tu d e n t's p ro fe ssio n a l p la n s, o th er coursework, personal r e la tio n s w ithin the supervisory system and the departm ent, and the stu d e n t's general w ell-being (each recommended by Benet, 1977). And other graduate requirem ents which a ls o have l i t t l e formal stru c tu re (as with the research requirem ent) were e a s ily incorporated into the

46 37 in d iv id u al meeting w ith the ad v iso r. A ssista n tsh ip ta sk s, the reading, w ritin g and generating of ideas th a t a lso are p rep arato ry to completing competency examinations and the review paper were discussed re g u la rly. But, reg ard less of the above b e n e f its, in order to implement th is supervisory approach in o th er graduate tra in in g programs, the a d v is o r's supervision time w ill have to be accounted fo r when developing acacemic loads (S priestersbach & Henry, 1978). S ocial V alidation Several procedures increased student support fo r the B ehavioral- System. Across seven s tu d ie s, the p o in t co n d itio n produced the h ighest performance; b u t, of th e two types of p o in ts used, only the p o sitiv e condition would seem most s u ita b le fo r the continued, long-term e f f e c t iveness of the system. Second, the opportunity to earn ex tra p o in ts was viewed as im portant, although no more than 10% ex tra task s were completed by students in any one of the experim ents. However, the stu d en ts valued the f l e x i b i l i t y of being ab le to in crease th e ir percentage over 100% or to r a is e a percentage th a t was below 90%. Fina l l y, the individual and group meeting ta sk s might be considered the "glue" th a t held the e n tir e system to g e th e r. The weekly in d iv id u al m eeting se t the occasion fo r re g u la r d isc u ssio n, problem -solving, and much a tte n tio n and feedback on the s tu d e n t's research progress. And students reported high support fo r the large-group meeting because of two requirem ents each student p resented h is or her research during the sem ester and each audience-member had to make one comment (or more)

47 38 on the research ju s t p resen ted. G enerally, p a rtic ip a n ts in th e Behavioral-System viewed th is stru c tu re d approach to re sea rc h -su p erv isio n as in th e ir b e st in te r e s ts. During one study, the experim enter discussed w ith the stu d en ts the p o s s ib ility of removing the e n tir e supervisory-system s tru c tu re in e f f e c t, rep lacin g the Behavioral-System w ith ty p ic a l Standard-System supervision. Following a group d iscu ssio n, the 11 stu d en ts who were in the experiment a t th a t time voted on the proposal. Eight of the stu d en ts (73%) approved the removal of the s tru c tu re, but only if i t needed to be removed fo r "research purposes". They generally regarded the removal as d etrim en tal to th e ir p rogress in completing th e ir re search. The remaining th ree stu d en ts voted a g ain st the removal under any circum stances; however, th e ir vote was confounded by the fa c t th a t th e percentages of p o in ts completed fo r these students were cu rre n tly below the 90% standard. Future Research Of fu rth e r experim ental in te r e s t would be the re p lic a tio n of th is supervisory system w ith n o n -behaviorally tra in e d graduate stu d en ts. Since the research on the p resen t system impacts mainly on the re q u ire ment of tim elin ess of research a c t i v i t y, research ev alu atin g procedures to improve the q u a lity of th e se s and d is s e r ta tio n s would a lso be imp o rta n t. From the ex p erim en ter's experience, the g re a te st area of p o te n tia l improvement in q u a lity might be in generating an ap p ro p riate research question, one th a t is of p u b lish ab le q u a lity. Perhaps a