American Economic Association - Baylor University

2y ago
15 Views
2 Downloads
487.19 KB
18 Pages
Last View : 17d ago
Last Download : 2m ago
Upload by : Camille Dion
Transcription

American Economic AssociationEvaluating the Econometric Evaluations of Training Programs with Experimental DataAuthor(s): Robert J. LaLondeReviewed work(s):Source: The American Economic Review, Vol. 76, No. 4 (Sep., 1986), pp. 604-620Published by: American Economic AssociationStable URL: http://www.jstor.org/stable/1806062 .Accessed: 15/08/2012 21:13Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at ms.jsp.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact support@jstor.org.American Economic Association is collaborating with JSTOR to digitize, preserve and extend access to TheAmerican Economic Review.http://www.jstor.org

Evaluating the Econometric Evaluations of Training Programswith Experimental DataBy ROBERT J. LALONDE*This paper compares the effect on trainee earnings of an employmentprogramthat was run as a field experiment whereparticipants were randomly assigned totreatment and control groups with the estimates that would have beenproduced byan econometrician. This comparison shows that many of the econometricprocedures do not replicate the experimentallydetermined results, and it suggests thatresearchers should be aware of the potential for specification errors in othernonexperimental evaluations.ric specifications.' The goal is to appraisethe likely ability of several econometricmethods to accurately assess the economicbenefits of employment and training programs.2Section I describes the field experimentand presents simple estimates of the program effect using the experimental data. Sections II and III describe how econometricians evaluate employment and trainingprograms, and compares the nonexperimental estimates using these methods to the experimental results presented in Section I.Section II presents one-step econometricestimates of the program's impact, whilemore complex two-step econometric estimates are presented in Section III. The re-Econometricians intend their empiricalstudies to reproduce the results of experiments that use random assignment withoutincurring their costs. One way, then, toevaluate econometric methods is to comparethem against experimentally determined results.This paper undertakes such a comparisonand suggests the means by which econometric analyses of employment and training programs may be evaluated. The paper compares the results from a field experiment,where individuals were randomly assigned toparticipate in a training program, against thearray of estimates that an econometricianwithout experimental data might have produced. It examines the results likely to bereported by an econometrician using nonexperimental data and the most modern techniques, and following the recent prescriptions of Edward Leamer (1983) and DavidHendry (1980), tests the extent to which theresults are sensitive to alternative economet-'These papers depict a more general crisis of confidence in empirical research. Leamer (1983) argues thatany solution to this crisis must divert applied econometricians from " the traditional task of identifyingunique inferences implied by a specific model to thetask of determining the range of inferences generated bya range of models." Other examples of this literature areLeamer (1985), Leamer and Herman Leonard (1983),and Michael McAleer, Adrian Pagan, and Paul Volker(1985).2Examples of nonexperimental program evaluationsare Orley Ashenfelter (1978), Ashenfelter and DavidCard (1985), Laurie Bassi (1983a,b; 1984), ThomasCooley, Thomas McGuire, and Edward Prescott (1979),Katherine Dickinson, Terry Johnson, and Richard West(1984), Nicholas Kiefer (1979a,b), and Charles Mallar(1978).*Graduate School of Business, University of Chicago,1101 East 58th Street, Chicago, IL 60637. This paperuses public data files from the National SupportedWork Demonstration. These data were provided by theInter-University Consortium for Political and SocialResearch. I have benefited from discussions with MariamAkin, Orley Ashenfelter, James Brown, David Card,Judith Gueron, John Papandreou, Robert Willig, andthe participants of workshops at the universities ofChicago, Cornell, Iowa, Princeton, and MIT.604

VOL. 76 NO. 4LALONDE: EVALUATING ECONOMETRIC EVALUATIONSsults of this study are summarizedin thefinal section.I. The ExperimentalEstimatesThe National Supported Work Demonstration (NSW) was a temporary employment program designed to help disadvantaged workers lacking basic job skillsmove into the labor market by giving themwork experience and counseling in a shelteredenvironment. Unlike other federally sponsored employment and training programs,the NSW program assigned qualified applicants to training positions randomly. Thoseassigned to the treatment group received allthe benefits of the NSW program, while thoseassigned to the control group were left tofend for themselves.3During the mid-1970s, the ManpowerDemonstrationResearch Corporation(MDRC) operated the NSW program in tensites across the United States. The MDRCadmitted into the program AFDC women,ex-drug addicts, ex-criminal offenders, andhigh school dropouts of both sexes.4 Forthose assigned to the treatment group, theprogram guaranteed ajob for 9 to 18 months,depending on the target group and site. Thetreatment group was divided into crews ofthree to five participants who worked to-3Findings from the NSW are summarized in severalreports and publications. For a quick summary of theprogram design and results, see Manpower Demonstration Research Corporation (1983). For more detaileddiscussions see Dickinson and Rebecca Maynard (1981);Peter Kemper, David Long, and Craig Thomton (1981);Stanley Masters and Maynard (1981); Maynard (1980);and Irving Piliavin and Rosemary Gartner (1981).4The experimental sample included 6,616 treatmentand control group members from Atlanta, Chicago,Hartford, Jersey City, Newark, New York, Oakland,Philadelphia, San Francisco, and Wisconsin. QualifiedAFDC applicants were women who (i) had to be currently unemployed, (ii) had spent no more than 3months in a job in the previous 6 months, (iii) had nochildren less than six years old, and (iv) had receivedAFDC payments for 30 of the previous 36 months. Theadmission requirements for the other participants differed slightly from those of the AFDC applicants. For amore detailed discussion of these prerequisities, seeMDRC.605gether and met frequently with an NSWcounselor to discuss grievancesand performance. The NSW programpaid the treatment group members for their work. Thewage scheduleofferedthe traineeslowerwagerates than they would have received on aregular job, but allowed their earnings toincrease for satisfactoryperformanceand attendance. The trainees could stay on theirsupportedworkjobs until their termsin theprogram expired and they were forced tofind regularemployment.Although these general guidelines werefollowed at each site, the agenciesthat operated the experiment at the local level provided the treatment group members withdifferentwork experiences.The type of workeven varied within sites. For example, someof the trainees in Hartfordworked at a gasstation, while others worked at a printingshop.5 In particular,male and female participantsfrequentlyperformeddifferentsortsof work. The female participants usuallyworked in service occupations,whereas themale participants tended to work in construction occupations. Consequently, theprogram costs varied across the sites andtarget groups. The programcost 9,100 perAFDC participantand approximately 6,800for the other targetgroups'trainees.6The MDRC collectedearningsand demographic data from both the treatment andthe control group membersat the baseline(when MDRC randomly assigned the participants) and every nine months thereafter,conducting up to four post-baseline inter-5Kemper and Long present a list of NSW projectsand customers (1981, Table IV.4, pp. 65-66). Thetrainees produced goods and services for organizationsin the public (42 percent of program hours), nonprofit(29 percent of program hours), and private sectors.6 The cost per training participant is the sum ofprogram input costs, site overhead costs, central administrative costs, and child care costs minus the value ofthe program's output. These costs are in 1982 dollars. Ifthe trainees' subsidized wages and fringe benefits areviewed as a transfer instead of a cost, the program costsper participant are 3,100 for the AFDC trainees and 2,700 for the other trainees. For a more detailed discussion of program costs and benefits, see Kemper,Long, and Thornton.

606THE AMERICAN ECONOMIC REVIEWSEPTEMBER 1986TABLE 1 -THESAMPLE MEANS AND STANDARD DEVIATIONS OFPRE-TRAINING EARNINGS AND OTHER CHARACTERISTICS FORTHE NSW AFDC AND MALE PARTICIPANTSFull National Supported Work SampleAFDC ParticipantsVariableAgeYears of SchoolProportionHigh School DropoutsProportion MarriedProportion BlackProportion HispanicReal Earnings1 year BeforeTrainingReal Earnings2 years BeforeTrainingHours Worked1 year BeforeTrainingHours Worked2 years BeforeTrainingMonth of Assignment(Jan. 78 0)Number .46).02(.15).84(.37).12(.32) 393(1,203)[43] 0Male 0.27(2.00).69(.46).04(.20).82(.39).13(.33) 395(1,149)[41] e: The numbers shown in parentheses are the standard deviations and those in thesquare brackets are the standard errors.views. Many participants failed to completethese interviews, and this sample attritionpotentially biases the experimental results.Fortunately the largest source of attritiondoes not affect the integrity of the experimental design. Largely due to limited resources, the NSW administrators scheduleda 27th-month interview for only 65 percentof the participants and a 36th-month interview for only 24 percent of the non-AFDCparticipants. None of the AFDC participants were scheduled for a 36th-month interview, but the AFDC resurvey during thefall of 1979 interviewed 75 percent of thesewomen anywhere from 27 to 44 months afterthe baseline. Since the trainee and controlgroup members were randomly scheduledfor all of these interviews, this source ofattrition did not bias the experimentalevaluation of the NSW program.Naturally, the program administrators didnot locate all of the participants scheduledfor these interviews. The proportion of participants who failed to complete scheduledinterviews varied across experimental group,time, and target group. While the responserates were statistically significantly higherfor the treatment as opposed to the controlgroup members, the differences in responserates were usually only a few percentagepoints. For the 27th-month interview, 72percent of the treatments and 68 percent ofthe control group members completed interviews. The differences in response rates were

607LALONDE: EVALUATING ECONOMETRIC EVALUATIONSVOL. 76 NO. 4EARNINGS OF NSW TREATMENTS, CONTROLS, ANDTABLE 2-ANNUALEIGHT CANDIDATE COMPARISON GROUPS FROM THE PSID AND THE CPS-SSAComparison Groupa,bYear19751976197719781979Number PSID-4CPSSSA-1CPSSSA-2CPSSSA-3CPSSSA-4 895(81) 1,794(99) 6,143(140) 4,526(270) 4,670(226) 877(90) 646(63) 1,518(112) 2,885(244) 0058559517311825511,1322411,59487aThe Comparison Groups are defined as follows: PSID-1: All female household heads continuously from 1975through 1979, who were between 20 and 55-years-old and did not classify themselves as retired in 1975; PSID-2:Selects from the PSID-1 group all women who received AFDC in 1975; PSID-3: Selects from the PSID-2 all womenwho were not working when surveyed in 1976; PSID-4: Selects from the PSID-1 group all women with children,none of whom are less than 5-years-old; CPS-SSA -1: All females from Westat CPS-SSA sample; CPS-SSA -2:Selects from CPS-SSA-1 all females who received AFDC in 1975; CPS-SSA-3: Selects from CPS-SSA-1 all femaleswho were not working in the spring of 1976; CPS-SSA -4: Selects from CPS-SSA-2 all females who were not workingin the spring of 1976.bAll earnings are expressed in 1982 dollars. The numbers in parentheses are the standard errors. For the NSWtreatments and controls, the number of observations refer only to 1975 and 1979. In the other years there are fewerobservations, especially in 1978. At the time of the resurvey in 1979, treatments had been out of Supported Work foran average of 20 months.larger across time and target group. Forexample, 79 percent of the scheduled participants completed the 9th-month interview, while 70 percent completed the 27thmonth interview. The AFDC participantsresponded at consistently higher rates thanthe other target groups; 89 percent of theAFDC participants completed the 9th-monthinterview as opposed to 76 percent of theother participants. While these response ratesindicate that the experimental results may bebiased, especially for the non-AFDC participants, comparisons between the baselinecharacteristics of participants who did anddid not complete a 27th-month interviewsuggest that whatever bias exists may besmall.77This study evaluates the AFDC females separatelyfrom the non-AFDC males. This distinction is commonin the literature, but it is also motivated by the differences between the response rates for the two groups.Table 1 presents some sample statisticsdescribing the baseline characteristics of theAFDC treatment and control groups as wellas those of the male NSW participants inthe other three target groups.8 As would beexpected from random assignment, theThe Supported Work Evaluation Study (Public UseFiles User's Guide, Documentation Series No. 1, pp.18-27) presents a more detailed discussion of sampleattrition. My working paper (1984, tables 1.1 and 2.3),compares the characteristics and employment history ofthe full NSW sample to the sample with pre- andpostprogram earnings data. Randall Brown (1979) reports that there is no evidence that the response ratesaffect the experimental estimates for the AFDC womenor ex-addicts, while the evidence for the ex-offendersand high school dropouts is less conclusive.'The female participants from the non-AFDC targetgroups were not surveyed during the AFDC resurvey inthe fall of 1979 and consequently do not report 1979earnings and are not included with the AFDC sample.Excluding these women from the analysis does notaffect the integrity of the experimental design.

SEPTEMBER 1986THE AMERICAN ECONOMIC RE VIEW608TABLE 3-ANNUAL EARNINGS OF NSW MALE TREATMENTS,CONTROLS,ANDSIX CANDIDATECOMPARISONGROUPS FROMTHE PSID AND D-1PSID-2PSID-3CPS-SSA-1CPS-SSA-2CPS-SSA-31975 3,066(283) 4,035(215) 6,335(376) 5,976(402) 3,027(252) 2,121(163) 3,403(228) 7(431)2,493253128197619771978Number ofObservations29742515,9921,283305aThe Comparison Groups are defined as follows: PSID-1: All male household heads continuously from 1975through 1978, who were less than 55-years-old and did not classify themselves as retired in 1975; PSID-2: Selectsfrom the PSID-1 group all men who were not working when surveyed in the spring of 1976; PSID-3: Selects fromthe PSID-1 group all men who were not working when surveyed in either spring of 1975 or 1976; CPS-SSA-1: Allmales based on Westat's criteria, except those over 55-years-old; CPS-SSA-2: Selects from CPS-SSA-1 all males whowere not working when surveyed in March 1976; CPS-SSA-3: Selects from the CPS-SSA-1 unemployed males in1976 whose income in 1975 was below the poverty level.bAll earnings are expressed in 1982 dollars. The numbers in parentheses are the standard errors. The number ofobservations refer only to 1975 and 1978. In the other years there are fewer observations. The sample of treatments issmaller than the sample of controls because treatments still in Supported Work as of January 1978 are excluded fromthe sample, and in the young high school target group there were by design more controls than treatments.means of the characteristics and pretraininghours and earnings of the experimentalgroups are nearly the same. For example, themean earnings of the AFDC treatments andthe AFDC controls in the year before training differ by 2, the mean age of the twogroups differ by 3 months, and the meanyears of schooling are identical. None of thedifferences between the treatment's and control's characteristics, hours, and earnings arestatistically significant.The first two columns of Tables 2 and 3present the annual earnings of the treatmentand control group members.9 The earningsof the experimental groups were the same inthe pre-training year 1975, diverged duringthe employment program, and converged tosome extent after the program ended. The9A11earnings presented in this paper are in 1982dollars. The NSW Public Use Files report earnings inexperimental time, months from the baseline, and notcalendar time. However, my working paper describeshow to convert the experimental earnings data to theannual data reported in Tables 2 and 3.post-training year was 1979 for the AFDCfemales and 1978 for the males.10Columns 2 and 3 in the first row of Tables4 and 5 show that both the unadjusted andregression-adjusted pre-training earnings ofthe two sets of treatment and control groupmembers are essentially identical. Therefore,because of the NSW program's experimentaldesign, the difference between the post-training earnings of the experimental groups is anunbiased estimator of the training effect, andthe other estimators described in columns5-10(11) are unbiased estimators as well.The estimates in column 4 indicate that theI0The number of NSW male treatment group members with complete pre- and postprogram earnings ismuch smaller than the full sample of treatments or thepartial sample of control group members. This difference is largely explained by the two forms of sampleattrition discussed earlier. In addition, however, (i) thispaper excludes all males who were in Supported Workin January 1978, or entered the program before January1976; (ii) in one of the sites, the administrators randomly assigned .4 instead of one-half of the qualifiedhigh school dropouts into the treatment group.

VOL. 76 NO. 4LALONDE: EVALUA TING ECONOMETRIC EVALUATIONS609TABLE 4-EARNINGSCOMPARISONS AND ESTIMATED TRAINING EFFECTS FOR THE NSWPARTICIPANTS USING COMPARISON GROUPS FROM THE PSID AND THE CPS-SSAaAFDCbDifference inComparisonName 311)233(64)1,595(360)1,207(166)1,684(524)NSW Treatment EarningsLess Comparison , ,467(216)-77(202)

American Economic Association Evaluating the Econometric Evaluations of Training Programs with Experimental Data Author(s): Robert J. LaLonde Reviewed work(s): Source: The American Economic Review, Vol. 76, No. 4 (Sep., 1986), pp. 604-620 Published by: American Economic Association

Related Documents:

Baylor Scott & White Heart & Vascular Hospital - Dallas Baylor Scott & White Medical Center - Uptown Baylor University Medical Center North Central Surgical Center Baylor Scott & White Medical Center - Sunnyvale Approved by: Baylor Scott & White Health - North Texas Operating, Policy and Procedure Board on June 25, 2019

Doctor of Nursing Practice Program Orthotics & Prosthetics Program (Nurse Anesthesia) Baylor College of Medicine . Baylor College of Medicine One Baylor Plaza, MS BCM115 . One Baylor Plaza, MS BCM115 DeBakey Bldg., Suite M108 . Debakey Building, Suite M108 Houston, Texas 77030 (713) 798-8650 (713) 798-3098

american PuBlic Wave III Baylor Religion Survey September 2011 A Research Project funded by Baylor University with support from the National Science Foundation and the John M. Templeton Foundation Conducted by the Department of Sociology, College of Arts and Sciences, and Hankamer School of Business, Baylor University Research Group

GAYNOR YANCEY Office Address: Home Address: Baylor University School of Social Work 907 Morning Sun Lane . Baylor University’s Board of Regents *Baylor University Faculty Ombudsperson, 2014-2017 *Designated as Master Teacher by Baylor University, 2016 *Selected as an Outstanding Mentor by the Council of Social Work Education, 2016 *Selected .

American Economic Association U.S. Economic Growth since 1870: One Big Wave? Author(s): Robert J. Gordon Reviewed work(s): Source: The American Economic Review, Vol. 89, No. 2, Papers and Proceedings of the One Hundred Eleventh Annual Meeting of the American Economic Association (May, 1999), pp. 123-128 Published by: American Economic Association

New Insights to the Depth and Complexity of Religion in the US Selected Findings from The Baylor Religion Survey September 2006 A Research Project funded by the John M. Templeton Foundation Conducted by The Baylor Institute for Studies of Religion and Department of Sociology, Baylor University Research Group Christopher Bader Kevin Dougherty .

2-3 // About Baylor 4-5 // Majors & Minors 6 // Engagement 7 // Faith & Learning 8-9 // A Campus to Call Home 10-11 // Traditions 12-13 // Athletics 14-15 // Applying for Admission 16 // Tips for Transfers 17 // Financial Aid 18-19 // Backing Your Success 20 // Explore Waco 21 // Visit Baylor IN THESE PAGES THE BEST & THE BRIGHTEST SHINE AT BAYLOR. On this campus, academic excellence is elevated,

Introduction The three-year (2010-2013) ‘Fields of Britannia’ project, funded by the Leverhulme Trust, aims to explore the transition from Roman Britain to medieval England and Wales from a broad landscape perspective, reaching beyond the traditional site-based approach. One of the most distinctive features of the British landscape today is its intricate pattern of fields, and it is their .