Afterschool Programs Make a Difference: Findings From the Harvard Family Research Project
Published in SEDL Letter Volume XX, Number 2, August 2008, Afterschool, Family, and Community
A recent study showed significant gains in math test scores for students who participated in high-quality afterschool programs.
In February, the Harvard Family Research Project (HFRP) published After School Programs in the 21st Century: Their Potential and What It Takes to Achieve It (Little, Wimer, & Weiss, 2008), a brief that summarizes 10 years of research on afterschool programs and discusses implications for the future. Featured in the brief are studies that evaluate large afterschool programs with experimental or quasi-experimental designs. The authors, Priscilla M. D. Little, Christopher B. Wimer, and Heather B. Weiss, drew on those evaluations to address two primary questions: 1) Does participation in after school programs make a difference, and, if so, 2) What conditions appear to be necessary to achieve positive results? In this article, we summarize their findings and discuss the characteristics of programs leading to positive student outcomes.
Does participation in afterschool programs make a difference?
According to Little, Wimer, and Weiss,
The short answer is yes. . . .A decade of research and evaluation studies, as well as large-scale, rigorously conducted syntheses looking across many research and evaluation studies, confirms that children and youth who participate in after school programs can reap a host of positive benefits in a number of interrelated outcome areas—academic, social/emotional, prevention, and health and wellness. (2008, p. 2)
Afterschool programs can have an impact on academic achievement. Improved test scores are reported in evaluations of The After-School Corporation (TASC) programs in New York City (Reisner, White, Birmingham, & Welsh, 2001; White, Reisner, Welsh, & Russell, 2001) and in Foundations, Inc. elementary school programs (Klein & Bolus, 2002). A more recent longitudinal study showed significant gains in math test scores for elementary and middle-school students who participated in high-quality afterschool programs (Vandell, Reisner, & Pierce, 2007), and a meta-analysis of 35 studies of at-risk youth found that out-of-school time programs had a positive effect on reading and math achievement (Lauer, Akiba, Wilkerson, Apthorp, Snow, & Martin-Glenn, 2006).
The HFRP brief emphasizes that many studies “repeatedly underscore the impact of supporting a range of positive learning outcomes, including academic achievement, by affording children and youth opportunities to learn and practice new skills through hands-on, experiential learning,” (p. 3) citing evaluations of Citizen Schools (Espino, Fabiano, & Pearson, 2004; Fabiano, Pearson, & Williams, 2005) and of LA’s BEST (Huang, Coordt, La Torre, Leon, Miyoshi, & Pèrez, et al., 2007), among others. These programs not only offered academic support to improve academic performance, but also combined it with other enrichment activities to achieve positive academic outcomes. Little, Wimer, and Weiss noted,
Thus, extra time for academics by itself may be necessary but may not be sufficient to improve academic outcomes. Balancing academic support with a variety of engaging, fun, and structured extracurricular or co-curricular activities that promote youth development in a variety of real-world contexts appears to support and improve academic performance. (2008, p. 4)
Social and Emotional Development
Programs with a strong intentional focus on improving social and personal skills were found to improve students’ self-esteem and self-confidence (Durlak & Weissberg, 2007). Examples include Go Grrls, an Arizona program of structured group sessions that helps improve girls’ body image, assertiveness, self-efficacy, and self-liking (LeCroy, 2003) and mentoring programs such as Across Ages (Taylor, LoSciuto, Fox, & Hilbert, 1999), which pairs older adults with students.
Prevention of Risky Behaviors
The hours after school, between 3 p.m. and 6 p.m., offer opportunities for juvenile crime, sexual activity, and other risky behaviors such as drug and alcohol use. Research and evaluation studies have shown that participation in afterschool programs have a positive impact on juvenile crime and help reduce pregnancies, teen sex, and boys’ marijuana use (Goldschmidt, Huang, & Chinen, 2007; Philliber, Kaye, & Herrling, 2001; Philliber, Kaye, Herrling, & West, 2002).
Health and Wellness
The afterschool setting presents an opportunity to address the growing problem of obesity among children and youth. Research has shown that afterschool programs can contribute to healthy lifestyles and increased knowledge about exercise and nutrition. Girlfriends for KEEPS (Story, et al., 2003) and the Medical College of Georgia’s FitKid program (Yin, Gutin, Johnson, Hanes, Moore, Cavnar, et al., 2005) are two such programs that benefit their participants; similar results are reported in a longitudinal study of more than 650 students who participated in 25 Connecticut afterschool programs (Mahoney, Lord, & Carryl, 2005).
What conditions appear to be necessary to achieve positive results?
Little, Wimer, and Weiss wrote that while afterschool programs “have the potential to impact a range of positive learning and development outcomes,” some programs do not maximize this potential. They identified the following three factors as critical to achieving positive youth outcomes:
- Access to and sustained participation in the program
- Quality programming and staffing
- Strong partnerships among the program and other places where students are learning, such as their schools, their homes, and other community institutions
Access to and Sustained Participation
The HFRP brief discussed a number of research syntheses (American Youth Policy Forum, 2006; Redd, Cochran, Hair, & Moore, 2002; Simpkins-Chaput, Little, & Weiss, 2004) and evaluations such as those of the After School Matters program in Chicago (Goerge, Cusick, Wasserman, & Gladden, 2007), Louisiana’s 21st Century Community Learning Center (CCLC) program (Jenner & Jenner, 2004), and LA’s Best (Huang, et al., 2007) that show that students experience greater gains if they participate regularly in afterschool programs, with greater frequency (more days per week), and in a sustained manner over a number of years.
Much like gaps among students in regular day school, Little, Wimer, and Weiss noted differences among students whose families have higher incomes and more education and those students whose families are less advantaged. They wrote (p. 6) that students whose families have higher incomes and more education:
- Are more likely to participate in afterschool activities
- Do so with greater frequency during the week
- Participate in a greater number of different activities within the week or month
- Are more likely to participate in enrichment programs, whereas disadvantaged students are more likely to participate in tutoring programs
Quality Programming and Staffing
According to Little, Wimer, and Weiss, research on the quality of afterschool programs is mostly descriptive, with only “a handful of rigorously designed studies.” They have drawn from a set of studies they describe as “a small but powerful set of studies.”
Regarding program structure and supervision, Little, Wimer, and Weiss (p. 6) conclude, “Without the structure and supervision of focused and intentional programming, youth participants in after school programs, at best, can fail to achieve positive outcomes and, at worst, can begin to perform worse than their peers” (Vandell, Pierce, Brown, Lee, Bolt, & Dadisman, 2006; Pearson, Russell, & Reisner, 2007). They continue, “In fact, some research finds that when youth are concentrated together without appropriate structure and supervision, problematic behavior follows, suggesting that focused, intentional activities with appropriate structure and supervision are necessary to keep youth on an upward trajectory and out of trouble” (Jacob & Lefgren, 2003).
In a meta-analysis of the impact of 73 afterschool programs, Durlak and Weissburg (2007) found that programs missing any of the following four characteristics did not achieve positive results:
- Sequenced – Used sequenced set of activities designed to achieve skill development objectives
- Active – Used active forms of learning to help students develop skills
- Focused – Devoted program components to developing personal or social skills
- Explicit – Targeted explicit personal or social skills
Other studies (Gerstenblith, Soule, Gottfredson, Lu, Kellstrom, Womer, et al., 2005; Arbreton, Goldsmith, & Sheldon, 2005) found that programs with structured and focused, well-organized activities foster engagement and facilitate high quality learning opportunities.
According to Little, Wimer, and Weiss, the quality of a program’s staff is one of the most critical features of a high-quality afterschool program. A follow-up study to the TASC evaluation found that positive relationships were found in sites where staff modeled positive behavior, actively promoted student mastery of the skills or concepts presented in activities, listened attentively to participants, frequently provided individualized feedback and guidance during activities, and established clear expectations for mature, respectful peer interactions (Birmingham, Pechman, Russell, & Mielke, 2005). Other research found that in low-quality programs, the staff “engaged in negative and punitive interactions with youth” instead of “engaging in supportive behavior and practicing positive behavior management techniques” (Vandell, Shumow, & Posner, 2005; Gerstenblith, et al., 2005).
Little, Wimer, and Weiss also found:
Strong partnerships are more than a component of program quality. . .
Programs are more likely to exhibit high quality when they effectively develop, utilize, and leverage partnerships with a variety of stakeholders like families, schools, and communities. However, strong partnerships are more than a component of program quality: they are becoming a nonnegotiable element of supporting learning and development across all the contexts in which children learn and develop. (p. 8)
After School in the 21st Century
Little, Wimer, and Weiss summarize:
The research and evaluation studies and syntheses highlighted in this brief demonstrate how complex a task it is to provide high quality, effective supports for youth and their families, but they also provide powerful evidence that after school programs do work when key factors are addressed—factors of access, sustained participation, program quality and strong partnerships. (p. 10)
They also conclude that the research and evaluation results from the past decade raise the following important questions about the future of afterschool programs and their role:
- Moving forward, how can the research-based practices known to be effective in after school programs be adopted more broadly within after school programs and other expanded learning models?
- How can after school programs work with schools, families, and other community and health supports to ensure a complementary array of learning and developmental supports across the day, the year, and the developmental continuum from kindergarten through high school?
- Moving forward, how can and should “success” of after school programs be measured, particularly as the field moves toward greater emphasis on shared responsibility and partnerships?
- How can choice be built into after school and extended day options to ensure that programs are responsive to the needs of working families and youth participants alike?
About the Studies Included in HFRP Brief
According to lead author Priscilla Little, the authors of the brief did not do an exhaustive review of all studies available or conduct a systematic review, but rather took a “seminal studies” approach to examine and include those studies which best represented a range of outcomes, practices, and settings. The set of studies was then reviewed by leading researchers in the field who validated that this set was indeed representative of the wealth of available information.
- American Youth Policy Forum. (2006). Helping youth succeed through out-of-school time programs. Washington, DC: Author.
- Arbreton, A. J. A., Goldmith, J., & Sheldon, J. (2005). Launching literacy in after-school programs: Early lessons from the CORAL initiative. Philadelphia: Public/Private Ventures.
- Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of highperforming after-school programs: A follow-up to the TASC evaluation. Washington, D. C.: Policy Studies Associates.
- Durlak, R., & Weissberg, R. (2007). The impact of after-school programs that promote personal and social skills. Chicago: CASEL.
- Espino, J., Fabiano, L., & Pearson, L. M. (with Kirkwood K. P., Afolabi, K., & Pasatta, K.). (2004). Citizen Schools: Evidence from two student cohorts on the use of community resources to promote youth development. Phase II report of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates.
- Fabiano, L., Pearson, L. M., & Williams, I. J. (2005). Putting students on a pathway to academic and social success: Phase III findings of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates.
- Gerstenblith, S., Soule, D., Gottfredson, D., Lu, S., Kellstrom, M., Womer, S., et al. (2005). Afterschool programs, antisocial behavior, and positive youth development: An exploration of the relationship between program implementation and changes in youth behavior. In J. Mahoney, J. Eccles, & R. Larson (Eds.), Organized activities for contexts for development: Extracurricular activities, after-school, and community programs (pp. 457-477). Mahwah, NJ: Erlbaum.
- Goerge, R., Cusick, G. R., Wasserman, M., & Gladden, R. M. (2007). After-school programs and academic impact: A study of Chicago’s After School Matters. Chicago: Chapin Hall Center for Children.
- Goldschmidt, P., Huang, D., & Chinen, M. (2007). The long-term effects of after-school programming on educational adjustment and juvenile crime: A study of the LA’s BEST after-school program. Los Angeles: UCLA/CRESST. Available at http://www.lasbest.org/resourcecenter/LASBEST_DOJ_Study_Brief.pdf
- Huang, D., Coordt, A., La Torre, D., Leon, S., Miyoshi, J., Pérez, P., et al. (2007). The afterschool hours: Examining the relationship between afterschool staff-based social capital and student engagement in LA’s BEST. Los Angeles: UCLA/CRESST. Available at http://www.cse.ucla.edu/products/reports/R712.pdf
- Jacob, B. A., & Lefgren, L. (2003). Are idle hands the devil’s workshop? Incapacitation, concentration, and juvenile crime. American Economic Review, 93, 1560-1577.
- Jenner, E. J., & Jenner, L. W. (2004). Academic outcomes in Louisiana’s 21st Century Community Learning Centers. Baton Rouge, LA: Policy & Research Group.
- Klein, S. P., & Bolus, R. (2002). Improvements in math and reading scores of students who did and did not participate in the Foundations After School Enrichment Program during the 2001–2002 school year. Santa Monica, CA: Gansk & Associates.
- Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.
- LeCroy, C. W. (2003). Experimental evaluation of “Go Grrls.” Tucson, AZ: Author.
- Little, P. M.D., Wimer, C. , & Weiss, H. B. (2008, February). After school programs in the 21st century: Their potential and what it takes to achieve it. Issues and Opportunities in Out-of-School Time Evaluation Brief No. 10. Cambridge, MA: Harvard Family Research Project.
- Pearson, L. M., Russell, C. A., & Reisner, E. R. (2007). Evaluation of OST programs for youth: Patterns of youth retention in OST programs, 2005–06 to 2006–07. Washington, DC: Policy Studies Associates.
- Philliber, S., Kaye, J. W., & Herrling, S. (2001, May). The national evaluation of the Children’s Aid Society Carrera-Model Program to prevent teen pregnancy. Accord, NY: Phillber Research Associates. Available at http://www.childrensaidsociety.org/media/general/cas-Teen_Pregnancy_Prevention.pdf
- Philliber, S., Kaye, J. W., Herrling, S., & West, E. (2002). Preventing pregnancy and improving health care access among teenagers: An evaluation of the Children’s Aid Society—Carrera Model. Perspectives on Sexual and Reproductive Health, 34(5), 244–251. Available at http://www.agi-usa.org/pubs/journals/3424402.pdf
- Redd, Z., Cochran, S., Hair, E., & Moore, K. (2002). Academic achievement programs and youth development: A synthesis. Washington, DC: Child Trends.
- Reisner, E. R., White, R. N., Birmingham, J., & Welsh, M. (2001). Building quality and supporting expansion of After-School Projects: Evaluation results from the TASC After- School Program’s second year. Washington, DC: Policy Studies Associates.
- Simpkins-Chaput, S., Little, P. M. D., & Weiss, H. B. (2004). Understanding and measuring attendance in out-of-school time programs. Issues and Opportunities in Out-of-School Time Evaluation Brief No. 7. Cambridge, MA: Harvard Family Research Project.
- Story, M., Sherwood, N. E., Himes, J. H., Davis, M., Jacobs, Jr., D. R., Cartwright, Y., et al. (2003). An after-school obesity prevention program for African-American girls: The Minnesota GEMS Pilot Study [Supplement 1]. Ethnicity & Disease, 13(1), 54–64.
- Taylor, A., LoSciuto, L., Fox, M., & Hilbert, S. (1999). The mentoring factor: An evaluation of Across Ages. Intergenerational program research: Understanding what we have created. Binghamton, NY: Haworth.
- Vandell, D., Reisner, E., & Pierce, K. (2007). Outcomes linked to high-quality afterschool programs: Longitudinal findings from the study of promising practices. Irvine, CA: University of California and Washington, DC: Policy Studies Associates. Available at http://www.gse.uci.edu/docs/PASP%20Final%20Report.pdf
- Vandell, D., Pierce, K., Brown, B., Lee, D., Bolt, D., Dadisman, K., et al. (2006, March). Developmental outcomes associated with the afterschool contexts of low-income children and adolescents. Paper presented at the Society for Research on Adolescence Annual Meeting, San Francisco.
- White, R. N., Reisner, E. R., Welsh, M., & Russell, C. (2001). Patterns of student-level change linked to TASC participation, based on TASC projects in Year 2. Washington, DC: Policy Studies Associates.
- Yin, Z., Gutin, B., Johnson, M., Hanes, J., Jr., Moore, J. B., Cavnar, M., et al. (2005). An environmental approach to obesity prevention in children: Medical College of Georgia FitKid Project year 1 results. Obesity Research, 13, 2153–2161.
Next Article: SEDL's Afterschool Work Deepens with California Demonstration Program
Homework – an eight letter word likely to spark an immediate and impassioned response.
As students, we've all had to do it. We've handed it in on time, late, or not at all (sometimes offering up creative and amusing excuses into the bargain).
Most of you reading this will have set homework, marked it and, at one time or another, questioned whether it's worth the time and effort ... for you and for your students.
It continues to be a hotly debated topic, not just among those in the school community, but in the academic community too.
'The debate about the effectiveness of homework as a tool of learning has continued for more than a century. There have been more than 130 studies published related to the subject and these have reached different and, at times, quite contradictory conclusions,' a report from the Inquiry into the approaches to homework in Victorian schools points out.
After conducting its own literature review, receiving 32 submissions and hearing from 16 expert witnesses during three days of public hearings, the inquiry committee had this to say: 'It is not possible from the available data to make unequivocal statements about the effectiveness of homework overall in assisting student learning.’
It does, however, want the state education department to support schools and teachers in this area by explaining current research.
So, what research is it referring to?
Most of the studies deal with the United States and Europe. 'There is limited research undertaken in Australia, however a book published in 2012 by two Australian academics [Associate Professor Richard Walker and Professor Michael Horsley], Reforming Homework, provided valuable context for the inquiry,' the report says.
Supporters of homework argue it not only has academic benefits, but also helps youngsters develop important study and time management skills, and gives parents a chance to engage in their child’s learning.
On the other side of the debate: ‘For those opposed to homework, many feel that it creates unnecessary pressure on students for limited or disputed academic benefit, robs children of time to develop other life skills, through recreational and artistic activities and social interaction, and places pressure on family life,’ the report says.
Walker, of the University of Sydney, told one of the public hearings that research on homework tends to focus on three things: student learning and achievement; the development of student learning skills; and parental involvement.
In Reforming Homework, the academics comment on the differing conclusions reached by the studies. 'Researchers have variously concluded that homework is beneficial (Cooper et al.) or harmful (various), that homework has no effects (Kohn), that it has complex effects or that the research is too sparse or too problematic to be able to justify the drawing of strong conclusions.'
As John Buell (2004) puts it: ‘… for a practice as solidly entrenched as homework, the scholarly case on its behalf is surprisingly weak and even contradictory.’
The inquiry report does highlight research (Cooper, 2007) showing that students are more likely to complete homework if they know teachers are keeping track of their progress and giving feedback on errors and areas for improvement.
Walker told the committee that's not an easy task. 'Providing every student with targeted feedback about their homework is very difficult for teachers, so it often falls between the cracks.'
Professor John Hattie, of the Melbourne Graduate School of Education, has famously calculated the 'effect-size' of more than 100 education innovations. He recently told the BBC that homework in primary school has an effect-size of around zero 'which is why we need to get it right, not why we need to get rid of it...'
He added homework does make a bigger difference in secondary school, mainly because the tasks are often about reinforcing and giving students another chance to practice what they've learnt. 'The worst thing you can do with homework is give kids projects, the best thing you can do is to reinforce something you've already learnt,' Hattie told the broadcaster.
How much homework?
This week The Australian highlighted OECD data showing Australian 15-year-olds are set six hours of homework a week - higher than the OECD average of 4.9 hours.
Walker told the inquiry committee that there is 'absolutely no research advice' on how much time students should spend doing homework. That hasn’t stopped some Australian education departments from making recommendations on the 'optimum' amount of homework.
Victorian DEECD guidelines (2012) indicate no more than 30 minutes per day and no homework during weekends or holidays for Prep to Year 4; 30 to 45 minutes per day in Year 5 extending to 45 to 90 minutes by Year 9; one to three hours per night for Year 10 to 12, plus six hours on weekends during peak VCE periods.
Queensland education department guidelines in 2012 suggest: no homework for Prep students and weekly limits of one hour for Years 1 to 3; two to three hours for Years 4 and 5; three to four hours for Years 6 and 7; and no more than five hours a week for Years 8 and 9. For Years 10 to 12 it says hours will vary according to individual learning needs.
The remaining state and territory education departments all have homework policies but do not make recommendations on hours.
Even if you look beyond the fact that guidelines on ‘optimum hours’ aren’t backed by research advice, suggesting set times - as the inquiry committee points out - is about the quantity of work set, rather than the quality. Not everyone works at the same pace and has access to the same resources and support network.
On the question of quality, Hattie is urging schools to think about outcomes. He told the committee that education departments shouldn’t encourage schools to adopt a set policy. 'Rather than prescribing a particular way of doing it, let us ask them to provide evidence that the homework policy the school is adopting is improving the outcomes for kids.'
Discussing whether or not homework has more of an impact according to the subject area, the academic says it does have more influence in maths, again because often tasks set for this subject are about ‘deliberate practice’ of things learnt in class as opposed to a ‘project’.
Researchers Ozkan Eren and Daniel J Henderson found assigning homework in subjects like Science, English and History has little to no impact on test scores.
And so the debate continues. As the inquiry report committee concludes: ‘Until a causal relationship between homework and academic achievement and personal development is established in studies in Australia, based on Australian educational and cultural structures, this argument is likely to remain unresolved and will continue to be one of perception.’
Findings from the Victorian inquiry
- Feedback on homework is a crucial step in the learning process and without timely feedback some of the learning benefits of homework may be reduced;
- New teachers in Victorian schools may currently lack support to identify and set quality homework;
- Homework can reduce the amount of time available to pursue other activities and interests which may have equal or greater long term benefit;
- Flipped learning offers a new way of engaging children in education and may allow for a better use of time in the classroom;
- There is strong evidence and general agreement that homework at the primary school level has little impact on academic performance, but may play an important transitional role in preparing students for secondary school and beyond;
- Measuring homework by the time spent doing it is an imprecise and inadequate measure that does not take into account the quality of the work or the ability of the student or, increasingly importantly, student access to technology;
- Homework can have the effect of helping a parent to understand the progress the child is making or otherwise and can therefore help make parent-teacher interviews more meaningful;
- Successful schools see education as a collaborative process between the student, parent and the school, and consider parents to be ‘partners’ in their children’s education. Schools that assist parents in providing support to their children tend to have better educational outcomes;
- Homework’s value is largely as a tool to develop the capacity of students, even when it has no mark or grade attached;
- The proliferation of private tutors may place undue financial pressure on families and has the potential to undermine the value of the assistance they can provide, by shifting the focus from the work assigned by the teacher to work assigned by the tutor;
- Homework may need to be adapted for children with learning disabilities to ensure they obtain the same benefit from homework as their peers;
- Homework clubs provide a vital service for students who experience a form of disadvantage. They engage students who may otherwise drop out of the system.
Buell, J. (2004). Closing the book on homework: Enhancing public education and freeing family time. Temple University Press.
Cooper, H. (2007). The battle over homework. Corwin Press.
Eren, O., & Henderson, D.J. (2011). ‘Are we wasting our children’s time by giving them more homework?’ Economics of Education Review 5(30).
Horsley, M., & Walker, R. (2013). Reforming homework: practices, learning and policy. Melbourne: Palgrave Macmillan.
How is homework improving outcomes for your students? What evidence do you have?
Research shows timely feedback is an important part of the process - how and when do you provide feedback to students on homework tasks?
Visit the Victorian Parliament website to access the inquiry committee's full report.