Our past two posts covered both the “why” of measuring implementation and some of the common challenges to doing so. In this third and final post, we’ll look at what is most useful to measure.
Implementation measures are particular to each program and should take into account the specific actions expected of program participants: who is doing what, when, where, how often, etc. Participants may be teachers, students, administrators, parents, advocates, tutors, recruiters, or institutions (e.g., regional centers, schools, community organizations). Specific measures should help stakeholders understand whether, how, and with what intensity a program is being put into place. Moreover, for programs with multiple sites or regions, understanding differences among them is critical.
In our last post, we shared four reasons why educators should be measuring implementation: here we’ll look at four common challenges to strong implementation measurement.
1. Differential definitions. What happens when different units of your program operate with different working definitions of a measure?
Take tutoring, for example, in a multi-site program, where each site is asked to report the number of hours per week a participant is tutored. Site A takes attendance and acknowledges that, although the after school program runs for 1.5 hours, only .5 hours are spent tutoring. So Site A reports the number of days a student attends, multiplied by .5: e.g., if Jose attends for 3 days, Site A reports 1.5 hours of tutoring. Site B calculates 1.5 hours of tutoring per day times 5 days per week, per participant: So if Jose is a participant that week, regardless of how often he attends, Site B reports 7.5 hours of tutoring. (more…)
Continue reading
Asked at a conference what I thought was the best book on education research I’d read recently, I was quick to answer, “Moneyball.” Moneyball? But that’s a baseball book! Well, yes and no. Michael Lewis’s story tells how Oakland A’s General Manager Billie Bean got the lowest payroll baseball team in America to challenge the American League record for consecutive wins; the A’s went on to repeated success by dispensing with preconceived notions of what makes for a good baseball player and letting comprehensive data analysis inform their decision making throughout the organization.
.” Moneyball? But that’s a baseball book! Well, yes and no. Michael Lewis’s story tells how Oakland A’s General Manager Billie Bean got the lowest payroll baseball team in America to challenge the American League record for consecutive wins; the A’s went on to repeated success by dispensing with preconceived notions of what makes for a good baseball player and letting comprehensive data analysis inform their decision making throughout the organization.Many of the insights offered in the book are good re-tellings of the classic writings of baseball statistician William James. Here is just a sampling of insights from James that can be applied to education research: (more…)
Continue reading