Our clients are by and large genuine change-makers, motivated to measure and achieve the positive outcomes they seek. And one of our most important jobs is helping them develop and use appropriate data to enhance discovery, analysis, insight and direction. But client commitment and our professional responsibility don’t always avoid some common data collection pitfalls.
Through countless evaluations of school and district level educational programs, as well as multi-site, statewide initiatives, we have identified the pitfalls that follow. They may seem like no-brainers. But that’s what makes them so easy to fall into, even for seasoned evaluators and educational leaders. We highlight them here as a reminder to anyone looking to accurately measure their impact:
1. Asking leading a question versus a truly open-ended one. If you aim for honesty, you must allow respondents to give negative responses as well as positive ones. For instance, asking:
“How would putting an iPad into the hands of every student in this district improve teaching and learning outcomes?”
…assumes teaching and learning outcomes will be improved, at least to some degree. (more…)
Continue readingAs educators, we talk about data, collect data, wade through data, analyze data, and draw conclusions from data that hopefully demonstrate how and why our interventions led to the achievement of our goals. But sometimes there seems to be so much data, so many things we could measure, that it’s difficult to know where to start.
Burying one’s head in the sand – i.e., not planning for the appropriate collection and use of data to drive decision-making – is clearly not the answer. But where to begin? In a guest blog post for ASCD, 30-year educator, administrator and author Craig Mertler shared his top five ways to achieve strategic data use in planning and decision-making. We’ve adapted them here:
1. Find your focus. Planning starts with identification. Mertler suggests zeroing in on a specific “problem of practice” that you want to improve or otherwise address and using that to brainstorm about the types of data you may wish to collect. (more…)
Continue readingOur past two posts covered both the “why” of measuring implementation and some of the common challenges to doing so. In this third and final post, we’ll look at what is most useful to measure.
Implementation measures are particular to each program and should take into account the specific actions expected of program participants: who is doing what, when, where, how often, etc. Participants may be teachers, students, administrators, parents, advocates, tutors, recruiters, or institutions (e.g., regional centers, schools, community organizations). Specific measures should help stakeholders understand whether, how, and with what intensity a program is being put into place. Moreover, for programs with multiple sites or regions, understanding differences among them is critical.
Data mapping is about more than key points; it’s about the many sources of data that can help predict future outcomes. For a project in Clark County Public Schools, we sought a way to show how various data sources across the district could be useful in informing progress toward their strategic objective of producing graduates who are “Ready by Exit.” That meant pulling together data about program participation, special services provided during and after school, as well as community input and educator professional learning. In short, it meant we needed to create a shorthand way to map those things that informed their strategic plan. We thought it was a useful exercise, that it was worth sharing, and that it might form the basis for discussion within other organizations. We’d love to hear your thoughts! (Click the picture to view it full size)
The Family Educational Rights and Privacy Act (FERPA) is designed to protect students and families from unlawful use and release of private data. It firmly undergirds how Arroyo Research Services handles data confidentiality and security, and it most often requires strict de-identification and/or prior written consent, if any access is allowed, before data is provided to third parties for the purposes of independent research.
Unfortunately, we find FERPA is often misinterpreted to mean that identifiable or non-consented data may never be released. That’s not actually the case. (more…)
Continue reading