Understanding implementation is critical to both program improvement and program evaluation. But measuring implementation is typically undervalued and often overlooked. This post is one of three in a series that focuses on measuring implementation when evaluating educational programs.
“Fidelity of implementation” ranks next to “scientifically based research” on our list of terms thrown about casually, imprecisely, and often for no other reason than to establish that one is serious about measurement overall. Sometimes there isn’t even a specified program model when the phrase pops up, rendering fidelity impossible. Other times we think all stakeholders are on the same page and so don’t bother to measure implementation at all.
That women are traditionally underrepresented in technology-related careers will surprise no one. It is something we’ve seen in our work, our research, and with our own children. In the 2015 ARS evaluation of STEM, Inc., a coding and entrepreneurship project designed for middle school students, only 32% of participants were girls and they were almost half as likely as boys to have any prior coding or robotics experience (35% versus 65%).
Moreover, a recent survey of over 5,700 middle school students found that boys agreed more with the statement they are good at solving computer problems. Boys are also more likely than girls to say they plan to study computers in college; they are more likely to create technology; and they demonstrate a more positive attitude toward computers and computer classes. Among our own middle and high school aged children, we note significantly more external encouragement toward coding and technology among boys than among girls, manifest in recruitment and participation in after school coding clubs and in AP Computer Science course participation. All of these factors contribute to the significant decline in young women’s pursuit of computer science degrees and the current lack of gender parity in the technology workforce. (more…)
Data mapping is about more than key points; it’s about the many sources of data that can help predict future outcomes. For a project in Clark County Public Schools, we sought a way to show how various data sources across the district could be useful in informing progress toward their strategic objective of producing graduates who are “Ready by Exit.” That meant pulling together data about program participation, special services provided during and after school, as well as community input and educator professional learning. In short, it meant we needed to create a shorthand way to map those things that informed their strategic plan. We thought it was a useful exercise, that it was worth sharing, and that it might form the basis for discussion within other organizations. We’d love to hear your thoughts! (Click the picture to view it full size)
Together with Drs. Raymond Barclay and Barbara Weschke, Managing Director Kirk Vandersall recently released results from an Arroyo Research Services study of online teacher education in the Journal of Asynchronous Learning Networks published by the Online Learning Consortium (formerly the Sloan Consortium). The study presents findings from an investigation of the impact of teachers who graduated from a fully online master’s degree program with training in pedagogy and a content-specialization in elementary reading and literacy (oERL) on reading achievement in a large urban public school system in the northwestern United States. The research team used a non-equivalent group design and matched pairs of teachers based on degree, grade-level taught, and teaching experience to construct the study on three years of student and teacher data. The study consisted of 70 teachers and 3,828 student observations. Hierarchical linear modeling was employed to understand the teachers’ effects on student learning over time. Results indicate there was a significant positive effect of the oERL on student achievement. Broadly, this study is an example of a serious attempt to ascertain the impact of a high demand and fully online program on the community where graduates are employed. More narrowly, these results support the view that a fully online program aimed at training teachers can provide opportunities for those teachers to obtain the pedagogical content knowledge that can positively influence instructional effectiveness. Full text of the study can be found here via ERIC.