Superintendent Eakins Makes Data Driven Decision on Teacher Evaluation. Bravo.

Eakins with teachersAfter six years and more than $180 million, the Hillsborough County school system plans to move beyond the teacher evaluation system developed with funding from the Bill & Melinda Gates Foundation to a system of educator support more emblematic of strategies advanced by Learning ForwardThe feeling was apparently mutual, with Gates choosing not to fund the final $20 million.

The move was announced in an email from incoming superintendent Jeff Eakins to the more than 260 Peer Evaluators and Mentors who form the core of the system. Said Eakins:

Much of the latest research points to job-embedded professional development and non-evaluative feedback from colleagues as the mechanisms which create a quality professional growth environment within organizations.

Although some components of the Empowering Effective Teachers program developed under prior Superintendent MaryEllen Elia will remain, including the student value added models embedded in Florida state law, Eakins is convening a work group to guide a transition toward a system that includes:

  • Non-evaluative systems of support by colleagues.
  • Fully released intensive systems of support for teachers who need it the most, such as brand new teachers, new teachers to HCPS and struggling teachers.
  • Job-embedded professional development.
  • Site-based Teacher-Leaders who support both students, through great instruction, and teachers, through timely observations and feedback.
  • Model and/or Demonstration Classrooms where the most effective teachers can share best practices with other colleagues.

“Relationships are key in teachers growing professionally,” Eakins said. He believes such relationships should exist largely within the school walls, and that collaboration will also cut down on the turnover that plagues the district’s poorest schools.

Why the change?

Three reasons:

  1. Limited program efficacy. Gaps in student performance between poor and black students and others in the district persist. Hillsborough aimed to address the achievement gap for poor and black students, and to have 90 percent of its third-grade and eighth-grade students testing at or above grade level in reading and math. But 2014 FCAT proficiency rates were between 53 and 59 percent (as low as 33 percent for black students), and lower-income schools continue to hire the newest and least qualified teachers.
  2. Projected Costs. The current program is projected to cost more to sustain (upwards of $52 million per year) than Hillsborough can afford. Eakins learned recently that Hillsborough’s reserve fund was evaporating at an alarming rate, affecting the district’s ability to borrow money. Costs associated with the Gates initiative are a contributing factor, including as much as $100 million dollars in district funds for new evaluator and mentor positions, data systems, and district analysts.
  3. Lack of evidence that you can measure your way to improvement. Though it did include some mentoring and coaching, EET was, at its core, a measurement project, and mentoring was provided by the same people who did the measuring. You don’t need an advanced degree in psychology – or to be an educator – to think that might not work as intended.

Data 2

What We Think

We do measurement, research and evaluation. We like to work with data to drive decisions. We believe there is a place for value-added measurement and the use of multiple data points to understand teaching and learning. But not every useful research design for understanding an education problem or the efficacy of an education program is appropriate to evaluating individual educators. This is one of those cases.

We co
mmend Eakins’ decision on multiple levels. First, it makes use of the full range of data appropriate for the decision, including student performance, staffing patterns, fiscal implications, educator feedback, and consideration of research. Second, the new model builds on the institutional strengths developed through the EET program. Third, the model focuses on building the professional practices of Hillsborough educators required for sustained improvement through collaboration, mentoring, and individual growth. Finally, the proposed new direction values educators, which is as important systemically as it is within each school to building the trust, openness and professional engagement required for success.

Hillsborough isn’t running from data. They are embracing it.

Posted in Funding Streams, News Tagged with: , , , , , , , ,

John Kucsera on Segregation in Schools

Photo credit: Gillian Laub

Photo credit: Gillian Laub

Arroyo Research Services Senior Associate John Kucsera has been attracting considerable attention for his work with UCLA professor Gary Orfield on the extreme segregation still found in our nation’s school systems 60+ years after the landmark Brown v. Board of Education court ruling. And contrary to what some may expect, the biggest problem is not found in the American South. In fact, a recent study by Kucsera and Orfield, who co-directs the Civil Rights Project at UCLA, discovered that the country’s worst segregation rates in modern day can be found in New York state’s schools.

Photo credit Warren K Leffler

Photo credit Warren K Leffler

And the problem doesn’t end there; demographic shifts in places like Southern California, which are representative of changes happening elsewhere in the country, signal a need for more proactive policies that favor integration across the country (see, for example, “Are We Segregated and Satisfied?” in the journal, Urban Education). While progress has been made since Brown for some students, issues like a growing Latino population, increasing socioeconomic disparity, school choice, and finance reform all have an impact on segregation rates.

Research shows that desegregated education can offer substantial benefits to students from all backgrounds, both in school and later in life. Yet Kucsera and Orfield readily admit it’s not a panacea, nor is it always possible to implement policies that encourage greater integration in every school. But, they say, “where it is possible…desegregation properly implemented can make a very real contribution to equalizing educational opportunities and preparing young Americans for the extremely diverse society in which they will live and work and govern together.” And policymakers – in New York and elsewhere – are taking note.

Posted in Arroyo Research Services News, News Tagged with: , , , , , , , , ,

Efficacy of Online Teacher Education

OLCTogether with Drs. Raymond Barclay and Barbara Weschke, Managing Director Kirk Vandersall recently released results from an Arroyo Research Services study of online teacher education in the Journal of Asynchronous Learning Networks published by the Online Learning Consortium (formerly the Sloan Consortium). The study presents findings from an investigation of the impact of teachers who graduated from a fully online master’s degree program with training in pedagogy and a content-specialization in elementary reading and literacy (oERL) on reading achievement in a large urban public school system in the northwestern United States. The research team used a non-equivalent group design and matched pairs of teachers based on degree, grade-level taught, and teaching experience to construct the study on three years of student and teacher data. The study consisted of 70 teachers and 3,828 student observations. Hierarchical linear modeling was employed to understand the teachers’ effects on student learning over time. Results indicate there was a significant positive effect of the oERL on student achievement. Broadly, this study is an example of a serious attempt to ascertain the impact of a high demand and fully online program on the community where graduates are employed. More narrowly, these results support the view that a fully online program aimed at training teachers can provide opportunities for those teachers to obtain the pedagogical content knowledge that can positively influence instructional effectiveness. Full text of the study can be found here via ERIC.


Posted in Digital Learning, Our Work

Effective Data Destruction: Shred, Bed, Red

The Super ShredderMuch more often than you would expect, Arroyo Research Services has to destroy confidential hard copy documents. We specify how we do so in our Data Confidentiality Plan, but that section is really a secret test for our clients: if they read that part, they will surely ask about it! The Texas Education Agency, for instance, never commented on this line in a Data Destruction Report: Read more ›

Posted in Our Work

FERPA-allowed Confidential Data

The Family Educational Rights and Privacy Act (FERPA) is designed to protect students and families from unlawful use and release of private data. It firmly undergirds how Arroyo Research Services handles data confidentiality and security, and it most often requires strict de-identification and/or prior written consent, if any access is allowed, before data is provided to third parties for the purposes of independent research.

Unfortunately, we find that FERPA is often misinterpreted Read more ›

Posted in Our Work

Small Firm, Big Commitment to Data Security

As education researchers and evaluators, data and its security are the central to our work. That’s why, at Arroyo Research Services, we are committed to the strongest data security policies and practices to assure the confidentiality of student and teacher data. Just like the larger organizations, we adhere to a detailed Data Confidentiality Policy that keeps us in compliance with FERPA and related federal and state guidance. But we go a bit further, so we thought we’d share how we think about protecting student data and some of the practical steps we take to do so. Read more ›

Posted in Our Work, Tips and Tools

Baseball Statistics and Education Research

To celebrate the re-opening of our blog, I am republishing our most popular piece, originally published in 2005 –  long before the movie version. Enjoy! And enjoy the new content in progress.

MoneyBallAsked at a recent conference what I thought was the best book on education research I’d read recently, I was quick to answer:  “Moneyball.”  Moneyball?  But that’s a baseball book!  Well, yes and no.  Michael Lewis tells the story of how Oakland A’s General Manager Billie Bean brought the lowest payroll baseball team in America to challenge the American League record for consecutive wins and experience repeated success by dispensing with preconceived notions of what makes for a good baseball player and letting comprehensive data analysis inform decision making throughout the organization.

Many of the insights offered in the book are good re-tellings of the classic writings of baseball statistician William James.  Here is just a sampling of insights from William James that can be applied to education research: Read more ›

Posted in Practical Evaluation

Congratulations, Mike Matsuda

Mike Matsuda

Mike Matsuda, Superintendent, Anaheim Union School District

We smiled when learning that Mike Matsuda was named Superintendent of Anaheim Union School District. We have been so pleased to work with him through our work evaluating the Transforming Academic and Cultural Identidad through Biliteracy project, a National Science Foundation Math Science Partnership with California State University Fullerton. Mike is one of the most decent human beings you will ever encounter, and exemplifies the impact that educators can have when they fully value each student and person around them. Working with educators like Mr. Matsuda is one of the great rewards of working every day for public school systems.

Posted in Client and Partner News

Master’s Degrees and Teacher Effectiveness

Arroyo Research Services Study of Masters Degrees and Teacher EffectivenessIn a research brief summarizing the most recent findings in a three year study examining the effects of master’s degrees on teacher effectiveness, Arroyo Research Service found that elementary students whose teachers held a master’s degree performed statistically significantly better in both reading and language arts than students whose teachers did not hold a master’s degree.

The study was conducted in a single large suburban district in Georgia, using student, teacher and university data from 2004 through 2010, including performance on the Georgia CRCT assessments. The study includes analysis of over 200,000 student data points and more than 4,100 teachers. The study and associated frequently asked questions can be found here:

Master’s Degrees and Teacher Effectiveness: New Evidence from State Assessments

Master’s Degrees and Teacher Effectiveness: Frequently Asked Questions

Posted in Our Work, Professional Learning

Cost Effective Dropout Recovery in Texas

Arroyo_Research_Services_Dropout_Recovery_Report_CoverWe were pleased to release the Arroyo Research Services Evaluation of the Texas Dropout Recovery Pilot Program: Cycles 1 and 2 in May 2011. Conducted from 2008 through 2011, the evaluation assisted the Texas Education Agency in examining the effects of the TDRPP pay for performance model that directly tied project payments to demonstrated student academic progress and program completion. Full results and program descriptions are included in the link below; summary results are presented in the Executive Summary. Key findings from the evaluation include:

– Grantees served 4,141 students, twice as many as projected.
– 1,283 students completed the program by earning a high school diploma or demonstrating college readiness.
– The average TDRPP graduate is expected to earn $246,348 more in his or her lifetime than a high school dropout.
– TDRPP is expected to save the state $95.3 million in current dollars after accounting for initial program expenditures.

Read more ›

Posted in Our Work