Asked at a recent conference what I thought was the best book on education research I'd read recently, I was quick to answer:  "Moneyball."  Moneyball?  But that's a baseball book!  Well, yes and no.  Michael Lewis tells the story of how Oakland A's General Manager Billie Bean brought the lowest payroll baseball team in America to challenge the American League record for consecutive wins and experience repeated success by dispensing with preconceived notions of what makes for a good baseball player and letting comprehensive data analysis inform decision making throughout the organization.

Educators, like baseball fans, are convinced that they know good teaching when they see it, they can assess student ability and performance without systematic measurement, and that careful observation and understanding of individual student and teacher differences must be taken into account.  Of course there is obvious (partial) truth to this, but there is much to be learned by carefully structured observation and measurement over time that is impossible to see in any one classroom experience or snapshot measure.

As in baseball, so in education, we must be watchful to assure that the statistics we gather and the analyses we pursue tell true stories of what is significant in the processes and the outcomes they produce. This comes from not only assuring that the measures we use are gathered without bias, but that they are in fact the appropriate measures in the first place! 

To the extent that we fail to account for the overall circumstances of our students, teachers, and schools, we will miss important lessons, and may ultimately draw incorrect conclusions about effective instructional strategies.  Example:  I recently toured the most well-regarded school in a local school district, where I was considering placing my son.  Standardized test scores were higher than the district average.  The school was neat and orderly.  The teachers followed the curriculum.  And the students were practically falling asleep at their desks out of sheer boredom.  It was painful to watch.  The Assistant Principal went on about how their test scores and successful secondary school placements outperformed the rest of the district.  The school was a magnet school, had less then half the district average for students receiveing free or reduced lunch, and more closely resembled local private schools in the composition of the student body than any other school in the district. It should have been performing even better than it was, and it could with more engaging curriculum and instruction.