Let's bring out a great article from the archives, Daniel J. Cohen and Roy Rosenzweig's "No Computer Left Behind."
To Christian Sandvig's brilliant blog on research methods (http://www.hastac.org/blogs/cathy-davidson/statistical-significance-v-su...), I would like to add a pedagogical note. This is tangential but not irrelevant. Both as research and as pedagogy, many ways that we quantify data are rooted less in research outcomes than in circular assumptions about data that don't bear scrutiny as we look at them more closely.
The article I refer to here is one of my favorites, by Daniel J. Cohen (a member of the HASTAC Steering Committee) and the late, beloved Roy Rosenzweig, then both of George Mason University. Dan continues to teach there as a history professor and part of the Center for History and New Media. Providing us with a history (and it is a relatively recent one, going back to the 1920s) of multiple-choice testing, they then go build themselves a a "bot," H-Bot, created by Dan with the help of a talented high school student. Even in 2006 when "No Computer Left Behind" appeared in the Chronicle Review of the Chronicle of Higher Education (Feb 24, 2006), H-Bot could score 82% on a multiple-choice history test designed for fourth-graders by NAEP. I'm sure H-Bot would be 100% accurate today, given Google's ever expanding databases and ever-more significant algorithms for datamining.
Their point was that in the same way calculators made rudimentary calculations like long division unnecessary and could free up math teachers to concentrate on higher understanding of math, H-Bot could finally (a) take history out of the hands of the 150M plus annual testing industry and the Scantron monopoly; and (b) put emphasis on real historical research methods instead of the manufactured methodology of multiple choice that evolved for cheap, efficient scorekeeping not real learning. Or, to quote an article cited in this article, refering to an early study of students' knowledge of history based on multiple-choice testing, "While perhaps the first instance, this was not the last in which ease of measurement--not priority of subject-matter understanding--determined the shape and contour of a research program."
Given the devastating effect that No Child Left Behind and "standards-based education reform" have had on American education since 2002, winding up H-Bot again to assist Diane Ravitch in her campaign against standardized testing might not be a bad thing.