Do you remember this ad slogan for Trident gum?
“4 out of 5 dentists surveyed would recommend sugarless gum to their patients who chew gum.”
I was a kid when that slogan aired on national television stations in the early 1980’s. Even then, I remember thinking to myself, what exactly does that last phrase mean? Incidentally, for this post I found this information from The Straight Dope, about the study. As you would expect, that fifth dentist probably refused to recommend chewing gum at all. And for those of you who had the other question that I had, apparently 1200 dentists were surveyed. Compared to most research in education, the Trident commissioned study of dentists was air tight.
It was when I attended grad school in science education that I was first exposed to the concept of a quasi-experimental study. Prior to that I had studied physics, and my concept of research was divided into two worlds. There were hard sciences where you limited the scope of your study, and therefore the scope of your findings, by controlling every possible variable so that you simply had one independent and one dependent variable to examine. Oh, to be an undergrad! The second part of my concept of the research world was social science where qualitative data was gathered using heuristic, and inherently limited methods.
My view of what makes for a valuable contribution to a given field of study has changed since then, and so has research in general. I was ill conceived back then in my belief that the physical sciences had a lock on truth, limited as it may have been. Perhaps because of the burden placed on them to demonstrate their worth, social scientists like Richard Shavelson, Professor Emeritus, Stanford University, took statistical analysis to places that many in the physical sciences would do well to emulate. The preponderance, ease of access, and reward for quality analysis of data in our digital age are making it easier for researchers in any field to do quality analysis that can guide our policy and local decision making.
Unfortunately, despite the availability of treasure troves of data in education, very little high quality research happens. This is about to change.
Little Data Lots of Air
In recent years, the numbers that have guided education policy could hardly be called research results at all. The 1983 report entitled A Nation at Risk is the most oft cited study in education circles and is believed to be the major driver behind increases in both federal funding of and intervention into public K12 education. The study cited international standardized exam scores that convinced policy makers the US was going to lose The Cold War to the Russians. We know how that turned out. National comparisons of US student scores on the PISA exam have been the preferred Skinnerian stick of late that privatization advocates use to prod their policy maker allies in Washington. Well respected education researchers, like Linda Darling Hammond, Stanford University, have had to take their more holistic and sensible analyses of the plight of American education to alternative venues as Darling Hammond has done in these sharp analyses positions in The Nation.
The Winds of Change Are Blowing
Economists, were the first to the data feast. Levitt and Dubner, authors of Freakanomics surprised us by drawing a statistical association with big data between the number of books on the shelves of a home (or at least the number a parent stated on a survey) and the performance of the child in that house – apparently, controlling for socioeconomic status. More books on shelf equals higher achieving kids – go figure. Malcolm Gladwell, author of Tipping Point, also using big data, pointed out that success in school may be more closely associated with your astrological sign than with your average daily attendance. I jest, but his analysis surfaced the importance of being older than your peers. Cutoff dates for school enrollment determine, to a certain extent, who will achieve at the highest level. Older kids rule.
The edtech startup will be the next party to arrive at the big data feast. Google Apps, still in it’s K12 infancy has the power to provide large education districts with a waterfall of data that will not simply be the uni-dimensional and barely actionable standardized test scores from bubble exams that drive decision making currently. Philanthropic, education-focussed venture funds like the New Schools Venture Fund and the SCE Foundation are providing the capital needed for startups and nonprofits alike to use data meaningfully to craft products and services that can show holistic improvement in student achievement. With access to more data these organizations and others like them are expecting better analysis and response from their funded partners. It will soon be common practice for organizations to demonstrate how their intervention impacted the reading level or the word production output of the students who were targeted.
Don’t get too excited. There are still a lot of dentists out there who will be willing to answer a loaded question, and there are plenty of bubble gum manufacturers who would be more than happy to publish half truths in support of bubble gum sales. My advice to the data consumer.. don’t chew gum.