A critical examination of School of One pilot study

School of One is all over the blogosphere. I took a close look at their website tonight for the first time. The program is interesting. It appears that the company is using a comprehensive algorithm that is not only adaptive inside of the box, but outside as well. The algorithm considers how best to meet a student’s individual learning needs (in math) by prescribing group instructional tasks, direct instructional time with a teacher, and remote instruction with an online tutor, as well as adaptive learning with the computer alone. Their videos are impressive, and their gains might be as well. The research they use to support their claims – not so much.

I will be the first shoulder you can cry on if you want to complain about how hard it is to control educational research. I have been involved in three long-term studies of my own as a teacher. Nonetheless, I took a close look at the research results published on the School of One website, and I have to say that the study was not well controlled enough for them to make any claims about the effectiveness of their algorithm, other than to say that School of One, as an after school program, increases student achievement more than students who do not attend the School of One after school program.

In the frothy world of EdTech verve, them’s fightin’ words, I know. So here you go.

First, in each of the schools in which the School of One study occurred, a control group of students uninvolved in the School of One schedule was used to assess the gains of the students in the experimental condition. The graphic below shows the ethnic breakdown of the students in each of the control and experimental groups at the three schools. Look carefully.

The College Board publishes SAT score data annually. Consistently, students of asian descent post the highest scores, followed by caucasian students, then latino students, and then african american students. In the graphic shown, it appears that the racial break down of each of the experimental groups is stacked with a statistically significant proportion of students who are statistically more likely to score higher on the SAT. The only exception to this is in school 1, which has a higher proportion of white students in the control group. You might quip that this was a gain study. Fair enough, but if we interpolate achievement backwards from the SAT to birth, this means that the gains of those with the highest scores on the SAT will be greater during any given time period of their schooling up to that point.

Next, students self selected to participate in the School of One program. It has been demonstrated countless times in education research that choice plays a significant role in student achievement. I did not discover in the report if teachers were also allowed to self select, but if that is the case, this uncontrolled factor makes the results even more suspect.

Finally, there is no discounting the effect that being a part of something new, with shiny new toys and a lot of attention, has on student achievement. We must assume that students were initially impacted by the influx of new resources in their environment. And I think it is a fair inference to make that this impact would decrease over time. Unfortunately, the study lasted only a few months. This would most certainly not have been enough time for this factor to be discounted.

I am a technophile. It is not my intention to bring down School of One. I applaud the organization for their inventiveness, and their willingness to actually test their product in a real school setting. Much of the education research cited by edtech companies is either commissioned or compares students using the product in charter schools to their traditional school peers (who, again, did not make a choice to join a charter school). I sincerely hope that the School of One algorithm is wildly successful and that they make great strides toward closing the achievement gap. As a true technophile, however, I think it is imperative that we critically examine all claims that technology is changing the game. And in sum, I am as yet unconvinced that School of One is doing that in the regular school day.

About Jack West

Teacher, team member, father, neighbor.

3 Responses

  1. It’s critical we develop a better basis for evaluating emerging technologies. It’s pretty easy to say that many many studies are tweaked to demonstrate the best possible results. At the same time, education as a sector has to come to terms with the fact that any new technology is likely to be mediocre at best–at first. It takes multiple iterations of working with students, in different environments, etc., to figure out how to make a new tool “work” — and in fact, what “working” even means. We don’t like to do that in education. Few people are willing to let their kids “work out the bugs,’ or be the guinea pigs for a new idea. Some new ideas will bomb; and a few will be successful — but only over time. We’re forcing all technologies to “declare success” early — simply to have a chance to find their way.

    Is that good? Noooo, but I’m not sure there’s another path.

    So I’d argue we need to:
    1. Be honest about what we’re trying to achieve — and honest about the results.
    2. Be willing to celebrate incremental progress for a while — in hopes of great progress in the future.
    3. Be willing to engage in give & take: the technologists are experimenting, the teachers are experimenting. Both need to talk to one another more!

What do you have to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s