A Harvard AI tutor more than doubled what an active-learning classroom taught. A chatbot handed to high schoolers left them seventeen percent behind kids who never had it. Same class of tool. The difference was what each design made the human do before the answer arrived. This episode returns to chapter six of AI Empowered, […] Below is the full script of this episode as an article.
Picture a Harvard student. Intro physics. It’s a weeknight, she’s at her kitchen table with a laptop, and instead of going to class she’s working through a lesson on surface tension with a chat bot.
And this bot is annoying. Not in the way you’d expect. It has the full worked solution sitting right there in its instructions. It just won’t hand it over. It walks her through the problem one piece at a time, and she has to work each piece before it moves on. When she’s wrong, it says why, right then.
Meanwhile, the rest of her section is in the classroom. Active learning. Small groups, hard questions, the way the course already ran. The same teaching approach the bot was built on, delivered the way it had always been delivered.
Next lesson, they swap. Everybody gets both.
A hundred and ninety-four students. A quiz after every lesson. And the students who learned from the bot gained more than double what the students in the classroom gained. More than double, on the median. And they did it in forty-nine minutes against an hour in the room. They reported being more engaged. Eighty-three percent said the bot’s explanations were as good as or better than their instructors’.
What the study actually tested
That study came out of Harvard, it went through peer review, and it ran in Scientific Reports last year. Names are in the show notes. When the early version got around in twenty twenty-four, the headline everywhere was the same. AI tutor doubles learning.
I get it. Great headline. It’s also the wrong takeaway, and the authors’ own limitations section tells you why.
Because here’s what did NOT happen in that study. Nobody tested an AI against a human. Both groups got the same research-based teaching. What changed was the design of the delivery. One design was a room with instructors. The other was a bot that had been built, deliberately, to make the student do the thinking first. Scaffolding. Managing how much lands on you at once. Feedback right when the mistake happens. Your own pace.
And nobody tested a plain chat bot, the kind you’d get if you just handed students a login. That arm doesn’t exist in this study.
So the honest headline is something like… design was the variable. Worse headline. Better finding.
The other half of the story
Back in July I did an episode on the deskilling trap, and the study at the center of it was a different experiment. About a thousand high school math students in Turkey. Some kids got open access to a chat bot during practice, some got a scaffolded version that gave hints and asked questions instead of handing over solutions, and some got nothing.
During practice, both AI groups did better. Open access, forty-eight percent better. Scaffolded, a hundred and twenty-seven percent. Then the researchers took the tool away for the exam. The open access kids landed seventeen percent behind the kids who never had it. The scaffolded kids mostly held level with them. The gain during practice didn’t turn into a debt.
That episode was about loss. And if that’s the only story you’ve heard about scaffolding, you’d come away thinking it’s a tax. Something you pay to avoid getting hurt.
The Harvard result is the other half. Same discipline of design. Opposite direction. Scaffolding wasn’t the insurance policy there. It was the whole product.
Design was the variable
Put the two studies side by side and the variable isolates itself.
Neither one is about how smart the model is. The Turkish study’s two AI arms were the same kind of chat bot. One had guardrails. That’s the whole difference. The Harvard study used a strong model, and it was told to guide, one step at a time, instead of answer. In both cases the tool’s capability was a constant. What changed was what the interaction made the human do before the answer showed up.
And the same class of tool went both directions. One design left kids seventeen percent behind. Another design more than doubled a Harvard classroom’s gains on an intro physics quiz.
This is the heart of chapter six of the book. The collaboration demands more of me, not less. In the chapter I make that case with the Turkish study, and I lean on something a psychologist named Robert Bjork worked out in the nineties. Desirable difficulties. Conditions that make practice harder in the moment and make you better over time. Spacing practice out. Quizzing yourself. Making yourself commit to an answer before anyone shows you one.
The scaffolded bot, in both studies, is a desirable difficulty wearing a chat window. It’s harder on purpose. And the harder version won twice, for two different reasons. Once by keeping the skill from leaking out. Once by building more of it, faster, than a well-run active learning classroom did on the same quiz.
What this can’t hold
Before you carry this anywhere, here’s what it can’t hold.
The Harvard study only measured right after the lesson. No retention test. It says nothing about deskilling. That half of the argument comes from the Turkish study, and I want to keep the two findings apart, because the whole point depends on it.
The Harvard bot lesson happened at home, at the student’s own pace, one on one. The classroom lesson didn’t. So self-pacing and undivided attention are tangled up with the design, and the authors say so.
The material was introductory. Understand it, apply it, analyze it. The authors are careful not to claim this for synthesis or higher-order judgment. So when I carry it into the kind of work you and I do… strategy, diagnosis, the calls where you’re paid for judgment… that’s my extension, and the book’s. The direction of the evidence supports it. Nobody has run this in a boardroom.
And both of these are students. You’re not a student. Your Monday isn’t a physics quiz. That gap is real, and I’m naming it. What carries across is the mechanism.
In my own work
Let me tell you where I see this in my own work, because I think it’s the same rule.
At Era, we’re building an app. The next piece in the spec is a coach. A conversation that helps you get better at working with your own AI tools. And there’s a rule in the spec for that coach that I want to describe, because it’s the Harvard design pointed at adults.
The coach asks what you think before it offers a view. It asks for your evidence. If you ask it to write the thing you’re working on, it declines. I won’t write that for you. Here’s the question you should ask your tool, and here’s what to check in what comes back. And every exchange has to end with a decision you make, not one the coach makes for you.
There’s even a second check behind it. If a reply does your task for you, or doesn’t end with something for you to decide, it gets flagged and generated again.
Here’s the thing, though. Every one of those rules makes the app harder to use than the alternative. My bet is a tool that just answers would test better early. But there’s a line in the spec worth saying out loud. Dependency on the system is a failure mode. If you stop needing the coach because you got better, that’s the win. If you keep coming back because you can’t decide without it… we built the seventeen percent.
It’s the same reason client strategy work at Auspicious doesn’t start with a prompt. It starts with my own call, on paper, and the model gets to argue with that.
The Monday version
So what’s the Monday version of this?
You’re designing the scaffold whether you know it or not. If you rolled AI out as access… licenses, a policy, maybe a training day… that is a design. That’s your open access arm, or the closest thing you’ve got to one. The version that looked great during practice and cost the kids on the exam.
Pick one AI workflow you own. Just one. Ask a single question about it. What does it make the human do before the answer arrives?
If the answer is nothing… if the first move is paste the task, take the output… you’re in the arm of the experiment nobody wants to be in, and a better model probably won’t move you out of it. The fix is a design decision, and it’s one you already have the authority to make.
Where you make the human commit first, ask the tool to argue instead of answer, or check the output against something they wrote down before they saw it… that’s the scaffolded arm. It’s slower at first. If the mechanism carries over to judgment work the way I think it does, it’s the arm that’s still standing when the tool goes quiet.
Same tool, two designs, opposite outcomes. You’re choosing one of them either way.
—
This show extends the book AI Empowered. Read the first chapter free at https://www.aiempoweredbook.com, or get the book on Amazon.
Aaron Douglas runs Auspicious, a fractional marketing practice.
Blog: https://auspicious.llc/insights/
X: https://x.com/aaron_douglas
LinkedIn: https://www.linkedin.com/in/aaronddouglas/
