It’s Not the AI Tutor. It’s The Practice That Counts.
A large randomized evaluation of AI tutoring in U.S. public schools set out to test Khan Academy’s AI tutor, Khanmigo. But the study’s most interesting finding is about the power of well-structured practice.
Economists Philip Oreopoulos and Nina Low randomly assigned some middle-school students to use Khan Academy paired with Khanmigo during their existing daily remedial math block, while the control group kept their usual mix of teacher-led practice and worksheets.
The students given the tutoring posted real gains, but those gains were essentially indistinguishable from what Khan Academy can produce with no AI involved at all. The gains were also broadly in line with what computer-assisted math software has produced for two decades. As the authors put it, “nothing in the achievement data requires the AI tutor to explain it.”
In fact, the students barely used the AI tutor at all. The median student messaged it on only a third of the days they practiced. When students did engage with the AI, they were mostly submitting one-word answers or clicking on suggested prompts. Only about one in seven contained an actual math question or piece of mathematical thought.
So what actually moved scores? One, kids got more practice. Students logged roughly 30 extra minutes of practice a week, climbing up to 45 minutes in the second year.
And two, that practice was better targeted. Khan Academy automatically pulled each student’s latest test scores and slotted them into a personalized sequence of exercises. That’s the individualized diagnosis that makes human tutoring so effective and so expensive to scale. The Khan Academy platform did it automatically, at a cost of about $15 per student per year.
That’s a helpful takeaway for school and district leaders. They need to find a good tool, yes, but they don’t necessarily need the most expensive or fanciest models. In fact, the most important thing is figuring out how to boost engagement and make sure that students actually put in the work and do the practice.
Chad Aldeman is a nationally recognized expert on education policy, including school finance; teacher preparation, evaluation, and compensation; and state standards, assessment, and accountability.
It’s Not the AI Tutor. It’s The Practice That Counts.
A large randomized evaluation of AI tutoring in U.S. public schools set out to test Khan Academy’s AI tutor, Khanmigo. But the study’s most interesting finding is about the power of well-structured practice.
Economists Philip Oreopoulos and Nina Low randomly assigned some middle-school students to use Khan Academy paired with Khanmigo during their existing daily remedial math block, while the control group kept their usual mix of teacher-led practice and worksheets.
The students given the tutoring posted real gains, but those gains were essentially indistinguishable from what Khan Academy can produce with no AI involved at all. The gains were also broadly in line with what computer-assisted math software has produced for two decades. As the authors put it, “nothing in the achievement data requires the AI tutor to explain it.”
In fact, the students barely used the AI tutor at all. The median student messaged it on only a third of the days they practiced. When students did engage with the AI, they were mostly submitting one-word answers or clicking on suggested prompts. Only about one in seven contained an actual math question or piece of mathematical thought.
So what actually moved scores? One, kids got more practice. Students logged roughly 30 extra minutes of practice a week, climbing up to 45 minutes in the second year.
And two, that practice was better targeted. Khan Academy automatically pulled each student’s latest test scores and slotted them into a personalized sequence of exercises. That’s the individualized diagnosis that makes human tutoring so effective and so expensive to scale. The Khan Academy platform did it automatically, at a cost of about $15 per student per year.
That’s a helpful takeaway for school and district leaders. They need to find a good tool, yes, but they don’t necessarily need the most expensive or fanciest models. In fact, the most important thing is figuring out how to boost engagement and make sure that students actually put in the work and do the practice.
Chad Aldeman
Recent posts
A Low-Cost, Highly Effective Math Program
Let the March Mathness Begin
Automatic Enrollment Policies Check All the Boxes
Virginia Pushes to Accelerate Students in Math