The evidence on AI tutoring: what helps, what hurts

[ EVIDENCE ]

The short answer

Guardrails stop the harm. Structure and teachers create the gains. When AI does the thinking, students skip the practice that builds skill, and it shows on tests without AI. Hint-first AI prevents that harm but rarely adds gains on its own. The clearest gains come when AI sits inside structured practice or helps a person who is teaching.

Everyday use points the same way

In a 30-month study of 26,811 Chinese students in grades 7-12, adopting AI raised homework scores 18% and cut monthly closed-book exam scores 20% within six months. Students who kept spending normal time on homework lost little. It is a working paper and not randomized, and losses grew with self-reported hours of AI use.

On PISA 2025, students who used AI to draft, summarize or research scored lower in science than non-users; students who used AI as a general learning aid scored about the same. This is a correlation.

Hint-first alone rarely adds gains

In a two-year randomized trial in 18 middle schools, Khan Academy with Khanmigo raised achievement about 0.06-0.08 SD a year, similar to Khan Academy practice without AI. The median student messaged the AI tutor on a third of practice days. Engagement, not model quality, set the limit.

Gains come with structure or a person

• AI that helps a tutor: in a randomized trial with 900 tutors and 1,800 K-12 students, students whose tutors had AI advice were 4 points more likely to master topics (preprint).
• AI inside mastery practice: with 6,000+ middle schoolers, AI support improved the next attempt after a mistake, and AI inside a mastery workflow gave marginally significant gains on practiced material (NBER working paper).
• Structured tutoring in one course: a Harvard physics tutor with expert-written solutions and a forced step order raised post-test scores; the authors built it, and retention was not measured.
• Teacher-guided use: a six-week after-school program with teacher guidance in Nigeria gained 0.31 SD overall (World Bank working paper).

Claims to treat with care

• A widely shared meta-analysis claiming large learning gains from ChatGPT was retracted on Apr 22, 2026.
• Large gains reported for one university physics course came from a tutor its authors built, and retention was not measured; it is not a general result.
• Brain-scan claims about ChatGPT rest on a preprint with 54 adults; EEG is not a measure of learning.
• Vendor studies that measure usage or satisfaction do not measure learning.

How to read a study in two minutes

Ask four questions. Was it randomized, or a correlation? Was the outcome measured without AI, after a delay? Who built the tool, and who ran the study? Is it peer-reviewed, a preprint or a working paper? A result that passes all four is rare, and worth more than many that pass none.

What it means for choosing a tool

Ask vendors for outcomes measured without AI, after a delay, with a comparison group and raw counts. Nexus has no outcome data yet and makes no learning claim. It is built to keep students doing the thinking, and our pilots measure delayed unaided success rather than usage.