[ TEMPLATE ]
AI pilot evaluation rubric (template)
A four-point rubric for deciding whether to continue after a pilot, with the weekly data to collect, survey items with national comparisons and a continue rule.
How to use this rubric
Score each row from 1 to 4 at the end of a pilot, using data you agreed to collect before it started. A pilot cannot prove a learning gain; with 200 students the smallest detectable effect is about 0.40 SD. Use the rubric to judge reliability and usefulness, and compare products on the same rubric.
The rubric
| Criterion | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| Learning rules hold | Final answers before an attempt are common in Hints or Guided | Some slips; fixed after reports | Rare slips in weekly spot checks | Near zero across weekly spot checks of sampled replies |
| Students can do it alone | No unaided checks run | Pre- and post-checks only | Day-7 rechecks show practiced concepts solved alone | Day-7 and 2-4 week checks both show concepts solved alone |
| Help after an attempt | Not measured | Measured once | Measured weekly | Rising share of help requests follow the student's own attempt |
| Teacher usefulness | Teachers rarely say it helped | Some yes answers, no examples | Yes in half of weekly check-ins | Yes in half or more from week 3, with specific uses named |
| Visibility and privacy | Students unsure who sees chats | Notice exists but unclear | Students told; staff reads logged | Students can explain what is visible; logs reviewed |
| Safety route | No route | Route exists, untested | Tested with the designee | Every flag reached a person within the agreed time |
| Cost | Over budget | At the ceiling | Under the ceiling | Under the ceiling with predictable weekly spend |
| IT fit | Blocked or unstable on school devices | Works with workarounds | Works on school devices and Wi-Fi | Works everywhere students need it, with allow-list tested |
| Equity | Not checked | Checked for one group | Checked by group | No group falls behind; small groups suppressed |
Data to collect each week
- A two-minute teacher check-in: did it help this week, and how?
- A spot check of sampled replies in each class for answers given before an attempt.
- Day-7 unaided rechecks of practiced concepts.
- Model spend per active student.
- Safety flags and the time each took to reach a person.
- Per assignment: "Did you also use another AI tool?"
Student survey items
Ask before and after, using Common Sense Media's wording so you can compare with national figures:
- "I feel I miss out on learning when I use AI." (National: 39% of teen AI users.)
- "AI helps me understand." (National: 66%.)
- "When I get stuck, the first thing I do is..." (National: 10% go to AI first.)
Continue rule
Agree on it before the pilot starts. Continue only if: teachers say yes in at least half of weekly check-ins from week 3 on and name specific uses; learning rules hold; every safety flag reached a person in time; and cost stayed inside the ceiling.
[ FAQ ]
Questions and answers
How do you evaluate an AI pilot in a school?
Score reliability and usefulness: do learning rules hold, can students do the work alone a week later, do teachers find specific uses, does the safety route work, and does cost stay inside the ceiling.
Can a pilot show that AI improves learning?
Rarely. With 200 students the smallest detectable effect is about 0.40 SD, larger than published AI-tutoring effects.
What survey questions should students answer?
Use items with national comparisons, such as Common Sense Media's: missing out on learning, whether AI helps them understand, and their first move when stuck.
[ SOURCES ]
Sources
Facts checked Oct 18, 2026.
- 1Common Sense Media, Teens in the AI Era: Schoolwork and Skills That MatterAug 18, 2026 · commonsensemedia.org
- 2Oreopoulos and Low, One Click Away, NBER working paper w356202026-08 · nber.org
[ AI FOR SCHOOLS ]
The AI your school can approve.