Our data
We asked 16 AI models to teach certification exams from memory. None passed.
If you ask a chatbot to teach you RBT, forklift or CDL air brakes material without handing it the source, expect it to get some test-sized details wrong. We had 16 models write the same 9 lessons from memory, and every one of them got between 2 and 9 lessons critically wrong. Given the official text, four of them wrote all 9 without one.
We make ClassMetric's lessons with AI, so we needed the answer for ourselves. On November 12 we had to pick the model that writes them, and decide whether it needs the source in front of it.
The test we ran
We picked three subjects where one wrong number can cost you a question, or worse: the RBT exam, forklift operator training under OSHA's rule, and the air brakes section of the CDL manual. For each we wrote three lesson briefs. Each one held excerpts from the official source, the 13 or 14 facts a good lesson must teach, and 5 or 6 tempting wrong claims we call traps.
Then 16 models from 10 families each wrote one lesson per brief, twice. Once with the source excerpts in the prompt, and once with nothing but the topic, from memory. Two judge models read every lesson claim by claim against the sources and marked each one as supported, unsupported or contradicted, and whether an error was critical: the kind that would teach you the wrong answer on the test or the wrong move on the job.
Every model slipped from memory
From memory, 77 of 144 lessons we asked for had at least one critical error. With the source, 30 of 143 did.
The numbers as a table
| Model | With the source | From memory | Claims that hold, from memory |
|---|---|---|---|
| GLM 5.3 | 3 of 9 | 9 of 9 | 84% |
| MiniMax M3.1 Flash | 3 of 9 | 8 of 9 | 77% |
| Gemini 3.8 Flash | 7 of 8 | 7 of 9 | 90% |
| Claude Haiku 5.5 | 4 of 9 | 7 of 9 | 90% |
| Qwen 3.8 27B | 4 of 9 | 7 of 9 | 80% |
| Gemini 4 Carbon | 3 of 9 | 5 of 9 | 94% |
| Claude Sonnet 5.5 | 1 of 9 | 5 of 9 | 91% |
| Claude Opus 5.5 | 0 of 9 | 5 of 9 | 95% |
| Grok 4.7 | 2 of 9 | 4 of 9 | 96% |
| DeepSeek V4.1 Flash | 0 of 5 | 4 of 6 | 96% |
| Mistral Large 4 | 0 of 4 | 4 of 6 | 90% |
| GPT-6.1 Terra | 1 of 9 | 3 of 9 | 95% |
| GPT-6.1 Sol | 0 of 9 | 3 of 9 | 97% |
| GPT-6 Luna | 2 of 9 | 2 of 9 | 95% |
| Claude Fable 5.2 | 0 of 9 | 2 of 9 | 98% |
| Kimi K3.1 Pro | 0 of 9 | 2 of 9 | 96% |
The memory lessons read well, and that's the problem. On average 91.5% of their claims held up, against 97.7% with the source. Claude Fable 5.2 got 98% of its claims right from memory and still had 2 lessons with a critical error. A lesson can be right nine sentences in ten, and the tenth is a number or a rule the exam asks about, written in the same confident voice as the rest.
Claude Opus 5.5 wrote all 9 lessons with no critical error when it had the source, and 5 of 9 with one when it didn't. GPT-6.1 Sol went from 0 to 3. Traps, the wrong claims we planted in the answer key, showed up 3 times from memory and never with the source.
One mistake, up close
GPT-6.1 Sol, writing its RBT lesson on professional conduct from memory, gave its trainee Maya this rule.
"Monthly supervision covers at least 5% of Maya’s behavior-analytic service hours: two real-time, face-to-face contacts, including one observation of service delivery. At least half the supervision must be individual."
Almost all of that is right. The last sentence isn't. The RBT Handbook (page 17) says: "At least one of the two supervision sessions must be individual (i.e., no other RBTs or trainees present), but the other may occur in a small-group meeting." The handbook counts sessions, not hours: of the two monthly sessions, one has to be just you and your supervisor, and the other can be a group. “Half the supervision” sounds close enough, and that closeness is what makes it a good wrong answer on a test. Both judges flagged it as critical, and it is exactly the sort of detail a supervision question turns on.
Claude Sonnet 5.5, writing a CDL lesson from memory, told drivers that at 55 mph "you travel roughly 40 feet" during air brake lag. Half a second at 55 mph does come to about 40 feet, but the CDL manual (section 5.4.4) says the lag "adds about 32 feet," and that's the number the test asks for. Neither mistake would look wrong to someone who is still learning the material, and that's who reads these lessons.
With the handbook text in its prompt, GPT-6.1 Sol wrote every lesson without a critical error, at about 2.5 cents a lesson. It's the model that now writes ClassMetric's lessons, always with the source in the prompt.
How we ran it, and what it doesn't show
The run was on November 12, 2026, and cost $23.63 in API calls, most of it for the judges. Gemini 4 Carbon and Grok 4.7 judged the lessons written with the source, with GPT-6.1 Terra standing in when the writer came from either family. The memory lessons were judged earlier by two of GPT-6.1 Terra, Claude Sonnet 5.5 and Gemini 4 Carbon. Either way, no model graded its own family. Before judging, a simple phrase check looked for the must-have facts and the traps. A model passed only if, across its 9 lessons, at least 95% of claims held, no lesson had a critical error, no trap appeared and at least 80% of the key facts were taught. From memory, none passed. With the source, four did: GPT-6.1 Sol, Kimi K3.1 Pro, Claude Opus 5.5 and Claude Fable 5.2.
It's a small test. Three subjects and 9 lessons per model, one try each, so a model's count could move by a lesson or two on another run. The judges are models too. They agreed on whether a lesson had a critical error 84% of the time, and some of their calls are strict. Both marked Claude Opus 5.5's "a red light or buzzer" as a critical error because the manual requires a warning you can see, though a federal rule allows one you can hear. And we tested models writing lessons, not a chatbot that searches the web while it answers. If you want the full method and every lesson's verdicts, write to us at hello@classmetric.com.
If you're studying with a chatbot
A chatbot is a patient explainer, and our data says it gets most things right. The trouble is that you can't tell which sentence is the wrong one, and exams are written around exactly those sentences: thresholds, counts, who may do what, and words like "only" and "at least."
So give it the source. For the RBT, that's the BACB's Test Content Outline and the RBT Handbook, both free. For forklifts, OSHA's rule at 29 CFR 1910.178. For air brakes, section 5 of your state's CDL manual. Paste the part you're studying and ask it to answer only from that text and quote the line it's using. When it states a number or a rule, ask where the source says so. If it can't quote it, treat it as unchecked. Use it to quiz you and to explain the why, and let the official document settle any argument.
In our run, having the source text was the difference between four passing models and none.
Try it on one lesson
Every ClassMetric lesson is written with the source text in the prompt and checked against it before it ships. Lesson 1 of each class is free in your browser, with no account: the RBT exam prep class, forklift operator and CDL air brakes. If you're comparing RBT study options, we also wrote up how the RBT prep options compare and an RBT study plan built on where the exam's points are.