AI detectors for teachers: reading a score before you act on it

An AI detector does not know who wrote anything. It has no record of authorship, no browser telemetry and no access to the document's history. It reads the prose and estimates how statistically typical of machine writing it looks — mostly through perplexity, how predictable each next word is, and burstiness, how much sentence length and structure vary.

Human writing tends to be uneven. Model writing tends toward smooth and medium-everything. That difference is the whole mechanism, and it is why the tools struggle with the specific kinds of human writing that happen to be even and conventional.

So a score of 80% does not mean an 80% chance the student used AI. It means the text carries features the model associates with machine writing. Those are not the same claim, and the gap between them is where nearly every unfair accusation lives.

In 2023, Weixin Liang and colleagues at Stanford published a study in Patterns testing seven detectors against 91 TOEFL essays written by real people — non-native English speakers sitting an English exam. The detectors flagged an average of 61.22% of those essays as AI-generated. Eighty-nine of the 91 were flagged by at least one tool.

The same detectors classified essays by US 8th-grade students almost perfectly. The variable was not whether a machine wrote the text. It was whose English was being read.

Turnitin states its own false positive rate is under 1%, and that figure deserves to be reported accurately: it applies only to documents where at least 20% of the qualifying text is flagged. Below that threshold Turnitin shows an asterisk rather than a percentage, because its own testing found more false positives in that range. Any summary quoting the sub-1% figure without that condition is quoting it wrongly.

Turnitin also funded research testing roughly 2,000 samples from English language learners, reporting false positive rates of 0.014 for ELL writers against 0.013 for native speakers on documents over 300 words. That is a real result and a direct answer to the bias charge. It was also paid for by Turnitin, which does not make it wrong but is worth knowing when the strongest evidence for a tool comes from the company selling it.

The full evidence review is here, including where the studies disagree and why.

An accusation is not a neutral act. Students who are wrongly accused describe it as one of the worst experiences of their education, and the damage does not undo cleanly when the case is dropped. The base rates above mean that if you act on scores alone across a cohort, you will be wrong about some students — and disproportionately about the ones already at a disadvantage.

A related trap: students who did nothing wrong sometimes confess anyway, to end the discomfort. If your process makes admission the fastest route out of a difficult meeting, it will produce admissions regardless of what happened.

None of this is an argument for ignoring AI use. It is an argument for treating the score as the beginning of an enquiry rather than the end of one.

The teachers who report the fewest disputes tend to have changed the assignment rather than the enforcement. A few patterns that come up repeatedly:

Ask for the process. Outlines, annotated drafts and a short reflection on what changed between versions are hard to fabricate convincingly and useful to read regardless.

Anchor to specifics only they have. A seminar discussion, a local case, a dataset from your own module. Generic prompts get generic answers, from people and models alike.

Write the policy precisely. State which uses are fine, which need disclosure, and which are not permitted — and say explicitly whether spellcheck and grammar tools count. Most disputes trace back to a policy that said "no AI" and left everyone guessing.