AI Detection Thresholds: The Dial That Decides Who Gets Flagged

An AI detection threshold is the cutoff score above which a detector calls text "AI." It is not discovered by science — it is chosen by a vendor, and the choice is a business decision that trades false accusations against missed AI. Move the dial one way, you accuse innocent writers; move it the other, AI slips through. Every vendor picks a spot, and the spot tells you who they're afraid of disappointing.

Most writing about AI detectors argues about accuracy percentages. Almost nobody writes about the dial those percentages depend on. That's a strange omission, because the threshold is where the interesting decisions live: it's the single number that converts a model's raw probability into an accusation, a rejection, or an all-clear. Two vendors can run near-identical models and behave like different products purely because of where they set it — which, incidentally, is the biggest reason the same text scores differently across tools. That companion post covers why scores scatter; this one is about the dial itself: what it is, who sets it, and what a well-set one looks like.

Disclosure, since we're naming competitors throughout: we build a detector and humanizer ourselves at HumanFlow. Our editorial policy is to quote vendors' claims with their conditions attached and criticize only with evidence — judge accordingly.

What a threshold actually is

Strip away the interface and every statistical detector does the same two-step. Step one: the model reads your text and produces a score — call it a suspicion level — based on measurements like perplexity (how predictable each next word is) and burstiness (how much sentence rhythm varies). Step two: that score gets compared to a line. Above the line, the tool says "AI." Below it, "human," or "uncertain," or an asterisk.

Step one is machine learning. Step two is policy.

The score itself is continuous — a document might sit at 0.61 suspicion, another at 0.87. Nothing in the math says where "AI" begins. A vendor could draw the line at 0.5, 0.7, or 0.95, and each choice produces a different product from the same model. Lower line: catch more AI, flag more humans. Higher line: protect humans, miss more AI. There is no setting that does both, for the same reason a smoke alarm can't simultaneously be impossible to false-trigger and guaranteed to catch every smolder. Sensitivity and specificity pull against each other, always.

So when a vendor says "our detector is 99% accurate," the honest follow-up question is: at what threshold, and which error did you sacrifice to get there? Accuracy claims without threshold context are like fuel-economy claims without saying highway or city.

The trade-off has a shape (ROC intuition, no jargon required)

Researchers draw this trade-off as a curve. You don't need the plot or its acronym; you need the intuition, and the smoke alarm gets you most of the way.

Imagine your detector's sensitivity dial. At one extreme — maximum paranoia — it flags everything. It catches 100% of AI text, because it catches 100% of all text, including every innocent human. At the other extreme it flags nothing: zero false accusations, zero catches. Both extremes are useless. Every real detector lives somewhere between, and the curve simply records what pairing you get at each dial position: at this setting, you catch 92% of AI while wrongly flagging 3% of humans; nudge it up, 96% and 8%; nudge again, 99% and 15%.

Two things fall out of that picture, and they explain most of the industry's behavior.

First: near the aggressive end, the exchange rate turns brutal. Early dial movement is cheap — you gain a lot of AI catches for a small false-positive cost. But squeezing out the last few percent of missed AI costs disproportionately many innocent flags, because what's left uncaught is precisely the AI text that most resembles human writing. Chasing 99% detection means dragging in the human writers whose prose happens to look machine-typical: non-native English speakers, students trained into rigid five-paragraph structures, technical writers, heavy self-editors. That's not speculation — it's the documented false-positive risk ranking, and it's why Liang et al. (Patterns, 2023) found seven detectors falsely flagging human-written TOEFL essays 61.22% of the time on average while sailing through essays by native-speaking US 8th graders. Aggressive thresholds don't distribute their errors evenly. They concentrate them on predictable groups of innocent people.

Second: a better model moves the whole curve; a threshold only moves you along it. This is the part vendors would prefer you not conflate. Genuine model improvement gives you better pairings at every dial position. Threshold adjustment just picks a different pairing from the same menu. A vendor can "reduce false positives" overnight without improving anything, simply by turning the dial down — and quietly missing more AI. Or advertise a spectacular catch rate achieved by turning it up — and quietly flagging more humans. Both moves generate honest-sounding numbers. Neither makes the detector better.

Why "just make it more accurate" isn't an option

The obvious retort: fine, then build a model so good the trade-off disappears. Vendors are trying. It hasn't happened, and there are structural reasons to expect it won't fully happen.

The core problem is that the two distributions overlap. Detectors measure how machine-typical text is, not who typed it — and some human writing genuinely is machine-typical (formulaic, low-variance, predictable), while some AI writing genuinely is human-typical, especially after editing. Where the distributions overlap, no threshold anywhere can separate them; every possible line either flags the machine-typical humans or passes the human-typical AI. A better model shrinks the overlap. Nothing eliminates it, because the overlap isn't a modeling failure — it's a fact about how text is.

And the overlap is growing on one side: frontier models are explicitly trained to write more like people, and every generation resets detector training data (the vintage problem). OpenAI's own classifier is the standing exhibit — retired in July 2023 after catching just 26% of AI text while falsely flagging 9% of human text, by OpenAI's own announcement. The model's maker, with unmatched access to its outputs, couldn't find a usable point on its own curve.

There's also a nastier constraint that thresholds can't fix: base rates. Suppose a detector is genuinely excellent — 95% catch rate, 99% specificity (1% false positives). Now feed it a course where only 10% of 1,000 submissions involve AI. It catches about 95 of the 100 AI papers. But it also flags about 1% of the 900 honest papers — roughly 9 innocent students. One false accusation for every ten true catches, from a detector performing exactly as advertised. Scale that to Turnitin's volume — 200M+ papers screened in its first year — and a 1% false-positive rate is an enormous absolute number of innocent flags, which is the arithmetic Vanderbilt University published when it disabled Turnitin's AI indicator in August 2023. No threshold setting escapes this. It's division.

Case study in conservative tuning: Turnitin

If thresholds are business decisions, Turnitin's business is not getting its university customers sued. Its tuning shows it, and — credit where due — the engineering is genuinely responsible even if you distrust the category.

Three documented choices define Turnitin's conservative posture. First, the asterisk floor: AI scores between 1% and 19% display as an asterisk, not a number, because Turnitin itself says low-range scores are less reliable. A vendor voluntarily suppressing its own output in the range where it's weakest is rare, and it deserves to be called what it is: honest. Second, the conditional accuracy claim: the famous "98% accuracy, under 1% false positives" applies only to documents where more than 20% of the text is flagged. Turnitin puts the condition in its own FAQ; most people quoting the 98% leave it out. Third, deliberate under-flagging: Turnitin's chief product officer, Annie Chechitelli, put it to BestColleges in April 2023 as "we find about 85% of it. We let probably 15% go by". Read that as a threshold confession — Turnitin chose a dial position that knowingly misses about one AI sentence in seven because the alternative was accusing more innocent students.

Add the ~300-word minimum (below which scores aren't produced at all — the short-text problem in vendor-policy form) and you get a coherent philosophy: when unsure, say less. The cost of that philosophy is equally real, and Turnitin's own scale data hints at it — in its first year, roughly 11% of 200M+ papers showed ≥20% AI writing. A tool tuned to under-flag is, by design, undercounting. Educators who treat a low Turnitin score as proof of human authorship are misreading a conservative instrument as a clean bill of health.

Was it enough? Vanderbilt didn't think so — even sub-1% false positives multiplied across thousands of students was too many wrongful flags for a score that can't be independently audited. Conservative tuning reduces the injustice rate; it can't reduce it to zero, and it can't change the base-rate math.

Case study in aggressive tuning: Originality.ai

Now the other end of the dial. Originality.ai sells primarily to publishers, agencies, and SEO teams — customers for whom the expensive error is missed AI, since undisclosed AI content can cost them clients or search rankings. A false positive, in that market, means an awkward conversation with a freelancer, not a misconduct tribunal.

The tuning follows the customer. Originality.ai advertises 99%+ accuracy on flagship models, and — to its genuine credit — publishes far more methodological detail than most rivals, including per-model false-positive figures. Its current documentation (fetched during writing) lists a "Lite" model at a 0.5% false-positive rate, a "Turbo" model at 1.5% with a slightly higher catch rate, claims of up to 97% detection on "humanized" content, and configurable "AI allowance" settings where the false-positive rate visibly shifts with the setting — 0.5% at one allowance level, 1.5% at another. That is the threshold trade-off sold openly as a product menu: pick Turbo, catch more, flag more; pick Lite, the reverse. You could not ask for a cleaner public demonstration that the dial is a choice.

Quote their caveat too, because it's the fine print that matters: Originality.ai's own accuracy page says no detector will ever be 100% perfect and states that detection scores "should not be used for academic honesty purposes and disciplinary action" — even a low false-positive rate, they write, is too high to rely on for discipline. An aggressively tuned vendor telling schools not to use it for enforcement is the whole thesis of this article in one sentence: the dial that's right for content QA is wrong for accusing a nineteen-year-old, and the vendor knows it.

Neither company is the villain of this story. Turnitin's dial is defensible for education; Originality's is defensible for publishing. The failure mode is transplanting a threshold across contexts — a teacher using a marketer-tuned tool on essays, or a publisher using an education-tuned tool and wondering why AI keeps slipping through.

What a well-chosen threshold looks like

Since the right setting depends on who bleeds when the tool is wrong, "well-chosen" differs by audience. A few honest benchmarks:

Three properties mark a well-set threshold regardless of audience. It's disclosed — the vendor tells you the false-positive and false-negative rates at the setting you're using, not a blended best-case. It's conditional — claims come with their text-length, language, and AI-share conditions attached. It matches the decision — nobody's dial should be doing work its error profile can't support, which is why "advisory only" labels on education tools aren't cowardice but correctness.

For what it's worth, our position at HumanFlow follows from all this: our AI detector shows sentence-level results rather than a single verdict, and we publish no accuracy percentage for it — because an accuracy number without published methodology and threshold conditions is marketing, not measurement. It also doesn't promise to beat any other detector, because nobody can honestly promise that.

How to use this when you're on the receiving end

If a score is being used on you — or you're the one wielding it — the threshold lens gives you exact questions to ask. What threshold produced this flag, and what are the error rates at that threshold? Is the claim conditional (Turnitin's 98% only applies above 20% AI share — was this document above it, or in asterisk territory)? Was the text long enough for the score to mean anything? Is the tool tuned for this context, or is a marketer's dial being pointed at a student? And is a probability being treated as proof — because no threshold, anywhere on the curve, converts a statistical suspicion into an eyewitness.

A threshold is a bet about which mistake matters less. Vendors place that bet for you, mostly invisibly. The least you're owed is to know where the dial is set — and the broader evidence on how these instruments perform under independent testing is collected in our detector accuracy hub, with the Turnitin-specific story at /ai-detection/turnitin.

FAQ

What is an AI detection threshold? It's the cutoff score at which a detector's internal suspicion level becomes a public verdict of "AI-generated." The underlying model outputs a continuous probability; the vendor chooses where to draw the line. Different lines on the same model produce very different products.

Why don't all detectors use the same threshold? Because their customers punish different errors. Education tools tune conservative since false accusations are catastrophic there; publisher-focused tools tune aggressive since missed AI is what costs their customers money. The threshold encodes the business model.

Can a vendor improve accuracy just by moving the threshold? No — moving the threshold only trades one error type for the other. A lower false-positive rate achieved by threshold alone means more missed AI, and vice versa. Only a better underlying model improves both at once, and even then an overlap zone remains where errors are unavoidable.

Is Turnitin's threshold stricter than other detectors'? Turnitin is documented as conservatively tuned: it hides scores under 20% behind an asterisk, applies its 98%/<1% claims only above that line, requires roughly 300 words, and has said, in its chief product officer's words, that it finds "about 85%" of AI text and lets "probably 15% go by". Marketer tools like Originality.ai are openly tuned in the opposite direction.

What's a good false positive rate for an AI detector? There's no universal number — it depends on volume and stakes. Even 1% means about ten innocent flags per thousand honest documents, which is why Vanderbilt disabled Turnitin's indicator in 2023 and why Originality.ai itself says detection scores shouldn't drive disciplinary action. For enforcement decisions, arguably no current false-positive rate is low enough on its own.

Why does the same detector sometimes change its verdict on my text? Vendors retrain models and adjust thresholds without notice, and some (like Originality.ai) offer multiple models or settings with different tunings. Same text, different dial position, different verdict. This is one of five reasons detectors disagree, covered in our companion piece on conflicting scores.

Does a stricter threshold mean a detector is more accurate? No — it means the vendor chose to accept more missed AI in exchange for fewer false accusations. That's often the right choice, especially in education, but it's a values decision about which error is worse, not an accuracy improvement.

Key facts

• Turnitin's 98% accuracy / <1% false-positive claims apply only to documents with more than 20% flagged AI text; scores of 1–19% display as an asterisk instead of a number (Turnitin AI writing FAQ).
• Turnitin deliberately leaves roughly 15% of AI text unflagged to reduce false accusations — chief product officer Annie Chechitelli: "we find about 85% of it. We let probably 15% go by" (BestColleges, April 2023).
• Originality.ai publishes per-setting false-positive rates — 0.5% (Lite) vs 1.5% (Turbo/higher allowance) — a public menu of threshold trade-offs, alongside 99%+ accuracy claims on flagship models (Originality.ai accuracy page, fetched 2026).
• Originality.ai's own documentation says detection scores should not be used for academic disciplinary action, calling even low false-positive rates too high to rely on for discipline.
• OpenAI retired its own classifier in July 2023 at 26% detection and 9% false positives — no usable threshold existed on its curve (OpenAI announcement).
• At 99% specificity, 10,000 human documents still yield ~100 innocent flags — the base-rate arithmetic behind Vanderbilt's August 2023 decision to disable Turnitin's AI indicator.
• Liang et al., Patterns (2023) found aggressive detection concentrated its errors: 61.22% average false-flag rate on non-native English TOEFL essays across seven detectors, versus near-perfect results on native-speaker essays.

Sources

1. Turnitin — AI writing detection FAQ / transparency page (conditional accuracy claims, asterisk range, ~300-word minimum).
2. BestColleges — interview with Annie Chechitelli, Turnitin chief product officer, April 2023.
3. Originality.ai — "AI Content Detection Accuracy" study page (per-model false-positive rates, AI-allowance settings, disciplinary-use caveat; fetched during writing, 2026).
4. OpenAI — announcement retiring its AI text classifier, July 2023.
5. Liang, W., et al. — "GPT detectors are biased against non-native English writers," Patterns (Cell Press), 2023.
6. Vanderbilt University — statement on disabling Turnitin's AI detection, August 2023.
7. Turnitin — first-anniversary press data, April 2024 (200M+ papers; ~11% with ≥20% AI writing).