How to test an AI detector yourself: our open replication framework

A trustworthy AI detector test needs five things almost no online "we tested them" article has: a published corpus, dated model versions, disclosed detector settings, pre-committed scoring rules, and scheduled reruns. This post gives you that protocol in full — the same discipline we already committed to in our published humanizer benchmark, turned on detectors — so you can replicate it, criticize it, or beat us to it.

Disclosure first, since this whole post is about conflicts of interest: we build a detector and humanizer ourselves. Here's our editorial policy — judge accordingly. In fact, judge especially accordingly, because a detection company publishing a detection benchmark is exactly the situation the protocol below is designed to make safe. The controls that keep us honest are the same ones that would keep anyone honest.

The reproducibility crisis nobody calls by name

Search "best AI detector 2026" and you'll find dozens of articles announcing that somebody "tested" ten tools and crowned a winner. Read them closely — actually closely — and a pattern emerges. The corpus is never published. "We used 20 AI samples and 20 human samples" — which ones? Written by whom, when, generated by what model, at what version, with what prompt? Silence. The test date is missing, though every major detector retrains continuously, so an undated result describes a product that no longer exists. Settings go unreported: most detectors expose sensitivity options or report scores against tunable thresholds, and the difference between "flagged at 50%" and "flagged at 90%" is the difference between opposite conclusions. Scoring rules get invented after the results arrive — is 47% AI on a human essay a pass or a fail? Depends what makes the affiliate-linked winner look good.

That last point isn't a cheap shot; it's the economic engine of the genre. A large share of detector round-ups earn commission on the tools they rank, and almost none disclose which results would have changed their recommendation. Even the honest ones are unreproducible, which makes them unfalsifiable, which makes them — in any scientific sense — not tests at all. They're anecdotes with tables.

The vendors themselves sometimes do better. Originality.ai, to its credit, publishes recurring accuracy studies with named datasets and methodology notes, including studies of its competitors — and it is still a company grading its own homework, with every incentive to choose favorable corpora. Pangram publishes third-party university evaluations. These are steps in the right direction that no one independent has systematically checked. Meanwhile the one genuinely rigorous public instrument, the RAID benchmark (ACL 2024) — over 6 million generations across 11 models, 8 domains, and 11 adversarial attacks — found that detectors advertising 99%+ accuracy were "easily fooled by adversarial attacks, variations in sampling strategies, repetition penalties, and unseen generative models." The gap between the blog-post genre and the academic result is the reason this manifesto exists.

The stakes aren't abstract. Institutions buy detectors based on these claims, then point them at students — and the peer-reviewed record shows what happens when nobody checks. Liang et al. (Patterns, 2023) found seven detectors falsely flagged non-native English speakers' TOEFL essays 61.22% of the time on average; we've dissected that paper line by line in our deep-dive on the Stanford TOEFL study. The detectors that failed that test all had glowing informal reviews somewhere.

What makes a detector test reproducible

Strip away the specifics and a legitimate test has a short list of properties. Each one closes a specific door that sloppy or motivated testing walks through.

Provenance-verified human text. You cannot measure false positives without text you know is human, and "I wrote this myself last night" is not verification — it's testimony. The gold standard is text that provably predates generative AI, or fresh text written under observation with signed attestation.

Fully specified AI text. "We used ChatGPT" is not a specification. Model name, version identifier, date of generation, full prompt, temperature and sampling settings, and any post-editing — all of it, or the AI side of the corpus can't be rebuilt by anyone else.

Pre-committed scoring rules. Decide before scanning what counts as a pass: which threshold, whose default, how partial scores on mixed text are handled. Publish the rules first, then the results, in that order. This is the pre-registration principle from clinical research, and it exists because humans — all humans, including us — rationalize after seeing data.

Versioning and dates. Every scan is a photograph of a moving target. A result without a date is noise; a result with one is history, and history accumulates into trend lines.

Published raw data. Not just the summary table — the per-document scores, as CSV, so a skeptic can recompute every number and slice the data along axes we didn't think of.

Everything below is these five principles turned into a working procedure.

The HumanFlow open protocol

We run one benchmark on this basis already: our humanizer benchmark protocol is published in full, with no results yet and the first run due Q4 2026. What follows is the same discipline applied to detectors — quarterly, versioned, raw data downloadable. Here is the entire method, published before anyone has run it, which is the point.

Corpus construction

The corpus has three arms, each targeting a different failure mode.

Arm A — verified pre-2022 human text (n = 150). Human writing whose provenance predates ChatGPT's November 2022 release, drawn from dated, archived sources: essays and articles with archive.org captures from 2021 or earlier, published student writing from pre-2022 corpora, and pre-2022 forum and blog posts with verifiable timestamps. Fifty documents each from three registers — academic/student prose, professional/journalistic prose, and casual web writing — at 300 to 1,200 words, since most detectors state minimums around 300 words. One caveat we flag now rather than in a footnote: public pre-2022 text may appear in detectors' training data, which could deflate false positive measurement. That is precisely why Arm B exists.

Arm B — fresh volunteer human text (n = 75). Newly written prose from adult volunteers who attest, in writing, that no AI touched the draft, composed in an editor whose version history we retain. Volunteers span native and non-native English writers, and we will report those subgroups separately — the central lesson of the Liang paper. This arm cannot have leaked into anyone's training data, because it will not exist until the week we run the test.

Arm C — multi-model AI text (n = 150). Generated fresh each cycle from at least five current models across at least three providers (specific models and version strings published with each run), using three prompt styles per model: plain ("write an essay about X"), styled ("write in a personal, informal voice"), and instruction-matched to the human arms' topics. Default temperature unless stated. Topic-matched to the human arms so detectors can't succeed by learning topics instead of authorship. Sub-arm C2 (n = 45, drawn separately): AI drafts lightly human-edited under a fixed budget — up to 10% of words changed, no sentence added or deleted — because mixed text is the documented hard case, the one Turnitin itself acknowledged struggling with in the Washington Post's April 2023 test.

Total corpus per run: 420 documents. Every document's text (or, for volunteer work, an irreversible fingerprint plus register metadata where contributors decline publication) ships in the public repository.

Scoring rules — committed now

Each detector scans each document once, on the vendor's default settings, on a recorded date, from a standard account tier we pay for ourselves. We record the raw percentage or verdict exactly as displayed. A human document counts as a false positive if the detector reports it as AI at or above the vendor's own default decision boundary — the boundary the vendor's product presents to an ordinary user, documented per tool in the results. An AI document counts as detected by the same boundary, symmetric rules. Mixed-arm documents are reported separately and never blended into headline rates. We publish per-document raw scores, so anyone who prefers a different threshold can recompute everything. Confidence intervals: Wilson score, 95%, alongside every rate — at n = 150 per cell, rates carry meaningful uncertainty, and pretending otherwise is how marketing numbers happen. Where a detector reports sentence-level rather than document-level results, we record the document-level summary score and note the display style.

What we will report, in framing terms: rates, intervals, and dated version context — never a crowned "most accurate detector," because a single quarterly snapshot cannot support that sentence honestly.

Cadence and versioning

The full protocol reruns quarterly. Each run gets a version tag (BENCH-2026-Q4 style), a frozen corpus manifest, and a changelog listing any protocol amendments — amendments announced in the changelog before the run they affect, never after. Arm C regenerates with current models every run; Arms A and B persist so false positive trends stay comparable across time. Detector list per run: the majors covered on our accuracy hub, plus tools readers nominate, capacity permitting.

What the tables will hold

Each run reports two tables. Table 1 gives headline rates per detector — version and scan date, human documents scanned, false positives, FP rate with a Wilson 95% interval, AI documents scanned, detections, detection rate with its interval. Table 2 splits the hard cases: false-positive rate for native volunteers, for non-native volunteers and for pre-2022 archive text, against detection rates on unedited and on lightly edited AI.

When those tables fill, we will report exactly what they show — including any result that makes our own detector look bad, which gets scanned under identical rules and printed in the same table. Until then this article claims no measurements of its own. Everything above rests on the published record cited throughout, and the protocol stands without the numbers.

An invitation to replicate

The protocol above is free to take. Run it yourself — all of it or any arm — and publish what you find, whether you're a student with a weekend, a journalist with a deadline, or a vendor with a grievance. We ask only what the method asks: publish your corpus, your dates, your settings, and your scoring rules before your conclusions. If your numbers disagree with ours, that disagreement is signal, not embarrassment; detectors change monthly, corpora differ, and mapping that variance is the entire point of replication. We'll link serious independent replications from the benchmark page, including ones that contradict us. Especially ones that contradict us — a benchmark only we can pass is called marketing, and there's plenty of that already.

If you want to sanity-check the mechanics before committing to a full run, our own AI detector is free for 10,000 detection words a month with sentence-level readouts — and in the spirit of everything above: we publish no accuracy percentage for it, because until this benchmark's methodology produces one in public, any number we quoted would be exactly the kind of claim this post tells you to distrust. It doesn't promise to beat any detector either, because nobody can honestly promise that.

What to demand from anyone's detector test

You'll read ten more "we tested AI detectors" articles this year. Here's the interrogation checklist — five questions, one minute.

Any article failing three or more isn't a test — file it under entertainment. And one structural warning that applies even to honest efforts: a single-run test cannot see the failure modes that matter most at scale. The base-rate arithmetic means a detector with a seemingly small false positive rate still generates streams of innocent flags across thousands of documents, and subgroup failures — the Liang paper's 61.22% on non-native writers — vanish inside healthy-looking averages. That's why our protocol reports subgroups separately, and why enterprise buyers should demand the same before signing anything.

Why we're doing this in public

The detection market runs on asymmetric information. Vendors publish accuracy numbers few can check; content mills publish rankings nobody should check; institutions make consequential decisions downstream of both. The academic corrections — the Liang paper, RAID, OpenAI's own July 2023 admission that its classifier caught just 26% of AI text — arrive late, land hard, and get absorbed slowly. A standing public benchmark, run on a schedule, with frozen methodology and open data, is the boring, unglamorous fix: it converts a market of claims into a record of measurements.

We are not neutral, and won't pretend to be — a company that sells detection benefits when detection is taken seriously, and a company that sells humanization benefits when detectors look fallible. Both incentives point at us at once, which is either a fatal conflict or, handled in the open, a decent set of opposing checks. The protocol is our answer: every rule above was chosen so that our conflicts can't quietly steer the results, and every raw score ships so you don't have to take that on faith. Our benchmark page is where that work gets published.

FAQ

What is a good AI detector test methodology? One a stranger could rerun: a published corpus with verified human provenance, AI text specified down to model version and prompt, detector settings and decision thresholds disclosed, scoring rules committed before scanning, scan dates recorded, and raw per-document results released. Missing any of these, results can't be checked — and unchecked results are marketing.

Why do AI detector reviews disagree so much? Different corpora, different dates, different thresholds, and different incentives. Detectors retrain continuously, so two honest tests months apart measure different products; the same text also scores differently across tools because each vendor sets its own decision threshold. Undisclosed affiliate relationships do the rest.

How many samples do you need to test an AI detector? Enough that rates carry tight confidence intervals — with 100 human documents, a measured 5% false positive rate spans roughly 2–11% at 95% confidence. Our protocol uses 420 documents per run with Wilson intervals reported. Twenty samples, the round-up standard, can't distinguish a good detector from a coin with a limp.

Can I run this protocol myself? Yes — that's the invitation. Use the three-arm corpus design (verified pre-2022 human text, fresh attested human text, version-documented AI text), commit scoring rules first, record dates, and publish raw scores. Scaled-down versions are fine if every rule is disclosed.

Why does human text need to be pre-2022 or freshly attested? Because false positive measurement collapses if any "human" document might be machine-touched. Text archived before ChatGPT's November 2022 release is provably clean; fresh text written under attestation with version history is verifiably clean and can't be in any detector's training data. Everything else is trust, not verification.

What's wrong with vendor-published accuracy studies? They're not worthless — Originality.ai's, with named datasets, are more checkable than most blog tests. But a vendor chooses its own corpus, threshold, and comparison set, and grades itself. Independent benchmarks like RAID (ACL 2024) consistently find real-world robustness far below advertised figures.

When will HumanFlow publish its first benchmark results? Our humanizer benchmark has its protocol published and its first run due in Q4 2026. The detector protocol in this post is committed now, in advance, so that whenever it runs — by us or by anyone who takes it — the results can be judged against a method that was fixed beforehand, not fitted afterwards. We would rather publish a dated method with no results than an undated result with no method.

Key facts

• The RAID benchmark (ACL 2024): 6M+ generations, 11 generator models, 8 domains, 11 adversarial attacks — detectors claiming 99%+ accuracy were "easily fooled" by attacks, sampling changes, and unseen models.
• Liang et al. (Patterns, 2023): 61.22% average false positive rate on 91 non-native speakers' TOEFL essays across seven detectors; 89 of 91 flagged by at least one tool.
• OpenAI retired its own AI text classifier in July 2023 after it caught only 26% of AI text and falsely flagged 9% of human text.
• Turnitin displays an asterisk instead of a score below 20% flagged content and requires roughly 300 words — vendor-side acknowledgment that low-range and short-text results are unreliable.
• The Washington Post's April 2023 test found Turnitin struggled most on mixed human/AI drafts — the condition our protocol isolates as Arm C2.
• HumanFlow's protocol: 420 documents per run across three arms, vendor-default thresholds, Wilson 95% intervals, quarterly versioned reruns, raw CSVs published.
• Statistical reality: with n = 100, a measured 5% false positive rate carries a 95% confidence interval of roughly 2–11% — small tests cannot support strong rankings.

Sources

1. Dugan, L., Hwang, A., Trhlik, F., Ludan, J.M., Zhu, A., Xu, H., Ippolito, D., Callison-Burch, C. "RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors," ACL 2024 (arXiv:2405.07940).
2. Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., Zou, J. "GPT detectors are biased against non-native English writers," Patterns 4(7), Cell Press, 2023.
3. OpenAI — "New AI classifier for indicating AI-written text" (January 2023; July 2023 retirement update).
4. Turnitin — AI writing detection FAQ / transparency documentation (asterisk policy, ~300-word minimum, claim conditions).
5. Fowler, G. "We tested a new ChatGPT-detector for teachers. It flagged an innocent student." The Washington Post, April 2023.
6. Originality.ai — self-published accuracy study series (originality.ai/blog/ai-accuracy), referenced as an example of vendor-run methodology.
7. Pangram Labs — published third-party evaluations (pangram.com), referenced as an example of vendor-commissioned external testing, checked August 2026.