The humanizer benchmark: protocol first, results later

200 human-written source documents across five categories, 40 per cell. Every document is written by a person, with authorship attested, and none has appeared on the public web — a corpus a detector may have trained on measures the wrong thing.

The fifth cell exists because of the published false-positive research. If detectors misclassify non-native English writing at the rates Liang and colleagues reported, then a benchmark drawn only from native-speaker prose will overstate how well everything works.

Every humanizer on our comparison list that offers a testable tier — 13 tools including our own — run over every AI-generated document, at each tool's default setting and, where one exists, its strongest setting. Both are reported separately.

Running rivals on defaults while running your own product on its best setting is the most common way these comparisons are rigged. Reporting both columns for every tool removes the option.

Each detector returns a score on its own scale. We record the raw value and, separately, a pass or fail against that detector's own published threshold — never a threshold we invented, because choosing one is choosing the answer.

Three figures publish per tool per detector: the human baseline false-positive rate, the unmodified AI detection rate, and the post-humanizer detection rate. The first is the one most benchmarks omit, and it is the one that tells you whether a detector is worth listening to at all.

A meaning-preservation check runs alongside. Output that scores well by destroying the argument is not a success, so every sample is scored for whether the claims survived. Any tool can defeat a detector by returning nonsense.

Every run records the date, each detector's version identifier where exposed, and each humanizer's plan and settings. Runs are archived rather than overwritten, so an old result stays readable next to the new one.

Published alongside each run: the full corpus, per-sample raw outputs as CSV, the scoring code, and the summary. If any of those four cannot be released, the run does not publish.

The CSV commitment matters most. A summary table can be argued with; per-sample data can be checked. Anyone who thinks our conclusion is wrong should be able to find the specific samples that show it.