Confidence interval

In plain English

Measure anything on a small sample and the result is a rough guess wearing a decimal point. The interval says how rough.

It is the difference between "we measured 5%" and "the real number is somewhere between about 1% and 24%, and 5% is our best guess".

A worked example

The same measured rate — one wrongly flagged document — from three test sizes. Intervals are 95% Wilson score intervals, the standard method for proportions.

  1 error in     20 documents  = 5.00%   plausibly  0.9% – 23.6%
  1 error in    200 documents  = 0.50%   plausibly  0.1% –  2.8%
  10 errors in 10,000 documents = 0.10%  plausibly  0.1% –  0.2%

  same kind of finding, three completely different amounts
  of information

The first row is the one to be careful with. A blog post reporting "5% false positive rate" from twenty documents is compatible with the true rate being under 1% or being nearly a quarter — it has barely narrowed the question.

The third row is what a claim like "1 in 10,000" needs behind it to mean anything, and it is why sample size is the first thing to ask about any published rate.

Nothing here says the small test was done badly. A small sample is simply a small sample, and reporting its point estimate without its interval is what turns an honest measurement into an overstatement.

Why it matters for AI detection

Because this site and every other one repeats detector accuracy figures, and the ones drawn from small independent tests carry intervals wide enough to overlap with figures they are quoted as contradicting.

It is also the fair reading of vendor claims in both directions. A vendor's low rate measured on a large private corpus may be tighter than an independent test on 50 documents, even though the independent test sounds more trustworthy — sample size and independence are separate virtues.

For an appeal, the practical use is narrow but real: if a figure is being quoted at you from a small study, the size of that study is a legitimate question, and so is whether anyone has reported the interval.

Commonly confused with

Confidence score

A detector's per-document "confidence" is the model's own certainty about one text. A confidence interval describes uncertainty about a rate measured across many. Unrelated quantities, similar words.

False positive rate

The rate is the point estimate; the interval is how much that estimate could be wrong. A rate published without one is an assertion about a sample presented as a fact about a tool.

Read next

• The accuracy research, reviewed
• False positive rate
• Pangram, and what its figures rest on