Lexical diversity

In plain English

Count the different words, divide by the total number of words, and you have the type-token ratio. That is the measure almost every tool reports.

It has a flaw that matters more than it sounds: the number falls as text gets longer, in every text, no matter who wrote it.

A worked example

The same passage measured at increasing lengths. Nothing about the writing changes; only how much of it is being measured.

first  50 words   →  TTR 0.70
first 100 words   →  TTR 0.61
first 200 words   →  TTR 0.52
first 400 words   →  TTR 0.44

MATTR (50-word window), whole text  →  0.71

The decline is arithmetic, not stylistic. Function words like "the" and "of" recur indefinitely, while genuinely new words arrive more and more slowly, so the ratio of distinct to total must fall with length.

The consequence is that a short abstract will nearly always out-score a long chapter by the same author, and comparing them says nothing.

MATTR avoids this by measuring a fixed 50-word window and sliding it across the text, averaging the results. Because every measurement covers the same number of words, document length drops out and the figure stays put.

Why it matters for AI detection

Because lexical diversity gets quoted as evidence in arguments about authorship, and the raw type-token ratio is not evidence of anything unless both texts being compared are the same length.

It is one of the features stylometric analysis actually uses, which is why it appears in discussions of AI detection at all — though no mainstream detector reports it to you directly.

There is no good score. Technical and academic writing repeats its key terms deliberately, and a thesaurus pass that raises the number makes the writing worse. Use it to compare drafts of the same piece, not to hit a target.

Commonly confused with

Perplexity

Lexical diversity counts how many different words you used. Perplexity measures how predictable each word was given the ones before it. A text can repeat a small vocabulary in a completely unpredictable order.

Stylometry

Stylometry is the broader practice of identifying authorship from measurable style features. Lexical diversity is one of the features it uses, not a method in itself.

Read next

• Lexical diversity calculator
• Stylometry
• Word frequency counter