humanflow

Token

A token is the unit of text a language model actually reads — usually a word fragment rather than a whole word, which is why a model's limits are measured in tokens and not in words.

Last reviewed 15 August 2026 · The HumanFlow team

In plain English

Models do not see letters or words. They see a sequence of numbered chunks, where common words are one chunk and rarer words get split into several.

The practical consequence is that a document's token count is not its word count, and unusual vocabulary costs more tokens than ordinary vocabulary does.

A worked example

One sentence, split the way a typical tokeniser splits it. The | marks a boundary.

  words   (6)   The researcher's hypothesis was unfalsifiable.

  tokens (11)   The| research|er|'s| hypothesis| was| unf|als|ifi|able|.

  common word   "hypothesis"      → 1 token
  rare word     "unfalsifiable"   → 4 tokens

Six words became eleven tokens, and one rare word accounts for four of them. "Hypothesis" is common enough in the training data to be a single unit; "unfalsifiable" is not, so it gets rebuilt from fragments.

That asymmetry is the useful part. Rare, precise vocabulary is expensive in tokens and unlikely in prediction — which is the same property that makes it read as human to a detector.

It also explains why word limits and model limits never quite line up. A 1,000-word document might be 1,300 tokens of plain prose or 1,700 of technical writing.

Why it matters for AI detection

Every measurement a detector makes happens at the token level. When a tool reports how surprising your text was, it is reporting how surprising each token was given the tokens before it.

It explains a specific unfairness. Writers using specialised or non-English-derived vocabulary produce more tokens per word, and each fragment is an additional prediction the model can find unsurprising — one of several mechanisms by which plain, formulaic prose and unusual prose land at opposite ends of a score.

For anyone hitting a length limit: limits quoted in words are approximations of a limit that is really in tokens, which is why the same word count sometimes fits and sometimes does not.

Commonly confused with

Word
A word is a unit of language; a token is a unit of processing. Common words are one token, rare words are several, and punctuation is usually its own.
Perplexity
Tokens are what gets measured. Perplexity is the measurement — how predictable each token was given the ones before it.

Read next

Part of the AI detection glossary.