How models work

What is Token?

The small chunks of text an AI model reads and writes — roughly a word or part of a word. Limits and pricing are counted in tokens.

Models don't see words the way we do. They split text into tokens: common words are often one token, longer or rarer words become several. Hindi and other Indian languages often use more tokens per word than English.

Tokens matter because a model's memory (its context window), its speed and API pricing are all measured in them. A long chat or a big document uses up tokens quickly.

Rule of thumb

In English, 100 tokens is roughly 75 words. A 1,500-word article is about 2,000 tokens.

Related terms

In the news