What is a token in AI? A plain‑English explanation for 2026

A token in AI is a small chunk of text, usually a word, part of a word or a punctuation mark, that a language model reads and writes one piece at a time. Models never see your letters directly: they see a list of token IDs, and every limit and price you run into is counted in tokens. In English, OpenAI puts the rough rule at about four characters, or three-quarters of a word, per token.

That's the whole idea. The rest of this piece covers how the splitting works, what it costs you in real numbers (checked against vendor docs on 2 November 2026), and where the rules of thumb fall apart.

Short answer: what is a token in AI

A token is the unit of text a large language model works in. The model turns your prompt into tokens, predicts the next token, appends it, and repeats until it's done. Your input is billed per token, the reply is billed per token, and the context window (how much the model can hold at once) is measured in tokens too.

If you only remember one number, make it this: 100 tokens is roughly 75 English words on OpenAI's models. It's different on other models, and that difference matters more than most people expect.

From text to numbers: the mechanics

Before training, a model's makers build a tokenizer, which is a fixed vocabulary of text pieces plus rules for chopping any input into those pieces. Most modern tokenizers descend from byte pair encoding, a compression trick that Rico Sennrich and colleagues adapted for language models in a 2015 paper on rare words. The method starts with single characters, then keeps merging the pairs that show up most often in the training text until the vocabulary hits a target size. Common words end up as one token; rare words get spelled out from smaller pieces.

So "the" is almost certainly one token. A word like "unbelievable" might come out as three pieces, something like "un", "believ" and "able", though the exact split depends on the tokenizer. Small details count as well. OpenAI's token guide points out that "red", "Red" and " red" (with a leading space) can all map to different tokens.

Each piece then becomes an integer ID. The model looks up a learned vector for each ID, runs those vectors through its transformer layers, and outputs a probability for every token in its vocabulary. It picks one. Then it does the whole thing again for the next token.

That loop explains a lot of odd behaviour. Since the model sees "strawberry" as a couple of chunks rather than ten letters, counting the r's is harder for it than you'd think.

Tokens aren't only text, by the way. Google's Gemini docs say all input and output is tokenized, images and other non-text inputs included, and that's why a single photo can eat part of your budget.

Real examples with real numbers

Every vendor gives a slightly different rule of thumb, and they don't agree. Here's what each one says for English text:

Anthropic's number stands out. Its glossary explains that Claude 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same text than earlier Claude models did. Same words, more tokens.

Worked example: summarising 1,000 reports

Say you run a small tool that takes a 1,500-word report and returns a 300-word summary, and it handles 1,000 reports a month. Two mid-range models carry the same sticker price, as of 2 November 2026: OpenAI's GPT-6.1 Sol and Anthropic's Claude Sonnet 5.5 both cost $2 per million input tokens and $10 per million output tokens on standard pricing (OpenAI rates, Claude rates).

Same price per token; roughly 35 percent more spent on the Claude side, purely because its tokenizer cuts English into smaller pieces. These are estimates built on each vendor's rule of thumb, not exact counts, and I'd guess the real gap moves around a fair bit with the kind of text you feed in. OpenAI makes the same point itself: a lower price per million tokens doesn't necessarily mean a lower total cost.

Now add reasoning. On OpenAI's API, a reasoning model's hidden thinking tokens count and are billed as output tokens. If GPT-6.1 Sol spends 3,000 thinking tokens per report before writing its 400-token summary, output jumps to 3,400 tokens and the bill for 1,000 reports goes from about $8 to about $38. Honestly, that line is where I'd look first on any surprising invoice.

Benefits and limitations of tokens

I think tokens are a sensible compromise. A vocabulary of whole words couldn't handle new names, typos or code; a model that reads single characters would need far longer sequences for the same text. Subword tokens sit in the middle, which is why the Sennrich approach stuck.

Failure modes

• Other languages cost more. A 2023 study by Aleksandar Petrov and colleagues found the same text translated into different languages can need up to 15 times as many tokens, which means higher prices and less room in the context window for those speakers.
• Letter-level tasks go wrong. Spelling, counting characters and rhyming are awkward when the model never sees individual letters.
• Rules of thumb drift. Anthropic's tokenizer change shows the ratio can shift between model versions from the same company.
• Long prompts can get pricier per token. OpenAI charges 2x input and 1.5x output on GPT-6.1 Sol once a prompt passes 272K input tokens, and Claude Haiku 5.5 goes from $0.10 to $0.50 per million input tokens above 100,000 tokens.

The language gap bothers me most. It's a cost that falls on users who never chose the tokenizer.

Practical takeaways for you

If you use ChatGPT, Claude or Gemini in a browser on a flat subscription, tokens mostly show up as limits: a conversation that gets too long, or a file that's too big. Claude's current models have a 1M-token context window and 128K tokens of max output; GPT-6.1 Sol lists a 1,050,000-token context window, with up to 922,000 input tokens. By Anthropic's ratio, a 90,000-word novel comes to roughly 160,000 tokens, so it fits easily.

If you pay per token through an API, a short checklist helps:

• Count before you send. OpenAI has a tokenizer tool and the tiktoken library, Google has a count_tokens call, and Anthropic has a token counting endpoint.
• Compare the total cost of a task, not the price per million tokens.
• Cap output length with max_output_tokens or the equivalent setting, and leave room for reasoning tokens.
• Cache repeated context. Claude cache hits on Sonnet 5.5 cost $0.10 per million tokens against $2 for fresh input.
• Trim the prompt. Repeated instructions and pasted boilerplate are paid tokens on every call.

Tighter prompts also tend to get better answers, which is a nice side effect; our guide to writing better prompts for ChatGPT, Claude and Gemini has more on that. For background on the models themselves, see our explainer on what an AI model is, and our Claude review covers the consumer plans.

Next for tokenizers

Some researchers want to drop fixed tokenizers altogether. The Byte Latent Transformer paper by Artidoro Pagnoni and colleagues, from December 2024, reads raw bytes and groups them into patches of varying size, and the authors report it matched tokenizer-based models at scale (up to 8B parameters) with better inference efficiency.

I don't expect the big commercial models to switch soon, though. Vendors are still tuning conventional tokenizers, as Anthropic's tokenizer change shows, and every price sheet I read for this piece is still written per million tokens. My guess is you'll be counting tokens for a good while yet, so it's worth knowing roughly how many words your favourite model squeezes into each one.

FAQ

What is an AI token?

An AI token is the basic unit of text a language model processes: a whole word, a piece of a word, a character or a punctuation mark. Your prompt is split into tokens, the model generates its reply one token at a time, and API pricing and context limits are both counted in tokens.

What are tokens for in AI?

Tokens turn text into numbers a model can do maths on, since each one maps to an ID and a learned vector. They're also the accounting unit: OpenAI, Google and Anthropic bill per million tokens.

How many words is a token?

It depends on the model. OpenAI says one token is about three-quarters of an English word, so 100 tokens is about 75 words. Google says 100 tokens is about 60 to 80 words for Gemini. Anthropic's current Claude tokenizer fits roughly 555,000 words into one million tokens, about 55 words per 100 tokens.

Is an AI token the same as a crypto token?

No. Tokens in language models are pieces of text, and they don't exist outside the model or have any market value. Crypto projects sometimes call their coins "AI tokens", but that's a separate thing entirely. If you're comparing chatbots or APIs, the tokens that matter are the ones listed next to each model's input and output price, such as $2 per million input tokens for GPT-6.1 Sol. Those are a billing unit, not something you can buy, hold or trade.

Do tokens matter if I only use a chatbot app?

A little. On a flat monthly plan you don't pay per token, but tokens still set how long a conversation or uploaded file can be before the app starts forgetting earlier parts or refuses the upload. For most people, that's the only place tokens seem to show up at all.