Tokens, Context Windows, and Why AI Chatbots Forget

In 2023, a team of researchers at Stanford, UC Berkeley, and a startup called Samaya AI ran a deceptively simple experiment on the language models of the day. They took questions with known, verifiable answers, buried the passage containing each answer inside a stack of ten, twenty, or thirty other documents, and asked the model to find it. The only thing they varied was where the right document sat in the stack: first, last, or somewhere in the middle.

The results, published as “Lost in the Middle: How Language Models Use Long Contexts,” traced a shape that has since become one of the most cited findings in AI research: a U. Models were consistently best at using information placed at the very start or the very end of what they’d been given, and consistently worse at using the exact same information when it sat in the middle of a long input. Not because the information was hidden. Not because the model ran out of room. It could see every word, every time. It just used the middle of it worse.

That single finding explains more about why AI chatbots feel forgetful than almost anything else you’ll read on the subject. It isn’t really about “memory,” at least not in the way that word works for a person. It’s about tokens, the small pieces of text a model actually reads. It’s about a hard architectural ceiling called a context window. And it’s about the fact that even inside that ceiling, not all information gets treated equally. Once you understand those three things, the moments where an AI assistant loses the thread mid-conversation stop feeling like glitches and start feeling like exactly what you’d predict.

Quick answer: A token is the small chunk of text, often smaller than a word, that an AI model actually reads and writes. A context window is the maximum number of tokens a model can hold in view at once, and it covers everything: your instructions, the whole conversation so far, any documents you’ve pasted in, and its own reply. When a conversation grows past that limit, or when the information you need is buried deep inside a long one, the model doesn’t handle it as reliably, which is what shows up to you as the AI “forgetting” something you told it.

Here’s what you’ll walk away knowing:

• Models don’t read words or letters. They read tokens, and that difference explains a surprising number of everyday AI quirks.
• A context window is a hard ceiling on how much text a model can hold in view at once, not a separate memory store somewhere else.
• Context windows have grown roughly 500-fold since 2020, but “how big” and “how well-used” are two different questions.
• There’s a real, physical reason for the ceiling: the math behind attention gets more expensive, quadratically, the longer the input gets.
• Even well inside the limit, models use information at the start and end of a long context more reliably than information buried in the middle, a pattern researchers call “lost in the middle.”

What a token actually is

Start with the part everyone gets wrong first: an AI model doesn’t see your message the way you typed it. Before anything else happens, your text gets chopped into tokens, chunks of text that are often smaller than a word and sometimes exactly one word, depending on how common that word is. “The” is almost always its own token. “Antidisestablishmentarianism” is not; it gets carved into several smaller pieces that, put back together, spell the whole word.

OpenAI’s own guidance on the subject offers a serviceable rule of thumb for English: one token is roughly four characters, or about three-quarters of a word. So a 100-word paragraph of ordinary English prose usually runs somewhere around 130 tokens, not 100. That gap between “words” and “tokens” is small enough to ignore in casual conversation and large enough to matter the moment you’re trying to reason about limits, costs, or why an AI is behaving strangely.

We’ve written before about how large language models actually work, including the mechanics of how tokens turn into predictions. This article picks up a thread that piece deliberately left for later: what happens once you understand that a model only ever “sees” a limited number of these tokens at a time, and what that limit does to the conversation as it grows.

Why chunks, not words

The reason models use tokens instead of whole words comes down to an algorithm called byte pair encoding (BPE), and its close cousins like SentencePiece. The basic idea: start with individual characters, then repeatedly merge whichever pair of characters or character-chunks shows up most often across a huge body of training text, building up a vocabulary of common pieces over many rounds. Do this enough times and you end up with a vocabulary where extremely common words become a single token, while rarer words, unusual names, typos, and made-up terms get split into two or more subword pieces.

This buys the model two things at once. A fixed-size vocabulary can still represent essentially any string of text, even words it never saw during training, by falling back to smaller and smaller pieces. And common patterns get compressed into single units, which makes the whole system more efficient than reading character by character. It’s the same core insight as a zip file: represent frequent patterns compactly, and spell out the rare stuff.

The catch is that “frequent” is measured against the training data, and the training data for most major tokenizers is disproportionately English. That has a real, measurable consequence, and it’s worth seeing with actual numbers rather than taking on faith.

The non-English tax: why some languages cost more tokens than others

To make this concrete, we took a single sentence, Article 1 of the Universal Declaration of Human Rights, in its official English, Chinese, Spanish, Russian, and Arabic translations, and ran each one through OpenAI’s o200k_base tokenizer, the encoding behind current GPT-4o-generation models. Same meaning, same legal text, five different scripts.

English came in cheapest at 33 tokens. Arabic needed 44 tokens for the identical sentence, a third more, despite being 47 characters shorter. Chinese is the interesting outlier: at just 43 characters for the whole sentence, it needed only 35 tokens, because a single Chinese character already carries roughly a syllable’s worth of meaning, so the tokenizer doesn’t need to split it into much. Alphabetic non-Latin scripts like Russian and Arabic don’t get that same density benefit, and their words also weren’t merged into single tokens as aggressively during tokenizer training, so they end up costing more.

This isn’t a fluke of one sentence. It’s a documented, structural pattern: languages and scripts that were rare in a tokenizer’s training data never earned the efficient multi-character merges that common English words did, so their text tends to fall back to smaller, less efficient pieces, sometimes close to one token per character for scripts far outside the training distribution. Researchers evaluating tokenization across languages have found the same directional effect using different methods and different text, even if the exact multiplier varies by tokenizer and sentence.

The practical upshot is not trivial. If you’re chatting in a language other than English, or working with documents in one, you’re quietly using up more of your context window and, on paid APIs, more of your budget, for the same amount of meaning. It’s not a policy choice by any AI company, more a byproduct of what the internet’s text looks like and how these systems were built, but it’s a real cost that falls unevenly depending on what language you happen to work in.

What a context window actually is

Now the second concept. A context window is the maximum number of tokens a model can hold in view for a single request. Anthropic’s own documentation puts it plainly: the context window is a model’s “working memory,” distinct from the enormous corpus of text the model was trained on. Training data shapes what the model generally knows and how it writes. The context window is what it can actually see, right now, for this specific reply.

Crucially, that number is a shared budget, not a separate allowance for input and output. It covers the system prompt, every prior message in the conversation, any files or documents you’ve pasted or uploaded, tool definitions if the assistant is using tools, and the model’s own response as it’s being generated. Everything in the request counts toward it, including things you never see, like a hidden system prompt or extended internal reasoning some models produce before answering. A long back-and-forth conversation, a large pasted document, and a lengthy answer are all drawing from the exact same pool.

It’s also worth being precise about what a context window is not. It is not a database. It is not the model recalling something it learned during training, the way it might recall a well-known historical date. And critically, it resets. A brand-new chat starts with an empty context window every time; nothing from your previous conversation carries over unless a separate feature, like a memory system, deliberately re-inserts it, which we’ll get to later in this piece.

How we got from 2,000 tokens to over a million

Context windows haven’t always been generous. The trajectory over the past six years is one of the more startling growth curves in the entire field, and it’s worth walking through concretely, because “the limit went up” undersells just how constrained early systems were.

GPT-3, released in 2020, worked with a context window of 2,048 tokens, roughly three or four pages of text, total, for everything: your prompt and the model’s entire reply, combined. ChatGPT’s launch model, GPT-3.5, doubled that to around 4,096 tokens in late 2022. GPT-4, in March 2023, launched with an 8,000-token standard version, alongside a larger 32,000-token variant for select users. Then, in November 2023, GPT-4 Turbo jumped to 128,000 tokens, about 300 pages, in a single release. The climb from a few thousand tokens to well over a hundred thousand took less than a year.

As of this writing, in mid-2026, the frontier has moved again. Anthropic’s Claude Opus 4.8, Opus 4.7, Opus 4.6, and Sonnet 5 all ship with a 1-million-token context window as the default, at standard API pricing, no special access required. Other current Claude models, including Sonnet 4.5 and Haiku 4.5, remain at 200,000 tokens, which is a useful reminder that “the newest model” and “the biggest context window” aren’t automatically the same thing, even within one company’s own lineup. Industry trackers report OpenAI’s current flagship models sitting around 1 million tokens as well, with a pricing step-up for requests above roughly 272,000 tokens, and Google’s largest current Gemini model reportedly reaching up to 2 million tokens, with its faster, cheaper sibling models capped closer to 1 million.

Model family	Roughly current (mid-2026)	Context window
Anthropic Claude	Opus 4.8, Sonnet 5	1M tokens
OpenAI GPT	Current flagship	~1M tokens (reported)
Google Gemini	Largest current model	Up to 2M tokens (reported)

A caveat worth stating outright: context window sizes change with nearly every model release across all three companies, sometimes within the same month. The Claude figures above come directly from Anthropic’s own current API documentation; the OpenAI and Google figures are based on third-party reporting rather than a document we could verify firsthand, since both companies’ specs shift often enough that a printed number can be stale within weeks. Treat this table as a snapshot of the general order of magnitude, not a permanent spec, and check the provider’s own docs before making a decision that depends on the exact figure.

It helps to make a number like “a million tokens” concrete. Going by OpenAI’s own rough conversion of about three-quarters of a word per token, a million tokens is somewhere in the neighborhood of 750,000 words. Anthropic’s own materials describe its smaller, 200,000-token project workspaces as holding “the equivalent of a 500-page book,” so scale that up roughly fivefold and a million-token window is in the range of five books that size, stacked together, or one very long one. That’s an enormous amount of room compared to 2020’s 2,048-token ceiling, which struggled to hold a single long magazine article without spilling over. It’s also, as the next section explains, not the same thing as a guarantee that a model will use every one of those tokens equally well.

Why is there a limit at all?

Here’s the question that trips people up: if context windows keep growing, why is there a ceiling at all? Why not just make it infinite?

The honest answer is computational cost, and it comes from how the attention mechanism at the heart of every modern language model actually works. Attention is what lets a model weigh every other token in its context against the one it’s currently predicting, no matter how far apart they are in the text. That’s what makes long-range understanding possible in the first place. But it has a price: for every new token added to the input, the model has to compute its relationship to every token that came before it. Add a token, and the number of these pairwise comparisons doesn’t grow by one. It grows by however many tokens are already there.

Picture a large meeting where, for good decision-making, every new comment needs to be weighed against every comment made so far in the room. With ten comments already on the table, a new remark requires ten comparisons. With ten thousand comments on the table, one more remark requires ten thousand comparisons. The room isn’t getting linearly harder to follow: it’s getting quadratically harder, because the cost is proportional to how many prior comments there are, and that number keeps climbing as the meeting goes on. That’s roughly the shape of the problem: doubling the length of an input doesn’t double the computational cost of processing it, it roughly quadruples it.

In practice, engineers get around recomputing everything from scratch at every single step using a trick called a KV cache, short for key-value cache: a running set of notes the model keeps about everything it’s already processed, so it doesn’t have to re-derive those relationships each time it wants to generate the next token. Without this cache, generating the thousandth token of a response would require redoing the relationship math for all 999 tokens before it, every single time, which would make anything beyond a short reply impractically slow. The cache solves the speed problem, but not the space problem: the notes it keeps take up memory that scales directly with how many tokens are in play, so a longer context still means more computer memory reserved for a single conversation, on top of the extra processing time.

That’s the real, physical reason a bigger context window isn’t just a switch a company can flip for free. It costs more to compute, it costs more memory to hold in view, and both of those costs show up somewhere: in how much a provider charges for long requests, in how fast a model responds, or in how much specialized hardware a company needs to serve you at all. The 500-fold growth in context windows since 2020 didn’t happen because the ceiling turned out to be arbitrary. It happened because engineers found real, substantial ways to make that quadratic cost more manageable, through more efficient attention variants, smarter caching, and simply throwing more hardware at the problem, and those innovations keep shipping.

How far can this actually be pushed?

Context windows don’t grow only through more hardware. Architecture matters too, and one useful illustration of how far it can be stretched comes from MiniMax, an AI lab whose MiniMax-Text-01 model, released in January 2025, was trained with a context window of up to 1 million tokens and demonstrated it could extrapolate to handle 4 million tokens at inference time, at what its creators described as an affordable computational cost. That’s not a claim about beating any particular frontier model on quality, and by mid-2026 standards it’s a model from an earlier generation, not a leaderboard entry. What it demonstrates instead is that the quadratic-cost ceiling described above is a real engineering constraint to design around, not a fixed law of physics. With the right architectural choices, ways of approximating full attention rather than computing every single pairwise comparison exactly, a model can be pushed to hold far more text in view than a naive implementation would allow, without a proportional explosion in compute. That’s a meaningful part of why context window numbers have kept climbing across the industry rather than plateauing.

Why AI seems to “forget”: lost in the middle and context rot

This is where the two ideas, tokens and context windows, meet the experience you actually have when a chatbot loses the thread. And it turns out there isn’t just one mechanism at work, there are two, and they’re easy to mix up.

The first is running out of room outright, which we’ll cover in the next section. The second, and the more surprising one, is the “lost in the middle” effect from the opening of this article: models are demonstrably less reliable at using information that’s sitting in the middle of a long context, even when that information is fully present and the model is nowhere near its limit. This U-shaped pattern, strong recall at the start and end of a context, weaker recall in the middle, was first documented in 2023 and has been repeatedly reconfirmed on newer benchmarks since, including on today’s million-token-context models. Researchers still debate exactly why it happens, but that it happens is not seriously in question.

Anthropic has given a related, broader phenomenon an official name in its own documentation: context rot. As the company puts it plainly, “as token count grows, accuracy and recall degrade,” and that’s a distinct issue from simply running out of space. A model can have hundreds of thousands of tokens of room left and still get measurably worse at using what’s already in front of it, purely because there’s more of it to sift through. Anthropic’s framing is worth sitting with: what you put in context matters just as much as how much room is available.

This isn’t a 2023 artifact that better engineering has since made irrelevant, either. Independent long-context evaluations built specifically to stress-test newer, much larger context windows, benchmarks with names like RULER, LongBench v2, and HELMET, have continued to find versions of the same pattern on today’s frontier models, including ones with million-token windows. The exact numbers and root causes are still actively debated in the research community, and it would be overstating the evidence to pin the effect on any single mechanism. But the general shape, information gets used less reliably the deeper it’s buried in a long context, has held up across model generations rather than disappearing as windows got bigger.

Put those two findings together and you get the honest picture that vendors don’t always lead with in their marketing: a model’s nominal context window, the number in the spec sheet, and its effective context window, how much of that it actually uses well, are two different numbers. A million-token window is real capacity. It is not a guarantee that everything inside it gets equal attention. This is exactly why the article you’re reading right now, and the one on our site about RAG versus fine-tuning versus prompting, both land on the same practical conclusion from different directions: bigger context windows have genuinely reduced how often you need workarounds like retrieval, but they haven’t eliminated the value of keeping what you hand a model focused and relevant, rather than exhaustive.

What happens when you actually run out of room

Separate from context rot, there’s the more literal version of forgetting: the conversation actually exceeds the context window, and something has to give.

Different products handle this differently, and it’s worth knowing that the underlying mechanics aren’t always visible to you as the user. Anthropic’s own documentation describes one concrete, real approach: chat interfaces such as claude.ai can manage the context window “on a rolling first in, first out basis,” meaning that once a long conversation approaches the limit, the oldest messages are the first to be dropped to make room for new ones. That’s the practical, mechanical answer to a question a lot of people ask without quite framing it this way: if you’ve been chatting with an AI for an hour and it suddenly seems to have no idea what you told it at the very beginning, there’s a real chance that message is, quite literally, no longer part of what the model can see.

The newer, gentler alternative to blunt truncation is summarization. Rather than silently dropping old messages, some systems now compress them: Anthropic calls this compaction, a feature that automatically summarizes earlier parts of a long conversation on the server so it can continue past the context window limit without simply chopping off the beginning. The tradeoff is inherent to the idea: a summary preserves the gist of what was said while discarding the specific wording, so a detail you mentioned in passing thirty messages ago may not survive being condensed, even though the broad topic does.

For anyone building on top of these models directly through an API, the overflow behavior is stricter and more visible: if a request’s input alone already exceeds the model’s context window, the system returns an outright error rather than guessing what to drop. That’s a deliberate design choice, favoring a clear failure over a silently degraded one, and it’s the reason developers are told to estimate token usage before sending a large request rather than finding out the hard way. As a regular chat user you won’t see that error message directly, but the underlying constraint is the same one shaping what you experience as an assistant that quietly stops remembering things.

Memory features aren’t the same thing as a context window

This is the single most common point of confusion once people start understanding context windows, so it’s worth being direct about it: a chatbot’s “memory” feature and its context window are two different systems doing two different jobs.

ChatGPT’s memory feature, when turned on, lets the assistant “automatically remember useful context from your chats, files, and connected apps to personalize your experience,” so you don’t have to repeat basic facts about yourself every time you start a new conversation. Claude’s rough equivalent is Projects, dedicated workspaces where you can attach reference documents and set standing instructions that every chat inside that project can draw on. Anthropic’s own help documentation is specific about the scale involved: each project still runs on a 200,000-token context window, “the equivalent of a 500-page book.” That detail matters, because it shows exactly how these features actually work under the hood: they don’t grant the model some separate, unlimited memory bank. They store useful information somewhere outside the live conversation, and then, when relevant, insert a relevant slice of it back into the same finite context window every other request has to share.

That’s a genuinely useful design, and it solves a real problem: without it, you’d need to re-explain your job, your preferences, and your ongoing projects at the start of every single conversation. But it’s an addition on top of the context window limit, not an exception to it. If you’ve told an AI assistant a hundred different facts about yourself over months of use, a memory feature is making a judgment call about which handful of those facts are worth re-inserting into any given conversation, because it, too, only has finite room to work with.

What this means for how you actually use AI

None of this is trivia. It should change a few habits if you regularly work with ChatGPT, Claude, or Gemini on anything longer or more involved than a quick question.

Restate what actually matters, rather than assuming it’s still in view. If a conversation has run long and something you said early on is genuinely important, especially a specific number, name, or constraint, repeat it close to where you need the model to use it. Don’t rely on it being recalled perfectly from fifty messages back, particularly if it wasn’t near the very start or the very end of the conversation.

Start a fresh conversation for a genuinely new task. Every irrelevant message sitting in a long conversation’s history is still consuming context window space and, per the “lost in the middle” pattern, potentially diluting the model’s attention on what you’re asking right now. A clean start isn’t just tidier, it’s often functionally better for accuracy on the new task.

Don’t paste a giant document in just because the window is technically big enough. A model with a million-token context window can accept a huge file, but “can accept it” and “will reason about all of it equally well” are different claims, exactly per the context rot research above. If you only need answers about a few sections, tools built around retrieval-augmented generation, or RAG, that search a document and pull in only the relevant passages tend to outperform dumping everything in at once, and they cost less to run besides.

Watch for usage or context indicators in the product you’re using. Many chat interfaces now show some signal, even a rough one, of how much of the context window a long conversation has used. Treat it the way you’d treat a fuel gauge: useful information for deciding whether to keep going or start fresh.

For code, lean on tools built for the problem. Coding assistants like Cursor and GitHub Copilot don’t typically try to stuff an entire codebase into context. They index the project and selectively retrieve the files and functions relevant to what you’re working on, which is the same underlying idea as RAG, applied to source code instead of documents.

Put your most important instructions at the start or the end, not buried in the middle. Given the “lost in the middle” pattern, a critical constraint, like a formatting rule or a hard requirement, is more likely to be followed if it opens or closes your prompt than if it’s sandwiched inside three paragraphs of background material. This is a small, free habit that directly works around a documented weak spot rather than fighting it.

Remember that non-English text costs more of your budget. If you’re working in a language other than English, especially one with a non-Latin script, budget extra room mentally: the same information can take meaningfully more tokens to express, which means it also fills up the context window faster and, on metered tools, costs more per message.

Common misconceptions

“A bigger context window means a smarter model.” They’re different properties. Context window size is capacity, how much a model can hold in view. Reasoning quality, knowledge, and reliability are separate dimensions, and a model can have an enormous context window while still reasoning worse than a smaller-window competitor on a given task.

“The AI remembers everything I’ve ever told it, by default.” Only if you’re using a product with a memory feature explicitly turned on, and even then, only a distilled subset of what you’ve said, not a verbatim transcript of every past conversation. A plain chat session, without memory enabled, starts from zero every time.

“Once something scrolls off the visible chat, it’s gone forever.” Not necessarily, and this is worth separating from context window mechanics specifically. Most chat products still store your full conversation history on their servers, and you can typically scroll back and read it yourself. What may no longer be true is that the model is actively considering that old material when generating its next reply. The message can still exist in the app while being functionally invisible to the AI in that moment.

“Since long context windows exist now, techniques like RAG are obsolete.” The Lost in the Middle research and Anthropic’s context rot findings are exactly why this isn’t true yet. A bigger window reduces how often you strictly need retrieval, but it doesn’t erase the accuracy cost of handing a model more than it needs. For focused, cost-sensitive, or accuracy-critical use, keeping context small and relevant remains a real advantage, not a workaround for a problem that’s already been solved.

“Tokens are basically just words with extra steps, so the difference doesn’t matter.” It matters in three concrete ways covered in this article: it’s why letter-counting and spelling tasks trip models up, it’s why non-English text can quietly cost more of your budget for the same meaning, and it’s the actual unit that a context window, and most AI pricing, is measured in. Treating “tokens” and “words” as interchangeable will make every one of those numbers look wrong.

The bottom line

Go back to that 2023 experiment one more time. The researchers weren’t testing whether models could technically fit information in their context, they could, every time. They were testing whether the models actually used it well, and the answer, for information stuck in the middle of a long input, was: not as well as you’d hope. That gap between “can technically see it” and “actually uses it reliably” is the whole story of why AI chatbots feel like they forget.

Tokens are the currency these systems spend just to read what you write. The context window is the finite budget of that currency they’re given per conversation. And even within budget, a model’s attention isn’t evenly distributed across everything you’ve handed it. None of that makes these tools less useful, context windows a thousand times larger than 2020’s are a genuinely remarkable engineering achievement, and most everyday conversations never get anywhere near the limit. But the next time an assistant loses track of something you said twenty minutes ago, you’ll know it’s not being careless. It’s a token-counting machine, doing exactly what a token-counting machine does when you ask it to hold too much in view at once.

Frequently asked questions

How many words is one token?

For English text, a useful rule of thumb from OpenAI is that one token is roughly four characters, or about three-quarters of a word, so 100 tokens is around 75 words. That ratio isn't fixed: common short words often become a single token, while rare words, typos, and numbers get split into more pieces, and non-English text almost always uses more tokens per word than English does.

Why does a chatbot forget something I told it earlier in a long conversation?

Two different things cause this. If the conversation has grown past the model's context window, the oldest messages may have been trimmed or summarized to make room, so they're no longer in view at all. But even well inside the limit, research shows models use information sitting in the middle of a long context less reliably than information at the start or end, so it can seem to forget something that's technically still there.

Does a bigger context window make an AI model smarter?

Not by itself. Context window size measures how much text a model can hold in view, not how well it reasons or how much it knows. A model with a million-token window can still misuse or overlook information inside that window, a pattern well documented in research on long-context performance. Bigger context is more raw capacity, not a separate axis of intelligence.

Why does writing in a language other than English use up my limit faster?

Tokenizers are built by finding the most common chunks of text in their training data, and that training data skews heavily English. Words in less-represented languages and scripts often get broken into more, smaller pieces to be represented, so the same sentence can cost meaningfully more tokens in, say, Arabic or Russian than it does in English, even though it says exactly the same thing.

Is a chatbot's 'memory' feature the same as its context window?

No. A context window is a hard technical limit on a single conversation. A memory feature, like ChatGPT's saved memories or Claude's Projects, is a separate system that stores facts or reference files outside that one conversation and quietly re-inserts relevant bits into a future conversation's context window. It works around the limit by feeding it useful information, but it doesn't remove the limit itself.

What can I do if what I need an AI to read is longer than its context window?

Break the material into focused chunks and ask about one part at a time, or use a tool built for exactly this, like a retrieval system (RAG) that searches a large set of documents and only pulls the relevant passages into context instead of pasting everything at once. For code specifically, most AI coding assistants already do this kind of selective retrieval across a codebase automatically.

• #Tokens
• #Context Window
• #Tokenization
• #Large Language Models
• #AI Explained
• #AI for Beginners