Fenêtres contextuelles et jetons dans l'IA : une explication simple

Context windows et tokens are two foundational concepts in AI, especially in Large Language Models (LLMs). Understanding these concepts will make more sense of why AI models seem to “forget” parts of the conversation once developed enough as well as why complex prompts can be misinterpreted by the model.

What are Tokens ?

In AI, a token is a unit of data that may represent a character, word or phrase. You can think of it as a building blocks that models use to create their answers. Usually, in the English language, each token is equivalent to 4 characters, although this isn’t a rule. Other languages have different tokenization equivalence and it may also depend on the AI systems themselves.

Example of tokens:

• Complete words (“house,” “cat,” “wonderful”)
• Parts of words (“un-” “expect-” “-ed”)
• Individual characters (especially for punctuation like “.” or “?”)
• Spaces between words

For instance, the sentence “I love learning about AI!” might be broken down into tokens like:

“I”
“ love”
“ learning”
“ about”
” AI”
“!”

Why Tokens Matter ?

Tokens help AI systems to process language more efficiently. They aren’t trying to understand every single word you might enter, but instead work with a finite vocabulary of tokens. It makes their task much more manageable.

What is a Context Window ?

Context Windows are the limit to which certain models will keep in memory the information you provide in a conversation, to then answer.

The context window represents how much text (measured in tokens) an AI model can “see” and consider at once when generating a response. It includes:

• Your current question or request
• Previous messages in the conversation
• Any documents or information you’ve shared
• The AI’s own previous responses

Why do Context Windows Matter ?

Context Windows matter in the sense that they impact the output / answer quality. Let’s say an AI has a context window of 8000 tokens (roughly 6k words), it can only consider that amount of context to generate the answer. Everything outside this context window will be effectively invisible to the AI.

Some Practical Benefits of Understanding These Concepts:

• If you analyze a long document, you might need to summarize it within the context window, split it in smaller sections to then analyze those and focus on the important parts
• For creative writing, let’s say working on a story, you might want to keep track of how much of the context window is being used, refresh the conversation from time to time and summarize previous chapters to ensure continuity while saving tokens.
• For coding projects, you want to ensure you provide only the necessary context to achieve your goal. As someone who’s been programming with AI, it’s easy to get lazy and provide too much or too little information about a problem you’re trying to solve. That however, doesn’t output the best results and might even hinder you more than anything.

Strategies for Working with Context Windows

1. Be concise: Communicate clearly but efficiently to save tokens.
2. Prioritize information: Include the most important details first.
3. Summarize when possible: Instead of pasting an entire document, consider sharing a summary with key points.
4. Clean up: In long conversations, consider starting fresh when the context becomes cluttered with irrelevant information.
5. Structure your questions: Well-organized, specific questions help the AI understand your needs without requiring additional back-and-forth.

Conclusion

Tokens and Context Windows are important concepts to learn when trying to work with AI. Understanding those will help you communicate with AI model better, working up to their fullest potential and within their constraints. As AI technology advances, context windows are growing larger and token processing become more efficient, but these concepts remains at the heart of how we interact with language AI.