Jobescape
AI glossary

Token

A token is a small chunk of text - often a word or part of a word - that an AI model reads and generates one piece at a time.

Start Free QuizBrowse courses

In this guide

  1. 1What Token means
  2. 2Why Token matters

What Token means

AI models do not process whole documents at once. They break text into tokens, which are small pieces - a short word might be one token, while a longer word splits into several. The model reads and writes these tokens in sequence.

As a rough guide, one token is about three-quarters of a word in English, so 100 words is roughly 130 to 150 tokens. Both your input and the AI's response are counted in tokens.

Why Token matters

Tokens are how AI usage is measured and priced, so the concept matters once you build real tools. Understanding it helps you manage cost and stay within limits.

AI service pricing is usually based on tokens used
A model's context window is measured in tokens
Knowing this helps you keep automations affordable
It explains limits on how much text an AI can handle at once

Frequently asked questions

Tokens are the exact units an AI model processes, so they measure usage precisely. Words vary in length and structure, while token counts directly reflect the work the model does.

More AI terms

Context WindowThe maximum amount of text an AI model can consider at once, including your input and its reply.AI HallucinationWhen an AI produces information that sounds confident and plausible but is actually incorrect or made up.Zero-Shot LearningWhen an AI completes a task it was not given any examples for, relying only on its general training.Few-Shot LearningGuiding an AI by including a few examples in your prompt so it follows the pattern you want.Chain-of-Thought PromptingA prompting technique that asks an AI to reason step by step, improving accuracy on complex tasks.Prompt ChainingBreaking a task into several prompts where each AI step feeds its output into the next.

Ready to build the AI skills your future depends on?

Take the free 5-minute quiz and get a personalized learning plan built around your goals, schedule, and experience.

Start Free QuizBrowse courses