大模型
pub.towardsai.net ·
@“The AI Engineer”
为什么 LLM 预测 token 而不是单词
文章解释语言模型以介于字符与单词之间的子词 token 为最小处理单位，tokenizer 将文本片段映射为整数 ID，提示词、上下文窗口和定价均以 token 计量。作者指出单词级词表会因专有名词、术语、拼写错误和多语言膨胀到数百万条，且无法表示未见过的词，因此采用子词切分；这一设计会直接影响上下文预算、延迟和费用，并因用户语言不同而放大成本。
AI推荐理由 面向开发者的科普文章，解释 LLM 为何以 token 而非单词或字符为预测单位，并指出 tokenization 直接影响上下文预算、延迟和 API 成本，且不同语言会放大成本差异。正文片段仅覆盖词表规模与未登录词问题，未给出具体数据或实验，属概念性解释而非新发现；来源为 Medium 个人专栏，权威性一般。
62
正文
Member-only story
Artificial Intelligence
Machine Learning
LLM
Ai Engineering
NLP
Why LLMs Predict Tokens Instead of Words
“The AI Engineer”
5 min read
·
13 hours ago
--
Listen
Share
Press enter or click to view image in full size
image created by GEMINI
Here’s the friend link….
A design choice with real costs, and who actually pays for it.
Ask most developers why a language model predicts tokens instead of words and you will usually get a shrug. Tokens, words, close enough. That shrug is expensive. Tokenizing text into subword units instead of whole words sets your context budget, your latency, and your bill, and depending on which language your users type in, it can quietly multiply your costs before a single API call returns an answer.
A token is the smallest unit of text a language model actually reads. Not a character, not a word, a chunk somewhere between the two, produced by a tokenizer that maps each piece to an integer ID the model was trained on. Prompts, context windows, pricing, all of it is measured in these units, not in words.
Why Not Whole Words, Why Not Characters
Words look like the obvious unit until you count them. English has roughly 170,000 words in active use. Add proper nouns, jargon, typos, and every other language, and a word level vocabulary balloons into the millions, each entry needing its own row in an embedding table. However large you make it, a user will eventually type something it has never seen, and a strictly word level model has no way to represent that.