BPE 风格分词器：决定 LLM 能看到什么的小算法

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. When you type: unbelievableness Enter fullscreen mode Exit fullscreen mode an LLM does not see the word. It sees something more like: ["un", "believ", "ableness"] Enter fullscreen mode Exit fullscreen mode Or perhaps: ["un", "believe", "ness"] Enter fullscreen mode Exit fullscreen mode Or, depending on the tokenizer: ["un", "bel", "iev", "ab", "leness"] Enter fullscreen mode Exit fullscreen mode That difference is not cosmetic. Tokenization determines the length of the model's input sequence, which affects context usage, inference cost, attention computation, vocabulary size, handling of rare words, programming-language behavior, multilingual performance, and even some model failure modes. And one of the most widely used ideas behind modern LLM tokenizers has an unusually non-LLM origin: a 1994 data-compression algorithm by a programmer named Philip Gage. The basic idea is remarkably simple: Find things that occur together often, and give them a reusable symbol. That idea eventually went from C programmers doing data compression, to neural machine translation, to GPT-2 and the tokenization machinery surrounding today's language models. This article builds the idea from intuition to implementation, then looks at the less obvious engineering consequences. 1. What problem is a tokenizer actually solving? A neural network wants numbers. Your input is text: The server returned HTTP 500. Enter fullscreen mode Exit fullscreen mode The model needs: [the, server, returned, HTTP, 500, .] Enter fullscreen mode Exit fullscreen mode which eventually becomes integer IDs such as: [464, 2126, 4710, ...] Enter fullscreen mode Exit fullscreen mode The obvious question is: Why not make every word a token? Suppose the vocabulary contains: cat dog server database running ... Enter fullscreen mode Exit fullscreen mode Now consider: microarchitectural microarchitectures microarchitecturally Enter fullscreen mode Exit fullscreen mode You immediately run into the open-vocabulary problem. There are infinitely many possible strings. New product names appear. Developers invent identifiers. People misspell things. Languages generate long compounds. Users paste URLs, hashes, code, emojis and arbitrary Unicode. A word-level tokenizer therefore needs some fallback mechanism. At the other extreme, we could tokenize one character at a time: m i c r o a r c h i t e c t u r a l Enter fullscreen mode Exit fullscreen mode Now everything is representable, but sequences become much longer. That creates a fundamental tradeoff: word tokens small fragments -> common morpheme-like units -> common words -> common multi-character sequences Enter fullscreen mode Exit fullscreen mode But an important distinction: BPE does not understand morphology. It does not know that: walk walking walked walker Enter fullscreen mode Exit fullscreen mode share a linguistic stem. It only knows that certain byte sequences occur frequently enough to be worth merging. That distinction matters when people say things like "the tokenizer understands prefixes." It does not. It has learned a segmentation that is useful according to its training statistics. A useful example Imagine a corpus where: hyperparameter Enter fullscreen mode Exit fullscreen mode occurs 50,000 times. Then the tokenizer has an economic incentive, in vocabulary terms, to represent something like: hyperparameter Enter fullscreen mode Exit fullscreen mode compactly. But suppose: hyperparametrix Enter fullscreen mode Exit fullscreen mode appears once. A BPE tokenizer can still represent it: hyper + parameter + ix Enter fullscreen mode Exit fullscreen mode or some other decomposition. This is the main advantage over word-level tokenization. It gets compression for common patterns without making the vocabulary responsible for every possible word. 5. Byte-level BPE: the trick that removes Ordinary character-level BPE has an awkward problem. Unicode is enormous. If you want every possible Unicode character to be a base symbol, your initial vocabulary is already huge. GPT-2 instead starts from bytes. There are exactly: 256 Enter fullscreen mode Exit fullscreen mode possible byte values. Any Unicode string encoded as UTF-8 becomes a byte sequence: text -> UTF-8 -> bytes -> BPE merges -> token IDs Enter fullscreen mode Exit fullscreen mode This has an important consequence: there is always a fallback representation. Even if a tokenizer has never seen a particular Unicode string during training, the raw bytes can still be represented. For example, an emoji such as: 👍 Enter fullscreen mode Exit fullscreen mode is represented internally by its UTF-8 bytes: F0 9F 91 8D Enter fullscreen mode Exit fullscreen mode The tokenizer may have learned to merge those bytes, partially merge them, or leave them separate. But it does not need a vocabulary entry literally corresponding to every possible Unicode character. That is a powerful design decision. There is another subtlety Naively running BPE over raw bytes has undesirable behavior. Suppose your corpus contains: dog dog. dog! dog? Enter fullscreen mode Exit fullscreen mode Frequency-based BPE may learn variants of entire sequences that are statistically frequent, wasting vocabulary entries on punctuation-specific combinations. GPT-2's approach therefore constrained which byte sequences could merge, while treating spaces specially. The objective was to retain the generality of byte-level representation without allowing the greedy learner to spend too much vocabulary capacity on accidental boundary variants. (OpenAI CDN) This is a recurring theme in tokenizer engineering: The basic algorithm is simple. Most of the engineering is deciding where the simple algorithm is allowed to operate. 6. The developer consequences: tokens are an economic unit This is where tokenization stops being an NLP curiosity. Consider a model with a context window of: 128,000 tokens Enter fullscreen mode Exit fullscreen mode If your tokenizer turns a piece of text into: 100,000 tokens Enter fullscreen mode Exit fullscreen mode you have room for approximately: 28,000 tokens Enter fullscreen mode Exit fullscreen mode of additional context. If another tokenizer represents exactly the same text as: 80,000 tokens Enter fullscreen mode Exit fullscreen mode you now have approximately: 48,000 tokens Enter fullscreen mode Exit fullscreen mode left. That is a 71% increase in remaining context. The difference gets even more important for long-context workloads. Attention cost For standard full self-attention, the interaction matrix is approximately: n x n Enter fullscreen mode Exit fullscreen mode so the dominant attention computation scales approximately as: O(n^2) Enter fullscreen mode Exit fullscreen mode Suppose tokenizer A gives you: n = 10,000 Enter fullscreen mode Exit fullscreen mode tokens. Tokenizer B produces 20% more: n = 12,000 Enter fullscreen mode Exit fullscreen mode The ratio of pairwise attention work is approximately: 12,000^2 / 10,000^2 = 1.44 Enter fullscreen mode Exit fullscreen mode So a 20% increase in token count can imply roughly: 44% more Enter fullscreen mode Exit fullscreen mode pairwise attention work. That is not a property of BPE itself. It is a consequence of the fact that tokenization controls sequence length. This gives us a useful engineering principle: characters -> tokenizer -> token count -> context utilization -> compute + memory + latency Enter fullscreen mode Exit fullscreen mode Token efficiency is also model capacity Imagine two representations of the same sentence: Tokenizer A: 12 tokens Tokenizer B: 18 tokens Enter fullscreen mode Exit fullscreen mode The model using B has to predict a longer sequence. At training time that means more prediction positions. At inference time it means more autoregressive steps. For APIs, token count also becomes a billing and capacity unit because providers commonly meter usage in tokens. So tokenizer quality is not merely: "Does the text tokenize?" Enter fullscreen mode Exit fullscreen mode It is also: "How economically does this representation use the model's finite sequence budget?" Enter fullscreen mode Exit fullscreen mode Code exposes the problem Consider: def calculate_monthly_revenue(customer_transactions): ... Enter fullscreen mode Exit fullscreen mode A tokenizer that is optimized around English prose may discover useful units such as: calculate monthly revenue customer Enter fullscreen mode Exit fullscreen mode But source code contains many patterns that have different frequency distributions: __init__ HTTPRequest std::unordered_map get_user_profile ===> Enter fullscreen mode Exit fullscreen mode Programming languages are therefore an interesting tokenizer workload because identifiers, punctuation, whitespace, delimiters and repeated syntactic fragments all compete for vocabulary capacity. The result is one reason why "tokenizer efficiency" should be evaluated on the actual distribution your model serves, not only on generic English text. 7. At inference time, BPE is a deterministic compression dictionary Once training is finished, the tokenizer no longer needs to "discover" anything. It has two important artifacts: vocabulary merge rules Enter fullscreen mode Exit fullscreen mode For example, imagine the merge ranking contains: 1. e s 2. es t 3. n e 4. ne w 5. new est ... Enter fullscreen mode Exit fullscreen mode Now given: newest Enter fullscreen mode Exit fullscreen mode the encoder applies the learned rules in their defined priority. Conceptually: n e w e s t Enter fullscreen mode Exit fullscreen mode then perhaps: ne w e s t Enter fullscreen mode Exit fullscreen mode then: ne w est Enter fullscreen mode Exit fullscreen mode then eventually: new est Enter fullscreen mode Exit fullscreen mode depending on the learned merge table. The output is something like: [new, est] Enter fullscreen mode Exit fullscreen mode The exact implementation used by modern tokenizers is optimized considerably beyond this toy procedure. A naive implementation that rescans an entire corpus after every merge would be unnecessarily expensive. But the conceptual model remains: base symbols + ordered merge rules = tokenizer Enter fullscreen mode Exit fullscreen mode And that has a subtle consequence for developers: token IDs are meaningless without the tokenizer definition that produced them. Token ID: 12345 Enter fullscreen mode Exit fullscreen mode does not inherently mean "hello" or "database." It means whatever entry 12345 refers to in a particular tokenizer vocabulary. This is also why changing tokenizers can invalidate embeddings, model inputs, cached token sequences and various pieces of preprocessing infrastructure. The tokenizer is effectively part of the model's interface contract. 8. What BPE does not solve BPE solves one problem very well: How do we turn arbitrary text into a finite vocabulary while giving common sequences compact representations? It does not solve everything. It does not guarantee linguistically meaningful boundaries. It does not guarantee equal token efficiency across languages. It does not make arithmetic easy. It does not make code identifiers naturally interpretable. It does not prevent pathological tokenizations. And it certainly does not give the model a semantic understanding of the pieces. You can see this clearly with a made-up identifier: calculateUserMonthlyNetRevenueExcludingRefunds Enter fullscreen mode Exit fullscreen mode The tokenizer might produce something like: calculate User Monthly Net Revenue Excluding Refund s Enter fullscreen mode Exit fullscreen mode Or something considerably less intuitive. That is perfectly fine from the tokenizer's perspective. Its job is not to discover what the identifier "means." Its job is to produce a sequence that fits within the vocabulary and represents the input efficiently according to patterns learned from its corpus. This also explains an important phenomenon when working with LLM APIs: two strings that humans consider almost identical can have materially different token counts. For example: camelCaseIdentifier Enter fullscreen mode Exit fullscreen mode and: snake_case_identifier Enter fullscreen mode Exit fullscreen mode may produce different segmentations because their character sequences and punctuation patterns have different statistics. Likewise: hello world Enter fullscreen mode Exit fullscreen mode and: hello_world Enter fullscreen mode Exit fullscreen mode are linguistically related but are not equivalent objects to a frequency-based tokenizer. The model ultimately sees the tokens, not our intuitive notion of "the same phrase." Conclusion: The tokenizer is the first compression algorithm in your LLM stack There is a useful way to think about the whole system. Your original text contains enormous redundancy. BPE performs a kind of learned compression: raw bytes | v frequent local patterns | v reusable subword tokens | v shorter sequence | v Transformer Enter fullscreen mode Exit fullscreen mode The irony is that the algorithm is not particularly sophisticated. Count adjacent pairs. Merge the frequent ones. Repeat. Yet that small mechanism sits directly in front of billions of neural-network parameters. And its decisions propagate everywhere: tokenizer -> sequence length -> context capacity -> attention computation -> inference latency -> memory usage -> training efficiency -> API cost -> multilingual behavior -> code handling Enter fullscreen mode Exit fullscreen mode That makes tokenization one of those pieces of infrastructure that is easy to ignore precisely because it works so well. The most interesting lesson may be historical. Philip Gage was trying to compress bytes in 1994. Sennrich, Haddow and Birch were trying to solve rare-word problems in neural translation in 2016. GPT-2 then adapted the idea to byte-level language modeling. A concept that began as a compact data-compression trick became part of the interface between human language and modern neural networks. That is a useful reminder for developers building ML systems: sometimes the important abstraction is not the complicated algorithm in the middle, but the small transformation that determines what the algorithm gets to see. What tokenization behavior have you found most counterintuitive in an LLM—code, multilingual text, numbers, punctuation, or something else? Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down. I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. ⭐ Star it on GitHub: HexmosTech / LiveReview Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff. blast-radius-demo.mp4 LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer. The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score How does Blast Radius scoring work? (a more technical explanation) Here's the goal: A 3-line fix in a function used by 40 other files, that also writes to a database, should score high. A 300-line UI change in one file, fully covered by… View on GitHub Click below to try LiveReview with your codebase: