重新思考意图路由：从 TF-IDF 到微调 Transformer

AI Intelligent Routing Fine Tuning Tf Idf Encoder Rethinking Intent Routing: From TF-IDF to Fine-Tuned Transformers A practical experiment in routing enterprise chatbot queries across APIs, LLM + RAG, and an explicit out-of-domain class — and in learning not to trust a pretty accuracy number too quickly. Camilo Medina 9 min read · Just now -- Listen Share Source: Image generated with AI. One chatbot. Three execution paths. The router decides where the question should go before anything tries to answer it. Code, experiments, and evaluation artifacts: github.com/CamiloMedina28/Intelligent-query-router One Chatbot, Different Ways to Answer When building an enterprise chatbot, it is tempting to treat every user message the same way: send it to a language model and let the model figure it out. Convenient? Yes. Always the right architecture? Not really. Consider two requests that could reach the same interface: “What are the currently active massive incidents?” “Explain the root cause of incident INC123456.” They look similar because both are written in natural language and both belong to the same operational domain. But they require very different execution paths. The first is a structured-data problem: an API can retrieve the answer directly. The second requires context — ticket notes, history, or related documents — and is better handled by an LLM backed by retrieval. That made the first problem for the chatbot surprisingly simple to state: not “What is the answer?”, but “Which component should be responsible for answering this?” Press enter or click to view image in full size Source: Image by the author. The router has three destinations: API, LLM, and UNDETERMINED. API covers deterministic or structured operations. LLM covers requests that need retrieval and interpretation. UNDETERMINED represents queries that do not reliably belong to either supported domain. That third class matters. The earliest version was binary: every query had to become either API or LLM, even when neither made sense. Eventually UNDETERMINED became a learned class rather than a confidence-threshold fallback. In the current model, the classifier produces three logits, applies softmax, and takes the argmax. No extra threshold is involved. A Real Dataset, with Real Constraints The dataset comes from real enterprise queries. That made it useful — and impossible to publish. The raw questions can contain internal terminology, customer references, ticket identifiers, and operational context covered by strict confidentiality agreements. Every example in this article is therefore synthetic or anonymized while preserving the original routing intent. The final dataset contains 9,189 queries, balanced across the three classes: 3,063 for LLM, 3,063 for API, and 3,063 for UNDETERMINED. The split used for the experiments was 6,432 training examples, 1,378 validation examples, and 1,379 test examples. Press enter or click to view image in full size Source: Image by the author. Because the classes are balanced, accuracy is easy to interpret and Macro F1 remains useful because it gives each route equal weight. More importantly, all four models were evaluated on the same test split, which makes the relative comparison meaningful. From TF-IDF to Full Fine-Tuning I did not start with the largest model I could fine-tune. I started with a baseline strong enough to be annoying. TF-IDF + Logistic regression The first experiment used TF-IDF with unigrams and bigrams, followed by multiclass Logistic Regression. It is simple, fast, interpretable, and particularly good at exploiting recurring domain phrases such as “active incidents” or “root cause.” Press enter or click to view image in full size Source: Image by the author. TF-IDF + Logistic Regression architecture The baseline reached 96.01% accuracy and a Macro F1 of 0.9601. That number changed the rest of the experiment: any Transformer now had to beat something that was cheap and already very good. Frozen MiniLM Next, TF-IDF was replaced by paraphrase-multilingual-MiniLM-L12-v2. Each query became a dense 384-dimensional representation, while a small MLP learned the three routing classes. The encoder itself stayed frozen. The result: 94.78% accuracy. The Transformer arrived and promptly lost to TF-IDF. Not exactly the dramatic entrance one expects from 2020s NLP. The lesson was useful: pretrained semantic representations are richer, but richer does not automatically mean better for a specific decision boundary. Press enter or click to view image in full size Source: Image by the author. frozen encoder and trainable linear head architecture Partial Fine-Tuning The next step unfroze the final two Transformer blocks while keeping the rest of the encoder fixed. About 3.6 million parameters became trainable, together with the classification head. Accuracy improved to 95.58% and Macro F1 to 0.9556. Better than the frozen encoder, still slightly worse than TF-IDF. Partial adaptation was moving in the right direction, but the baseline was still sitting there, being inconvenient. Full Fine-Tuning Finally, the entire MiniLM encoder was allowed to adapt to the routing objective. The architecture did not change; the important difference was where learning could happen. Full fine-tuning made roughly 117.7 million parameters trainable. Press enter or click to view image in full size Source: Image by the author. Full-fine tuning architecture This was the first neural configuration to clearly beat the baseline: 98.04% accuracy and 0.9804 Macro F1. Get Camilo Medina’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in I am deliberately not turning this into a claim that full fine-tuning is universally better for intent routing. The narrower statement is the useful one: for this dataset, this base model, and this routing task, full fine-tuning performed best among the four approaches tested. The Comparison That Actually Matters Putting the four approaches side by side makes the story clearer than four separate walls of metrics: Press enter or click to view image in full size Source: Image by the author. The progression is not monotonic once the classical baseline is included. Moving from TF-IDF to a pretrained Transformer initially made the result worse. Partial fine-tuning recovered some performance. Only full fine-tuning produced a clear improvement over the baseline. Complexity had to earn its place. This is why the baseline mattered. If I had compared only the neural models, partial fine-tuning would have looked like an obvious success. TF-IDF changed the question from “Did this improve over the last Transformer?” to “Did this justify the extra complexity at all?” Looking Beyond 98% Accuracy 98.04% looked like a victory lap. Then I audited the test set. There were zero literal duplicate queries across the splits. After normalization, however, 87 test rows matched training examples, and 327 test queries had a training neighbor with cosine similarity of at least 0.95. The audit also found three normalized query pairs with contradictory labels, affecting two test rows. High semantic similarity is not automatic proof of leakage, but it is a signal that the test set contains families of queries closely related to examples already seen during training. That changes what the headline number can honestly claim. 98.04% is evidence that full fine-tuning performed best on this evaluation setup. It is not evidence of 98.04% accuracy on completely new query families. As a diagnostic, I removed the 87 normalized matches and recalculated the full fine-tuned model on the remaining 1,292 rows. Accuracy became 97.99%. That is reassuring, but it is not a new independent test: the filtering happened after the original evaluation and semantic overlap still exists. The point is not to make the result look worse. It is to understand what the number measures. A metric without the data story behind it is mostly decoration. What the Router Still Gets Wrong Press enter or click to view image in full size Source: Image by the author. Confusion matrix for the full fine-tuned model. Rows are actual routes; columns are predictions. Those 15 cases were worth more than another decimal place of accuracy. Eleven contained spelling errors, three used abbreviations, two were elliptical or incomplete, and one was a broad request. The categories overlap, but the pattern is clear: real users do not write benchmark-friendly sentences. More interestingly, nine of those 15 wrong predictions assigned at least 0.99 probability to UNDETERMINED. The model was not slightly wrong. It was confidently, enthusiastically wrong. That is a useful reminder that softmax probability is not the same thing as calibrated certainty. In this router there is no threshold after softmax; the highest-scoring class wins. A prediction near 1.0 simply means one class dominates the model’s learned score distribution. The review also exposed label ambiguity. One short query equivalent to “active?” had contradictory labels relative to similar variants elsewhere in the data. Some errors therefore reflect the model, some the input, and some the data definition itself. A confusion matrix cannot tell those stories. Reading the errors can. Why UNDETERMINED Is a Learned Class The first version of the router only knew API and LLM. That implicitly assumed every query belonged to one of those two execution paths. Real input quickly disagreed. Instead of keeping the classifier binary and inventing a rule such as if confidence < threshold: reject, the current design teaches the model a third class directly. UNDETERMINED is not the absence of confidence; it represents examples outside the supported routing domains. Those are different ideas. A model can be uncertain between two valid routes, or extremely confident about an invalid one. The error analysis gave us examples of the latter. Historically, introducing the third class improved the router, but the old binary experiment was not preserved under a directly comparable dataset and evaluation protocol. So I will not manufacture an improvement percentage for the sake of a cleaner story. The architectural motivation is enough: the system needed a way to learn “neither.” What I Learned Building the Router A baseline should be capable of ruining your favorite idea TF-IDF was not there to lose politely. It beat both the frozen encoder and partial fine-tuning. That made the final result more meaningful because full fine-tuning had to outperform something genuinely competitive. A Transformer is not a performance guarantee The frozen encoder was semantically richer and empirically worse. Partial adaptation helped. Full adaptation helped more. The experiment shows an association between more domain adaptation and better performance here; it does not prove a universal law or isolate the exact causal mechanism. Accuracy is the start of evaluation, not the end The 98.04% score was useful, but the audit and failure analysis were more informative. They exposed train-test redundancy, contradictory labels, noisy language, and highly confident mistakes — none of which is visible in a single headline metric. Routing is a systems problem wearing an NLP hat A wrong prediction is not merely a wrong label. It selects the wrong execution path: the wrong retrieval strategy, the wrong tool, or the wrong subsystem. That is why routing quality has to be evaluated in architectural terms, not only classifier terms. Final Thoughts Could I build this router differently today? Of course. An LLM can classify intents directly. Embedding-based routers can reduce custom training. Agent and tool-orchestration frameworks can move routing into a higher-level abstraction. Managed services can make most of this disappear behind an API. If the only goal were the shortest path to a production prototype, I might choose one of those options. But the important result of this project was not finding the shortest path. Building the router step by step forced me to understand what the abstractions hide: how labels encode architecture decisions, why a baseline matters, what frozen representations can and cannot do, what fine-tuning actually changes, how a test split can flatter a model, and why a probability of 0.99 can still be wrong. Tools will get better. Some of the code in this project will eventually become unnecessary. That is fine. Knowing what happens underneath those tools makes it easier to choose them, evaluate them, and recognize when they fail. The final model was useful. Understanding why it worked — and where it did not — was the real result. Project repository → Intelligent-query-router on GitHub