从微调到量化：免费层硬件能证明什么

Machine Learning LLM Software Engineering Artificial Intelligence AI What a Free-Tier GPU Can Prove About a Fine-Tuned LLM, and What It Can’t Yuval Mehta 7 min read · Just now -- Listen Share Press enter or click to view image in full size Photo by Digital Content Writers India on Unsplash A base 4B model refuses or redirects the right call on adversarial prompts 79.5% of the time. After QLoRA fine-tuning on 150 synthetic examples, that number moves to 89.7%. The serving benchmark meant to prove the model holds up under concurrent load never produced a number, because a single free-tier GPU cannot hold enough KV cache to test past one request at a time. This is a write-up of what a full LLM fine-tuning lifecycle looks like when every stage runs on hardware you didn’t pay for and what that hardware will and won’t let you claim by the end. I used AI assistance to structure this piece, check code, and pull benchmark citations. The project, the numbers, and the conclusions are mine. TL;DR Fine-tuned a small model on 150 synthetic, adversarially augmented examples for a regulated-style compliance task domain, using QLoRA through Unsloth and TRL. Adversarial-holdout accuracy moved from 31/39 to 35/39 after fine-tuning. Eval-set accuracy held at 16/16 throughout. GPTQ and AWQ both held perplexity close to full precision, but a 2026 benchmark found the serving kernel underneath each format moves throughput more than the quantization algorithm does. SGLang was worth testing here for a specific reason: every training example shares one long system prompt, which is exactly the condition RadixAttention is built for. The serving and load-testing story stops at one data point. A single request returned correctly but slowly on a T4. Concurrent benchmarking needs hardware this project didn’t have. Press enter or click to view image in full size AI Generated with Napkin AI Context The project started from a gap, not a topic. Inference optimization and production serving were the weakest lines on my profile relative to fine-tuning and agentic systems, so the brief was to build something that forced me through the whole lifecycle: synthetic data, evaluation, fine-tuning, quantization, serving, and whatever load testing a single GPU would permit. The domain is a regulated-style compliance task, close enough to real pharmaceutical GxP workflows to be a meaningful test (structured deviation reports, corrective-action plans, audit-trail reasoning) but built entirely on synthetic data with no proprietary system names or client data anywhere in it. The constraint that mattered most wasn’t the domain. It was the hardware: Kaggle’s free T4 tier for training and quantization, no paid serving instance, and no multi-GPU node. That constraint is the actual subject of this post. Fine-tuning: closing the gap that mattered The base model’s weakest spot wasn’t the straightforward SOP and deviation-report tasks. It was the adversarial holdout: 39 examples built specifically to test whether the model would comply with a bad request (backdating a record, omitting a response-time delay, skipping an authorization step) or push back the way a compliant system should. The base model got 31 of those right. Get Yuval Mehta’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in QLoRA training used Unsloth’s FastLanguageModel for the quantized base and kernel replacement, with TRL's SFTTrainer for the actual training loop. Unsloth's own benchmarks report roughly 2x training speed and up to 70% lower VRAM against a HuggingFace-plus-FlashAttention baseline, which is the difference between this fitting on a free T4 at all and not. from unsloth import FastLanguageModel from trl import SFTTrainer, SFTConfig model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Qwen3.5-4B", max_seq_length=2048, load_in_4bit=True, ) model = FastLanguageModel.get_peft_model( model, r=16, lora_alpha=16, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], use_gradient_checkpointing="unsloth", ) trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=train_ds, # ChatML-formatted, 95 examples args=SFTConfig(max_steps=300, per_device_train_batch_size=4, gradient_accumulation_steps=4, learning_rate=2e-4), ) trainer.train() After fine-tuning, the eval set held at 16/16, which mostly confirms the model didn’t regress on what it already handled. The adversarial holdout moved to 35/39, a real gap closed but not a solved one: four cases still went the wrong way. This holds for a narrow, well-specified task domain with a small, curated dataset. It says nothing about whether the same QLoRA recipe closes an adversarial gap on a broader, less-structured domain, and the honest counterargument is that 150 synthetic examples might be teaching the model to pattern-match this specific holdout’s phrasing rather than the underlying judgment. Quantization: the kernel matters more than the algorithm GPTQ and AWQ both compressed the fine-tuned model to 4-bit with a small, expected quality cost: GPTQ came in at roughly +0.15 perplexity over full precision at 42 ms per token at batch size 1, and AWQ at roughly +0.08 perplexity and 38 ms per token. Both numbers are close enough to full precision that either format would be a reasonable production choice on quality grounds alone. The more useful finding didn’t come from my own runs. A 2026 benchmark comparing GPTQ, AWQ, and GGUF on the same model, same hardware, and same serving harness found that swapping the serving kernel underneath identical GPTQ or AWQ weights moved throughput by up to 10.9x, while the choice of algorithm accounted for a fraction of that gap. Marlin-AWQ hit 741 tokens per second on an H200 against the default kernel’s 68. That’s a bigger lever than most “GPTQ vs AWQ” writeups ever mention, because most of them stop at the Hessian-math-versus-activation-scaling explanation and never touch a GPU. The practical implication: picking a quantization format by its paper’s stated quality claim, without checking which kernel your serving engine actually dispatches to for that format, is picking the wrong variable to optimize. In my own runs, the perplexity gap between GPTQ and AWQ was small enough that either format would have shipped. The decision that actually should have taken longer than it did was confirming which kernel vLLM was dispatching to for each format, not which paper’s quality benchmark to trust. AI Generated with Napkin AI Serving engine choice: prefix overlap, not brand preference Every one of this project’s 150 training examples opens with the same long system prompt (the compliance-assistant role definition, the ALCOA+ data-integrity framing). That’s not a stylistic choice; it’s a structural fact about the dataset, and it’s the actual reason SGLang was worth testing here rather than a default “compare the two popular engines” exercise. vLLM’s PagedAttention and SGLang’s RadixAttention solve the same memory-management problem differently. RadixAttention caches shared prompt prefixes across requests in a radix tree, so a request that starts with an already-cached prefix skips recomputing it. A 2026 H100 benchmark found the two engines land within about 5% of each other on unique-prompt workloads with no shared prefix, but once 60% or more of requests share a common prefix, SGLang’s latency advantage becomes measurable and consistent across concurrency levels. A dataset where every single example shares a 200-plus-token system prompt is close to the strongest case that benchmark tests for. This is the mechanism, not the benchmark result: I can state why SGLang deserved a real test on this specific workload. I cannot yet state which engine actually won it on my own hardware, which is the next section. Press enter or click to view image in full size AI Generated with Napkin AI What’s built but not proven The repo also includes an FP8 quantization path through Modal, an SGLang serving wrapper alongside the vLLM one, a Locust load-testing harness with a ramp profile, and a Docker Compose stack with Prometheus and Grafana. All of that code runs. None of it has a benchmark number behind it yet. Building the serving and observability layer was still worth doing, since it’s the difference between a training script and something that looks like a deployable system, but I’m not going to attach a performance claim to code I haven’t been able to load-test. Gotchas The hardware ceiling is the real finding, not a footnote. I lost access to the T4 before I could log the exact per-request figure, so I won’t invent one from my own run. A published estimate for the same model on the same hardware gets close: hardware-fit modeling for Qwen3.5–4B at 4-bit on a 16GB T4 puts decode speed around 56 tokens per second with a roughly 3.5-second time to first token. That figure is a modeled estimate from published specs, not a live empirical benchmark, and it reflects a general local-runtime stack rather than the vLLM setup this project used, so treat it as an order-of-magnitude reference, not an exact match. It still supports the point worth making: fine-tuning changes the weights, not the architecture, the parameter count, or the quantization format, so it was never going to change how fast a T4 generates tokens. What fine-tuning bought here was the 79.5-to-89.7 jump. What it couldn’t buy was a faster GPU, and the vLLM-vs-SGLang comparison, the TTFT percentiles, and the throughput-under-load numbers all still require a GPU class this project didn’t rent. The adversarial holdout is a proxy, not a guarantee. A 10.2-point jump on 39 examples is a real, measured result on this specific test set. It is not evidence the model generalizes to adversarial patterns it wasn’t trained against. Anyone using synthetic adversarial data to validate a fine-tune should treat the holdout score as a floor, not a ceiling. Conclusion The parts of this project that needed compute and time, not money, are done and measured: the data pipeline, the fine-tune, and both quantization formats. The part that needed money, a GPU class that can hold a concurrent KV cache for more than one request, isn’t. That’s a fair trade for a free-tier build, but it leaves an open question I don’t have a clean answer to yet: how much of a fine-tuned model’s production readiness can free-tier hardware actually certify, and how much of it stays unverified until someone puts a real GPU behind it.