Laya 对决 Gemma：谁能在 Doom 游戏中胜出？

Artificial Intelligence Games Generative Ai Tools AI Genai Laya vs Gemma: Who Wins the Doom Game? A “System One” decision model and a small general-purpose LLM, trained on the same 979 examples, play the same Doom levels on the same GPU. The answer is less exciting than the launch posts, and more useful. Gian Luca Bailo, Ph.D. 13 min read · Just now -- Listen Share Press enter or click to view image in full size doomLaya (left) and doomGemma (right) on FreeDoom MAP01, seed 48, seconds 4.5 to 9.5. The panel shows the probability each model assigns to its options. Ten days ago TypeSafe AI launched Jev, a model that does not write. You give it a state and a set of typed questions, and it returns probabilities over the answers you allowed, in a single pass. They call it a System One model, after Kahneman, and the launch numbers are the kind that make me reach for a notebook: up to 193.6 times faster and 444.6 times cheaper than an LLM on “System One tasks”. Within days an open alternative appeared: Laya, from Convai Innovations. It is a 421M-parameter ModernBERT-large encoder with a small decision head, released under Apache 2.0, and it runs on anything. Then the demos started, and a surprising number of them were Doom. One of them, azalio’s doomLaya, fine-tuned Laya on the game and showed it finishing FreeDoom MAP01 faster than Jev. I have no problem with the demo. I have a problem with the sentence people read into it: that the architecture is the reason. Two different things could explain that result. One is the architecture, a bidirectional encoder that scores every option at once. The other is much more boring: a small model, trained on the task. These two can be separated. You take a general-purpose LLM, train it on exactly the same data, and see what happens. So that is what I did. The short version With the same 979 examples, a 27-minute LoRA on a consumer GPU brings Gemma 4 E2B level with Laya v3 in the game (6 out of 6 against 6 out of 6 on MAP01), and ahead of it on single decisions. At equal size a decoder is as fast as Laya (22 ms against 23); at equal game result, Laya is still about three times faster. On a level neither model has seen, nobody finds the exit, not even the rule-based teacher that produced the labels, so on this bench generalization cannot be measured at all. What already existed Before running anything I checked what had been done. doomLaya compares Laya v3 with Jev on MAP01, on one seed, with no trained LLM, and its authors write that a new level’s geometry has not been tested. system-one-open replicates Jev on Gemma 4 E2B with a LoRA, but plays Doom untrained and reports no numbers. SauerkrautLM-Doom pits a 1.3M model against untrained LLMs, which is not a fair fight. anth.us trains Laya on Jev’s labels, for sentiment classification. An issue on tensor-logic proposes exactly this comparison, but nobody ran it. I could not find a comparison with the same data as Laya v3, a LoRA on a generative LLM, several seeds, an unseen level, calibration and latency on the same hardware. The bench The game side is doomLaya’s code, unchanged: ViZDoom 1.3.1, freedoom2.wad, 35 tics per second. The agent turns the game state into text (health, weapons, visible enemies with id and distance, items, doors) and asks two multiple-choice questions. The first is the command: shoot enemy #N, pick up item #N, open the door, go to the exit, retreat, explore, wait. The second is the weapon. A deterministic executor then aims, plans the path and presses the keys. Every choice belongs to the model; the executor only carries it out. Requests are asynchronous, at most one every 0.5 s, and the previous command stays valid for 2 s. The game does not wait for the model. A slow model simply reacts late. To put LLMs in the same seat I wrote a small bridge with the same /predict API as Laya's server. Each option becomes a letter, and a grammar allows exactly one token, chosen among the valid letters. The probabilities come from the top log-probabilities, renormalised over the options. This is single-pass scoring, the prompt read once plus one token, not free generation. It also means the famous "no type errors" property is not exclusive to System One models: any decoder gets it with a grammar. Everything ran on the lab machine: Ryzen 5 5600X, RTX 4070 12 GB plus RTX 2070 SUPER 8 GB, Ubuntu 24.04. Every latency that gets compared was measured on the 4070. Same data, 27 minutes The training data are doomLaya’s own files: 979 questions for training and 130 for validation, the exact set used for Laya v3. Seeds 42 and 43 for training, 44 for validation, all on MAP01. The labels were not written by people. The game’s rules assign them offline, so every model here is learning to imitate a rule-based teacher. The LoRA is ordinary: rank 16, alpha 32, on all attention and MLP projections, loss only on the answer letter, and, as doomLaya does, I kept the epoch that scored best on validation. On Gemma 4 E2B that is 24.2M trainable parameters at 550 s per epoch on the 4070, best epoch 2 of 3. I also trained Gemma 3 270M the same way, 101 s per epoch. After merging and converting to GGUF Q8_0, doomGemma weighs 4.9 GB and doom-270M 292 MB. Laya v3 is the checkpoint azalio published, verified against its checksums and not retrained. It was trained in three phases, the head first and then the top three encoder layers. My LoRA touches every layer. Keep that in mind for the single-decision numbers below. Two snags worth recording. Gradient checkpointing throws a CheckpointError on Gemma 4, so it stayed off. And E2B's per-layer embedding table weighs 4.4 GiB, more than the transformer layers themselves; I kept it in system RAM and the compute on the 4070 alone. Round one: one question at a time Press enter or click to view image in full size Accuracy on the 130 validation questions. Untrained, Gemma 4 E2B beats both Laya checkpoints; after training on the same 979 examples, doomGemma reaches 0.98 against 0.82 for Laya v3. Untrained, nobody is good, but the order is not the one the launch posts suggest. Gemma 4 E2B, with no training at all, answers 62% of the validation questions correctly; both Laya checkpoints stop at 48%. I tested the English base checkpoint as well as typed-decisions, because the model card warns that typed-decisions is specialised on four synthetic workflows. Same result. The typical untrained mistake is choosing “pick up” when the right answer was “exit”. After training, doomGemma reaches 0.98, doom-270M 0.93, Laya v3 0.82. Two cautions. The validation set was also used to pick the best epoch, for every model, so the absolute numbers are optimistic. And with a lighter fine-tuning protocol on Laya v3, the gap on single decisions may say as much about the protocol as about the model. Round two: does the confidence mean anything? Press enter or click to view image in full size Calibration before and after temperature scaling. Untrained LLMs are overconfident; after the LoRA doomGemma has the lowest ECE and Brier score, though at 0.98 accuracy the Brier score is the more telling number. Calibration is the other half of the System One promise: when the model says 80%, it should be right about 80% of the time. I fitted a temperature for each model with 2-fold cross-validation on the validation set, so no number in the table is measured on the data it was tuned on. Untrained LLMs are badly overconfident. Gemma 4 E2B answers with probability 1.00 even when it is wrong, and its confidence has to be cooled by a factor of about 4.4. Laya base has the lowest untrained ECE, 0.085, and it would be easy to write that it “knows what it doesn’t know”. It doesn’t. Its low ECE comes from low, flat confidences, not from telling right answers from wrong ones: accuracy is 0.48, and its Brier score, 0.65, is the same as typed-decisions. In the game it chooses “wait” 350 times out of 350. Calibrated, but barely discriminating. Its own model card says the opposite, that the base checkpoint ships overconfident (mean ECE 0.466 on their benchmark). That it is not overconfident here is a difference worth flagging, not a confirmation. After the LoRA, doomGemma is the best calibrated model in the group, even before temperature scaling. With 0.98 accuracy, though, a low ECE comes almost for free; the Brier score is the more solid number. Round three: the game Press enter or click to view image in full size Every player on MAP01, seed 48. Only the fine-tuned models, Jev and the rule-based teacher reach the exit; no untrained model does. The untrained models share an obsession with picking things up. Gemma 4 E2B chose “pick up” 271 times out of 350, Gemma 4 26B-A4B 274 out of 349. When enemies are in view they shoot less than half the time. On seed 48, the one doomLaya used, doomGemma exits in 56.6 s against 71.0 for Laya v3 and 69.0 for Jev (that last figure is doomLaya’s; the API is paid and I did not reproduce it). It is a tempting headline. It is also one seed. Press enter or click to view image in full size Six seeds on MAP01. Laya v3 and doomGemma both exit every time, in 59.5 and 60.0 seconds on average: a tie. The teacher that produced the labels is slower, at 81.6 seconds. Over six seeds, five of which are in neither training nor validation, the headline disappears. Both exit six times out of six, in 59.5 s against 60.0 on average. Laya v3 is more variable, doomGemma more consistent, and the 56.6 against 71.0 of seed 48 was partly luck. This is also why a single seed is not a benchmark, including the one in the Laya-versus-Jev demo. Get Gian Luca Bailo, Ph.D.’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in The last row is the teacher. I copied the labelling function from doomLaya’s dataset builder, unchanged, and let it play through the same executor at the same cadence. It exits six times out of six, but in 81.6 s on average, slower than both students. The teacher does not optimise exit time: its rules include collecting useful items, and it collects more (17.7 on average, up to 31) and kills more. The models exit earlier because they imitate it imperfectly and pick up less. That is a side effect of imperfect imitation, not a merit of the models. On exit time alone, 60 s is not the ceiling of this bench. The model that aced the exam and failed the game Press enter or click to view image in full size MAP02, a level no model saw in training. Nobody exits, the rule-based teacher included: the pipeline stalls in front of a door that needs the yellow key. doom-270M has 0.93 on validation, better than Laya v3, and exits only two games out of six. The telemetry explains it. The executor could not carry out its command 60% of the time on MAP01, mostly because the path was blocked. Each answer, taken alone, looks right; strung together, they ask for things that are impossible at that moment. Questions scored one at a time do not measure coherence from one moment to the next. On that metric Laya v3 is the best of the group, at 13%, ahead of doomGemma at 22% and even of the teacher. That point goes to Laya. Latency: two sentences that are both true Press enter or click to view image in full size Share of time in which the executor cannot carry out the chosen command. doom-270M scores 0.93 on validation but asks for the impossible 60% of the time; Laya v3 is the most coherent at 13%. I replayed the same 1,416 real game packets on the 4070, end to end. At equal size, the architecture gives no intrinsic speed advantage. Gemma 3 270M after the LoRA, run directly in PyTorch as one pass with both questions batched, answers in 22 ms. Laya v3 through its own server answers in 23. If anything the comparison favours Laya, since my script has no HTTP in the way. At equal game result, the picture changes. The decoder as fast as Laya exits two games out of six. The decoder that ties Laya in the game is doomGemma, at 59 ms through llama.cpp and 84 ms run directly: 2.5 to 3.7 times slower, and 4.9 GB against less than 1. With these data there is no decoder yet that is both as fast as Laya and as good as Laya. What I cannot say is why. Either the bidirectional encoder gets more out of each parameter on this kind of task, or nobody has tried to squeeze a decoder between 270M and 2B. Both are plausible. Two side notes. Through llama.cpp, the 270M costs about as much as E2B (71 ms against 59): past a certain point the time goes into the serving chain, HTTP, the grammar, log-probabilities over a 262,000-token vocabulary, not into the model. And Gemma’s latency through llama.cpp has two humps, around 59 and 108 ms, which come from the prompt cache being reused between similar states. Finally, the cadence. The game asks for a decision at most every half second, so between 23 and 109 ms nothing much changes in play. The tie does not show that latency doesn’t matter. It shows that this bench does not measure it. It would matter in a tighter control loop, a robot for instance. MAP02: the level nobody finishes Press enter or click to view image in full size Latency on the same RTX 4070, replaying 1,416 real game packets. At equal size a decoder matches Laya (22 ms against 23); the decoder that matches Laya in the game is about three times slower. The obvious last test was a level neither model had seen. Nobody exits: zero out of eighteen, teacher included. It is not for lack of trying. The “exit” option was always on the menu, because the navigator knows where the exit is from the map, and the models chose it: doomGemma 69 times and Laya v3 38 times on seed 48. The problem is downstream. The teacher chooses “open the door” about 500 times out of 580, and the executor spends 7,600 to 9,500 tics in front of the same door. Reading freedoom2.wad settles why. All six linedefs of that door carry special 27, a door that opens only if the player holds the yellow key. The yellow key is on MAP02, far from the door and out of sight. Three things combine into the deadlock: doomLaya's sensor treats key doors as openable and offers them as open_door; the teacher's rule opens any closed door within 4 m; and keys only appear among the options when they are visible within 12 m. So on MAP02 the whole chain fails, rules, executor and state description together, not the models’ ability to generalize. Before writing “they learned the level, not the game”, I had to run the teacher; it fails too, and the WAD says why. Measuring generalization here would first need actions that can go and look for keys, and a sensor that doesn’t offer doors that cannot be opened. Only then does it make sense to ask whether the models imitate the teacher there too. So, who wins the game? On MAP01, nobody: it is a tie. What I think these numbers support is narrower than any headline. Training on the task matters more than the architecture. Untrained, everybody loses, and Gemma 4 E2B is more accurate than Laya, though far less calibrated. With the same 979 examples, a 27-minute LoRA on a consumer card brings a general-purpose LLM level with the specialised model in the game, and ahead of it on single decisions. On latency the architecture has no intrinsic advantage at equal size, but at equal result Laya is about three times faster today, because it gets there with a smaller model. On this bench that difference does not show. Accuracy on single questions does not predict the game. The 270M model with 0.93 finishes two games out of six; Laya v3 with 0.82 finishes all of them. And generalization to new levels cannot be measured here, because the level defeats the whole pipeline, teacher included. What does it mean that a fine-tuned LLM gets there too? That the valuable part of a System One model is the contract, not the architecture: a closed answer space, typed questions, probabilities you can act on, one pass per decision. Any decoder can offer that contract with a grammar and a scoring pass, and on this task a few hundred labelled examples were enough to make it hold up in the game. The practical consequence is that the choice stops being “LLM or decision model” and becomes an ordinary engineering trade-off. If you already serve an LLM, you can train it on your decisions and keep one model in production. If memory and latency are the constraint, a small specialised encoder still does the same job three times faster in a fifth of the space, and here it was also the more coherent player from one moment to the next. Either way, the durable asset is the labelled data; the model on top of it is interchangeable. This says nothing about Jev’s closed architecture or about how general these models are without training, which I did not measure: it is one game, one level, and labels written by rules. A System One model, in practice, is a small model trained on your task. So is a LoRA. That is the part worth paying attention to. Five things that would have made this meaningless Laya on the wrong GPU. In the first round Laya ran on the 2070 SUPER at 38 ms and looked slower than it is. Every compared latency was measured again on the 4070. The prompt cache. On MAP02 doomGemma seemed faster, 42 to 62 ms. It was llama.cpp reusing the cache between similar states, not the model. The wrong Laya. The first cold comparison used the typed-decisions checkpoint, which its own card calls specialised. I ran the base checkpoint too: same 0.48. Blaming the models for MAP02. Without the teacher’s run, the zero out of fifteen would have become a conclusion about generalization. It is a locked door. Tuning calibration on the numbers you report. Temperatures come from cross-validation, so each half is scored with a temperature fitted on the other. Limits Training and validation are MAP01 only; MAP02 turned out to be unsolvable by the whole chain, and no other level was tried. There are 130 validation questions, so each calibration fold has 65 and the decimals deserve caution. Laya v3 was trained by someone else, with a different method, and I did not try to improve it. Jev’s numbers come from doomLaya and were not reproduced. The direct 270M latency comes from my own PyTorch script, not a production server. The bridge reads only the top 20 log-probabilities, and options outside that list get a floor of 1e-4; it had no visible effect on accuracy or calibration. The very first untrained E2B game used a prompt with both questions together; all later runs ask them separately, as in training. Gemma 3 270M comes from unsloth’s copy, since Google’s original needs an account and I could not compare checksums. All times are game time, and the game runs in real time. Method note The experiments were run by an agentic pipeline (Claude Code) on my own hardware. I verified every figure against the raw files; the analysis and the conclusions are mine. Models and Code: doomGemma, Gemma 4 E2B + LoRA https://huggingface.co/dexmac/doomgemma-e2b-lora Gemma 3 270M + LoRA, 15 MB : https://huggingface.co/dexmac/doom-gemma3-270m-lora Code: https://github.com/dexmac221/doomgemma