Latent-GRPO：在连续思维空间中进行强化学习

When you sit down to solve a complex puzzle or plan three moves ahead in chess, do you narrate every synaptic firing to yourself in full, grammatically correct English sentences? Of course not. Human cognition operates across continuous, multi-dimensional mental representations—spatial, relational, intuitive. We only convert our thoughts into human language when we need to speak to someone else. Yet the dominant paradigm in AI reasoning today (from DeepSeek-R1 to OpenAI o1) forces neural networks to do the exact opposite. We train models to emit thousands of discrete text tokens enclosed in ... blocks: “Wait, let me think... Actually, let me reconsider... Hmm, what if I try...” During reinforcement learning (RL) post-training, this imposes an astronomical token tax. Models burn 80% to 90% of their generation time emitting sprawling English prose that no human will ever read. In multi-turn engineering workflows (such as multi-step SQL querying, hardware synthesis, or complex network routing), verbose text reasoning exhausts context limits, balloons the KV-cache, and stretches training runs from hours into days. Worse: in our experiments, up to a third of all model rollouts got chopped off mid-sentence by token limits, scoring zero reward and poisoning gradient updates. Latent-GRPO asks the obvious question: Why should an artificial neural network think in human words? By replacing discrete vocabulary tokens with continuous recurrent thought vectors in embedding space, models can reason internally across continuous manifold dimensions. Start here Understand why discrete Chain-of-Thought creates massive rollout bottlenecks, how continuous thought recurrence works in embedding space, how to solve the RL exploration dilemma with tractable action densities, and how two-pass gradient replay enables exact backpropagation. Embedding: A dense vector of real numbers representing a concept in continuous semantic space. Hidden state: The internal activation vector produced by intermediate transformer layers. Latent thought step: A recurrent model cycle where continuous vectors loop internally without emitting vocabulary logits. GRPO: Group Relative Policy Optimization: policy gradients computed relative to a group of sampled attempts. Two-pass replay: Collecting rollouts in eval mode with active KV cache, followed by an exact gradient forward pass. 1. The Token Tax: How Textual Chain-of-Thought Kills Rollouts In discrete Chain-of-Thought (CoT), reasoning is forced through an artificial bottleneck: the model’s vocabulary projection layer. For a modern foundation model like Qwen3.6-27B: The Autoregressive Penalty: Every single intermediate thought word requires a full forward pass through 64 transformer layers, a massive projection across 152,064 vocabulary logits, and an addition to the active KV-cache state. Context Window Choke: In agent tasks, environment observations and schema definitions already consume 1,500 to 3,000 tokens. Generating another 3,000 tokens of internal soliloquy per turn blows right past sequence limits. The Truncation Disaster: In our gft-studio multi-hop routing runs with discrete CoT, 19% to 34% of all generated trajectories hit the sequence ceiling mid-sentence. In reinforcement learning, a truncated completion fails its verifier and receives a reward of exactly 0. Your trainer ends up penalizing a model not because its reasoning was wrong, but because it didn’t finish talking before the bell rang. Continuous latent thinking eliminates this bottleneck: instead of projecting hidden states onto discrete words, the model feeds its internal recurrent representations directly back into itself in continuous space (d = 5,120): The short version of the same two paths. Left, the state is squeezed through one token. Right, the full vector is passed on. ht=fθ(context,z 0): Ensures that noise boosts promising paths without accidentally degrading probabilities on valuable tokens. Winner-Take-All Reinforcement: When multiple rollouts in a group solve the problem, only the top-scoring trajectory’s initial latent vector receives positive gradient reinforcement, avoiding muddy representation drift. 3.3 CoLaR: Direct Gaussian Latent Heads Instead of constraining thoughts to the vocabulary hull, CoLaR (Tan et al., NeurIPS 2025) equips the transformer with an MLP head that predicts mean and variance (μt, σt) directly in representation space: z=μθ(s)+σϵ,ϵ∼N(0,I) z=\mu_\theta(s)+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,I) z=μθ​(s)+σϵ,ϵ∼N(0,I) Sampling zt = μt + σt ⊙ ε yields a clean, closed-form analytical Gaussian log-density: log⁡πθ(z∣s)=−12∑j[(zj−μθ,j(s)σj)2+2log⁡σj+log⁡(2π)] \log \pi_\theta(z\mid s)=-\frac{1}{2}\sum_j\left[\left(\frac{z_j-\mu_{\theta,j}(s)}{\sigma_j}\right)^2+2\log\sigma_j+\log(2\pi)\right] logπθ​(z∣s)=−21​j∑​[(σj​zj​−μθ,j​(s)​)2+2logσj​+log(2π)] 4. Systems Engineering: The Two-Pass Gradient Replay In our research platform (gft-studio), we implemented this progression step-by-step: L0 (Deterministic Loop, Answer-Only GRPO): A simple residual ThoughtProjector feeds the last hidden state back: et+1 = et + Projector(ht). The model takes 8 deep thought steps silently. GRPO gradients only optimize the subsequent discrete action tokens. L1 (Dynamic Thought Budgets): The model learns boundary tokens allowing it to decide dynamically whether a problem needs 2 latent cycles or 12. L2b (Gaussian Latent Policy): The projector predicts Gaussian parameters, allowing full policy gradient backpropagation through the internal thought cycles. The Engineering Wall: Two-Pass Gradient Replay Here is a practical systems trap that cost us days of debugging: gradient checkpointing deletes KV caches inside checkpointed transformer layers. If you try to extend an existing cached sequence during backward passes in PyTorch, you get violent tensor shape mismatch errors between cached keys and attention masks. We solved this with a Two-Pass Replay Architecture: Pass 1 (Thought Generation in Eval Mode): Recompute the exact continuous thought vectors without gradients using an active KV cache (mirroring rollout inference). Pass 2 (Unified Embedding Forward): Stitch the tensors into a single unified sequence [prompt_embeds, thought_embeds, answer_embeds] and run a standard forward pass with gradient checkpointing, computing exact loss and gradients across all answer tokens in one shot. 5. Benchmarks: Discrete CoT vs. Latent Thinking on Qwen3.6-27B We pitted discrete Chain-of-Thought against continuous latent thinking in a controlled 4-arm study on a complex multi-hop-routing-gym using Qwen3.6-27B with QLoRA on NVIDIA H100 and H200 clusters: Experimental Arm Rollout Engine & Silicon Completed Steps Avg Generation Time / Step Avg Total Step Time Avg Emitted Tokens 1. Discrete CoT (Visible ) Hugging Face / 1x H100 5 / 50 (Aborted: Too Slow) 1,878.4 s (~31.3 min) 1,932.7 s ~4,131 tokens 2. No-Think Baseline (Direct Action) Hugging Face / 1x H100 50 / 50 (SUCCESS) 690.6 s (~11.5 min) 722.8 s ~680 tokens 3. Discrete CoT (Visible ) vLLM + Graphs / 1x H100 50 / 50 (Parity Failed) 612.4 s (~10.2 min) 660.7 s ~4,050 tokens 4. Latent Thinking (L2b Continuous Loop) vLLM + Graphs / 1x H200 50 / 50 (SUCCESS; Parity Passed) 132.7 s (~2.2 min) 171.2 s ~688 tokens What the Data Actually Tells Us The 14x Speedup Dissected: In our initial runs, the Latent L2b arm finished rollouts in 132.7 seconds compared to 1,878.4 seconds for Hugging Face Discrete CoT—a staggering 14.2x speedup. To be completely honest, that jump comes from two combined factors: moving from H100 to H200 memory bandwidth (4.8 TB/s) and compressing sequence length from ~4,131 chatty tokens down to ~688 action tokens. Continuous Thoughts Consume Zero Token Budget: This is the true superpower. While the discrete CoT arm repeatedly truncated its second reasoning cycle and forfeited its reward, the Latent arm executed 8 deep recurrent thought cycles without consuming a single token from the 1,024-token sequence ceiling. Total Training Wallclock: Over the full 50-step run, Latent L2b finished all training in 2.38 hours of total cluster time, compared to a projected 26+ hours for discrete CoT. 6. The Road Ahead: Pure Latent Cognition Teaching language models to reason by making them write endless stream-of-consciousness essays was a brilliant stepping stone. But as agents tackle multi-step tools, deep code refactors, and complex simulations, forced textualization becomes an unacceptable bottleneck. The future of agent post-training isn’t models that talk faster. It is models that think silently, deeply, and continuously in latent space—and only speak when they have something to say. Originally published at g-ftech.com.