colibri v1.11.0 发布：新增 DeepSeek V4.1 Flash CPU 流式推理引擎

Official release notes (https://github.com/JustVugg/colibri/releases/tag/v1.11.0) 56 pull requests since v1.10.2. A ninth model family, five real bugs closed across four engines, and the two platforms the C tests never built on now building them in CI. ### A ninth engine: DeepSeek V4.1 Flash - **#1453**: DeepSeek V4.1 Flash (552B, 510 GB on disk) runs on a CPU box streaming experts from an SSD, with no conversion: the released checkpoint is read natively, fp8 dense with 32x32 ue8m0 tiles and fp4 experts whose layout is byte-identical to the mxfp4 the Kimi K3 engine already reads. Everything in the architecture that is not V4 is in: Engram (two n-gram memories of 384M rows, 203 GB, that never enter RAM), the DSA indexer with its two-level candidate source, hyper-connections, the compressor, the 32-layer vision tower with its aligner, DSpark speculative decoding (3 stages, blocks of 5, verified in one batched forward with a rollback for the rejected rows) and tool calling in the checkpoint's own DSML format. - Held token-exact in CI against a torch-only CPU reference (`c/tools/dsv41_ref.py`, written because the vendor's own forward needs tilelang GPU kernels), at three cache capacities, on a short and a 40-token prompt, under all three speculative modes; the vision tower is matched to 5e-06. The CI also asserts that the vendor's index-key republication policy and the tidy reading disagree, so the shipped default cannot be "corrected" by accident (`V41_INDEX_OWNER`). - Measured on the released checkpoint, cold, caches dropped before every run, same prompt and seed: a turn went from 78.7 s to 25.1 s during the work (0.305 to 0.957 tok/s) through batched expert reads (`V41_READ_DEPTH`, default 8 is the measured knee), attention matrices read once per block of positions instead of once per token, and an expert-major MoE. Every step is bit-exact against what it replaced. A five-turn chat session runs at 1.14 to 1.58 tok/s; an `image_url` part goes through the gateway into the vision tower on the same engine. - Two ways of hiding the expert reads behind the matmuls were built, measured worse, and removed; both are written up in `docs/deepseek-v41.md` with the numbers, so the next attempt starts from them. - `coli chat`, `coli serve` and `coli web` work; `coli run` is deliberately unwired for this family, as for qwen36 and qwen38. Per-turn accounting is behind `V41_STATS`, off by default, so the chat stays clean. ### Fixed - **#1390** (@crichalchemist): the qwen36 GPU tier's `qt_fill_wait` returned when the upload queue was empty, but the uploader frees the ring slot at dequeue, before the backend copies anything, so the warmstart freed the RAM int8 copy of experts still in flight. An in-flight count that the uploader decrements only once the slot is resident (#1360). - **#1434** (@njloof): `fmt=0` is raw f32 and has no scale array, but the Metal sizing helper returned a per-row scale size for it, so `coli_metal_matmul` wrapped a NULL pointer and segfaulted inside `newBufferWithBytes`, or silently produced all-zero output when the size happened to be a page multiple. The kernel no longer applies a scale for `fmt=0`, and a fail-closed guard sized off the same helper falls back to the CPU path for any format that needs scales and got none. - **#1389** (@bherald): GLM-5.3 teardown leaked the lazily allocated dashboard HITS table. - **#1461**: `coli cluster worker` could never start without an explicit `--cap`: the default is `None`, and `str(None)` reached the engine's argument check as the literal `"None"` (#1452). The worker now resolves the cap the way every other launcher does. - **#1460**: olmoe's serve-mode `PROF` line published five literal zeros, so `/profile` reported `expert_disk_s = 0.0` on turns where the expert reads were the workload. The disk phase is measured now; the other four fields stay zero because they are unmeasured, not because a guess would look better (#1449, half of it). - **#1445**: olmoe's `--ram` was inert and "no `--cap`" meant a constant eight slots per layer regardless of memory; the cap is derived from the RAM budget (#1443). On the reporter's box a 2,291-token prompt went from 675 s to 353 s with no flags (#1442). - **#1377**: available RAM is measured on every platform (glm53 read `/proc/meminfo` unconditionally, so on Windows and macOS it read 0 and the expert budget collapsed to 1 GB, which is how `coli tune` went OOM in #1375), and the Windows commit limit is respected. - **#1381**: the context ceiling is announced, a request that cannot be honoured is refused up front, and the number reported is the right one (#1376). - **#1393**: GLM-5.3 replies always start inside the think block (#1278). - **#1414** (@dajiaohuang): the GLM-5.3 serve loop honours `CANCEL` while the turn is still running (#1332). - **#1423** (@kreuzzelg): the lazy HITS table is built privately and published under a lock in qwen36, kimi_k3, glm53 and qwen38; the parallel warmstart segfaulted on first touch about one run in twelve (#1422). - **#1404** (@Petsku01): the qwen36 tier offers int8 experts on the decode path (#1391). **#1388** (@crichalchemist): `QT_MAX_ROWS` replaces seven literal 32s that had to agree, and the int8 copy is freed by ownership rather than by format. - **#1421** (@bherald): `coli` no longer overwrites an explicit `CUDA_EXPERT_GB` when it decides to place the dense trunk on the card. - **#1244** (@Unknown-Findout): the CUDA expert tier is charged real VRAM, not logical bytes (#687). **#1394**: the VRAM prefix is priced per row, not at the container's widest (#1351). **#1410**, **#1411**: on a single GPU the VRAM prefix is sized from the VRAM budget and the measured headroom, not from the RAM pin plan capped by the LRU reserve (#1409, #1405). - **#1447** (@trigger2k20): `coli mirror verify` accepts a complete mirror copied by other means, with no receipt, and fails closed on a short or corrupt shard. - **#1456** (@trigger2k20): the planner reports `memory.unified` from the host, not from whether the selected engine has a placement-capable GPU, so a CPU-only engine on Apple Silicon no longer reads as non-unified; and `coli tune` stops suggesting `DRAFT`/`PIPE`/`PIN`/`NUMA` to glm53, which ignores them. - **#1396** (@dmoraesrs): the web reasoning selector drops "Medium" for GLM 5.3, which the engine renders identically to "High". - **#1428**, **#1435** (@iiEliJas), **#1433** and **#1432** (@texasich), **#1440**, **#1459**, **#1458** (@trigger2k20), **#1463**: `setenv` visible to `getenv` in-process on Windows, `malloc_trim` guarded to glibc so the engines build on musl, a missing bench recipe restored, the per-test `_putenv_s` helpers dropped, tests including `compat.h` directly, and Clang warnings cleaned up. ### Added - **#1462** (@trigger2k20): an opt-in Metal path for GLM-5.3-Flash's routed experts on Apple Silicon (`COLI_METAL=1`): gate, up, clamped SwiGLU, down and the route-weighted scatter on the GPU, CPU path unchanged when Metal is off. Validated against the CPU implementation to 2.8e-09 and measured on an M4 Max: 0.469 to 0.621 tok/s at 32 tokens, up to 2.15 tok/s with the expert cache tuned. Also fixes `coli run --cap N` being silently ignored on the glm53 one-shot path. - **#1457** (@trigger2k20): GLM-5.3-Flash feeds the shared routing telemetry and writes `.coli_usage`, so usage-driven placement and partial-mirror planning have data for it. - **#1454** (@yuripourre): the qwen36 GPU tier compiles with `HIP=1`. - **#1399** (@SebaWag): `TRUNK_RESIDENT_LAYERS=N` streams the dense trunk through mmap (#826). - **#1361**, **#1374** (@kreuzzelg): the qwen36 dense trunk places itself (`COLI_PLACE=auto`, priced in bytes saved per byte of VRAM), fp8 streaming mode, VRAM accounting at allocator granularity, thread affinity, `COLI_GPU`. - **#1383**, **#1385**, **#1386**, **#1387**: Brain and Profile tabs (EMAP, HITS, PROF) on every engine. **#1382** (@dmoraesrs): a reasoning depth selector in the web UI (#1311). - **#1407**, **#1408**, **#1406**: CI builds the portable Windows CUDA DLL and publishes it as an artifact; the Makefile refuses the 32-bit `cl.exe` before nvcc runs (#1405). - **#1436**: CI builds against musl (Alpine) and compiles the environment tests on Windows, the two holes that let #1430 and #1420 ship. - **#1360** (@kreuzzelg): tier invariants on the fake backend, and the reservation leak they found. **#1419**, **#1415**, **#1418** (@dajiaohuang): the V4 and fp8 test harnesses build on Windows. - **#1439** (@bherald): the planner tests no longer materialise four 3 GB zero-filled payloads. **#1358** (@monotophic): a raw-evidence adapter for the scoring corpus. **#1238** (@ZacharyZcR): reproducible performance records are validated. ### Docs - **#1446**: the qwen36 `--ram` section said the engine does not stream at all. **#1392**: Azure GLM-5.2 benchmark rows (#1379, #1380, #1384), the rANS cross-reference (#1273), and the int8 probe pinned in CI (#1331). - **#1427** (@ZH1995): README.zh-CN formatting. **#1438** (@iiEliJas): the `malloc_trim` comment. Repository README (current repository documentation, not release changes) Website · Discord · English · 简体中文 · 繁體中文 · Italiano **Tiny engine, immense model.** Run **frontier MoE models — 744B to 2.8T parameters** — on consumer and heterogeneous hardware, in pure C with zero engine dependencies, by treating storage, RAM, and VRAM as a single inference hierarchy (AI memory multitiering). Nine families run today: **GLM-5.2/5.3** (744B), **GLM-5.3-Flash** (321B, with vision), **Inkling** (975B), **Kimi K3** (2.8T), **DeepSeek V4 Flash** (284B), **DeepSeek V4.1 Flash** (552B, with vision), **Qwen3.8-Flash-Next** (125B + 51B n-gram), **Qwen3.6** (35B-A3B) and **OLMoE** (7B) — one C file each, the same `coli chat` / `coli serve` / `coli web` front end. [Full roster ↓](#other-supported-models) > **Colibrì is an inference engine you can run today, and an open research > platform.** Its primary goal is to pursue inference-side performance across > the entire software/hardware boundary — model formats, memory hierarchy, > storage I/O, placement, scheduling, kernels, speculation, and CPU/GPU > overlap — so large models depend less on scarce hardware and cost less to run. Colibrì treats VRAM, RAM, and storage as a single multitier hierarchy, and it is deliberately a place to test aggressive systems ideas — so there is **no SLA on speed, and a hard guarantee on semantics**: experiments must earn their place through reproducible end-to-end measurements, and the default policy **never silently changes model precision or router semantics**. Insufficient fast memory may reduce speed; it must not quietly redefine the model. ``` $ ./coli chat 🐦 colibri v1.11.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU ✓ ready in 32s · resident 9.9 GB › ciao! ◆ Ciao! 😊 Come posso aiutarti oggi? ``` ## See it running The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 — full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown, the VRAM/RAM/disk tier bar and the live mini-brain in the corner. The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity. The Atlas page: the measured expert atlas as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin. ## The research mission With Colibrì, private frontier model access is not limited by availability of hyperscaler-class hardware. With its multitiering features Colibrì **removes proprietary hardware dependencies aggressively optimizing functional inference engine pipelines**. Our operational mission includes changing how weights are represented and moved, deciding what lives in VRAM, RAM, or storage, overlapping heterogeneous compute, reducing launch and synchronization overhead, exploiting sparsity and reuse, and testing new decoding algorithms. Nothing is protected merely because it is conventional; nothing is adopted merely because a microbenchmark looks fast. The deciding result is end-to-end inference on real machines, with correctness and quality measured alongside throughput, latency, memory, and cost. The practical consequence is **accessibility**: run a 744B-parameter model on hardware you already own, watch every expert fire in real time, and change the code that does it. Not renting intelligence behind an API — *holding* it: probing it, measuring it, improving it. The engine is deliberately small enough that the next useful optimization can come from anyone willing to measure it. ## Core techniques and measured findings - **One hierarchy, not limited by tier capacity.** VRAM, RAM, and NVMe are placement tiers for the same weights; limited fast memory changes speed, not model semantics. - **A JIT for weights.** Measured routing heat drives a per-layer LRU, a learned pinned hot-store, and one-layer-ahead prefetch instead of loading every expert. It wins on repeatable workloads; history can overfit, and lookahead can lose on some hosts, so both remain measurable policies rather than promises. - **I/O is part of the engine.** Batched expert unions, overlapped reads and compute, `O_DIRECT`, and weighted dual-SSD striping attack the streaming path rather than pretending storage latency is free. `O_DIRECT` is drive-dependent, and dual-SSD still needs broader end-to-end community A/Bs. - **Heterogeneous execution.** CPU, CUDA, Metal, NUMA memory, and partial or full expert residency share one runtime and can be combined according to the machine; the profitable combination depends on compute, bandwidth, residency, and workload. - **Compressed state without a different model.** Token-exact forward validation, 57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep optimization tied to correctness. These are memory, latency, and correctness properties — not a blanket throughput claim. - **Speculation that must earn its keep.** Native MTP and grammar-forced drafts are measured end to end and can be disabled when acceptance does not repay verification. ## Open hypotheses, experiments, and how to help Colibrì treats an optimization as a hypothesis until a controlled end-to-end A/B shows otherwise. These are the main questions now: | hypothesis | evidence so far | experiment still needed | |---|---|---| | Routing history can place experts better than plain LRU | learned pins improve repeated workloads, but can overfit a prompt | held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads | | Multiple SSDs can turn independent bandwidth into decode speed | weighted mirror/split routing is implemented and validated; the bandwidth model is sound | cold-cache one-drive vs two-drive GLM-5.2 runs on real, independent controllers | | A hardware-aware planner can approach each machine's best configuration automatically | RAM/VRAM budgets and several backends are detected today | compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems | | Lossless or quality-bounded representations can reduce weight movement enough to matter | format and quantization ablations exist, with correctness/quality gates | reproduce quality, bytes moved, latency, and cost per useful token together — not compression ratio alone | | Routing-aware speculation can pay before near-full residency | MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit | map the break-even surface across acceptance, expert hit rate, batch union, and draft depth | | CPU/GPU overlap can hide transfer and synchronization rather than merely move the bottleneck | CUDA and Metal wins exist, but fast CPUs and low residency can erase them | per-stage profiles and one-variable A/Bs across PCIe, unified-memory, and full-resident machines | Want to help? Pick one row and publish the negative results too. Record the hardware, commit, model/container, exact command, prompt, cache state, throughput, TTFT, expert hit rate, bytes read, and quality check; change one variable, repeat the run, and attach raw logs. Start with [CONTRIBUTING.md](CONTRIBUTING.md), compare against [the benchmark protocol](docs/benchmarks.md), then [open an experiment issue](https://github.com/JustVugg/colibri/issues/new). A well-controlled failure is more valuable here than an unexplained fast number. ## The idea A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts): So the model doesn't need to *fit* in fast memory — it needs to be **placed**: - the **dense part** (attention, shared experts, embeddings — ~17B params) stays **resident in RAM at int4** (~9.9 GB); - the **19,456 routed experts** (75 MoE layers × 256 + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier. Think of the core algorithm as **a JIT, but for weights**. A compiler JIT never compiles the whole program — it watches what actually runs and compiles the hot paths, just in time. colibrì makes the same bet about a 744B parameter space: parameters are not resident state to be held, they are **data to be staged** across a heterogeneous storage hierarchy (VRAM / RAM / NVMe), exactly when the router proves they are needed. Measured routing heat decides which experts earn which tier, the router runs a layer ahead so prefetch hides the staging latency, and — like a JIT — the engine learns your workload: the more you run, the hotter the right experts get. It works because routing has measurable structure (see the [expert atlas](https://github.com/JustVugg/colibri/issues/175)) — and structure is cacheable. The engine is a single C file (`c/colibri.c`) plus small headers. No BLAS, no Python at runtime, no GPU required. ### Local cluster mode The coordinator keeps token generation, routing, and KV state local while disk-backed expert workers execute routed FFNs on other Macs. A layer's routed batch-union is sent as one persistent TCP request, so a token does not incur one round trip per expert. Start the optional registration service: ```bash ./coli cluster coordinator --host 0.0.0.0 --port 8765 ``` On each worker, with the same converted model available locally: ```bash ./coli cluster worker --model /nvme/glm52_i4 --port 9100 \ --coordinator http://COORDINATOR:8765 --advertise-host WORKER_IP ``` Run the coordinator with discovery, or provide `--cluster-workers HOST:PORT,...` for a static setup: ```bash ./coli serve --model /nvme/glm52_i4 \ --cluster-coordinator http://127.0.0.1:8765 ``` The transport is disabled unless workers are configured, so the existing single-machine path remains unchanged. Dense-layer sharding and browser/WebGPU workers are separate follow-up seams. ## How it works ### The per-token path Every layer of every token walks the same five steps. The design goal is that **placement only ever decides speed** — the router's decisions and the weights' precision are the same whether an expert answered from VRAM or from disk. ### One memory hierarchy instead of one memory requirement ### Dual-SSD: two copies of the model, twice the read bandwidth Decode is disk-bound on most machines, and expert reads are read-only — so if you have a **second SSD**, put a full copy of the model on it and let the engine stream from both drives at once: ```bash COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4 ./coli chat COLI_DISK_WEIGHTS=9,3 ... # optional: primary,mirror bandwidth ratio (else measured at startup) ``` Each expert is routed to one drive by a deterministic hash, weighted by the two drives' measured (or declared) bandwidth, so readahead/PILOT prefetch and the demand read always hit the same drive and nothing is cached twice. The aggregate bandwidth is the sum of both drives — a 9 GB/s + 3 GB/s pair reads experts ~33% faster than the fast drive alone, and the OMP-parallel pin/warmup load streams from both. Details worth knowing: - the mirror is **validated at startup** (per-file size + safetensors header must be byte-identical to the primary); divergent or missing files silently stay on the primary, so a **partial mirror is fine** — a smaller second SSD holding only some shards still helps; - the mirror is **never written**: `.coli_usage`, `.coli_kv` and all sidecars stay on the primary; - a read error on the mirror falls back to the primary (one warning, no crash), so unplugging the second drive mid-run degrades instead of killing the server; - routing never changes tokens — both copies are byte-identical, and the per-run `MIRROR:` stats line shows GB served per drive. The same engine spans the whole range: on a 25 GB laptop everything streams from disk (slow but correct); on a large host the entire expert set becomes resident (`CUDA_EXPERT_GB=auto PIN_GB=all`) and disk drops out of the decode path entirely. Between the tiers sits a **learning cache**: the engine records which experts *your* workload routes to (`.coli_usage`, updated every turn) and pins the hottest ones automatically — colibrì literally gets faster the more you use it. On multi-socket hosts, `COLI_NUMA=1` interleaves the resident weights across memory controllers ([#82](https://github.com/JustVugg/colibri/issues/82)). For a second drive that cannot hold the whole model, Colibri can rank a partial mirror from the expert history it already learns. Run a few representative prompts first so `.coli_usage` reflects the workload, then plan, stage, and verify the mirror: ```bash ./c/coli mirror plan --model /fast/glm52_i4 --mirror /second/glm52_i4 \ --budget-gib 200 --reserve-gib 20 ./c/coli mirror stage --model /fast/glm52_i4 --mirror /second/glm52_i4 \ --budget-gib 200 --reserve-gib 20 ./c/coli mirror verify --model /fast/glm52_i4 --mirror /second/glm52_i4 ``` The planner reads safetensors headers directly, follows split-model directories from `COLI_MODEL_DIRS`, and prioritizes shards that can serve the hottest routed experts. Staging never changes the primary model: it copies through temporary files, preserves the requested free-space reserve, verifies every shard with SHA-256, never deletes an existing mirror shard, and atomically publishes a receipt only after the selected mirror is ready. ### Never wait for the disk twice Misses are expensive, so the engine spends most of its cleverness avoiding and overlapping them: each expert's three matrices are stored adjacent and read in one `pread`; a bounded async I/O pool (`PIPE=1`, default) loads missing experts while resident ones compute; batched positions read each unique expert once (**batch-union**); and a router-lookahead thread (`PILOT=1`) prefetches the next layer's experts — routing is measurably **71.6% predictable one layer ahead**. On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual stream on-device across layers so the CPU expert loop runs uninterrupted; on Apple Silicon an experimental [Metal backend](docs/metal.md) does the batched expert math on the unified-memory GPU; and a [Vulkan backend](docs/vulkan.md) brings the expert tier, dense projections, and the MLA attention core to any GPU with a Vulkan 1.2 driver — including AMD cards via Mesa/RADV (the only backend for cards the vendor stacks no longer support, like the RX 580, and competitive with ROCm on RDNA4 — see [the benchmarking notes](docs/vulkan.md)). > **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is > often a large win on drives with DRAM cache and bandwidth headroom (+34% > decode measured with `PIPE=1` on a Blackwell/Windows box; 4.25→9.69 GB/s in > iobench on a GB10) — but it is drive-dependent: QLC/DRAM-less or virtualised > disks can be neutral to negative. Try it first; keep what your hardware > rewards. ### Faithful model, compressed state The forward pass is validated against a `transformers` oracle (teacher-forcing typically 30-32/32; two tiny-oracle positions are floating-point near-ties and toolchain-dependent). MLA attention stores a compressed KV state — 576 floats/token instead of 32,768 (**57× smaller**) — and persists it across restarts (`.coli_kv`): conversations reopen warm with zero re-prefill, byte-identical to an uninterrupted session. DSA sparse attention (GLM-5.2's lightning indexer) is implemented faithfully and validated by forcing full-key selection to reproduce dense attention exactly. ### Speculative decoding, honestly GLM-5.2's native MTP head drafts tokens that the main model verifies in one batched forward — 2.2–2.8 tokens/forward when it pays. Two hard-won rules ship as defaults: the MTP head must be **int8** (int4 heads collapse to 0–4% acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)), and draft and verify must compute **the same function** — `SPEC_PIN=1` pins both to one kernel family ([#163](https://github.com/JustVugg/colibri/issues/163) is the full forensic story). Grammar-forced drafts ([`GRAMMAR=file.gbnf`](docs/grammar-draft.md)) add ~free acceptance on constrained JSON output. Whether speculation is a net win depends on your cache temperature — measure, and use `DRAFT=0` when it doesn't pay. ## What it achieves Same engine, same int4 container — the hardware only changes where the experts live. Highlights from the [full benchmark tables](docs/benchmarks.md): - **6× RTX 5090, full residency:** 5.8–6.8 tok/s decode, TTFT ~13 s ([experiment log](docs/experiments/glm52-6x5090-2026-07-12.md)); - **128 GB CPU-only desktop:** ~1.8 tok/s warm ([#200](https://github.com/JustVugg/colibri/issues/200)); - **single RTX 5070 Ti laptop-class box:** 1.07 tok/s via the GPU-resident pipeline ([#273](https://github.com/JustVugg/colibri/issues/273)); - **25 GB dev box:** 0.05–0.1 tok/s cold — the proven floor where this project started, and still the honest baseline. Quality is measured, not assumed: the int4 container's quantization cost and the scale-granularity/rotation ablations live in [docs/benchmarks.md](docs/benchmarks.md#quality-benchmark) and [#108](https://github.com/JustVugg/colibri/issues/108)/[#81](https://github.com/JustVugg/colibri/issues/81). ## Get started You need two things: **the program** (a few hundred KB) and **the model** (372 GB). Step-by-step for every platform in the [Quick Start guide](docs/quickstart.md). ### 1. Get colibri **Download a prebuilt release** — Linux, macOS and Windows, no compiler needed. Take the archive for your platform from [Releases](https://github.com/JustVugg/colibri/releases) and unpack it: ```bash mkdir colibri && tar xzf colibri-v1.8.0-linux-x86_64.tar.gz -C colibri && cd colibri python3 coli info # engine ready ✓ ``` Inside you get the engine (`colibri`, `colibri.exe` on Windows), the `coli` launcher and its Python helpers. Nothing to rename or configure — `coli` finds the engine next to itself. You only need [Python 3](https://www.python.org/downloads/) installed: the launcher and the API gateway are Python scripts, while the engine itself is pure C with zero dependencies. **Or build from source** — needs `gcc` (or clang) with OpenMP: ```bash git clone https://github.com/JustVugg/colibri && cd colibri/c ./setup.sh # checks gcc/OpenMP, builds, self-tests ``` Want `coli` on your PATH? From a checkout, `pip install -e .` registers it (the engine still lives in `c/` — an editable install from the clone, not a wheel). ### 2. Get the model A pre-converted **GLM-5.2 int4** container is on Hugging Face — use the **group-scaled (gs64)** build with the **int8 MTP head**. It is about **372 GB**, so put it on a disk with the room, ideally a fast one: **https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp** > ⚠️ Use the **gs64** container above, not the older per-row int4 mirrors > (`mateogrgic/…`, `jlnsrk/…`): those measure ~9pp worse on quality and are the > root cause of the original think-mode loops and never-terminating generations > in [#455](https://github.com/JustVugg/colibri/issues/455). The gs64 container > fixed those controlled per-row A/Bs, but it is not a general repetition or > EOS-starvation guard. The MTP head must also be **int8, not int4** > (int4 → 0% draft acceptance, [#8](https://github.com/JustVugg/colibri/issues/8)): > `ls -l /out-mtp-*` — int8 (correct) is `3527131672 / 5366238584 / 1065950496`. Or convert from the FP8 source yourself — one resumable command that never needs the full 756 GB on disk at once: ```bash ./coli convert --model /nvme/glm52_i4 # download+convert shard by shard (python, one-time) ``` #### Other supported models GLM-5.2 is the reference model, but the same streaming approach runs six more families. Each is a **sibling engine** — one C file, its own architecture, the same `coli chat` / `coli serve` / `coli web` front end (the launcher picks the binary from the model's `config.json`): > **What each one needs.** These differ a lot, and reading two of them together > has confused people into thinking the requirements contradict each other > ([#191](https://github.com/JustVugg/colibri/issues/191)). They do not — they > are different models. **None of them needs a GPU.** > > | Model | Disk for the weights | RAM | GPU | > |---|---|---|---| > | **OLMoE** | ~7 GB (int8 container) | 8 GB | not needed | > | **GLM-5.2/5.3** | ~372 GB | 16 GB min, 24 GB comfortable | not needed | > | **GLM-5.3-Flash** | ~195 GB converted | 25 GB (12 GB weights at int4 + expert cache) | not needed | > | **Inkling** | ~469 GB | 25 GB with the int4 dense container, ~120 GB without | not needed | > | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed | > | **DeepSeek V4 Flash** | ~167 GB (REAP 150B: ~85 GB) | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster | > | **Qwen3.8-Flash-Next** | ~185.5 GB (official FP8 checkpoint) | 16 GB min, 24 GB comfortable at the default context | not supported; CPU only | > | **Qwen3.6-35B-A3B** | ~20 GB (int4-gs64 container) | 24 GB (needs full RAM residency) | optional; the CUDA VRAM expert tier measured **1.44 -> 10.05 tok/s (7.0x)** on two 8 GB cards, output bit-identical to CPU | > > A GPU only ever makes it faster. Speed is set by your disk, because the experts > are streamed from it — expect a fraction of a token per second on a slow drive > and a few per second on a fast one with the cache warm. | Family | Total / active | Weights | Build | Docs | |---|---|---|---|---| | **GLM-5.2/5.3** | 744B / 40B | [`mastouri/…-int4-g64-with-int8-mtp`](https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp) (372 GB) | `make -C c glm` | this page | | **Inkling** (Thinking Machines) | 975B / 41B | [`nbeerbower/Inkling-colibri-int4`]