Strata v0.1.31：在 64GB 内存 PC 上实验性运行 Unsloth 4-bit 大模型，AMD 解码提速最高 15%

Official release notes (https://github.com/Niko1221/Strata/releases/tag/v0.1.31) **Unsloth's Q4 model runs on a 64 GB PC (experimental), AMD decode up to 15% faster, models load twice as fast on Windows, and a batch of fixes.** **Experimental: Unsloth UD-Q4_K_XL on a normal PC.** Strata can now run Unsloth's 4-bit Qwen3.8-Flash-Next (UD-Q4_K_XL, 111 GB in 4 files, 72 GiB of experts) on a PC whose RAM is smaller than the model: - The model's four files are read as they are: a missing file is an error, a layer split across files is fine. - Its expert formats (Q4_K, Q5_K, Q5_1, Q8_0) run natively on the GPU and the CPU; the GPU dequantizers match llama.cpp's bit for bit, the dot products match ggml's CPU results within float rounding. - Three tiers for the experts: the most used in VRAM, the next ones in RAM (`--resident-budget-gib N`), the rare ones read straight from the model files on the SSD, prefetched one layer ahead from the router's prediction. Measured on an RTX 5070 12 GB with 64 GB RAM, a 40 GiB RAM budget: **about 7-8.5 tokens/s** for answers, with correct output. A big-RAM PC holds more in RAM and is faster. Not in setup yet: see `docs/UNSLOTH_Q4.md` for the manual import. Ported from eddoursul's fork (#245), with pieces from gopinath87607 (#255) and jagsan-cyber (#247). **Low-RAM mode without the 30-70 GB copy.** The low-RAM mode (`--mmap-experts`) now reads a native pack's experts straight from the model's GGUF files when the pack has no `experts.bin` - the same tokens and logits as with the copy. **Faster:** - Windows: models without `experts.bin` load about twice as fast (the expert reader used 4 KB reads; #230, reported by activeing123). IQ3_XXS starts in ~30 s instead of ~47 s here. - AMD RDNA4: decode +15% on the Radeon AI PRO R9700 and +5-7% on the RX 9070 XT (byte-level intrinsics from #262, ttio2tech); the live test on the R9700 went from 67 to 82 tokens/s. - The Q2_0 expert kernels (#241, gputier) and the IQ expert kernels (#242, gputier) do less work per token - bitwise identical output. - RTX 20 (Turing): long prompts ~12% faster - the prompt attention runs on tensor cores (#270, kenh0u; output on RTX 20 cards changes slightly, as for RTX 30+ before), and the hyper-connection kernel stages smaller tiles (#258, hireymage, bitwise identical). **Fixes:** - Server: a request that ended late could clear the next request's status and drop its answer (#266, tonykee). - Server: a long unbroken run of Chinese/Japanese/Korean text (or a URL) no longer takes seconds to tokenize (#268, cha0yang: 1.6 s -> 5 ms for 2,000 characters). - Tool calls cut off by the end of the answer are reported as unfinished instead of complete, and their broken arguments are not passed on (#211, #231 by alphastorm). - Windows: when a verify window stalls, the watchdog now releases the GPU's waits before it ends the engine, so the GPU is not left "lost" until a reboot (#267, 1593914054). - Linux multi-GPU: the whole expert arena is pinned again (the 8 GiB cap was meant for Windows only; it cost prompt speed 3x on a 4090 + 3060; #253, bettercallcaleb). `STRATA_ARENA_PIN_GIB=N` sets a cap. - An MCP server that fails to start always says why. - A stall report names the prompt chunk and layer it stopped at (#251). - Setup: says why the low-RAM mode uses one GPU, and `--low-ram off --yes` installs without asking (#250); a checkout installs the engine of its own version, pinned model revisions and pinned Python packages (#214, alphastorm). - A CPU without AVX2 is refused at start with a clear message instead of crashing. - The cuBLAS setup failures are named instead of reported as one line (#240, lukmanfauzie). **New, all off unless you turn them on:** - `reasoning_budget_tokens` in a request (or the config): caps the thinking, then lets the model answer (#123). - `STRATA_GR_V3=1`: the hyper-connection read in two kernels, +2-4% decode here, another summation order (#186, q8atnight). - `STRATA_ARENA_PIN_GIB=auto` (Windows): keeps the sliced pin below the GPU's shared-memory budget, for a PC where later allocations fail after pinning (#243, icegita: please try it). - `"split_skip_if_fits": true` (multi-GPU config): no split when the first card holds every expert. - `STRATA_LOGPOS`: per-position log-probabilities for quality tests (#235, enkynakamura). **AMD:** setup installs a layer split over several AMD cards (`--gpus 1,0`); the RX 7800 XT (gfx1101, #254 by jhohertz) and RX 9060 XT (gfx1200, #256 by Efeisot) are community-validated. **Also:** a "Where things are stored" section in the README (#226); a community benchmark guide with RTX 5090 results (#233, hagope) and RTX 3090 / dual 3090 results (#234, mad9home); `iq_parity` runs from a fresh checkout (#264, j-luwierski); an experimental build flag admits Volta (#236, cardonja; not in the ready-made engine). **Checked before the release:** - Every merged change was gated on its own: byte-identical to 0.1.30 with a fixed cache on all four quants on this PC (Q2_0, IQ3_XXS, IQ3_S, the Coder), including the prompt path's internal state, in nine rounds; default-settings speed A/Bs on Q2_0 and IQ3_S (equal or faster). Two changes were reworked because they cost speed here: the Windows pin cap (#243, now opt-in) and a polling wait in the verify window (#267, now the watchdog's job); #257 was taken out again (not faster on an AVX2-only CPU). - Real use at a 64K context on all five models (Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, the Coder), through the server: - a fact found in a 57,479-token document; - the follow-up turn reusing the whole prompt; - a tool call; - a cancelled long prompt followed by a new request; - a sampled answer. None of them restarted the engine or ran low on VRAM. - The Unsloth Q4 model end to end on this PC (correct answers at 24, 32 and 40 GiB RAM budgets); the new format kernels against ggml-cpu and llama.cpp's dequantizers; the split-file loader and pack tests; the low-RAM mode reading the GGUF in place gives the same tokens and logits as `experts.bin`. - The server's tests (100) and setup's tests. - Linux (WSL, RTX 5070): builds, and Q2_0 is byte-identical to 0.1.30 with a fixed cache (10/10); the default settings pass. - AMD (RX 9070 XT and R9700): the HIP build with its tests (all pass but the two that need a model fixture or an AVX-512 CPU), the live server test on each card (R9700 82 tokens/s, 9070 XT 34 tokens/s), and the two-card split. **Updating:** get the latest files (`git pull`, or download and unzip anywhere), then run `START-HERE.bat` (Linux: `./setup.sh`). Setup installs engine 0.1.31. The ready-made Strata engine for Windows (RTX 20 / 30 / 40 / 50: sm_75, sm_86, sm_89, sm_120 + PTX), CUDA 13.0. You don't need to download this yourself: `START-HERE.bat` fetches it (and NVIDIA's cuBLAS from pip), so no compiler or CUDA Toolkit is needed, only an NVIDIA driver 580 or newer. Contents: `strata.exe` (the engine), `strata-vision.exe` (the optional image encoder), `BUILD.json`. Repository README (current repository documentation, not release changes) Strata Run a 125-billion-parameter AI model on a normal gaming PC one NVIDIA card (12-24 GB) + 64 GB of RAM · Windows or Linux · one click to install A voxel pagoda garden, 1 shot prompt running on an RTX 5070 with Strata (IQ3_S, 128K context) · full video (49 s) Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** - a large, smart AI model that normally needs a server - on your own PC. It writes its answers at **60-95 tokens per second** (a token is about ¾ of a word): faster than you can read. - **Free and open source.** > **Jump to:** [How fast?](#how-fast-is-it) · [Which model?](#which-model-should-i-pick) · [Install](#install) · > [Using it](#using-it) · [Problems?](#something-went-wrong) · [How it works](#how-does-it-work) · > [All the details](docs/DETAILS.md) --- ## How fast is it? Measured on an RTX 5070 (12 GB), a Ryzen 5 7600 and 64 GB of RAM: | Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt | | --- | ---: | ---: | ---: | | **Q2_0** | 93 tokens/s | 74 tokens/s | 2,170 tokens/s | | **IQ2_XS** | 79 tokens/s | 63 tokens/s | 2,090 tokens/s | | **IQ3_XXS** | 62 tokens/s | 49 tokens/s | 1,750 tokens/s | | **IQ3_S** | 53 tokens/s | 46 tokens/s | 1,620 tokens/s | | **Coder** (IQ1_M) | 55 tokens/s | 43 tokens/s | 2,180 tokens/s | - **Writes answers** = how fast the reply appears (tokens per second). - **Reads your prompt** = how fast it takes in what you send (long documents, code, chat history), measured on a 32K-token prompt; a 4K prompt reads at 910-1,580 tokens/s. A 32K prompt takes about 15 seconds with Q2_0. A card with more VRAM is faster, because more of the model fits on the GPU: an RTX 3090 (24 GB) should do roughly 100-140 tokens per second. All measurements, long-context numbers and estimates for other cards are in the [details](docs/DETAILS.md#speed-measured). Every PC is different: `START-HERE.bat --calibrate` measures a few engine settings on yours and keeps the fastest (about 5-10 minutes; on the PC above it made the Coder 7% faster). Measured Strata on your own PC? See [Community benchmark results](docs/COMMUNITY_BENCHMARKS.md) for a report template and how to share your results in a pull request. **Two or three NVIDIA cards?** Just run `START-HERE.bat`: it lists your cards, says which ones Strata can use, and asks whether to share the model across them (recommended when two can). An install made on one card asks once at its next start. Or choose yourself: `START-HERE.bat --gpus 0,2` (both, remembered), `--gpus all`, or `--gpu 0` (one card, this start only). Each card keeps the experts of its own layers, and prompts flow through the cards in a pipeline: on an RTX 5080 + RTX 3090 prompts were read 18-20% faster than on the 5080 alone, decoding on par. Every card must be an RTX 20 series or newer with 8 GB or more. See [docs/MULTI_GPU.md](docs/MULTI_GPU.md). ## Which model should I pick? **The size** (the same model, compressed more or less): | Model | RAM+VRAM Requirements | Speed | Quality | | --- | ---: | --- | --- | | **Q2_0** | 37.6 GB | fastest | good | | **IQ2_XS** | 39.2 GB | fast | better (**recommended**) | | **IQ3_XXS** | 47.0 GB | slower | great | | **IQ3_S** | 54.8 GB | slowest | best: matches the full model on the published tests (original model only) | **Will it fit?** Shard 1 is the part of the model that gets loaded when it starts: its experts go into your **RAM**, the rest onto your graphics card (the second shard, a 29 GB lookup table, stays on the SSD). So it fits when your **RAM is at least shard 1 + about 10 GB** for Windows and your other programs. With 64 GB of RAM every size fits (IQ3_S with little else open); with 48 GB, Q2_0 and IQ2_XS. A bigger graphics card makes it faster, but it doesn't lower the RAM needed. **The version:** - **Qwen3.8-Flash-Next** - the original. - **[Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** - ISTA-DASLab's coding version: half of the experts removed, keeping the ones that code, tool use and images need (91% of the full model's SWE-bench Verified score, 99% of LiveCodeBench, by its authors). One size (IQ1_M: its experts stored like IQ3_S): shard 1 is **29.6 GB**, so it fits a PC with **32 GB of RAM**, runs 262K context on 64 GB, and reads long prompts the fastest of all. Weaker outside coding. - **[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI that thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per token, and about the same RAM as the same size of the original (no IQ3_S). Its own license applies (see its page). Not sure? Take **IQ2_XS** - or the **Coder** if you mainly write code, or have 32-48 GB of RAM. You can add another one later with `SETUP.bat` (the same as `START-HERE.bat --setup`; on Linux `./setup.sh --setup`). For **OrcaRouter's Flash-Next Uncensored IQ3_XXS**, see the [manual compatibility setup](docs/ORCA.md). It needs an explicit packing conversion and is not an installer menu option. An **AMD Radeon RX 7900 XT / XTX, RX 9070 / 9070 XT or Radeon AI PRO R9700 on Linux** works too (experimental; the RX 7800 XT / 7700 XT and RX 9060 XT were validated by their owners): `./setup.sh --backend hip`, chosen by itself on a PC with no NVIDIA card Strata can use. It installs ROCm without sudo and compiles the engine (no images yet; several cards with `--gpus`). Details: [AMD HIP](docs/AMD_HIP.md). ## Install **You need:** an NVIDIA RTX 20, 30, 40 or 50 card with 12 GB of VRAM or more (RTX 20 since 0.1.27), enough RAM for the size you pick (above; a big GPU makes up for less RAM - the [low-RAM mode](docs/DETAILS.md)), ~80 GB of free disk space (an SSD makes the first start much faster), and Windows 10/11 or Linux. The only thing you install yourself is a current **NVIDIA driver** ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). Everything else - Python, the engine, the model - is set up for you. **Windows** 1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it). 2. Double-click **`START-HERE.bat`**. 3. Answer a few questions - or just press Enter each time for the recommended choice: - **Which model and size?** The original or Swift 1.5, and Q2_0, IQ2_XS, IQ3_XXS or IQ3_S - see [above](#which-model-should-i-pick) - **How much context?** How much text it can keep in mind at once (it suggests one for your card). 384K and 512K (experimental) extend the model past its trained 262K by rope scaling - the setup turns it on itself (yarn and a covering factor; `--rope-scaling`/`--rope-scale` override) ([details](docs/DETAILS.md)) - **Images?** Whether it should also read pictures - **Experimental speed projection?** Off unless you say yes - [read what it does](docs/DETAILS.md#experimental-speed-projection-experimental-off-by-default) first Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`. > **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strata > loads 35-55 GB into your RAM and locks part of it for the graphics card. That's normal - wait, and don't close the > window. The window tells you what it is doing. **Next time**, just double-click `START-HERE.bat` again: it starts right away, nothing is downloaded twice. Close its window to stop the model. **Updating:** download the new version and unzip it anywhere (or `git pull`), then run `START-HERE.bat` in it. The model files are kept in a `Strata-data` folder next to your Strata folder, so a new copy finds them and sets itself up the same way - nothing big is downloaded again. **Linux:** run `./setup.sh` - same questions, same result. **Docker (Linux):** the same idea, in a container. 1. Host: Docker with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) and a driver **580 or newer** (CUDA 13.0). 2. Build (this compiles the engine into the image, so the container never compiles): `docker build -t strata .` `docker build -t strata --build-arg CUDA_ARCHITECTURES=89 .` builds for one card only (faster). The default covers RTX 30 (86), RTX 40 (89), RTX 50 (120) and A-series (80); a card outside that set needs a rebuild with its own arch. Add `--build-arg BUILD_VISION=0` to skip the image encoder. 3. Run (the first start downloads the ~70 GB model, then starts; later starts go straight to serving): `docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 -v strata-data:/data strata` The setup choices are env vars: `-e MODEL=IQ2_XS -e FAMILY=qwen -e CONTEXT=32768 -e VISION=no` (or `MODEL=Q2_0|IQ3_XXS|IQ3_S`, `FAMILY=swift|coder`; the defaults above are the recommended ones). `-e VISION=cpu` keeps the image encoder on the CPU. `-e KV=int8|q4_0|k8v4` picks the KV cache precision; `k8v4` is INT8 K with 4-bit V and keeps its KV in VRAM from 64K up. Only the model files, the prepared pack, the MTP layer and the install config live in the `strata-data` volume; the engine is part of the image. Switching between models already on the volume needs no setup pass: `-e MODEL=Q2_0 -e FAMILY=coder` picks that model's config. Add `-e REINSTALL=1` only to change settings for a model already set up (context, vision, KV, host, api_key, LOW_RAM), since those are recorded in its config. Strata loads 32-62 GB into RAM. `--gpus all` on a host with two usable cards takes both: the layer split is setup's recommended default ([docs/MULTI_GPU.md](docs/MULTI_GPU.md)), and a volume set up for one card switches to the pair on its first start there. Pin one card with `-e GPU=0`, or name them with `-e GPUS=0,2` and where the later card's layers start with `-e LAYER_SPLIT=18`. A memory limit needs `-e LOW_RAM=on`, which maps the model's experts from the pack instead of keeping them in RAM: setup.py measures the host's RAM, not the container's limit, so it cannot see a cap. LOW_RAM runs on one card. The server listens on `0.0.0.0:8080` by default; set `-e API_KEY=` before exposing the port to a network. The image has a `HEALTHCHECK` on `/health`, so `docker ps` shows the container healthy once the model is loaded, and `GET /v1/status` says what it is running. ## Using it The Strata app's Monitor (left) while a coding agent writes the pagoda garden from the video (right) - **In the browser:** `http://127.0.0.1:8080` - the Strata app (it opens by itself when the model starts): **Chat**, a live **Monitor** of the model and your GPU/CPU/RAM, and **About** with the settings and addresses. - **Chat in the terminal:** `.venv\Scripts\python chat.py` - **Your apps and coding agents:** add it as an "OpenAI-compatible" provider with base URL **`http://127.0.0.1:8080/v1`**, any API key and any model name. Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages`. - **Thinking:** the model thinks before it answers. Choose **off, low, medium or high** - in the chat page menu, with `/think low` in `chat.py`, or with your app's "reasoning effort" setting. Off is fastest; high is best for hard questions. - **Pictures:** in the chat page click **Picture**; in `chat.py` type `/image `; in apps just attach them. - **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key `, then open the address the server window prints; see the [details](docs/DETAILS.md#using-it). - **Experimental speed projection (off by default):** an experimental control vector that setup can turn on; it changes how the model answers - read [what it does](docs/DETAILS.md#experimental-speed-projection-experimental-off-by-default) first. **Good to know:** it answers one request at a time. The first message of a chat is read in full (about 1 minute per 30,000 tokens); after that it keeps the conversation and reads only what is new, so follow-ups start in seconds. ### Where things are stored - **Your chats: only in your browser.** The Chat tab keeps the conversation, its settings and the API key you typed in the browser's local storage (`strata.*` keys) - not on the server and not in the Strata folder. Pictures are not kept, only their names. Another browser or a private window starts empty; clearing the site's data deletes them. - **How the model starts:** `strata-.json` in the Strata folder (context, GPUs, host, API key, ...), written by setup; next to it `run-.bat` / `.sh`, the log `strata-.log` and, when you use "Use for other apps too", `strata-.shared-settings.json`. - **The model files** (`models/`, `packs/`, `mtp/`, 70-120 GB): in **`Strata-data` next to the Strata folder**, or wherever `--data-dir` put them. - **Where that data folder is:** `%APPDATA%\Strata\settings.json` on Windows, `~/.config/strata/settings.json` on Linux ([details](docs/DETAILS.md)). ## Something went wrong? **My PC froze, or got very slow, the first time Strata started.** That's normal while it starts, most of all the first time. Strata loads 35-55 GB into your RAM, locks part of it for the graphics card, and works out how much of the model fits on your GPU. The mouse can freeze for a few minutes. **Wait, and don't close the window.** The next starts are much faster. Still frozen after 10 minutes? Restart the PC, close other programs (browsers use a lot of RAM) and try again. If it keeps happening, pick a smaller size (Q2_0 or IQ2_XS). **It stopped while downloading or installing.** Run `START-HERE.bat` again. It continues where it stopped. **It says the NVIDIA driver is too old.** Update it (NVIDIA App or [nvidia.com/drivers](https://www.nvidia.com/drivers)), restart the PC, and run `START-HERE.bat` again. **It says port 8080 is already in use.** Strata is already running. Look for its window. **It's very slow and the disk light keeps blinking.** Your PC is out of free RAM. Close other programs, or pick a smaller size (Q2_0 or IQ2_XS). **An answer stopped with "the engine stopped unexpectedly".** Usually not enough RAM (on Linux the system then stops the engine). Just send your message again: Strata starts the engine by itself. If it keeps happening, close other programs or pick a smaller size. **It says the prompt exceeds the context.** The conversation is longer than the context you chose. Start a new chat, or run `SETUP.bat` and pick more context. **Still stuck?** Look in the [full troubleshooting table](docs/DETAILS.md#troubleshooting), or open an issue and attach `strata-.log` from the Strata folder. ## How does it work? Models like this one normally run on servers with hundreds of gigabytes of graphics memory. Your graphics card has 12-24 GB. Strata makes it fit by **sharing the work across your whole PC** - the same idea as a kitchen, where the things you use all the time stay on the counter and the rest waits in the pantry. - **The model is a team of 24,576 small specialists ("experts"),** and each word it writes needs only 10 of them. So it doesn't have to have all of them on the graphics card at once. - **Your graphics card** does the part of the work needed for every word, and keeps the few thousand experts that are asked most often. It keeps learning which ones those are while you use it. - **Your RAM** holds every expert. When a word needs one the card doesn't have, **your processor** works on it - at the same time as the graphics card, so neither waits for the other. - **Your SSD** holds a big lookup table; the model only reads a few small rows of it per word. - **Guess, then check.** A small, fast helper built into the model guesses the next few words, and the big model checks all the guesses in one go. It keeps the ones it agrees with and writes the next word itself - so one step often produces several words. The helper only guesses - the big model decides every word - so you get the same quality answer, 1.6-1.8x sooner. - **Long texts are read in big pieces** (up to 8,192 tokens - pieces of words - at a time), which is why a long document or code base is read at over 1,000 tokens per second. Want the full picture? The [details](docs/DETAILS.md#how-it-works) explain every part and its numbers, and the [paper](docs/paper/Strata-Paper.pdf) tells the whole story, with the measurements behind it. ## Credits - Model: [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; compressed versions by [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF); [Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) by UkisAI. Their licenses apply to the model files. - Built with parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT). Ideas from [Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and [HyperQwen](https://github.com/syv-ai/HyperQwen). More in the [details](docs/DETAILS.md#credits-and-licenses). ## License Strata is open source under the [MIT License](LICENSE). A few parts carry their own licenses: `third_party/ggml` (MIT, llama.cpp / ggml), the web app's font (SIL Open Font License 1.1) and the experimental speed projection's vector in `data/experimental-speed-projection` (Qwen Community License 1.0, from the model's activations). The models are not part of this repository; each model's own license applies to its files.