Strata v0.1.38 发布：提示词加速、--kv q4_0 张量核心支持与无 API key 安全加固

Official release notes (https://github.com/Niko1221/Strata/releases/tag/v0.1.38) **Faster prompts, `--kv q4_0` prompts on tensor cores, a 6 GB card that starts, and a large batch of community PRs.** **Security: the server without an API key (DNS rebinding, cross-site requests).** Without `api_key`, the server now answers only requests addressed to a name it knows (`localhost`, an IP address, the address it listens on and, when it listens beyond this PC, this PC's name), and a browser page from another site can no longer send it requests (`403`). `/unload` and `/load` take only JSON: `curl -X POST -H "Content-Type: application/json" localhost:8080/unload`. Clients without an `Origin` header (curl, the OpenAI and Anthropic SDKs, other servers) are not affected, and with an API key the key decides. **Behind a reverse proxy, a tunnel or Docker under another name, without a key:** add the name, `"allowed_hosts": ["strata.example.com"]` in `strata-.json` (or `STRATA_ALLOWED_HOSTS`), or set an API key. Details: docs/DETAILS.md. **Faster prompts:** - The prompt path gathers a group of experts in one launch with one wait (#372), reads the first chunk's n-gram rows beside layer 0 instead of before it (#374), and runs the DeltaNet recurrence three heads per thread (#413). The answers are the same bits as before. - Measured on an RTX 5070 12 GB (3 alternating pairs each against 0.1.37): Q2_0 prompts +10% at 4K, +2.7% at 32K, +3.0% at 128K; IQ2_XS +1.9% at 32K and +6.4% at 128K (4K: within the run-to-run spread). - Decode time per verify round is the same or a little shorter. Since #463 a greedy answer no longer depends on when the adaptive tier's expert copies land, so on some long prompts the drafts are accepted a little more or less often than before (Q2_0 at 128K: 70.9 instead of 75.1 tokens/s on the measured prompt, IQ2_XS at 32K: 73.0 instead of 69.7). **`--kv q4_0` (#452):** prompts read their Q4_0 K/V on tensor cores (RTX 30 and newer; RTX 20 keeps the old kernel for now): a 32K prompt on Q2_0 in 14.3 s instead of 18.8 s. Against an FP16 K/V reference, over 2,000 teacher-forced positions after the prompt: at 32K, KL 0.026 and top-1 95.1% (0.031 / 93.6% with the old kernel); at 8K, KL 0.079 and top-1 89.3% (0.077 / 90.0%). The default `--kv int8` is unchanged. **A 6 GB card starts (#496):** when the default VRAM reserve leaves the expert cache too little room, the engine lowers the reserve (down to 300 MiB) until the cache fits, and warns if the card ends nearly full. If even that is not enough, the start stops with how many MiB are short and what frees them; setup gives the same tip on cards under 8 GB. **More formats and CPUs:** - **Q5_0 experts on the GPU** (#473): community Q4_K_M GGUFs that put Q5_0 on the down projections now load. Q5_0 n-gram tables also read with `--ple-io direct`. - **IQ4_XS on AVX-2-only CPUs** (#415): the multi-token kernel the other i-quants already had. AVX-512 CPUs are unchanged. **Loading and memory:** - **Windows, short on RAM** (#357 #362): the experts are read past the file cache when it cannot keep them anyway, at start and in the RAM-budget tier (faster starts and decode on 32-64 GB PCs, the same tokens). - **Linux** (#488): `STRATA_NO_LARGEPAGES=1` works there too, and a hugetlb pool that is too small is named. **AMD:** hipBLASLt tables for gfx1201 and gfx1200 with hipBLASLt 1.2.2 (system ROCm 7.2.x; ~1.5-2x prompt speed there, #386 #387), and the draft layer's prompt pass runs per group on HIP, which fixes a prompt hang on gfx1201 (#382). **New options (off by default; the default output is unchanged):** - `--peer-device N` (#531): a second GPU as an extra expert cache (needs P2P, e.g. NVLink); see docs/SECOND_GPU.md. - `--adapt-decay F` (#407): how fast the adaptive tier forgets (0.7, as before). - `STRATA_GR_DOWN_MAX4=1` (#443): a smaller decode kernel variant for short windows (the same bits). **Also:** a native pack with `STRATA_PF_FUSED=1` whose fused kernels don't cover every layer (Unsloth UD-IQ4_XS) no longer gives garbage or hangs on long prompts; the verify window's PCIe call launches less (#363), the adaptive tier waits for its expert copies so a decode no longer depends on their timing (#463), a stager race with unpinned blobs is closed (#385), RTX 20 cards pick a faster top-k at long contexts (#512), and `--dump-logits` no longer writes a header without rows (#463). **Fixes from your reports:** - **AMD on a Linux desktop** (#560 #516): when the desktop or apps crash once the model is loaded, the expert cache has filled the card the desktop needs. Setup and the server now recommend `--vram-reserve-mib 3072` there (nothing is changed for you). - **A slow CUDA build** (#542): MMQ reads the GPU's limits in a way a mismatched CUDA runtime cannot shift, and the engine warns when its runtime is older than the toolkit it was built with. - **The weight arena does not fit** (#486): the error says how much VRAM was free at that moment. - **Setup and the server** (#549 #545 #530 #564): `setup --update` skips a `strata-*.json` that is no model config; the "exceeds the context" error says how to get past it; a reply that reaches its token limit while still thinking is logged with what helps; each start prints the settings it uses. - **Docs:** an OpenCode config example (#543); MODELS.md says plainly that the Coder is the 32 GB fit, weaker outside code, and recommends the full models for general use. - A parity test waits for its uploads (#548). Thanks to everyone who sent PRs for this release, and especially to sergqwer and BlueKingMuch for the prompt-path and decode work. **Checked before the release:** - **The same answers as 0.1.37 on all four quants** (Q2_0, IQ3_XXS, IQ3_S, the Coder), 10/10 each with a fixed cache, and the prompt path's internal state at 4K and 20K tokens (Q2_0, the Coder). - **Speed:** the 5-pair A/B on Q2_0 and IQ3_S: Q2_0 +4.0% / +1.9%, IQ3_S +0.2% / +0.1%, the same expert slots; the prompt/decode table above. - **Kernel parity tests** (the new DeltaNet, grouped-expert, top-k and Q4_0 attention tests included), **real use at a 57K-token prompt** (Q2_0, the Coder), **Linux** (WSL) Q2_0 identical 10/10. - **AMD on Windows:** the HIP zip builds; the AMD changes are untested on an AMD card here (we have none). - **After the report fixes were merged:** Q2_0 and IQ3_XXS still identical 10/10 and at 4K, and the Q2_0 speed A/B +0.7% / +0.6% (the new start-up checks cost nothing). - **Tests:** tools/setup 287, server 197 (the new security tests included). **Updating:** run `UPDATE.bat` (Linux: `./update.sh`). Setup installs engine 0.1.38. The ready-made Strata engines for Windows, which `START-HERE.bat` fetches by itself: - `strata-windows-x64.zip`: NVIDIA (RTX 20 / 30 / 40 / 50: sm_75, sm_86, sm_89, sm_120 + PTX), CUDA 13.0, needs an NVIDIA driver 580 or newer. Contents: `strata.exe`, `strata-vision.exe` (the optional image encoder), `BUILD.json`. - `strata-windows-x64-hip.zip`: AMD (gfx1100, gfx1101, gfx1102, gfx1200, gfx1201, gfx1030), ROCm 10.2.0a20260930 from AMD's TheRock builds, needs a current AMD driver. Contents: `strata.exe`, `strata-device.exe`, the HIP runtime next to them, `BUILD.json`, `rocm\` (the ROCm libraries and their licenses). --- Strata is free and open source. If it runs well on your PC, a coffee keeps the work on it going: Repository README (current repository documentation, not release changes) Strata Run a 125-billion-parameter AI model on your own gaming PC NVIDIA or AMD graphics card (12 GB or more) · Windows or Linux · free and open source A voxel pagoda garden, 1 shot prompt running on an RTX 5070 with Strata (IQ3_S, 128K context) · full video (49 s) Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** - a large, smart AI model that normally needs a server - on a normal PC. It chats, writes code, reads pictures and works with your apps and coding agents, and nothing leaves your PC. ## How fast is it? Measured on two ordinary gaming PCs. "Writes answers" is how fast the reply appears in a short chat; "reads your prompt" is how fast it takes in what you send (a 32K-token document, code or chat history). A token is about ¾ of a word, so 60 tokens per second is faster than you can read. NVIDIA: RTX 5070 (12 GB), Ryzen 5 7600, 64 GB RAMAMD: RX 9070 XT (16 GB), Ryzen 9 3900X, 47 GB RAM | Size | Writes answers | Reads your prompt | | --- | ---: | ---: | | **Q2_0** | 94 tokens/s | 2,650 tokens/s | | **IQ2_XS** | 79 tokens/s | 2,090 tokens/s | | **IQ3_XXS** | 62 tokens/s | 1,750 tokens/s | | **IQ3_S** | 53 tokens/s | 1,620 tokens/s | | **Coder** | 55 tokens/s | 2,180 tokens/s | | Size | Writes answers | Reads your prompt | | --- | ---: | ---: | | **Q2_0** | 60 tokens/s | 1,160 tokens/s | | **IQ2_XS** | 52 tokens/s | 1,110 tokens/s | | **Coder** | 44 tokens/s | 1,420 tokens/s | A card with more VRAM is faster: an RTX 3090 (24 GB) should write roughly 100-140 tokens per second. Long chats, other cards: [speed of each model](docs/MODELS.md#how-fast-is-each-size), [community results](docs/COMMUNITY_BENCHMARKS.md). Strata is free. If it runs well on your PC, a coffee keeps the work on it going. ## What you need | | | | --- | --- | | **Graphics card** | **NVIDIA** GeForce RTX 20, 30, 40 or 50 series, or **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 or RX 6800 / 6900 series - with **12 GB of VRAM or more** | | **RAM** | 32 GB or more - how much decides [which model](#which-model-should-i-pick) fits; 64 GB runs every size | | **Disk** | about 80 GB free, on an SSD if you can (the first start is much faster) | | **System** | Windows 10 / 11 or Linux, and a current graphics driver from NVIDIA or AMD | Everything else is installed for you. Two or three cards can share the model ([multi-GPU](docs/MULTI_GPU.md)). The full list: [docs/INSTALL.md](docs/INSTALL.md#what-you-need). ## Install ### Let your AI set it up Use an AI coding assistant (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Paste this into it: ```text Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository. ``` It checks your graphics card, RAM and disk, picks the model that fits, installs it, starts it and tells you how to connect your apps. AI tools can also install, start and stop Strata themselves through its [MCP server](docs/MCP_SERVER.md). ### Or do it yourself [Download Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it). **Windows:** double-click **`START-HERE.bat`**. **Linux:** run **`./setup.sh`** in the Strata folder. The same steps for NVIDIA and AMD: the installer finds your card and sets up the right engine for it. It asks which model, which size, how much context (how much text it keeps in mind) and whether it should read pictures - press Enter each time for the recommended answer. Then it downloads the model (~70 GB; you can stop and it continues where it left off) and starts it. Your browser opens the Strata app at `http://127.0.0.1:8080`. > **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strata > loads 35-55 GB into your RAM and locks part of it for the graphics card. That's normal - wait, and don't close the > window. The window tells you what it is doing. **Next time**, run `START-HERE.bat` (or `./setup.sh`) again: it starts right away, nothing is downloaded twice. Close its window to stop the model. `UPDATE.bat` (`./update.sh`) updates Strata without starting it. Updating, Docker, several cards, where the files go and every option: [docs/INSTALL.md](docs/INSTALL.md). ## Which model should I pick? The installer recommends one for your RAM. The same model comes in sizes that are compressed more or less: smaller is faster, larger is a bit smarter. | Your RAM | Take | Why | | --- | --- | --- | | **32 GB** | **Coder** | it fits 32 GB, and it is made for code (with a 24 GB card, Q2_0 and IQ2_XS run too) | | **48 GB** | **IQ2_XS** (or Q2_0, the fastest) | the larger sizes do not fit | | **64 GB** | **IQ2_XS** (recommended), or IQ3_XXS / IQ3_S | every size fits; IQ3_S is the best, and the slowest | | **96 GB or more** | **IQ3_S**, or Unsloth's 4-bit (experimental) | room for the largest sizes with everything else open | - **[Coder](docs/MODELS.md#coder)** - a coding version with half of the experts removed: 91% of the full model's SWE-bench Verified score (by its authors), fits 32 GB of RAM. Weaker outside code, including Chinese and other CJK text (#438): for those, take Q2_0, IQ2_XS or IQ3_S, which keep every expert. - **[Swift 1.5](docs/MODELS.md#swift-15)** - a fine-tune that thinks much shorter before it answers, so you get the answer sooner, at about the same quality. - **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental) - the closest to the full model, but most of it is read from the SSD while it answers: 7-8.5 tokens/s on a 64 GB PC. - **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs)** - a manual setup, not in the installer's menu. Sizes, downloads and what fits where: [docs/MODELS.md](docs/MODELS.md). You can add another model later with `SETUP.bat` (Linux: `./setup.sh --setup`). ## Using it The Strata app's Monitor (left) while a coding agent writes the pagoda garden from the video (right) - **In the browser:** `http://127.0.0.1:8080` - **Chat**, a live **Monitor** of the model and your GPU/CPU/RAM, and **About** with the settings and addresses. - **Your apps and coding agents:** add an "OpenAI-compatible" provider with base URL **`http://127.0.0.1:8080/v1`**, any API key and any model name. Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages` (Claude Code: `ANTHROPIC_BASE_URL=http://127.0.0.1:8080`). - **Thinking:** choose **off, low, medium or high** in the chat menu or your app's "reasoning effort". Off is fastest; high is best for hard questions. - **Pictures:** say yes to "Images?" in setup, then click **Picture** in the chat, or attach them in your app (AMD cards: on Linux through the processor, not on Windows yet). - **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key ` - always with a key. - **Good to know:** it answers one request at a time. The first message of a chat is read in full (about 1 minute per 30,000 tokens); follow-ups start in seconds. More: [where your chats are stored](docs/INSTALL.md#where-things-are-stored), [the API](docs/DETAILS.md#using-it). ## Something went wrong? - **My PC froze the first time Strata started.** Normal while it loads the model: wait, don't close the window. Still frozen after 10 minutes? Restart the PC, close other programs and try again, or pick a smaller size. - **It stopped while downloading or installing.** Run `START-HERE.bat` (or `./setup.sh`) again: it continues where it stopped. - **It's very slow and the disk light keeps blinking, or "the engine stopped unexpectedly".** Not enough free RAM: close other programs (browsers use a lot), or pick a smaller size (Q2_0 or IQ2_XS). - **It says port 8080 is already in use.** Strata is already running - look for its window. More problems and their fixes: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Still stuck? Open an [issue](https://github.com/Niko1221/Strata/issues) and attach `strata-.log` from the Strata folder. ## How does it work? Models like this one normally run on servers with hundreds of gigabytes of graphics memory. Your graphics card has 12-24 GB. Strata makes it fit by **sharing the work across your whole PC** - like a kitchen, where the things you use all the time stay on the counter and the rest waits in the pantry. - **The model is a team of 24,576 small specialists ("experts"),** and each word needs only 10 of them. - **Your graphics card** keeps the few thousand experts that are asked most often; **your RAM** holds all of them, and **your processor** works on the rest at the same time. **Your SSD** holds a big lookup table. - **Guess, then check:** a small helper guesses the next few words and the big model checks them all at once, so you get the same answer, 1.6-1.8x sooner. - **Long texts are read in big pieces** (up to 8,192 tokens at a time): over 1,000 tokens per second. The longer explanation: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Every part and its numbers: [the details](docs/DETAILS.md#how-it-works) and the [paper](docs/paper/Strata-Paper.pdf). ## Credits and license The model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team, compressed by [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5) and Unsloth; Strata is built with parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). All credits: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata is open source under the [MIT License](LICENSE); a few parts and every model carry their own licenses ([which ones](docs/HOW_IT_WORKS.md#license)). ## Support Strata Strata is free and open source. If it is useful to you, you can support its development: