Human-Agent-Society/reef v0.1.1 发布：新增 vLLM token 级捕获与权重版本标记连接器

Official release notes (https://github.com/Human-Agent-Society/reef/releases/tag/v0.1.1) ## What's Changed * [Feat] Add vLLM token-native capture and runtime kind by @ruizhang0101 in https://github.com/Human-Agent-Society/reef/pull/571 * fix(reefine): rename /evolve command to /reefine by @BobbyZhouZijian in https://github.com/Human-Agent-Society/reef/pull/589 * SAO example: batch 128, paper-shaped runs, results and limitations by @Xuan-1998 in https://github.com/Human-Agent-Society/reef/pull/563 * fix(docs): retain homepage statistics when GitHub requests fail by @BobbyZhouZijian in https://github.com/Human-Agent-Society/reef/pull/590 * fix(runtime): place the training coordinator on the trainer's node by @MikeZheng777 in https://github.com/Human-Agent-Society/reef/pull/587 * feat(records): batch import and storage-owned capacity eviction by @BobbyZhouZijian in https://github.com/Human-Agent-Society/reef/pull/588 * fix(harness): the dsh session reader lifts each event's data beside its envelope by @Benjamin-eecs in https://github.com/Human-Agent-Society/reef/pull/594 * feat(inference): add the vLLM connector that stamps tokens with their weight version by @ruizhang0101 in https://github.com/Human-Agent-Society/reef/pull/601 * docs(harness): clarify how to write a harness method by @YihongT in https://github.com/Human-Agent-Society/reef/pull/606 * fix(docs): load pinned local fonts for reliable builds by @BobbyZhouZijian in https://github.com/Human-Agent-Society/reef/pull/609 * feat(scenario): named release components, one trainer per component, composite recipes (#15 phases 1-3) by @BobbyZhouZijian in https://github.com/Human-Agent-Society/reef/pull/537 * perf(artifact): publish a carried component without re-hashing or cloning it by @BobbyZhouZijian in https://github.com/Human-Agent-Society/reef/pull/652 * refactor(train): name the scenario step scenario_step and keep rollout_id for the checkpoint index by @BobbyZhouZijian in https://github.com/Human-Agent-Society/reef/pull/653 * docs(harness): clarify the harness adapter guide by @YihongT in https://github.com/Human-Agent-Society/reef/pull/607 * feat(reefine): /reefine on every adapter with a command surface, and a proposer that writes what each harness takes by @Benjamin-eecs in https://github.com/Human-Agent-Society/reef/pull/596 * Fix harness release installation and large inference requests by @Xuan-1998 in https://github.com/Human-Agent-Society/reef/pull/600 ## New Contributors * @YihongT made their first contribution in https://github.com/Human-Agent-Society/reef/pull/606 **Full Changelog**: https://github.com/Human-Agent-Society/reef/compare/v0.1.0...v0.1.1 Repository README (current repository documentation, not release changes) Continual learning infra for self-improving agents [](https://github.com/Human-Agent-Society/reef/actions/workflows/ci.yml) [](https://pypi.org/project/reef-infra/) [](pyproject.toml) [](LICENSE) English | [中文](README.zh.md) Reef is the first open-source infrastructure for continual self-improving agents. It connects agent inference, feedback, learning, and versioned delivery. Use it to train model weights with Slime and SGLang, or improve an agent's harness, including its prompts, rules, and skills. **🚀 [Get started](https://reefinfra.ai/docs/getting-started/quickstart/) | 🗺️ [Roadmap](https://github.com/Human-Agent-Society/reef/issues/25) | 📣 [Launch post](https://x.com/ao_qu18465/status/2094867930081337730) | 💬 [Join Discord](https://discord.gg/5y8e5f937k) | 📱 [Join WeChat Group](docs/community/wechat.md)** ## 🎯 When to use Reef Use Reef when you want your agent to keep improving simply by learning from how you interact with your agent. | Your goal | Learning path | What you need | |---|---|---| | Keep getting stronger model designed for you | Model weight training | A trainable model, a supported GPU stack, and feedback your recipe can use | | Get your harness to self-improve | Harness optimization | A model endpoint, representative tasks, and an evaluator; no local training GPUs | | Scientific discoveries | Test-time training | An execution environment, a correctness checker, and a measurable objective | ## 🧩 How Reef fits your stack | Ability | Inference engine (vLLM, SGLang, …) | RL training framework (Slime, veRL, AReaL, …) | **Reef** | |---|:---:|:---:|:---:| | Serves live traffic | ✅ | ❌ | ✅ | | Trains weights | ❌ | ✅ | ✅ | | Version management | ❌ | ❌ | ✅ | | Stays live through updates | ❌ | ❌ | ✅ | | Evolves beyond weights (skills, harness) | ❌ | ❌ | ✅ | ## 🔄 How it works Reef processes each learning cycle in four steps. The table also shows which modules implement each step. | Step | What happens | Where it lives | |---|---|---| | **1 · Serve** | Serve agent requests and record interactions. | [`service/`](reef/service) — agent requests and interaction records [`runtime/`](reef/runtime) — inference and artifact updates | | **2 · Observe** | Match feedback to recorded interactions. | [`storage/records.py`](reef/storage/records.py) — stored interactions and feedback [`train/processors/`](reef/train/processors) — feedback matching and eligibility | | **3 · Grow** | Produce an update from eligible records. | [`recipe/`](reef/recipe) — recipe integration [`train/`](reef/train) — batches and update jobs | | **4 · Commit** | Apply the configured selection policy and publish accepted updates. | [`train/evaluation/`](reef/train/evaluation) — candidate evaluation [`artifact/`](reef/artifact) — version history [`surface/`](reef/surface) — artifact delivery | ## 📦 Installation > 💡 **Note** > > Reef's artifact and checkpoint functionality requires the `git-lfs` system > package. Reef initializes Git LFS locally for its artifact repositories. We recommend [uv](https://docs.astral.sh/uv/) for managing packages, and the commands below use it. ### From PyPI ```bash uv venv && source .venv/bin/activate uv pip install reef-infra python3 -c "import reef; print(reef.__version__)" ``` ### From source ```bash git lfs install git clone https://github.com/Human-Agent-Society/reef.git cd reef uv venv && source .venv/bin/activate uv pip install -e . python3 -c "import reef; print(reef.__version__)" ``` Use the source checkout for development and for the training examples below. ## 🔧 Using Reef Reef supports two learning surfaces: model **weights** and agent **harnesses**. The deployment's recipe determines which surface its scenarios update. As a minimal example, start Reef as a pure inference server: ```bash uv run reef serve --inference.model-path Qwen/Qwen2.5-1.5B-Instruct ``` ### Weight-training deployment #### Start the deployment The following example starts the SAO (arXiv:2607.07508) example deployment. Run it from a Reef checkout in an environment that satisfies the GPU requirements in [Evolve your model](https://reefinfra.ai/docs/user-guide/evolve-your-model/). ```bash uv pip install -e ".[slime]" && uv pip install --no-deps --group runtime export MODEL_PATH="Qwen/Qwen2.5-1.5B-Instruct" export REEF_TOKEN="reef-local" reef serve -c recipes/sao/examples/imo_answerbench/serve.yaml \ --inference.model-path "$MODEL_PATH" \ --reef.port "8900" curl -f http://127.0.0.1:8900/healthz # ready to serve ``` #### Send an inference request and report feedback Send inference requests through Reef and report a score for each response. The SAO recipe uses each eligible scored rollout to run a training step. Reef's inference endpoint is OpenAI- and Anthropic-compatible: `/v1/chat/completions` and `/v1/messages` take the provider's own request body. A request includes the `x-reef-scenario` header; a new name creates a scenario using the deployment's configured recipe. Requests do not select recipes. The response body uses the provider's OpenAI-compatible format. Reef adds the `x-reef-agent-record-id` response header. Its value is the **receipt** that a later report uses to identify this interaction. A report can contain a numeric `score`, textual or structured `feedback`, and the receipts it evaluates. This example reports both a score and a short explanation. ```python import os import httpx reef = httpx.Client( base_url="http://127.0.0.1:8900", headers={"Authorization": f"Bearer {os.environ['REEF_TOKEN']}", "x-reef-scenario": "hello-reef"}, timeout=300, ) # Send a provider-compatible inference request response = reef.post( "/v1/chat/completions", json={ "model": os.environ["MODEL_PATH"], "messages": [{"role": "user", "content": "Return exactly: reef is ready"}], }, ) response.raise_for_status() receipt = response.headers["x-reef-agent-record-id"] answer = response.json()["choices"][0]["message"]["content"] # Sending report about the inference matched = answer.strip() == "reef is ready" reef.post( "/reef/report", json={"score": float(matched), "feedback": "matched" if matched else "wrong answer", "references": [receipt]}, ).raise_for_status() ``` `feedback` carries the richer signal, plain text or a structured object, for recipes that read more than a scalar. The endpoint will validate the **report schema** ([`reef/core/reports/`](reef/core/reports)). #### Watch it learn and grow Once the recipe has enough feedback, it runs a training step and synchronizes the updated weights to the serving runtime. Later inference requests use the current version without restarting Reef. ### Harness-evolving deployment Refine a coding harness from plain-language asks, using a model API instead of GPUs. Reefine is the built-in harness-refinement recipe and includes a deployment configuration; specify the provider URL and model. From your Reef checkout and activated Python environment: ```bash reef serve --recipe reefine \ --inference.upstream-url http://127.0.0.1:11434 \ --inference.upstream-model gemma4:26b ``` For another provider, change `--inference.upstream-url` and `--inference.upstream-model`, and set `REEF_UPSTREAM_API_KEY` if authentication is required. With this configuration, Reef listens on `127.0.0.1:8901` without authentication (set `REEF_TOKEN` before starting it to require that token) and keeps its state under `.reef/reefine/` (`--recipe harness-evolve`, the former name, starts the same configuration). To change anything else, copy [the deployment configuration](reef/service/profiles/reefine.yaml) and pass your copy with `-c`. In another terminal with the same Python environment activated (the install bakes that terminal's `python3` into `reef-pi`), create a scenario, install the harness, and ask for a change: ```bash curl -fsS -H "Content-Type: application/json" \ -d '{"name": "my-harness"}' http://127.0.0.1:8901/reef/scenarios curl -fsS -H "x-reef-scenario: my-harness" \ 'http://127.0.0.1:8901/reef/harness/install?adapter=pi' | bash reef-pi evolve "when I ask you to fix a bug, reproduce it with a failing test first" ``` Inside a `reef-pi` session, `/reefine ` files the same ask. The served model writes the change as a skill, a rules entry, an agent command, or a pi extension. Where the host can isolate it (Linux with `bwrap` and `pasta`, as a non-root user), or with `REEF_PROPOSER_SANDBOX=none` on a machine you trust, it works as a coding agent that runs the changed harness before handing the change back. The next session's update notice offers the install; a step that settles while you are between turns offers its install right away. Review the versions with `/versions`, which opens a step's page, and install one with `/versions install`. To change the model, restart `reef serve` with another `--inference.upstream-model` and rerun the install command: installation writes the model ID into the local harness configuration. See the [Reefine tutorial](tutorials/reefine/README.md) for scripted bug-fix and research demos and the [Reefine guide](docs/user-guide/recipes/reefine.rst) for configuration. ## 📚 Recipes and examples Pick a recipe by the **task type** of your workload and by **what it should evolve**, model weights or the agent harness. Weight recipes need the GPU training stack, while harness recipes need only a model endpoint. Each recipe below links to its guide and each measured benchmark links to its results page, and the [recipe catalog](https://reefinfra.ai/docs/user-guide/recipes/) adds the code and example for every recipe. Reefine ships with `reef-infra`, and the other implementations live in this repository's `recipes/` cookbook, selected by dotted class reference and not shipped in the Reef wheel. | Task type | Task shape | Evolves the model | Evolves the harness | Standard benchmarks | |---|---|---|---|---| | Scientific discovery | Repeated attempts at one hard problem with a measurable objective | [TTT-Discover](https://reefinfra.ai/docs/user-guide/recipes/tttd/), [Guidance-TTT](recipes/tttd/examples/guidance_ttt/README.md) | None yet | Measured: [TriMul](recipes/tttd/examples/guidance_ttt/results/README.md), [circle packing](recipes/tttd/examples/tttd/README.md#formal-8x64-results), [Erdős minimum overlap](recipes/tttd/examples/tttd/README.md#formal-8x64-results). | | Continual learning on a task stream | A stream of independent tasks that a verifier scores one by one | [SAO](https://reefinfra.ai/docs/user-guide/recipes/sao/) | [Meta-Harness](recipes/meta_harness/README.md), [GEPA](https://reefinfra.ai/docs/user-guide/recipes/gepa/) | Measured: [AIME 2025](recipes/gepa/examples/aime/README.md#the-validation-contract), [IMOAnswerBench](recipes/sao/examples/imo_answerbench/README.md#results), [CEO-Bench](recipes/sao/examples/ceobench/README.md#results), [Terminal-Bench](recipes/meta_harness/examples/terminal_bench/README.md#results). | | Learning from usage | Real interaction where no one reports a score or feedback arrives late | [OpenClaw-RL](https://reefinfra.ai/docs/user-guide/recipes/openclawrl/) | [SkillClaw](https://reefinfra.ai/docs/user-guide/recipes/skillclaw/), [Reefine](docs/user-guide/recipes/reefine.rst) | Measured: [simulated student with GSM8K task stream](recipes/openclawrl/examples/openclawrl/README.md#results), [WildClawBench](recipes/skillclaw/README.md#the-2026-08-29-results-glm-53-flash-preliminary). | [`recipes/basic/`](recipes/basic/) is the record-only starting stack and stays outside the catalog. For a small walkthrough of feedback, candidate edits, and publication, start with [the coding harness tutorial](tutorials/evolve-your-harness/README.md). Each result page documents its task, evaluation setup, measurements, and limitations. ## 📐 Architecture ## 📖 Learn more The [documentation](https://reefinfra.ai/docs/) is organized in the following order: - [Quickstart](https://reefinfra.ai/docs/getting-started/quickstart/): install Reef, connect a client, and inspect the version history - [HTTP API](https://reefinfra.ai/docs/reference/http-api/): use the HTTP API and report feedback - [Write a recipe](https://reefinfra.ai/docs/developer-guide/write-a-recipe/): configure how Reef processes data and produces updates - [Evolve your harness](https://reefinfra.ai/docs/user-guide/evolve-your-harness/): evolve a harness instead of model weights - [Evolve your model](https://reefinfra.ai/docs/user-guide/evolve-your-model/): configure and operate a training deployment - [Recipes](https://reefinfra.ai/docs/user-guide/recipes/): the catalog of cookbook recipes by task type, with code, docs, example, and results for each - [The core loop](https://reefinfra.ai/docs/getting-started/core-loop/): The core loop of Reef - [Glossary](https://reefinfra.ai/docs/reference/glossary/): Explanation of the terminologies used ## 🤝 Community & Contributing Working on continual self-improving agent? - [Join Discord](https://discord.gg/5y8e5f937k) to share your recipes, ask implementation questions, and discuss new features. - [Join the WeChat group](docs/community/wechat.md): the group is full, so add the assistant and it will invite you. - Join the [GitHub Discussions](https://github.com/orgs/Human-Agent-Society/discussions) to ask questions, share ideas, and connect with the community. - Start contributing with the [contribution guide](CONTRIBUTING.md). - Propose designs through an [RFC issue](https://github.com/Human-Agent-Society/reef/issues/new?template=rfc.yml). - Report suspected vulnerabilities privately by following the [security policy](SECURITY.md). If Reef looks useful to you, please give it a ⭐ — it helps the community to discover and contribute to the project. ## 👥 The Team Reef brings together people exploring how agents can learn from experience and improve over time. The people below help turn that idea into working infrastructure. This list is non-exhaustive, with team members listed alphabetically by last name: [Wenhao Chai](https://github.com/wenhaochai), [Shuangrui Ding](https://github.com/Mark12Ding), [Shiyi Zoe Du](https://github.com/zoedsy), [Hao He](https://github.com/hehaodele), [Haoze He](https://github.com/HectorHHZ), [Chonghe Jiang](https://github.com/Chonghe-Jiang), [Nan Jiang](https://github.com/nanjiangwill), [Xuan Jiang](https://github.com/Xuan-1998), [Xiaochen Li](https://github.com/SeuperHakkerJa), [Paul Liang](https://github.com/pliang279), [Bo Liu](https://github.com/Benjamin-eecs), [Boyuan Long](https://github.com/BoyuanLong), [Qiuyang Mang](https://github.com/joyemang33), [Zhenting Qi](https://github.com/zhentingqi), [Ao Qu](https://github.com/quao627), [Mingruo Qu](https://github.com/workhardforcoding), [Zhaokai Wang](https://github.com/wzk1015), [Xuezhi Yan](https://github.com/yanxz), [Hanfei Yu](https://github.com/hanfeiyu), [Haofei Yu](https://github.com/lwaekfjlk), [Simon Yu](https://github.com/simonucl), [Han Zheng](https://github.com/MikeZheng777), [Kaichen Zhou](https://github.com/kaichen-z), [Zijian Zhou](https://github.com/BobbyZhouZijian), [Jiacheng Zhu](https://github.com/Jiacheng-Zhu-AIML), [Dingyi Zhuang](https://github.com/ZhuangDingyi), [Xinkai Zou](https://github.com/jayzou3773). ## ⭐ Star History ## 🙏 Acknowledgements We are particularly grateful to these projects which power important parts of Reef: - [SGLang](https://github.com/sgl-project/sglang) — high-performance inference - [slime](https://github.com/THUDM/slime) — model weight training - [cordis](https://github.com/cordiverse/cordis) — harness evolution