智能体
x.com ·
@aiDotEngineer
推理工程专场要点：成本度量、路由与缓存实践
该帖整理 AI Engineer 推理工程专场的若干要点：Meta 按每成功任务成本而非每 token 衡量推理；OpenAI 推理负载均衡器曾因反馈回路导致流量震荡，后由全局优化器替代；同一编码 Agent 在闭源前沿模型与开源权重模型上均能完成塔防游戏，FriendliAI 称开源权重方案约便宜 5.5 倍；Superlinked 的自动调优以 80 美分训练出微调模型，在德语法律文本检索上提升 18%（概念验证）；Red Hat 演示中缓存命中使第二轮请求从约 3 秒降至约 1 秒。
AI推荐理由 同场会议的要点笔记，包含若干具体工程细节：Meta 以每成功任务成本而非每 token 衡量推理、OpenAI 推理负载均衡器反馈回路问题及优化器替代、开源权重模型运行成本约为闭源前沿模型的 5.5 倍便宜、Superlinked 微调 80 美分提升德语法律检索 18%、Red Hat 缓存命中使首轮 3 秒降至 1 秒。均为转述的会议要点，非一手论文或官方数据，需保留其未经独立验证的性质。
70
浏览 303
点赞 0
评论 0
转发 0
正文
Notes:
- Stream 200 tokens to a user, then lose the GPU to maintenance, and you cannot just retry. Meta measures inference by cost per successful task, not per token.
- OpenAI's inference load balancer used to set its routing weights from a feedback loop. Shift traffic off a hot engine and it cools, the controller reads cool as spare capacity, and the traffic comes back. An optimizer with a view of every engine replaced it.
- The same coding agent built the same tower defense game on a closed frontier model and on an open weight model. Both games worked. FriendliAI puts the open weight run at about five and a half times cheaper.
- Superlinked's automated tuning loop produced a fine tune that cost 80 cents to train and lifted retrieval on German legal text by 18 percent, as a proof of concept.
- In Red Hat's demo a first request took about three seconds. The next turn reused the same system prompt, hit the cache on the same machine, and took about one. A new system prompt on a different machine paid the full three again.
Inference Engineering playlist:
[youtube.com/watch?v=7c9FSU…](https://www.youtube.com/watch?v=7c9FSUVcXR0&list=PLcfpQ4tk2k0W3_JZkL2SRVKTVOj7DoWtZ)