智能体
arxiv.org ·
服务前先筛选：面向1.4亿规模生产客户体验AI智能体的仿真
该论文提出一种假设驱动的仿真工作流，用于在部署前筛选客户体验（CX）智能体。方法使用合成客户对智能体回复做出反应，并以模拟工具输出支撑多步 agentic 流程，无需调用生产后端。作者在 Nubank 的 Card Delivery 智能体及其后继 Card Management（巴西最高流量的聊天客服智能体）上使用 Snowglobe 模拟器验证：4 个已部署版本的仿真与生产版本级二值评估分数高度相关；仿真引导迭代在线上 A/B 测试中使 tNPS 提升 36.69 分；在超过 16000 次仿真对话中筛选开放权重配置后，线上 A/B 测试中选出的模型使自助服务率（SSR）提升 8.82 个百分点至 Nubank 观测到的最高水平，tNPS 无统计显著变化。作者认为仿真使模型、推理设置与提示词的广泛探索在不暴露客户的前提下变得可行。
AI推荐理由 arXiv 论文，作者团队来自 Nubank 等，摘要给出可验证的量化结果：在 Nubank 巴西最高流量的 Card Management 客服智能体上，仿真与生产版本级二值评估分数高度相关；仿真引导迭代在线上 A/B 测试中使 tNPS 提升 36.69 分；在 16000+ 仿真对话中筛选开放权重配置后，线上 A/B 测试使自助服务率（SSR）提升 8.82 个百分点且 tNPS 无统计显著变化。新信息在于提出假设驱动的仿真筛选工作流，用合成客户与模拟工具输出在不调用生产后端的情况下跑多步 agentic 流程，为受监管行业部署前评估提供可复用的方法。局限是摘要未披露相关性系数、样本量与统计细节，且结果来自单一公司场景，外推需谨慎。
82
正文
Computer Science > Artificial Intelligence
arXiv:2609.30137 (cs)
[Submitted on 24 Sep 2026]
Title:Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
Authors:Edesio Alcoba, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Conceição Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath
View a PDF of the paper titled Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale, by Edesio Alcoba and 15 other authors
View PDF
HTML (experimental)
Abstract:Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust.
We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation. Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test. We also screened open-weight configurations in over 16,000 simulated conversations. In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS. Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure, enabling production improvements that would have been impractical to pursue through live experimentation alone.
Comments:
Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:
arXiv:2609.30137 [cs.AI]
(or
arXiv:2609.30137v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.30137
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Kevin Rossell [view email]
[v1]
Thu, 24 Sep 2026 17:07:38 UTC (2,728 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale, by Edesio Alcoba and 15 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
cs.CL
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)