智能体
全球精选 ·
@Snir Cohen
HPE Zerto基于Amazon Bedrock构建智能故障排查系统
HPE Zerto与AWS合作，在客户本地环境部署了一套基于Amazon Bedrock的智能故障排查系统。该系统采用多智能体架构，通过自然语言交互，帮助运维人员快速从问题定位到可操作建议，支持灾难恢复和网络弹性管理。文章详细介绍了系统架构、本地部署模式（基于Strands Agents）以及将智能体与实时灾难恢复数据结合的工程挑战。
AI推荐理由 AWS官方博客与HPE Zerto联合撰写的案例，详细描述了基于Amazon Bedrock的agentic故障排查系统的多智能体架构、本地部署模式及工程挑战，信息密度高，来源可信，对Agent落地有参考价值。
82
正文
本文由AWS和HPE Zerto团队共同撰写。如果您管理混合云和多云基础设施，您可能已经在使用AI系统来评估健康状况、调查问题并更快地采取行动。HPE Zerto通过构建一个由Amazon Bedrock驱动的智能故障排除系统来应对这一挑战。HPE Zerto软件帮助您加强网络弹性、灾难恢复和持续数据保护。您可以使用这些功能大幅减少数据丢失和停机时间，同时支持从勒索软件事件、中断和其他干扰中快速恢复。本文介绍了我们选择的系统架构以及在开发由Amazon Bedrock驱动的智能故障排除系统时面临的挑战。我们将其部署在本地环境中，对操作信号、环境上下文和产品知识进行受控且安全的访问。该系统提供自然语言体验，帮助您快速从问题转向可操作的见解，以支持更快、更明智的恢复决策。业务挑战数据保护和灾难恢复团队在规模、复杂性和业务关键性不断增长的环境中运作。他们必须持续监控保护健康状况、验证服务级别协议（SLA）合规性、解释警报、调查故障并为恢复事件做好准备。这项工作通常跨越多个站点和大量受保护的工作负载。回答操作问题所需的信息通常分散在产品警报、事件和文档中，迫使团队在采取行动前手动收集上下文。操作员在仪表板、报告和知识源之间切换以了解正在发生的事情以及下一步该做什么，浪费了时间。经验较少的管理员可能难以解释症状或确定最安全的修复路径。管理人员可能无法快速访问风险、SLA暴露或恢复就绪状态的清晰摘要。在中断或网络事件期间，这些延迟变得更加严重，因为不确定性的成本很高，团队需要快速获得清晰、可信的指导。因此，我们的客户需要一种方法，将复杂的弹性数据转化为可操作的答案，而不会增加操作负担，同时部署在不同的环境中。解决方案概述从根本上讲，我们设计该系统旨在：作为支持助手，帮助您解决配置问题并进行故障排除，而无需创建支持工单。提醒您系统健康问题并协助缓解。代表您执行任务以加速设置时间和功能采用。我们将系统作为Zerto产品中的一个pod部署在您的本地环境中，可直接从您日常操作中已使用的同一UI访问。您可以使用自然语言进行交互，系统旨在推理、行动并以基于您环境实时状态的上下文相关、可操作的答案进行响应。我们选择Amazon Bedrock是因为它能够支持安全的企业部署、模型灵活性和操作控制。关键考虑因素包括：安全与治理——Amazon Bedrock支持对基础模型的受控访问，帮助我们将AI使用与企业安全、合规和数据保护要求保持一致。模型灵活性——该服务使我们能够访问和评估多个基础模型，并根据质量、延迟和成本选择最合适的模型。操作集成——Amazon Bedrock与用于护栏、可观测性和知识检索的AWS服务集成，支持更易于管理的生产架构。我们使用Amazon Bedrock Guardrails强制执行内容安全，以确保您的提示和系统输出符合公司政策、合规性和安全要求。下图显示了架构的关键组件：UI层——您通过嵌入在现有Zerto UI中的聊天界面与系统交互。使用服务器发送事件（SSE）的实时流式传输显示调查进度，提供更好的用户体验。智能体层——使用Strands Agents框架构建的智能体作为pod在您的本地环境中运行，可访问以下内容：会话历史——本地存储的历史聊天对话和上下文。内部工具——本地模型上下文协议（MCP）服务器，公开ZVM（Zerto Manager）API，智能体可以调用这些API以结构化方式访问实时环境数据。外部工具——使用Amazon Bedrock Knowledge Bases智能获取公共文档、相关运行手册和操作知识，这是一种集中式、完全托管的检索增强生成（RAG）功能，具有内部语义搜索以进行上下文检索。智能层——由三个主要组件组成：推理——请求发送到Amazon Bedrock，后者将其路由到选定的基础模型。使用AWS身份和访问管理（IAM）角色标签进行每租户请求标记，每个租户承担这些标签。安全——在推理前后应用Amazon Bedrock Guardrails以强制执行内容边界，并帮助验证响应保持在操作主题范围内。限制和配额——结合使用Amazon CloudWatch、Amazon DynamoDB和AWS Lambda来强制执行每租户配额和限制，并维护成本效益高的系统。可观测性层——从系统发送到Amazon CloudWatch的遥测数据提供对智能体性能、错误率和使用模式的可见性。图1：解决方案架构下图显示了嵌入在Zerto产品中的智能体系统。图2：HPE Zerto管理控制台中的嵌入式智能体系统智能体流程系统由多个子智能体组成，由编排器智能体控制，编排器智能体可以根据手头的任务将每个子智能体用作工具。反过来，每个子智能体都有自己的一套工具和单独的上下文来处理其子任务。图3显示了智能体之间的流程和层次结构。编排器决定何时委派工作。常规问题直接回答，而深入的、日志密集的调查则路由到专门的子智能体，范围限定在相关组件。每个子智能体使用自己的新上下文、专用系统提示、自己的工具运行，并使用更强的模型进行推理密集型工作。其最终响应是紧凑、提炼的报告，不会用原始日志和调查草稿污染或膨胀父级的上下文。编排器智能体——有两个目的。您可以使用它检查保护组状态、查看最近的警报或询问配置最佳实践。当给定系统中当前存在的问题时，智能体协调子智能体之间的故障排除流程。它在从响应中剥离秘密或原始日志后将最终答案转发给您。ZVM智能体——负责对ZVM相关问题进行故障排除，例如服务崩溃、网络问题和升级失败。使用提供领域特定知识和特定日志模式的专用技能，它通过MCP分析日志并添加环境数据。这提供了对当前问题及其可能原因的彻底分析。VRA智能体——负责对VRA（Zerto复制引擎）相关问题进行故障排除，例如复制问题和延迟、站点之间的网络和安装。它还具有领域特定知识技能，提供结构化的日志模式上下文，供智能体与日志以及通过MCP可用的环境数据进行匹配。图3：智能体流程挑战构建HPE Zerto智能故障排除系统需要应对几个关键的工程决策，包括模型选择、评估方法和本地部署的约束。模型选择作为开发过程的一部分，我们比较并组合了多个模型进行推理和推断，考虑了质量、性能和成本之间的权衡。我们首先评估了更大、更强的模型用于初始概念验证（POC），以证明可行性，后来实现了智能体评估机制（在下一节中描述），并比较了其他模型，其中一些更快、更小、更便宜。最终，如前所述，选择了多模型架构。随着新模型的推出，我们继续评估它们。有关按AWS区域划分的模型可用性，请参阅Amazon Bedrock中按AWS区域划分的支持模型。智能体评估我们使用PyTest和Strands Agents Evals SDK来系统地评估智能体，并在系统升级过程中持续进行评估。我们还创建了运行完整评估套件的作业，以查看智能体行为中的回归和改进。创建了三种类型的评估：基本查询评估——单轮查询-响应评估，其中智能体被给予查询作为测试，并评估响应。多轮评估——完整对话评估，其中智能体被给予初始查询作为测试，智能体响应，用户模拟器大语言模型（LLM）充当实际用户。此技术使用Strands Agents ActorSimulators。最后评估完整对话。动态测试评估——为了评估未覆盖和动态用例，我们使用Strands Agents Experiment Generator创建了完全动态的测试套件，该生成器根据特定标准创建独特的测试套件。对于每个测试，有四个评估器：响应评估器——根据特定的预定义标准评估智能体响应，并使用可解析以提取相关数据的结构化输出。轨迹评估器——根据特定的预定义标准评估智能体的工具选择。此评估器也使用结构化输出。延迟评估器——评估智能体延迟并检查其是否在定义的阈值内。令牌评估器——评估智能体令牌使用情况并检查其是否保持在边界内。“Strands Agents消除了构建AI智能体的复杂工作——没有僵化的工作流定义，没有样板工具链代码，只需一个模型、一个提示和您需要的工具。凭借内置的评估和可观测性，我的团队比我们尝试过的任何框架都更快地构建、测试和迭代智能体。它涵盖了完整的开发生命周期，因此我们可以专注于解决实际问题，而不是编写粘合代码。”——Shahak Gabay，HPE Zerto高级软件工程师本地部署模型我们的智能故障排除系统部署的约束之一是它必须在您自己的环境中运行。灾难恢复（DR）基础设施通常是气隙的、对延迟敏感的，或受严格的数据驻留要求约束。我们使用Strands Agents SDK实现了智能体运行时，该SDK完全在ZVM设备内的本地运行。只有模型推理请求和知识库查询通过HTTPS发送到AWS。业务成果借助HPE Zerto智能体系统，您可以在加强恢复就绪性的同时提高运营效率。通过将响应基于实时环境数据和可信的产品文档，系统帮助您回答操作问题、调查问题、识别可能的根本原因，并优先处理需要采取行动的风险。自2026年第二季度发布以来，已有超过20%的HPE Zerto客户采用并积极使用该智能体系统。这表明对AI辅助灾难恢复操作的需求强劲。在生产使用中，我们观察到智能体系统支持的工作流的支持案例减少了10%。您现在可以通过引导式自助服务直接在产品中调查和解决更常见的操作问题。影响不仅限于减少支持案例。通过减少重复的数据收集和手动调查，系统有助于加速您的故障排除和根本原因识别，同时提高管理员生产力。您可以更快地生成操作摘要，更早地识别SLA和保护风险，并在日常操作和事件响应期间做出更明智的决策。随着采用率的持续增长，这些功能可以帮助您减少操作摩擦、提高恢复就绪性，并更有效地扩展灾难恢复操作，而无需按比例增加专业知识。在内部，我们的工程师已开始使用智能体系统的故障排除功能来调查和解决常见的操作问题。此外，我们的QA自动化团队已将智能体系统集成到他们的工作流中，现在在测试期间发现的问题上进行更快、更准确的调查和解决。经验教训构建和部署此解决方案揭示了几点见解，这些见解塑造了我们对智能体架构的方法。这些见解集中在本地部署约束、智能体分解和实时用户反馈上。本地部署从根本上改变了架构。没有客户数据可以离开您的本地网络的约束影响了每个架构决策，从我们选择Strands Agents SDK进行本地编排到用于本地数据访问的MCP服务器设计。MCP服务器是将智能体基于实时操作数据的有效模式。将现有产品API包装在MCP服务器中为智能体提供了对实时环境数据的结构化、类型化访问，并显著加速了我们的智能体开发。多智能体分解提高了复杂任务的可靠性。将工作流分解为多个专门的智能体，每个智能体具有有限的职责，使每个组件可独立测试，整个系统更易于调试。我们将子智能体以中心辐射拓扑连接。每个委派都通过父级流动，发现问题属于其他地方的子智能体向父级报告，而不是直接调用对等体。这种设计为安全规则和最终答案保持单一控制点，并保持流程线性且易于在遥测中跟踪。它还有助于防止失控的智能体到智能体递归，这可能会耗尽令牌预算。结果是在复杂问题上提供更好的答案，而不会牺牲日常聊天的速度和可预测性。流式传输对用户信任很重要。通过SSE实时显示调查进度显著改善了用户体验。用户可以看到每一步的发生，而不是在不可见的情况下等待。结论HPE Zerto Software将值得信赖的弹性专业知识、操作数据和AI驱动的协助与Amazon Bedrock生成式AI服务相结合，帮助您以更大的信心进行保护、恢复和操作。通过将智能指导嵌入日常保护和恢复工作流，我们帮助您将复杂的弹性信号转化为清晰、可操作的见解，支持更快的决策和更强的SLA就绪性。借助HPE Zerto，您可以超越时间点恢复就绪性，构建更智能、更有弹性的持续业务保护方法。要了解更多信息并深入探讨，请探索以下相关资源：评估用于生产的AI智能体：Strands Evals实用指南使用Strands Agents的多智能体协作模式超越基础：全面的基础模型选择框架关于作者Snir Cohen解决方案架构师，与全球企业组织合作，加速云中现有工作负载的采用和现代化。高度专注于生成式AI项目，领导赋能、架构设计和最佳实践的使用，以支持客户从构思到生产。Oz AltagarAWS原型架构师，与客户动手构建生成式AI工作负载。Oz直接与客户团队合作，通过动手开发、架构设计和最佳实践推动生成式AI领域的创新，特别是智能体AI。Idan Aharoni软件工程师和团队负责人，为HPE Zerto开发先进的AWS和AI解决方案。高度专注于智能体AI计划，领导架构、开发和智能系统的采用，同时推动云和AI技术的最佳实践和创新。Shahak Gabay高级软件工程师，AI智能体和AWS领域负责人。领导Zerto产品AI计划的技术方面，并负责AWS DR产品。Marianna Zemelman在HPE Zerto领导跨云和AI的产品战略和开发，在技术、客户需求和业务优先级之间架起桥梁。与工程团队密切合作，塑造现代化灾难恢复、提高恢复就绪性和加强运营弹性的能力。
获得自己的一套工具和单独的上下文来处理其子任务。图3显示了智能体之间的流程和层次结构。编排器决定何时委派工作。常规问题直接回答，而深入的、日志密集的调查则路由到专门的子智能体，范围限定在相关组件。每个子智能体使用自己的新上下文、专用系统提示、自己的工具运行，并使用更强的模型进行推理密集型工作。其最终响应是紧凑、提炼的报告，不会用原始日志和调查草稿污染或膨胀父级的上下文。编排器智能体——有两个目的。您可以使用它检查保护组状态、查看最近的警报或询问配置最佳实践。当给定系统中当前存在的问题时，智能体协调子智能体之间的故障排除流程。它在从响应中剥离秘密或原始日志后将最终答案转发给您。ZVM智能体——负责对ZVM相关问题进行故障排除，例如服务崩溃、网络问题和升级失败。使用提供领域特定知识和特定日志模式的专用技能，它通过MCP分析日志并添加环境数据。这提供了对当前问题及其可能原因的彻底分析。VRA智能体——负责对VRA（Zerto复制引擎）相关问题进行故障排除，例如复制问题和延迟、站点之间的网络和安装。它还具有领域特定知识技能，提供结构化的日志模式上下文，供智能体与日志以及通过MCP可用的环境数据进行匹配。图3：智能体流程挑战构建HPE Zerto智能故障排除系统需要应对几个关键的工程决策，包括模型选择、评估方法和本地部署的约束。模型选择作为开发过程的一部分，我们比较并组合了多个模型进行推理和推断，考虑了质量、性能和成本之间的权衡。我们首先评估了更大、更强的模型用于初始概念验证（POC），以证明可行性，后来实现了智能体评估机制（在下一节中描述），并比较了其他模型，其中一些更快、更小、更便宜。最终，如前所述，选择了多模型架构。随着新模型的推出，我们继续评估它们。有关按AWS区域划分的模型可用性，请参阅Amazon Bedrock中按AWS区域划分的支持模型。智能体评估我们使用PyTest和Strands Agents Evals SDK来系统地评估智能体，并在系统升级过程中持续进行评估。我们还创建了运行完整评估套件的作业，以查看智能体行为中的回归和改进。创建了三种类型的评估：基本查询评估——单轮查询-响应评估，其中智能体被给予查询作为测试，并评估响应。多轮评估——完整对话评估，其中智能体被给予初始查询作为测试，智能体响应，用户模拟器大语言模型（LLM）充当实际用户。此技术使用Strands Agents ActorSimulators。最后评估完整对话。动态测试评估——为了评估未覆盖和动态用例，我们使用Strands Agents Experiment Generator创建了完全动态的测试套件，该生成器根据特定标准创建独特的测试套件。对于每个测试，有四个评估器：响应评估器——根据特定的预定义标准评估智能体响应，并使用可解析以提取相关数据的结构化输出。轨迹评估器——根据特定的预定义标准评估智能体的工具选择。此评估器也使用结构化输出。延迟评估器——评估智能体延迟并检查其是否在定义的阈值内。令牌评估器——评估智能体令牌使用情况并检查其是否保持在边界内。“Strands Agents消除了构建AI智能体的复杂工作——没有僵化的工作流定义，没有样板工具链代码，只需一个模型、一个提示和您需要的工具。凭借内置的评估和可观测性，我的团队比我们尝试过的任何框架都更快地构建、测试和迭代智能体。它涵盖了完整的开发生命周期，因此我们可以专注于解决实际问题，而不是编写粘合代码。”——Shahak Gabay，HPE Zerto高级软件工程师本地部署模型我们的智能故障排除系统部署的约束之一是它必须在您自己的环境中运行。灾难恢复（DR）基础设施通常是气隙的、对延迟敏感的，或受严格的数据驻留要求约束。我们使用Strands Agents SDK实现了智能体运行时，该SDK完全在ZVM设备内的本地运行。只有模型推理请求和知识库查询通过HTTPS发送到AWS。业务成果借助HPE Zerto智能体系统，您可以在加强恢复就绪性的同时提高运营效率。通过将响应基于实时环境数据和可信的产品文档，系统帮助您回答操作问题、调查问题、识别可能的根本原因，并优先处理需要采取行动的风险。自2026年第二季度发布以来，已有超过20%的HPE Zerto客户采用并积极使用该智能体系统。这表明对AI辅助灾难恢复操作的需求强劲。在生产使用中，我们观察到智能体系统支持的工作流的支持案例减少了10%。您现在可以通过引导式自助服务直接在产品中调查和解决更常见的操作问题。影响不仅限于减少支持案例。通过减少重复的数据收集和手动调查，系统有助于加速您的故障排除和根本原因识别，同时提高管理员生产力。您可以更快地生成操作摘要，更早地识别SLA和保护风险，并在日常操作和事件响应期间做出更明智的决策。随着采用率的持续增长，这些功能可以帮助您减少操作摩擦、提高恢复就绪性，并更有效地扩展灾难恢复操作，而无需按比例增加专业知识。在内部，我们的工程师已开始使用智能体系统的故障排除功能来调查和解决常见的操作问题。此外，我们的QA自动化团队已将智能体系统集成到他们的工作流中，现在在测试期间发现的问题上进行更快、更准确的调查和解决。经验教训构建和部署此解决方案揭示了几点见解，这些见解塑造了我们对智能体架构的方法。这些见解集中在本地部署约束、智能体分解和实时用户反馈上。本地部署从根本上改变了架构。没有客户数据可以离开您的本地网络的约束影响了每个架构决策，从我们选择Strands Agents SDK进行本地编排到用于本地数据访问的MCP服务器设计。MCP服务器是将智能体基于实时操作数据的有效模式。将现有产品API包装在MCP服务器中为智能体提供了对实时环境数据的结构化、类型化访问，并显著加速了我们的智能体开发。多智能体分解提高了复杂任务的可靠性。将工作流分解为多个专门的智能体，每个智能体具有有限的职责，使每个组件可独立测试，整个系统更易于调试。我们将子智能体以中心辐射拓扑连接。每个委派都通过父级流动，发现问题属于其他地方的子智能体向父级报告，而不是直接调用对等体。这种设计为安全规则和最终答案保持单一控制点，并保持流程线性且易于在遥测中跟踪。它还有助于防止失控的智能体到智能体递归，这可能会耗尽令牌预算。结果是在复杂问题上提供更好的答案，而不会牺牲日常聊天的速度和可预测性。流式传输对用户信任很重要。通过SSE实时显示调查进度显著改善了用户体验。用户可以看到每一步的发生，而不是在不可见的情况下等待。结论HPE Zerto Software将值得信赖的弹性专业知识、操作数据和AI驱动的协助与Amazon Bedrock生成式AI服务相结合，帮助您以更大的信心进行保护、恢复和操作。通过将智能指导嵌入日常保护和恢复工作流，我们帮助您将复杂的弹性信号转化为清晰、可操作的见解，支持更快的决策和更强的SLA就绪性。借助HPE Zerto，您可以超越时间点恢复就绪性，构建更智能、更有弹性的持续业务保护方法。要了解更多信息并深入探讨，请探索以下相关资源：评估用于生产的AI智能体：Strands Evals实用指南使用Strands Agents的多智能体协作模式超越基础：全面的基础模型选择框架关于作者Snir Cohen解决方案架构师，与全球企业组织合作，加速云中现有工作负载的采用和现代化。高度专注于生成式AI项目，领导赋能、架构设计和最佳实践的使用，以支持客户从构思到生产。Oz AltagarAWS原型架构师，与客户动手构建生成式AI工作负载。Oz直接与客户团队合作，通过动手开发、架构设计和最佳实践推动生成式AI领域的创新，特别是智能体AI。Idan Aharoni软件工程师和团队负责人，为HPE Zerto开发先进的AWS和AI解决方案。高度专注于智能体AI计划，领导架构、开发和智能系统的采用，同时推动云和AI技术的最佳实践和创新。Shahak Gabay高级软件工程师，AI智能体和AWS领域负责人。领导Zerto产品AI计划的技术方面，并负责AWS DR产品。Marianna Zemelman在HPE Zerto领导跨云和AI的产品战略和开发，在技术、客户需求和业务优先级之间架起桥梁。与工程团队密切合作，塑造现代化灾难恢复、提高恢复就绪性和加强运营弹性的能力。
减少重复的数据收集和手动调查，系统有助于加速您的故障排除和根本原因识别，同时提高管理员生产力。您可以更快地生成操作摘要，更早地识别SLA和保护风险，并在日常操作和事件响应期间做出更明智的决策。随着采用率的持续增长，这些功能可以帮助您减少操作摩擦、提高恢复就绪性，并更有效地扩展灾难恢复操作，而无需按比例增加专业知识。在内部，我们的工程师已开始使用智能体系统的故障排除功能来调查和解决常见的操作问题。此外，我们的QA自动化团队已将智能体系统集成到他们的工作流中，现在在测试期间发现的问题上进行更快、更准确的调查和解决。经验教训构建和部署此解决方案揭示了几点见解，这些见解塑造了我们对智能体架构的方法。这些见解集中在本地部署约束、智能体分解和实时用户反馈上。本地部署从根本上改变了架构。没有客户数据可以离开您的本地网络的约束影响了每个架构决策，从我们选择Strands Agents SDK进行本地编排到用于本地数据访问的MCP服务器设计。MCP服务器是将智能体基于实时操作数据的有效模式。将现有产品API包装在MCP服务器中为智能体提供了对实时环境数据的结构化、类型化访问，并显著加速了我们的智能体开发。多智能体分解提高了复杂任务的可靠性。将工作流分解为多个专门的智能体，每个智能体具有有限的职责，使每个组件可独立测试，整个系统更易于调试。我们将子智能体以中心辐射拓扑连接。每个委派都通过父级流动，发现问题属于其他地方的子智能体向父级报告，而不是直接调用对等体。这种设计为安全规则和最终答案保持单一控制点，并保持流程线性且易于在遥测中跟踪。它还有助于防止失控的智能体到智能体递归，这可能会耗尽令牌预算。结果是在复杂问题上提供更好的答案，而不会牺牲日常聊天的速度和可预测性。流式传输对用户信任很重要。通过SSE实时显示调查进度显著改善了用户体验。用户可以看到每一步的发生，而不是在不可见的情况下等待。结论HPE Zerto Software将值得信赖的弹性专业知识、操作数据和AI驱动的协助与Amazon Bedrock生成式AI服务相结合，帮助您以更大的信心进行保护、恢复和操作。通过将智能指导嵌入日常保护和恢复工作流，我们帮助您将复杂的弹性信号转化为清晰、可操作的见解，支持更快的决策和更强的SLA就绪性。借助HPE Zerto，您可以超越时间点恢复就绪性，构建更智能、更有弹性的持续业务保护方法。要了解更多信息并深入探讨，请探索以下相关资源：评估用于生产的AI智能体：Strands Evals实用指南使用Strands Agents的多智能体协作模式超越基础：全面的基础模型选择框架关于作者Snir Cohen解决方案架构师，与全球企业组织合作，加速云中现有工作负载的采用和现代化。高度专注于生成式AI项目，领导赋能、架构设计和最佳实践的使用，以支持客户从构思到生产。Oz AltagarAWS原型架构师，与客户动手构建生成式AI工作负载。Oz直接与客户团队合作，通过动手开发、架构设计和最佳实践推动生成式AI领域的创新，特别是智能体AI。Idan Aharoni软件工程师和团队负责人，为HPE Zerto开发先进的AWS和AI解决方案。高度专注于智能体AI计划，领导架构、开发和智能系统的采用，同时推动云和AI技术的最佳实践和创新。Shahak Gabay高级软件工程师，AI智能体和AWS领域负责人。领导Zerto产品AI计划的技术方面，并负责AWS DR产品。Marianna Zemelman在HPE Zerto领导跨云和AI的产品战略和开发，在技术、客户需求和业务优先级之间架起桥梁。与工程团队密切合作，塑造现代化灾难恢复、提高恢复就绪性和加强运营弹性的能力。
查看原文（中文正文为 AI 翻译）
This post was co-written by AWS and the HPE Zerto team. If you manage hybrid and multi-cloud infrastructures, you may already be turning to AI systems to assess health, investigate issues, and act on problems faster. HPE Zerto addressed this challenge by building an agentic troubleshooting system powered by Amazon Bedrock. HPE Zerto Software helps you strengthen cyber resilience, disaster recovery, and continuous data protection. You can use these capabilities to substantially reduce data loss and downtime, while supporting rapid recovery from ransomware events, outages, and other disruptions. This post describes the system architecture we selected and the challenges we faced when developing an agentic troubleshooting system powered by Amazon Bedrock. We deployed it in an on-premises environment, with controlled and secure access to operational signals, environmental context, and product knowledge. The system provides a natural language experience that helps you move quickly from questions to actionable insights to support faster, more informed recovery decisions. Business challenge Data protection and disaster recovery teams operate in environments that are growing in scale, complexity, and business criticality. They must continuously monitor protection health, validate service level agreement (SLA) compliance, interpret alerts, investigate failures, and prepare for recovery events. This work often spans multiple sites and large numbers of protected workloads. Information needed to answer operational questions is often fragmented across product alerts, events, and documentation, forcing teams to manually gather context before they can act. Operators lose time moving between dashboards, reports, and knowledge sources to understand what is happening and what to do next. Less experienced administrators may struggle to interpret symptoms or determine the safest remediation path. Managers may lack fast access to clear summaries of risk, SLA exposure, or recovery readiness. During outages or cyber events, these delays become even more significant because the cost of uncertainty is high and teams need clear, trusted guidance quickly. Our customers therefore need a way to turn complex resilience data into actionable answers without adding more operational burden while deployed in different environments. Solution overview Fundamentally, we designed the system to: Serve as a support assistant helping you solve configuration issues and troubleshoot problems in a short-circuit without needing to create a support ticket. Alert you on system health problems and assist with mitigation. Perform tasks on your behalf to accelerate setup time and feature adoption. We deploy the system as a pod in the Zerto product inside your on-premises environment, accessible directly from the same UI you already use for day-to-day operations. You can interact in natural language, and the system is designed to reason, act, and respond with contextual, actionable answers grounded in the live state of your environment. We selected Amazon Bedrock because of its ability to support secure enterprise deployment, model flexibility, and operational control. The key considerations included: Security and governance – Amazon Bedrock supports controlled access to foundation models, helping us align AI usage with enterprise security, compliance, and data protection requirements. Model flexibility – The service made it possible to access and evaluate multiple foundation models and select the best fit based on quality, latency, and cost. Operational integration – Amazon Bedrock integrates with AWS services used for guardrails, observability, and knowledge retrieval, supporting a more manageable production architecture. We enforce content security using Amazon Bedrock Guardrails to make sure your prompts and system output are aligned with company policy, compliance, and security requirements. The following figure shows the key components of the architecture: UI layer – You interact with the system through the chat interface embedded in the existing Zerto UI. Real-time streaming using Server-Sent Events (SSE) shows investigation progress as it happens, providing a better user experience. Agentic layer – Agents built using the Strands Agents framework run as a pod inside your environment on-premises, with access to the following: Session history – Locally stored historical chat conversations and context. Internal tools – Local Model Context Protocol (MCP) server exposing ZVM (Zerto Manager) APIs the agent can call for structured access to live environment data. External tools – Smart fetching of public documentation, relevant runbooks and operational knowledge using Amazon Bedrock Knowledge Bases, a centralized, fully managed Retrieval Augmented Generation (RAG) capability with internal semantic search for context retrieval. Intelligence layer – Comprised of three main components: Inference – Requests are sent to Amazon Bedrock which routes them to the selected foundation model. Per-tenant request tagging is done using AWS Identity and Access Management (IAM) role tags each tenant assumes. Security – Amazon Bedrock Guardrails are applied before and after inference to enforce content boundaries and help verify responses remain scoped to operational topics. Limits and quotas – Amazon CloudWatch, Amazon DynamoDB, and AWS Lambda are combined to enforce per-tenant quotas and limits and maintain a cost-efficient system. Observability layer – Telemetry data sent from the system to Amazon CloudWatch provides visibility into agent performance, error rates, and usage patterns. Figure 1: Solution architecture The following figure shows the agentic system embedded in the Zerto product. Figure 2: The embedded agentic system in the HPE Zerto management console Agentic flow The system consists of multiple sub-agents, controlled by an Orchestrator agent which can use each of those sub-agents as tools based on the task in hand. In turn, each sub-agent gets its own set of tools and separate context to handle its sub-task. Figure 3 shows the flow and hierarchy between the agents. The Orchestrator decides when to delegate work. Routine questions are answered directly, while deep, log-heavy investigations are routed to a specialized sub-agent, scoped to the relevant component. Each sub-agent runs with its own fresh context, dedicated system prompt, its own tools, and uses a stronger model for the reasoning-heavy work. Its final response is a compact, distilled report, without raw logs and investigation scratch-work polluting or inflating the parent’s context. Orchestrator agent – Serves two purposes. You can use it for checking protection group status, reviewing recent alerts, or asking about configuration best practices. When given an issue that currently exists in the system, the agent orchestrates the troubleshooting flow between the sub-agents. It relays the final answer to you after stripping secrets or raw logs from the response. ZVM agent – Responsible for troubleshooting ZVM related issues such as service crashes, networking issues, and upgrade failures. Using dedicated skills that provide domain-specific knowledge and specific log patterns, it analyzes logs and adds environmental data through MCP. This provides a thorough analysis of the issue at hand and its possible causes. VRA agent – Responsible for troubleshooting VRA (Zerto Replication Engine) related issues such as replication problems and delays, networking between sites and installations. It also has domain-specific knowledge skills that provide structured log patterns context for the agent to match against the logs and the environmental data available through MCP. Figure 3: Agentic flow Challenges Building the HPE Zerto agentic troubleshooting system required navigating several key engineering decisions, including model selection, evaluation methodology, and the constraints of on-premises deployment. Model selection As part of our development process, we compared and then combined several models for reasoning and inference, considering the tradeoff between quality, performance, and cost. We evaluated bigger, stronger models first for the initial proof of concept (POC), to prove feasibility, and later implemented an agent evaluations mechanism (described in the following section) and compared other models, some of them faster, smaller, and cheaper. Eventually, a multi-model architecture was selected as described earlier. We continue to evaluate newer models as they become available. For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock. Agent evaluations We used PyTest and Strands Agents Evals SDK for evaluating agents systematically and on an ongoing basis as we keep upgrading the system. We have also created jobs for running the full evaluations suite to see regressions and improvements in the agent behavior. Three types of evaluations were created: Basic query evaluation – Single-turn query-response evaluation where the agent is given a query as a test and the response is evaluated. Multi-turn evaluation – Full conversation evaluation where the agent is given an initial query as a test, the agent responds and a user simulator large language model (LLM) acts as an actual user. This technique uses Strands Agents ActorSimulators. The full conversation is evaluated at the end. Dynamic tests evaluation – To evaluate uncovered and dynamic use cases, we created a fully dynamic test suite using the Strands Agents Experiment Generator, which creates a unique test suite based on specific criteria.For each test there are four evaluators: Response evaluator – Evaluates the agent response based on specific predefined criteria and using structured output that can be parsed to extract relevant data. Trajectory evaluator – Evaluates the agent tools selection based on specific pre-defined criteria. This evaluator also uses structured output. Latency evaluator – Evaluates the agent latency and checks it stands within the threshold defined. Token evaluator – Evaluates the agent token usage and checks it remains within boundaries. “Strands Agents eliminates the complex work of building AI agents — no rigid workflow definitions, no boilerplate tool-chaining code, just a model, a prompt, and the tools you need. With built-in evaluations and observability, my team builds, tests, and iterates on agents faster than with any framework we’ve tried. It covers the full development lifecycle so we can focus on solving real problems, not writing glue code.” — Shahak Gabay, Senior Software Engineer, HPE Zerto On-premises deployment model One of the constraints around deployment of our agentic troubleshooting system is that it must run inside your own environment. Disaster recovery (DR) infrastructure is frequently air-gapped, latency-sensitive, or subject to strict data residency requirements. We implemented the agent runtime using the Strands Agents SDK, which runs entirely on-premises within the ZVM appliance. Only model inference requests and knowledge base queries go out to AWS over HTTPS. Business outcomes With the HPE Zerto agentic system, you can improve operational efficiency while strengthening recovery readiness. By grounding responses in live environment data and trusted product documentation, the system helps you answer operational questions, investigate issues, identify likely root causes, and prioritize risks that require action. Since its release in Q2 of 2026, more than 20 percent of HPE Zerto customers have already adopted and are actively using the agentic system. This demonstrates strong demand for AI-assisted disaster recovery operations. Across production usage, we have observed a 10 percent reduction in support cases for workflows supported by the agentic system. You can now investigate and resolve more common operational issues directly within the product through guided self-service. The impact extends beyond support-case reduction. By reducing repetitive data gathering and manual investigation, the system helps accelerate your troubleshooting and root-cause identification while improving administrator productivity. You can generate operational summaries faster, identify SLA and protection risks earlier, and make more informed decisions during both routine operations and incident response. As adoption continues to grow, these capabilities can help you reduce operational friction, improve recovery readiness, and scale disaster recovery operations more effectively without requiring a proportional increase in specialized expertise. Internally, our engineers have started using the agentic system’s troubleshooting capabilities to investigate and resolve common operational issues. Moreover, our QA automation team has integrated the agentic system into their workflows, now performing faster and more accurate investigation and resolution of issues found during testing. Lessons learned Building and deploying this solution surfaced several insights that shaped our approach to agentic architectures. These insights center on on-premises constraints, agent decomposition, and real-time user feedback. On-premises deployment changes the architecture fundamentally. The constraint that no customer data could leave your on-premises network shaped every architectural decision, from our choice of the Strands Agents SDK for local orchestration to the MCP server design for local data access. MCP servers are an effective pattern for grounding agents in live operational data. Wrapping an existing product API in an MCP server gives agents structured, typed access to live environment data and significantly accelerated our agent development. Multi-agent decomposition improves reliability for complex tasks. Breaking the workflow into multiple specialized agents, each with a bounded responsibility, made each component independently testable and the overall system easier to debug.We wired the sub-agents in a hub-and-spoke topology. Every delegation flows through the parent, and a sub-agent that discovers the issue belongs elsewhere reports back to the parent rather than calling a peer directly. This design keeps a single point of control for safety rules and the final answer, and it keeps the flow linear and straightforward to trace in telemetry. It also helps prevent runaway agent-to-agent recursion that could exhaust the token budget.The result is better answers on complicated problems without sacrificing the speed and predictability of everyday chat. Streaming matters for user trust. Showing investigation progress in real time through SSE significantly improved the user experience. Users see each step as it happens rather than waiting without visibility. Conclusion HPE Zerto Software brings together trusted resilience expertise, operational data, and AI-driven assistance, with Amazon Bedrock generative AI service, to help you protect, recover, and operate with greater confidence. By embedding intelligent guidance into everyday protection and recovery workflows, we help you turn complex resilience signals into clear, actionable insights, supporting faster decisions and stronger SLA readiness. With HPE Zerto, you can move beyond point-in-time recovery readiness and build a more intelligent, resilient approach to continuous business protection. To read more and go deeper, explore these related resources: Evaluating AI agents for production: A practical guide to Strands Evals Multi-Agent collaboration patterns with Strands Agents Beyond the basics: A comprehensive foundation model selection framework for generative AI About the authors Snir Cohen Solutions Architect, working with Global Enterprise organizations, accelerating adoption and modernization of existing workloads in the cloud. Highly focused on generative AI projects, leading enablement, architecture design and use of best practices to support customers get from ideation to production. Oz Altagar Prototype Architect at AWS, working hands-on with customers to build generative AI workloads. Oz works directly alongside customer teams, driving innovation across generative AI and specifically Agentic AI through hands-on development, architecture design, and best practices. Idan Aharoni Software Engineer and Squad Lead, working on advanced AWS and AI solutions for HPE Zerto. Highly focused on Agentic AI initiatives, leading architecture, development, and adoption of intelligent systems while driving best practices and innovation across cloud and AI technologies. Shahak Gabay Senior software engineer and domain lead of AI Agents and AWS domains. Leading the technical side of Zerto’s product AI initiatives and responsible for the AWS DR product. Marianna Zemelman Leads product strategy and development across Cloud and AI at HPE Zerto, working at the intersection of technology, customer needs, and business priorities. Partners closely with engineering teams to shape capabilities that modernize disaster recovery, improve recovery readiness, and strengthen operational resilience.