大模型
blog.bytebytego.com ·
@ByteByteGo
如何处理 LLM 应用中的错误和故障
ByteByteGo 的文章概述了 LLM 应用中错误处理的要点，包括重试、超时、熔断等，适合开发者参考，但需注意赞助内容。
AI推荐理由 ByteByteGo 文章介绍 LLM 应用的错误处理，内容系统，但包含赞助内容，且摘要未提供完整细节。
65
正文
一体化智能云（赞助）编写代码很快，但发布仍然困难。Railway 的一键式计算、存储和网络专为小型和超大规模软件而构建。人类和代理都在我们自己的硬件上操作一个垂直集成的系统。获得卓越的速度、更好的经济性和更多的平静。通过下面的链接注册即可获得免费云额度，让你的代理大显身手。立即部署 什么是 LLM 驱动的应用程序？它就像任何软件应用程序一样。但有一个主要区别。除了正常处理外，应用程序还将数据发送到大型语言模型（LLM）。然后，它使用模型的响应来执行任务。例如，客户支持聊天机器人可能依赖 LLM 来回答用户查询。文档处理系统可以使用 LLM 从上传的文档中提取姓名、日期和发票金额。编码助手可能会要求 LLM 编写一段代码并将其集成到逻辑流程中。从表面上看，我们可能会觉得构建这样的应用程序非常简单：用户发送请求。应用程序向 LLM 发送提示。LLM 返回答案。应用程序显示该答案或将其用于其他处理。然而，这些步骤中的任何一步都可能失败。网络可能不可用。LLM 提供商可能因各种原因拒绝用户请求。请求可能成功，但模型可能返回无效的 JSON。LLM 可能误解指令并因幻觉而编造虚假信息。它们也可能需要很长时间才能响应。弹性和错误处理技术有助于应用程序以最佳方式处理这些情况。以下是本文将要涵盖的内容：错误处理和弹性的含义是什么？为什么 LLM 应用程序需要特殊的错误处理？请求在 LLM 驱动的应用程序中如何流动？错误和失败的主要类型 如何对失败进行分类以采取适当的行动？如何正确重试请求？如何管理超时和截止时间？如何处理 LLM 驱动的应用程序中的业务规则？如何管理回退和优雅降级？断路器在此设置中如何工作？速率限制、队列和并发控制 如何使用幂等性使 LLM 工具调用安全？如何流式传输 LLM 的响应？错误处理和弹性的含义 在我们深入细节之前，让我们了解错误处理和弹性究竟意味着什么。你可以将错误处理视为程序的一部分，负责在出现问题时决定适当的操作。例如，考虑一个调用 LLM API 的应用程序。如果请求成功，程序以正常方式处理响应。但如果失败，程序可能必须从各种选项中选择，例如重试请求、显示解释性消息、使用备用模型或记录失败以供进一步调查。这些方法中的任何一种都比让整个应用程序崩溃要好得多。但我们也不能以同样的方式对待每个问题。真正的生产应用程序应该能够区分不同类型的失败，以便我们采取正确的行动。因此，良好的错误处理需要理解什么失败了以及为什么失败。相比之下，弹性是应用程序即使在系统的某些部分失败时也能继续执行其工作的能力。我们不需要应用程序完美运行才能具有弹性。失败是可以的。但它们应该以受控的方式发生。这也称为优雅降级。例如，考虑一个使用 LLM 为旅行者构建个性化行程的旅行助手。如果它依赖的主要 LLM 不可用，应用程序可以使用较小的备用模型来创建行程。如果该模型也不可用，应用程序可以显示以前为相同地点创建的目的地指南。当然，这些答案不会是个性化的，但用户不会只看到无用的错误消息。错误处理处理单个失败，而弹性更关心整个系统的行为，即使事情正在失败。为什么 LLM 应用程序需要特殊的错误处理？传统软件应用程序按照明确定义的规则工作。如果我们编写一个将 5 和 7 相加的函数，它将始终返回 12。如果函数成功返回，则结果被视为有效。对于 LLM，我们没有这种奢侈。这是因为 LLM 的输出是概率性的。换句话说，相同的提示每次都会产生不同的答案。此外，对 LLM 的成功 API 请求并不保证我们收到的响应是正确的甚至可用的。据我们所知，响应可能是完全胡言乱语。这意味着 LLM API 调用可能基于状态码看起来成功，但在逻辑上可能是失败。响应可能有多处问题：模型可能忽略了部分指令。当应用程序期望 JSON 时，它可能返回了纯文本响应。LLM 可能产生幻觉并编造一个在真实数据中没有依据的产品、政策或信息。模型可能因为达到其令牌限制而返回不完整的响应。LLM 可能因为误解上下文而拒绝无害的请求。LLM 可能选择错误的工具或使用无效参数调用工具。最后，LLM 可能返回违反某些重要业务规则的答案。我们在这里可以大致注意到两类失败：技术失败：当系统甚至无法完成操作时发生。这涉及网络错误、超时、服务不可用、无效凭据等。语义失败：这些类型的失败显示
即使操作在技术上成功，也可能出现故障。但结果可能不正确、不安全、不相关，甚至不可用。传统的错误处理侧重于处理技术故障。然而，基于LLM的应用程序需要同时处理技术故障和语义故障。请求在基于LLM的应用中如何流动？基于LLM的应用包含的远不止一个LLM。我们有许多不同的组件组合在一起，使应用程序完整。见下图：例如，考虑一个根据用户偏好检查产品的购物聊天机器人。它首先在产品数据库上运行搜索，并将结果返回给LLM。然后，它要求模型为特定用户选择合适的产品，然后调用库存API以查找这些产品的可用性。在此应用程序的每个边界都可能发生故障。用户可能提供一些无效信息。产品搜索可能返回无结果。由于外部原因，对LLM的请求可能超时。模型可能返回未知的产品标识符。库存服务可能宕机。最后，即使以某种方式生成了响应，在向用户流式传输时也可能中断。要使应用程序真正具有弹性，我们不能将LLM视为孤立的组件。我们需要从头到尾考虑请求的路径，并识别潜在问题。错误和故障的主要类型现在让我们看看在此类应用程序中可能发生的主要错误和故障类型。无效的用户输入某些故障甚至在调用LLM之前就可能出现。例如，用户可能提交空消息、上传不可接受的文件格式、提供超大文档或输入无效日期。在这种情况下，应用程序应能够验证明显的要求。无需调用LLM来查明是否缺少必填的电子邮件地址或上传的文件是否超过大小限制。如果我们提前拒绝无效请求，可以节省时间和处理成本。由于规则是固定的，我们还可以向用户提供更清晰的错误消息。网络和连接故障应用程序通过互联网与LLM提供商通信。但是，连接可能会中断。我们可能会遇到DNS查找失败。响应可能会丢失。此类故障是暂时的。我们可以重试请求。但应用程序不能无限期重试。我们还需要考虑重复操作是否安全，以及是否存在任何不可预见的副作用。超时在这种情况下，LLM响应所需的时间可能超过应用程序可以等待的时间。如果没有超时，请求可能会在消耗资源的同时保持卡住。超时值定义了应用程序可以等待多长时间。例如，交互式聊天应用程序的超时时间可能为20秒。但离线文档分析作业可能允许几分钟。此外，超时时间应取决于用户体验。例如，等待屏幕上的用户应比夜间运行的后台任务有更短的截止时间。速率限制LLM提供商对特定账户在时间段内可以使用的请求或令牌数量有限制。如果应用程序超过此限制，LLM提供商可能返回速率限制错误。这由HTTP状态码429表示。收到速率限制响应并不意味着服务已损坏。这意味着我们的应用程序发出的请求多于提供商在给定时间点可以处理的请求。为了处理这种情况，应用程序可能需要延迟请求、降低并发性、将工作放入队列，或切换到可能具有可用容量的另一个模型。提供商和服务器故障在某些情况下，LLM提供商可能因临时内部问题而宕机。这些故障通常由HTTP 5xx状态码表示。在这种情况下，有限的重试尝试可能是个好主意，因为故障可能很快消失。但是，如果故障持续存在，重复的重试调用只会增加已经不堪重负的服务的负载。这是合理的，因为故障可能很快消失。应用程序应停止重试，并在可能的情况下切换到替代执行路径。身份验证和权限故障如果应用程序提供过期或无效的API密钥，也可能导致LLM提供商返回401或403响应。在这种情况下，重试几乎没有意义。应用程序应记录故障、提醒相关方，并向用户返回适当的消息。它必须注意不要向用户暴露API密钥或其他内部安全细节。上下文长度故障每个语言模型对单个请求中可以处理的文本量都有限制。提示、对话历史、检索到的文档、工具定义和预期答案都是此上下文窗口的一部分，并占用空间。如果应用程序向LLM发送大量信息，LLM可能拒绝请求，或者可能没有足够的空间来生成答案。设计时考虑弹性的应用程序通常在发送请求前计算令牌。它还可以删除旧对话消息、总结早期内容、检索更少的文档，或将大作业分成小块以保持在限制内。格式错误或意外输出假设我们的应用程序期望以下格式的结果。{“customer_name”：“Adam Smith”，“priority”：“high”}然而，模型可能回复如下：“客户姓名似乎是Adam Smith，这看起来紧急。这样的答案对人类来说可能合理，但期望JSON响应的应用程序无法处理。为了解决这个问题，如果模型提供商支持，应用程序应使用结构化输出功能。它还应在使用响应之前根据模式验证响应。不正确或虚构的信息最危险的LLM故障以看似合理但完全错误的答案形式出现。这是由于模型幻觉造成的。例如，客户支持助手可能编造不存在的退款政策。研究工具可能生成不存在的来源。库存助手可能推荐目录中甚至不存在的产品。此类问题无法通过捕获异常来检测，因为没有发生技术异常。为了处理这些类型的问题，应用程序需要额外的检查、可信数据源或某种形式的人工干预才能继续。工具故障和部分完成LLM现在也用作可以调用API、搜索数据库、发送消息或执行事务的代理。这引发了一个主要问题。LLM采取的特定操作可能成功，即使周围的工作流失败。例如，假设LLM助手调用支付服务。支付成功，但在应用程序收到确认之前网络连接中断。如果应用程序盲目重试支付，我们可能最终向客户收取两次费用。这就是基于工具的系统需要幂等性、状态跟踪和恢复机制的原因。如何对故障进行分类以采取适当行动那么，我们如何处理基于LLM的应用程序中所有这些不同类型的错误？第一步是对它们进行分类。主要有几类：瞬时错误是暂时的，通常如果我们再次尝试操作就会消失。例如，网络问题、速率限制和服务器故障。永久错误会持续存在，直到请求或系统发生变化。此类错误包括无效凭据、不支持的文件类型、权限问题和格式错误的请求。语义错误发生在响应技术上正确但未满足应用程序要求时。例如，无效JSON、不支持的工具参数、幻觉等。一旦我们能够对错误进行分类，就可以适当地处理它。对于瞬时错误，我们应该在短暂延迟后重试或使用后备方案。对于永久错误，我们需要向能够纠正问题的相关方报告。对于语义错误，我们需要验证、修复或请求人工审查。当然，并非每个错误也完全适合单个类别。例如，上下文长度错误对于当前提示是永久的。但是，可以通过缩短提示来解决。如何正确重试请求？让我们看一些处理故障的技术。第一个要检查的是重试请求。重试是提高应用程序弹性的最简单方法之一。如果操作因短暂网络问题而失败，很可能在应用程序等待一段时间并再次尝试后会正常工作。要防止的主要事情是创建无限重试循环。这会增加成本并使故障更严重。限制尝试次数的重试方法更安全。每次尝试之间的时间延迟应在每次失败后增加。这种技术称为指数退避。一种简单的方法可能是在第一次重试前等待1秒，第二次前等待2秒，第三次前等待4秒，依此类推。应用程序还经常在每个延迟中添加少量随机时间。这称为抖动。如果没有抖动，数千个失败的请求可能同时重试。这可能导致另一个流量峰值，并抵消重试的好处。重试应用于超时、临时网络故障、429响应以及某些特定的5xx响应。我们不应将它们用于无效凭据、禁止的请求、错误输入或其他每次都会产生相同结果的问题。如何管理后备和优雅降级？在应用程序中，后备路径是替代路径。当正常首选路径因某些故障而失败时使用。应用程序可以按以下方式使用后备方法：尝试使用主要模型，通常质量较高。如果该模型不可用，则切换到较小的后备模型。如果没有模型可用，则返回预定义的后备响应或缓存信息。最后，如果无法通过任何选项安全地处理请求，则将其转交给人工进行进一步操作。见下图：最佳后备方法取决于任务。例如，我们可能可以使用较小的模型来总结内部会议。但是，对于解释复杂的法律文件，这可能不是好选择。类似地，缓存响应对于一般FAQ类型场景可能足够好。但对于当前账户余额之类的内容则不适用。换句话说，后备策略不应违反操作的原始要求。我们不能只是将任何可用的模型插入流程中。避免依赖单点故障也很重要。例如，如果主要和后备模型都通过同一提供商访问，则一次中断可能使两者都不可用。要实现真正的冗余，我们应该在两个路径之间进行分离。在此设置中，断路器如何工作？考虑LLM提供商不可用的场景。每个请求都超时。在这种情况下，继续发送更多请求是不明智的。它们只会浪费资源，让用户等待失败。断路器可以暂时停止对失败服务的调用。它通常有三种状态：在关闭状态下，我们可以按正常节奏发送请求。在打开状态下，请求会立即被拒绝或重定向。这是因为服务被认为不健康。在半开状态下，允许少量测试请求通过。它们有助于确定服务是否已在一定程度上恢复。如果半开状态下的测试请求成功，断路器可以关闭。但如果它们失败，断路器将回到打开状态。通过这种方式，断路器有助于消除重复故障对应用程序其余部分的影响。它们还允许我们配置适当的后备行为。速率限制、队列和并发控制基于LLM的应用程序不能接受无限的LLM工作，仅仅因为用户可以从UI提交。例如，如果同时到达10K个请求，立即将它们全部转发给LLM提供商可能会耗尽提供商限制、数据库连接、内存或应用程序的预算。使用速率限制，我们可以控制用户或客户端提交请求的频率。使用并发控制，我们可以限制用户或客户端可能提交请求的频率。此外，并发控制限制了应用程序可以同时处理的请求数量。队列存储额外的工作，直到容量可用。用户等待响应的交互式请求通常应优先于后台工作。例如，等待聊天响应的用户不应因为后台正在总结数千份文档而被阻塞。应用程序还应对令牌使用、文档大小、检索段落数量、工具调用次数和总工作流持续时间设置限制。结论在本文中，我们详细研究了基于LLM的应用程序的弹性和错误处理方面。关键要点是，基于LLM的应用程序可能具有与传统应用程序不同的故障。这是因为LLM固有的概率方法。重要的是对故障类型进行分类，以便应用程序采取适当的行动。尽管错误类型存在差异，我们仍然可以从传统应用程序中借用许多错误处理技术。例如，重试、后备、断路器、速率限制和排队等技术可以提供帮助。
最危险的LLM故障以看似合理但完全错误的答案形式出现。这是由于模型幻觉造成的。例如，客户支持助手可能编造不存在的退款政策。研究工具可能生成不存在的来源。库存助手可能推荐目录中甚至不存在的产品。此类问题无法通过捕获异常来检测，因为没有发生技术异常。为了处理这些类型的问题，应用程序需要额外的检查、可信数据源或某种形式的人工干预才能继续。工具故障和部分完成LLM现在也用作可以调用API、搜索数据库、发送消息或执行事务的代理。这引发了一个主要问题。LLM采取的特定操作可能成功，即使周围的工作流失败。例如，假设LLM助手调用支付服务。支付成功，但在应用程序收到确认之前网络连接中断。如果应用程序盲目重试支付，我们可能最终向客户收取两次费用。这就是基于工具的系统需要幂等性、状态跟踪和恢复机制的原因。如何对故障进行分类以采取适当行动那么，我们如何处理基于LLM的应用程序中所有这些不同类型的错误？第一步是对它们进行分类。主要有几类：瞬时错误是暂时的，通常如果我们再次尝试操作就会消失。例如，网络问题、速率限制和服务器故障。永久错误会持续存在，直到请求或系统发生变化。此类错误包括无效凭据、不支持的文件类型、权限问题和格式错误的请求。语义错误发生在响应技术上正确但未满足应用程序要求时。例如，无效JSON、不支持的工具参数、幻觉等。一旦我们能够对错误进行分类，就可以适当地处理它。对于瞬时错误，我们应该在短暂延迟后重试或使用后备方案。对于永久错误，我们需要向能够纠正问题的相关方报告。对于语义错误，我们需要验证、修复或请求人工审查。当然，并非每个错误也完全适合单个类别。例如，上下文长度错误对于当前提示是永久的。但是，可以通过缩短提示来解决。如何正确重试请求？让我们看一些处理故障的技术。第一个要检查的是重试请求。重试是提高应用程序弹性的最简单方法之一。如果操作因短暂网络问题而失败，很可能在应用程序等待一段时间并再次尝试后会正常工作。要防止的主要事情是创建无限重试循环。这会增加成本并使故障更严重。限制尝试次数的重试方法更安全。每次尝试之间的时间延迟应在每次失败后增加。这种技术称为指数退避。一种简单的方法可能是在第一次重试前等待1秒，第二次前等待2秒，第三次前等待4秒，依此类推。应用程序还经常在每个延迟中添加少量随机时间。这称为抖动。如果没有抖动，数千个失败的请求可能同时重试。这可能导致另一个流量峰值，并抵消重试的好处。重试应用于超时、临时网络故障、429响应以及某些特定的5xx响应。我们不应将它们用于无效凭据、禁止的请求、错误输入或其他每次都会产生相同结果的问题。如何管理后备和优雅降级？在应用程序中，后备路径是替代路径。当正常首选路径因某些故障而失败时使用。应用程序可以按以下方式使用后备方法：尝试使用主要模型，通常质量较高。如果该模型不可用，则切换到较小的后备模型。如果没有模型可用，则返回预定义的后备响应或缓存信息。最后，如果无法通过任何选项安全地处理请求，则将其转交给人工进行进一步操作。见下图：最佳后备方法取决于任务。例如，我们可能可以使用较小的模型来总结内部会议。但是，对于解释复杂的法律文件，这可能不是好选择。类似地，缓存响应对于一般FAQ类型场景可能足够好。但对于当前账户余额之类的内容则不适用。换句话说，后备策略不应违反操作的原始要求。我们不能只是将任何可用的模型插入流程中。避免依赖单点故障也很重要。例如，如果主要和后备模型都通过同一提供商访问，则一次中断可能使两者都不可用。要实现真正的冗余，我们应该在两个路径之间进行分离。在此设置中，断路器如何工作？考虑LLM提供商不可用的场景。每个请求都超时。在这种情况下，继续发送更多请求是不明智的。它们只会浪费资源，让用户等待失败。断路器可以暂时停止对失败服务的调用。它通常有三种状态：在关闭状态下，我们可以按正常节奏发送请求。在打开状态下，请求会立即被拒绝或重定向。这是因为服务被认为不健康。在半开状态下，允许少量测试请求通过。它们有助于确定服务是否已在一定程度上恢复。如果半开状态下的测试请求成功，断路器可以关闭。但如果它们失败，断路器将回到打开状态。通过这种方式，断路器有助于消除重复故障对应用程序其余部分的影响。它们还允许我们配置适当的后备行为。速率限制、队列和并发控制基于LLM的应用程序不能接受无限的LLM工作，仅仅因为用户可以从UI提交。例如，如果同时到达10K个请求，立即将它们全部转发给LLM提供商可能会耗尽提供商限制、数据库连接、内存或应用程序的预算。使用速率限制，我们可以控制用户或客户端提交请求的频率。使用并发控制，我们可以限制用户或客户端可能提交请求的频率。此外，并发控制限制了应用程序可以同时处理的请求数量。队列存储额外的工作，直到容量可用。用户等待响应的交互式请求通常应优先于后台工作。例如，等待聊天响应的用户不应因为后台正在总结数千份文档而被阻塞。应用程序还应对令牌使用、文档大小、检索段落数量、工具调用次数和总工作流持续时间设置限制。结论在本文中，我们详细研究了基于LLM的应用程序的弹性和错误处理方面。关键要点是，基于LLM的应用程序可能具有与传统应用程序不同的故障。这是因为LLM固有的概率方法。重要的是对故障类型进行分类，以便应用程序采取适当的行动。尽管错误类型存在差异，我们仍然可以从传统应用程序中借用许多错误处理技术。例如，重试、后备、断路器、速率限制和排队等技术可以提供帮助。
关闭。但如果它们失败，断路器将回到打开状态。通过这种方式，断路器有助于消除重复故障对应用程序其余部分的影响。它们还允许我们配置适当的后备行为。速率限制、队列和并发控制基于LLM的应用程序不能接受无限的LLM工作，仅仅因为用户可以从UI提交。例如，如果同时到达10K个请求，立即将它们全部转发给LLM提供商可能会耗尽提供商限制、数据库连接、内存或应用程序的预算。使用速率限制，我们可以控制用户或客户端提交请求的频率。使用并发控制，我们可以限制用户或客户端可能提交请求的频率。此外，并发控制限制了应用程序可以同时处理的请求数量。队列存储额外的工作，直到容量可用。用户等待响应的交互式请求通常应优先于后台工作。例如，等待聊天响应的用户不应因为后台正在总结数千份文档而被阻塞。应用程序还应对令牌使用、文档大小、检索段落数量、工具调用次数和总工作流持续时间设置限制。结论在本文中，我们详细研究了基于LLM的应用程序的弹性和错误处理方面。关键要点是，基于LLM的应用程序可能具有与传统应用程序不同的故障。这是因为LLM固有的概率方法。重要的是对故障类型进行分类，以便应用程序采取适当的行动。尽管错误类型存在差异，我们仍然可以从传统应用程序中借用许多错误处理技术。例如，重试、后备、断路器、速率限制和排队等技术可以提供帮助。
查看原文（中文正文为 AI 翻译）
The all-in-one intelligent cloud (Sponsored) Writing code is fast, shipping it is still hard. Railway’s push-button compute, storage, and networking is built for both small and hyperscale software. Humans and agents alike operate on one vertically integrated system on our own hardware. Get superior speed, better economics, and a lot more calm. Sign up with the link below for free cloud credits and let your agents cook. Deploy now What is an LLM-powered application? It’s just like any software application. But there is one major difference. Apart from normal processing, the application also sends data to a large language model (LLM). It then uses the model’s response to carry out a task. For example, a customer support chatbot might depend on an LLM to answer user queries. A document-processing system can use an LLM to extract names, dates, and invoice amounts from the uploaded documents. A coding assistant might ask an LLM to write a piece of code and integrate it into the logical flow. On face value, we might feel that building such an application is quite simple: The user sends a request. The application sends a prompt to an LLM. The LLM returns an answer. The application displays that answer or uses the same for some other processing. However, there are chances that any of these steps can fail. The network might be unavailable. The LLM provider might reject the user request for various reasons. The request might go through, but the model might return invalid JSON. The LLM can misunderstand the instructions and invent false information due to hallucinations. They might also take a long time to respond. The techniques for resiliency and error handling help prepare the application to handle these situations in the best possible manner. Here’s what we will cover in the article: What is the meaning of error handling and resiliency? Why do LLM applications need special treatment for error handling? How does a request flow in an LLM-powered application? Main types of errors and failures How to classify failures to take appropriate action? How to retry a request correctly? How to manage timeouts and deadlines? How to deal with business rules in an LLM-powered application? How to manage fallbacks and graceful degradation? How do circuit breakers work in this setup? Rate limiting, queues, and concurrency control How to make LLM tool calls safe with idempotency? How to stream the responses from the LLM? Meaning of Error Handling and Resiliency Before we get into the details, let us understand what error handling and resiliency exactly mean. You can think of error handling as a part of the program responsible for deciding the appropriate action when something goes wrong. For example, consider an application that calls an LLM API. If the request is successful, the program processes the response in a normal way. But if it fails, the program might have to choose from various options, such as retrying the request, showing an explanatory message, using a backup model, or recording the failure for further investigation. Any of these approaches is much better than allowing the entire application to crash. But we also cannot treat every problem in the same manner. A real production application should be able to distinguish between different types of failures so that we can take the right action. Therefore, good error handling needs an understanding of what failed and why. In contrast, resiliency is an application’s ability to continue doing its job even when some parts of the system are failing. We don’t need an application to operate perfectly to be resilient. Failures are fine. But they should happen in a controlled manner. This is also known as graceful degradation. For example, consider a travel assistant that uses an LLM to build personalized itineraries for travellers. If the primary LLM it depends on is unavailable, the application can use a smaller backup model for creating the itineraries. If that model also becomes unavailable, the application can display previously created destination guides for the same locations. Of course, these answers won’t be personalized, but the user won’t just see a useless error message. While error handling deals with an individual failure, resiliency is more concerned about the behavior of the entire system even when things are failing. Why do LLM Applications Need Special Treatment for Error Handling? Traditional software apps work according to well-defined rules. If we write a function that adds 5 and 7, it will always return 12. If the function returns successfully, the result is deemed valid. With LLMs, we don’t have this luxury. This is because the output from LLMs is probabilistic. In other words, the same prompt can produce different answers every single time. Also, a successful API request to an LLM doesn’t guarantee that the response we receive is correct or even usable. For all we know, the response might be utterly gibberish. This means that an LLM API call might appear successful based on status code, but it might be a failure in logical terms. There are several things that could be wrong with the response: The model might have ignored some part of the instructions. It might have returned a pure text response when the application expected JSON. The LLM might hallucinate and invent a product, policy, or piece of information that has no basis in real data. The model might return an incomplete response because it reached its token limit. The LLM can refuse a harmless request because it misunderstood the context. The LLM might select the wrong tool or call a tool with invalid arguments. Lastly, the LLM might return an answer that violates some important business rule. We can broadly notice two categories of failures over here: Technical Failures: They happen when the system can’t even complete the operation. This involves stuff like network errors, timeouts, unavailable service, invalid credentials, and so on. Semantic Failures: These types of failures show up even when the operation is technically successful. But the result might be incorrect, unsafe, irrelevant, or even unusable. Conventional error handling focuses on dealing with technical failures. However, LLM-powered applications need to handle both technical and semantic failures. How Does a Request Flow in an LLM-powered App? LLM-powered apps contain much more than an LLM. We have many different components that come together to make an application complete. See the diagram below: As an example, consider a shopping chatbot that checks products based on the user’s preference. It first runs a search on a product database and gives the results back to an LLM. It then asks the model to select the right products for a particular user and then calls an inventory API to find out the availability of those products. We can have failures at every boundary within this application. The user might provide some invalid information. The product search may return no results. The request to the LLM might time out due to external reasons. The model may return an unknown product identifier. The inventory service might go down. Lastly, even if the response is somehow generated, it might be interrupted while streaming to the user. To make an application truly resilient, we can’t treat the LLM as an isolated component. We need to consider the path of the request from the beginning to the end and identify potential problems. Main Types of Errors and Failures Let us now look at the main types of errors and failures that can occur in such an application. Invalid User Input Some failures can show up even before the LLM is called. For example, a user may submit an empty message, upload an unacceptable file format, provide an extremely large document, or enter an invalid date. In such cases, the application should be able to validate obvious requirements. There is no need to call an LLM to find out if a mandatory email address is missing or if an uploaded file exceeds the size limit. If we reject invalid requests early, we can save time and processing costs. We can also provide much clearer error messages to the users since the rules are fixed. Network and Connection Failures An application talks to an LLM provider over the internet. However, the connections can be interrupted. We can have DNS lookup failures. Responses might be lost. Such type of failures are temporary. We can retry the request. But the application cannot retry indefinitely. We also need to consider whether it is safe to repeat the operation and whether there are any unforeseen side effects. Timeouts In this case, the LLM may take longer to respond than the application can wait. If there’s no timeout, the request can remain stuck while consuming resources. The timeout value defines how long the application can wait. For example, an interactive chat application might have a timeout of 20 seconds. But an offline document-analysis job might allow several minutes. Also, the timeout should depend on the user experience. For example, a user waiting on a screen should have a shorter deadline than a background task that runs overnight. Rate Limits LLM providers have a limit to how many requests or tokens a specific account can use within a time period. If the application goes above this limit, the LLM provider may return a rate-limit error. This is represented by the HTTP status code 429. Getting a rate limit response doesn’t mean that the service is broken. It means that our application is making more requests than what the provider can handle at a given point in time. To handle this, the application may need to delay the requests, reduce concurrency, put the work in a queue, or switch to another model that might have available capacity. Provider and Server Failures In certain situations, the LLM provider might be down due to a temporary internal problem. These failures are commonly represented by HTTP 5xx status codes. In this case, a limited retry attempt could be a good idea because the failure might vanish quickly. However, if the failures continue, repeated retry calls only increase the load upon an already struggling service. is reasonable because the failure may disappear quickly. The application should stop retrying and switch to an alternative execution path if possible. Authentication and Permission Failures If the application provides an expired or invalid API key, it might also result in a 401 or 403 response from the LLM provider. In this case, there is very little point to retrying. The application should record the failure, alert the relevant parties, and return an appropriate message to the user. It must take care not to expose API keys or other internal security details to the user. Context-length Failures Every language model has a limit on how much text it can process in one request. The prompt, conversation history, retrieved documents, tool definitions, and expected answer are all part of this context window and consume space. If the application sends a lot of information to the LLM, the LLM may reject the request, or it may have too little space for the answer. An application designed with resiliency in mind usually counts tokens before sending the request. It can also remove old conversation messages, summarize earlier content, retrieve fewer documents, or divide a large job into smaller pieces to stay within the limit. Malformed or Unexpected Output Let’s say our application expects a result in the following format. { “customer_name”: “Adam Smith”, “priority”: “high” } However, the model might reply something like: “The customer’s name appears to be Adam Smith, and this looks urgent. Such an answer might make sense to a human being, but the application that expects a JSON response can’t process it. To get around this, the application should use a structured-output feature if the model provider supports it. It should also validate the response against a schema before using it. Incorrect or Invented Information The most dangerous LLM failures happen in the form of plausible answers that are completely wrong. This occurs due to model hallucination. For example, a customer-support assistant might invent a refund policy that doesn’t exist. A research tool might generate a source that doesn’t exist. An inventory assistant might recommend a product that is not even there in the catalog. Such problems can’t be detected by catching exceptions because no technical exception occurred. To handle these types of issues, the application needs additional checks, trusted data sources, or some form of human intervention to proceed further. Tool Failures and Partial Completion LLMs are also now used as agents that can make API calls, search databases, send messages, or perform transactions. This gives rise to a major problem. The specific action taken by the LLM might succeed even when the surrounding workflow fails. For example, suppose the LLM assistant calls a payment service. The payment succeeds, but the network connection breaks before the application gets the confirmation. If the application retries the payment blindly, we might end up charging the customer twice. This is the reason tool-based systems need idempotency, state tracking, and recovery mechanisms. How to Classify Failures to Take Appropriate Action So how do we deal with all these different types of errors in an LLM-powered application? The first step is to classify them. There are main categories: A transient error is temporary and usually disappears if we attempt the operation again. For example, network problems, rate limits, and server failures. A permanent error continues until there is a change in the request or the system. Such errors include invalid credentials, unsupported file types, permission issues, and malformed requests. A semantic error occurs when the response is technically correct, but doesn’t solve the application’s requirements. Examples are invalid JSON, unsupported tool arguments, hallucinations, and so on. Once we are able to classify an error, we can deal with it appropriately. For a transient error, we should retry after a short delay or use a fallback. For permanent errors, we need to report to the concerned parties who can correct the problem. For semantic errors, we need to validate, repair, or request human review. Of course, not every error also fits neatly into a single category. For example, a context-length error is permanent for the current prompt. However, it can be solved by making the prompt shorter. How To Retry a Request Correctly? Let us look at some techniques to deal with failures. The first one to check out is retrying a request. Retries are one of the simplest ways to improve an application’s resiliency. If an operation fails because of a brief network problem, chances are that it will work fine once the application waits for a while and tries again. The main thing to guard against is creating an infinite retry loop. It can increase costs and make the outage worse. A safer approach to retrying limits the number of attempts. The time delay between each attempt should increase after each failure. This technique is known as exponential backoff. A simple approach could be to wait 1 second before the first retry, 2 seconds before the second, and 4 seconds before the third, and so on. Applications also often add a small random amount of time to each delay. This is known as jitter. Without jitter, thousands of failed requests may all retry at the same time. This can cause another traffic spike and negate the benefit of retrying. Retries should be used for timeouts, temporary network failures, 429 responses, and some specific 5xx responses. We shouldn’t use them for failures such as invalid credentials, prohibited requests, bad input, or other problems that will produce the same result every single time. How to Manage Fallbacks and Graceful Degradation? In an application, the fallback path is an alternative path. It is used when the normal preferred path fails due to some failure. An application can use the fallback approach in the following manner: Attempt to use the primary model, which is usually of high quality. If that model is unavailable, switch to a smaller backup model. If no model is available, return a pre-defined fallback response or cached information. Lastly, if the request can’t be handled safely through any option, transfer it to a human for further action. See the diagram below: The best fallback approach depends on the task. For example, we might be okay to use a smaller model for summarizing an internal meeting. However, it might not be a good choice for interpreting a complicated legal document. Similarly, a cached response may be good enough for a general FAQ type scenario. But it won’t be suitable for something like the current account balance. In other words, the fallback strategy should not violate the original requirements of the operation. We can’t just plug any available model into the flow. It’s also important to avoid dependency on a single point of failure. For example, if both the primary and backup models are accessed through the same provider, an outage can disable both. For real redundancy, we should have separation between the two pathways. How Do Circuit Breakers Work in this Setup? Consider a scenario where the LLM provider is unavailable. Every request is timing out. In such cases, it is not wise to keep sending more requests. They only waste resources and make users wait for failures. A circuit breaker can temporarily stop calls to a failing service. It commonly has three states: In the closed state, we can send the requests at normal cadence. In the open state, requests get rejected or redirected immediately. This is because the service is considered unhealthy. In the half-open state, a small number of test requests are allowed to go through. They help determine whether the service has recovered to some extent. If the test requests in the half-open state succeed, the circuit can close. But if they fail, the circuit goes back to the open state. In this way, circuit breakers help get rid of repeated failures from infecting the rest of the application. They also let us configure the appropriate fallback behaviour. Rate Limiting, Queues, and Concurrency Control An LLM-powered application can’t accept unlimited LLM work just because users can submit it from the UI. For example, if 10K requests arrive at once, forwarding all of them to the LLM provider immediately might exhaust provider limits, database connections, memory, or the application’s budget. Using rate limiting, we can control how frequently a user or client can submit requests. With concurrency control, we can limit how frequently a user or client might submit requests. Also, concurrency control puts a cap on how many requests the application can process simultaneously. A queue stores extra work until capacity becomes available. Interactive requests where a user is waiting for the response should usually receive higher priority over background work. For example, a user waiting for a chat response should not be blocked because thousands of documents are being summarized in the background. The application should also place limits on token usage, document size, number of retrieved passages, number of tool calls, and total workflow duration. Conclusion In this article, we have taken a detailed look at resiliency and error handling aspects when it comes to LLM-powered applications. The key takeaway is that LLM-powered applications can have failures that are different from traditional applications. This is because of the inherent probabilistic approach of the LLMs. The important thing is to classify the type of failure so that proper action can be taken by the application. Despite the differences in the types of errors, we can still borrow many of the techniques for error handling from traditional applications. For example, things like retries, fallbacks, circuit breakers, rate limiting, and queuing can help.