使用 WebSocket 和耐用对象进行大规模实时 AI 推理

2024 年 10 月，我们讨论了使用 AI Gateway 存储来自 AI 应用程序的数十亿条日志，以及我们如何使用 Cloudflare 的开发者平台来做到这一点。

In October 2024, we covered how to store billions of logs from your AI application using AI Gateway, and how we used Cloudflare’s Developer Platform to do this.

2024 年 10 月，我们介绍了如何使用 AI Gateway 存储来自 AI 应用程序的数十亿条日志，以及如何使用 Cloudflare 的开发者平台来执行此操作。

With AI Gateway already processing over 3 billion logs and experiencing rapid growth, the number of connections to the platform continues to increase steadily. To help developers manage this scale more effectively, we wanted to offer an alternative to implementing HTTP/2 keep-alive to maintain persistent HTTP(S) connections, thereby avoiding the overhead of repeated handshakes and TLS negotiations with each new HTTP connection to AI Gateway. We understand that implementing HTTP/2 can present challenges, particularly when many libraries and tools may not support it by default and most modern programming languages have well-established WebSocket libraries available.

AI网关已处理超过30亿条日志，且增长迅速，平台连接数持续稳定增长。为了帮助开发人员更有效地管理这种规模，我们希望提供一种替代方案来实现 HTTP/2 keep-alive 来维持持久的 HTTP(S) 连接，从而避免与 AI Gateway 的每个新 HTTP 连接重复握手和 TLS 协商的开销。我们知道，实现 HTTP/2 可能会带来挑战，特别是当许多库和工具默认情况下可能不支持它，并且大多数现代编程语言都有完善的 WebSocket 库可用时。

With this in mind, we used Cloudflare’s Developer Platform and Durable Objects (yes, again!) to build a WebSockets API that establishes a single, persistent connection, enabling continuous communication.

考虑到这一点，我们使用 Cloudflare 的开发者平台和 Durable Objects（是的，又是！）来构建 WebSockets API，该 API 建立单个持久连接，从而实现持续通信。

Through this API, all AI providers supported by AI Gateway can be accessed via WebSocket, allowing you to maintain a single TCP connection between your client or server application and the AI Gateway. The best part? Even if your chosen provider doesn’t support WebSockets, we handle it for you, managing the requests to your preferred AI provider.

通过此 API，AI 网关支持的所有 AI 提供程序都可以通过 WebSocket 进行访问，从而允许您在客户端或服务器应用程序与 AI 网关之间维护单个 TCP 连接。最好的部分？即使您选择的提供商不支持 WebSockets，我们也会为您处理，管理对您首选 AI 提供商的请求。

By connecting via WebSocket to AI Gateway, we make the requests to the inference service for you using the provider’s supported protocols (HTTPS, WebSocket, etc.), and you can keep the connection open to execute as many inference requests as you would like.

通过 WebSocket 连接到 AI Gateway，我们使用提供商支持的协议（HTTPS、WebSocket 等）为您向推理服务发出请求，您可以保持连接打开以执行任意数量的推理请求。

To make your connection to AI Gateway more secure, we are also introducing authentication for AI Gateway. The new WebSockets API will require authentication. All you need to do is create a Cloudflare API token with the permission “AI Gateway: Run” and send that in the cf-aig-authorization header.

为了让您与AI网关的连接更加安全，我们还引入了AI网关的身份验证功能。新的 WebSockets API 将需要身份验证。您所需要做的就是创建一个具有“AI Gateway: Run”权限的 Cloudflare API 令牌，并将其发送到 cf-aig-authorization 标头中。

In the flow diagram above:

在上面的流程图中：

1. When Authenticated Gateway is enabled and a valid token is included, requests will pass successfully.

1. 当启用身份验证网关并包含有效令牌时，请求将成功通过。

2. If Authenticated Gateway is enabled, but a request does not contain the required cf-aig-authorization header with a valid token, the request will fail. This ensures only verified requests pass through the gateway.

2. 如果启用了身份验证网关，但请求不包含具有有效令牌的所需 cf-aig-authorization 标头，则请求将失败。这确保只有经过验证的请求才能通过网关。

3. When Authenticated Gateway is disabled, the cf-aig-authorization header is bypassed entirely, and any token — whether valid or invalid — is ignored.

3. 禁用经过身份验证的网关时，将完全绕过 cf-aig-authorization 标头，并且忽略任何令牌（无论有效还是无效）。

How we built it

我们是如何构建它的

We recently used Durable Objects (DOs) to scale our logging solution for AI Gateway, so using WebSockets within the same DOs was a natural fit.

我们最近使用持久对象 (DO) 来扩展 AI 网关的日志记录解决方案，因此在同一 DO 中使用 WebSocket 是很自然的选择。

When a new WebSocket connection is received by our Cloudflare Workers, we implement authentication in two ways to support the diverse capabilities of WebSocket clients. The primary method involves validating a Cloudflare API token through the cf-aig-authorization header, ensuring the token is valid for the connecting account and gateway.

当我们的 Cloudflare Workers 收到新的 WebSocket 连接时，我们以两种方式实施身份验证，以支持 WebSocket 客户端的不同功能。主要方法涉及通过 cf-aig-authorization 标头验证 Cloudflare API 令牌，确保该令牌对于连接帐户和网关有效。

However, due to limitations in browser WebSocket implementations, we also support authentication via the “sec-websocket-protocol” header. Browser WebSocket clients don't allow for custom headers in their standard API, complicating the addition of authentication tokens in requests. While we don’t recommend that you store API keys in a browser, we decided to add this method to add more flexibility to all WebSocket clients.

但是，由于浏览器 WebSocket 实现的限制，我们还支持通过“sec-websocket-protocol”标头进行身份验证。浏览器 WebSocket 客户端不允许在其标准 API 中使用自定义标头，这使得在请求中添加身份验证令牌变得复杂。虽然我们不建议您在浏览器中存储 API 密钥，但我们决定添加此方法以便为所有 WebSocket 客户端增加更多灵活性。

After this initial verification step, we upgrade the connection to the Durable Object, meaning that it will now handle all the messages for the connection. Before the new connection is fully accepted, we generate a random UUID, so this connection is identifiable among all the messages received by the Durable Object. During an open connection, any AI Gateway settings passed via headers — such as cf-aig-skip-cache (which bypasses caching when set to true) — are stored and applied to all requests in the session. However, these headers can still be overridden on a per-request basis, just like with the Universal Endpoint today.

在这个初始验证步骤之后，我们将连接升级到持久对象，这意味着它现在将处理该连接的所有消息。在新连接被完全接受之前，我们会生成一个随机的 UUID，因此该连接在持久对象接收到的所有消息中都是可识别的。在开放连接期间，通过标头传递的任何 AI 网关设置（例如 cf-aig-skip-cache（设置为 true 时绕过缓存））都会被存储并应用于会话中的所有请求。但是，这些标头仍然可以根据每个请求进行覆盖，就像今天的通用端点一样。

How it works

它是如何运作的

Once the connection is established, the Durable Object begins listening for incoming messages. From this point on, users can send messages in the AI Gateway universal format via WebSocket, simplifying the transition of your application from an existing HTTP setup to WebSockets-based communication.

一旦建立连接，持久对象就开始侦听传入消息。从此时起，用户可以通过 WebSocket 以 AI Gateway 通用格式发送消息，从而简化应用程序从现有 HTTP 设置到基于 WebSocket 的通信的转换。

When a new message reaches the Durable Object, it’s processed using the same code that powers the HTTP Universal Endpoint, enabling seamless code reuse across Workers and Durable Objects — one of the key benefits of building on Cloudflare.

当新消息到达持久对象时，将使用支持 HTTP 通用端点的相同代码对其进行处理，从而实现跨工作线程和持久对象的无缝代码重用——这是在 Cloudflare 上构建的主要优势之一。

For non-streaming requests, the response is wrapped in a JSON envelope, allowing us to include additional information beyond the AI inference itself, such as the AI Gateway log ID for that request.

对于非流式请求，响应被包装在 JSON 信封中，使我们能够包含 AI 推理本身之外的其他信息，例如该请求的 AI 网关日志 ID。

Here’s an example response for the request above:

以下是上述请求的响应示例：

For streaming requests, AI Gateway sends an initial message with request metadata telling the developer the stream is starting.

对于流请求，AI Gateway 会发送一条初始消息，其中包含请求元数据，告诉开发人员流正在启动。

After this initial message, all streaming chunks are relayed in real-time to the WebSocket connection as they arrive from the inference provider. Note that only the eventId field is included in the metadata for these streaming chunks (more info on what this new field is below).

在此初始消息之后，所有流数据块在从推理提供程序到达时都会实时中继到 WebSocket 连接。请注意，这些流数据块的元数据中仅包含 eventId 字段（有关此新字段的更多信息如下）。

This approach serves two purposes:

这种方法有两个目的：