使用 WebSocket 和耐用物件進行大規模即時 AI 推理

2024 年 10 月，我們討論了使用 AI Gateway 儲存來自 AI 應用程式的數十億個日誌，以及我們如何使用 Cloudflare 的開發者平台來做到這一點。

In October 2024, we covered how to store billions of logs from your AI application using AI Gateway, and how we used Cloudflare’s Developer Platform to do this.

2024 年 10 月，我們介紹如何使用 AI Gateway 儲存來自 AI 應用程式的數十億條日誌，以及如何使用 Cloudflare 的開發者平台來執行此操作。

With AI Gateway already processing over 3 billion logs and experiencing rapid growth, the number of connections to the platform continues to increase steadily. To help developers manage this scale more effectively, we wanted to offer an alternative to implementing HTTP/2 keep-alive to maintain persistent HTTP(S) connections, thereby avoiding the overhead of repeated handshakes and TLS negotiations with each new HTTP connection to AI Gateway. We understand that implementing HTTP/2 can present challenges, particularly when many libraries and tools may not support it by default and most modern programming languages have well-established WebSocket libraries available.

AI網關已處理超過30億個日誌，且成長迅速，平台連線數持續穩定成長。為了幫助開發人員更有效地管理這種規模，我們希望提供一種替代方案來實現HTTP/2 keep-alive 來維持持久的HTTP(S) 連接，從而避免與AI Gateway 的每個新HTTP 連接重複握手和TLS 協商的開銷。我們知道，實作 HTTP/2 可能會帶來挑戰，特別是當許多函式庫和工具預設可能不支援它，而且大多數現代程式語言都有完善的 WebSocket 函式庫可用時。

With this in mind, we used Cloudflare’s Developer Platform and Durable Objects (yes, again!) to build a WebSockets API that establishes a single, persistent connection, enabling continuous communication.

考慮到這一點，我們使用 Cloudflare 的開發者平台和 Durable Objects（是的，又是！）來建立 WebSockets API，該 API 建立單一持久連接，從而實現持續通訊。

Through this API, all AI providers supported by AI Gateway can be accessed via WebSocket, allowing you to maintain a single TCP connection between your client or server application and the AI Gateway. The best part? Even if your chosen provider doesn’t support WebSockets, we handle it for you, managing the requests to your preferred AI provider.

透過此 API，AI 網關支援的所有 AI 提供者都可以透過 WebSocket 進行訪問，從而允許您在客戶端或伺服器應用程式與 AI 網關之間維護單一 TCP 連線。最好的部分？即使您選擇的提供者不支援 WebSockets，我們也會為您處理，管理對您首選 AI 提供者的請求。

By connecting via WebSocket to AI Gateway, we make the requests to the inference service for you using the provider’s supported protocols (HTTPS, WebSocket, etc.), and you can keep the connection open to execute as many inference requests as you would like.

透過 WebSocket 連接到 AI Gateway，我們使用提供者支援的協定（HTTPS、WebSocket 等）為您向推理服務發出請求，您可以保持連線開啟以執行任意數量的推理請求。

To make your connection to AI Gateway more secure, we are also introducing authentication for AI Gateway. The new WebSockets API will require authentication. All you need to do is create a Cloudflare API token with the permission “AI Gateway: Run” and send that in the cf-aig-authorization header.

為了讓您與AI網關的連線更加安全，我們也引進了AI網關的身份驗證功能。新的 WebSockets API 將需要身份驗證。您所需要做的就是建立一個具有「AI Gateway: Run」權限的 Cloudflare API 令牌，並將其傳送到 cf-aig-authorization 標頭中。

In the flow diagram above:

在上面的流程圖中：

1. When Authenticated Gateway is enabled and a valid token is included, requests will pass successfully.

1. 當啟用身份驗證網關並包含有效令牌時，請求將成功通過。

2. If Authenticated Gateway is enabled, but a request does not contain the required cf-aig-authorization header with a valid token, the request will fail. This ensures only verified requests pass through the gateway.

2. 如果啟用了身份驗證網關，但請求不包含具有有效令牌的所需 cf-aig-authorization 標頭，則請求將失敗。這可確保只有經過驗證的請求才能通過網關。

3. When Authenticated Gateway is disabled, the cf-aig-authorization header is bypassed entirely, and any token — whether valid or invalid — is ignored.

3. 停用經過驗證的網關時，將完全繞過 cf-aig-authorization 標頭，並且忽略任何令牌（無論有效或無效）。

How we built it

我們是如何建構它的

We recently used Durable Objects (DOs) to scale our logging solution for AI Gateway, so using WebSockets within the same DOs was a natural fit.

我們最近使用持久性物件 (DO) 來擴展 AI 閘道的日誌記錄解決方案，因此在同一 DO 中使用 WebSocket 是很自然的選擇。

When a new WebSocket connection is received by our Cloudflare Workers, we implement authentication in two ways to support the diverse capabilities of WebSocket clients. The primary method involves validating a Cloudflare API token through the cf-aig-authorization header, ensuring the token is valid for the connecting account and gateway.

當我們的 Cloudflare Workers 收到新的 WebSocket 連線時，我們以兩種方式實作身份驗證，以支援 WebSocket 用戶端的不同功能。主要方法涉及透過 cf-aig-authorization 標頭驗證 Cloudflare API 令牌，確保該令牌對於連接帳戶和網關有效。

However, due to limitations in browser WebSocket implementations, we also support authentication via the “sec-websocket-protocol” header. Browser WebSocket clients don't allow for custom headers in their standard API, complicating the addition of authentication tokens in requests. While we don’t recommend that you store API keys in a browser, we decided to add this method to add more flexibility to all WebSocket clients.

但是，由於瀏覽器 WebSocket 實現的限制，我們也支援透過「sec-websocket-protocol」標頭進行身份驗證。瀏覽器 WebSocket 用戶端不允許在其標準 API 中使用自訂標頭，這使得在請求中新增身份驗證令牌變得複雜。雖然我們不建議您在瀏覽器中儲存 API 金鑰，但我們決定新增此方法以便為所有 WebSocket 用戶端增加更多靈活性。

After this initial verification step, we upgrade the connection to the Durable Object, meaning that it will now handle all the messages for the connection. Before the new connection is fully accepted, we generate a random UUID, so this connection is identifiable among all the messages received by the Durable Object. During an open connection, any AI Gateway settings passed via headers — such as cf-aig-skip-cache (which bypasses caching when set to true) — are stored and applied to all requests in the session. However, these headers can still be overridden on a per-request basis, just like with the Universal Endpoint today.

在這個初始驗證步驟之後，我們將連線升級到持久對象，這意味著它現在將處理該連線的所有訊息。在新連線被完全接受之前，我們會產生一個隨機的 UUID，因此該連線在持久性物件接收到的所有訊息中都是可識別的。在開放連線期間，透過標頭傳遞的任何 AI 閘道設定（例如 cf-aig-skip-cache（設定為 true 時繞過快取））都會被儲存並套用於會話中的所有請求。但是，這些標頭仍然可以根據每個請求進行覆蓋，就像今天的通用端點一樣。

How it works

它是如何運作的

Once the connection is established, the Durable Object begins listening for incoming messages. From this point on, users can send messages in the AI Gateway universal format via WebSocket, simplifying the transition of your application from an existing HTTP setup to WebSockets-based communication.

一旦建立連接，持久性物件就開始偵聽傳入訊息。從此時起，使用者可以透過 WebSocket 以 AI Gateway 通用格式傳送訊息，從而簡化應用程式從現有 HTTP 設定到基於 WebSocket 的通訊的轉換。

When a new message reaches the Durable Object, it’s processed using the same code that powers the HTTP Universal Endpoint, enabling seamless code reuse across Workers and Durable Objects — one of the key benefits of building on Cloudflare.

當新訊息到達持久性物件時，將使用支援 HTTP 通用端點的相同程式碼對其進行處理，從而實現跨工作執行緒和持久物件的無縫程式碼重用——這是在 Cloudflare 上建置的主要優勢之一。

For non-streaming requests, the response is wrapped in a JSON envelope, allowing us to include additional information beyond the AI inference itself, such as the AI Gateway log ID for that request.

對於非串流請求，回應被包裝在 JSON 信封中，使我們能夠包含 AI 推理本身之外的其他信息，例如該請求的 AI 網關日誌 ID。

Here’s an example response for the request above:

以下是上述請求的回應範例：

For streaming requests, AI Gateway sends an initial message with request metadata telling the developer the stream is starting.

對於串流請求，AI Gateway 會傳送初始訊息，其中包含請求元數據，告訴開發人員流正在啟動。

After this initial message, all streaming chunks are relayed in real-time to the WebSocket connection as they arrive from the inference provider. Note that only the eventId field is included in the metadata for these streaming chunks (more info on what this new field is below).

在此初始訊息之後，所有流資料區塊在從推理提供者到達時都會即時中繼到 WebSocket 連線。請注意，這些流資料區塊的元資料中僅包含 eventId 欄位（有關此新欄位的詳細資訊如下）。

This approach serves two purposes:

這種方法有兩個目的：