Cloudflare exposed data between customers on Workers AI's GLM-5.3: an SGLang bug and a misconfiguration

On October 26, Cloudflare published an incident report that anyone serving language models to multiple customers on the same infrastructure should read. Between August 27 and September 9, 2026, the Workers AI GLM-5.3 model (@cf/zai-org/glm-5.3) returned responses to some customers that contained data from other customers' requests made to the same model shortly before. According to Cloudflare, 155 responses were affected, all to prompts longer than 681,280 tokens. The cause was a combination of a misconfiguration on their side and a bug in SGLang, the open-source inference engine they use. The report came out as a coordinated disclosure with the SGLang team, which published its own technical analysis the same day.

What happened, in short

On September 8, researcher Gadi Evron, from Knostic, reported through Cloudflare's HackerOne program that very long prompts sent to GLM-5.3 sometimes returned text from other users' sessions. Cloudflare reproduced the problem the next morning, declared a security incident, made the model private (requests started returning HTTP 400), and restored it that same afternoon with the corrected configuration and a patch to the engine.

Cloudflare says that, excluding its own tests and the researcher's, the affected requests match in timing and volume what you'd expect from normal long-context usage, and that it found no evidence of deliberate exploitation. It also states that no other Workers AI model was exposed, because in the rest of its deployments the cache pool is larger than the context window.

Two bugs found in the worst possible place

Workers AI serves GLM-5.3 with SGLang in disaggregated mode: one group of GPUs does the prefill (processes the prompt and generates the KV cache, the attention keys and values the model needs to keep generating) and another group does the decode (generates the response). The cache travels from one group to the other over RDMA, using Mooncake. That cache lives in a shared pool of fixed-size pages that are not zeroed out between requests: the next request that gets a page simply overwrites it. Cloudflare explains that this is common among inference engines because it's necessary for performance, but it leaves all the responsibility for knowing whose page is whose on the engine.

Bug one: the configuration. Cloudflare announced GLM-5.3 with a context window of 1,048,576 tokens, but the KV cache pool on its prefill GPUs only had room for 681,280. SGLang calculates that pool at startup from the memory left over after loading the weights; the context window is a separate value and nothing forces them to match. Cloudflare admits it knew about the difference before launch, but assumed the engine would reject anything that didn't fit with an error. Its long-context tests only checked HTTP status codes and performance, not whether the response made sense, and they all returned 200.

Bug two: the SGLang bug. When a prompt didn't fit in the pool, the scheduler marked it as rejected and replaced it with a single-token placeholder, so the request could go through the normal flow without reserving memory. On a single node, that worked: the client received an error. But in disaggregated mode, the admission queues checked whether the request was finished, not whether it was rejected, and a rejected request kept being reported as alive. The result, according to the sequence Cloudflare describes:

• The decode reserved pages for the whole prompt (8 in their example).
• The prefill processed only the placeholder token and sent one page.
• The decode received "transfer complete" and generated the response by reading all 8 pages, even though 7 still held what earlier requests had left behind: prompts and partial generations from other users.
• The client received HTTP 200 instead of 400, with a coherent response that arrived faster than usual and contained someone else's content.

Cloudflare points out that another user had already reported this bug in SGLang as issue #30233 on July 6, 2026, and that they only saw it once they investigated the incident.

How they fixed it

Cloudflare attacked both fronts. First it adjusted the GLM-5.3 configuration to match GLM-5.2's, which freed up enough memory for the prefill pool to hold around 2.04 million tokens, roughly double the context window. That way, any valid prompt fits and never reaches the rejection path.

Since relying on pool size is fragile (any change in quantization, tensor parallelism, or model version moves it), it also patched the engine so that a rejected request is removed at the admission queue, with its original error, before any reservation or transfer. It proposed this upstream as PR #38704. The SGLang maintainers ended up merging two changes: #36988, which removes aborted requests before their result reaches the decode, and #38935, which rejects them at admission following Cloudflare's approach.

The part that goes beyond Cloudflare

The most useful thing about the report, in my reading, is that this bug belongs to a larger family. Both Cloudflare and the SGLang team describe another race condition in disaggregated deployments: if a request is aborted while the prefill is still writing its cache over RDMA, those writes already in flight can't be canceled. If the decode frees the pages right away and assigns them to another request, the late writes land in that other request's cache. SGLang says that, within disaggregated deployments, the default configurations were exposed to this race from the v0.4.x line (early 2025) through v0.5.21, or v0.5.22 if you use MORI as the transfer backend. It adds that the risk grows with long context, that it leaves no errors or log lines, and that an operator has no reliable way of knowing whether it has already happened to them.

The main fix is deferred release: the decode holds on to the pages of an aborted request until each prefill rank confirms it has no writes in flight, or until a timeout expires. According to SGLang's tests on 2×H100, the throughput impact was statistically indistinguishable from not having it, with differences within ±2%.

What you should do

If you use Workers AI: according to Cloudflare, the problem was limited to GLM-5.3, only affected prompts of more than 681,280 tokens, and was fixed on September 9. Neither Cloudflare's report nor SGLang's says whether they notified the affected customers directly.

If you run SGLang in disaggregated prefill/decode mode, Cloudflare recommends:

1. Compare the KV pool size (it shows up in the startup log) with the context window on every node, after any change in weights, quantization, parallelism, or memory fraction. If the pool is smaller, lower the window. Don't assume the engine will reject whatever doesn't fit.
2. Upgrade to SGLang v0.5.22 or later, or merge #36988 and #38935.
3. Run with deferred release in both roles (it's the default behavior since v0.5.21) and limit Mooncake batch sizes.
4. Enable corruption detection where the model supports it, with --disaggregation-enable-kv-checksum and --kv-canary. Heads up: they don't cover every architecture yet; Cloudflare says it's working with SGLang to extend them to more, including GLM 5.3, GLM 5.3 Flash, and DeepSeek V4. Check your logs to confirm requests are actually being verified.
5. Configure the proxy or load balancer to let requests proceed until the first token and not generate unnecessary aborts mid-transfer.

SGLang adds that, after upgrading, it's worth watching the metric sglang:decode_deferred_kv_release_total{outcome}: in a healthy deployment almost all holds should end as drained, not by timeout. If you serve on a single node, with no cache transfer between engines, this particular race doesn't apply to you.

My take

The uncomfortable thing about this case is that nothing blew up. There were no 500 errors or alerts, and the response was coherent. The lesson Cloudflare itself admits is that testing an AI system only with status codes and latency isn't enough: you have to validate the content of the output, for example with canary markers like the ones they used to confirm the fix. For anyone building their own multi-tenant inference infrastructure, the rule almost writes itself: isolation between customers can't be delegated to "the secure engine handles it." You have to measure it.

Long term, Cloudflare says it's working with SGLang on per-page tags that get verified on every read (their prototype cost less than 1% of performance, but they disabled it because of reliability issues) and on formal verification of the RDMA transfer protocol. We'll have to see how much of that makes it to production.

Sources

• Cloudflare: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
• LMSYS / SGLang: SGLang security update: improving request isolation
• SGLang v0.5.22 on GitHub
• Cloudflare Status: Workers AI increased errors (September 9, 2026)