Streaming LLM Responses in Production: Handling Backpressure and Partial Failures

The Problem With Naive Streaming

When I started building Snapfix, the initial pitch was simple: detect a Python traceback in your editor, stream fix suggestions in real time. The user types, the extension buffers the error, and Claude or GPT-4 starts writing a fix while they're still reading the problem.

That sounds straightforward until you realize what happens when the LLM is slow, the user closes the tab mid-stream, or the API quota hits. Naive streaming—just piping the LLM response directly to the client—breaks catastrophically under load or latency spikes.

The core issue: streaming doesn't mean "fast." It means you're sending partial data as it arrives. If the client disconnects, the server keeps generating tokens. If the LLM API rate-limits you, your entire request queue backs up. If one model is slow, all downstream consumers wait.

Building Backpressure Into the Stream

I modeled the architecture around three principles:

1. The server should never generate faster than the client can consume.
2. A slow client shouldn't starve other requests.
3. Model failures should trigger fallback routing, not cascade.

Here's the FastAPI service layer I built:

from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from litellm import acompletion
import asyncio
from typing import AsyncGenerator
import logging

app = FastAPI()
logger = logging.getLogger(__name__)

class StreamBuffer:
    """Bounded queue with backpressure signaling."""
    def __init__(self, max_size: int = 100):
        self.queue = asyncio.Queue(maxsize=max_size)
        self.closed = False
    
    async def put(self, token: str):
        """Put token; blocks if buffer full (backpressure)."""
        if self.closed:
            return
        try:
            await asyncio.wait_for(self.queue.put(token), timeout=5.0)
        except asyncio.TimeoutError:
            logger.warning("Stream buffer timeout; client likely disconnected")
            self.closed = True
    
    async def get(self):
        return await self.queue.get()
    
    def mark_complete(self):
        self.closed = True

async def stream_with_fallback(
    traceback: str,
    primary_model: str = "gpt-4",
    fallback_models: list = None
) -> AsyncGenerator[str, None]:
    """Stream LLM response with automatic fallback on failure."""
    if fallback_models is None:
        fallback_models = ["claude-3-sonnet", "ollama/neural-chat"]
    
    models_to_try = [primary_model] + fallback_models
    buffer = StreamBuffer(max_size=50)
    
    async def generate_tokens():
        """Background task: call LLM and feed buffer."""
        for model in models_to_try:
            try:
                logger.info(f"Attempting {model} for traceback analysis")
                response = await acompletion(
                    model=model,
                    messages=[
                        {"role": "system", "content": "You are a Python debugging assistant. Provide concise fixes."},
                        {"role": "user", "content": f"Fix this error:\n{traceback}"}
                    ],
                    stream=True,
                    timeout=15
                )
                
                async for chunk in response:
                    if chunk.choices[0].delta.content:
                        token = chunk.choices[0].delta.content
                        await buffer.put(token)
                
                buffer.mark_complete()
                logger.info(f"Successfully streamed response from {model}")
                return
            
            except asyncio.TimeoutError:
                logger.warning(f"{model} timeout; trying next")
                continue
            except Exception as e:
                logger.warning(f"{model} failed: {e}; trying next")
                continue
        
        # All models failed
        await buffer.put("[Error: All LLM providers failed. Check your API keys and rate limits.]")
        buffer.mark_complete()
    
    # Start background generation
    generate_task = asyncio.create_task(generate_tokens())
    
    try:
        while True:
            try:
                token = await asyncio.wait_for(buffer.get(), timeout=30)
                if token is None:  # Sentinel for completion
                    break
                yield token
            except asyncio.TimeoutError:
                logger.warning("Client read timeout; ending stream")
                break
    finally:
        buffer.mark_complete()
        if not generate_task.done():
            generate_task.cancel()

@app.post("/fix")
async def get_fix(traceback: str):
    """Endpoint: POST traceback, GET streaming fix."""
    if not traceback or len(traceback) > 5000:
        raise HTTPException(status_code=400, detail="Invalid traceback")
    
    return StreamingResponse(
        stream_with_fallback(traceback),
        media_type="text/event-stream"
    )

Why This Matters

Backpressure via bounded queue: The StreamBuffer has a fixed size. If the client is slow (or disconnected), the buffer fills up. put() blocks, which pauses token generation. This prevents memory bloat and stops the server from burning through tokens on a dead connection.

Timeout on reads: The wait_for(buffer.get(), timeout=30) ensures that if the client vanishes, the stream loop exits cleanly instead of hanging forever.

Fallback routing: If GPT-4 times out, we try Claude, then Ollama. Each attempt gets its own timeout window. The user sees a degraded but functional response instead of a 500 error.

Cancellation cleanup: The finally block cancels the background generation task if the client disconnects early. Without this, the task keeps running and wastes tokens.

The Client Side: Respecting the Stream

The VS Code extension (TypeScript) mirrors this discipline:

async function streamFix(traceback: string): Promise<void> {
  const response = await fetch('/fix', {
    method: 'POST',
    body: JSON.stringify({ traceback }),
    headers: { 'Content-Type': 'application/json' }
  });

  if (!response.body) throw new Error('No response body');
  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      
      const text = decoder.decode(value, { stream: true });
      // Append to editor; UI rendering naturally throttles consumption
      updateFixSuggestion(text);
    }
  } catch (err) {
    if (err.name === 'AbortError') {
      console.log('Stream cancelled by user');
    } else {
      console.error('Stream error:', err);
    }
  } finally {
    reader.releaseLock();
  }
}

The browser's ReadableStream API naturally handles backpressure. If the UI is slow to render, reader.read() waits. The server only generates as fast as the client pulls.

Lessons From Production

After deploying Snapfix, I discovered a few hard truths:

Token budget is real. A single slow client can consume your entire monthly quota if you don't timeout. I set a 15-second per-model limit and a 30-second per-client limit. Beyond that, it's not worth the tokens.

Ollama is a lifeline. When OpenAI or Anthropic APIs are rate-limited, falling back to a local Ollama instance (running on the same VPS) means the extension still works. It's slower, but it works. LiteLLM makes this routing transparent.

Partial responses are better than no responses. If the stream fails after 3 tokens, send those 3 tokens. The user gets something and can retry. A 500 error on an empty response is worse.

Monitor queue depth. If your StreamBuffer is consistently full, your clients are slower than your LLM. This is a signal to reduce model complexity or increase client-side buffering.

Applying This Pattern Elsewhere

This pattern generalizes. In my attendance system, I use similar backpressure when broadcasting GPS updates to 300+ concurrent WebSocket connections. In the lead scraper dashboard, I apply the same timeout-and-fallback logic to Selenium retries.

The principle: streaming is not a throughput optimization. It's a UX pattern that requires careful resource management. Build your streams with bounded buffers, timeouts, and fallback routing from day one. Your future self (and your cloud bill) will thank you.

Next Steps

If you're building streaming LLM integrations, start with a bounded queue and explicit timeouts. Test with slow clients (throttle your network in DevTools). Measure token consumption per request. Add fallback models early—retrofitting them later is painful.

Snapfix is open source on GitHub if you want to see the full implementation. The extension handles streaming, the backend handles backpressure, and LiteLLM handles the routing. It's not fancy, but it works at scale.