Navigating the LLM Wild West: Dynamic Rate Limit Management Across 45+ Providers

The Problem: The LLM Wild West of Rate Limits

Building Dronzer AI Gateway was always about abstracting away the complexity of integrating with a multitude of LLM providers. We aimed for zero vendor lock-in and robust failover across 45+ different APIs, including OpenAI, Anthropic, Groq, Google Gemini, and Mistral. The idea was simple: if one provider goes down or gets slow, Dronzer seamlessly routes your request to another. Sounds great on paper.

Then reality hit. Each of these 45+ providers has its own set of rate limits. Requests Per Minute (RPM), Tokens Per Minute (TPM), concurrent requests — you name it, they've got a limit for it. And these limits aren't static. They vary by API key, by subscription tier, by time of day, and sometimes, it feels like they vary just because the wind changed direction. Some providers are explicit, others are vague, and some don't document them at all, forcing you to discover them through trial and error.

Our initial approach was to either hardcode the published limits or use simple exponential backoff on 429 Too Many Requests errors. This worked fine for low traffic or when hitting a single provider. But as Dronzer grew and started handling significant throughput across many providers simultaneously, it became a nightmare.

We'd see cascades of 429 errors. One instance of Dronzer would hit a limit, back off, but another instance, completely unaware, would continue hammering the same provider with the same API key, quickly exhausting its capacity. Our failover logic, while sound for outages, struggled with rate limits because all providers could potentially be rate-limited if not managed carefully. The result was degraded service, increased latency, and a frustrating experience for our users who expected a smooth, uninterrupted AI experience. We were effectively spending money on API calls that were just getting rejected, eating into our operational budget and hurting our reliability metrics.

Hardcoding was clearly unsustainable. Manual adjustments were impossible at scale. We needed a system that was adaptive, distributed, and intelligent enough to navigate this ever-changing landscape of provider limitations without constant human intervention.

Root Cause Analysis: Why Static Limits Fail in a Dynamic World

The core issue was a mismatch between a dynamic, unpredictable external environment (LLM providers' rate limits) and our static, reactive internal mechanisms. Here’s a breakdown of why our initial approach fell short:

1. Brittleness of Hardcoded Limits: Even when a provider published limits, they were often generic. Actual limits could vary based on the specific endpoint, the model being used, or even internal provider-side load. Hardcoding these meant we were always playing catch-up, or worse, over-provisioning and under-utilizing. Any change on the provider's side required a code change and redeployment, which was slow and error-prone.

2. Lack of Real-time Adaptability: Static limits don't account for dynamic usage patterns. If a provider temporarily lowers its limits due to an internal issue, our system would continue to send traffic at the 'known' rate until we started seeing 429s, by which point it was already too late. Conversely, if a provider increased its limits, we wouldn't immediately take advantage of the increased capacity.

3. Ineffective Simple Retry Logic: Our initial retry mechanism for 429s was basic exponential backoff. While necessary, it was insufficient. If an instance hit a 429, it would back off, but other instances of Dronzer (running in a distributed environment) would have no knowledge of this event. They'd continue sending requests, effectively creating a distributed denial-of-service against the provider's rate limit, leading to prolonged service degradation.

4. No Global Coordination: Dronzer runs across multiple instances for high availability and scalability. Each instance operated independently regarding rate limits. This meant that while one instance might have a token bucket for a specific API key, other instances would have their own, separate buckets. There was no shared state, leading to race conditions and simultaneous exhaustion of limits across the cluster.

5. Failover Limitations: Our failover mechanism was designed for complete provider outages or severe latency. While it could detect a provider returning 429s and route traffic elsewhere, it didn't prevent the 429s in the first place, nor did it dynamically adjust the rate at which we tried the failed provider again. It was a blunt instrument for a nuanced problem.

We needed a system that could centralize rate limit knowledge, dynamically adjust based on real-time feedback, and coordinate those adjustments across all running instances of Dronzer. This pointed towards a distributed, adaptive rate limiting architecture.

The Solution: A Distributed, Adaptive Token Bucket System

To tackle this, I engineered a multi-layered solution centered around a distributed, adaptive token bucket algorithm, coordinated through Redis. Here’s how it works:

1. Centralized Rate Limit Registry

First, we needed a single source of truth for all known and observed rate limits. This wasn't just for hardcoded values but for dynamically adjusted ones. I created a simple PostgreSQL table to store provider_id, api_key_hash, limit_type (e.g., rpm, tpm), initial_rate, current_rate, and last_updated_at. This allowed us to persist learned limits and use them as a starting point even after a service restart.

2. Adaptive Distributed Token Bucket Implementation

The core of the solution is a custom AsyncTokenBucket class. This isn't just a local token bucket; its state (tokens available, last refill time, current rate) is stored and updated in Redis, making it accessible and consistent across all Dronzer instances.

The AsyncTokenBucket operates on the principle of capacity (max tokens) and fill_rate (tokens per second). Every time a request comes in, it attempts to consume a token. If tokens are available, the request proceeds. If not, it waits or fails over.

Here's a simplified version of the AsyncTokenBucket:

# core/rate_limiter.py

import time
import asyncio
from typing import Optional

import redis.asyncio as redis
from pydantic import BaseModel, Field

class RateLimitState(BaseModel):
    current_tokens: float = Field(..., description="Current tokens in the bucket")
    last_refill_time: float = Field(..., description="Timestamp of the last token refill")
    fill_rate: float = Field(..., description="Tokens per second")
    capacity: float = Field(..., description="Maximum tokens in the bucket")
    provider_id: str
    api_key_hash: str
    limit_type: str

class AsyncTokenBucket:
    REDIS_KEY_PREFIX = "dronzer:rate_limit:bucket"
    REDIS_CHANNEL_PREFIX = "dronzer:rate_limit:updates"

    def __init__(self, 
                 redis_client: redis.Redis, 
                 provider_id: str, 
                 api_key_hash: str, 
                 limit_type: str, 
                 initial_rate: float, 
                 capacity: float):
        self.redis = redis_client
        self.provider_id = provider_id
        self.api_key_hash = api_key_hash
        self.limit_type = limit_type
        self._initial_rate = initial_rate # Store original for reset/reference
        self._capacity = capacity
        self._key = f"{self.REDIS_KEY_PREFIX}:{provider_id}:{api_key_hash}:{limit_type}"
        self._channel = f"{self.REDIS_CHANNEL_PREFIX}:{provider_id}:{api_key_hash}:{limit_type}"

    async def _get_state(self) -> RateLimitState:
        state_data = await self.redis.get(self._key)
        if state_data:
            return RateLimitState.model_validate_json(state_data)
        
        # Initialize if not found
        initial_state = RateLimitState(
            current_tokens=self._capacity,
            last_refill_time=time.time(),
            fill_rate=self._initial_rate,
            capacity=self._capacity,
            provider_id=self.provider_id,
            api_key_hash=self.api_key_hash,
            limit_type=self.limit_type
        )
        await self.redis.set(self._key, initial_state.model_dump_json())
        return initial_state

    async def _save_state(self, state: RateLimitState):
        await self.redis.set(self._key, state.model_dump_json())

    async def consume(self, tokens: float = 1.0, timeout: Optional[float] = None) -> bool:
        start_time = time.time()
        while True:
            async with self.redis.lock(f"{self._key}:lock", timeout=5):
                state = await self._get_state()
                now = time.time()
                
                # Refill tokens
                time_passed = now - state.last_refill_time
                refilled_tokens = time_passed * state.fill_rate
                state.current_tokens = min(state.capacity, state.current_tokens + refilled_tokens)
                state.last_refill_time = now

                if state.current_tokens >= tokens:
                    state.current_tokens -= tokens
                    await self._save_state(state)
                    return True
                else:
                    if timeout is None or (now - start_time) > timeout:
                        return False # No tokens, and no timeout or timeout reached
                    
                    # Calculate time to wait for next token
                    tokens_needed = tokens - state.current_tokens
                    time_to_wait = tokens_needed / state.fill_rate if state.fill_rate > 0 else float('inf')
                    
                    await self._save_state(state) # Save current state before waiting
                    await asyncio.sleep(min(time_to_wait + 0.01, timeout - (now - start_time) if timeout else time_to_wait + 0.01))
                    
            if timeout and (time.time() - start_time) > timeout:
                return False

    async def adjust_rate(self, new_rate: float, capacity_factor: float = 1.0):
        # Ensure capacity scales with rate to avoid immediate exhaustion
        # We make capacity proportional to the rate, so higher rates allow for bigger bursts
        new_capacity = new_rate * capacity_factor * 10 # Example: 10 seconds worth of tokens as burst capacity

        async with self.redis.lock(f"{self._key}:lock", timeout=5):
            state = await self._get_state()
            state.fill_rate = max(0.1, new_rate) # Ensure rate doesn't drop to zero
            state.capacity = max(1.0, new_capacity)
            state.current_tokens = min(state.current_tokens, state.capacity) # Cap current tokens if capacity reduced
            await self._save_state(state)
        
        # Publish update to other instances
        update_message = {
            "provider_id": self.provider_id,
            "api_key_hash": self.api_key_hash,
            "limit_type": self.limit_type,
            "new_rate": new_rate,
            "new_capacity": new_capacity
        }
        await self.redis.publish(self._channel, RateLimitState.model_dump_json(update_message))

    async def get_current_state(self) -> RateLimitState:
        return await self._get_state()

Key aspects of this implementation:

• Redis for State: The _get_state and _save_state methods read and write the RateLimitState to Redis. This makes the bucket state shared across all Dronzer instances.
• Distributed Lock: self.redis.lock ensures that only one Dronzer instance updates a specific bucket's state at any given moment, preventing race conditions when multiple instances try to consume or update tokens concurrently.
• Refill Logic: Tokens are refilled based on the fill_rate and the time elapsed since the last_refill_time. The current_tokens never exceed capacity.
• consume with Timeout: Requests can wait for tokens up to a specified timeout. If no tokens become available, it returns False, allowing for immediate failover.
• adjust_rate: This method is critical. When a 429 is detected, we call this to dynamically reduce the fill_rate for that specific provider/API key. I also ensure that capacity scales with fill_rate to maintain a proportional burst allowance.

3. Dynamic Limit Learning and Adjustment

This is where the 'adaptive' part comes in. When Dronzer makes an API call to an LLM provider using httpx, it wraps the call with error handling that specifically looks for 429 Too Many Requests status codes. If a 429 is received, we trigger a rate adjustment.

# core/llm_client.py (simplified)

import httpx
import logging
from core.rate_limiter import AsyncTokenBucket # Assuming AsyncTokenBucket is defined
from core.config import settings # For default retry values

logger = logging.getLogger(__name__)

class LLMProviderClient:
    def __init__(self, 
                 base_url: str, 
                 api_key: str, 
                 rate_limiter: AsyncTokenBucket):
        self.base_url = base_url
        self.api_key = api_key
        self.rate_limiter = rate_limiter
        self.client = httpx.AsyncClient(base_url=base_url, headers={"Authorization": f"Bearer {api_key}"})

    async def make_request(self, endpoint: str, data: dict, max_retries: int = 3, initial_backoff: float = 1.0) -> httpx.Response:
        retries = 0
        while retries <= max_retries:
            # First, try to consume a token
            if not await self.rate_limiter.consume(timeout=settings.LLM_RATE_LIMIT_WAIT_TIMEOUT):
                logger.warning(f"Rate limit exceeded for {self.rate_limiter.provider_id} (type: {self.rate_limiter.limit_type}). Failing over or waiting.")
                # Here you'd integrate with Dronzer's failover logic
                raise httpx.RequestError(f"Rate limit token unavailable for {self.rate_limiter.provider_id}")

            try:
                response = await self.client.post(endpoint, json=data, timeout=settings.LLM_API_TIMEOUT)
                response.raise_for_status() # Raises HTTPStatusError for 4xx/5xx responses
                return response
            except httpx.HTTPStatusError as e:
                if e.response.status_code == 429:
                    logger.warning(f"Received 429 from {self.rate_limiter.provider_id}. Adjusting rate limit and retrying (attempt {retries+1}/{max_retries+1}).")
                    
                    # Reduce the effective rate limit for this bucket
                    current_state = await self.rate_limiter.get_current_state()
                    new_rate = current_state.fill_rate * settings.RATE_LIMIT_DECREASE_FACTOR # e.g., 0.8
                    await self.rate_limiter.adjust_rate(new_rate)

                    # Apply exponential backoff before next retry
                    backoff_delay = initial_backoff * (2 ** retries)
                    await asyncio.sleep(min(backoff_delay, settings.MAX_RATE_LIMIT_BACKOFF_DELAY))
                    retries += 1
                else:
                    logger.error(f"HTTP error from {self.rate_limiter.provider_id}: {e.response.status_code} - {e.response.text}")
                    raise
            except httpx.RequestError as e:
                logger.error(f"Network or request error to {self.rate_limiter.provider_id}: {e}")
                raise

        logger.error(f"Max retries ({max_retries}) exhausted for {self.rate_limiter.provider_id}.")
        raise httpx.RequestError(f"Max retries exhausted for {self.rate_limiter.provider_id}")

This make_request method first attempts to consume a token. If it can't, it raises an error, allowing Dronzer's routing logic to pick another provider. If a 429 is received, it immediately adjusts the fill_rate downwards for that specific bucket and applies an exponential backoff before retrying. The RATE_LIMIT_DECREASE_FACTOR (e.g., 0.8) ensures a controlled reduction.

4. Distributed Coordination with Redis Pub/Sub

The adjust_rate method in AsyncTokenBucket doesn't just update the local Redis key; it also publishes a message to a Redis Pub/Sub channel. All other Dronzer instances are subscribed to this channel. When they receive an update, they immediately adjust their local view of that specific token bucket's rate and capacity.

# main.py or wherever Dronzer initializes

import asyncio
import redis.asyncio as redis
import json
from core.rate_limiter import AsyncTokenBucket, RateLimitState

redis_client = redis.Redis(host='localhost', port=6379, db=0)

async def rate_limit_listener():
    pubsub = redis_client.pubsub()
    await pubsub.psubscribe(f"{AsyncTokenBucket.REDIS_CHANNEL_PREFIX}:*") # Subscribe to all rate limit update channels

    async for message in pubsub.listen():
        if message['type'] == 'pmessage':
            try:
                data = json.loads(message['data'].decode('utf-8'))
                provider_id = data['provider_id']
                api_key_hash = data['api_key_hash']
                limit_type = data['limit_type']
                new_rate = data['new_rate']
                new_capacity = data['new_capacity']

                # Re-instantiate a bucket object to update its state
                # In a real system, you'd fetch the existing bucket instance from a registry
                temp_bucket = AsyncTokenBucket(
                    redis_client=redis_client,
                    provider_id=provider_id,
                    api_key_hash=api_key_hash,
                    limit_type=limit_type,
                    initial_rate=new_rate, # Use new_rate as initial for this update
                    capacity=new_capacity
                )
                
                # Acquire lock and update state directly, bypassing publish loop
                async with redis_client.lock(f"{temp_bucket._key}:lock", timeout=5):
                    state = await temp_bucket._get_state()
                    state.fill_rate = new_rate
                    state.capacity = new_capacity
                    state.current_tokens = min(state.current_tokens, state.capacity)
                    await temp_bucket._save_state(state)
                    logger.info(f"Received and applied rate limit update for {provider_id}/{limit_type}: new rate {new_rate:.2f} tokens/sec.")

            except Exception as e:
                logger.error(f"Error processing rate limit update message: {e}")

# In your FastAPI app startup event:
# @app.on_event("startup")
# async def startup_event():
#     asyncio.create_task(rate_limit_listener())

This listener runs as a background task. When a message is received, it parses the new rate and capacity and applies it to the corresponding AsyncTokenBucket in that Dronzer instance's memory (or rather, updates its representation in Redis). This ensures near real-time synchronization of rate limit adjustments across the entire Dronzer cluster.

5. Integration with Dronzer's Failover Strategy

The AsyncTokenBucket.consume() method returning False (due to no tokens or timeout) is a critical integration point. Instead of immediately retrying the same provider, Dronzer's intelligent routing layer interprets this as a signal to try an alternative, healthy provider. This means that if a provider is consistently rate-limiting, Dronzer will gracefully degrade by routing traffic away from it until its rate limit bucket has recovered, without user intervention.

This forms a powerful feedback loop:

1. Proactive: Requests check the token bucket before hitting the external API, preventing unnecessary 429s.
2. Reactive: If a 429 still occurs (e.g., if the provider's limits changed unexpectedly, or initial limits were too high), the system immediately reduces its perceived rate for that provider.
3. Distributed: All instances learn and adapt simultaneously, preventing a 'thundering herd' problem against an already struggling provider.
4. Resilient: If a provider is persistently rate-limited, Dronzer's failover routes traffic to other available providers, maintaining service availability.

Lessons Learned

This journey into dynamic rate limit management taught me a few crucial lessons about building resilient, distributed systems:

1. Static Configurations Are a Liability in Dynamic Environments: Relying on hardcoded values for external service parameters is a recipe for disaster. Systems need to be able to observe, learn, and adapt to the real-world behavior of their dependencies. This applies not just to rate limits but to latency, error rates, and even API schema changes.

2. Distributed State Management is Hard, But Essential: For truly robust, scalable systems, individual service instances cannot operate in isolation. Shared context, like dynamically adjusted rate limits, requires a reliable, low-latency mechanism for distributed state synchronization. Redis Pub/Sub, combined with atomic operations and locks, proved to be an effective pattern here.

3. Observability Drives Adaptability: You can't adapt to what you can't see. Detailed logging, metrics (tokens consumed, 429s received, rate adjustments), and alerting were indispensable in understanding how our system was interacting with providers and refining the adjustment factors. Without clear visibility into when and why 429s were happening, our adaptive logic would have been blind.

4. Proactive Prevention Trumps Reactive Recovery: While robust retry and failover are non-negotiable, preventing issues before they occur is always better. The token bucket acting as a pre-flight check significantly reduced the number of actual 429 errors we saw, leading to a smoother experience and less wasted API budget.

5. Graceful Degradation is Key: Knowing when to back off, when to retry, and when to completely switch providers is a delicate balance. Building a system that can gracefully degrade performance or shift load away from troubled dependencies ensures that core functionality remains available, even when parts of the external ecosystem are struggling. Our adaptive rate limiting directly fed into Dronzer's existing failover, making it much more intelligent and responsive.

Engineering Dronzer to handle the 'wild west' of LLM provider rate limits was a significant challenge, but it reinforced the importance of building systems that are not just performant, but genuinely resilient and adaptive to the unpredictable nature of external dependencies. It's about building intelligence into the infrastructure itself, so the developers and users can focus on what matters: leveraging AI effectively.