Architecting Dynamic LLM Routing: Intelligent Model Selection in Production

When you're building serious applications with large language models, simply picking one provider and sticking with it often isn't enough. The landscape is moving fast. New models emerge, prices change, and different LLMs excel at different tasks. For a while, the conversation was about integrating multiple providers to avoid vendor lock-in. That's a good start, but it's only half the battle. The real challenge comes with intelligently deciding which LLM to use for each specific request.

I've spent a lot of time wrestling with this while building Dronzer AI Gateway, an LLM orchestration platform designed to route requests across dozens of providers. The goal isn't just multi-provider support; it's task-aware intelligence and bulletproof failover. This means the system needs to dynamically select the optimal model based on a complex interplay of factors like cost, latency, capability, and even real-time provider status.

This isn't a simple if/else statement. It's an architectural problem that demands a robust solution. Let's break down how to build a truly intelligent LLM routing system.

The Problem with Static Routing

Many initial multi-LLM integrations start with static routing. You might have a configuration that says: "For creative writing, use Claude. For code generation, use GPT-4. For simple summarization, use a cheaper, smaller model." This works for basic use cases, but it quickly falls apart in a dynamic production environment for several reasons:

1. Cost Inefficiency: A premium model might be overkill for a simple prompt, leading to unnecessary expenses. Conversely, a cheaper model might perform poorly on a critical, complex task, requiring re-runs or human intervention.
2. Latency Issues: Some models are faster than others. For real-time user-facing applications, latency is paramount. Statically routing a high-latency model to a time-sensitive request degrades user experience.
3. Capability Drift: LLM capabilities evolve. A model that was best for a task last month might not be today. New, more specialized models emerge, or existing ones get updates. Static configurations don't adapt.
4. Context Window Limitations: Different models have different context length limits. A long prompt might fail on a model with a smaller window but succeed on another, more capable one.
5. Provider Outages & Rate Limits: APIs go down. Providers impose rate limits. A static setup means your application breaks when its designated provider fails, even if other perfectly capable models are available.
6. Lack of Experimentation: Without a dynamic layer, A/B testing new models or fine-tuned versions in production is cumbersome and risky.

We needed something that could look at an incoming request, understand its requirements, and then consult a dynamic registry of available models to make the best decision in milliseconds. That's what intelligent routing is all about.

Defining "Intelligent" Routing: Key Factors

To build an intelligent routing system, we first need to define what "intelligent" means in this context. It means making an informed decision based on several critical factors:

• Cost: The monetary cost per token (input and output) or per request. This is often a primary driver for optimization.
• Latency: The typical response time of the model. Crucial for real-time applications.
• Capability / Quality: How well a model performs on specific task types (e.g., reasoning, code, creativity, summarization). This can be subjective but can also be informed by internal evaluations or benchmarks.
• Context Length: The maximum number of tokens a model can handle in a single request.
• Availability / Reliability: The current operational status of the model's provider. Is it up? Is it healthy? Are there known issues?
• Special Features: Does the model support specific features like function calling, JSON mode, or multi-modal inputs?.
• User / Application Tier: For internal tools or multi-tier applications, higher-priority users or critical application paths might justify using more expensive, higher-quality models.

Architecting the Dynamic Model Selection Engine

Building this requires a few core components:

1. Request Metadata Capture: Enriching incoming prompts with critical information.
2. Model Registry: A real-time, centralized source of truth for all LLM options.
3. Decision Engine: The core logic that consumes metadata and registry data to select a model.
4. Adaptive Strategies: Mechanisms for failover, load balancing, and dynamic adjustments.

1. Request Metadata Capture

Every incoming request needs to carry enough information for the decision engine to work. This isn't just the user's prompt; it's metadata about the intent and constraints of the request.

For example, when a user asks Snapfix (my VS Code extension) for a code fix, the system knows it's a code_generation task. If it's for a quick docstring, it might be summarization. If a user is on a "premium" tier, they might get access to higher-quality models.

from typing import Literal, Optional, Dict, Any
from pydantic import BaseModel

class RequestContext(BaseModel):
    task_type: Literal[
        "code_generation", 
        "creative_writing", 
        "summarization", 
        "qa", 
        "function_call"
    ]
    priority_tier: Literal["low", "medium", "high", "premium"]
    max_latency_ms: Optional[int] = None
    max_cost_per_request: Optional[float] = None
    required_features: list[str] = []
    input_token_count: int
    # Any other relevant context, e.g., user ID, source application
    metadata: Dict[str, Any] = {}

# Example usage:
example_context = RequestContext(
    task_type="code_generation",
    priority_tier="high",
    max_latency_ms=5000, # 5 seconds
    input_token_count=250,
    required_features=["function_calling"]
)

This RequestContext object becomes the input to our decision engine.

2. The Model Registry

This is the backbone of dynamic routing. It's a database (or a cached in-memory structure) that holds all the up-to-date information about every LLM we can access. For Dronzer, this is critical because we support over 45 providers.

Each entry in the registry needs to include:

• Model ID / Name: Unique identifier.
• Provider: OpenAI, Anthropic, Google, Groq, Mistral, Ollama, etc.
• Base Cost: Input and output token pricing.
• Average Latency: Historical data, updated regularly.
• Capabilities: Which task_type it excels at, or specific features it supports (e.g., supports_json_mode).
• Context Window: Max tokens.
• Status: Healthy, degraded, offline. This is crucial for real-time failover.
• Tier: A predefined quality/performance tier (e.g., "premium", "standard", "economy").

from datetime import datetime

class LLMModel(BaseModel):
    id: str
    provider: str
    model_name: str
    input_cost_per_million_tokens: float # e.g., $1.00
    output_cost_per_million_tokens: float # e.g., $3.00
    avg_latency_ms: int # Updated via monitoring
    context_window: int # Max tokens
    capabilities: list[str] # e.g., ["code_generation", "function_calling"]
    quality_tier: Literal["premium", "standard", "economy"]
    status: Literal["healthy", "degraded", "offline"] # Updated via health checks
    last_updated: datetime = datetime.utcnow()

class ModelRegistry:
    def __init__(self):
        self._models: Dict[str, LLMModel] = {}
        # In a real system, this would load from a DB and be periodically refreshed
        self._load_initial_models()

    def _load_initial_models(self):
        # Simulate loading from a database or config file
        self._models["gpt-4o"] = LLMModel(
            id="gpt-4o", provider="openai", model_name="gpt-4o",
            input_cost_per_million_tokens=5.00, output_cost_per_million_tokens=15.00,
            avg_latency_ms=800, context_window=128000, capabilities=["code_generation", "creative_writing", "function_calling"],
            quality_tier="premium", status="healthy"
        )
        self._models["claude-3-opus"] = LLMModel(
            id="claude-3-opus", provider="anthropic", model_name="claude-3-opus-20240229",
            input_cost_per_million_tokens=15.00, output_cost_per_million_tokens=75.00,
            avg_latency_ms=1200, context_window=200000, capabilities=["creative_writing", "reasoning"],
            quality_tier="premium", status="healthy"
        )
        self._models["llama-3-8b-it"] = LLMModel(
            id="llama-3-8b-it", provider="groq", model_name="llama-3-8b-it",
            input_cost_per_million_tokens=0.05, output_cost_per_million_tokens=0.10,
            avg_latency_ms=100, context_window=8192, capabilities=["general", "fast_inference"],
            quality_tier="economy", status="healthy"
        )
        self._models["mixtral-8x7b"] = LLMModel(
            id="mixtral-8x7b", provider="mistral", model_name="mistral-large-latest",
            input_cost_per_million_tokens=4.00, output_cost_per_million_tokens=12.00,
            avg_latency_ms=500, context_window=32768, capabilities=["code_generation", "reasoning"],
            quality_tier="standard", status="healthy"
        )

    def get_all_healthy_models(self) -> list[LLMModel]:
        return [model for model in self._models.values() if model.status == "healthy"]

    def update_model_status(self, model_id: str, status: Literal["healthy", "degraded", "offline"]):
        if model_id in self._models:
            self._models[model_id].status = status
            self._models[model_id].last_updated = datetime.utcnow()
            print(f"Updated status for {model_id} to {status}")

# Instantiate the registry (in a real app, this would be a singleton or dependency-injected)
model_registry = ModelRegistry()

The status field is critical. This is where real-time health checks and circuit breaker patterns come into play. If a provider's API starts returning 5xx errors or experiences high latency, its models' statuses in the registry are updated, and the decision engine will automatically route requests away from it.

3. The Decision Engine

This is where the magic happens. The decision engine takes the RequestContext and queries the ModelRegistry to find the best fit. The logic can be simple rule-based or more complex, potentially using heuristics or even machine learning.

Here's a simplified example of a rule-based decision engine:

from typing import List, Optional

class LLMDecisionEngine:
    def __init__(self, model_registry: ModelRegistry):
        self.model_registry = model_registry

    def select_model(self, context: RequestContext) -> Optional[LLMModel]:
        eligible_models = self.model_registry.get_all_healthy_models()

        # 1. Filter by hard requirements (context window, features)
        filtered_models = []
        for model in eligible_models:
            if model.context_window < context.input_token_count:
                continue # Too small context window
            if not all(feature in model.capabilities for feature in context.required_features):
                continue # Missing required features
            filtered_models.append(model)

        if not filtered_models:
            print("No models meet hard requirements.")
            return None

        # 2. Prioritize by task type and quality tier (soft requirements)
        # Define a scoring function or preference order
        def score_model(model: LLMModel) -> float:
            score = 0

            # Task type preference (example: code_generation prefers GPT-4o, Mixtral)
            if context.task_type == "code_generation":
                if model.id == "gpt-4o": score += 100
                elif model.id == "mixtral-8x7b": score += 80
            elif context.task_type == "creative_writing":
                if model.id == "claude-3-opus": score += 100
                elif model.id == "gpt-4o": score += 80
            elif context.task_type == "summarization":
                if model.id == "llama-3-8b-it": score += 70 # Prefer cheaper, faster for summarization

            # Quality tier preference (higher tier is better)
            tier_scores = {"premium": 30, "standard": 20, "economy": 10}
            score += tier_scores.get(model.quality_tier, 0)

            # Latency consideration (penalize higher latency if max_latency_ms is set)
            if context.max_latency_ms and model.avg_latency_ms > context.max_latency_ms:
                score -= 50 # Significant penalty for exceeding latency tolerance
            elif model.avg_latency_ms < 200: # Reward very fast models
                score += 15

            # Cost consideration (penalize higher cost if max_cost_per_request is set)
            # This is a simplification; real cost calculation needs output tokens too
            # For now, just a general preference for lower cost
            if model.input_cost_per_million_tokens < 1.00: score += 20

            # Priority tier of the request (higher priority requests can use more expensive models)
            if context.priority_tier == "premium" and model.quality_tier == "premium": score += 20

            return score

        # Sort models by score in descending order
        scored_models = sorted(filtered_models, key=score_model, reverse=True)

        if scored_models:
            print(f"Selected model: {scored_models[0].id} with score {score_model(scored_models[0])}")
            return scored_models[0]
        
        print("No optimal model found after scoring.")
        return None

# Example usage:
engine = LLMDecisionEngine(model_registry)

# Request for a quick, cheap summarization
summarization_request = RequestContext(
    task_type="summarization", priority_tier="low", input_token_count=100,
    max_cost_per_request=0.01
)
selected_model_summarization = engine.select_model(summarization_request)

# Request for high-quality code generation with function calling
code_gen_request = RequestContext(
    task_type="code_generation", priority_tier="premium", input_token_count=500,
    required_features=["function_calling"],
    max_latency_ms=1500
)
selected_model_code_gen = engine.select_model(code_gen_request)

# Simulate a provider outage (e.g., Groq goes down)
model_registry.update_model_status("llama-3-8b-it", "offline")

# Re-attempt summarization request - should now pick something else or fail
selected_model_summarization_after_outage = engine.select_model(summarization_request)

The score_model function is where the core business logic lives. This can be as sophisticated as needed, incorporating weighted averages, thresholds, and even dynamic adjustments based on system load or user feedback.

4. Adaptive Strategies and Failover

A static select_model function is good, but real-world systems need to be adaptive. This means building in resilience.

Tiered Fallback

Instead of just picking one best model, the decision engine can return a prioritized list. If the primary model fails, the system automatically tries the next one in the list. This is a common pattern I use in Dronzer.

class AdaptiveLLMRouter:
    def __init__(self, decision_engine: LLMDecisionEngine):
        self.decision_engine = decision_engine

    async def route_request(self, context: RequestContext, prompt: str) -> str:
        # In a real system, this would involve calling the actual LLM API
        # and handling exceptions.

        # Get a list of potential models, not just the single 'best'
        # Modify select_model to return a ranked list or iterate internally
        potential_models = self.decision_engine.get_ranked_models(context) # Hypothetical method

        for model in potential_models:
            try:
                print(f"Attempting request with model: {model.id}")
                # Simulate LLM API call
                if model.id == "some-unstable-model" and random.random() < 0.3:
                    raise Exception("Simulated API error")
                
                # Replace with actual LLM API call
                response = f"Response from {model.model_name} for task '{context.task_type}': {prompt[:20]}..."
                return response

            except Exception as e:
                print(f"Model {model.id} failed: {e}. Trying next option.")
                # Update model status in registry (e.g., to 'degraded')
                self.decision_engine.model_registry.update_model_status(model.id, "degraded")
                continue # Try the next model
        
        raise Exception("All LLM providers failed for this request.")

Circuit Breakers

Tiered fallback handles transient errors, but what if a provider is consistently failing? That's where circuit breakers come in. If a model (or provider) experiences a certain threshold of errors or timeouts within a timeframe, the circuit breaker "trips," marking that model as degraded or offline in the ModelRegistry. All subsequent requests will bypass it for a cool-down period. After the cool-down, a single "test" request is allowed through. If it succeeds, the circuit closes; if not, it stays open.

This prevents hammering a failing service and allows it time to recover, while your application continues to function using other models.

Load Balancing

If multiple models are equally suitable for a given request (e.g., two "economy" tier models from different providers), you can load balance between them. This distributes traffic, prevents any single provider from being overwhelmed, and provides a natural form of resilience.

A/B Testing and Shadowing

With a dynamic routing layer, you can easily introduce new models or configurations. You can route a small percentage of traffic (e.g., 5%) to a new model to observe its performance and cost characteristics without impacting the majority of users. "Shadowing" involves sending requests to a new model but only using the response from the current production model, allowing you to compare outputs offline.

Monitoring and Evaluation

An intelligent routing system is only as good as the data it's fed and the feedback loop it maintains. You need robust monitoring for:

• Latency: Per-model and per-request latency.
• Cost: Actual token usage and dollar cost per request/model.
• Error Rates: Per-model API errors, timeouts.
• Output Quality: This is harder to automate. Human evaluation, A/B testing user satisfaction, or programmatic checks (e.g., for structured output formats) are necessary.
• Model Status: Active health checks against all integrated LLM APIs.

This data feeds back into the ModelRegistry (updating avg_latency_ms, status) and helps refine the scoring logic in the DecisionEngine.

Beyond Simple Rules: The Future

While rule-based decision engines are powerful and practical, the next frontier involves more sophisticated approaches:

• Reinforcement Learning: An agent could learn to pick the optimal model by maximizing a reward function (e.g., minimizing cost while staying below a latency threshold and maintaining quality).
• Contextual Bandits: Similar to RL, but focused on exploring different model choices and exploiting the best ones, especially useful for A/B testing and adapting to changing model performance.
• Embedding-Based Routing: For highly nuanced tasks, embedding the user's prompt and comparing it to embeddings of model capabilities or historical best-performing models could offer more granular routing.

These are complex to implement and maintain, but they represent the pinnacle of adaptive LLM orchestration.

Lessons Learned from Dronzer

Building Dronzer AI Gateway has hammered home a few key points:

1. Observability is Non-Negotiable: Without detailed metrics on cost, latency, and errors for each model and each request, you're flying blind. You can't optimize what you can't measure.
2. The Registry is King: Keep your ModelRegistry up-to-date and easily accessible. It's the single source of truth for your LLM ecosystem. Automate its updates as much as possible.
3. Start Simple, Iterate: Don't try to build a full RL-powered router on day one. Start with clear, rule-based logic. Add complexity (like advanced failover or more nuanced scoring) as your needs grow and you gather more data.
4. Cost is a First-Class Citizen: LLM costs can escalate quickly. Integrating cost as a primary factor in your routing logic from the beginning saves significant money down the line.
5. Resilience is Paramount: APIs will fail. Models will degrade. Your system needs to gracefully handle these scenarios without falling over. Adaptive strategies like circuit breakers and tiered fallback are essential.

Architecting for dynamic model selection and adaptive routing is a complex but incredibly rewarding engineering challenge. It transforms your LLM integrations from brittle, static configurations into resilient, cost-effective, and high-performing systems. It's how you truly unlock the power of a multi-LLM strategy in production.