Treating Prompts as Code: Managing LLM Instructions in Production

When I first started playing with LLMs, it felt like magic. Throw a few words at it, and boom, something useful comes out. But moving from a local playground to a production system is where the real work starts. The 'magic' quickly turns into a question of consistency, reliability, and maintainability.

One of the biggest lessons I've learned is that prompt engineering isn't just about crafting a good initial prompt. It's about treating those instructions as a first-class citizen in your codebase. Just like any other piece of logic, prompts need to be versioned, tested, and dynamically managed.

Take something like Snapfix, the VS Code extension I built. Its core job is to analyze a Python traceback in real-time and provide a sensible, actionable fix. This isn't a one-off query. It's happening constantly, with wildly varying inputs. If the prompt isn't robust, consistent, and able to adapt, the suggestions become useless, or worse, dangerous.

The Problem with Static Prompts

At the beginning, it's easy to just hardcode a prompt string directly into your llm_call() function. Maybe you use an f-string to inject a variable or two. That works for a prototype.

But as soon as you want to:

• Iterate and improve: You tweak a word, then another. How do you track what worked? How do you revert if a change makes things worse?
• Test different variations: You want to A/B test a 'concise' prompt against a 'verbose' one. Copy-pasting functions isn't scalable.
• Maintain consistency: Different parts of your application might need similar prompt structures. Copy-pasting leads to divergence.
• Manage complexity: Real-world prompts often need a lot of context. Injecting a traceback, relevant code snippets, user preferences, even previous conversation history – that quickly turns a simple f-string into a tangled mess.

Eventually, you realize you're not just writing a string; you're designing an interface for an AI. And interfaces need proper engineering.

My Approach: Prompt Engineering as a First-Class Citizen

What I mean by 'first-class citizen' is simple: prompts should be treated with the same rigor as your Python classes, database schemas, or API endpoints. This means:

1. Version Control: Prompts live in Git. Every change is tracked.
2. Templating: Use a system to inject dynamic context cleanly.
3. Separation of Concerns: Keep the prompt text separate from the application logic that uses it.
4. Testability: You should be able to run tests against your prompts, just like any other code.

For Snapfix, this was critical. We need to tell the LLM, "Here's a traceback, here's some code, tell me how to fix it." But the exact phrasing, the emphasis, the constraints on the output (e.g., "only provide code in a markdown block"), all need to be carefully controlled and easily modifiable.

Implementing Prompt Management in Practice

Let's look at how this can be structured, using Python and LiteLLM as an example.

1. Prompt Templating

Instead of inline strings, I use separate files for prompt templates. These can be simple text files with placeholders, or leverage templating engines like Jinja2 for more complex logic. For many cases, simple .format() placeholders are enough.

Consider a prompts directory in your project:

# prompts/fix_suggestion.txt
You are an expert Python developer. Your goal is to provide a concise, actionable fix for a Python traceback.
Analyze the following traceback and the accompanying code context.
Provide only the code changes needed to fix the issue, enclosed in a markdown code block.
Explain your reasoning briefly, in one sentence.

Traceback:
{traceback_content}

File content (excerpt around the error, if available):
{file_content_excerpt}

Consider the following:
- What is the most likely root cause?
- How can this be fixed with minimal changes?
- Is the fix safe and idiomatic Python?

This template clearly defines the LLM's role, the expected input, and the desired output format. The placeholders {traceback_content} and {file_content_excerpt} are where the dynamic data comes in.

2. Loading and Dynamic Context Injection

Your application code then loads these templates and injects the actual data at runtime. This keeps your application logic clean and focused on data retrieval, not prompt wording.

import litellm
import os
from typing import Dict, Any

def load_prompt_template(template_name: str) -> str:
    """Loads a prompt template from the 'prompts' directory."""
    # Adjust this path based on your project structure
    current_dir = os.path.dirname(os.path.abspath(__file__))
    template_path = os.path.join(current_dir, "prompts", f"{template_name}.txt")
    
    try:
        with open(template_path, "r", encoding="utf-8") as f:
            return f.read()
    except FileNotFoundError:
        raise ValueError(f"Prompt template '{template_name}.txt' not found at {template_path}")

async def get_fix_suggestion(traceback: str, file_excerpt: str, model: str = "gpt-4-turbo") -> str:
    """
    Generates a fix suggestion for a Python traceback using a dynamically loaded prompt.
    """
    try:
        template = load_prompt_template("fix_suggestion")
    except ValueError as e:
        print(f"Error loading prompt: {e}")
        return "Error: Prompt template missing."
    
    # Dynamic context injection using format()
    formatted_prompt = template.format(
        traceback_content=traceback,
        file_content_excerpt=file_excerpt
    )

    messages = [
        {"role": "user", "content": formatted_prompt}
    ]

    try:
        response = await litellm.acompletion(
            model=model, # e.g., "gpt-4-turbo", "claude-3-opus", "ollama/llama2"
            messages=messages,
            temperature=0.7,
            max_tokens=500
        )
        return response.choices[0].message.content
    except Exception as e:
        print(f"Error calling LLM with model {model}: {e}")
        return "Could not generate a fix suggestion at this time due to an API error."

# Example usage (for demonstration purposes)
async def main():
    sample_traceback = """
Traceback (most recent call last):
  File "/app/main.py", line 7, in <module>
    result = divide(10, 0)
  File "/app/main.py", line 4, in divide
    return a / b
ZeroDivisionError: division by zero
    """
    sample_file_excerpt = """
1 def divide(a, b):
2     if b == 0:
3         raise ValueError("Cannot divide by zero!")
4     return a / b
5
6 if __name__ == "__main__":
7     result = divide(10, 0)
    """
    
    print("\n--- GPT-4 Turbo Suggestion ---")
    fix_gpt = await get_fix_suggestion(sample_traceback, sample_file_excerpt, model="gpt-4-turbo")
    print(fix_gpt)

    # You could easily swap models here if configured with LiteLLM
    # print("\n--- Claude Suggestion ---")
    # fix_claude = await get_fix_suggestion(sample_traceback, sample_file_excerpt, model="claude-3-opus-20240229")
    # print(fix_claude)

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

This setup allows:

• Easy updates: To change the prompt, you edit fix_suggestion.txt. No code deployment needed (if you load from a config service, even better).
• Readability: The prompt's intent is clear.
• Contextual relevance: The LLM gets exactly the information it needs for the task.

3. Versioning Prompts with Git

Storing your prompts as files in your repository means they automatically benefit from Git's version control. You get a full history of changes, who made them, and when. This is invaluable when you deploy a new prompt version and suddenly see a drop in output quality – you can pinpoint the change and roll it back.

For more advanced scenarios, especially in larger teams or applications needing rapid iteration without code deploys, you might consider storing prompts in a dedicated database or a remote configuration service. This allows for A/B testing different prompt versions in real-time and rolling out changes instantly. But for many projects, especially starting out, plain old Git is fantastic.

4. Model Agnostic Prompts (with LiteLLM)

One of the benefits of abstracting your LLM calls with a library like LiteLLM (which I use in Snapfix) is that your prompt logic remains largely independent of the specific model or provider you're using. The core prompt structure for getting a fix suggestion is the same, whether it's GPT-4, Claude, or a local Ollama model.

This separation means you can focus on refining the prompt's effectiveness without worrying about API differences. LiteLLM handles the routing and translation, letting you concentrate on the instruction itself.

Lessons Learned and Future Thoughts

• Prompt engineering is an ongoing process. It's not a 'set it and forget it' task. You'll constantly be tweaking, testing, and refining your prompts as models evolve and your understanding of user needs deepens.
• Observability is key. Log your prompts and the corresponding LLM responses. This data is gold for debugging, identifying prompt weaknesses, and understanding how the model interprets your instructions.
• Guardrails are necessary. Especially for an application like Snapfix, you need to think about what happens if the LLM generates an unsafe fix, or hallucinates. Your prompts should guide the model, but your application logic needs to validate its output. For example, in Snapfix, I'd have additional checks on the suggested code before presenting it.
• Consider prompt chaining. For complex tasks, breaking down a problem into smaller, sequential prompts can often yield better results than one monolithic prompt. Each step can have its own versioned template.

Treating prompts as code might seem like overkill at first. But once you move beyond trivial examples and into production applications where reliability and consistency matter, it quickly becomes indispensable. It's about bringing engineering discipline to the art of communicating with AI, making your LLM integrations robust, scalable, and maintainable for the long haul.