Prompt Diff Tool

Prompt Engineering Is Iteration; Iteration Requires Diff

LLM applications live or die by their prompts. The difference between a flaky AI feature and a reliable one is usually 5-20 iterations of prompt refinement, each guided by observing the output, hypothesizing a change, applying it, and measuring the difference. This iterative loop has the same problem any iterative engineering loop has: as you make changes, it's easy to lose track of WHAT you changed and WHY. The Prompt Diff Tool gives you the same workflow for prompts that git diff gives you for code.

Prompts Are Code

Treat prompts as code. They have:

• Logic, conditional instructions, control flow ("if the input contains X, do Y").
• Structure, sections, hierarchy, ordering.
• Inputs and outputs, variable interpolation, expected response format.
• Bugs, edge cases that fail, surprising failure modes, subtle behaviors.
• Performance, token cost, latency, quality.

Every code-engineering best practice translates: version control, code review, testing, documentation, modularity (composable prompt fragments). Most teams under-invest in this and pay later when "the prompt we shipped 3 months ago is broken, and we don't know what was in it then."

Why Diff Matters Specifically

Prompts have a property normal code doesn't: non-local effects. Editing a single word can change the model's entire output distribution. There's no compiler to catch the breakage; you find out only by running the prompt and observing.

This makes the relationship between edit and outcome opaque. You can't reason from "this prompt looks correct" to "this prompt will produce the right output." You have to empirically test. And to test scientifically, you need to know what changed.

A diff tool answers: "Between Version A (worked well) and Version B (started failing), what exactly differs?" Without the diff, you'll guess wrong and waste hours.

The Iteration Workflow

Concrete loop for prompt engineering:

1. Set up evaluation. Have a set of 10-50 test inputs and the expected behavior for each (a "golden set"). Some inputs are easy, some are edge cases.
2. Run baseline. Execute the current prompt against all test inputs. Score outputs (manually or with an LLM judge).
3. Hypothesize. Pick one specific behavior to improve. "On test #7, the output is too verbose. I want it shorter."
4. Change. Edit ONE thing in the prompt. Maybe add "Be concise" to the system message. Maybe shorten the examples. Pick one lever and pull it.
5. Re-run. Execute the new prompt against all test inputs.
6. Compare outputs. What got better? What got worse? Did test #7 improve? Did anything regress?
7. Diff the prompts. Confirm the change you made was the change you intended, sometimes copy-paste accidents introduce extra changes.
8. Decide. Keep, revert, or refine. If keep, save as a new version; if revert, return to previous.
9. Repeat.

This is the same TDD loop as software engineering, just slower because LLM outputs require evaluation.

What to Look for in a Diff

When you diff prompts, look for:

• Wording changes in instructions: did you weaken or strengthen a constraint?
• Order changes: moving an instruction up or down changes its attention weight in the model.
• Example changes: few-shot examples are heavy signal. Adding/removing/editing examples is high-impact.
• Format changes: any change to the expected output structure (JSON schema, markdown, XML tags) is high-impact.
• Persona changes: the system prompt's role definition affects every response.
• Constraint additions: "Never X" changes are particularly impactful because the model now must avoid X across all responses.
• Whitespace changes: surprisingly, formatting consistency (always two-space indents vs mixed) can affect output. Tabs/spaces matter to models.

Common Pitfalls

Multiple changes at once. You change three things, output changes, you don't know which change caused the change. Make one change at a time during iteration.

Subjective evaluation. "It's better now" without measurable criteria. Define what "better" means with concrete examples, "test #7 went from 200 words to 80 words."

Overfitting to test cases. Tuning the prompt to pass your test set without checking it still works on production inputs. Keep a holdout test set you don't tune against.

Forgetting versions. "What did we change in v17?" You don't know. Version control is the answer.

Reactive vs proactive. Only iterating when production fails. Proactively running test sets after every change catches regressions earlier.

Prompt Management at Scale

For teams running LLM features in production:

• Source control. Prompts in .md/.txt files in your repo. Diff via git.
• Templating. Use libraries (LangChain PromptTemplate, OpenAI Function calling templates, Anthropic XML tag conventions) to keep prompts modular.
• Variable interpolation. Separate prompt structure from runtime data. {{user_input}} placeholders are filled at request time.
• Versioned tracking. Tag prompt versions; correlate with output quality metrics. Tools: LangSmith, PromptLayer, Humanloop.
• A/B testing. Run two prompts against the same input distribution; compare aggregate quality.
• Regression testing. Run a golden set through every deploy; fail the deploy if quality drops.

Models Drift, Too

A subtle issue: the same prompt sent to the same model on the same day can produce different outputs (temperature > 0). Across model updates (OpenAI's GPT-4 became GPT-4-turbo became GPT-4o), the same prompt can behave significantly differently.

This means prompt versioning isn't enough, you also need to track WHICH model version the prompt was tested against. Otherwise "this prompt worked last month" loses its meaning.

When you find behavior changing without a prompt change, suspect the model update.

Diff vs Eval: Different Tools

Diff shows what CHANGED between prompts. Eval shows whether the change IMPROVED things.

You need both. Diff alone tells you what you did; eval alone tells you whether it worked. Pair them: diff to attribute outcomes, eval to measure outcomes.

This tool does the diff part. The eval part is your set of test inputs run through your LLM API, that's a separate workflow specific to your use case.

Privacy

Diff runs as pure text comparison in your browser using a JavaScript diff library (similar to the algorithm behind git diff). Prompts stay in JavaScript memory. Open DevTools Network during use: zero outbound requests. Critical because prompts often encode proprietary business rules, system-level instructions that constitute IP, and competitive insights, they should never leak to third parties.