The prompt that broke our production pipeline was not wrong. It was better. It produced more detailed code summaries with richer context. The developer who wrote it tested it manually against five representative repositories and it worked correctly every time.
What they did not test was the output format. The original prompt produced responses that started with `{`. The new prompt, requesting 'additional context,' caused Gemini to prepend a one-sentence prose introduction before the JSON block. Every `JSON.parse()` call in three downstream analytics services threw a `SyntaxError: Unexpected token` exception. The services continued running — they just silently swallowed the error and wrote `null` into the analytics database instead of the parsed summary.
We discovered the breakage 6 hours later when a customer emailed to report that their repository dashboard was showing blank summaries. The fix took 4 minutes. The detection took 6 hours.
The migration that followed took two weeks. Every prompt in Gitloom was moved to a versioned registry file — a TypeScript module that exported prompt templates as typed functions with explicit input parameter schemas and Zod-validated output schemas:
```typescript // prompts/code-summary.ts export const CodeSummaryPrompt = { version: '2.1.0', template: (context: CodeContext): string => ` Analyze the following code and return ONLY valid JSON matching the schema. Do not include any prose before or after the JSON object. Schema: ${JSON.stringify(CodeSummarySchema.shape)} Code: ${context.snippet} `, outputSchema: CodeSummarySchema, // Zod schema, validated at inference boundary }; ```
The output schema validation runs at the inference call site, not in the consuming service. If the model returns a response that does not match the schema, the error is thrown at the source with the full raw output attached — not silently swallowed 3 service hops later.
Every prompt change now requires updating the version string. The CI pipeline runs a 40-query eval suite against each prompt version before merging — comparing accuracy scores, schema compliance rates, and token consumption against the previous version's baseline. If any metric regresses by more than 5%, the merge is blocked.
After the migration: zero downstream parsing exceptions across 14 subsequent prompt and model updates. Three silent accuracy regressions caught by the eval suite that would have reached production undetected. The developer who made the original breaking change said afterward: 'I didn't know I needed a test for that.' Now the CI tells you before the customer does.