Evaluating LLM Applications: Moving Beyond Vibe Checks

The Vibe Check Problem

"Looks good to me" is not an evaluation strategy. But it's what most teams do: manually inspect 10 outputs, ship if they feel right. This doesn't scale and it doesn't catch regressions.

Level 1: Unit Tests for Prompts

Deterministic assertions: JSON schema validation, required field presence, forbidden phrases, length constraints. Fast, cheap, run on every commit.

Level 2: Golden Dataset + LLM-as-Judge

Curate 100-500 representative inputs with expected outputs (or quality criteria). Use a strong LLM (GPT-4, Claude) as judge with a structured rubric. Calibrate the judge against human labels first — aim for >0.85 Cohen's kappa.

Level 3: Adversarial & Edge Cases

Automatically generate: prompt injections, out-of-distribution inputs, multi-turn context confusion, hallucination triggers. Run nightly.

Level 4: Production Monitoring

Log every (input, output, latency, tokens). User feedback (thumbs up/down) as implicit labels. Drift detection on embedding distributions. Alert on quality regression >5%.

CI Integration

# .github/workflows/llm-eval.yml
name: LLM Evaluation
on: [push, pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run eval suite
        run: python -m pytest eval/ --threshold=0.85
      - name: Comment PR with results
        uses: actions/github-script@v7
        with:
          script: |
            // post eval summary to PR

The Payoff

We caught a 12% quality drop from a prompt "improvement" before it hit production. The eval suite runs in 3 minutes. It's the highest-ROI test suite we own.