Skip to content

feat: add examples/model_drift — answer-drift gate for model migrations - #58

Merged
Therealdk8890 merged 1 commit into
mainfrom
DK/model-drift-example
Jul 21, 2026
Merged

feat: add examples/model_drift — answer-drift gate for model migrations#58
Therealdk8890 merged 1 commit into
mainfrom
DK/model-drift-example

Conversation

@Therealdk8890

@Therealdk8890 Therealdk8890 commented Jul 21, 2026

Copy link
Copy Markdown
Owner

What

A reusable examples/model_drift/ recipe: check whether a replacement model drifts from a soon-to-be-discontinued one on a fixed prompt set.

It records the old model's answers as a golden run and the new model's as a candidate (one CRITICAL generation event per prompt carrying {prompt_id, model, response}), then gates the candidate with the kit's RegressionGate + a pluggable answer-equivalence evaluator. A per-prompt answer below your threshold surfaces as a critical regression (drift); the CLI exits 1 on drift, 0 on parity — drops straight into CI.

Why

Model deprecations are a canonical migration risk with no first-class recipe in the repo. dpk compare / gate already do golden-vs-candidate gating; this fills the gap of driving two models and judging answer equivalence.

Design & principles

  • Core stays stdlib-only. Vendor SDKs live only in this example and are lazily imported; adds an opt-in openai extra to pyproject.toml. No change to dprovenancekit/.
  • Pluggable DriftEvaluator — stdlib LexicalSimilarityEvaluator (default, difflib) + LLMJudgeEvaluator. The judge fails closed: any reply that isn't a clean [0, 1] score (an error string, an out-of-range number like "Error 503" or "9 out of 10") reads as drift, so a flaky judge can never silently pass a regression.
  • Capture-now, gate-laterrecord_baseline(...) / --record-golden + --golden-db pin the old model's golden while it's still callable, then gate against it after retirement (the old model is never re-called on the reuse path).
  • Runs in CI with zero setup — deterministic offline FakeModelClient (parity/drift modes); no network, no key.

Tests / verification

  • 21 deterministic offline tests (tests/test_model_drift.py): parity passes, drift fails and names the drifted prompts, CLI exit codes, imports + runs with openai absent, judge fails-closed on unparseable/out-of-range replies, saved-golden reuse never calls the old model.
  • ruff clean; mypy clean on the example and on dprovenancekit/.

Usage

# offline smoke test (no setup)
python -m examples.model_drift --old old --new new --fake drift   # exits 1

# live migration check
pip install "dprovenancekit[openai]"; export OPENAI_API_KEY=...
python -m examples.model_drift --old <old-model> --new <new-model> \
    --prompts my_prompts.jsonl --live --judge llm --threshold 0.85

Check whether a replacement model drifts from a soon-to-be-discontinued one on a fixed prompt set. Records the old model's answers as a golden run and the new model's as a candidate, then gates with RegressionGate + a pluggable answer-equivalence evaluator (stdlib lexical or LLM-as-judge). The LLM judge fails closed: any reply that isn't a clean [0,1] score reads as drift, so a flaky judge can't silently pass a regression.

record_baseline(...) / --record-golden + --golden-db let you pin the old model's golden while it's still callable and gate against it after retirement (the old model is never re-called on the reuse path). Ships a deterministic offline FakeModelClient so the flow runs in CI with no key; the OpenAI client and its opt-in 'openai' extra are lazily imported, keeping dprovenancekit/ stdlib-only and unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Therealdk8890
Therealdk8890 merged commit cb9c999 into main Jul 21, 2026
9 checks passed
@Therealdk8890
Therealdk8890 deleted the DK/model-drift-example branch July 21, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant