Guides · ← All guides
Model Behaviour Changes: Silent Updates and How to Catch Them
Published: November 7, 2026 · Reviewed November 2026 · 8 min read
By the Ingenious AI team
The short answer
A model can change without changing its name — an alias is repointed, or the model behind a fixed ID is swapped, and your tuned prompts quietly degrade while every health check stays green. You catch it with a canary: a fixed prompt set with reference outputs recorded from the live API, re-run and compared on four deterministic metrics. ModelMinder's alarm needs 3+ probe runs before calling drift (composite ≥0.25). Method: ModelMinder's downgrade alarm, as implemented; reviewed November 2026.
What are the ways a model changes silently?
Alias repointing. A bare alias — gpt-4, claude-sonnet-4 — points wherever the provider aims it; the two failure modes are laid out in pinned vs alias. The silent swap. The vendor replaces the model behind a fixed ID or endpoint: the new model answers, passes health checks, and charges the same, and no deprecation calendar catches it — there is no date to put on a calendar. Schedule drift. The model stays, but its shutdown date moves — sometimes earlier than the notice you planned around. Tokenizer and artifact drift. The machinery around the model changes, shifting how your text is counted and priced. Each shape needs a different sensor, which is the point of the next section.
How does the vital-sign pulse work?
ModelMinder's watch is a set of vital signs, each read the same way every time — the pulse is the rhythm, not a metaphor. The output pulse is the Silent Downgrade Alarm's canary method:
| Vital sign (metric) | What it measures | Weight in the composite |
|---|---|---|
| Token-length ratio drift | Word-count ratio between the new output and the reference — answers getting shorter or longer | 0.30 |
| N-gram overlap drift | Jaccard distance on word 3-grams — the wording itself changing | 0.40 |
| Refusal-pattern change | A refusal marker present in exactly one of the two outputs — a new "I can't" where none was, or one gone | 0.15 |
| Formatting-marker drift | Markdown structure per output — headers, bold, lists, code blocks, tables, links | 0.15 |
The reference outputs are captured at registration, from the live API, never synthesized. The verdicts are deliberately conservative: STABLE below a composite of 0.25, DRIFT-SUSPECTED at 0.25 (strong at 0.45), and INSUFFICIENT-DATA until at least 3 probe runs exist — a single noisy run never triggers a verdict. Around the output pulse sit the other signs: the Schedule-Drift Sentinel diffing each vendor's published retirement schedule against its previous snapshot (a date moved 14+ days earlier is HIGH; moved later is recorded, never alerted), and tokenizer/artifact hash recording, where a changed hash on a tracked model raises a drift alert. When it is time to move models entirely, the same determinism runs the other way: a migration eval diffs the replacement's outputs on your real prompts — unified diff, similarity, keyword retention — with pass / attention / fail verdicts, per the migration playbook.
What does a drift verdict not prove?
That the vendor swapped the model — and ModelMinder prints that caveat on every verdict rather than in a footnote: prompting changes, temperature jitter, retrieval-backend drift, and content filters all move these metrics too. DRIFT-SUSPECTED is an instruction to investigate: check the vendor's changelog, re-run the canary suite twice, and only then conclude. The honest framing is what makes the alarm usable — a detector that cried "swap" at every wobble would be muted within a month, like the teams in how teams get burned who learned to ignore the notices that mattered. More from the fleet: Ingenious AI.
Frequently asked questions
Can an AI model change without its name changing?
Yes, two ways. An alias (gpt-4, claude-sonnet-4) is repointed by the provider whenever they choose, and a vendor can swap the model behind a fixed ID or endpoint. The new model answers, passes health checks, and charges the same — but tuned prompts quietly degrade. No deprecation calendar catches that, which is why drift detection exists.
How does the canary drift method work?
At registration you record a canary suite: a fixed set of prompts (up to 12) plus the reference outputs the model produced that day, captured from the live API. Each later probe re-sends the same prompts and compares the new outputs to the references on four deterministic metrics — token-length ratio, word 3-gram overlap, refusal-pattern change, and formatting-marker drift.
When does ModelMinder call it drift?
The four metrics combine into a composite score with published weights (length 0.30, n-gram distance 0.40, refusal change 0.15, format drift 0.15). At 0.25 or above the verdict is DRIFT-SUSPECTED — strong at 0.45 — below it, STABLE. No verdict fires before 3 probe runs; fewer is INSUFFICIENT-DATA, so one noisy run never triggers an alarm.
Does DRIFT-SUSPECTED prove the vendor swapped the model?
No, and every verdict says so. DRIFT-SUSPECTED means outputs differ enough that an investigation is warranted. Temperature jitter, retrieval-backend drift, content-filter updates, and your own prompt edits move these metrics too. Before concluding anything, check the vendor's changelog and re-run the canary suite twice.
What else changes silently besides outputs?
Schedules and tokenizers. Vendors move shutdown dates — ModelMinder's Schedule-Drift Sentinel flags a date pulled 14+ days earlier as HIGH severity — and tokenizer or artifact changes are caught by recording their hashes and alerting when a hash changes on a model you track.