Zero-Downtime LLM Migrations: Failure Modes, Latency Budgets, and CI/CD Evals (Part 1)

You can't escape from model deprecation! But you can survive it. A provider announces that a production model Gemini 3.5 Flash will reach End-of-Life in 90 days. In other instances, there is no advance notice. Overnight, the hosted model endpoint is updated with a new backend quantization pass, an "improved" safety alignment, and a slightly different inference server configuration. The model ID stays the same, but the answers change.

In asynchronous batch applications or standard web interfaces, subtle behavioral shifts usually cause minor friction. In a real-time conversational voice bot running over telephony or WebRTC, model drift breaks the entire pipeline. A 200ms latency degradation breaks turn-taking cadence. A change from digits ("452") to spelled-out words ("four fifty-two") breaks downstream speech synthesizers. A slight increase in verbosity causes callers to talk over the bot.

This two-part guide provides the engineering blueprint for zero-downtime model migrations in mission-critical voice systems.

[!NOTE] This is Part 1 of a two-part series. Part 1 covers model drift failure modes, voice latency budgets, phonetic regressions, and Tier 1 declarative CI/CD test suites.

Continue to Part 2: User Simulation, Shadow Traffic, and Canary Rollouts for multi-turn simulation, live dark traffic routing, and automated fallback gateways.

The Anatomy of Model Drift and Deprecation

When an engineering team swaps an incumbent model for a candidate model (or when an upstream provider updates a snapshot), regressions emerge across four distinct axes.

1. Prompt Sensitivity and Instruction Alignment

Prompts calibrated using standard prompt engineering patterns for an older model architecture frequently degrade on newer architectures. Delimiters, few-shot examples, and markdown headers tuned for one tokenizer can dilute attention in another.

Tokenizer vocabularies differ significantly across model families. For instance, a prompt template optimized with XML tags (<context>, <instructions>) for Claude or Gemini may be parsed with different subword boundaries in a newer model version. This alters attention weights over critical system constraints.

Furthermore, newer foundation models often feature stronger Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) safety tuning. This alignment frequently causes over-refusals on benign user utterances or inverts negative constraints. A system instruction explicitly commanding the model to "never output conversational filler" may be ignored by a model trained to prioritize polite conversational preambles.

2. Verbosity Expansion and Preamble Bloat

Newer foundation models frequently default to longer, more elaborate responses. While higher token counts improve benchmark scores on reasoning evaluations like MMLU or MATH, they directly degrade voice systems.

In voice bots, output length directly dictates synthesis duration. A jump from 25 tokens to 65 tokens per dialogue turn doubles audio playback time from 2.5 seconds to over 6 seconds. Human callers do not tolerate monologues on the phone. Long bot turns prevent the user from clarifying their intent and spike call abandonment rates.

3. Structured Output and Tool Signature Drift

Telephony agents rely on function calling to query backends, transfer calls, execute CRM updates, and trigger DTMF tones. Newer models can alter parameter types, omit optional schema properties, or hallucinate parameters that violate strict Pydantic schemas.

In the regressed output above, flight_number was coerced from a string to an integer, seat_count was renamed to seats, and boolean confirm became a string "yes". In a standard API, this triggers a schema validation error. In a live telephone call, this halts execution, resulting in dead air and an abrupt call drop.

4. Latency Distribution Variance

Average latency is a misleading metric for conversational systems. An upgraded model may report a lower mean inference time on static benchmark datasets, but its P95 and P99 Time-to-First-Token (TTFT) under concurrent load can exhibit severe jitter. Voice pipelines require strict deterministic latency ceilings rather than favorable average throughput.

The Voice Bot Latency Budget

A real-time voice bot operates as an asynchronous streaming pipeline connected over WebSockets or WebRTC. Total turnaround latency (the pause between the caller finishing their sentence and hearing the bot's initial audio) must remain strictly under 1,000ms. Latencies above 1,200ms feel unnatural, while delays beyond 1,800ms trigger conversational collisions.

TTFT vs TTFS: The Sentence Chunking Bottleneck

In text chatbots, Time-to-First-Token (TTFT) governs perceived performance. In voice bots, Time-to-First-Sentence (TTFS) is the governing metric.

Modern speech synthesizers (such as Cartesia Sonic, ElevenLabs Flash, or Deepgram Aura) require a complete phrase or clause before synthesizing audio with natural prosody. If an LLM streams 15 tokens with a 120ms TTFT but outputs a 30-word run-on sentence without punctuation, the sentence chunker cannot send tokens to the TTS engine. Audio generation remains blocked until a period, comma, question mark, or semicolon is produced.

1Token Stream: "Certainly [20ms] I [20ms] can [20ms] help [20ms] you [20ms] check [20ms] your [20ms] flight [20ms] status [20ms] for [20ms] tomorrow [20ms] morning [20ms] if [20ms] you [20ms] give [20ms] me [20ms] your [20ms] confirmation [20ms] code."

In this scenario:

• TTFT: 180ms
• TTFS (Chunk Delivered to TTS): 180ms + (19 tokens * 20ms) = 560ms
• Total Audio Latency: 560ms + 120ms (TTS generation) + 150ms (STT) + 120ms (VAD) = 950ms

If a candidate model adds conversational filler ("Certainly, I would be delighted to assist you with checking your flight details today..."), TTFS increases by 400ms, pushing turnaround time over the 1,200ms threshold.

Phonetic and Audio Drift: The Hidden Regressions

When migrating models in pure text environments, formatting variations are cosmetic. In voice systems, text formatting changes completely alter how the TTS engine pronounces words over telephony channels.

1. Numeric and Temporal Formats

• Currency: An incumbent model outputs "one hundred and twenty dollars". A candidate model outputs "$120.00". A naive TTS engine synthesizes the latter as "dollar sign one hundred twenty point zero zero".
• Phone Numbers & Account IDs: If the model outputs "8005550199", the TTS engine may read it as "eight billion five million five hundred fifty thousand one hundred ninety-nine" instead of individual digits.
• Dates & Times: "07/12/2026" may be read as "zero seven slash twelve slash twenty twenty-six" rather than "July twelfth, twenty twenty-six".

2. Acronyms and Technical Initialisms

• St.: Could mean "Saint" or "Street" depending on sentence context ("320 Main St." vs "St. Jude Hospital").
• LLM, SQL, NASA, API: Some acronyms should be spelled out character-by-character ("S-Q-L" or "A-P-I"), while others should be pronounced as whole words ("NASA"). Upgraded models that insert hyphens or lowercase characters ("sql", "L.L.M.") alter phoneme mappings in speech engines.

3. Special Character and Markdown Leaks

Standard LLMs frequently inject markdown formatting into text streams:

• Asterisks (**bold** or *italic*)
• Markdown links ([account dashboard](https://...))
• Bullet characters (-, •, 1.)
• Code block delimiters (````json)
• Emoji characters (😊, ✈️, 📞)

When sent over WebRTC to a speech synthesizer, these characters are either read aloud verbatim ("asterisk asterisk urgent asterisk asterisk") or cause audio engine buffer crashes. A regression test suite must enforce zero tolerance for markdown tokens in voice responses.

4. Sentence Boundary Regex and Punctuation Edge Cases

Sentence chunking engines slice streaming token streams into discrete synthesis packets. Simple splitters using single periods fail on standard conversational edge cases:

• Decimal Numbers: An utterance like "The temperature is 98.6 degrees" must not split at 98. as a sentence boundary.
• Honorifics and Initials: Titles like "Dr. Smith" or "e.g." must be processed as single tokens rather than terminating a clause.
• Ellipses and Trailing Hesitations: Natural human dialogue often contains trailing pauses ("Let me see..."). The sentence chunker must buffer trailing dots until subsequent words clarify whether the turn is complete.

Streaming voice gateways should employ lookahead regex chunkers or fast token-level sentence boundary classifiers (e.g. PySBD or lightweight Rust token scanners) to prevent premature audio dispatch.

Interruption Handling and Context Truncation

A fundamental challenge in real-time voice bots is handling caller interruptions (barge-in). When a user speaks while the bot is outputting audio, the pipeline must execute three coordinated actions within milliseconds:

1. In-Flight LLM Stream Abort

When the voice activity detector triggers mid-turn, the orchestration server must send an immediate cancellation signal to the candidate LLM client. If using Python's asyncio, the task running the streaming generator must be cancelled immediately. If generation continues in the background, it consumes expensive inference tokens and introduces race conditions in session state.

2. Audio Buffer Flushing

The WebRTC media server must drop queued audio packets within 60ms. If the buffer is not flushed cleanly, the caller hears residual audio for several hundred milliseconds after they start speaking, creating the impression that the bot is ignoring them.

3. Context History Synchronization

The most common state-corruption bug in voice bot migrations is recording unuttered tokens in conversation history.

1Full LLM Generation: "Your appointment is confirmed for Friday at three PM, and I have emailed the confirmation receipt to your address on file."
2Spoken Portion Before Interruption: "Your appointment is confirmed for Friday at three PM..."
3Caller Interruption: "Wait, can we make it four PM instead?"

If the entire unuttered response is stored in the conversation history, the candidate model in subsequent turns will assume the email receipt was already discussed, leading to hallucinations and confusing dialogue.

The Four-Tier Testing Harness

To migrate models safely without risking live telephony traffic, we implement a four-tier testing strategy.

Tier 1: Declarative and Unit-Level Regression Testing

Tier 1 executes inside the CI/CD pipeline on every commit, prompt edit, or upstream model version bump. It consists of deterministic unit tests, JSON schema validation, latency thresholds, and LLM-as-a-judge rubrics.

1. Declarative Testing with Promptfoo

Promptfoo is an open-source CLI and evaluation engine that allows teams to declare model configurations, test cases, and assertion matrices in YAML.

1# promptfooconfig.yaml
2description: "Voice Bot Migration: Gemini 2.5 Flash -> Gemini 3.5 Flash"
3
4prompts:
5  - file://prompts/voice_system_prompt.json
6
7providers:
8  - id: "google:gemini-2.5-flash"
9    label: "Incumbent Baseline"
10  - id: "google:gemini-3.5-flash"
11    label: "Candidate Release"
12
13defaultTest:
14  options:
15    transform: "output.trim()"
16
17tests:
18  - description: "Flight cancellation tool call and argument schema"
19    vars:
20      user_utterance: "I need to cancel my flight UA 782 tomorrow afternoon."
21    assert:
22      - type: is-json
23        value: true
24      - type: javascript
25        value: "output.tool_call === 'cancel_flight' && output.parameters.flight_id === 'UA782'"
26      - type: latency
27        threshold: 420 # Hard ceiling of 420ms TTFT
28
29  - description: "Phonetic number formatting (no raw numerals or dollar signs)"
30    vars:
31      user_utterance: "How much do I owe on my current billing cycle?"
32    assert:
33      - type: not-regex
34        value: "\\$[0-9]+(\\.[0-9]{2})?"
35      - type: not-regex
36        value: "[0-9]{3,}"
37      - type: llm-rubric
38        value: "Verify that all currency and quantities are spelled out as spoken words suitable for text-to-speech synthesis (e.g. 'one hundred and forty dollars')."
39
40  - description: "Voice conciseness constraint (<35 words)"
41    vars:
42      user_utterance: "What documents do I need to bring to my passport appointment?"
43    assert:
44      - type: javascript
45        value: "output.text.split(/\\s+/).length <= 35"
46      - type: not-contains
47        value: "Here are the documents you will need:"

Running this test suite outputs a side-by-side diff across both models:

1npx promptfoo@latest eval
2npx promptfoo@latest view

2. Pytest-Native Testing with DeepEval

For teams with existing Python test automation, DeepEval provides unit test fixtures with built-in LLM metrics.

1# test_voice_model_regression.py
2import pytest
3from deepeval import assert_test
4from deepeval.test_case import LLMTestCase, LLMTestCaseParams
5from deepeval.metrics import GEval, LatencyMetric
6from pipeline.voice_orchestrator import run_turn
7
8# Metric 1: Spoken Conversational Tone (GEval)
9voice_tone_metric = GEval(
10    name="Voice Appropriateness",
11    criteria="Evaluate if the response is conversational, natural when read aloud, and avoids robotic lists, markdown syntax, or URLs.",
12    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
13    threshold=0.85
14)
15
16# Metric 2: Hard Turn Latency ceiling (under 400ms)
17latency_metric = LatencyMetric(max_latency=0.40)
18
19@pytest.mark.parametrize("caller_input,expected_tool", [
20    ("Book a table for four at seven PM tonight", "reserve_table"),
21    ("What time do you close on Sundays?", "query_business_hours"),
22    ("Transfer me to an agent right now", "transfer_to_human"),
23])
24def test_candidate_voice_model(caller_input, expected_tool):
25    # Execute turn against candidate model snapshot
26    response, latency, tool_called = run_turn(model="gemini-3.5-flash", user_text=caller_input)
27
28    test_case = LLMTestCase(
29        input=caller_input,
30        actual_output=response,
31        latency=latency
32    )
33
34    # Assert tool selection match
35    assert tool_called == expected_tool, f"Expected {expected_tool} but candidate triggered {tool_called}"
36
37    # Assert voice rubric and latency metrics
38    assert_test(test_case, [voice_tone_metric, latency_metric])

Statistical Hypothesis Testing for Offline Evaluations

When evaluating candidate models against baseline models across hundreds of test cases, raw pass rates are insufficient. If an incumbent model passes 94/100 tests and a candidate passes 96/100, is the new model genuinely superior, or is the difference random noise?

1. McNemar's Test for Paired Categorical Assertions

Because both models are tested on the exact same dataset, their outcomes are paired. McNemar's test specifically evaluates cases where the models disagree:

• b: Incumbent passed, Candidate failed (regressions)
• c: Incumbent failed, Candidate passed (improvements)

$\chi^2 = \frac{(|b - c| - 1)^2}{b + c}$

If $p < 0.05$, the performance difference between models is statistically significant rather than stochastic variance.

1# stats_eval.py
2from statsmodels.stats.contingency_tables import mcnemar
3
4def evaluate_model_significance(incumbent_results: list[bool], candidate_results: list[bool]):
5    # Contingency Table: [[both_pass, inc_pass_cand_fail], [inc_fail_cand_pass, both_fail]]
6    table = [[0, 0], [0, 0]]
7    for inc, cand in zip(incumbent_results, candidate_results):
8        if inc and cand:
9            table[0][0] += 1
10        elif inc and not cand:
11            table[0][1] += 1
12        elif not inc and cand:
13            table[1][0] += 1
14        else:
15            table[1][1] += 1
16
17    result = mcnemar(table, exact=True)
18    print(f"McNemar Statistic: {result.statistic:.4f}, p-value: {result.pvalue:.4f}")
19    if result.pvalue < 0.05:
20        print(">> Significant difference detected between model versions.")
21    else:
22        print(">> No statistically significant difference detected.")

2. Two-Sample Kolmogorov-Smirnov Test for Tail Latencies

To ascertain whether a candidate model's latency distribution has shifted (especially at P95 and P99), apply the Kolmogorov-Smirnov (KS) test to the latency samples:

1from scipy.stats import ks_2samp
2
3def test_latency_distribution_shift(incumbent_latencies: list[float], candidate_latencies: list[float]):
4    ks_stat, p_val = ks_2samp(incumbent_latencies, candidate_latencies)
5    print(f"KS Statistic: {ks_stat:.4f}, p-value: {p_val:.4f}")
6    if p_val < 0.05:
7        print("WARNING: Significant latency distribution shift detected.")

What's Next in Part 2

Unit tests and static assertions are only the first defense layer. A model that passes 100% of single-turn unit tests can still collapse during live multi-turn telephone calls due to context saturation, mid-turn interruptions, and unpredictable user disfluencies.

In Part 2: Simulation, Shadow Traffic, and Canary Rollouts, we implement:

• Tier 2 Bot-to-Bot User Simulation: Simulating adversarial personas and barge-in interruptions over virtual telephony channels.
• Tier 3 Production Shadow Traffic (Dark Launching): Cloning live STT audio streams to candidate models asynchronously using WebSocket event dispatchers and ClickHouse schemas.
• Tier 4 Canary Rollouts & Circuit Breakers: Staged traffic shifts (5% >> 25% >> 100%) with automated rollbacks.
• Multi-Model Fallback Architecture: Building low-latency failover gateways with local edge models on vLLM / SGLang.

References

1. Promptfoo Documentation
2. DeepEval LLM Evaluation Framework
3. Braintrust AI Evaluation Platform
4. Langfuse Open Source LLM Observability
5. Cartesia Sonic Real-Time Voice Synthesis
6. Deepgram Real-Time Speech Recognition
7. WebRTC Audio Processing Architecture