When the Model Is Smarter Than Your Test Fixture — Building a TTS↔STT Round-Trip Suite for AgentZero Lite

Series: AgentZero Lite — Part 8. The previous post (Part 7 — Step 2 Landed: Akka.NET Streams Now Carry the Whole Voice Path) put a streaming substrate under the voice subsystem. This one puts a measurement instrument on top — so we know whether it's getting better or worse between commits.

TL;DR

• 5-case xUnit suite that runs TTS (Windows SAPI default voice) → WAV → resample → STT (Whisper.net medium, CPU) and compares the transcript to the input.

• Pure-model, file-based, no Akka, no streams. Deterministic baseline; the streaming variant becomes measurable as a delta against it.

• Comparison folds case + punctuation + whitespace before equality so tokeniser cosmetics ("hello" → "Hello.") aren't reported as content drift.

• 4 / 5 pass. The one fail is the interesting case — Whisper auto-corrected the user's input typo 모래 (sand) to the contextually correct 모레 (day after tomorrow). Edit distance 1, similarity 96.8 %. The test reports the drift honestly; the post argues it should.

• Six lessons worth writing down so the next person doesn't re-derive them. The middle four are technique; the bookends are philosophy.

Why we needed this now

• No mic, no speaker. The acoustic path has its own problems (Windows mic enhancements, room noise, hardware variance). Validating the raw model pair means a file-based round-trip — synthesise to WAV, hand the WAV to STT.

• No actors, no streams. Those are operational concerns. The question here is "what do TTS and STT do to a string when neither has any other moving parts in the way?"

• Real providers. Mocking either end defeats the point — we want to know the actual hallucination rate of the actual deployed pipeline, not the rate of a mock.

flowchart LR
    A["input string"] --> B["WindowsTts.SynthesizeAsync"]
    B --> C["WAV bytes"]
    C --> D["File.WriteAllBytes<br/>*-tts.wav"]
    C --> E["WavToPcm.To16kMono<br/>(NAudio resample)"]
    E --> F["PCM 16k mono"]
    F --> G["File.WriteAllBytes<br/>*-stt-input-16k.wav"]
    F --> H["WhisperLocalStt.TranscribeAsync<br/>(medium, CPU)"]
    H --> I["transcript string"]
    A --> J["Normalize<br/>(case + punct + WS)"]
    I --> J
    J --> K{"== ?"}
    K --> L["VERDICT + diff + timings"]

    classDef io fill:#1f2937,stroke:#0ea5e9,color:#e5e7eb
    classDef pure fill:#0b3d2e,stroke:#10b981,color:#d1fae5
    class B,H,E pure
    class D,G io

[Fact] public Task Korean_short_안녕하세요()         => RunRoundTrip("안녕하세요", language: "ko", caseId: "ko-short");
[Fact] public Task Korean_question_today_weather()  => RunRoundTrip("오늘의 날씨는 어때?", language: "ko", caseId: "ko-question");
[Fact] public Task Korean_long_multi_clause()       => RunRoundTrip(
    "내일의 날씨는 말고 모래의날씨는 흐리고 그리고 주간내내 비올예정입니다.",
    language: "ko", caseId: "ko-long");
[Fact] public Task English_short_hello()            => RunRoundTrip("hello", language: "en", caseId: "en-short");
[Fact] public Task English_question_how_are_you()   => RunRoundTrip("how are you?", language: "en", caseId: "en-question");

Lesson 1 — Comparison: == is too strict, Contains is too loose

private static string Normalize(string s)
{
    if (string.IsNullOrEmpty(s)) return string.Empty;
    var sb = new StringBuilder(s.Length);
    foreach (var c in s)
    {
        if (char.IsWhiteSpace(c)) continue;
        if (char.IsPunctuation(c)) continue;   // covers .,?!;:'\"() and Unicode kin
        sb.Append(char.ToLowerInvariant(c));    // no-op for Korean
    }
    return sb.ToString();
}

• char.IsPunctuation instead of a hard-coded list. It picks up smart quotes, em-dashes, and full-width punctuation that Whisper happily emits for Korean text.

• ToLowerInvariant is unconditional. Hangul has no case so applying it costs nothing, and it means the same Normalize works for both languages. Branchless code that's trivially right is worth more than language-aware cleverness.

Lesson 2 — Whisper auto-corrects the fixture's typo. Test still fails. Good.

─── COMPARISON ────────────────────────────────────────────────────────
normalised input   : "내일의날씨는말고모래의날씨는흐리고그리고주간내내비올예정입니다"
normalised output  : "내일의날씨는말고모레의날씨는흐리고그리고주간내내비올예정입니다"
exact match        : False
edit distance      : 1
similarity         : 96.8%

VERDICT            : ✗ FAIL — see comparison

"It's only one character. Bump the threshold to >= 0.95 PASS. Move on."

1. The 1-character drift is exactly the signal you wanted. The whole point of the suite is to know whether TTS+STT is preserving content. Whisper changed the content. That's the answer to the question. Hiding it because the change is small and arguably correct doesn't preserve the answer; it discards it.

2. The next change won't be benign. Today it's a homophone correction nobody objects to. Tomorrow it's 송파구 → 송파의 구 (Songpa's borough) because the model thinks the user meant a possessive — also a single substitution, also "smart", also breaks downstream code that's looking for an exact district name. A fuzzy threshold won't tell you which one is happening.

3. The console dump tells the operator everything. normalised input and normalised output printed side-by-side — if the diff is 모래 → 모레, the reviewer sees "ah, the model fixed my typo" and can move on. If it's 송파구 → 송파의 구, the reviewer sees the actual problem. Both are 1-edit-distance failures; they're not the same kind of problem.

Lesson 3 — Dual output channel: ITestOutputHelper and Console.WriteLine

private void Log(string line)
{
    try { _output.WriteLine(line); } catch { /* helper may be disposed during failed assert teardown */ }
    Console.WriteLine(line);
}

Lesson 4 — Always save the audio as evidence

%TEMP%\agentzero-tts-stt-tests\
  20260429-180920-162-ko-long-tts.wav             ← what SAPI produced
  20260429-180920-162-ko-long-stt-input-16k.wav   ← what Whisper actually heard

Lesson 5 — Performance baseline. Record now, regress later.

• xRT improves on longer audio. Whisper's per-call overhead — model load, prompt setup, beam-search initialisation — dominates short clips. The 1.68× on the 5.93-second utterance is closer to the steady-state CPU performance of the medium model. Don't draw conclusions from the ~5× xRT on a half-second clip.

• TTS is two orders of magnitude faster than STT. Synthesis is rarely the bottleneck. If the round-trip is slow, look at STT first.

• First test in a run pays the model load (~9 s prep). Subsequent tests in the same suite hit cache (prep: 0 ms in the log). A 5-test suite costs ~50 seconds wall-clock end-to-end — fine for local iteration, fine for CI.

• a different STT (cloud Whisper, Webnori Gemma, on-device Gemma)

• a different model size (tiny / small / large)

• GPU acceleration (Vulkan / CUDA)

Lesson 6 — InternalsVisibleTo is for state, not for helpers

What the console actually looks like

═══════════════════════════════════════════════════════════════════════
Round-trip case: ko-question
═══════════════════════════════════════════════════════════════════════
input              : "오늘의 날씨는 어때?"
language           : ko
input chars        : 11
TTS provider       : WindowsTTS (SAPI default voice)
STT provider       : WhisperLocal (model=medium)
OS                 : Microsoft Windows NT 10.0.26200.0
.NET               : 10.0.7

TTS voices available: Microsoft Heami Desktop, Microsoft Zira Desktop, ...
[STAGE 1/3] TTS    : 33 ms · 96150 bytes
WAV saved          : C:\...\agentzero-tts-stt-tests\20260429-180759-870-ko-question-tts.wav
[STAGE 2/3] decode : 1 ms · 69736 pcm bytes (~2.18s audio)
PCM level          : peak=-5.2 dBFS · rms=-21.3 dBFS
STT-input WAV saved: C:\...\agentzero-tts-stt-tests\20260429-180759-870-ko-question-stt-input-16k.wav
STT prep           : 0 ms · ready=True
[STAGE 3/3] STT    : 8915 ms · 4.09x realtime
transcript         : "오늘의 날씨는 어때?"
transcript chars   : 11

─── COMPARISON ────────────────────────────────────────────────────────
normalised input   : "오늘의날씨는어때"
normalised output  : "오늘의날씨는어때"
exact match        : True · edit distance: 0 · similarity: 100.0%

TIMING TOTAL       : 8949 ms (33 synth + 1 decode + 8915 STT)
VERDICT            : ✓ PASS
═══════════════════════════════════════════════════════════════════════

What this enables next

• Cloud Whisper swap — same suite with OpenAiWhisperStt instead of WhisperLocalStt. We get a direct latency / quality / cost trade-off table.

• Whisper small vs medium — currently the running app defaults to small for cost/speed; the test suite uses medium for quality measurement. Add a [Theory] row that runs both per fixture and we can see the small-vs-medium accuracy gap on the actual deployed fixtures, not on synthetic benchmarks.

• Streaming-graph regression — once the Akka.Streams output graph chunks per sentence and runs progressive TTS, the same five fixtures in a streaming-flavor variant will catch any regression vs the file-based reference here.

• GPU acceleration — the xRT column in the baseline becomes the "is this faster" check when we wire the Vulkan / CUDA Whisper.net runtimes.

Closing

• psmon

NEXT : TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki