The Test Tool That Became a Debugger — From a Unit-Test Baseline to a Live Voice Pipeline

Series: AgentZero Lite — Part 9. The previous post (Part 8 — When the Model Is Smarter Than Your Test Fixture: Building a TTS↔STT Round-Trip Suite) built a unit-test baseline that proved the model pair (Windows SAPI + Whisper.net) preserves Korean and English content end-to-end on file-based round-trips. This post is what happened when we put that pair on a live mic.

TL;DR

• Unit tests said "the model pair is fine." The live AskBot mic said otherwise. Six rounds of investigation closed the gap.

• The biggest single win was a buffer leak between utterances — captured PCM was 9–17 seconds for 2 seconds of speech, dragging Whisper into YouTube-creator-outro hallucinations on the silence padding.

• RMS-over-whole-clip is the wrong silence measure for streaming captures (silence padding dilutes it). Peak + voice-activity-ratio is right.

• The same user complaint kept recurring across multiple "fixes" because "bypass" was a bad name — my framing of the feature didn't match the user's mental model. Resolved by collapsing two modes into one.

• A WPF thread-affinity gotcha (Application.Current.Windows is dispatcher-owned) hid behind one of those mode-unification fixes.

• The test tool I built to drive the pipeline ended up being more useful as a debugger than as a tester. Every fix was discovered through a single new log line in it.

Why the unit test wasn't enough

• VAD threshold and how it's set (sensitivity slider, max'd by users who want to catch quiet speech).

• Pre-roll ring (the second of audio captured before the VAD trigger, to preserve the initial consonant).

• Utterance hangover (40 frames ≈ 2 s of trailing silence before declaring the utterance complete).

• Buffer accumulation between utterances.

• Windows mic enhancements (echo cancellation, noise suppression).

• Real-time HTTP latency to OpenAI.

• Thread affinity inside the WPF dispatcher.

Round 1 — Instrument the streaming pipe before guessing

[t0] utterance-start
[t1] utterance-end | t1-t0={Nms} · pcm={N bytes} (~Ns audio)
[t2] pipeline-start | enqueue-lag={Nms} · provider=X · lang=Y · pcm=~Ns
[stage] STT prep | {Nms} · ready=True
[stage] STT transcribe | {Nms} · {N.NN}x realtime · chars={N}
[t3] DONE | end-to-end {Nms} (utt-end → terminal) · enqueue · prep · transcribe · dispatch

Round 2 — The silent killer: buffer leak between utterances

[t1] utterance-end | t1-t0=2049ms · pcm=544000 bytes (~17.00s audio)
[t1] utterance-end | t1-t0=1999ms · pcm=312000 bytes (~9.75s audio)
[t1] utterance-end | t1-t0=2100ms · pcm=403200 bytes (~12.60s audio)

// VoiceCaptureService.OnDataAvailable — every audio frame, every time:
if (BufferPcm)
{
    lock (_pcmBuffer)
    {
        _pcmBuffer.AddRange(chunk);  // <- always appends. Always.
    }
}

if (above)
{
    if (!_inUtterance)
    {
        // Utterance starts — SeedBufferWithPreRoll on the consumer
        // side clears the buffer and adds pre-roll.
        _inUtterance = true;
        UtteranceStarted?.Invoke();
    }
    else
    {
        // Continuing in utterance, above threshold.
        if (BufferPcm) lock (_pcmBuffer) { _pcmBuffer.AddRange(chunk); }
    }
}
else if (_inUtterance)
{
    // Trailing silence — still inside utterance window per hangover.
    if (BufferPcm) lock (_pcmBuffer) { _pcmBuffer.AddRange(chunk); }
    _utteranceSilenceFrames++;
    if (_utteranceSilenceFrames >= UtteranceHangoverFrames)
    {
        _inUtterance = false;
        UtteranceEnded?.Invoke();
    }
}
// else: outside utterance, below threshold. Drop.

[t1] utterance-end | t1-t0=2050ms · pcm=97600 bytes (~3.05s audio)
[t1] utterance-end | t1-t0=2000ms · pcm=96000 bytes (~3.00s audio)
[t1] utterance-end | t1-t0=2399ms · pcm=96000 bytes (~3.00s audio)

Round 3 — Whisper hallucinated YouTube outros even on good audio

text="시청해 주셔서 감사합니다."
text="시청해주셔서 감사합니다! 구독 좋아요 댓글 부탁드려요!"
text="오늘도 시청해 주셔서 감사합니다!"

const double MinSpeechPeakDbfs     = -38.0;   // any audible moment in the clip
const double MinFrameLoudDbfs      = -45.0;   // per-frame "active" threshold
const double MinVoiceActivityRatio =  0.10;   // ≥ 10% of 50 ms frames active

if (peakDb < MinSpeechPeakDbfs)             return drop;  // pure silence
if (var50ms < MinVoiceActivityRatio)        return drop;  // single click

[t2] pipeline-start | ... · peak=-27.3dBFS · rms=-52.3dBFS · VAR=16.7%
[stage] STT transcribe | ... · text="안녕하세요"
[t3] DONE | ...

[t2] pipeline-start | ... · peak=-22.0dBFS · rms=-58.0dBFS · VAR=1.7%
dropped — VAR gate (active-frame ratio=1.7% < 10% · likely brief click/tap, not sustained speech)

Round 4 — Mute and volume as user-controlled prerequisites

• MicMuted: bool — soft mute. Capture stays alive (level meter still moves) but VAD/STT pipeline ignores frames. Persisted; applied on every mic-on.

• MicVolumePercent: int (-1 = unset). Slider 0–100 controlling the Windows default capture endpoint's master volume via WASAPI (MMDeviceEnumerator + AudioEndpointVolume.MasterVolumeLevelScalar). System-wide effect — same as moving the slider in Sound Settings.

Round 5 — Mode unification: "bypass" was a bad name

• Bypass (default): synthesise → STT → show in popup. Don't drive AskBot.

• Acoustic loop (off by default): synthesise → speaker → mic → STT → AskBot terminal.

1. Added auto-mute checkbox so AskBot wouldn't pick up speaker echo during bypass tests.

2. Rebuilt acoustic loop to do direct STT (no mic round-trip) but still play through the speaker for human verification.

3. Updated the description text to explain the new semantics.

가상음성은 ... 음성재생을 일으켜 ... AskBot이 음성입력모드일때 자연스럽게 감지되어 반응함

• Collapse the two modes. There is now ONE turn shape: synth → STT → inject into AskBot.

• The single remaining checkbox controls only "play through speaker for verification" — defaults to off. The transcript injection happens regardless.

• Mode line below the active-models row reads either "silent (no speaker playback)" or "ALSO speaker playback (verification)". No more "bypass."

Round 6 — A WPF thread-affinity bug hiding inside the unification

"자원 점유 에러나서 안되는것같음, 테스트용은 생성 하나더 해야할것같음 (동시사용 성문제로보임)"

System.InvalidOperationException: 다른 스레드가 이 개체를 소유하고 있어
   at System.Windows.Application.get_Windows()
   at TestToolsWindow.FindAskBot() — line 182
   at <RunTurnAsync>b__0.MoveNext() — inside Task.Run

// Before Task.Run, on the UI thread:
var bot = FindAskBot();

await Task.Run(async () =>
{
    // ... synth, decode, STT ...
    bot?.SendVoiceTranscript(transcript);  // safe — dispatcher-marshals internally
});

What the test tool actually taught

• Per-stage timing log was the primary instrument. It surfaced the buffer leak, the energy-gate-vs-RMS mismatch, the thread-affinity throw, and the structural 2-second VAD hangover. Every fix was triggered by one specific log line.

• Acoustic loops are unreliable on Windows. The native echo-cancellation + noise-suppression filters can zero out the speaker output reaching the mic. For automated voice testing, don't depend on the round trip — feed the synthesised audio directly to STT and play through the speaker only as human verification.

• RMS-over-whole-clip is the wrong silence measure when the audio is structurally "speech burst inside a silence envelope." Peak + voice-activity-ratio is right.

• Naming is requirements, not implementation. When the same complaint keeps surfacing across multiple "fixes," the disagreement is what the feature is for, not how it's coded. Renaming and collapsing modes can resolve more bugs than another patch.

• Capture WPF dispatcher-owned objects on the UI thread before Task.Run. Application.Current.Windows and a few other globals throw on background-thread access. Most member calls then marshal back via Dispatcher.BeginInvoke automatically, so closing over the reference is enough.

What's still on the table

• Whisper Small vs Medium accuracy comparison. The unit test uses Medium for quality measurement; the running app uses Small for speed. A [Theory] row that runs both on the same fixtures would quantify the trade-off on the actual deployed phrases.

• GPU acceleration. Vulkan / CUDA Whisper.net runtimes are wired but not on. The unit test's per-stage timings give a clean baseline to compare against once they are.

• Reactor token streaming. The original Akka.Streams Step 2 promise (Part 7) was per-token streaming all the way to the speaker. Currently AskBot waits for the full reactor result before TTS. The plumbing exists; the wiring is one more cycle of work.

• VAD hangover tuning. Currently 40 frames (2 s). At 20 frames (1 s) you'd shave a second off the perceived latency at the cost of mid-utterance pauses occasionally splitting into two segments. Worth measuring.

Closing

• psmon

TECH LINKS

• 𝕏 @webnori

• 📘 Akka Labs (Facebook group)

• 📝 More writings — webnori wiki