Realtime systems / Applied ML

Speech Relay

  • Python
  • faster-whisper
  • IndicTrans2
  • WebRTC
  • ONNX
  • asyncio
2505 ms mean · 2580 ms p50
  1. Mic
  2. VAD
  3. ASR
  4. MT
  5. TTS
  6. WebRTC out
2505msmean end to end
Overview
Spoken translation needs more than a good model. I built an English-to-Hindi speech pipeline over WebRTC, connecting recognition, translation, and synthesis with an instrumented backend.
Architectural decision
Let speech boundaries drive the pipeline. Silero-VAD replaces fixed-window flushing to reduce the wait between speaking and hearing a translation.
Measured outcome
23.6% lower mean latency in the separate VAD comparison: 4,708 ms fixed-window → 3,597 ms VAD.
Implementation detail
The full-pipeline CPU profile measured 2,505 ms mean and 2,580 ms p50 across 11 utterances using faster-whisper tiny/int8, OPUS-MT, and edge-tts. These profile measurements are separate from the VAD comparison. A shared ASR stream fans out to multiple listeners; WER/CER evaluation tracks recognition quality alongside timing.

Silero-VAD replaced fixed-window flushing. Benchmark context and results are linked in the engineering notes.