Realtime systems / Applied ML
Speech Relay
- Python
- faster-whisper
- IndicTrans2
- WebRTC
- ONNX
- asyncio
- Mic
- VAD
- ASR
- MT
- TTS
- WebRTC out
2505msmean end to end
- Overview
- Spoken translation needs more than a good model. I built an English-to-Hindi speech pipeline over WebRTC, connecting recognition, translation, and synthesis with an instrumented backend.
- Architectural decision
- Let speech boundaries drive the pipeline. Silero-VAD replaces fixed-window flushing to reduce the wait between speaking and hearing a translation.
- Measured outcome
- 23.6% lower mean latency in the separate VAD comparison: 4,708 ms fixed-window → 3,597 ms VAD.
- Implementation detail
- The full-pipeline CPU profile measured 2,505 ms mean and 2,580 ms p50 across 11 utterances using faster-whisper tiny/int8, OPUS-MT, and edge-tts. These profile measurements are separate from the VAD comparison. A shared ASR stream fans out to multiple listeners; WER/CER evaluation tracks recognition quality alongside timing.
Silero-VAD replaced fixed-window flushing. Benchmark context and results are linked in the engineering notes.