Hugging Face Speech to Speech: Modular Open-Source Voice Agent Pipeline

Repositoryhuggingface/speech-to-speechTool

Hugging Face has released speech-to-speech, a modular open-source framework designed to build voice agents with open-source models. The pipeline spans VAD, STT, LLM, and TTS, exposing the core OpenAI Realtime GA event set over WebSocket and WebRTC.

Key points

  1. Architecture: Fully modular pipeline covering Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS).
  2. API Compatibility: Exposes core OpenAI Realtime GA events across WebSocket and WebRTC, enabling direct integration with clients like the official OpenAI Agents SDK.
  3. Model Support: Uses Parakeet TDT for STT and Qwen3-TTS for speech output out of the box; supports OpenAI-compatible LLMs via hosted endpoints or local engines like llama.cpp (e.g. Gemma 4) and vLLM.
  4. Production Validation: Operates in production as the conversation backend for thousands of Reachy Mini robots.

Why it matters

Proprietary real-time voice APIs frequently create vendor lock-in, data privacy hurdles, and recurring infrastructure costs. This project enables teams to deploy completely local or self-hosted open-source voice stacks while maintaining drop-in compatibility with OpenAI Realtime client code.

Analysis

The project is aimed at voice application developers, robotics engineers, and teams requiring low-latency conversational AI without cloud dependencies. Technically built on Python, WebRTC, WebSockets, Parakeet TDT, and Qwen3-TTS, it provides straightforward flexibility across hardware setups. With over 13,000 GitHub stars, PyPI distribution, and proven deployments across Reachy Mini robot fleets, the project shows solid maturity for production use.

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Similar in content, found by the AI search.

  • Repository:jamiepine/voicebox

    Voicebox: Open-Source Local AI Voice Studio and Dictation Tool

    Voicebox is a local-first, open-source voice studio combining text-to-speech, voice cloning, and system-wide dictation. It serves as a self-hosted alternative to proprietary cloud solutions such as ElevenLabs and WisprFlow.

    55.6kstarsTypeScript

    AI & LLMs· Tool

  • X post:MIKE

    Open-Source Local Alternative to ElevenLabs for Voice Cloning and TTS

    An X post highlights an open-source project by a solo developer that runs text-to-speech and voice cloning entirely on local hardware. The repository has accumulated 19.4k GitHub stars, presenting itself as a free self-hosted alternative to ElevenLabs.

    21.1kbookmarks1.1mviews

    AI & LLMs· Tool

  • Repository:calesthio/OpenMontage

    OpenMontage: Open-Source Agentic Video Production System

    OpenMontage is an open-source video production framework designed to turn AI coding assistants such as Cursor or Claude into automated video production studios. Built with Python, it provides 12 production pipelines, over 100 tools, and 700+ skill and knowledge files connecting speech, image, and video generation services with programmatic editing engines.

    56.2kstarsPython

    AI & LLMs· Tool

  • Article:Jakub Mroz, Henry Ndubuaku

    Whistle: Open-Source Speech-to-Text in 16.9 MB

    Cactus Compute has released Whistle, a lightweight speech recognition model packaged into a single 16.9 MB file. Running on CPU without external dependencies, it shares its C++ engine with Needle to convert audio directly into tool calls.

    AI & LLMs· Announcement

  • Repository:ShenSeanChen/waku-agent

    Waku Agent: Legible, Local-First AI Agent Harness in Python

    Waku Agent is a local-first AI assistant harness designed to avoid bloated framework abstractions. Built in straightforward Python, it implements an agent loop, tiered memory in SQLite, and built-in evaluations in an easily readable codebase.

    1.7kstarsPython

    AI & LLMs· Tool

Let's talk about your project

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.