'Hayami,' a real-time multilingual speech recognition system that operates solely on the CPU, can perform live subtitle display, browser display, speaker labeling, and translated subtitles with less than 2GB of memory, without using a GPU or cloud API.

' Hayamimi ' is a lightweight, real-time multilingual speech recognition system that operates solely on the CPU, without relying on GPUs or cloud APIs. It supports live subtitle display, speaker labeling, and translated subtitle generation even in environments with less than 2GB of memory, and is characterized by its high speed, with subtitles starting to appear during speech and finalizing approximately 100ms after the speaker finishes speaking.

oboroge0/hayamimi: Hayami - Real-time multilingual speech-to-text on CPU only. Live subtitles, browser dashboard, speaker labels, translation. No GPU, no cloud.
https://github.com/oboroge0/hayamimi

While conventional CPU-based speech recognition tends to rely on a single model, Hayamimi employs a unique approach that determines the language for each utterance and assigns it to the optimal dedicated model. The model used is in INT8 quantized ONNX format and runs on sherpa-onnx, eliminating the need for PyTorch or CUDA. It achieves a high accuracy of 'Character Error Rate (CER) 3.8%' with actual Japanese audio from television broadcasts and achieves speeds 10 to 50 times faster than real time on a 6-core desktop CPU.

Hayamimi offers a wide range of features, as follows:

• Five language catalogs : In addition to the five major languages (Japanese, Chinese, Korean, Cantonese, and English), 24 European languages are processed by dedicated models, while the remaining approximately 1600 languages fall back to Meta Omnilingual ASR.
• Draft subtitles : The draft text is updated approximately every 0.5 seconds during speech.
- High-speed confirmation : In the case of Japanese, the confirmation sentence is output approximately 100ms after the speaker finishes speaking.
- Two-stage pass correction : After a 2-second silence, the previous utterance is re-decoded to generate a 'cleaned' transcript, improving the Japanese CER from 15.5% to 12.0%.
- Speaker labeling : Real-time speaker tagging using CAM++ with the '--speakers' option, followed by full-scale re-separation using pyannote/segmentation-3.0 in the final output pass.
- Translation : The '--translate' option provides real-time translation of Japanese subtitles into the specified language (e.g., en, zh, ko, es).
- Hotwords : Use '--hotwords' to enhance proper noun recognition.
- Replacement dictionary : Use '--replace' to perform post-search and replace.
- Converting Chinese numerals to Arabic numerals : Converting Chinese numerals to Arabic numerals in Japanese, Chinese, and Cantonese.
- Runtime dictionary API : Allows replacement dictionaries and ITN (Inverse Text Normalization) exceptions/mandates to be swapped during a session.
• Structured events + runtime configuration : Events can be published to EventHub and listened to at each stage of the pipeline.
- OBS Overlay + Dashboard : Use '--serve' to launch a local HTTP server and provide a browser source overlay and live dashboard.
- Network voice input : '--input ws' accepts voice input via WebSocket.
- Speaker loopback input : '--input speaker' (Windows only) supports transcription of the PC's internal speaker audio, and '--input mix' supports transcription of a loopback stream that combines microphone and speaker audio.
- English v2 tier (opt-in) : Switched to Parakeet TDT v2 with '--en-tier v2' and improved English WER from 10.0% to 6.6%.
- 4-class Japanese punctuation (opt-in) : Switch to a single model that directly predicts punctuation using '--punct-model 4class'.
- Memory limit management : LRU model backup controls the resident model to stay within the limit (default is 2GB).

To run Hayamimi, you need Python 3.10 or later and ffmpeg included in your PATH. First, clone the repository to an appropriate location.
[code]
git clone https://github.com/oboroge0/hayamimi.git
[code]

Next, we will create a virtual environment and install dependencies.
[code]
python -m venv .venv

# For Windows
.venv\Scripts\pip install -r requirements.txt

# For macOS / Linux
.venv/bin/pip install -r requirements.txt
[code]

Finally, download the model.
[code]
# For Windows
.venv\Scripts\python scripts/download_models.py

# For macOS / Linux
.venv/bin/python scripts/download_models.py
[code]

Specifying the '--minimal' option allows for a configuration of approximately 1.1GB containing only Japanese and English.

To perform real-time recognition, execute the following command:
[code]
# Microphone input (Windows)
.venv\Scripts\python scripts/realtime_transcribe.py

# Microphone input (macOS/Linux)
.venv/bin/python scripts/realtime_transcribe.py

# PC internal speaker input (Windows only)
.venv\Scripts\python scripts/realtime_transcribe.py --input speaker

# Microphone + speaker combined input
.venv\Scripts\python scripts/realtime_transcribe.py --input mix

# Dashboard + OBS overlay included
.venv\Scripts\python scripts/realtime_transcribe.py --serve
[code]

Hayamimi also comes with a demo UI. The '--serve' option starts a local server, allowing you to access the following three pages from your browser.

• http://localhost:8833/dashboard : Dashboard: Displays the spoken text, confirmed lines, translation, and final transcript.
- http://localhost:8833/ : OBS Overlay: Setting this as the browser source URL for OBS will display subtitles on the streaming screen.
• http://localhost:8833/transcript : Transcript page: Only the final version of the transcript is displayed.

The limitations of Hayamimi as of the time of writing this article are as follows:

- Does not support speech containing multiple languages within a single sentence.
- Short speech immediately following sound effects or background music may result in incorrect language detection.
Two speakers speaking simultaneously cannot be separated.
- Translation quality depends on the size of the model, so there is a possibility of errors in numbers and amounts in Chinese and Korean translations.
- The opt-in 4-class punctuation model does not output exclamation marks '!' by default.

Hayamimi is a lightweight and fast real-time multilingual speech recognition system that will continue to be a force to be reckoned with.