Damien/audio-summary-with-local-LLM
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update Voxtral STT model to non-realtime batch variant

Browse Source 
Use Voxtral-Mini-3B-2507-bf16 across all tiers and set appropriate
max_tokens for full-file transcription
This commit is contained in:
Damien
2026-06-17 11:25:04 +02:00
parent 425d4db1e2
commit 701f5dd56e
2 changed files with 18 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
src/audio_summary/models/config.py
View File

@@ -9,7 +9,8 @@ threshold is *less than or equal to* the detected memory is selected (see
``audio_summary.device.select_models``).
STT engines:
* ``"voxtral"`` -> Mistral Voxtral via ``mlx-audio`` (streaming, long audio).
* ``"voxtral"`` -> Mistral Voxtral (non-realtime batch model) via ``mlx-audio``.
Honors the requested language and transcribes whole files in one pass.
* ``"whisper"`` -> ``mlx-whisper`` (lightweight fallback for low-RAM Macs).
"""
@@ -42,17 +43,17 @@ TIERS: list[Tier] = [
),
Tier(
min_ram_gb=16,
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit"),
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-3B-2507-bf16"),
summarization_repo="mlx-community/Qwen3-8B-4bit",
),
Tier(
min_ram_gb=24,
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-4B-Realtime-2602-fp16"),
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-3B-2507-bf16"),
summarization_repo="mlx-community/Qwen3-8B-8bit",
),
Tier(
min_ram_gb=32,
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-4B-Realtime-2602-fp16"),
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-3B-2507-bf16"),
summarization_repo="mlx-community/Qwen3-30B-A3B-4bit",
),
]

18
src/audio_summary/pipeline/transcription.py
View File

@@ -2,22 +2,30 @@
Two engines are supported, selected through ``config.STTModel.engine``:
* ``voxtral`` -> Mistral Voxtral via ``mlx-audio``. Outperforms Whisper
large-v3 on most benchmarks and natively handles long audio with streaming.
* ``voxtral`` -> Mistral Voxtral (non-realtime batch model) via ``mlx-audio``.
Outperforms Whisper large-v3 on most benchmarks. The *realtime* Voxtral
variant is deliberately not used here: it ignores the requested language and
truncates when run one-shot on a full file.
* ``whisper`` -> ``mlx-whisper`` fallback for memory-constrained Macs.
"""
from ..models import STTModel
# Upper bound on generated tokens for a full-file transcription. The non-realtime
# Voxtral ``generate`` defaults to only 128 tokens, which would truncate any real
# recording, so we raise it well above the length of a long-form transcript.
_VOXTRAL_MAX_TOKENS = 32768
def _transcribe_voxtral(file_path: str, model_repo: str, language: str | None) -> str:
"""Transcribe using Mistral Voxtral through mlx-audio."""
from mlx_audio.stt.utils import load
model = load(model_repo)
# Voxtral handles long audio natively; ``generate`` returns an object with a
# ``.text`` attribute. ``language`` is optional (auto-detected when None).
kwargs: dict = {}
# The non-realtime Voxtral model transcribes the whole file in one pass and
# honors ``language``. ``generate`` returns an object with a ``.text``
# attribute. ``language`` is optional (auto-detected when None).
kwargs: dict = {"max_tokens": _VOXTRAL_MAX_TOKENS}
if language:
kwargs["language"] = language
result = model.generate(file_path, **kwargs)