Damien/audio-summary-with-local-LLM
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'feat/mlx-native' (#2) from feat/mlx-native into main

Browse Source 
Reviewed-on: #2
This commit was merged in pull request #2.
This commit is contained in:
Damien Arnodo
2026-06-17 12:42:55 +00:00
parent bda2b516a4 383c1bd966
commit 5e7046ac5a
16 changed files with 1505 additions and 710 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored
View File

@@ -1,6 +1,7 @@
src/__pycache__
__pycache__/
*.pyc
.ruff_cache/
src/audio_summary_with_local_LLM.egg-info/
*.egg-info/
# Virtual Env
.venv

287
README.md
View File

@@ -1,60 +1,77 @@
# Audio Summary with Local LLM
# Audio Summary with Local LLM (Apple Silicon / MLX)
This tool is designed to provide a quick and concise summary of audio and video files. It supports summarizing content either from a local file or directly from YouTube. The tool uses Whisper for transcription and a local version of Llama3 (via Ollama) for generating summaries.
This tool provides a quick and concise summary of audio and video files. It supports
summarizing content either from a local file or directly from YouTube. Everything runs
**fully on Apple Silicon** through Apple's [MLX](https://github.com/ml-explore/mlx)
framework — no NVIDIA/CUDA, no PyTorch CUDA stack, no external server required.
> [!TIP]
> It is possible to change the model you wish to use.
> To do this, change the `OLLAMA_MODEL` variable, and download the associated model via [ollama](https://github.com/ollama/ollama)
- **Speech-to-Text**: Mistral [Voxtral](https://huggingface.co/mistralai) via
[`mlx-audio`](https://github.com/Blaizzy/mlx-audio) (with an
[`mlx-whisper`](https://github.com/ml-explore/mlx-examples) fallback for
memory-constrained Macs). Voxtral outperforms Whisper large-v3 on most transcription
benchmarks and natively handles long audio with streaming.
- **Summarization**: [Qwen3](https://huggingface.co/Qwen) via
[`mlx-lm`](https://github.com/ml-explore/mlx-lm) — the recommended family for
summarization on Apple Silicon.
## Automatic model selection
At startup the tool detects the machine's **unified memory** and automatically selects
the best models for the available RAM. The detected memory and chosen models are printed
before any work begins.
| RAM | STT model | Summarization model |
|---------|--------------------------------------------|---------------------------|
| 8 GB | `whisper-large-v3-turbo` (mlx-whisper) | `Qwen3-4B-4bit` |
| 16 GB | `Voxtral-Mini-4B-Realtime-2602-4bit` | `Qwen3-8B-4bit` |
| 24 GB | `Voxtral-Mini-4B-Realtime-2602-fp16` | `Qwen3-8B-8bit` |
| 32 GB+ | `Voxtral-Mini-4B-Realtime-2602-fp16` | `Qwen3-30B-A3B-4bit` (MoE)|
The selection table lives in [`src/audio_summary/models/config.py`](src/audio_summary/models/config.py)
and can be edited without touching the core logic. The highest tier whose RAM requirement
fits the detected memory is selected.
> [!NOTE]
> Models are downloaded from the Hugging Face Hub on first use and cached locally
> (`~/.cache/huggingface`). The first run for a given model may take a while.
## Features
- **YouTube Integration**: Download and summarize content directly from YouTube.
- **Local File Support**: Summarize audio/video files available on your local disk.
- **Transcription**: Converts audio content to text using Whisper.
- **Summarization**: Generates a concise summary using Llama3 (Ollama).
- **Transcript Only Option**: Option to only transcribe the audio content without generating a summary.
- **Device Optimization**: Automatically uses the best available hardware (MPS for Mac, CUDA for NVIDIA GPUs, or CPU).
- **Transcription**: Converts audio to text with Voxtral (or Whisper fallback).
- **Summarization**: Generates a concise summary with Qwen3.
- **Transcript Only Option**: Only transcribe, without generating a summary.
- **Automatic model selection**: Picks the best models for your Mac's unified memory.
## Prerequisites
Before you start using this tool, you need to install the following dependencies:
- Python 3.12 and lower than 3.13
- [Ollama](https://ollama.com) for LLM model management
- `ffmpeg` (required for audio processing)
- An Apple Silicon Mac (M1 or newer)
- Python 3.12 (and lower than 3.13)
- `ffmpeg` (required for audio extraction/processing)
- [uv](https://docs.astral.sh/uv/getting-started/installation/) for package management
Install `ffmpeg` with [Homebrew](https://brew.sh):
```bash
brew install ffmpeg
```
## Installation
### Using uv
Clone the repository and install the required Python packages using [uv](https://github.com/astral-sh/uv):
Clone the repository and install dependencies with [uv](https://github.com/astral-sh/uv):
```bash
git clone https://github.com/damienarnodo/audio-summary-with-local-LLM.git
cd audio-summary-with-local-LLM
# Create and activate a virtual environment with uv
uv sync
source .venv/bin/activate # On Windows: .venv\Scripts\activate
```
### LLM Requirement
[Download and install](https://ollama.com) Ollama to carry out LLM Management. More details about LLM models supported can be found on the Ollama [GitHub](https://github.com/ollama/ollama).
Download and use the Llama3 model:
```bash
ollama pull llama3
# Test the access:
ollama run llama3 "tell me a joke"
source .venv/bin/activate # activate the virtual environment
```
## Usage with uvx (no installation required)
You can run this tool directly without cloning the repository using `uvx`:
You can run the tool directly without cloning the repository using `uvx`:
```bash
# Summarize a YouTube video
@@ -70,172 +87,102 @@ uvx --from git+https://github.com/damienarnodo/audio-summary-with-local-LLM.git
uvx --from git+https://github.com/damienarnodo/audio-summary-with-local-LLM.git audio-summary --from-local <path-to-audio-file> --language fr --output my_summary.md
```
> [!NOTE]
> `uvx` automatically creates a temporary virtual environment and installs all dependencies. On first run, this may take a moment. Ensure `ffmpeg` and [Ollama](https://ollama.com) are installed and running on your system.
### Shell alias (optional)
To avoid typing the full `uvx --from ...` command every time, add an alias to your
`~/.zshrc`:
```bash
# Aliases
alias audio-summary="uvx --from git+https://github.com/damienarnodo/audio-summary-with-local-LLM.git audio-summary"
```
Reload your shell (`source ~/.zshrc`) and you can then call it directly:
```bash
audio-summary --from-youtube <YouTube-Video-URL>
audio-summary --from-local <path-to-audio-file> --language fr --output my_summary.md
```
## Usage
The tool can be executed with the following command line options:
The CLI options are:
- `--from-youtube`: To download and summarize a video from YouTube.
- `--from-local`: To load and summarize an audio or video file from the local disk.
- `--output`: Specify the output file path (default: ./summary.md)
- `--transcript-only`: To only transcribe the audio content without generating a summary.
- `--language`: Select the language to be used for the transcription (default: en)
- `--from-youtube`: Download and summarize a video from YouTube.
- `--from-local`: Load and summarize a local audio or video file.
- `--output`: Output markdown file path (default: `./summary.md`).
- `--transcript-only`: Only transcribe, do not summarize.
- `--language`: Language code for transcription (e.g. `en`, `fr`, `es`, or `auto` for
automatic detection). Default: `en`.
### Examples
1. **Summarizing a YouTube video:**
```bash
# Summarize a YouTube video
audio-summary --from-youtube <YouTube-Video-URL>
```bash
uv run python src/summary.py --from-youtube <YouTube-Video-URL>
```
# Summarize a local audio file
audio-summary --from-local <path-to-audio-file>
2. **Summarizing a local audio file:**
# Transcribe only (no summary)
audio-summary --from-youtube <YouTube-Video-URL> --transcript-only
```bash
uv run python src/summary.py --from-local <path-to-audio-file>
```
# Custom language and output file
audio-summary --from-local <path-to-audio-file> --language fr --output my_summary.md
```
3. **Transcribing a YouTube video without summarizing:**
```bash
uv run python src/summary.py --from-youtube <YouTube-Video-URL> --transcript-only
```
4. **Transcribing a local audio file without summarizing:**
```bash
uv run python src/summary.py --from-local <path-to-audio-file> --transcript-only
```
5. **Specifying a custom output file:**
```bash
uv run python src/summary.py --from-youtube <YouTube-Video-URL> --output my_summary.md
```
The output summary will be saved in a markdown file in the specified output directory, while the transcript will be saved in the temporary directory.
When running from a clone you can also use `uv run audio-summary ...` or
`uv run python -m audio_summary.cli ...`.
## Output
The summarized content is saved as a markdown file (default: `summary.md`) in the current working directory. This file includes a title and a concise summary of the content. The transcript is saved in the `tmp/transcript.txt` file.
The summary is written to a markdown file (default: `summary.md`) in the current working
directory, with a title and concise summary. The transcript is saved to
`tmp/transcript.txt`.
## Hardware Acceleration
## Project structure
The tool automatically detects and uses the best available hardware:
The code is organized into focused subpackages under `src/audio_summary/`:
- MPS (Metal Performance Shaders) for Apple Silicon Macs
- CUDA for NVIDIA GPUs
- Falls back to CPU when neither is available
| Module | Responsibility |
|------------------------------|-----------------------------------------------------------|
| `models/config.py` | Model selection table (RAM tiers → STT/summarization). |
| `models/device.py` | Unified-memory detection and tier selection (`psutil`). |
| `pipeline/download.py` | YouTube audio download (`yt-dlp` + `ffmpeg`). |
| `pipeline/transcription.py` | Speech-to-Text (Voxtral via `mlx-audio`, Whisper fallback).|
| `pipeline/summarization.py` | Summarization with Qwen3 via `mlx-lm`. |
| `utils/helpers.py` | Shared I/O, CLI argument, and console-output helpers. |
| `cli.py` | Argument parsing and orchestration (entrypoint). |
### Handling Longer Audio Files
## Customizing the models
This tool can process audio files of any length. For files longer than 30 seconds, the script automatically:
1. Chunks the audio into manageable segments
2. Processes each chunk separately
3. Combines the results into a single transcript
## Sources
- [YouTube Video Summarizer with OpenAI Whisper and GPT](https://github.com/mirabdullahyaser/Summarizing-Youtube-Videos-with-OpenAI-Whisper-and-GPT-3/tree/master)
- [Ollama GitHub Repository](https://github.com/ollama/ollama)
- [Transformers by Hugging Face](https://huggingface.co/docs/transformers/index)
- [yt-dlp Documentation](https://github.com/yt-dlp/yt-dlp)
Edit the `TIERS` table in [`src/audio_summary/models/config.py`](src/audio_summary/models/config.py).
Each tier declares a minimum RAM threshold, an STT model (engine + Hugging Face repo),
and a summarization repo. For example, to use a different Qwen3 size or a different
Voxtral quantization, just change the relevant repo string — no other code changes needed.
## Troubleshooting
### ffmpeg not found
If you encounter this error::
If you encounter:
```bash
yt_dlp.utils.DownloadError: ERROR: Postprocessing: ffprobe and ffmpeg not found. Please install or provide the path using --ffmpeg-location
yt_dlp.utils.DownloadError: ERROR: Postprocessing: ffprobe and ffmpeg not found.
```
Please refer to [this post](https://www.reddit.com/r/StacherIO/wiki/ffmpeg/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)
Install ffmpeg with `brew install ffmpeg`.
### Audio Format Issues
### Out of memory
If you encounter this error:
If a model is too large for your machine, lower the relevant tier's model in
`config.py` (e.g. switch a `fp16` STT model to its `4bit` variant, or pick a smaller
Qwen3 size). The smallest tier uses the lightweight `mlx-whisper` engine.
```bash
ValueError: Soundfile is either not in the correct format or is malformed. Ensure that the soundfile has a valid audio file extension (e.g. wav, flac or mp3) and is not corrupted.
```
## Sources
Try converting your file with ffmpeg:
```bash
ffmpeg -i my_file.mp4 -movflags faststart my_file_fixed.mp4
```
### Memory Issues on CPU
If you're running on CPU and encounter memory issues during transcription, consider:
1. Using a smaller Whisper model
2. Processing shorter audio segments
3. Ensuring you have sufficient RAM available
### Slow Transcription
Transcription can be slow on CPU. For best performance:
1. Use a machine with GPU or Apple Silicon (MPS)
2. Keep audio files under 10 minutes when possible
3. Close other resource-intensive applications
### Update the Whisper or LLM Model
You can easily change the models used for transcription and summarization by modifying the variables at the top of the script:
```python
# Default models
OLLAMA_MODEL = "llama3"
WHISPER_MODEL = "openai/whisper-large-v2"
```
#### Changing the Whisper Model
To use a different Whisper model for transcription:
1. Update the `WHISPER_MODEL` variable with one of these options:
- `"openai/whisper-tiny"` (fastest, least accurate)
- `"openai/whisper-base"` (faster, less accurate)
- `"openai/whisper-small"` (balanced)
- `"openai/whisper-medium"` (slower, more accurate)
- `"openai/whisper-large-v2"` (slowest, most accurate)
2. Example:
```python
WHISPER_MODEL = "openai/whisper-medium" # A good balance between speed and accuracy
```
For CPU-only systems, using a smaller model like `whisper-base` is recommended for better performance.
#### Changing the LLM Model
To use a different model for summarization:
1. First, pull the desired model with Ollama:
```bash
ollama pull mistral # or any other supported model
```
2. Then update the `OLLAMA_MODEL` variable:
```python
OLLAMA_MODEL = "mistral" # or any other model you've pulled
```
3. Popular alternatives include:
- `"llama3"` (default)
- `"mistral"`
- `"llama2"`
- `"gemma:7b"`
- `"phi"`
For a complete list of available models, visit the [Ollama model library](https://ollama.com/library).
- [MLX](https://github.com/ml-explore/mlx) — Apple's array framework for Apple Silicon
- [mlx-audio](https://github.com/Blaizzy/mlx-audio) — TTS/STT on MLX (Voxtral)
- [mlx-lm](https://github.com/ml-explore/mlx-lm) — LLM inference on MLX (Qwen3)
- [mlx-whisper](https://github.com/ml-explore/mlx-examples) — Whisper on MLX
- [yt-dlp](https://github.com/yt-dlp/yt-dlp) — YouTube downloader

32
pyproject.toml
View File

@@ -4,32 +4,28 @@ build-backend = "setuptools.build_meta"
[project]
name = "audio-summary-with-local-LLM"
version = "0.1.0"
description = 'Sum up your local or remote files with a local LLM'
keywords = ["audio", "summary", "local-llm", "ollama", "whisper"]
version = "0.2.0"
description = 'Transcribe and summarize audio/video files fully on Apple Silicon via MLX'
keywords = ["audio", "summary", "local-llm", "mlx", "apple-silicon", "voxtral", "qwen3"]
readme = "README.md"
requires-python = ">=3.12, <3.13"
authors = [
{ name = "darnodo", email = "sepales.pret0h@icloud.com" },
]
dependencies = [
"ffmpeg>=1.4",
"ollama>=0.4.7",
"openai-whisper>=20240930",
"torch>=2.6.0",
"torchaudio>=2.6.0",
"torchvision>=0.21.0",
"transformers>=4.50.2",
"mlx-lm>=0.21.0",
"mlx-audio>=0.2.0",
"mlx-whisper>=0.4.0",
"psutil>=6.0.0",
"yt-dlp>=2025.3.27",
"librosa>=0.11.0",
"mistral-common>=1.11.3",
]
[project.scripts]
audio-summary = "summary:main"
audio-summary = "audio_summary.cli:main"
[tool.setuptools]
py-modules = ["summary"]
[tool.setuptools.package-dir]
"" = "src"
[tool.setuptools.packages.find]
where = ["src"]
[tool.ruff]
# Exclude a variety of commonly ignored directories.
@@ -66,8 +62,8 @@ exclude = [
line-length = 88
indent-width = 4
# Assume Python 3.8
target-version = "py38"
# Match the project's required Python version.
target-version = "py312"
[tool.ruff.lint]
# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`) codes by default.

1
src/audio_summary/__init__.py Normal file
View File

@@ -0,0 +1 @@
"""Transcribe and summarize audio/video files on Apple Silicon via MLX."""

87
src/audio_summary/cli.py Normal file
View File

@@ -0,0 +1,87 @@
"""Command-line entrypoint.
Download (optional), transcribe, and summarize audio/video files fully on
Apple Silicon via MLX. Models are selected automatically from the detected
unified memory (see ``models.config`` and ``models.device``).
"""
import argparse
from pathlib import Path
from .models import select_models
from .pipeline import download_from_youtube, summarize_text, transcribe_file
from .utils import (
ensure_directory,
print_model_banner,
resolve_language,
write_summary,
)
def _parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Download, transcribe, and summarize audio or video files "
"on Apple Silicon (MLX)."
)
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--from-youtube", type=str, help="YouTube URL to download.")
group.add_argument("--from-local", type=str, help="Path to the local audio file.")
parser.add_argument(
"--output",
type=str,
default="./summary.md",
help="Output markdown file path.",
)
parser.add_argument(
"--transcript-only",
action="store_true",
help="Only transcribe the file, do not summarize.",
)
parser.add_argument(
"--language",
type=str,
help="Language code for transcription (e.g. 'en', 'fr', 'es', "
"or 'auto' for detection).",
)
return parser.parse_args()
def main() -> None:
args = _parse_args()
language = resolve_language(args.language)
# --- Automatic model selection based on detected unified memory ---
ram_gb, tier = select_models()
print_model_banner(ram_gb, tier)
# --- Working directory for intermediate files ---
data_directory = ensure_directory("tmp")
# --- Resolve input file ---
if args.from_youtube:
print(f"Downloading YouTube video from {args.from_youtube}")
file_path = download_from_youtube(args.from_youtube, str(data_directory))
else:
file_path = Path(args.from_local)
# --- Transcription ---
print(f"Transcribing file: {file_path}")
transcript = transcribe_file(
str(file_path),
str(data_directory / "transcript.txt"),
tier.stt,
language,
)
if args.transcript_only:
print("Transcription complete. Skipping summary generation.")
return
# --- Summarization ---
print("Generating summary...")
summary = summarize_text(transcript, tier.summarization_repo)
write_summary(args.output, summary)
if __name__ == "__main__":
main()

13
src/audio_summary/models/__init__.py Normal file
View File

@@ -0,0 +1,13 @@
"""Model configuration and hardware-based model selection."""
from .config import TIERS, STTModel, Tier
from .device import detect_ram_gb, select_models, select_tier
__all__ = [
"TIERS",
"STTModel",
"Tier",
"detect_ram_gb",
"select_models",
"select_tier",
]

59
src/audio_summary/models/config.py Normal file
View File

@@ -0,0 +1,59 @@
"""Model configuration for Apple Silicon (MLX).
This table maps the amount of unified memory detected at runtime to the best
Speech-to-Text (STT) and summarization models. It is intentionally kept in its
own module so the model line-up can be tuned without touching the core logic.
Each tier declares a ``min_ram_gb`` threshold. At runtime the highest tier whose
threshold is *less than or equal to* the detected memory is selected (see
``audio_summary.device.select_models``).
STT engines:
* ``"voxtral"`` -> Mistral Voxtral (non-realtime batch model) via ``mlx-audio``.
Honors the requested language and transcribes whole files in one pass.
* ``"whisper"`` -> ``mlx-whisper`` (lightweight fallback for low-RAM Macs).
"""
from dataclasses import dataclass
@dataclass(frozen=True)
class STTModel:
"""A Speech-to-Text model definition."""
engine: str # "voxtral" or "whisper"
repo: str # Hugging Face repository id
@dataclass(frozen=True)
class Tier:
"""A hardware tier: minimum RAM and the models it unlocks."""
min_ram_gb: int
stt: STTModel
summarization_repo: str
# Ordered from the smallest to the largest hardware tier.
TIERS: list[Tier] = [
Tier(
min_ram_gb=8,
stt=STTModel("whisper", "mlx-community/whisper-large-v3-turbo"),
summarization_repo="mlx-community/Qwen3-4B-4bit",
),
Tier(
min_ram_gb=16,
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-3B-2507-bf16"),
summarization_repo="mlx-community/Qwen3-8B-4bit",
),
Tier(
min_ram_gb=24,
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-3B-2507-bf16"),
summarization_repo="mlx-community/Qwen3-8B-8bit",
),
Tier(
min_ram_gb=32,
stt=STTModel("voxtral", "mlx-community/Voxtral-Mini-3B-2507-bf16"),
summarization_repo="mlx-community/Qwen3-30B-A3B-4bit",
),
]

32
src/audio_summary/models/device.py Normal file
View File

@@ -0,0 +1,32 @@
"""Hardware detection and automatic model selection.
Detects the amount of unified memory available on the Apple Silicon machine and
picks the most capable model tier that fits (see ``config.TIERS``).
"""
import psutil
from .config import TIERS, Tier
def detect_ram_gb() -> float:
"""Return the total unified memory in gibibytes."""
return psutil.virtual_memory().total / (1024**3)
def select_tier(ram_gb: float) -> Tier:
"""Select the highest tier whose RAM requirement fits ``ram_gb``.
Falls back to the smallest tier if the machine has less memory than any
declared threshold (the models simply run slower / may swap).
"""
eligible = [tier for tier in TIERS if tier.min_ram_gb <= ram_gb]
if eligible:
return max(eligible, key=lambda tier: tier.min_ram_gb)
return min(TIERS, key=lambda tier: tier.min_ram_gb)
def select_models() -> tuple[float, Tier]:
"""Detect memory and return ``(ram_gb, selected_tier)``."""
ram_gb = detect_ram_gb()
return ram_gb, select_tier(ram_gb)

11
src/audio_summary/pipeline/__init__.py Normal file
View File

@@ -0,0 +1,11 @@
"""The download -> transcribe -> summarize processing pipeline."""
from .download import download_from_youtube
from .summarization import summarize_text
from .transcription import transcribe_file
__all__ = [
"download_from_youtube",
"summarize_text",
"transcribe_file",
]

27
src/audio_summary/pipeline/download.py Normal file
View File

@@ -0,0 +1,27 @@
"""YouTube audio download helper (yt-dlp + ffmpeg)."""
from pathlib import Path
import yt_dlp
def download_from_youtube(url: str, path: str) -> Path:
"""Download the best audio track from ``url`` as an mp3 into ``path``.
Returns the path to the resulting ``to_transcribe.mp3`` file.
"""
ydl_opts = {
"format": "bestaudio/best",
"outtmpl": str(Path(path) / "to_transcribe.%(ext)s"),
"postprocessors": [
{
"key": "FFmpegExtractAudio",
"preferredcodec": "mp3",
"preferredquality": "192",
}
],
}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
ydl.download([url])
return Path(path) / "to_transcribe.mp3"

61
src/audio_summary/pipeline/summarization.py Normal file
View File

@@ -0,0 +1,61 @@
"""Text summarization with Qwen3 on Apple Silicon via mlx-lm."""
from mlx_lm import generate, load
SYSTEM_PROMPT = (
"You are an expert technical analyst who writes clear, faithful summaries of "
"transcribed audio. Rules you always follow:\n"
"- Write the summary in the SAME language as the transcript.\n"
"- Only use information present in the transcript; never invent facts, names, "
"or numbers. If something is unclear or cut off, say so rather than guessing.\n"
"- Preserve technical terms, product names, and acronyms exactly as spoken.\n"
"- The text is raw speech-to-text: ignore filler, repetitions, and transcription "
"noise, and focus on the substance."
)
USER_PROMPT_TEMPLATE = """Summarize the transcript below as a Markdown document with this exact structure:
# <a short, descriptive title capturing the main topic>
**TL;DR:** <2-3 sentence overview a busy reader could read alone>
## Key points
- <the most important ideas, one per bullet — aim for 5-8 bullets>
## Details that matter
- <specific facts, examples, definitions, numbers, or steps worth keeping>
## Takeaways
- <conclusions, recommendations, or open questions raised in the discussion>
Keep it concise and skimmable. Omit any section that the transcript genuinely does not support.
Transcript:
{text}"""
def summarize_text(text: str, model_repo: str, max_tokens: int = 2048) -> str:
"""Summarize ``text`` using the Qwen3 model at ``model_repo``."""
print(f"Loading summarization model: {model_repo}")
model, tokenizer = load(model_repo)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": USER_PROMPT_TEMPLATE.format(text=text)},
]
# Qwen3 supports a "thinking" mode; disable it for clean, direct summaries.
prompt = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
enable_thinking=False,
)
summary = generate(
model,
tokenizer,
prompt=prompt,
max_tokens=max_tokens,
verbose=False,
)
return summary.strip()

118
src/audio_summary/pipeline/transcription.py Normal file
View File

@@ -0,0 +1,118 @@
"""Speech-to-Text on Apple Silicon.
Two engines are supported, selected through ``config.STTModel.engine``:
* ``voxtral`` -> Mistral Voxtral (non-realtime batch model) via ``mlx-audio``.
Outperforms Whisper large-v3 on most benchmarks. The *realtime* Voxtral
variant is deliberately not used here: it ignores the requested language and
truncates when run one-shot on a full file.
* ``whisper`` -> ``mlx-whisper`` fallback for memory-constrained Macs.
"""
import os
import subprocess
import tempfile
from ..models import STTModel
# Upper bound on generated tokens for a full-file transcription. The non-realtime
# Voxtral ``generate`` defaults to only 128 tokens, which would truncate any real
# recording, so we raise it well above the length of a long-form transcript.
_VOXTRAL_MAX_TOKENS = 32768
# Sample rate Voxtral expects; we resample to it during the WAV conversion below.
_VOXTRAL_SAMPLE_RATE = 16000
def _to_wav_16k_mono(src: str) -> str:
"""Transcode ``src`` to a temporary 16 kHz mono WAV via ffmpeg.
Voxtral's processor loads audio through soundfile/libsndfile, which cannot
decode AAC/M4A (and several other compressed formats). ffmpeg (a project
prerequisite) handles those, so we normalize the input first. The caller is
responsible for deleting the returned path.
"""
fd, wav_path = tempfile.mkstemp(suffix=".wav")
os.close(fd)
subprocess.run(
[
"ffmpeg",
"-y",
"-i",
src,
"-ac",
"1",
"-ar",
str(_VOXTRAL_SAMPLE_RATE),
wav_path,
],
check=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
return wav_path
def _transcribe_voxtral(file_path: str, model_repo: str, language: str | None) -> str:
"""Transcribe using Mistral Voxtral through mlx-audio."""
from mlx_audio.stt.utils import load
model = load(model_repo)
# The non-realtime Voxtral model transcribes the whole file in one pass and
# honors ``language``. ``generate`` returns an object with a ``.text``
# attribute. ``language`` is optional (auto-detected when None).
kwargs: dict = {"max_tokens": _VOXTRAL_MAX_TOKENS}
if language:
kwargs["language"] = language
# Hand Voxtral a soundfile-readable WAV regardless of the source container.
wav_path = _to_wav_16k_mono(file_path)
try:
result = model.generate(wav_path, **kwargs)
finally:
os.remove(wav_path)
return result.text.strip()
def _transcribe_whisper(file_path: str, model_repo: str, language: str | None) -> str:
"""Transcribe using the mlx-whisper fallback engine."""
import mlx_whisper
result = mlx_whisper.transcribe(
file_path,
path_or_hf_repo=model_repo,
language=language, # None triggers auto-detection
)
return result["text"].strip()
def transcribe_file(
file_path: str,
output_file: str,
stt_model: STTModel,
language: str | None = None,
) -> str:
"""Transcribe ``file_path`` and persist the transcript to ``output_file``.
``language`` may be a language code (e.g. ``"en"``, ``"fr"``) or ``None`` for
automatic detection. Returns the full transcript text.
"""
print(f"Transcribing with {stt_model.engine} model: {stt_model.repo}")
if language:
print(f"Transcription language: {language}")
else:
print("Using automatic language detection")
print("Starting transcription (this may take a while for longer files)...")
if stt_model.engine == "voxtral":
text = _transcribe_voxtral(file_path, stt_model.repo, language)
elif stt_model.engine == "whisper":
text = _transcribe_whisper(file_path, stt_model.repo, language)
else:
raise ValueError(f"Unknown STT engine: {stt_model.engine!r}")
with open(output_file, "w") as tmp_file:
tmp_file.write(text)
print(f"Transcription saved to file: {output_file}")
return text

15
src/audio_summary/utils/__init__.py Normal file
View File

@@ -0,0 +1,15 @@
"""Cross-cutting helpers (I/O, formatting, CLI argument resolution)."""
from .helpers import (
ensure_directory,
print_model_banner,
resolve_language,
write_summary,
)
__all__ = [
"ensure_directory",
"print_model_banner",
"resolve_language",
"write_summary",
]

49
src/audio_summary/utils/helpers.py Normal file
View File

@@ -0,0 +1,49 @@
"""Cross-cutting helpers: CLI argument resolution, I/O, and console output.
These keep ``cli.py`` focused on orchestration and the ``pipeline`` modules
focused on the actual ML work.
"""
from pathlib import Path
from ..models import Tier
DEFAULT_LANGUAGE = "en" # Use "auto" on the CLI for automatic detection.
def resolve_language(arg_language: str | None) -> str | None:
"""Map the CLI ``--language`` value to a code or ``None`` (auto-detect)."""
language = arg_language if arg_language else DEFAULT_LANGUAGE
if language and language.lower() == "auto":
return None
return language
def ensure_directory(path: str | Path) -> Path:
"""Create ``path`` (and parents) if missing and return it as a ``Path``."""
directory = Path(path)
if not directory.exists():
directory.mkdir(parents=True)
print(f"Created directory: {directory}")
return directory
def print_model_banner(ram_gb: float, tier: Tier) -> None:
"""Print the detected hardware and the models selected for it."""
print("=" * 60)
print(f"Detected unified memory : {ram_gb:.1f} GB")
print(f"Selected hardware tier : >= {tier.min_ram_gb} GB")
print(f"STT model : {tier.stt.repo} ({tier.stt.engine})")
print(f"Summarization model : {tier.summarization_repo}")
print("=" * 60)
def write_summary(output_path: str | Path, summary: str) -> None:
"""Write ``summary`` to ``output_path``.
The summarization prompt already makes the model emit its own ``#`` title and
Markdown structure, so the text is written verbatim.
"""
with open(output_path, "w") as md_file:
md_file.write(summary)
print(f"Summary written to {output_path}")

167
src/summary.py
View File

@@ -1,167 +0,0 @@
import ollama
import argparse
from pathlib import Path
from transformers import pipeline
import yt_dlp
import torch
OLLAMA_MODEL = "llama3"
WHISPER_MODEL = "openai/whisper-large-v2"
WHISPER_LANGUAGE = "en" # Set to desired language or None for auto-detection
# Function to download a video from YouTube using yt-dlp
def download_from_youtube(url: str, path: str):
ydl_opts = {
'format': 'bestaudio/best',
'outtmpl': str(Path(path) / 'to_transcribe.%(ext)s'),
'postprocessors': [{
'key': 'FFmpegExtractAudio',
'preferredcodec': 'mp3',
'preferredquality': '192',
}],
}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
ydl.download([url])
# Function to get the best available device
def get_device():
if torch.backends.mps.is_available():
return "mps"
elif torch.cuda.is_available():
return "cuda"
else:
return "cpu"
# Function to transcribe an audio file using the transformers pipeline
def transcribe_file(file_path: str, output_file: str, language: str = None) -> str:
# Get the best available device
device = get_device()
print(f"Using device: {device} for transcription")
# Load the pipeline model for automatic speech recognition
transcriber = pipeline(
"automatic-speech-recognition",
model=WHISPER_MODEL,
device=device,
chunk_length_s=30, # Process in 30-second chunks
return_timestamps=True # Enable timestamp generation for longer audio
)
# Transcribe the audio file
# For CPU, we might want to use a smaller model or chunk the audio if memory is an issue
if device == "cpu":
print("Warning: Using CPU for transcription. This may be slow.")
# Set up generation keyword arguments including language
generate_kwargs = {}
if language and language.lower() != "auto":
generate_kwargs["language"] = language
print(f"Transcribing in language: {language}")
else:
print("Using automatic language detection")
# Transcribe the audio file
print("Starting transcription (this may take a while for longer files)...")
transcribe = transcriber(file_path, generate_kwargs=generate_kwargs)
# Extract the full text from the chunked transcription
if isinstance(transcribe, dict) and "text" in transcribe:
# Simple case - just one chunk
full_text = transcribe["text"]
elif isinstance(transcribe, dict) and "chunks" in transcribe:
# Multiple chunks with timestamps
full_text = " ".join([chunk["text"] for chunk in transcribe["chunks"]])
else:
# Fallback for other return formats
full_text = transcribe["text"] if "text" in transcribe else str(transcribe)
# Save the transcribed text to the specified temporary file
with open(output_file, 'w') as tmp_file:
tmp_file.write(full_text)
print(f"Transcription saved to file: {output_file}")
# Return the transcribed text
return full_text
# Function to summarize a text using the Ollama model
def summarize_text(text: str, output_path: str) -> str:
# Define the system prompt for the Ollama model
system_prompt = "I would like for you to assume the role of a Technical Expert"
# Define the user prompt for the Ollama model
user_prompt = f"""Generate a concise summary of the text below.
Text : {text}
Add a title to the summary.
Make sure your summary has useful and true information about the main points of the topic.
Begin with a short introduction explaining the topic. If you can, use bullet points to list important details,
and finish your summary with a concluding sentence."""
# Use the Ollama model to generate a summary
response = ollama.chat(
model=OLLAMA_MODEL,
messages=[
{
"role": "system",
"content": system_prompt,
},
{
"role": "user",
"content": user_prompt,
},
],
)
# Print the generated summary
return response["message"]["content"]
def main():
# Parse command line arguments
parser = argparse.ArgumentParser(description="Download, transcribe, and summarize audio or video files.")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--from-youtube", type=str, help="YouTube URL to download.")
group.add_argument("--from-local", type=str, help="Path to the local audio file.")
parser.add_argument("--output", type=str, default="./summary.md", help="Output markdown file path.")
parser.add_argument("--transcript-only", action='store_true', help="Only transcribe the file, do not summarize.")
parser.add_argument("--language", type=str, help="Language code for transcription (e.g., 'en', 'fr', 'es', or 'auto' for detection)")
args = parser.parse_args()
# Determine language setting
language = args.language if args.language else WHISPER_LANGUAGE
if language and language.lower() == "auto":
language = None # None triggers automatic language detection
# Set up data directory
data_directory = Path("tmp")
# Check if the directory exists, if not, create it
if not data_directory.exists():
data_directory.mkdir(parents=True)
print(f"Created directory: {data_directory}")
if args.from_youtube:
# Download from YouTube
print(f"Downloading YouTube video from {args.from_youtube}")
download_from_youtube(args.from_youtube, str(data_directory))
file_path = data_directory / "to_transcribe.mp3"
elif args.from_local:
# Use local file
file_path = Path(args.from_local)
print(f"Transcribing file: {file_path}")
# Transcribe the audio file
transcript = transcribe_file(str(file_path), data_directory / "transcript.txt", language)
if args.transcript_only:
print("Transcription complete. Skipping summary generation.")
return
print("Generating summary...")
# Generate summary
summary = summarize_text(transcript, "./")
# Write summary to a markdown file
with open(args.output, "w") as md_file:
md_file.write("# Summary\n\n")
md_file.write(summary)
print(f"Summary written to {args.output}")
if __name__ == "__main__":
main()

1251
uv.lock generated
View File

File diff suppressed because it is too large Load Diff