Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release MuScriptor to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres.
This article explains how MuScriptor works, what the benchmarks show, and how to run it.
What is MuScriptor?
At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme.
The release ships three weight variants on Hugging Face. Their sizes are small (103M), medium (307M, default), and large (1.4B). The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted.
How the Three-Stage Pipeline Works
MuScriptor’s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last.
- Pre-training uses D<sub>Synth</sub>, roughly 1.45M MIDI files. An on-the-fly pipeline synthesizes them during training. Augmentations include pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations.
- Fine-tuning uses D<sub>Real</sub>, an internal set of 170,000 recordings. Together they total more than 11,000 hours with aligned note annotations. Most alignments come from audio-symbolic synchronization using interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a maximum time-dilation factor.
- Reinforcement learning post-training uses D<sub>RL</sub>, 300 manually verified tracks. The team applies a GRPO-like method combining REINFORCE with group-relative advantage normalization. The reward sums three F-scores: onset, frame, and offset. As a result, the model learns to favor cleaner transcriptions.
Performance
For evaluation, the research team use D<sub>Test</sub>, 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the mir_eval library. Among them, Multi F1 is strictest, since it also requires the correct instrument.
The table below traces each training stage against the YourMT3+ baseline, using the large (~1.3B) model.
| Model (D<sub>Test</sub>) | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 |
|---|---|---|---|---|---|
| YourMT3+ (baseline) | 32.5 | 45.5 | 17.8 | 41.4 | 21.9 |
| MuScriptor · D<sub>Synth</sub> | 34.5 | 48.9 | 16.1 | 21.0 | 16.2 |
| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> | 54.4 | 69.3 | 42.3 | 43.3 | 41.6 |
| MuScriptor · D<sub>Synth</sub> + D<sub>Real</sub> + D<sub>RL</sub> | 60.4 | 73.3 | 49.0 | 50.2 | 48.2 |
Clearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding D<sub>Real</sub> then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing.
Cross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals.
Getting Started
Installation takes one command, and inference streams note events directly.
# pip install muscriptor (or: uv add muscriptor)
from pathlib import Path
from muscriptor import TranscriptionModel
# Downloads the default "medium" variant (also accepts "small" / "large")
model = TranscriptionModel.load_model()
# Stream note events; optionally condition on known instruments
for event in model.transcribe("audio.wav", instruments=["acoustic_piano", "drums"]):
print(event) # NoteStartEvent / NoteEndEvent / ProgressEvent
# Or write a MIDI file directly
Path("out.mid").write_bytes(model.transcribe_to_midi("audio.wav"))For the released models, keep cfg_coef at 1, since they are already RL post-trained. Additionally, uvx muscriptor serve launches a browser web UI with a live piano roll.
Use Cases with Examples
Because the output is standard MIDI, many workflows open up:
- Producers can extract a MIDI bassline from a mix, then re-voice it in a DAW.
- Musicologists can convert historical recordings into editable scores for analysis.
- MIR researchers can feed transcriptions into chord or key recognition systems.
- Educators can build practice tools showing a live piano roll during playback.
- Developers can transcribe only drums by passing instrument conditioning.
Strengths and Weaknesses
Strengths:
- Trained on 170k real recordings spanning classical to heavy metal.
- Open weights plus MIT-licensed inference code, in three size variants.
- Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D<sub>Test</sub>.
- Instrument conditioning customizes output and stabilizes cross-segment predictions.
- A streaming API emits note events and MIDI, alongside a browser web UI.
Weaknesses:
- Weights are CC BY-NC 4.0, so commercial deployment is restricted.
- The tokenizer drops velocity and cannot represent overlapping same-pitch, same-instrument notes.
- Onset and offset accuracy stay lower on chorals and similar styles.
- The large model wants a GPU for practical speed.
- The 5-second segment size limits long-range context and inference speed.
Check out the Paper, GitHub Repo and Model Weights. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
