How AI music separation works, and why it is not perfect

Once a song is mixed, the instruments are added together into one waveform (two, in stereo). There is no switch that takes them apart again. AI separation works by estimating what each instrument must have sounded like — very well, but not perfectly. Knowing how helps you get the most out of it.

Why separating a mix is hard

Instruments in a band share the same range of frequencies. A bass note's harmonics overlap the guitar; a snare hit spreads over almost every frequency; a singer and a guitar can play the same note at the same time. In the mixed recording these sounds are added together, and many different combinations of instruments could have produced the same mix. Older methods used simple tricks — for example removing what is in the center of a stereo mix to remove the vocals — which also removed the bass and the kick drum, and left the vocal reverb behind.

How the AI learns to separate

A separation model is trained on songs for which the separate recordings of each part (the “stems”) are available. During training it hears the mix, makes its guess for each part, and is corrected by comparing the guess with the real stems — many thousands of times, across many songs. It learns what a voice, a bass, and a drum kit typically sound like, and uses that knowledge to pull them apart in songs it has never heard.

Because the model learned from examples, it does best on music like what it was trained on — mainly pop and rock with a typical band setup — and can be less sure with unusual instruments or sounds.

The model: Demucs (htdemucs)

Band Stem Mixer uses Demucs, a music separation model published by Meta AI researchers, in its Hybrid Transformer version (htdemucs, Rouard, Massa, and Défossez, ICASSP 2023):

• Hybrid – It looks at the sound in two ways at once: as a waveform (how the signal moves over time) and as a spectrogram (which frequencies are present at each moment). Each view is better at some sounds: sharp drum hits are clear in the waveform, held notes in the spectrogram.
• Transformer – In its middle, a Transformer (the type of neural network used in many recent AI models) relates every moment of the section to every other, and the two views to each other.
• Training data – According to the Demucs documentation, it was trained on the MUSDB HQ dataset plus 800 extra songs.
• Accuracy – The documentation reports a signal-to-distortion ratio (SDR) of 9.00 dB on the MUSDB HQ test set. SDR measures how close the separated parts are to the real stems; it is a standard way to compare models, not a measure of how good it sounds to you.

The trained model (its “weights”) is provided by Meta for scientific purposes only and is not covered by the MIT license of the Demucs code. This is why Band Stem Mixer is non-commercial, with no ads.

Why four parts, and why guitar and keys stay together

The four parts — vocals, drums, bass, and other — come from the datasets the model was trained on, which provide stems in those four groups. “Other” is everything else: guitars, keys, synths, strings, horns. The model never learned to tell those apart, so they come out together.

Demucs also has an experimental six-part version that adds guitar and piano. Its own documentation says guitar quality seems okay but piano has a lot of bleeding and artifacts, and it needs more memory; it ran out of memory in a browser during our tests, so Band Stem Mixer uses the four-part model.

Why sounds leak between parts

The model's output is its best estimate, so some sounds end up in the wrong part, or in two parts at once (called bleeding):

• Sounds that share frequencies and timing are hardest: the kick drum and the bass, a sung note doubled by a guitar.
• Reverb and effects blur instruments into each other, so the echo of a voice can stay in “Other”.
• Loud, heavily compressed or distorted mixes give the model less to go on. Lossy files (low-bitrate MP3) have already lost detail.
• Instruments the model rarely heard in training can be split between parts — a synth bass may land partly in “Bass” and partly in “Other”.

For practice — muting your part or hearing a line clearly — this is rarely a problem. For tips, see how to get better separation.

How it runs in your browser

Usually a model like this runs on a server. Band Stem Mixer runs it on your own device, so your song is never uploaded:

• The browser downloads the model (about 172 MB, converted to the ONNX format) once and keeps it.
• ONNX Runtime Web runs the model, on the graphics chip (GPU) through WebGPU when the browser supports it, otherwise on the processor (CPU).
• This model takes 7.8 seconds of sound at a time, so the song is cut into sections that overlap by a quarter, and the overlaps are cross-faded so no seams can be heard. A 4-minute song is 42 sections — the count you see on the progress bar.
• Each section needs a lot of memory — about 1 GB while the model runs. That is why computers are much faster, and why some phones cannot finish.

Once separated, the four parts are played together in sync, and the mixer only changes their volumes, so moving a fader is instant.