Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems

Researchers introduce MixtureTT, a diffusion-based system for timbre transfer in polyphonic music that directly processes mixed audio rather than separating instruments first. The approach outperforms existing separate-then-transfer pipelines by modeling dependencies across multiple stems simultaneously, reducing inference costs and eliminating source separation artifacts.

MixtureTT addresses a fundamental limitation in music processing technology: the inability to efficiently and coherently transfer timbral characteristics across multiple instruments in mixed audio. Traditional approaches separate instruments, transfer timbre individually, and recombine them—a pipeline that introduces cumulative errors and produces incoherent results. This research inverts that workflow through joint stem diffusion, processing all instruments together while maintaining their harmonic relationships.

The technical innovation centers on a shared diffusion process that models cross-stem dependencies, essentially teaching the model to understand how instrumental timbres interact within a mixture. By eliminating the separation step entirely, MixtureTT reduces inference computational costs proportionally to the number of instruments—a significant efficiency gain for professional audio workflows. The use of a diffusion transformer architecture enables flexible, per-stem timbre control while maintaining musical coherence.

For the music production and AI audio communities, this represents a meaningful advancement in generative audio models. Current music production tools rely heavily on separate-then-resynthesize approaches, creating bottlenecks for artists and producers working with polyphonic material. MixtureTT's superior performance on objective metrics and subjective listening tests suggests practical applicability in professional workflows, from music arrangement to sound design.

The research validates an important principle: handling complex interdependencies at the system level outperforms sequential independent processing. This finding likely extends beyond audio to other multimodal domains. As generative audio models mature, frameworks that preserve inherent relationships between components will become increasingly valuable, positioning joint modeling approaches as standard practice in the field.

• →MixtureTT enables direct timbre transfer from polyphonic mixtures without requiring instrument separation, eliminating cascaded processing errors.
• →The joint stem diffusion approach reduces inference costs by a factor equal to the number of stems being processed.
• →Cross-stem dependency modeling proves essential for coherent multi-instrument timbre transfer, outperforming single-instrument baseline approaches.
• →The system demonstrates superior performance on both objective metrics and subjective listening evaluations across SATB choral datasets.
• →This research suggests that handling multimodal audio relationships at the system level yields better results than sequential independent processing pipelines.