GEWU-LAB · UNIFIED AUDIO EDITING

SteerSound

STEER Unified Audio Editing
Toward Reliable Source Selection

Edit the sound you mean. Keep the rest.

Youquan Fu1,2,*Dongnuan Cai1Yu Pan1Henghui Du2Yujia Zhang2Eric Liu2Xi Chen2Ping Luo3Yake Wei1,†Hongyuan Zhang3,†Di Hu1,†

1 Renmin University of China2 Tencent3 The University of Hong Kong

* Work done during an internship at Tencent · † Corresponding authors

Try SteerSound Code Weights Dataset Paper Coming soon
HEAR THE DIFFERENCE

“Remove the sound of background music.”

Explore all 12 examples
6generation & editing tasks
70.3%↓relative selective error reduction28.48% → 8.45% · Selective setting
93.67%human-rated editing success600 sampled examples · Complex setting

01 / LISTEN

One model. Six ways to shape sound.

Choose an operation, read the instruction, and compare the original recording with SteerSound’s output.

Remove the requested source while retaining the surrounding sound.

Loading audio examples…

02 / THE RESEARCH QUESTION

A good edit starts with the right source.

A successful edit changes the intended source and preserves the rest. Existing editors may miss the target, edit the wrong source, or change both. Our interventions show that early denoising states shape the final source choice: correcting an early trajectory can rescue a failed edit, while corrupting it can induce failure. This motivates STEER: Emphasis focuses training on critical early states to reduce selection errors, and Recovery learns to correct deviations from the model’s own trajectories.

Explore the evidence: corrective and corruptive prefix interventions

For failed edits, correct early prefixes can rescue the outcome. For successful edits, deliberately corrupted prefixes can induce failures. The model then resumes its own denoising under the original instruction. Interventions use 100 validation examples per task; the critical ranges are identified on validation data.

Read the abstract

Unified audio editors support diverse operations, yet often struggle to modify the intended source while preserving non-target content. To strengthen this selective editing capability, we propose Source-selection Training via Early-trajectory Emphasis and Recovery (STEER), a two-stage post-training framework, and use it to develop SteerSound, a unified audio editing model supporting six common tasks.

Intermediate-state observation and bidirectional prefix interventions show that early denoising states causally influence final source selection, while incorrect early decisions often persist through subsequent denoising. Guided by these findings, STEER-Emphasis emphasizes training on task-relevant early-trajectory states to reduce source-selection errors, while STEER-Recovery learns corrective denoising from self-generated rollout states to recover from early-trajectory deviations.

We further construct audio editing data in two complementary settings. The Selective setting enables large-scale automatic evaluation of target manipulation and non-target preservation, while the Complex setting contains richer multi-source scenes. Experiments in both settings show that STEER consistently improves selective editing reliability, reducing source-selection errors while improving overall editing performance. Further evaluations across different models and other selective-control tasks also support the broad generalizability of STEER and its underlying early-trajectory training rationale. SteerSound achieves advanced performance among open-source unified editors.

03 / THE METHOD

STEER the early trajectory.

Source-selection Training via Early-trajectory Emphasis and Recovery.

Pretrained base editor01 · Emphasis02 · RecoverySteerSound
01

STEER-EMPHASIS

Train where source
selection takes shape.

Standard logit-normal sampling emphasizes intermediate noise levels. STEER-Emphasis instead prioritizes the high-noise early states that matter for selective editing, while retaining the Flow Matching objective.

Remove
First 2 / 50 denoising steps
Replace · Pitch · Volume
First 15 / 50 denoising steps
Generate · Add
Standard logit-normal sampling
02

STEER-RECOVERY

Learn to recover
from early deviations.

The model rolls out its own early denoising states, then learns corrective velocities toward the desired clean edit. This exposes training to deviations the model actually encounters during generation.

The recovery depth increases progressively, retaining the best validation checkpoint. Generation and addition continue to use standard Flow Matching.

Recovery target = z(k) − z0σk
The editor beneath STEER 673M trainable backbone · 1.348B total parameters

A unified, dual-branch audio editor.

A SeedEdit-inspired DiT lets reference and generated audio exchange information through multimodal self-attention with shared attention parameters. It combines a 673.35M-parameter trainable backbone, T5-Base, an audio VAE, and a HiFi-GAN vocoder.

The same editor supports generation, addition, removal, replacement, pitch changes up to ±4 semitones, and volume changes up to ±5 dB.

04 / DATA & EVALUATION

Shared audio sources.
Two data settings.

Download figure

SHARED AUDIO SOURCE POOLS

~1M audio entries

635kClass-labeled audio
380kFine-captioned audio
Construct Complex data
Construct Selective data

↓ Construct two data settings

Complex setting

Editing in richer sound scenes

Dataset scale

Training
~600kNominal · online
Validation
6k
Test
6k

Source composition

Replace · Remove · Pitch · Volume

May overlap1–4 sources
1 source
16.8%
2 sources
62.1%
3 sources
15.5%
4 sources
5.5%

Reference audio

“Increase the pitch of dog barking.”

Selective setting

Isolating source-selection behavior

Dataset scale

Training
61kFixed
Validation
6k
Test
6k

Source composition

Replace · Remove · Pitch · Volume

No temporal overlap2 sources
2 sources
100%

Reference audio

“Increase the pitch of dog barking.”

What can go wrong in a selective edit?

Success

Target edited.
Other sources preserved.

Target missed

The requested change
does not happen.

Wrong-target edit

A different source is edited
while the target stays.

Unintended non-target edit

The target and at least one
non-target source are edited.

Explore the released dataset

05 / RESULTS

Unified capability.
Reliable source selection.

How does SteerSound compare with existing editors? And how much does STEER improve a pretrained model? We evaluate both questions.

01

STEERSOUND · OVERALL CAPABILITY

One model for six audio tasks.

Compare SteerSound with existing generation and editing methods on the test set of the Complex setting.

1.348Btotal parameters
02

STEER · EFFECTIVENESS

A consistent lift from STEER.

Follow each backbone from its pretrained baseline through Emphasis and Recovery. Choose an evaluation setting below.

ROBUSTNESS TO EARLY CORRUPTION

5 → 9 → 15 steps

The prefix length needed to induce a 50% failure rate increases from the base editor, to Emphasis, to full STEER.

HUMAN EVALUATION · COMPLEX SETTING

88.50% → 93.67%

Overall success improves while selective error counts fall from 40 to 17, across 100 sampled examples per task.

BEYOND CATEGORY-LEVEL EDITING

Early-trajectory emphasis also helps distinguish speakers by speech content. In separately adapted models, human-rated success rises from 58% to 92% for target speaker removal and from 54% to 87% for target speaker separation.

06 / EXPLORE & BUILD

Take SteerSound further.

Try an edit, inspect the implementation, or explore the data.

▷Online inferenceEdit your own audio ↗CodeTraining & inference ↗Model weightsHugging Face ↗DatasetData · both settings ↗

Paper figure