GEWU-LAB · UNIFIED AUDIO EDITING

SteerSound

Steer Unified Audio Editing Toward Reliable Source Selection

Edit the sound you mean. Keep the rest.

Youquan Fu1,2,* Dongnuan Cai1 Yu Pan1 Henghui Du2 Yujia Zhang2 Ping Luo3 Quande Liu4 Yake Wei1,† Xi Chen2,† Hongyuan Zhang3,† Di Hu1

1 Renmin University of China2 Tencent3 The University of Hong Kong4 The Chinese University of Hong Kong

* Work done during an internship at Tencent. · † Corresponding author.

HEAR THE DIFFERENCE
6generation & editing tasks
70.3%↓relative selective error reduction28.48% → 8.45% · Selective setting
93.67%human-rated editing success600 sampled examples · Complex setting

01 / LISTEN

One model. Six ways to shape sound.

Choose an operation, read the instruction, and compare the original recording with SteerSound’s output.

Remove the requested source while retaining the surrounding sound.

Loading audio examples…

02 / THE RESEARCH QUESTION

A good edit starts with the right source.

A successful edit changes the intended source and preserves the rest. However, existing editors can miss the target, edit the wrong source, or change both. We find that source-selection errors emerge early in denoising and often persist through later steps. Short corrective prefixes can rescue failed edits, while corruptive prefixes can induce failures, showing that early trajectories shape the final source choice.

These findings motivate STEER, a two-stage post-training framework. STEER-Emphasis strengthens training on task-critical early states to reduce source-selection errors. STEER-Recovery learns corrective denoising from the model’s own early rollout states to recover from remaining deviations. Applying both stages to our pretrained six-task editor yields SteerSound, combining broad editing capability with more reliable source selection.

Read the abstract

Unified audio editors support diverse operations, yet often struggle to modify the intended source while preserving non-target content. We show that source selection in audio editing is shaped by early denoising states, while incorrect early selections can hardly be corrected during subsequent denoising.

Motivated by these findings, we propose Source-selection Training via Early-trajectory Emphasis and Recovery (STEER), a two-stage post-training algorithm for reliable selective audio editing. STEER-Emphasis prioritizes training on task-critical early states to reduce source-selection errors, while STEER-Recovery learns corrective denoising from self-generated rollout states to recover from early deviations.

To systematically assess source-selection reliability, we introduce the Selective Audio Editing Benchmark, enabling large-scale automatic evaluation of target manipulation and non-target preservation. Applying STEER to our pretrained multitask base editor gives SteerSound, a unified audio editor with leading selective editing performance among open-source audio editors. Experiments on our base editor, UNISON, and SAM Audio demonstrate that STEER consistently improves both selective editing capability and overall editing performance. Early-trajectory emphasis further brings substantial improvements in transcript-conditioned target speaker removal and separation, highlighting the broader applicability of this training principle.

Observation: source-selection errors emerge early

We decode estimated clean audio from intermediate denoising states to inspect which source the model is tending to edit. In representative failed edits from UNISON and SAM Audio, incorrect source-selection tendencies are already perceptible within the first 1–5 of 50 denoising steps.

These observations locate when an error becomes visible. The interventions below test how changing early states affects the final edit.

Explore the evidence: corrective and corruptive prefix interventions

Do early states influence which source is ultimately edited? We test this with two interventions on UNISON and SAM Audio.

Corrective: rescue a failed edit

Replace the first k predicted denoising velocities with ones derived from the correct editing target.

Corruptive: disrupt a successful edit

Replace the first k predicted denoising velocities with ones derived from a target encoding incorrect source selection.

After either intervention, the model resumes its own denoising under the original instruction. We evaluate 100 validation examples per task.

Short correct prefixes can rescue failed edits, while short incorrect prefixes can induce failures. Extending the interventions into later steps adds limited effects. These findings motivate STEER-Emphasis to prevent early selection errors and STEER-Recovery to correct remaining deviations.

03 / THE METHOD

Reliable source selection with STEER.

STEER-Emphasis reduces early source-selection errors. STEER-Recovery learns to recover from remaining trajectory deviations.

Pretrained base editor01 · STEER-Emphasis02 · STEER-RecoverySteerSound

01 · STEER-Emphasis

Emphasize task‑critical early states.

STEER-Emphasis focuses training on the high-noise states critical to source selection. Corrective prefix interventions identify the early range for each task.

Training details

For selective tasks, noise levels are sampled uniformly within these ranges, using the standard Flow Matching objective.

Generate · Add
Standard logit‑normal sampling
Remove
First 2 / 50 denoising steps
Replace · Pitch · Volume
First 15 / 50 denoising steps

02 · STEER-Recovery

Recover from early trajectory deviations.

STEER-Recovery trains on the model’s own early rollout states, learning corrective denoising toward the desired clean edit.

Training details

Rollout depth increases progressively, and the best validation checkpoint is retained. Generate and Add use standard Flow Matching.

Corrective target = z(k) − z0σk
Base editor architecture 673M trainable backbone · 1.348B total parameters

A unified, dual-branch audio editor.

A SeedEdit-inspired DiT lets reference and generated audio exchange information through multimodal self-attention with shared attention parameters. It combines a 673.35M-parameter trainable backbone, T5-Base, an audio VAE, and a HiFi-GAN vocoder.

The same editor supports generation, addition, removal, replacement, pitch changes, and volume changes.

04 / DATA & EVALUATION

Shared audio sources.
Two data settings.

Download figure

SHARED AUDIO SOURCE POOLS

~1M audio entries

635kClass-labeled audio
380kFine-captioned audio
Construct Complex data
Construct Selective data

↓ Construct two data settings

Complex setting

Editing in richer sound scenes

Dataset scale

Training
~600kNominal · online
Validation
6k
Test
6k

Source composition

Replace · Remove · Pitch · Volume

May overlap1–4 sources
1 source
16.8%
2 sources
62.1%
3 sources
15.5%
4 sources
5.5%

Reference audio

“Increase the pitch of dog barking.”

Selective setting

Isolating source-selection behavior

Dataset scale

Training
61kFixed
Validation
6k
Test
6k

Source composition

Replace · Remove · Pitch · Volume

No temporal overlap2 sources
2 sources
100%

Reference audio

“Increase the pitch of dog barking.”

What can go wrong in a selective edit?

Success

Target edited.
Other sources preserved.

Target missed

The requested change
does not happen.

Wrong-target edit

A different source is edited
while the target stays.

Unintended non-target edit

The target and at least one
non-target source are edited.

Explore the released dataset

05 / RESULTS

Unified capability.
Reliable source selection.

We compare SteerSound with existing audio editors, then measure the gains from STEER across models and data settings.

01

STEERSOUND · OVERALL CAPABILITY

One model for six audio tasks.

Primary task metrics on the Complex setting’s test set.

1.348Btotal parameters
02

STEER · EFFECTIVENESS

A consistent lift from STEER.

Follow each backbone from its pretrained baseline through STEER-Emphasis and STEER-Recovery.

ROBUSTNESS TO EARLY CORRUPTION

5 → 9 → 15 steps

The prefix length needed to induce a 50% failure rate increases from the base editor, to STEER-Emphasis, to full STEER.

HUMAN EVALUATION · COMPLEX SETTING

88.50% → 93.67%

Overall success improves while selective error counts fall from 40 to 17, across 100 sampled examples per task.

BEYOND CATEGORY-LEVEL EDITING

STEER-Emphasis also helps distinguish speakers by speech content. In separately adapted models, human-rated success rises from 58% to 92% for target speaker removal and from 54% to 87% for target speaker separation.

06 / EXPLORE & BUILD

Take SteerSound further.

Try an edit, read the paper, or explore the code, model weights, and data.

Paper figure