Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
1
2
2
Loading representative videos...
2
We study reward fine-tuning for large-scale diffusion and flow models through a velocity-matching perspective. Instead of estimating policy ratios along denoising trajectories, reward-based velocity matching (RVM) applies reward feedback directly to the learned velocity field. The resulting objective combines a reward-weighted regression term with an anchor term that controls model drift. This view connects several recent velocity-based objectives while keeping training simple, trajectory-independent, and practical for video generation.
The loss is evaluated from generated samples x_0^(1:G) ~ pi_old and noisy states x_t^i | x_0^i ~ N((1-t)x_0^i, t^2 I), without storing or scoring full denoising rollouts x_(0:1).
The signed reward r_rvm^i attracts v_theta(x_t^i,t) toward the sample velocity v^i = epsilon^i - x_0^i for desirable samples and repels it from undesirable directions.
A detached anchor velocity v_anc(x_t^i,t) regularizes the update and recovers related velocity-based objectives under specific choices.
RAM shapes a detached reward target. With the frozen reference as anchor and unit anchor strength, its velocity-field update matches RVM.
DiffusionNFT contrasts positive and negative EMA-anchored branches. Its objective is an EMA-anchored RVM up to constants, with signed reward weight 2r_nft - 1.
RVM writes the shared structure directly: reward weighting chooses the preferred sample direction, while the anchor controls model drift.
Text-to-video. 53 frames, 480x832, 16-step DPM-2 ODE, VBench-official suite. The headline is the short VBench suite prompt; expand "extended prompt" for the self_forcing_extended caption the model was actually conditioned on.
Image-to-video. 53 frames, 400x640, 16-step ODE, cfg=1.0, VBench-I2V (vbench_i2v_a14b). The headline is the prompt the model was conditioned on, i.e., these runs used the original short prompts. The expandable rewrite comes from a separate eval_a14b_extended run and was not used for these clips. No CFG=5 baseline was collected for this model.
If you find our work useful, please consider citing it as follows.
@article{choi2026scaling,
title={Scaling Reinforcement Learning for Diffusion Models via Velocity Matching},
author={Choi, Jaemoo and Guo, Wei and Zhu, Yuchen and Vahdat, Arash and Tao, Molei and Berner, Julius and Chen, Yongxin},
journal={arXiv preprint arXiv:2608.23664},
year={2026}
}