Your Data Manifold is Secretly a Reward Model Shell-LCC for Text-to-Video Generation

A cost-free reward, hidden in your data

We argue that your data manifold is secretly a reward model. By modeling the manifold of high-quality SFT video patches and pulling generated latents onto it, Shell-LCC yields dense, differentiable, nearly free reward signals — no human labels, no external reward model.

Abstract

Recent text-to-video (T2V) diffusion models rely heavily on auxiliary reward signals (e.g., via reward models or DPO) to align generated content with human aesthetics and improve realism. These signals, however, incur substantial computational overhead, require costly human annotations, and often yield limited improvement in fine-grained local details. In this paper, we argue that your data manifold is secretly a reward model. By explicitly modeling the manifold structure of high-quality Supervised Fine-Tuning (SFT) data and encouraging video latents to lie on this manifold, we derive dense, differentiable, and nearly cost-free reward signals that significantly improve video quality, particularly in mitigating low-level distortions. Our modeling builds upon Local Coordinate Coding (LCC), which captures the ‘skeleton’ of the manifold. However, directly applying LCC suffers from mean regression, pulling latents toward the geometric mean and losing high-frequency details. We therefore extend it to Shell Local Coordinate Coding (Shell-LCC), which models the manifold ‘surface’ as an isotropic shell to align with the true high-density region. Experiments demonstrate that our approach improves realism, enhances high-frequency details, reduces over-smoothing artifacts, and alleviates motion blur.

Method: from data manifold to dense reward

1. Extract dense patches. Encode SFT videos with the frozen 3D VAE and flatten the latent volume into spatio-temporal patches — a single 5-second clip already yields 230,400 patches.
2. Learn the skeleton (LCC). Approximate the patch manifold with M=4096 learnable anchors and an amortized coordinate predictor: each patch is a sparse local linear combination of nearby anchors.
3. Learn the surface (Shell-LCC). Calibrate dimensions with a learnable scale σ (diagonal Mahalanobis) and constrain the normalized residual to a unit shell, with a log-σ regularizer — the negative log-likelihood of a shell-structured Gaussian.
4. Freeze & reward. The manifold distance \(R_{dist}\) of generated patches is a dense, differentiable reward: fine-tuning the T2V model to keep latents on the shell removes blur, noise and motion artifacts while keeping legitimate high-frequency detail.

Why a shell, not a point?

Two provable failure modes of naive manifold rewards motivate the shell.

Dropping the locality constraint collapses distinguishability (−9.5%): pure reconstruction degenerates into an identity map with no geometry. Shell-LCC trades a sliver of trivial accuracy (won by exploiting high-variance dimensions) for equal sensitivity to fine-grained, low-variance detail. And because every 5-second clip contributes 230k patches, 100 videos already reach 90.1% — the manifold is extremely data-efficient, and robust to hyperparameters (91.1–91.8% across M, τ₁, τ₂ settings).

Video comparisons — Wan-T2V-1.3B

Left: baseline. Right: + Shell-LCC. Same prompt, same seed. The finetuned Wan2.1-T2V-1.3B checkpoint is released on HuggingFace.

Scales to larger T2V models

The same Shell-LCC manifold transfers across model scales without retraining: on a 4.5B T2V model it sharpens fine detail while preserving composition, and on Wan2.1-T2V-14B it lifts overall visual quality and prompt fidelity.

Results: better imaging quality, nothing traded away

On a proprietary 4.5B model and two open-source backbones, Shell-LCC lifts VBench Imaging Quality while aesthetics, semantics and temporal consistency hold steady — the failure mode of preference-based baselines.

VBench scores in % (higher is better); green/red deltas are percentage points vs. the corresponding baseline. Shell-LCC sharply improves Imaging Quality (low-level fidelity) without trading off aesthetics, semantic alignment, or temporal consistency.

Controlled deblurring

Shell-LCC progressively removes motion blur, but over-optimization re-induces mean regression and collapse — so the reward is applied with early stopping.

Low-level distortions such as motion blur are entangled with genuine high-frequency information: over-aggressive optimization discards both and re-induces mean regression. We currently mitigate this with early stopping; combining the patch-level reward with preference-based methods (DPO) and scaling to larger backbones are natural next steps.

BibTeX

@inproceedings{zhang2026shelllcc,
  title     = {Your Data Manifold is Secretly a Reward Model:
               Shell-LCC for Text-to-Video Generation},
  author    = {Zhang, Shihao and Li, Yunzhi and Yan, Yuguang and
               Zhang, Junzhe and Zhao, Wei and Wang, Bohan and Zhang, Hanwang},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}