Chimera
Designing and Chinchilla-scaling hybrid visual diffusion transformers
Chimera is a hybrid visual diffusion backbone with a principled, compute-optimal scaling recipe for the regime where high-resolution images, long videos, and multimodal context make the quadratic cost of full attention prohibitive. It processes text, image, and video tokens in a single raster-ordered stream with no positional embeddings, and it is co-designed with a scaling methodology so architecture and scaling are developed together rather than as separate concerns.
Key ideas
- A unified, position-embedding-free stream. Text, image, and video tokens are processed in one raster-ordered sequence without positional embeddings, enabling direct cross-modal interaction in a single backbone.
- A hybrid attention design. Chimera combines Kimi Delta Attention (KDA) for long-context state tracking at O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context.
- Sparse mixture-of-experts. MoE layers expand capacity while keeping the activated compute controlled.
- HeteroP scaling. A module-wise scheme transfers hyperparameters across width and depth according to each tensor’s functional fan-in and model depth, yielding a consistently tuned family for fitting Chinchilla-style compute-optimal laws over activated model size, training tokens, and image–video data ratio.
Results
- Compute efficiency. Measured by pretraining diffusion loss, the dense backbone is 1.7× as compute-efficient as a matched full-attention Wan-2.1 2B baseline; the complete system reaches 7.3×.
- Zero-shot length extrapolation. Without length-specific fine-tuning, Chimera extrapolates from 5-second training clips to 30-second videos with only 6.5% FID degradation in the last five seconds.
- Compute-optimal recipe. Guided by the fitted laws, the team trains an 11B-parameter Chimera with 2B activated parameters; image pretraining splits compute nearly evenly between model size and tokens, while video pretraining modestly favors model size at higher budgets.
Scaling laws
Video demo
Zero-shot 30-second generation (16 fps, 481 frames) from the same prompt, compared across Wan2.1, HunyuanVideo, and Chimera. “motion” is the mean frame-to-frame pixel change — higher means more actual movement, while a value near zero is a near-frozen clip.
A ceramic bowl tips over on a smooth kitchen countertop, spilling rice grains that scatter and pile into a small mound.
A single yellow rose in full bloom is featured in a tranquil garden while the camera pans right as the petals slowly open.
Why it matters
Visual generation is following language models into a long-sequence regime, but the language recipe does not transfer directly — visual diffusion must preserve spatiotemporal locality and bidirectional cross-modal interaction. Chimera establishes a foundation for designing and scaling efficient long-context diffusion architectures whose behavior can be reasoned about ahead of expensive large-scale training.
Read the paper: Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers.