FLARE

FLARE is a systematic recipe for converting hybrid-attention autoregressive LLMs — those that mix softmax and linear attention — into capable, serving-efficient diffusion language models under a practical post-training budget. Starting from hybrid-attention seeds, it produces a single checkpoint that preserves autoregressive quality while unlocking parallel, real-time generation. The result is a family of models (FLARE-2B/4B/9B) that pairs the algorithmic efficiency of diffusion LMs with the architectural efficiency of hybrid attention.

Highlights

• One checkpoint, two decoding regimes. The same model exposes an AR-Trust path — speculative, AR-style verified decoding whose quality is provably bounded below by the AR teacher — and a Diffusion-Trust path that emits multiple tokens per forward through parallel block denoising.
• Transfer data is the dominant factor. Once the loss and mask design are aligned, transfer-data quality and distribution match outweigh further objective tweaks, with AR-SFT serving as a low-cost proxy for data selection.
• Hardware-aware Triton kernels. Diffusion visibility on linear-attention layers is realized through recurrent-state handoff rather than dense masking, closing the MFU gap to causal AR.
• State-of-the-art efficiency. FLARE-9B matches a 100B-A5B open diffusion LLM on most shared benchmarks at roughly 1/10 of the parameters.
• Real-time throughput. FLARE-2B reaches ~2,087 tokens/s on GSM8K at 8-way concurrency on a single A100 — about 2.2× and 4.8× faster than comparable diffusion baselines.

Why it matters

FLARE shows that parallel, real-time generation need not come at the cost of capability. By converting strong hybrid-attention LLMs into diffusion models under a modest budget, it brings low-latency, high-throughput serving to models that remain competitive on reasoning, math, and code — narrowing the long-standing gap between fast decoding and frontier quality.

Read the paper: FLARE: Diffusion for Hybrid Language Models.