Learnability-Guided Diffusion for Dataset Distillation

Standard distillation generates redundant samples. We partition the distilled dataset into data increments (e.g 10 images per class). With DiT, a model trained on one increment already achieves 98.0% accuracy on another—meaning each increment provides minimal new information.

Our approach generates complementary samples. By conditioning synthesis on the current model knowledge, we guide generation toward samples that complement existing data. The resulting increments achieve only 17.0% cross-increment accuracy, confirming they introduce substantial new learning signal.

Cross-validation across distilled data increments on ImageNette. Each heatmap shows accuracy when training on one increment and evaluating on another. DiT exhibits severe redundancy (avg. 94.7% off-diagonal accuracy), while our method yields only 57.65% on average—a 39.1% reduction in redundancy, confirming that our increments contain complementary information.

Problem Formulation

We partition the distilled dataset into \(K\) disjoint increments. At each stage, the model trained so far guides synthesis of the next increment by maximizing:

\[\mathcal{I}_i^* = \arg\max_{\mathcal{I}}\;\big[\mathcal{L}(\theta_{i-1}, \mathcal{I}) - \mathcal{L}(\theta^*, \mathcal{I})\big]\]

The first term targets samples hard for the current model; the second penalizes samples also hard for the reference—filtering out degenerate examples and focusing each increment on meaningful knowledge gaps.

Learnability Guidance

We integrate a per-sample learnability score directly into diffusion sampling:

\[\mathcal{S}(x, y) = \mathcal{L}(\theta_{i-1}, x, y) - \omega \cdot \mathcal{L}(\theta^*, x, y)\]

This score steers the denoising trajectory via gradient-based guidance:

\[\tilde{\epsilon}_\phi(x_t, t, y) = \epsilon_\phi(x_t, t, y) + \lambda \cdot \rho_t \cdot \nabla_{x_t} \mathcal{S}(x_t, y)\]

where \(\lambda\) controls guidance strength and \(\rho_t\) normalizes across timesteps. The result is an active learning mechanism that forms a curriculum of increasing difficulty.

Learnability Rank Selection

Guided sampling alone can still yield weakly learnable examples. We over-generate by a factor \(\kappa\), score each candidate with \(\mathcal{S}\), and retain the top-\(N_i\) per class:

\[\mathcal{I}_i = \operatorname{Top}_{N_i}\!\Big(\big\{(x_j^c,\; y_j^c,\; \mathcal{S}(x_j^c, y_j^c))\big\}\Big)\]

This two-stage approach—guidance during generation plus rank-based selection—yields high-learnability increments that complement rather than replicate existing data.

Our method achieves consistent improvements over prior state-of-the-art methods across both datasets and all architectures. On ImageNette at 100 IPC, we reach 87.2%, surpassing DiT+IGD by 2.7%. On ImageWoof at 100 IPC, we achieve 72.9% with ResNet-18.

ImageNet-1K at 50 IPC with ResNet-18. Our method achieves 60.1%, matching state-of-the-art performance while substantially outperforming the base diffusion approach (DiT: 52.9%, a 7.2% improvement).

Incremental training dynamics. (a–b) Our method produces stronger loss spikes after each new increment (avg. Δ=0.20 vs. 0.06 for DiT), confirming that added data is harder and complementary. (c) Our method achieves 85.2% vs. 79.9% (IGD) and 75.6% (DiT) at 50 IPC, with sustained accuracy gains across all stages.

Learning-dynamics visualization. DiT concentrates in easy regions (>80% with μ > 0.8). Our method distributes broadly across easy, informative, and hard regions. Jensen–Shannon divergence confirms alignment: our method achieves JS = 0.41 vs. DiT (2.04) and IGD (0.92)—5× and 2.2× improvements.

In-distribution and learning-dynamics analysis. Each point shows a sample's reference-model confidence (x-axis) and mean GT probability during training (y-axis). DiT clusters on easy, high-confidence samples. DiT+Loss (without our regularization) covers a broader range but introduces out-of-distribution samples. DiT+Ours achieves a balanced spread—capturing informative mid and hard examples while staying closely aligned with the in-distribution region.

Visual diversity in incrementally distilled datasets (Church class, 50 IPC total across 5 increments). DiT generates repetitive samples with similar architectural styles and lighting. IGD improves slightly but clusters around Gothic exteriors. Ours produces diverse samples spanning multiple architectural styles (traditional, modern, ornate), perspectives (exterior and interior views), and lighting conditions (day, night, golden hour)—reflecting our curriculum-based synthesis where early increments capture simpler structures and later increments progressively introduce complexity.