Awesome Alignment of Diffusion Models
The collection of awesome papers on the alignment of diffusion models.
If you are interested in the alignment of diffusion models, please refer to our survey paper "Alignment of Diffusion Models: Fundamentals, Challenges, and Future", which is the first survey on this topic to our knowledge.
We hope to enjoy the adventure of exploring alignment and diffusion models with more researchers. Corrections and suggestions are welcomed.
Alignment Techniques of Diffusion Models
RLHF
- ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023, [pdf]
- DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models, NeurIPS 2023, [pdf]
- Aligning Text-to-Image Models using Human Feedback. arXiv 2023, [pdf]
- Aligning Text-to-Image Diffusion Models with Reward Backpropagation. arXiv 2023, [pdf]
- Directly Fine-Tuning Diffusion Models on Differentiable Rewards. ICLR 2024, [pdf]
- PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models. CVPR 2024, [pdf]
- Feedback Efficient Online Fine-Tuning of Diffusion Models. ICML 2024, [pdf]
- Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control. arXiv 2024, [pdf]
- Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review. arXiv 2024, [pdf]
- Aligning Few-Step Diffusion Models with Dense Reward Difference Learning. arXiv 2024, [pdf]
DPO
- Diffusion Model Alignment Using Direct Preference Optimization. CVPR 2024, [pdf]
- Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model. CVPR 2024, [pdf]
- A Dense Reward View on Aligning Text-to-Image Diffusion with Preference. ICML 2024, [pdf]
- Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation. NeurIPS 2024, [pdf]
- Aligning Diffusion Models by Optimizing Human Utility. arXiv 2024, [pdf]
- Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step. arXiv 2024, [pdf]
- Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization. arXiv 2024, [pdf]
- Scalable Ranked Preference Optimization for Text-to-Image Generation. arXiv 2024, [pdf]
- Prioritize Denoising Steps on Diffusion Model Preference Alignment via Explicit Denoised Distribution Estimation. arXiv 2024, [pdf]
Training-Free Implicit Alignment
- Optimizing Prompts for Text-to-Image Generation. NeurIPS 2023, [pdf]
- RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions. CHI 2023, [pdf]
- Improving Text-to-Image Consistency via Automatic Prompt Optimization. TMLR 2024, [pdf]
- Dynamic Prompt Optimizing for Text-to-Image Generation. CVPR 2024, [pdf]
- ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization. NeurIPS 2024, [pdf]
- Towards Better Text-to-Image Generation Alignment via Attention Modulation. arXiv 2024, [pdf]
- Not All Noises Are Created Equally: Diffusion Noise Selection and Optimization. arXiv 2024, [pdf]
- Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding. arXiv 2024, [pdf]
- Inference-Time Alignment of Diffusion Models with Direct Noise Optimization. arXiv 2024, [pdf]
Alignment Beyond Text-to-Image Diffusion Models
- Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation. NeurIPS 2023, [pdf]
- AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model. ICLR 2024, [pdf]
- HIVE: Harnessing Human Feedback for Instructional Visual Editing. CVPR 2024, [pdf]
- InstructVideo: Instructing Video Diffusion Models with Human Feedback. CVPR 2024, [pdf]
- DreamReward: Text-to-3D Generation with Human Preference. arXiv 2024, [pdf]
- Alignment is Key for Applying Diffusion Models to Retrosynthesis. arXiv 2024, [pdf]
- Video Diffusion Alignment via Reward Gradients. arXiv 2024, [pdf]
- Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimization. arXiv 2024, [pdf]
- VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement. arXiv 2024, [pdf]
Benchmarks and Evaluation
- DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers. ICCV 2023, [pdf]
- Human Preference Score: Better Aligning Text-to-Image Models with Human Preference. ICCV 2023, [pdf]
- ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023, [pdf]
- Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. NeurIPS 2023, [pdf]
- LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation. NeurIPS 2023, [pdf]
- VPGen & VPEval: Visual Programming for Text-to-Image Generation and Evaluation. NeurIPS 2023, [pdf]
- Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. arXiv 2023, [pdf]
- GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. NeurIPS 2023 Datasets and Benchmarks, [pdf]
- Holistic Evaluation of Text-to-Image Models. NeurIPS 2023, [pdf]
- Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community. ICLR 2024, [pdf]
- Rich Human Feedback for Text to Image Generation. CVPR 2024, [pdf]
- Learning Multi-Dimensional Human Preference for Text-to-Image Generation. CVPR 2024, [pdf]
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better. NeurIPS 2024, [pdf]
- Measuring Style Similarity in Diffusion Models. arXiv 2024, [pdf]
Fundamentals of Human Alignment
Preference Modeling
- Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 1952, [pdf]
- Individual Choice Behavior. John Wiley 1959, [pdf]
- The Analysis of Permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics) 1975, [pdf]
- Learning-to-Rank with Partitioned Preference: Fast Estimation for the Plackett-Luce Model. AISTATS 2021, [pdf]
- Models of Human Preference for Learning Reward Functions. arXiv 2022, [pdf]
- Beyond Preferences in AI Alignment. arXiv 2024, [pdf]
RLHF
- Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022, [pdf]
- Constitutional AI: Harmlessness from AI Feedback. arXiv 2022, [pdf]
- RRHF: Rank Responses to Align Language Models with Human Feedback without Tears. NeurIPS 2023, [pdf]
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. TMLR 2024, [pdf]
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. ICML 2024, [pdf]
- Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024, [pdf]
DPO
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023, [pdf]
- Preference Ranking Optimization for Human Alignment. AAAI 2024, [pdf]
- A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024, [pdf]
- KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024, [pdf]
- LiPO: Listwise Preference Optimization through Learning-to-Rank. arXiv 2024, [pdf]
- ORPO: Monolithic Preference Optimization without Reference Model. arXiv 2024, [pdf]
Potential Challenges and Opportunities of Diffusion Alignment
- Scaling Laws for Reward Model Overoptimization. ICML 2023, [pdf]
- The Alignment Problem from a Deep Learning Perspective. ICLR 2024, [pdf]
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints. ICLR 2024, [pdf]
- Nash Learning from Human Feedback. ICML 2024, [pdf]
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint. ICML 2024, [pdf]
- Dense Reward for Free in Reinforcement Learning from Human Feedback. ICML 2024, [pdf]
- Position: A Roadmap to Pluralistic Alignment. ICML 2024, [pdf]
- Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications. ICML 2024, [pdf]
- MaxMin-RLHF: Alignment with Diverse Human Preferences. ICML 2024, [pdf]
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment. ICML 2024, [pdf]
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences. ICML 2024, [pdf]
- Generalized Preference Optimization: A Unified Approach to Offline Alignment. ICML 2024, [pdf]
- Human Alignment of Large Language Models through Online Preference Optimisation. ICML 2024, [pdf]
- Understanding the Learning Dynamics of Alignment with Human Feedback. ICML 2024, [pdf]
- Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. ICML 2024, [pdf]
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024, [pdf]
- BOND: Aligning LLMs with Best-of-N Distillation. arXiv 2024, [pdf]
- Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases, [pdf]
Citing
If you find the paper list useful for your research, you are highly welcome to cite our survey paper on this topic！
@article{liu2024alignment,
title = {Alignment of Diffusion Models: Fundamentals, Challenges, and Future},
author = {Liu, Buhua and Shao, Shitong and Li, Bao and Bai, Lichen, and Xu, Zhiqiang and Xiong, Haoyi and Kwok, James and Helal, Sumi and Xie, Zeke},
journal = {arXiv preprint arXiv 2024.07253},
year = {2024}
}
