This repo is the official PyTorch implementation for Improving Long-Text Alignment for Text-to-Image Diffusion Models (LongAlign)

by Luping Liu^1,2, Chao Du², Tianyu Pang², Zehan Wang^2,4, Chongxuan Li³, Dong Xu¹.

¹The University of Hong Kong; ²Sea AI Lab, Singapore; ³Renmin University of China; ⁴Zhejiang University

What does this work do?

To improve long-text alignment for T2I diffusion models, we propose LongAlign, which features a segment-level encoding method for processing long texts and a decomposed preference optimization method for effective alignment training. For decomposed preference optimization, we find the preference models can be decomposed into two components: a text-relevant part and a text-irrelevant part. We propose a reweighting strategy that assigns different weights to these two components, reducing overfitting and enhancing alignment.

CLIP-based preference decomposition

• (a) Schematic results for text embeddings. (b) Statistics of the projection scalar