xiaobul/mars

Official implementation of MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis

MARS: Mixture of Auto-Regressive Models for
Fine-grained Text-to-image Synthesis

Wanggui He^1,*, Siming Fu^1,*, Mushui Liu^2,*, Xierui Wang^2,+ , Wenyi Xiao, ^2,+ Fangxun Shu^1,+,
Yi Wang² , Lei Zhang², Zhelun Yu³, Haoyuan Li² , Ziwei Huang², LeiLei Gan², Hao Jiang^1,†
1 Alibaba Group      ² Zhejiang University      ³ Fudan University
^*Equal contribution        ⁺Core contributor        ^†Corresponding author

Overview

This repository contains the official implementation of the paper "MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis".

Auto-regressive models have made significant progress in the realm of language generation, yet do not perform on par with diffusion models in the domain of image synthesis. In this work, we introduce MARS, a novel framework for T2I generation that incorporates a specially designed Semantic Vision-Language Integration Expert (SemVIE). This innovative component integrates pre-trained LLMs by independently processing linguistic and visual information—freezing the textual component while fine-tuning the visual component. This methodology preserves the NLP capabilities of LLMs while imbuing them with exceptional visual understanding. Building upon the powerful base of the pre-trained Qwen-7B, MARS stands out with its bilingual generative capabilities corresponding to both English and Chinese language prompts and the capacity for joint image and text generation. The flexibility of this framework lends itself to migration towards any-to-any task adaptability. Furthermore, MARS employs a multi-stage training strategy that first establishes robust image-text alignment through complementary bidirectional tasks and subsequently concentrates on refining the T2I generation process, significantly augmenting text-image synchrony and the granularity of image details. Notably, MARS requires only 9% of the GPU days needed by SD1.5, yet it achieves remarkable results across a variety of benchmarks, illustrating the training efficiency and the potential for swift deployment in various applications.

Getting Started

Setup

git clone https://github.com/fusiming3/MARS.git   
cd MARS
pip install -r requirements.txt

Model

The code and ckpts are currently undergoing internal review. Please stay tuned!

Todo List

• paper
• inference code
• model weights
• training code

Acknowledgement

This repository is built based on the fancy, robust, and extensible work of Qwen and muse. We also thank HuggingFace for their contribution to the open source community.