VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix

Teng Wang1,2; Wenhao Jiang3; Zhichao Lu1; Feng Zheng1; Ran Cheng1; Chengguo Yin3; Ping Luo2

2022

38th International Conference on Machine Learning (ICML)

2640-3498

INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162

JUL 17-23, 2022

null,Baltimore,MD

1269 LAW ST, SAN DIEGO, CA, UNITED STATES

JMLR-JOURNAL MACHINE LEARNING RESEARCH

Existing vision-language pre-training (VLP) methods primarily rely on paired image-text datasets, which are either annotated by enormous human labors, or crawled from the internet followed by elaborate data cleaning techniques. To reduce the dependency on well-aligned imagetext pairs, it is promising to directly leverage the large-scale text-only and image-only corpora. This paper proposes a data augmentation method, namely cross-modal CutMix (CMC), for implicit cross-modal alignment learning in unpaired VLP. Specifically, CMC transforms natural sentences from the textual view into a multi-modal view, where visually-grounded words in a sentence are randomly replaced by diverse image patches with similar semantics. There are several appealing proprieties of the proposed CMC. First, it enhances the data diversity while keeping the semantic meaning intact for tackling problems where the aligned data are scarce; Second, by attaching cross-modal noise on uni-modal data, it guides models to learn token-level interactions across modalities for better denoising. Furthermore, we present a new unpaired VLP method, dubbed as VLMixer, that integrates CMC with contrastive learning to pull together the uni-modal and multi-modal views for better instance-level alignments among different modalities. Extensive experiments on five downstream tasks show that VLMixer could surpass previous state-of-the-art unpaired VLP methods. Project page: https: //github.com/ttengwang/VLMixer

First ; Corresponding

English

CPCI-S

National Natural Science Foundation of China["61972188","62122035","61906081","62106097"] ; China Postdoctoral Science Foundation["2021M691424","27208720","17212120"]

Computer Science

Computer Science, Artificial Intelligence

WOS:000900130203039

Web of Science

1.Department of Computer Science and Engineering, Southern University of Science and Technology
2.Department of Computer Science, The University of Hong Kong 3
3.Data Platform, Tencent

Teng Wang,Wenhao Jiang,Zhichao Lu,et al. VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix[C]. 1269 LAW ST, SAN DIEGO, CA, UNITED STATES:JMLR-JOURNAL MACHINE LEARNING RESEARCH,2022.