GLIP

Grounded Language-Image Pre-training

7 Dec 2021

CVPR2022 Oral

(in 33 best paper finalist)

https://github.com/microsoft/GLIP   

https://arxiv.org/abs/2112.03857

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, Jianfeng Gao

This paper presents a grounded language-image pre-training (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines. 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head.

本文提出了一种用于学习对象级、语言感知和语义丰富的视觉表示的扎根语言图像预训练（GLIP）模型。GLIP将目标检测和短语基础统一用于预训练。这种统一带来了两个好处：1）它允许GLIP从检测和基础数据中学习，以改进这两个任务，并引导一个良好的基础模型；2） GLIP可以通过以自我训练的方式生成接地框来利用大量图像-文本对，从而使学习到的表示语义丰富。在我们的实验中，我们在27M的基础数据上预训练GLIP，包括3M人类注释和24M网络爬网图像-文本对。学习到的表示方法显示出很强的零样本和少量样本可传递到各种对象级识别任务。1） 当直接在COCO和LVIS上进行评估时（在训练前没有看到COCO中的任何图像），GLIP分别达到49.8 AP和26.9 AP，超过了许多监督基线。2） 在COCO上进行微调后，GLIP在val上达到60.8 AP，在测试开发上达到61.5 AP，超过了之前的SoTA。3） 当转移到13个下游目标检测任务时，一个单发GLIP可以与一个完全监督的动态头部相媲美。