World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models

Abstract

Ziqiao Ma*, Jiayi Pan*, Joyce Chai. ⭐️ ACL 2023 Outstanding Paper.

We introduce Grounded Open Vocabulary Acquisition (GOVA) to examine grounding and bootstrapping in open-world language learning. We also propose object-oriented BERT (OctoBERT), a visually-grounded language model highlighting grounding as an objective. Our experiments demonstrate that OctoBERT is a more coherent and fast grounded word learner, and that the grounding ability helps the model to learn unseen words more rapidly and robustly.