World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models



Ziqiao Ma*, Jiayi Pan*, Joyce Chai. ⭐️ ACL 2023 Outstanding Paper.

We introduce Grounded Open Vocabulary Acquisition (GOVA) to examine grounding and bootstrapping in open-world language learning. We also propose object-oriented BERT (OctoBERT), a visually-grounded language model highlighting grounding as an objective. Our experiments demonstrate that OctoBERT is a more coherent and fast grounded word learner, and that the grounding ability helps the model to learn unseen words more rapidly and robustly.

The 61st Annual Meeting of the Association for Computational Linguistics
Jiayi Pan
Jiayi Pan