Framed Multi30K: Um dataset multimodal-multilíngue baseado em semântica de frames

Abstract

The combination of dierent communication modalities is one of the defining characteristics of human expression. However, much research has focused on analyzing the semantics of text and images separately. In recent decades, systems that process image and text data in a correlated way have been applied to computational tasks such as data retrieval, machine translation, and caption generation. In this work, we start from the premise that the development of such computational applications can benefit from a better understanding of the meanings that are established from the combination of textual and visual information. In particular, this dissertation contributes an innovative dataset that adds to a family of gold standard datesets for multimodal and multilingual NLP - Flickr30k, Multi30k and Flickr30k Entities - semantic information structured in terms of frames, as modeled in FrameNet Brasil. The resulting dataset, called Framed Multi30k (FM30K), contributes the following innovative data: (i) 150,000 conceptual descriptions originally written in Brazilian Portuguese for each of the 30,000 images in the Flickr30k dataset; (ii) 30,000 translations into Brazilian Portuguese of one of the descriptions originally written in English for each of the images in the Flickr30K; (iii) automatic annotations for frames of all the conceptual descriptions in the dataset into Brazilian Portuguese and English, totaling 330,000 semantically annotated descriptions; (iv) manual annotations for each of the bounding boxes from the dataset Flickr30k Entities in three dierent annotation conditions: annotation of entities with presence of description, annotation of entities without presence of description and annotation of events with presence of description. The resulting dataset was analized for formal aspects of the descriptions created in Brazilian Portuguese and for the cosine similarity between the semantic representations derived from the automatic and manual annotations carried out for the descriptions and images, respectively. Qualitative analyses were also carried out on the dierent perspectives encoded in the semantic representations generated for the images in each of the annotation conditions. The analyses corroborate the hypothesis that dierent annotation conditions regarding the interaction between modalities lead to dierent semantic representations for the images, which reinforces the argument in favor of adopting a perspectivist approach to human curation of datasets.