Anotação multimodal para copilotos de produção de tecnologias assistivas: uma proposta para a audiodescrição

Abstract

This research investigates how the multimodal event annotation in FrameNet Brazil (FN-Br), developed based on Frame Semantics (Fillmore, 1982), can contribute to the generation of audio description scripts by artificial intelligence systems acting as copilots for human audio describers. The study is based on the hypothesis that this type of annotation, by semantically mapping events and their participants in audiovisual productions, can support the creation of more accurate and contextually appropriate audio description scripts. The objective is to observe the performance of AI systems as assistants in the production of audio description scripts, comparing those based on frame annotation with those that do not use this approach, while also reflecting on the potential of this tool to support the work of audio describers. To this end, scenes from episodes 1 and 7 of the travel series Pedro Pelo Mundo, produced by GNT, were annotated using the FN-Br model, and preliminary audio description scripts were generated by AI based on these annotations. The resulting scripts were then compared and analyzed in terms of their alignment with the criteria outlined in the Guide for Accessible Audiovisual Productions (Naves et al., 2016). The results indicate that frame annotations act as a modulator of the model’s performance, more frequently favoring stronger alignment between the generated descriptions and the visual organization of the scenes, without compromising the quality of the scripts when no immediate gains are observed. From a qualitative perspective, the presence of metadata tends to guide descriptions according to the visual dynamics of the scenes, whereas their absence favors more stative and enumerative descriptions. These results reinforce the potential of multimodal annotation based on Frame Semantics as support for the production of audio description scripts by AI systems.