Semantic Technology Laboratory
Automatic Modeling of Social Concepts Evoked by Art Images as Multimodal Frames by Delfina Sol Pandiani
Social concepts referring to non-physical objects–such as revolution, violence or friendship–are powerful resources to describe, index, and query the content of visual data, including ever-growing collections of art images from the Cultural Heritage (CH) field. While much progress has been made towards the goal of complete image understanding in computer vision, automatic detection of these concepts, labeled “abstract concepts” in the field of cognitive neuroscience, is still far from real. This is partly due to the well-known semantic gap challenge, worsened for social concepts given their lack of unique physical features, and reliance on a larger proportion of unspecific features than concrete concepts.
In this presentation, I will present recent cognitive theories about abstract concept representation and my proposal to translate them into a software architecture that can learn to associate social concepts with art images. My approach focuses on the extraction, analysis, and integration of multimodal features from visual material tagged with the concepts of interest, in order to represent social concepts as multimodal frames. I will present experiments and results from an initial corpus of art images, taking the Tate Gallery’s collection as an empirical basis. Furthermore, I will discuss the Social Concepts Knowledge Graph currently under construction, whose data can support research from a variety of disciplines.
Source
