A picture is worth a thousand words, the adage reads. However, pictures cannot replace words in terms of their ability to efficiently convey clear (mostly) unambiguous and concise knowledge. Images and text, indeed, reveal different and complementary information that, if combined, result in more information than the sum of that contained in the single media. The combination of visual and textual information can be obtained by linking the entities mentioned in the text with those shown in the pictures. To further integrate this with agent background knowledge, an additional step is necessary. That is, either finding the entities in the agent knowledge base that correspond to those mentioned in the text or shown in the picture or, extending the knowledge base with the newly discovered entities. This complex task is called Visual-Textual-Knowledge Entity Linking (VTKEL). In this paper, we present a purely unsupervised algorithm for the solution of the VTKEL tasks. The evaluation on the VTKEL dataset —a dataset composed of about 30K pictures, annotated with visual and textual entities, and linked to the YAGO ontology— shows promising results.

On Visual-Textual-Knowledge Entity Linking

Rospocher, Marco;
2020-01-01

Abstract

A picture is worth a thousand words, the adage reads. However, pictures cannot replace words in terms of their ability to efficiently convey clear (mostly) unambiguous and concise knowledge. Images and text, indeed, reveal different and complementary information that, if combined, result in more information than the sum of that contained in the single media. The combination of visual and textual information can be obtained by linking the entities mentioned in the text with those shown in the pictures. To further integrate this with agent background knowledge, an additional step is necessary. That is, either finding the entities in the agent knowledge base that correspond to those mentioned in the text or shown in the picture or, extending the knowledge base with the newly discovered entities. This complex task is called Visual-Textual-Knowledge Entity Linking (VTKEL). In this paper, we present a purely unsupervised algorithm for the solution of the VTKEL tasks. The evaluation on the VTKEL dataset —a dataset composed of about 30K pictures, annotated with visual and textual entities, and linked to the YAGO ontology— shows promising results.
2020
978-1-7281-6332-1
Artificial intelligence , Knowledge Representation , Natural Language Processing , Computer Vision , Entity recognition and linking
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11562/1014825
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 4
  • ???jsp.display-item.citation.isi??? 3
social impact