Neural Semantic Pointers in Context

Alessio Plebe, Arianna Pavone

2020

Abstract

Resolving linguistic ambiguities is a task frequently called for in human communication. In many cases, such task cannot be solved without additional information about an associated context, which can be often captured from the visual scene referred by the sentence. This type of inference is crucial in several aspects of language, communication in the first place, and in the grounding of language in perception. This paper focuses on the contextual effects of visual scenes on semantics, investigated using neural computational simulation. Specifically, here we address the problem of selecting the interpretation of sentences with an ambiguous prepositional phrase, matching the context provided by visual perception. More formally, provided with a sentence, admitting two or more candidate resolutions for a prepositional phrase attachment, and an image that depicts the content of the sentence, it is required to choose the correct resolution depending on the image’s content. From the neuro-computational point of view, our model is based on Nengo, the implementation of Neural Engineering Framework (NEF), whose basic semantic component is the so-called Semantic Pointer Architecture (SPA), a biologically plausible way of representing concepts by dynamic neural assemblies. We evaluated the ability of our model in resolving linguistic ambiguities on the LAVA (Language and Vision Ambiguities) dataset, a corpus of sentences with a wide range of ambiguities, associated with visual scenes.

Paper Citation

in Harvard Style

Plebe A. and Pavone A. (2020). Neural Semantic Pointers in Context. In Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020) - Volume 1: NCTA; ISBN 978-989-758-475-6, SciTePress, pages 447-454. DOI: 10.5220/0010145904470454

in Bibtex Style

@conference{ncta20,
author={Alessio Plebe and Arianna Pavone},
title={Neural Semantic Pointers in Context},
booktitle={Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020) - Volume 1: NCTA},
year={2020},
pages={447-454},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010145904470454},
isbn={978-989-758-475-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020) - Volume 1: NCTA
TI - Neural Semantic Pointers in Context
SN - 978-989-758-475-6
AU - Plebe A.
AU - Pavone A.
PY - 2020
SP - 447
EP - 454
DO - 10.5220/0010145904470454
PB - SciTePress