Multi-modal grounded language learning

This atelier investigates early language learning from a multitude of perspectives ranging from multi-modal corpus studies, over NLP analysis of child language corpora to grounded robot experiments. The atelier focuses in particular on the integration of multiple modalities such as gaze, gesture, vocalisations and speech in models of language emergence in infants. Participants will learn 1) how to analyse multi-modal human-human, human-robot task description corpora, 2) how to extract patterns of child speech from large corpora and 3) how to build models for the emergence of gestures, vocabulary and early grammar. Models and analyses will be evaluated on robot data.


Date Lecture (11:00-13:00) Hands-on session (14:30-17:30)
5 Sept. Introduction (15:30-17:30) Tutorial on Lisp and Babel2 (18:00-20:00)
6 Sept. Grounded robot experiments Setting up a language game with grounded robot data
7 Sept. Corpora: annotation and NLP tools
8 Sept. Computational Cognitive Semantics Integration of day 2 and 3
9 Sept. Finalisation and public demonstration

Background materials

Lecture slides and tutorial files

Practical details

This atelier is part of the Summer School on Creativity and Evolution, running from Monday 5 to Friday 9 September at Lake Como (Italy). Every day consists of two atelier sessions. The morning sessions consist of a series of lectures on selected topics that directly feed into the practical exercises taking place in the afternoon. Participants will collaborate on a research project that will be presented to all members of the Summer School in a live demonstration session on Friday afternoon.

Prerequisites : A background in AI and machine learning, interest in natural language modelling.
Registration : Apply before 1 July 2016 on the main website of the Summer School.
This atelier is ran by Katrien Beuls, Michael Spranger and Brigitte Krenn