Multi-modal grounded language learning
This atelier investigates early language learning from a multitude of perspectives ranging from multi-modal corpus studies, over NLP analysis of child language corpora to grounded robot experiments. The atelier focuses in particular on the integration of multiple modalities such as gaze, gesture, vocalisations and speech in models of language emergence in infants. Participants will learn 1) how to analyse multi-modal human-human, human-robot task description corpora, 2) how to extract patterns of child speech from large corpora and 3) how to build models for the emergence of gestures, vocabulary and early grammar. Models and analyses will be evaluated on robot data.
Program
| Date | Lecture (11:00-13:00) | Hands-on session (14:30-17:30) |
|---|---|---|
| 5 Sept. | Introduction (15:30-17:30) | Tutorial on Lisp and Babel2 (18:00-20:00) |
| 6 Sept. | Grounded robot experiments | Setting up a language game with grounded robot data |
| 7 Sept. | Corpora: annotation and NLP tools | |
| 8 Sept. | Computational Cognitive Semantics | Integration of day 2 and 3 |
| 9 Sept. | Finalisation and public demonstration |
Background materials
Lecture slides and tutorial files
- Babel2 and the Naming Game
Literature
- Stephanie Gross and Brigitte Krenn. The OFAI Multimodal Task Description Corpus.
- Stephanie Schreitter, Brigitte Krenn and Matthias Scheutz. Multi-modal referring expressions in human-human task descriptions and their implications for human-robot interaction. Interaction Studies, 2016.
- Michael Spranger and Katrien Beuls. Referential Uncertainty and Word Learning in High-dimensional, Continuous Meaning Spaces . Accepted at the 2016 Joint IEEE International Conferences on Development and Learning and Epigenetic Robotics (ICDL-Epirob).
- Luc Steels. Agent-based Models for the Emergence and Evolution of Grammar. Phil Trans. Royal Society B, 2016.
- Michael Spranger and Luc Steels. Co-Acquisition of Syntax and Semantics - An Investigation in Spatial Language. In IJCAI'15: Proceedings of the 24st international joint conference on Artificial intelligence, pages 1909-1905, Palo Alto, 2015. AAAI Press.
- Michael Spranger and Luc Steels. Discovering Communication through Ontogenetic Ritualisation. In 2014 Joint IEEE International Conferences on Development and Learning and Epigenetic Robotics (ICDL-Epirob), pages 14-19. IEEE, 2014.
- Michael Spranger. Grounded lexicon acquisition - case studies in spatial language. In 2013 Joint IEEE International Conferences on Development and Learning and Epigenetic Robotics (ICDL-Epirob), pages 1-6. IEEE, 2013.
Software
Practical details
This atelier is part of the Summer School on Creativity and Evolution, running from Monday 5 to Friday 9 September at Lake Como (Italy). Every day consists of two atelier sessions. The morning sessions consist of a series of lectures on selected topics that directly feed into the practical exercises taking place in the afternoon. Participants will collaborate on a research project that will be presented to all members of the Summer School in a live demonstration session on Friday afternoon.
Prerequisites : A background in AI and machine learning, interest in natural language modelling.
Registration : Apply before 1 July 2016 on the main website of the Summer School.
This atelier is ran by Katrien Beuls, Michael Spranger and Brigitte Krenn