Talking Heads revisited

 

Context:
The Talking Heads experiment is a distributed multi-agent experiment that has been conducted in laboratories in Paris, Brussels and Amsterdam as well as in public settings in Antwerp, London and Cambridge in the years 1999 and 2000. Every location was equiped with multiple Talking Heads (in the form of cameras) looking at a whiteboard with objects on it while they were playing language games. By means of discrimination trees, objects could be distinguished and a game would succeed when the hearer can point to the object he thinks the speaker was talking about. Over the course of many games, agents would start to share the same vocabularies to describe different scenes. Agents could be transfered to different locations and align their vocabularies on a global level over the internet.

 

A language game can be considered as a microworld that operationalizes everything needed for modeling a routinized, communicative interaction: a situated context, two (or more) interlocutors, a communicative purpose, and so on. In a single game, the interlocutors go through multiple steps: conceptualization/interpretation, linguistic processing and learning and alignment. Over the last decade, a multi-agent language game framework, called Babel2, has been developed at the AI Lab to support experimentation in language evolution. It allows a user to implement a particular language game by building on the Babel2 abstractions, such as agent, experiment, world, etc. Specific processing components for conceptualization (IRL), language (FCG) and learning (reflective architecture) have been developed to allow powerful cognitive processing. Due to the modular nature of Babel2, the use of these components is optional.

Project:
This project focuses on building all the components required by the Talking Heads experiment within the Babel2 framework. Babel2 is written in the Common Lisp Object System (CLOS). The project thus requires the necessary abstractions to be provided, either by subclassing or creating new classes. Examples of classes are not only restricted to agents and actions but also to less common classes that are specific to Babel2 and the requirements of the Talking Heads experiment. On the processing side, the correct methods should be specialized or new generic functions have to be provided for all the steps required to successfully play a game. We provide you with visual data that has been recorded on Sony QRIO robots and processed by our object detection algorithms. Also the multi-agent language game framework Babel2 is available from the start of the project (the latest release can be downloaded here).

A second large part of the project is concerned with the implementation of an interactive graphical user interface in the web browser will enhance the accessibility of a game's internal states and improve the user-friendliness of the approach. Babel2 already supplies a basic web interface to visualize the output of the running system. This web interface should be further extended and the new components required by the Talking Heads experiment and specifically for interaction should be provided.

Testing both through integration and units tests is indispensable in successfully incorporating all components. Babel2 already supplies a unit testing framework the tests themselves need to be written as part of this project. At the lab we prefer a more iterative prototype-based software engineering approach.

Some parts of this project are AI specific, yet it also requires extensive software design and engineering. Furthermore, the web-based user interface integrates GUI and web technologies.

Basic Requirements:

  • Implement a new version of the Talking Heads experiment.
  • Get acquainted with the Babel2 framework (its core classes and methods). A manual is available (Loetzsch et al, 2008).
  • Build a conceptualization component that processes data coming from the vision system into discrimination trees in a similar fashion as in the original experiment.
  • Add a linguistic component that translates meanings derived from the conceptualization process into a linguistic form and reversely parses forms into possible meanings.
  • Develop a component for learning and alignment that allows to add a new vocabulary item and adapt existing items.
  • Develop a graphical user interface to monitor the actions in one game.
  • Get acquainted with the existing web interface (Loetzsch, 2012).
  • Display all steps in the game in the web browser.
  • Make the interface interactive so that the user can modify steps in the game or invent his own word for an object in the browser.
  • For each component, tests should be written and maintained.

Extras:

  • Integration of the discrimination trees conceptualization within the IRL module through conceptual primitives.
  • Integration of the linguistic component within FCG module to allow for more robust processing.

Background & Literature:

Babel2 resides on an SVN server, so knowledge of Subversion is an asset to start this project. Apart from that, you should be prepared to program in CLOS so basic Object Oriented Programming (polymorphism, inheritance) are an absolute must. Experience with Common Lisp will come in very useful. If you are not familiar with the Common Lisp programming language, we advise you to have a look at (Seibel, 2005) before starting the project. Lastly, basic knowledge of HTML is required (and preferrably AJAX).

Relevant publications

  • Keene, S.: Object-Oriented Programming in Common Lisp: A Programmar's Guide to CLOS. Addison-Wesley, Boston (MA) (1988)
  • Loetzsch, M., Wellens, P., De Beule, J., Bleys, J.,van Trijp, R.: The Babel2 manual. Technical Report AI-Memo 01-08, AI-Lab VUB, Brussels (2008).
  • Loetzsch, M.: Tools for Grammar Engineering. In Steels, L., ed.: Computational Issues in Fluid Construction Grammar. Springer Verlag, Berlin (2012).
  • Russell, S. and Norvig, P.: Artificial Intelligence: A Modern Approach. Prentice Hall (2003).
  • Seibel, P.: Practical Common Lisp. Apress (2005).
  • Spranger M., Pauw S., Loetzsch M. and Steels L.: Open-ended Procedural Semantics. In Steels, L., ed.: Language Grounding in Robots. Springer Verlag, Berlin (2012).
  • Steels, L.: The Talking Heads Experiment: Words and Meanings. Best of Publishing, Brussels (1999).
  • Steels, L., Loetzsch, M.: Babel: A tool for running experiments on the evolution of language. In Nolfi, S., Mirolli, M., eds.: Evolution of Communication and Language in Embodied Agents. Springer Verlag, Berlin (2010) 307-313.

For more information, please contact Pieter Wellens.