neuronalx/csl-bgm

Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots

Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots

Cross-Situational Learning with Bayesian Generative Model　(CSL-BGM)

These source code can be used in both iCub simulator and real iCub.　　

Introduction

Figure 1: This figure shows an overview of the cross-situational learning scenario as the focus of this study. The robot obtains multimodal information from multiple sensory-channels in a situation and estimates the relationships between words and sensory-channels.

Figure 2: This figure shows the procedure for obtaining and processing data.

Execution PC environment

• Ubuntu 14.04
• iCub software and YARP
• ODE 0.13.1 and SDL
• Python 2.7.6 (numpy,scipy,scikit-learn)
• OpenCV 3.1.0
• CNN feature extracter: Caffe (Reference model:Alex-net)

How to install iCub softwere

Plase see these sites.

• http://wiki.icub.org/wiki/ICub_Software_Installation
• http://wiki.icub.org/wiki/Linux:Installation_from_binaries

---

Abstract:
Human infants can acquire word meanings by estimating the relationships among multimodal information and words. In this paper, we propose a novel Bayesian generative model that can form multiple categories based on each sensory-channel and can associate words with any of four sensory-channels (action, position, object, and color). This paper focuses on cross-situational learning using the co-occurrence between words and information of sensory-channels in complex situations. We conducted a learning experiment using a simulator and a real humanoid iCub robot. In the experiments, a human tutor provided a sentence that describes an object of visual attention and an accompanying action to the robot. The experimental results showed that the proposed method was able to estimate the multiple categorizations and to learn the relationships between multiple sensory-channels and words accurately. In addition, we conducted an action generation task and an action description task based on word meanings learned in the cross-situational learning experiment. The experimental results showed the robot could successfully use the word meanings learned by using the proposed method.

Keywords: Bayesian model, cross-situational learning, lexical acquisition, multimodal categorization, symbol grounding, word meaning

Citation: Taniguchi A, Taniguchi T and Cangelosi A (2017) Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots. Front. Neurorobot. 11:66. doi: 10.3389/fnbot.2017.00066

Paper:
https://www.frontiersin.org/articles/10.3389/fnbot.2017.00066/full

Video:
https://youtu.be/SzyoWaj47Xc