ATTENTIVE

Reasons

Monitoring vulnerable individuals using sensors installed in buildings (cameras, microphones, etc.) is generally met with resistance because these sensors are too intrusive. Furthermore, integrating these sensors into the infrastructure requires costly investments, and their installation is often complex. To avoid this pitfall, we propose implementing a platform consisting of a robot that is clearly visible to people and equipped with a head that gives the impression of a companion. This companion robot prototype must be capable of moving, perceiving its environment, and analyzing complex situations, as well as focusing its attention and navigating within dynamic environments.

On this experimental platform, it is possible to design and evaluate innovative algorithms for data processing and fusion, target localization and tracking, situational analysis, and adaptation of the perception system. Ultimately, the goal is to advance research in the field of assisting vulnerable individuals through the use of mobile robots equipped with a perception system that is both suitable and acceptable to the person being observed.

Partners

Olivier Aycard, LIG/AMA

Catherine Garbay, LIG/AMA

Denis Pellerin, GIPSA-Lab/AGPIG

Michèle Rombaut, GIPSA-Lab/AGPIG

The AGPIG (Architecture, Geometry, Perception, Images, Gestures) team at the GIPSA-lab (Grenoble Images, Speech, Signal, Automation) has developed recognized expertise in methods for extracting and fusing information from video data, and is conducting research on the role of audio information as a complement to visual information.

The AMA team (Learning: Models and Algorithms) at the Grenoble Computer Science Laboratory (LIG) works on interpreting sensor data for the localization of mobile robots in their environment and the detection and tracking of moving objects. The team has expertise in classifying moving objects in images using machine learning techniques. It also specializes in the fusion of heterogeneous data and has developed abductive reasoning models implemented by distributed agents.

Both teams are interested in numerical and symbolic modeling (spatio-temporal data models (AGPIG) vs. normative distributed agents (AMA)). One of the major benefits of this collaboration is the complementary expertise of researchers from the GIPSA-lab and LIG laboratories, who specialize in the following areas: sensors and associated data models (laser (AMA), vision & audio (AGPIG), physiology & actimetry (AMA)), modeling uncertainty (Bayesian approaches (AMA) and credentialist approaches (AGPIG)), and numerical and symbolic modeling (spatio-temporal data models (AGPIG) vs. normative distributed agents (AMA)).

Results and Objectives

This project enabled us to support the initial research conducted as part of Quentin Labourey’s Master’s thesis. An application was developed for a Reeti robot (an expressive robotic head) manufactured by Robopec and equipped with microphones and cameras. Using a multimodal fusion method combining sound and image, the robot is capable of detecting the person speaking during a conversation among several participants. This work led to an initial publication [1].

The research conducted as part of Quentin Labourey’s dissertation focuses on recognizing situations of interest and defining movement strategies to improve situational awareness.

As part of this project, we have acquired a new Qbo robotic platform from Thecorpora, which is also equipped with microphones and cameras but also has the ability to move. It is on this platform that the research topics for this project are being developed:

• Multiscale data analysis to extract event signatures
• Attention Focus and Navigation in Dynamic Environments
• Technical and Application Evaluation

Positioning of the topic in relation to one of the Labex action work packages (ADM and SIM)

• Analysis of distributed, multimodal data streams and digital traces of events from sensors with different characteristics
• Merging data from different sources (audio, location, actigraphy, physiology, etc.),
• A combination of measures of a very different nature (taking into account prior knowledge about the environment),
• Analysis of interdependent variables at different spatial and temporal scales,
• Modeling uncertainty and accounting for various sources of uncertainty in the management of interpretive assumptions

Publications

[1] Labourey, Q., Aycard, O., Pellerin, D., and Rombaut, M., “Audiovisual data fusion for successive speaker tracking,” International Conference on Computer Vision Theory and Applications (VISAPP’2014), Lisbon, Portugal, January 2014.

[2] Song G., Pellerin D., Granjon L., “Different types of sounds influence gaze differently in videos,” Journal of Eye Movement Research, 6(4):1, 1–13, 2013

[3] TD. Vu, J. Burlet, O. Aycard, “Grid-based localization and local mapping with moving object detection and tracking.” *Information Fusion* 12(1): 58–69, 2011.

[4] B. Vettier, L. Amate, C. Garbay, J. Fontecave-Jallon, P. Baconnier, “Managing Multiple Hypotheses with Agents to Handle Incomplete and Uncertain Data,” URMASSN'2011, Belfast, Ireland, 2011.

[5] E. Ramasso, C. Panagiotakis, M. Rombaut, D. Pellerin, "Belief Scheduler based on model failure detection in the TBM framework. Application to human activity recognition." Int. J. Approx. Reasoning 51(7): 846–865, 2010