MYSSIV

Participants

Denis Pellerin, GIPSA-Lab; Christian Graff, LPNC;Sylvain Huet, GIPSA-Lab

Scientific Background

Sensory Substitution Devices (SSD)

Sensory substitution aims to compensate for impairments in one sense by transmitting, through another functioning sense, all or part of the information that the impaired sense can no longer perceive. SSDs designed to compensate for visual impairment generally consist of a camera that captures visual information about the user’s environment, a processor that processes this information, and actuators that generate tactile or auditory stimuli. Studies have shown that SSDs enable users to recognize patterns or objects, navigate hallways, and even recognize letters.

The AdViS (Adaptive Visual Substitution) system

The AdViS system we are working on is the result of an interdisciplinary project combining signal and image processing, visual perception, software and hardware architectures, neuroscience, and psychophysics. Launched in 2011, it involves the GIPSA-Lab (AGPIG team, Architecture, Geometry, Perception, Image, and Gesture) and the LNPC (PSM team, Perception and Sensory-Motor). This system is designed to assist the navigation of visually impaired individuals by converting spatial data from a depth camera into auditory information [1,2].

Barriers and challenges addressed in the MYSSIV project

We are continuing to develop the AdViS system along two innovative lines.

Spatial immersion in a virtual world

The development of a prototype sensory substitution device and its use in a real-world environment to evaluate a substitution strategy presents certain challenges: the acquisition of the substituted sensory modality is subject to sensor limitations and restricted to existing sensors; its evaluation can only be performed after building a complete prototype; and finally, real-world scenarios are complex or even dangerous, and setting up controlled scenarios is labor-intensive. To overcome some of these obstacles, we propose spatially immersing a user in a virtual 3D scene that they will perceive through hearing via the AdViS sensory substitution device.

Utilization of eye movement

The version of AdViS currently under development focuses on providing audio feedback based on an area of interest—referred to as the focal zone—on the depth map. The size and position of the focal zone can be adjusted. We plan to enable the adjustment of this focal point’s position using an eye tracker: by leveraging the residual eye movement present in certain categories of visually impaired individuals, we hope to restore a perception-action loop that is essential for spatial perception.

 Internships funded by the MYSSIV project

• Adrien Bardet, Improving the acoustic ergonomics of a visual-auditory sensory substitution system, Phelma^2nd-year engineering internship, May–July 2016
• Aziliz Guezou-Philippe, Spatial Immersion in a Virtual World for the Development of Sensory Substitution Devices, Master’s-level internship, February–August 2017

Scientific output

• Aziliz Guezou-Philippe, Sylvain Huet, Denis Pellerin, Christian Graff. Prototyping and Evaluating Sensory Substitution Devices by Spatial Immersion in Virtual Environments. VISIGRAPP, the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP 2018), Jan 2018, Funchal, Portugal. 2018. <hal-01663686>
• Sylvain Huet, Julien Doré, Zélie Buquet, Denis Pellerin, Christian Graff. Eye-movement in the dark for the exploration of virtual scenes encoded by sounds. 19th European Conference on Eye Movements (ECEM), Aug 2017, Wuppertal, Germany. <hal-01613512>

References

[1] C. Stoll, R. Palluel-Germain, V. Fristot, D. Pellerin, D. Alleysson, and C. Graff, "Navigating from a depth image converted into sound," *Applied Bionics and Biomechanics*, vol. 2015, article ID 543492, 2015. 9 pages.

[2] V. Fristot, J. Boucheteil, L. Granjon, D. Pellerin, and D. Alleysson, "Depth-Melody Substitution," in 20th European Signal Processing Conference (EUSIPCO-2012) (Eurasip, ed.), (Bucharest, Romania), Aug. 2012. 5 pages.