TAPIOCA

People involved

Céline Coutrix (LIG), Yann Laurillau (LIG), Jean-Claude Leon (LJK), Cédric Masclet (G-SCOP), Frank Quaine (GIPSA-lab), Damien Romer (LJK).

Project Summary

By exploring the convergence of the physical and digital worlds, this project aims to identify and design the interactive systems of the future based on gesture recognition and utilizing resizable mixed tools ( RMT). In this project, these tools are dedicated to the typical domain of designing physical artifacts, such as industrial products, and digital artifacts, such as 3D models.

Thanks to technological advancements, such as Microsoft’s Kinect, gesture-based interaction is now widely regarded as a more intuitive and natural way to interact with computers. It paves the way for a wide range of potential applications, such as healthcare and education. Furthermore, gestures also play an important role in thinking and collaboration, particularly in design activities.

In particular, tangible gesture interactions can play a positive role in collaborative design. However, designers end up using a large number of tangible devices of varying sizes in order to balance their need for input space with their need for physical space. It is important to be able to constantly balance these two needs, particularly when manipulating and visualizing 3D CAD (Computer-Aided Design) models, but the problem is difficult because it challenges current interaction paradigms and technologies.

At the heart of this project, RMTs are objects that combine physical and digital representations with resizing capabilities, used as tools for design activities. By supporting much more natural interactions and activities rooted in the physical world, we envision such a system that is (1) capable of adapting the interaction to the user’s needs, (2) capable of proactively assisting users in their tasks through gesture and intent recognition, and (3) facilitating and increasing the efficiency of artifact design activities.

Four major scientific challenges—closely interrelated and difficult to address, requiring an interdisciplinary approach—have been identified:

1. Robustly and accurately recognize forearm grasping gestures and identify users' intentions so that an interactive system can proactively assist users (such as designers) in their tasks.
2. Identify tangible and scalable interaction techniques based on RMT, and design interactive systems capable of adapting the interaction to the user's needs and gestures.
3. Develop appropriate design tools based on RMT that can be adapted to designers' needs and provide information on the grasping properties of the product being designed.
4. Provide 3D digital representations for design activities that support rich deformation modes combined with relevant interaction techniques based on RMT for manipulating deformable 3D models.

The challenges addressed span several scientific fields: human-machine interaction and tangible gesture interaction, as well as resizable user interfaces, collaborative design and design practices using CAD tools, EMG signal processing and classification, and 3D modeling and deformable 3D models. It encourages collaboration among researchers with interdisciplinary expertise from the GIPSA-lab (EMG), G-SCOP (cooperative design), LIG (human-computer interaction), and LJK (3D models) departments.