Similarities Between Physical and Digital Pointing in Light of Sensory-Motor Transfer

Abstract

Human-computer interaction (HCI) systems and methods are evolving to eliminate intermediaries between the body and digital objects, creating "natural" interactions inspired by the skills acquired for interacting with objects in the physical world. This evolution raises questions about the transfer of skills between the physical and digital worlds and their coexistence within the user’s sensorimotor brain. It also shifts the assessment of the “natural” nature of HMIs toward evaluating the transfer of skills between the gestures developed to interact with physical objects and those used to interact with digital objects. The question of learning transfer is a central issue in cognitive science for examining the nature of the brain mechanisms underlying movement control. Thus, the objective of this project is to adapt the sensorimotor learning paradigmto assess the degree of similarity between different pointing modalities (drag-and-drop) of a digital object and the pointing (drag-and-drop) of a physical object. The challenge of this project is to explore new avenues for evaluating HMIs within the framework of embodied and situated cognition, and to better understand how different control mechanisms are established and coexist in the user’s brain. Our main study was published at CHI’2015 (Bérard and Rochet-Capellan, The Transfer of Learning as HCI Similarity: Towards an Objective Assessment of the Sensory-Motor Basis of Naturalness, In ACM conference on Computer-Human Interaction (CHI), 2015, Seoul, Korea).

Background

Touch-based navigation in the user interface

In human-computer interaction (HCI), spatial pointing allows the user to indicate a position to the system, whether to select an object or move it. Indirect pointing with a mouse has long been the dominant form of spatial pointing. However, the development of interactive surfaces and the widespread adoption of mobile devices (such as smartphones and tablets) has led to the rise of direct spatial pointing via touch. Direct pointing is also the preferred interaction method for augmented reality applications (e.g.,ViewAR, which allows users to view 3D objects in a room).

Compared to the mouse, touch-based pointing eliminates a coupling step in the interaction (Figure 1): the user “touches” the digital object, making the interaction direct. Touch-based pointing is considered one of the building blocks of natural interfaces (Wigdor and Wixon, 2011). However, the “natural” nature of gestural interfaces remains poorly defined, and its benefits are a subject of debate (Norman, 2010). Furthermore, the diversification of interaction modalities and their accessibility to a broad audience raises questions about the user’s ability to adapt their gestures to each of these devices and the potential for transferring or applying skills developed for one device to another. The study of transfer mechanisms is central to the challenges of research on motor control.

Sensory-motor learning and transfer

Human movement is characterized by a high capacity for adaptation and rehabilitation, the mechanisms of which can be studied in the laboratory using the sensorimotor learning paradigm (Figure 2).

For example, to study the adaptation of a target-reaching movement, this paradigm involves:
(1) Characterizing the movement under normal reaching conditions (baseline); 
(2) Training the subject to reach for targets in the presence of a disturbance to the arm trajectory (see Gandolfo et al. 1996) or visual feedback (see Fernandes et al. 2012) so that the subject modifies the control of their movement to reach the target by compensating for the disturbance (training); 
(3) Test whether training has resulted in learning by studying (a) the same movement after removing the disturbance (post) or (b) a different movement or context; this is referred to as transfer. 

The study of transfer allows us to examine the nature of control mechanisms and their relationship to experience (in the sense of "lived experience"). If modifying one movement through sensorimotor learning affects the execution of another movement, we can assume that both movements rely, at least in part, on common control mechanisms, and vice versa. At present, the nature of the representations and mechanisms underlying the motor control of limbs or speech remains an open question. The main challenge is to understand how the brain generalizes action while being able to integrate contextual specifics (see Taylor et al. 2013). In particular, some recent studies suggest that daily computer use increases the capacity for generalization of visuomotor learning (see Wei et al. 2014). 

Research Questions and Objectives

Learning a new mode of interaction requires drawing on existing sensorimotor skills and/or developing new ones. Users demonstrate a great capacity to adapt to interaction tools. However, not all interaction modalities are equivalent, both in terms of performance and in terms of their relationship to the gestures developed to interact with objects in the physical world. In this project, we propose using the sensorimotor learning paradigm to study the similarity between gestures used on physical objects and those used on digital objects. This approach should, in particular, pave the way for an objective assessment of the “naturalness” of an interaction. To this end, we propose to quantify the transfer effects of motor skills in a task performed using different interaction modalities—involving physical versus digital objects—by adapting the sensorimotor learning paradigm.

Publications

Bérard, F., & Rochet-Capellan, A. (2015). The Transfer of Learning as HCI Similarity: Towards an Objective Assessment of the Sensory-Motor Basis of Naturalness, ACM CHI'15 (Computer-Human Interaction, acceptance rate ~23%)

Abstract (in English): Human-computer interaction should be natural. However, the concept of "natural" is called into question due to a lack of theoretical foundations and methods for objectively measuring the naturalness of an HCI. A frequently cited aspect of natural HCIs is their ability to draw on the knowledge and skills that users develop through their interactions with the real (non-digital) world. Among these skills, sensory-motor abilities are essential for operating many HCIs. This suggests that the transfer of these abilities between physical and digital interactions could serve as an experimental tool to assess the sensory-motor similarity between interactions, and could be considered an objective measure of the sensory-motor foundation of naturalness. Within this framework, we introduce a new experimental paradigm inspired by motor learning research to assess sensory-motor similarity, as revealed by the transfer of learning. We tested this paradigm in an empirical study to examine the naturalness of three HCIs: direct touch, mouse pointing, and absolute indirect touch. The study revealed how skill learning transfers from these three digital interactions to an equivalent physical interaction. We observed strong skill transfer between direct-touch and physical interaction, but no transfer from the other two interactions. This work provides a first objective assessment of the sensory-motor basis of direct-touch naturalness, and a new empirical approach to examining HCI similarity and naturalness.

Partners, Collaborations

Amélie Rochet-Capellan, PCMD team, Gipsa-lab

The research conducted by the PCMD team at Gipsa-lab focuses in particular on the relationships between sensorimotor functions and cognition, with a specific emphasis on the role of orofacial and manual sensorimotor mechanisms in the development of language and communication. There are also strong theoretical and experimental links between the study of manual and orofacial gestures (Grimme et al., 2010; Perrier, 2012). Thus, the sensorimotor learning paradigm, initially developed for the study of limb movements, has been adapted to the study of motor control of speech (Houde and Jordan, 1998; Tremblay et al., 2003), as well as to specifically investigate learning transfer (Rochet-Capellan and Ostry, 2011, Rochet-Capellan et al., 2012). HCI offers particularly interesting avenues for the study of motor control because the technologies it develops allow for the creation of new sensorimotor learning situations by controlling numerous parameters.

François Bérard, IIHM team, LIG

The Human-Computer Interaction (HCI) Engineering team at the LIG laboratory designs and evaluates new forms of interaction that are primarily aimed at improving user performance and reducing the time it takes to learn how to use the system (Bérard, 2003). Measuring user performance is an essential tool for comparing different interactions, but this measurement provides only a very limited view of how well an interaction suits users (Bérard et al., 2011, 2012). The shift in research toward touch-based input and augmented reality is steering HCI work toward gesture-based interfaces and the use of motion-tracking devices. A better understanding of the mechanisms of motion control and its evaluation is therefore a central challenge that requires expertise in cognitive science and motor control.

References

BERARD, F. The Magic Table: Computer-Vision-Based Augmentation of a Whiteboard for Creative Meetings. In IEEE International Workshop on Projector-Camera Systems (PROCAMS) (2003).

BERARD, F., AND ROCHET-CAPELLAN, A. Measuring the linear and rotational precision of touch pointing. In ACM International Conference on Interactive Tabletops and Surfaces (ITS) (2012), ACM, pp. 183–192.

BERARD, F., WANG, G., AND COOPERSTOCK, J. R. On the limits of human motor control precision: the search for a device’s human resolution. In IFIP Conference on Human-Computer Interaction, INTERACT (2011), Springer-Verlag, pp. 107–122.

FERNANDES, H. L., STEVENSON, I. H., AND KORDING, K. P. Generalization of stochastic visuomotor rotations. PLoS ONE 7, 8 (2012), e43016.

GANDOLFO, F., MUSSA-IVALDI, F., AND BIZZI, E. Motor learning by field approximation. Proceedings of the National Academy of Sciences 93, 9 (1996), 3843–3846.

GRIMME, B., FUCHS, S., PERRIER, P., SCHONER, G., ET AL. Limb versus speech motor control: A conceptual review. Motor Control 15, 1 (2011), 5–33.

HOUDE, J. F., AND JORDAN, M. I. Sensorimotor adaptation in speech production. Science 279, 5354 (1998), 1213–1216. NORMAN, D. A. The way I see it: Natural user interfaces are not natural. Interactions 17, 3 (2010), 6–10.

PERRIER, P. Gesture planning integrating knowledge of the motor system’s dynamics: A literature review from motor control and speech motor control. Speech Planning and Dynamics (2012), 191–238.

ROCHET-CAPELLAN, A., AND OSTRY, D. J. Simultaneous acquisition of multiple auditory–motor transformations in speech. The Journal of Neuroscience 31, 7 (2011), 2657–2662.

ROCHET-CAPELLAN, A., RICHER, L., AND OSTRY, D. J. Nonhomogeneous transfer reveals specificity in speech motor learning. Journal of Neurophysiology 107, 6 (2012), 1711–1717.

TAYLOR, J. A., AND IVRY, R. B. Context-dependent generalization. Frontiers in Human Neuroscience 7 (2013).

TREMBLAY, S., SHILLER, D. M., AND OSTRY, D. J. The somatosensory basis of speech production. Nature 423, 6942 (2003), 866–869.

WEI, K., YAN, X., KONG, G., YIN, C., ZHANG, F., WANG, Q., AND KORDING, K. P. Computer use alters the generalization of movement learning. Current Biology 24, 1 (2014), 82–85.

WIGDOR, D., AND WIXON, D. Brave NUI World: Designing Natural User Interfaces for Touch and Gesture. Elsevier, 2011.