Similarity between physical and digital pointing in light of sensorimotor transfer learning

Abstract

Human-Computer Interaction (HCI) systems and modalities are evolving to eliminate intermediaries between the body and digital objects, creating "natural" interactions inspired by skills acquired for interacting with objects in the physical world. This evolution raises the question of how skills transfer between the physical and digital worlds and how they coexist in the user's sensory-motor brain. It also directs the evaluation of the "naturalness" of HCIs toward assessing the transfer of skills between gestures developed for interacting with physical objects and those used for interacting with digital objects. The question of learning transfer is central in cognitive sciences for examining the nature of the cerebral mechanisms underlying movement control. Thus, the objective of this project is to adapt the sensory-motor learning paradigm to assess the degree of similarity between different pointing modalities (drag-and-drop) of a digital object and the pointing (drag-and-drop) of a physical object. The challenge of this project is to explore new ways of evaluating HCIs within the framework of embodied and situated cognition, and to better understand how different control mechanisms develop and coexist in the user's brain. Our main study was published at CHI’2015 (Bérard and Rochet-Capellan, The Transfer of Learning as HCI Similarity: Towards an Objective Assessment of the Sensory-Motor Basis of Naturalness, In ACM conference on Computer-Human Interaction (CHI), 2015, Seoul, Korea).

Background

Touch pointing in HCI

In Human-Computer Interaction (HCI), spatial pointing allows the user to designate a position to the system, whether to select an object or move it. Indirect mouse pointing has long been the dominant mode of spatial pointing. However, the development of interactive surfaces and the popularization of mobile devices ("smartphones" or tablets) have driven the growth of direct spatial touch pointing. Direct pointing is also the preferred interaction method for augmented reality applications (cf. ViewAR, which allows users to view 3D objects in a room).

Compared to a mouse, touch pointing eliminates a coupling step in the interaction (Figure 1): the user "touches" the digital object, making the interaction direct. Touch pointing is considered one of the core building blocks of natural interfaces (Wigdor and Wixon, 2011). However, the "natural" character of gestural interfaces remains poorly defined, and its benefits are a subject of debate (Norman, 2010). Furthermore, the diversification of interaction modalities and their accessibility to a broad audience raise questions about the user's ability to adapt their gestures to each of these devices, as well as the possibilities for transferring or independently applying skills developed on one device to another. Studying the mechanisms of transfer is central to the challenges of motor control research.

Sensorimotor learning and transfer

Human gestures are characterized by a high capacity for adaptation and re-adaptation, the mechanisms of which can be studied in the laboratory using the sensorimotor learning paradigm (Figure 2).

For example, to study the adaptation of a target acquisition gesture, this paradigm consists of:
(1) Characterizing the gesture under normal acquisition conditions (baseline);
(3) [sic] Training the subject to perform target acquisitions in the presence of a perturbation of the arm trajectory (cf. Gandolfo et al., 1996) or visual feedback (cf. Fernandes et al., 2012) so that the subject modifies their gesture control to reach the target by compensating for the perturbation (training);
(3) Testing whether the training resulted in learning by studying (a) the same gesture after removing the perturbation (post) or (b) a different gesture or context, which is referred to as transfer. 

Studying transfer allows us to examine the nature of control mechanisms and their relationship with experience (in the sense of "lived experience"). If modifying a gesture through sensorimotor learning has consequences for the execution of another gesture, we can assume that both gestures rely, at least in part, on shared control mechanisms, and vice versa. Currently, the nature of the representations and mechanisms underlying the motor control of limbs or speech remains an open question. The main challenge is to understand how the brain generalizes action while remaining capable of integrating contextual specificities (cf. Taylor et al., 2013). In particular, some recent studies suggest that daily computer use increases the capacity for generalizing visuomotor learning (cf. Wei et al., 2014). 

Issues and objectives

Learning a new interaction modality requires mobilizing existing sensorimotor skills and/or developing new ones. Users demonstrate a high capacity to adapt to interaction tools. However, not all interaction modalities are equivalent in terms of performance, nor in their relationship to the gestures developed for interacting with physical objects. In this project, we propose using the sensorimotor learning paradigm to study the similarity between gestures made on physical objects and those made on digital objects. This approach should, in particular, pave the way for an objective assessment of the "naturalness" of an interaction. To achieve this, we propose quantifying the transfer effects of motor skills in a task performed with different interaction modalities involving physical versus digital objects by adapting the sensorimotor learning paradigm.

Publications

Bérard F. & Rochet-Capellan, A. (2015). The Transfer of Learning as HCI Similarity: Towards an Objective Assessment of the Sensory-Motor Basis of Naturalness, ACM CHI'15 (Computer Human Interaction, acceptance rate ~23%)

English abstract: Human-computer interaction should be natural. However, the notion of natural is questioned due to a lack of theoretical background and methods to objectively measure the naturalness of an HCI. A frequently cited aspect of natural HCIs is their ability to benefit from knowledge and skills that users develop in their interaction with the real (non-digital) world. Among these skills, sensory-motor abilities are essential to operate many HCIs. This suggests that the transfer of these abilities between physical and digital interactions could be used as an experimental tool to assess the sensory-motor similarity between interactions, and could be considered as an objective measurement of the sensory-motor grounding of naturalness. In this framework, we introduce a new experimental paradigm inspired by motor learning research to assess sensory-motor similarity, as revealed by the transfer of learning. We tested this paradigm in an empirical study to question the naturalness of three HCIs: direct-touch, mouse pointing and absolute indirect-touch. The study revealed how skill learning transfers from these three digital interactions towards an equivalent physical interaction. We observed strong transfer of skill between direct-touch and physical interaction, but no transfer from the other two interactions. This work provides a first objective assessment of the sensory-motor basis of direct-touch naturalness, and a new empirical path to question HCI similarity and naturalness.

Partners, collaborations

Amélie Rochet-Capellan, PCMD team, Gipsa-lab

The work of the PCMD team at Gipsa-lab focuses in particular on the relationships between sensorimotor functions and cognition, examining the involvement of orofacial and manual sensorimotor mechanisms in shaping language and communication. Strong theoretical and experimental bridges also exist between the study of manual and orofacial gestures (Grimme et al., 2010; Perrier, 2012). Thus, the sensorimotor learning paradigm initially developed for studying limb movements has been adapted to study speech motor control (Houde and Jordan, 1998; Tremblay et al., 2003), also to specifically investigate the transfer of learning (Rochet-Capellan and Ostry, 2011; Rochet-Capellan et al., 2012). HCI offers particularly interesting means for studying motor control because the technologies it develops make it possible to create new sensorimotor learning situations while controlling numerous parameters.

François Bérard, IIHM team, LIG

The Human-Computer Interaction Engineering (IIHM) team at the LIG laboratory designs and evaluates new forms of interaction aimed primarily at improving user performance and reducing the learning curve for system use (Bérard, 2003). Measuring user performance is an essential tool for comparing different interactions, but this measurement provides only a very limited view of how well an interaction suits users (Bérard et al., 2011, 2012). The evolution of research towards touch pointing and augmented reality is directing HCI work toward gestural interfaces and the use of motion tracking devices. A better understanding of movement control mechanisms and their evaluation is therefore a key challenge that requires expertise in cognitive science and motor control.

References

BERARD, F. The magic table: Computer-vision based augmentation of a whiteboard for creative meetings. In IEEE International Workshop on Projector-Camera Systems (PROCAMS) (2003).

BERARD, F., AND ROCHET-CAPELLAN, A. Measuring the linear and rotational user precision in touch pointing. In ACM International Conference on Interactive Tabletops and Surfaces (ITS) (2012), ACM, pp. 183–192.

BERARD, F., WANG, G., AND COOPERSTOCK, J. R. On the limits of the human motor control precision: the search for a device’s human resolution. In IFIP Conference on Human-Computer Interaction, INTERACT (2011), Springer-Verlag, pp. 107–122.

FERNANDES, H. L., STEVENSON, I. H., AND KORDING, K. P. Generalization of stochastic visuomotor rotations. PloS one 7, 8 (2012), e43016.

GANDOLFO, F., MUSSA-IVALDI, F., AND BIZZI, E. Motor learning by field approximation. Proceedings of the National Academy of Sciences 93, 9 (1996), 3843–3846.

GRIMME, B., FUCHS, S., PERRIER, P., SCHONER, G., ET AL. Limb versus speech motor control: A conceptual review. Motor control 15, 1 (2011), 5–33.

HOUDE, J. F., AND JORDAN, M. I. Sensorimotor adaptation in speech production. Science 279, 5354 (1998), 1213–1216. NORMAN, D. A. The way i see it: Natural user interfaces are not natural. interactions 17, 3 (2010), 6–10.

PERRIER, P. Gesture planning integrating knowledge of the motor plant’s dynamics: A literature review from motor control and speech motor control. Speech Planning and Dynamics (2012), 191–238.

ROCHET-CAPELLAN, A., AND OSTRY, D. J. Simultaneous acquisition of multiple auditory–motor transformations in speech. The Journal of Neuroscience 31, 7 (2011), 2657–2662.

ROCHET-CAPELLAN, A., RICHER, L., AND OSTRY, D. J. Nonhomogeneous transfer reveals specificity in speech motor learning. Journal of Neurophysiology 107, 6 (2012), 1711–1717.

TAYLOR, J. A., AND IVRY, R. B. Context-dependent generalization. Frontiers in human neuroscience 7 (2013).

TREMBLAY, S., SHILLER, D. M., AND OSTRY, D. J. Somatosensory basis of speech production. Nature 423, 6942 (2003), 866–869.

WEI, K., YAN, X., KONG, G., YIN, C., ZHANG, F., WANG, Q., AND KORDING, K. P. Computer use changes generalization of movement learning. Current Biology 24, 1 (2014), 82–85.

WIGDOR, D., AND WIXON, D. Brave NUI world : designing natural user interfaces for touch and gesture. Elsevier, 2011.