Exploring Mobile Devices as Haptic Interfaces for Mixed Reality

While the visual and auditory capabilities of MR systems continue to advance [21, 61], the primary mode of interacting with virtual objects still relies on handheld controllers with traditional button types and with vibration as limited haptic feedback. Research proposed various novel systems with diverse form factors that extend the quality and range of possible inputs and feedback, such as handheld and wearable props, encounter-type interfaces, and the use of ultrasound for providing haptic rendering to mid-air gestures (see [7, 35, 39, 53] for detailed reviews). However, with the prevalence of standalone MR headsets shifting the interaction with virtual and augmented objects towards more mobile settings, these devices impose several limitations, including their bulkiness, the requirements for grounding, or simply being cumbersome to carry around. Consequently, research began to explore and advance the capabilities of ubiquitous devices we already carry with us for input sensing and feedback in MR, including smartphones, tablets, or smartwatches. Interactions often go beyond the known touch-based gestures and employ dexterous device-gestures. For example, prior works revealed the capacity of smartphones to sense various squeezing gestures [57] and hand postures [6, 15], 3D manipulations of the phone [56], as well as the orientation [52] and pressure [4] of individual fingers, illustrating the capabilities of existing mobile devices for sensing and simulating haptic experiences in MR.

The potential of ubiquitous, everyday devices such as smartphones to serve as substitutes for tracked controllers has been also investigated in both augmented and virtual reality settings. Previous work employed a wristband-like form factor and attached a mobile phone on users’ arms to provide visual display and enable spatial tracking of the hand [37].Samimi and Teather [44] improved menu selection in virtual reality (VR) using a smartphone’s multi-finger touch feature. Work by Surale et al. [49] explored the design space regarding the use of a tablet’s screen for 3D object manipulation in VR. Dong et al. [12] investigated the use of surface and 3D motion gestures with smartphones for input in AR, and Kari and Holz [20] recently showed the capacity of smartphones for effective bimanual input for VR in mobile or space-constrained settings. However, transitioning the interaction from a 3D virtual or augmented environment to touchscreen devices with distinct characteristics necessitates adaption. The question of how users would adjust their active object exploration to these devices and how they would like the devices to react and feel remains open.

Haptic rendering capabilities of commercial mobile devices are limited to vibrations, that are traditionally employed for notifications [38, 42] or haptic augmentation of mobile game effects [50].To enhance the interaction through more elaborated vibrations, a variety of easy-to-understand meaningful vibration patterns were designed [55] or a more usable and secure user authentication on an Apple Ipod Touch was investigated [2]. Previous work has also explored a variety of haptic mechanisms to equip these small and thin devices with more diverse haptic sensations, such as augmenting the device’s flat surfaces and edges. For example, Jang et al. augmented a mobile device’s edge with an actuated pin array that can change in terms of shape, size, and location to provide additional physical affordances during the interaction Jang et al. [19]. A simpler approach was used by Maiero et al. [29] , who integrated one movable pin into a custom 3D-printed case to explore force feedback at the back of a smartphone and augment the touch interactions on the front screen. Another approach of increasing the interactivity of the mobile phone’s edge incorporated button that was being operated with the thumb and that provided lateral skin stretch for interaction such as scrolling Luk et al. [28]. More recent work by Shultz and Harrison [45] pushed the dimensions of interactive surfaces and proposed a novel miniature shape-changing display that augments flat surfaces with rising and retracting bumps. Their flat panel haptic display is capable of simulating pop-up interfaces, providing pixelated displays, or being used beneath thin visual displays, such as smartphones, to create a deformable flat, pop-out, or sucked-in keyboards.

The exploration of altering the overall shape was thoroughly examined using various technologies.A deformation of the device’s corners was proposed by Rasmussen et al. [40]. Their flexible and shape-changing artefact "ReFlex" allows users to directly control the shape of a smartphone-like object. Their handheld prototype is equipped with servomotors at each corner, enabling it to lift both its corners. While Dimitriadis and Alexander [11] also bent the corners of their mobile phone prototype, they employed a rigid form factor that comprised movable parts. Moving the top cover upwards, e.g., allows the device to expand its volume.The usage of servo motors and pulse sensations in different work allowed the adaptation of the device’s physical appearance to simulate permanent tactile life-like actuation, e.g., breathing of the phone [17]. Later work of the same author attached a mobile phone to a box containing a wobbling plate to provide haptic sensations of tapering for, e.g. notifications [18]. Building upon the same concept of life-like devices, various materials were explored to animate the overall shape of a handheld mobile device to enable it to curl or crawl, mimicking evocative life-like gestures [10].

While prior research has introduced a range of haptic interfaces in mobile devices, their capacity for haptic feedback is often limited to a single sensation, e.g. perceiving changes in the overall shape or deformations on the surface. To deliver a more comprehensive haptic experience of virtual objects in MR, it is crucial to generate diverse stimuli, encompassing factors like the object’s temperature, weight or hardness. Such multi-haptic interfaces provide multiple simultaneous haptic sensations in a system. This approach allows for a more diverse range of haptic cues and has been employed in previous haptics research for multi-sensory experiences, e.g. [1, 27]. Understanding distinct haptic feedbacks users associate with an object’s haptic attributes is, therefore, part of this work’s goal.

In summary, previous research proposed various interaction methods with virtual objects for mixed reality. While our review highlights research efforts to expand input and feedback mechanisms, including handheld devices, wearable devices, and ultrasound for mid-air haptic rendering, MR controllers with vibration feedback are still the primary mode of interaction in MR. The move towards mobile, standalone MR experiences in public settings has prompted research into utilising everyday devices such as smartphones or wristbands for input sensing and haptic rendering. However, to date, there is limited research on how users can, and more importantly desire to, explore virtual objects and their properties in MR. As reviewed in Section 2.2, there is a plethora of research that investigated novel prototypes for haptic interactions and how existing form factors (e.g., those of smartphones) can be enhanced. However, what is missing is a comprehensive exploration of the interaction and design space of mobile devices when used for gestural object exploration in MR. Additionally, understanding how users’ needs and preferences influence the future design of mobile devices for haptic exploration in MR, especially when used in MR [16, 20], remains an important element to address.

After preparing the study material, we deployed a within-subjects online survey to understand people’s perceptions of the six haptic properties of each candidate object. These properties include size, shape, texture, hardness, temperature, and weight, which cover the six most important characteristics when identifying objects [22]. Each property manifests two characteristics on the level of each property. For example, for the property size, participants could rate on a continuous 0-to-100 scale from small (0) to large (100) using a slider. We then calculated means to identify two objects each at both ends of each property continuum. For example, if one object received the lowest mean score across all objects for size, we considered this object to be a suitable candidate for representing a “small object” in our follow-up gesture elicitation study. The same was done for the highest mean score, representing a suitable candidate for, e.g. a “large object”.

Fifty participants n = 50 (age ranging from 20 to 59, M = 28.65, SD = 8.41) were recruited using Prolific, an online crowd-sourcing platform that has found widespread application for empirical HCI research, e.g. [33, 36]. The study was distributed evenly to male and female participants using Prolific’s internal distribution feature. We added three attention-check questions to ensure data quality and excluded survey takers who clearly did not put in significant effort in filling out the survey. To further contribute towards high data quality, we added a 5-point Likert Scale asking how genuinely the participants filled in the survey. This was done after four initial pilot tests. We emphasised that the answer to this question has no impact on their reward. Overall, the online survey was scheduled for 30 minutes and eventually manifested a median of 34.35 minutes. None of the survey takers had to be excluded for the data analysis based on the attention checks, and the participants’ commitment was M = 4.89 (SD = 0.31). Participants received an average reward of £6,01/hr using Prolific’s reimbursement system.

After an initial screening by the ethics board of our University of St. Gallen, our research was exempt from a formal full review. All participants signed up for the study directly on Prolific and were then forwarded to our survey, which was hosted on SoSci Survey (https://www.soscisurvey.de/), a professional survey platform that allows designing and hosting surveys. Participants then provided informed consent of their participation and went through an example object that introduced them to their task (i.e., “In this survey, you will see images of objects and rate their properties. A short example at the beginning will illustrate the task.”). After going through the example, participants were asked to indicate their expectations of the size, shape, texture, temperature, weight, and stiffness of 100 objects using sliders from 0 to 100, as described in Section 3.1.

To identify objects that cover both ends of the category continuum (e.g., small size and large size), we normalised the dataset and calculated means and standard deviations (SDs) of participants’ indicated expectations of all six haptic properties of each object. This process identified the 12 objects that cover all properties (i.e., size, shape, texture, stiffness, temperature, weight) and all characteristics (e.g., small size or large size). For example, for size, the paper clip object was perceived as the smallest object and the umbrella object as the largest object out of all 100 objects by Klatzky et al. [22]. Table 1 shows the identified objects at both ends of the individual continua, including the means and the SDs from participants’ ratings. Overall, all participants rated their expected quality of a specific haptic attribute consistently in the online survey. In particular, while the visually displayed object sizes in the 2D copies are not in a realistic proportion, all size ratings across objects and participants show a low variability among the individual SDs with an average of 0.09 (SD = 0.06). This indicates that participants did not rate an object’s size as visually displayed in the 2D copy, but the expected actual object size, as intended by the study task.

As a next step, we then integrated the 12 objects in our haptic exploration study to investigate how people would elicit haptic sensations when interacting with them in MR. While our objects cover various object categories and properties (e.g., low temperature and high temperate) and are informed by previous user-centred object recognition research [22], we want to emphasise that our candidates do not represent a non-exhaustive list of possible MR objects; however, we aimed at identifying objects that cover a broad spectrum of characteristics in an initial step to then investigate possible related gestures for exploration, which we study in more depth in our gesture elicitation study.

We employed an exploratory study design with a within-subject design and a thinking-aloud protocol, similar to previous gesture elicitation studies [5, 12, 54]. This approach allowed us to capture participants’ intuitive hand movements when presented with a haptic exploration task involving a mobile device. In addition, participants verbally expressed ideas and explanations regarding their envisioned haptic sensations providing a deeper understanding of the meaningful haptic renderings for future mobile MR.

During the study task, participants observed inside an MR HMD a virtual object placed within the AR passthrough (see Figure 2 (a)). We asked participants to invent and perform a gesture, or multiple ones, involving a super-sensing artefact to explore a specific haptic attribute. The super-sensing artefact served as a physical representation of a future mobile device for MR and was incorporated into participants’ interactions and imaginations for haptic feedback. Based on the selection from the online survey, 12 objects in total were presented, with a counterbalanced order. Both objects representing the min and max variant of a specific haptic attribute were always shown in a consecutive and counterbalanced order. The process of gesture invention was guided by predetermined and ad-hoc questions ensuring a deeper understanding of participants’ verbally expressed design choices and consisted of two steps:

Step 1: Natural object exploration. We asked participants to visually inspect a particular haptic attribute, e.g. the texture of the virtual object in front of them and describe their expected haptic sensation upon touch: By just looking at the object, what would you expect the [haptic attribute] to feel? We then instructed them to use their hands and explore the haptic attribute to confirm their previously described expectation: What would you do with your hand to confirm your just mentioned expectation?, Can you explain what you are doing and why? and What would you expect to feel with your hand while performing the interaction? The purpose of this first step was to support participants in establishing an understanding of the involved haptic sensations and signals derived through haptic touch exploration. Additionally, the performed hand movements served as a starting point for the subsequent gesture adaption.

Step 2: Object exploration through tangible artefact interaction. Participants were asked to design an interaction with the super-sensing artefact that allowed them to experience the haptic sensation they explored in step 1. We guided participants to voice their thoughts by asking the same questions as in step 1 while adding a question specific to the artefact interaction: Can you explain how and why you adapted your hand interaction for the device?

After an initial screening by the ethics board of our University of St. Gallen, our research was exempt from a formal full review. Participants took part in the study voluntarily and did not receive monetary compensation. The study was conducted at two semi-public co-working spaces (see Figure 2 (b) for one of the spaces). At the beginning of the study, participants took a seat at a desk and were informed about the study’s purpose. Participants provided their consent and completed a demographic questionnaire. The study task was then explained to them and the mobile super-sensing artefact was introduced. The device was characterised as having limitless capabilities in sensing and providing haptic feedback, being capable of processing any imaginative request from participants. To ensure a shared understanding across all participants, we provided the following exemplary capabilities: the ability to detect touch, touch location, pressure, orientation, spatial position, and key pressed on physical buttons. After addressing any remaining questions, participants were equipped with the Meta Quest Pro HMD with enabled passthrough. The super-sensing artefact was positioned on the table, easily reachable for participants for step 2 of the gesture invention process. The study conductor then guided participants through the exploration of all 12 virtual objects, asking both predetermined questions and additional inquiries in response to participants’ descriptions. Upon completing all objects, the study concluded with a short semi-structured interview. The study lasted on average around 1 hour.

The Meta Quest Pro and its AR feature were used for displaying the virtual objects in front of the AR passthrough (see Figure 2 (a)). The HMD’s hand-tracking feature additionally displayed users’ hands. The study application was implemented in Unity 2022.3.4f1 and the HMD was connected via cable to the computer during the study. A Google Pixel 7, turned off, was used as the super-sensing artefact.

To capture participants’ hand movements, we employed a two-camera setup. The first camera was placed on a tripod and recorded from an overhead and right-side perspective. The second camera was on a small tripod on the table and focused on tracking finger movements on the back of the artefact. Nevertheless, videos from the second camera were omitted from the data analysis, given that the recordings from the first camera, in conjunction with participants’ comments, yielded ample information for our purposes. Alongside the video recordings, a wireless RØDE microphone captured participants’ expressed comments. Since the camera’s audio recording was of sufficient quality, we did not use the microphone’s data for the analysis. The consent and demographic questionnaire was hosted on SoSci Survey (https://www.soscisurvey.de/), which participants filled out before the study, either on their private device or a laptop provided by the study conductor.

Due to technical issues, one data set was excluded from the analysis reported in Section 5. Therefore, we will only provide the demographics of our final sample of 17 participants. All participants were recruited using word-of-mouth (6 female, 11 male) and they were aged between 23 and 32 (M=27.47, SD=2.48). All participants were right-handed (n=17) and seven had corrected vision using glasses or contact lenses. When asked for prior VR experience, most participants engaged with VR once or twice in total (n=6), or several times per year (n=7). One participant never used VR before, one used VR several times per month, one used VR several times per week, and one used VR daily. The AR experience was similar, with seven participants each having engaged in AR experiences once or twice in total or several times per year. The remaining three participants use AR several times per month (n = 2) or had never used it before (n = 1).

Participants’ design of gestures for exploring an object’s size was strongly impacted by an object’s large or small dimensions. Often, the large object was spatially "sampled" through motion gesture, and the small object was mapped onto the 2D surface of the artefact. When exploring the large object, the artefact was often used as a poking tool and moved through space to detect the object’s outline. Envisioned haptic feedback upon collision included force feedback, resisting arm movement, or vibration. A variant incorporating both hands "hugged" the object’s outline and involved force feedback at the second hand, which was not holding the artefact or feeling the umbrella’s round outline along the arms. The relative distance between both arms also provided a proprioceptive sensation for the object’s large dimensions. Another way to sense the size of the umbrella involved grasping the artefact as a substitute for the umbrella’s handle and moving it up and down. Participants then described the size sensation as the vertical resistive force coming from the air resistance. A concrete idea for implementing such vertical force feedback utilised magnetic repelling force between the grasping hand and an envisioned second magnet positioned at the top of the virtual umbrella.

In contrast to gestures heavily relying on spatial movement, participants mainly mapped the small virtual paperclip onto the artefact’s 2D surface, focusing the interaction on the exploration of the flat surface. The paperclip’s 2D outline was then examined through a pinching movement, squeezing the artefact’s front and back, guided by resistive force upon outline collision. For a more in-depth exploration, finger-pads either stroked or were pressed onto the surface to sense the size through envisioned surface deformation or additionally attached wire. Nonetheless, the paperclip’s outline was also experienced through the artefact resting inside the palm, with the size outline pressing into the palm’s skin via surface deformation on the artefact’s back. A motion gesture, similar to the exploration of the large umbrella, was also employed for the paperclip. In this context, the grip was restricted to the artefact’s corner to enable a smaller grip pose appropriate for the object while also providing passive haptic feedback from the physical feature.

Haptic knowledge about shape was mainly retrieved through gestures that allowed participants to enclose their hands or fingers around the physical shape of the artefact, serving as a substitute for the virtual object. The simple and straight shape of the toothpick was often mapped onto the edges or flat surface of the artefact. To experience the toothpick’s elongated form, participants commonly pinched either the device’s long edge from top to bottom or rotated the device by 90 degrees to pinch the short edge of the device. To capture the sensation of the toothpick’s sharp ends, a wire was suggested that would come out of both ends of the artefact’s edges. Alternatively, a deformation was described that mimics both cone-shaped ends. The toothpick’s thin diameter was investigated by pinching the round edge, providing passive haptic feedback similar to the object’s original shape. Additional rolling of both fingers then mimicked the natural rolling of the toothpick between the fingers. It was also suggested to incorporate one of the physical buttons on the side instead of the artefact’s whole edge, as the button’s shape is closer to the toothpick’s. In scenarios, in which participants mapped the toothpick onto the flat surface, the index and thumb were in contact with the artefact’s front and back side, feeling the shape through surface deformation recreating the toothpick’s thin diameter. The same surface deformation just on the front side was also explained as a meaningful simulation of the toothpick’s shape, which was then felt through stroking.

For the complex shape of the baseball glove, both hands were often used to grip the artefact from opposing sides. This approach was complemented by a desired shape-changing feature, aiming at enhancing the ability to perceive the object’s volume within both palms. Furthermore, participants applied pressure to the artefact, bending it to gain insights into the shape’s flexibility, conveyed by haptic feedback simulating the level of artefact deformation. Participants were also curious about the glove’s opening for the hand and suggested a gesture for pushing individual fingers through openings in the artefact. "Wearing" the artefact as a glove in this way would then restrict the individual fingers’ movements and convey the arched shape.

Central concepts in the device interaction for exploring the level of hardness involved squeezing and pushing the object, with the primary gestures Grip, Two-handed grip, Pinch and Press (see Figure 3). Grip gestures, one-handed and two-handed, squeezed the artefact’s width and aimed for haptic feedback related to soft or firm device deformation. Additionally, two-handed grip gestures were suggested to bend the artefact and convey a sense of hardness by responding with a break-resistance sensation. Another variant of the two-handed squeezing compressed the virtual object between the user’s second hand and the hand holding the artefact. Participants described a resistive force sensation that would then restrict both hands from approaching each other further and convey a level of softness. In the context of motion gestures, participants gripped the artefact again as a poking tool to push into the virtual object and sense its hardness upon collision through resistive force feedback. Here, participants exhibited different preferences for hand poses while holding the device.

Participants’ investigations of hardness on an object’s local scale were often conducted through a Pinch gesture. During the interaction, participants focused, e.g. on the concept of compressing the thin and pliable sheet of toilet paper. To convey the sheet’s flexibility and deformation, sensing the pushback of the opposing finger was highlighted as an integral component of the tactile encounter, which was complemented by material deformation and ripping sensation. The expected sensation when pressing the artefact’s flat surface with a single finger mostly targeted the feeling of a change in the device material’s softness. Furthermore, proposed gestures utilising the unique tactile qualities of the artefact involved pressing a physical button on the artefact’s side with envisioned adaptive button resistance. Additional ideas for higher hardness involved stroking the edges of the device or its camera component (considerably sticking out from the used Google Pixel 7) to perceive the tactile hardness of the co-located virtual clamp’s edge.

The texture was mainly explored through Pinch and Stroke for which the virtual object was mapped onto the artefact’s surface or other physical features. Notably, these gestures targeted a rather small area of the artefact, highlighting a more local haptic exploration.

Participants frequently engaged in stroke gestures, executing long lateral motions using their index, thumb, or multiple fingers, primarily to explore scratchy surfaces. Their expectations centred around experiencing vibration and surface deformation to convey the granularity of the sandpaper’s texture effectively. Moreover, a unique idea emerged, involving placing an index finger on the back of the device and virtually touching the object. In this scenario, the index finger would sense the device’s scratchiness upon collision. In the case of the sandpaper, Some participants used the device as a tool for scratching the object’s surface, combining grip and move gestures to create desired vibrations and force feedback. Participants also employed pinch gestures, often combined with rolling or stroking motions, to explore the texture. The rice grain was sometimes mapped onto the edge or physical button, to take advantage of their smooth surface as well as of the round shape of the edge and size of the button, which was described to be similar to the rice grain. Pinching the artefact’s flat front and back side was more common for the sandpaper, as participants then expected both sides to respond with different texture sensations, rough on top and smooth on the bottom. Moreover, some participants even suggested substituting parts of the device with small squares of fabric to provide the appropriate sensation.

The prevalent gestures for experiencing weight through vertical force feedback were grip and rocking, forming a large portion of participants’ interactions (see Figure 3). Various techniques for holding the artefact were exhibited, including gripping the device and moving it up and down, gripping and rotating the wrist, holding the thumb onto the screen or pushing a physical button to simulate grasping the string attached to a balloon and moving the device downward. Additionally, a touch-based gesture involved stroking downwards, pulling a virtual string attached to a balloon, which was mapped onto the screen. This gesture allowed participants to perceive the sensation of vertical upward force through their thumb, adding to the diversity of interactions explored.

In a contrasting approach, some participants opted for unsupported holding of the device, effectively using it as a scale and moving it up and down to engage with the vertical force feedback. Similarly, one participant rested the device on a table and placed their palm on it, expecting an upward push from the device to simulate the negative weight of the balloons. For interactions involving the hammer, participants primarily grasped the handle, focusing on sensing the imbalanced weight distribution. An interesting concept was also introduced, wherein a second hand was used to create a magnetic repelling force, pushing down on the hand holding the device to simulate a unique haptic experience.

Participants primarily utilised Hover and Tap to sense the temperature of the virtual objects. In the context of Hover, the radiating heat of the burning candle was mapped onto the artefact and explored by hovering one or two hands above the device. We observed different variations in the artefact’s orientation. In one scenario, the device was placed on a table, emanating the heat from its front side, while in another, the device was held upright, with heat radiating from the top section to mimic the flame. An additional airflow coming out of the top complimented the haptic sensation of the hot flame.

In contrast to hovering, some participants interacted with hot and cold temperatures through contact with the entire artefact by gripping, tapping, pressing, or resting the device in the palm. For the cold key, participants projected the key’s shape onto the flat surface as the only cold area. To provide localised feedback, temperature panels were suggested that are incorporated into the artefact. Another contact-based gesture aimed to replicate the sensation of the highest temperature at the artefact’s top by pinching its upper edge. One participant suggested tapping the flashlight of the artefact as the source of the heat. The lower sections of the artefact’s surface represented the colder wax, which participants explored through stroking and press movements. Additionally, surface deformation was mentioned to simulate the varying surface tensions of wax at different temperatures.

A key design aspect across all gestures is the variation in the positioning of the mobile device relative to the virtual object and the user’s hand. Based on our observations, we identified three distinct modes that the device can adopt during the interaction: (1) object mode, (2) hand mode, and (3) tool mode (see  Figure 1). In the following, we explore each of these roles and examine the impact of object properties on these gesture variations.

Object mode. In the object mode, the device acts as a physical representation of the virtual object. Here, touching the device is interpreted as touching the virtual object. The device remains stationary in front of the user, with the hand holding it. Participants described the object as either co-located with the handheld device or remaining in the augmented environment. In our study, we further observed three variations of the object mode. First, when the virtual object is smaller than the mobile device, the gesture utilises only a part of the device. For instance, when the small paper clip or cold key was mapped onto a partial area of the flat surface, the simple shape of the toothpick was aligned with the device’s edge or the rice’s texture was explored by touching the device’s similarly shaped physical button. The remaining physical components of the device do not serve as interactive elements in the artefact-gesture. Second, when the virtual object is larger than the device, the device depicts only a part of the object that is relevant to the explored haptic attribute. For example, gripping the mobile device in landscape orientation with the entire hand was associated with gripping the middle part of the clamp positioned on the side. Third, the mobile device depicts only the haptic attribute itself and does not portray the object. For example, the tapping of the mobile phone’s flashlight as a heat source of the candle. Here, the phone only conveys the temperature, but not the candle. All three variations of the object mode are not always clearly distinguishable, and sometimes, more than one can be observed. For example, one could argue that the device did not take on the clamp’s shape during the Grip gesture and, therefore, did not accurately assume the part of the object but rather the sensation of the hard material. However, the positioning of the hand mirrors the one that would be used on the virtual clamp. Therefore, we consider the device representing the clamp’s middle part to be more dominant in the gesture design than its representation of the isolated hardness. While the third variation of the object mode of isolating a specific haptic attribute might be influenced by our study task, it provides a valuable approach to simplifying a comprehensive simulation of object properties and conveying only the haptic property most relevant for the MR experience.

Hand mode. In the hand mode, the object takes a physical representation of the user’s hand. As the device is cradled in the open palm with fingers slightly curled to stabilise it, this mode can also be interpreted as extending the inside of the hand and enabling its sensing capabilities towards the virtual object. The envisioned haptic feedback mainly is restricted to the palm. During the object interaction, the device hand moves around the space, approaches the object and establishes physical contact with its outer shape. For example, the softness of the toilet paper was tested through a two-handed grip, with one hand holding the device and pressing against the paper roll. Other examples included using Grip + Move to press down the lightweight balloon or to "sample" the outer boundaries of the large umbrella and feel the collisions inside the palm.

Tool mode. In the tool mode, the device is used as a tool for indirect interaction with the object. The spatial movement of the device during interaction is similar to that in the hand mode and elicits haptic feedback upon colliding with the outer boundaries of the object. However, the device is held more with a harder grip and the haptic sensation focuses strongly on proprioceptive feedback such as force feedback/resistance in mid-air. For example, when poking the clamp with gripping + moving, the collision expressed the hardness of the clamp or the large dimensions of the umbrella.

Form factors of the device in the gesture design. In some instances, participants held the device either in portrait or landscape orientation while performing the same type of gesture, such as during the Pinch gesture to explore the size of the paper clip or to trace the shape of the toothpick along the short or long device edge. However, in these interactions, the device’s orientation appeared to be more of a personal preference rather than a deliberate design decision. Therefore, allowing different device orientations would ensure coverage for users’ individual preferences. The recent reintroduction of foldable mobile phones and tablets in the consumer market, however, will likely impact participants’ choices in gesture design. Unfolded mobile devices offer larger dimensions and an extended flat surface, likely prompting more two-handed types of gestures for larger objects. In contrast, when folded, the devices become compact and small, potentially encouraging gestures that prioritize one-handed gestures and exploration of small and fine-grained haptic properties. To understand the impact of a foldable form factor on gesture design, further research is needed.

To cover the variety of haptic features involved in the object’s tactile perception, participants suggested various hardware mechanisms and concepts. Overall, they can be broadly categorised into active feedback, passive haptics (also established through shape-change or appendages), arm poses for proprioceptive sensation and adding a second mobile device. Vibration as the nowadays most commonly used haptic feedback in consumer MR systems was suggested for only two attributes (size and texture). Going beyond vibration, a plethora of previous works have already explored different approaches to equip an artefact with form factors of a mobile device with enhanced haptic rendering capabilities, as reviewed in Section 2.2. As exhibited by recent work from Shultz and Harrison, a shape-changing display can already render bumps and holes [45], and further advancements might be able to achieve a more fine-grained deformation of the device’s surface to create a local shape outline of a key or to render distinct lines representing the individual wires of a paperclip. An existing method that could create sharper elevations and edges on the small form factor of a mobile device is a pin-array [19]. By repositioning the pins to the mobile device’s edges, users could experience the sharpness of, e.g. a virtual toothpick during a pinch gesture. Other mechanisms could simulate a change in the device’s global shape [40] or the temperature gradient of a virtual candle through heat panels [58]. However, incorporating this variety of hardware components will be highly complex and increase the device’s weight, which goes against the vision of everyday, mobile MR experiences. Considering scalable solutions, a flexible setup that allows users to add or remove elaborated haptic rendering capabilities could be in the form of haptic mobile cases, as done in the work by Maiero et al. [29]. Furthermore, software-based concepts known from VR inducing pseudo-haptic effects such as the manipulation of the control-display ratio [43, 48] or hand-redirection [59] could be explored for haptic MR interaction. Manipulating the movement gain when moving the device up and down could convey a sense of weight, or redirecting arm movements while hugging a large object could induce a proprioceptive sensation of different sizes. However, in contrast to VR, where users only see the manipulated hands, MR displays users’ physical hands, making pseudo-haptic illusions challenging. A solution might be blink-suppressed redirecting [60] combined with manipulating the video passthrough, to shift users’ displayed arms unnoticed during a blink. Overall, future work should explore the design of mobile devices as a multi-haptic interface to enable haptic experiences in mobile MR. While software-based pseudo-haptic approaches offer more flexibility and scalability, their application and perceptual mechanisms in MR have to be further researched.

As we singled out individual haptic attributes for eliciting exploration gestures in our study, we gained an in-depth understanding of the core hand movements and haptic sensations. However, our derived gestures and haptic renderings could be combined and integrated into a multi-haptic sensation to achieve even more realistic and comprehensive haptic experiences of virtual objects. The wish to not only sense one single haptic property was indeed expressed by a small number of participants in our study. For example, weight was reported as a desired indicator for the size of the umbrella or that simulating the metal texture for the hard surface of the clamp could improve the hardness perception. To understand meaningful and appropriate combinations of attributes, gestures and haptic feedback, we can leverage our identified relations between one type of gesture and multiple haptic attributes, illustrated in Table 3. In the following, we illustrate the use of our design space using three concrete applications.

There are a few limitations that are worth discussing. First, we investigated the use of a mobile phone as an initial important step towards haptic mobile MR interactions. We did not explore other form factors of mobile devices, such as tablets or smartwatches, which might impact the design of user-centred gestures and add to our design space. While we acknowledge that our design space for haptic mobile MR interactions is not exhaustive, we do believe it provides a comprehensive and detailed overview of gestures that are employed by users when using their smartphones for object exploration in MR. Second, we explored individual haptic attributes to construct an initial design space of user-centred gestures in MR. Exploring multiple haptic attributes simultaneously might reveal additional gestures, that can further broaden our design space in Figure 3. Nonetheless, as our findings show overlaps among the gestures and targeted object properties, our findings show a multi-attribute coverage of the gesture types. Furthermore, with future applications increasingly taking place in shared public spaces, making mid-air gestures less preferred by users [32, 34, 51], social acceptance becomes a critical consideration in designing haptic MR interactions for mobile contexts [13]. While our participants expressed that the presence of others around them in the semi-public study setup did not impact their gesture design, future studies should explore this aspect in a more controlled setting. Finally, all our participants in the gesture elicitation study were right-handed; therefore, gestures performed by users with their left hand being the dominant hand are not covered in our design space. While we believe this does not significantly impact the design space per se, it remains an unexplored area in our research that surely deserves attention in the future, including people with mixed handedness.