Keywords

1 Introduction

Over the last few years, the emerging technologies for full-body interactive experiences and games have grown. Technologies that support full-body interaction can generally categorized based on our sensations such as vision, audio, and haptic. In addition to technologies for vision or audio, new mechanisms have been proposed to achieve different types of haptic feedback. Therefore, the integration of technologies for full-body interaction requires various types of hardware and software. These range from the HTC Vive [8] for immersive experience to the Nintendo Switch [9], which displays vision on screen for multiplayer, and include systems that aim at providing specific exergames, enabling the engagement of friends, and instructing a group of elders to execute simple movements to maintain their health or rehabilitate them.

A well-designed virtual environment is capable of providing users with not only the short-term emotional impression, but also an immersive experience related to personal [6] and cultural experience as well, even when they only receive feedback in vision [2]. In this study, we propose virtual kayaking experience (VKE), an effort-saving and safe system that provides an immersive first-person kayaking experience (Fig. 1). VKE enhances the sense of presence of virtual kayaking experience by importing real scenes in the vision of users, and it augments the sense of immersion by applying a haptic device that can provide full-body force feedback. Although there are several kayaking games and arcade games nowadays, we aim to realize an advanced virtual kayaking experience by augmenting multimodal feedback and evoking emotional connection by linking traditional spirit. Hence, we designed a visual and haptic feedback system focusing on the essential elements of kayaking activity within the” coming-of-sixteen” (COS) ritual, a local rite of passage that derives from the Tainan city of Taiwan [17].

Fig. 1.
figure 1

Virtual kayaking experience (VKE) allows user paddling in virtual environment.

Full size image

In this study, we provide three main contributions: (1) a proposed design process to enable kayaking experience in a virtual environment, called VKE; (2) a description of the implementation of the VKE system; and (3) two formative user studies to evaluate the potential of the VKE system for the rehabilitation of elders or as a COS archive approach.

2 Related Work

2.1 Full-Body Interaction in Games and VR Experience

Previous works show that the use of full-body interaction in games and VR experience undeniably benefits many aspects of rehabilitation. First, full-body interaction in games is incipiently proved to allow users to have an advanced experience in the virtual environment [10], effectively enhancing the sense of immersion and increasing the motivation of patients to perform rehabilitation exercise [7, 11]. Second, one of the advantages of employing virtual reality related technologies is that they can accurately instruct users to perform correct movements of health promotion related exercise [12], and moreover visualize the skeleton of users to remind them to maintain specific postures for core muscle training [1]. Last, through positioning technology, it takes no effort to collect the movement data of users [13], and even physical and cognitive hybrid activities data when they are playing games [14]. Consequently, there is great potential in applying full-body interaction to elderly rehabilitation.

Multimodal feedback is a critical factor within full-body interaction. Apart from visual feedback, haptic feedback plays an important role in enhancing the sense of immersion. As a unique approach of providing haptic feedback, some research use propellers or rotors to generate force. Thor’s Hammer [19] consists of six motors and propellers in a cube to generate 3-DOF force feedback. LevioPole [4] allows users to sense full-body scale kinesthesia by connecting two rotor units using a rod. These approaches can be integrated into full-body interaction in VR. For example, the virtual super-leaping [3] experience, an immersive virtual experience that provides the feeling of extreme jumping in the sky, uses the method used in LevioPole.

2.2 VR for Culture-Based Experience

VR has great potential for comprehensively presenting nonverbal elements within a scenario, and hence, it is an appropriate story-telling technology to realize experience with abundant contexts, such as sightseeing [21] and museum visits [22]. In addition, a virtual environment provides opportunities to make mistakes without taking any risk in some situations. One such scenario is surgical education [23]. Furthermore, the competence of users can be automatically recorded and measured as data using the VR software [24] for further evaluation. According to these cases, attractive elements exist in high-risk cultural activities, and extreme sports can be abstracted and recompose into safer VR games.

3 Crystallized in Virtual Kayaking Experience

In this section, we thoroughly explain how kayaking is selected as the representative activity of the COS tradition of the Tainan city, and, how we map our sensations to decide the architecture of the VKE system. Finally, appropriate technologies are used to implement the system.

3.1 Cultural Elements Analysis

We applied a cultural design model [5] to decompose the COS culture of the Tainan city. Figure 2 shows the results of our analysis.

Fig. 2.
figure 2

Analysis of coming-of-sixteen (COS) culture

Full size image

The harbor lifestyle of Tainan in the past is believed to give impetus to the COS ritual [18]. Because of the prosperity of the harbor business in the past, child labor was recruited with half pay. Nevertheless, the employer should provide full pay to children once they are 16 years old. Hence, this ritual is important with respect to both spiritual meaning and reality aspects. There are many social expectations within the COS ritual, such as one should pursue their goals with perseverance. Thus, the Tainan government is highly proactive to integrate different activities with this ritual to enrich the cultural deposit. For instance, finishing kayaking in the Tainan canal of 10 km [15, 16] can prove that participants are tough enough to conquer challenges in the future.

3.2 Design of Virtual Kayaking Experience System

At the first stage, we aim at realizing kayaking activity in a virtual environment. Therefore, the main challenge is how to kayak while still escaping from limitations of weather and canal conditions. Thus, because of the development of advanced technologies to augment our perceptions in a virtual environment, we decided to adopt VR related techniques to build a VKE system. Furthermore, to provide this multimodality somatic experience into virtual reality, we focus on three key perceptions: vision, haptic, and sense of balance as shown in Fig. 3.

Fig. 3.
figure 3

(a) Some key sensations from real kayaking, and mapping each sensation to feedback elements such as (b) 360-degree vision, (c) resistance of water, and (d) waves of the boat. (e) VKE includes these elements and provides feedback to users using VR techniques.

Full size image

3.3 Implementation

As shown in Fig. 4, the overall architecture of the proposed VKE system can be generally categorized into hardware and software sections.

Fig. 4.
figure 4

Components of VKE system

Full size image

To allow users to intuitively manipulate the rod-like controller and engage bystanders in the VKE system, we utilize the form variation approach to explore the feasible structure of the physical device after sorting critical parameters of the kayaking activity. The vision while kayaking is implemented via the Unity program. Next, a linear force feedback device and a rocking chair are applied as somatic props to increase the sense of immersion, and eventually integrated into the Unity program.

Vision Making.

First, to present the real canal scene of Tainan to users, a drone (DJI Mavic Pro) and a 360-degree camera (Insta360 ONE X) were applied to video shooting. Figure 5(a) shows the actual setup of video shooting. The reason why we did not use a remote control boat for shooting is because of the consistency of perceptions. Because the effects of rippling and splashing while paddling must be consistent with the movements of users, we prefer a still water surface rather than a waving surface for easier adjustment of the consistency of vision and movement (Fig. 5(b)). For haptic feedback, a handheld device is employed to provide full-body linear force feedback and interaction in the virtual kayaking activity. We shot 360-degree videos that are arranged and handled using the Unity software, as shown in Fig. 5(c).

Fig. 5.
figure 5

(a) 360-degree camera attached above the drone for video shooting. (b) Drone flying 80 cm above the surface of the river. (c) Unity software handling 360-degree videos for the content

Full size image

Haptic Devices.

This rod-like device consists of two units Fig. 6(a) shows the device components such as propellers, motors, and speed controllers, which are the same as previous work [3]. Benefited from this mechanical design, the handheld device serves as a paddle-like controller in this virtual kayaking experience and provides linear forces to simulate force feedback of paddling. Among the sense of balance, proprioception is the sensation of awareness about an ambient environment, which means the simulation of the unstable surface of the canal is needed for an immersive experience. Therefore, a human-scale interaction rocking chair was designed to create the feeling of unstable waves.

Fig. 6.
figure 6

(a) Size of haptic device. (b) Electric components. (c) Size of rocking chair

Full size image

4 Formative User Study

The research group has conducted two informal user studies to record and observe how people interact with VKE system (Fig. 7). The first is a public demonstration that is open to any participant to experience it. Next, 9 elderly people with an average age of 72.5 years are recruited to join the second user testing.

Fig. 7.
figure 7

Setup overview of VKE system for user study

Full size image

4.1 Public Demonstration

We demonstrated the VKE system at a Taiwan domestic exhibition for a day and had 30 participants experienced it, as shown in Fig. 8. In this event, users were briefly instructed about the experiencing sequences and how to manipulate the haptic device before starting the VKE system. At the beginning stage, the users were explicitly informed to start the experience after the propellers were activated. Next, user should make sure that the paddle is inserted into water to push the water backward to move forward. Based on the on-site observations, three noteworthy behavior patterns were summarized: (1) users from the Tainan city curiously look around from the viewpoint on virtual canoe although they are familiar with this area; (2) some users try to move backwards by reverse stroke; (3) bystanders can interact with user through appropriate human-scale hardware design.

Fig. 8.
figure 8

Bystanders interact with user through human-scale hardware.

Full size image

4.2 User Study of Elderly People

Participants.

Nine participants with an average age of 72.5 years were recruited through friends (three males aged 72–86 years, and six females aged 66–73 years). All of these participants had no experience of using VR and had not previously used force feedback devices. Furthermore, all participants were born residents of the Tainan city, but never attended any kayak activity that was held by the local government. Moreover, all the elderly participants were first timers of paddling or kayaking.

Procedure.

Before starting to experience the VKE system, a staff member demonstrates the whole process from wearing the head mounted display to kayaking till the end of experience. Next, the participants are guided one after the other to go through the whole virtual kayaking experience. Finally, a semi-structured interview is conducted to gather feedback from them (Fig. 9).

Fig. 9.
figure 9

(a) Explaining experiment process before conducting formative user study. (b) A researcher demonstrating the whole experience process. (c) Guiding participant to adjust grip width and correcting her posture.

Full size image

Findings.

Three interesting findings were observed from the user study of the elderly people: (1) Cultural Engagement from Local Scene. It was an active atmosphere of discussing the history of the real scene in Tainan while one experienced VKE (Fig. 10(c)). Participants shared their personal memories related to what they saw with others; (2) Priorities of Multimodal Feedback. We found that all participants were focused on what they saw rather than how long they repeatedly paddled; (3) Various Types of Paddling Behavior. An important issue in the VKE interaction process is how a user paddles using a hand-held haptic device. To observe the real behavior patterns of the user, we do not restrict the frequency of paddling, direction, and any other way of using the haptic device (Fig. 10(a)). Therefore, we found that most of the elderly participants had incoordination problem while paddling (Fig. 10(b)). Therefore, it is necessary to fix the hand-held device if the users are required to complete a certain movement perfectly in the rehabilitation scenario.

Fig. 10.
figure 10

(a) Participant kayaking slowly. (b) Participant having incoordination problem while paddling. (c) Bystanders discussing the historical context of this scene.

Full size image

5 Discussion and Implications

5.1 Virtual Kayaking vs. Kayaking

The focus of our first prototype was not only to simulate real kayaking movements as much the same as possible, but also to explore the feasibility of applying immersive exergames to enhance the motivation of doing exercise with repeated movements. Hence, there is still scope of improvement in the paddling movements design. For example, the proposed VKE system only offers forward paddling feedback, but real kayaking is composed of various movements.

5.2 VKE for Rehabilitation

Our findings from user study indicate some further issues in three aspects of the VKE system, which are usability, engagement, and feasibility of applying VKE in the rehabilitation scenario.

In terms of usability, first, we designed a standard posture for VKE, as shown in Fig. 11(a). However, users often rotate the haptic device in the wrong direction or grip an inappropriate width of the rod because they are not familiar with kayaking (Fig. 11(b)). Although we already showed users hints in their vision, the results suggest that we need to restrict the way they hold the haptic device by adding a tangible handle on the rod.

Fig. 11.
figure 11

(a) Standard posture of paddling. (b) Gripping of the narrow width. (c) Shrugging posture of the participant

Full size image

In terms of the user engagement, we found that participants paid almost all of their attention to experiencing content, and hence, they often forgot how long they were experiencing it. The elderly participants engaging in our user study showed high acceptability and interests towards our VKE system, which is a positive response to feasibility evaluation. On the other hand, some elders had vision problems, and different types of media to represent vision content are necessary.

In terms of the feasibility of the application, we considered the nature posture of users during the experience. Apart from gripping the rod in wrong direction, participants also shrug while kayaking, which affected the posture and has to be avoided during the experience, as shown in Fig. 11(c). The shrug problem could be solved by placing a stand to support the center of mass of the haptic device. As for the rocking chair, it could not swing naturally because users are used to step on the ground while sitting on the chair. Hence, in future, a pair of the pedal is necessary to better present sense of proprioception.

5.3 Extendibility of the VKE System

One of the purposes of our study was to explore how bystanders interact with users through the design of the VKE system. Hence, we decided to film a well-known local scene as the visual content of the VKE system in an attempt to evoke a cultural connection between users and bystanders. Furthermore, the handle of the rocking chair is designed for bystanders to shake the chair (Fig. 12). Through the experiencing process, a culture-related discussion continuously popped up, which is a good phenomenon for occupational therapists while doing reminiscence therapy with a group of elders. We optimistically anticipate the somatic experience with cultural contents, and exergames can further improve the qualities of both physical rehabilitation and fulfill social needs of the elderly. Moreover, based on our user study, we found that compared to the traditional archiving approaches, such as sound and video files, the integration of surround vision and behavior patterns shifts from passive preservation to active promotion of intangible culture.

Fig. 12.
figure 12

Rocking chair of VKE system creating another interaction pattern between bystanders and a user.

Full size image

6 Conclusion

In this study, we proposed VKE that allows users to experience immersive kayaking activity in VR. To abstract the determining factors of real kayaking experience, we decomposed our body feedback of kayaking based on different sensations. We then integrated different technologies in a virtual environment according to selected multimodal feedback, including vision, hearing, haptics, and proprioception. Therefore, this VKE system was accomplished. The VKE system consists of an HMD-based visual feedback and a haptic device that is constructed using two rotor units and a rod to provide full-body scale kinesthesia. Besides, a rocking chair serves as a simulator of waves to provide the sense of proprioception. We described the reasons behind the decision making of the representative activities within the COS tradition to fit properly in the historic context. Furthermore, the technical details of the system were reported to articulate how we achieved our system design. We conducted two formative user studies to evaluate the feedback of users after experiencing the VKE system and observe the behavior pattern of elderly people who are the future target group of the VKE system. The findings were reported and some phenomena from observation were discussed. In future work, we propose some perspectives for improvements: (1) utilize computer graphics-related techniques to enrich the visual content for the historical perspective to evoke more emotional connections; (2) to simplify the manipulation process, the haptic device and rocking chair can be integrated into an arcade game machine form. On the other hand, as for the elderly people, the fixed position of the seat and handle of the haptic device help them to maintain the correct posture while experiencing, such that they can perform the designed rehabilitation movements much more precisely; (3) some elders have vision or hearing problems. Hence, different medias of presenting vision and hearing content are necessary.