Development of a Reminiscence Therapy System for the Elderly Using the Integration of Virtual Reality and Augmented Reality

Abstract

Through virtual interaction and multivariate communication, the social interaction and life satisfaction of elderly people can be improved. This study integrated virtual reality (VR) and augmented reality (AR) to construct a visualized reminiscence therapy system, which provides not only reminiscence entertainment but may also aid in dementia prevention for healthy and sub-healthy (a therapeutic working concept which defines an intermediate stage between health and disease) elderly people. Images taken by an LG 360 CAM are primarily introduced into Power Director 360 as the guide interface. The special effects are joined using After Effects. With the model constructed under 3ds Max, the reminiscence therapy system integrated by virtual reality and augmented reality is then established by the Unity engine. It is shown that the developed 3D ancient building model can be used by Samsung Note 5 smartphones as well as VR CASE glasses to increase the immersive experience for the elderly. Through nostalgic elements triggered by the AR/VR three-dimensional model and video/audio interaction, the feasibility of our integrated system for reminiscence therapy is thus verified. Through reminiscence therapy, memories and thoughts can fully be activated as a therapy for elderly people. The visualized reality system developed in this study can further promote the social interactional satisfaction of elderly people.

1. Introduction

Following a decline in Taiwan’s birth rate and the evolution of medical technology, the proportion of elderly people in the country has risen annually. In fact, Taiwan has become one of the fastest-growing aged societies worldwide. Consequently, problems related to aged societies have surfaced. For example, cases of aging-related diseases, such as dementia, have increased, causing problems and difficulties for elderly people in their daily lives. In particular, the prevalence of dementia among elderly people may exacerbate Taiwan’s economic burden. Therefore, efforts must be made to prevent and alleviate dementia in elderly people in Taiwan. Numerous studies have reported that reminiscence therapy is highly effective in this regard. Reminiscence therapy is a psychological treatment that can be applied to patients of all ages. Psychologically speaking, elderly people, being aware of their limited life span, may lose interest in their future. Physiologically speaking, the short-term memory of elderly people declines as their long-term memory strengthens relatively. Accordingly, past events often appear clearly in the minds of elderly people, and thus, reminiscence therapy is particularly suitable for them.

Cutler et al. [1] employed a highly interactive mobile platform to enhance the communication abilities and positive emotions of elderly people. Through incorporating digital somatosensory games into reminiscence therapy, the researchers effectively enhanced users’ healthy daily living activities and intergenerational interactions within their families, thereby generating positive perceptions. Jo et al. [2] indicated that reminiscence therapy not only decelerates physiological and psychological deterioration caused by aging but also enables elderly people to engage in interpersonal and social interactions through positive and active approaches. Through doing so, it improves their social behaviors and interpersonal relationships as well as delaying the regression of their cognitive, verbal, and communication skills. Subramaniam et al. [3] indicated that reminiscence therapy not only improves elderly users’ psychological health but also enhances their life satisfaction through interpersonal interactions and experiential exchange through mutual participation and consensus. Fujiwara et al. [4] maintained that the focus group approach adopted in reminiscence therapy enables participating elderly people to improve their social adaptation skills, interaction abilities, social emotions, and interpersonal relationships. Participation in sufficiently frequent and diverse social activities can alleviate loneliness and alienation. Huang et al. [5] explored the applicability of reminiscence therapy for improving cognition and social emotions in elderly people with dementia from the nursing perspective. The results revealed that most reminiscence activities involve guiding elderly people through experiences with nostalgic physical items, images, videos, and music. Through these objects, elderly people can reminisce about past events and share their experiences. In addition, aged care facilities have been decorated to invoke nostalgic scenarios for elderly people. Sellers et al. [6] reported that reminiscence therapy enables elderly people to integrate their life experiences with positive physiological, psychological, and social functions, thereby improving their physiological and psychological health and quality of life.

Following relevant studies, the present study examined the applicability of reminiscence therapy for improving cognition and social emotions in elderly people with dementia using nostalgic physical items, images, a video, and music as assistive devices. Based on the above, this study integrates virtual reality (VR) and augmented reality (AR) techniques to present nostalgic story modes, including nostalgic oldies, family-related nostalgic videos, nostalgic implements, etc., for elderly people to enjoy immersive memories of old days. Primarily, the images captured using an LG 360 CAM were imported into Power Director 360 to design a navigation interface, and subsequently, Adobe After Effects was employed to implement special effects. Next, 3ds Max was used to establish the model, and the video game engine Unity was employed to complete the AR- and VR-based reminiscence therapy system. Thus, this study developed an AR interaction system that is intended to improve life satisfaction for elderly people through social interactions.

2. Research Method

2.1. Reminiscence Therapy

Reminiscence therapy is a common and effective technique involving nursing measures and activities for elderly care. It is focused on reducing patients’ senses of alienation and loneliness, increasing their socialization level, promoting positive feedback, and improving their self-identity [7,8].

Reminiscence therapy is suitable for elderly people because it encourages them to recall or reconstruct their past actions or thoughts. Through reviewing their past life experiences, elderly people can strengthen their self-affirmation and construct self-awareness. Moreover, the therapy assists elderly people to reflect on and learn multiple approaches to solve their current life problems [9]. When therapists perform reminiscence therapy, they can help elderly people to have positive self-experiences by conducting memory recall activities with each individual or a focus group. Furthermore, diverse experiences can be created through small-group interactions, thereby evoking memories of specific topics or times. In summary, reminiscence therapy enables elderly people to share interesting and critical experiences that entail the development of emotional connections as well as to engage in social interactions that enhance their self-worth, thereby improving their life satisfaction.

After researchers confirm their reminiscence activities and implementation, they must proceed to the design of an actual implementation plan [10]. Reminiscence therapy also helps elderly people to reduce boredom and a sense of meaninglessness in their self-perception and interpersonal interactions derived from aging. During therapy, users recall painful life events and experiences of being unfairly treated. By recalling stories from their lives, users can reconstruct their overall awareness of their life’s value, philosophy, and experiences. This process allows elderly people to redevelop their self-care, cognition, and social interaction abilities. Thus, participating elderly people can adopt strategies that they can use solve existing problems and, in turn, improve their self-identity and satisfaction with their psychological health.

In brief, reminiscence therapy, if applied to residents of aged care facilities, can improve the psychological and social interaction skills of elderly people as well as improve their physiological functions and delay their aging process. Based on the aforementioned literature and the categorization of functions in reminiscence therapy proposed by Parker (1995), namely individual and social functions, this study determined that reminiscence therapy has the individual functions of improving communication skills, reducing negative emotions, and maintaining the existing skills of elderly people. These individual functions are definitely worthwhile as therapy for dementia and help elderly people to develop their self-identity. Regarding the social functions of reminiscence therapy, it emphasizes increases in interpersonal interactions, self-identity, and self-affirmation and reduces loneliness, thereby providing social support to elderly people.

2.2. VR

In a VR system, virtual image spaces can be set according to various requirements without limitations on location or time. VR can be used to develop military, educational, and entertainment equipment at a relatively low cost and risk and with high efficiency. Figure 1 presents the three components of a standard interactive VR system, whereas Figure 2 depicts the structure of a VR system comprising the following components: 360 video/audio capture, a VR software system, a VR cloud, a user interface, a scene request, a response evaluation, and media systems that enhance lifelike effects for the user.

Currently, VR is constructed through any of the following three approaches:

• Geometry-based VR: Also known as graphic-based VR, this involves the creation of objects required for a three-dimensional (3D) simulated scenario using 3D model-building software, such as 3ds Max, TrueSpace, AutoCAD, or Superscape. Users can view the scenario through any angle of view and path, and trigger events can be appropriately implemented into model objects for message delivery and trigger interactions. Editing functions in VR software can be used to define the properties of objects as well as to drive specific devices for various levels of interaction.
• Image-based VR: This was introduced to solve problems related to the frame rate in graphic-based VR, which can be exacerbated by bandwidth insufficiency in networks. The earliest manifestation of image-based VR was Quick Time VR, which was developed by Apple using image-processing technology. Quick Time VR was originally designed for Mac, yet was subsequently extended to PC.
• Hybrid VR: This combines the advantages of graphic- and image-based VR, both of which exhibit shortcomings in practical applications, to store images and data for video playback. Hybrid VR may be applicable as a mainstream network VR system in the future. Similar to image-based VR, hybrid VR captures multiple, continuous images through shooting 360° scenes. These images are combined into panoramic images through image processing. Subsequently, objects that did not belong in the original scenario can be added according to user requirements, and the properties of these objects can be established through simple programming.

2.3. AR

AR is a technology derived from VR, and its display modes are depicted in Figure 3. AR requires the use of VR devices to project virtual objects and scenes onto real-world images, thereby causing users to perceive the coexistence of reality and VR [11]. In other words, real-world environments and computer-simulated images are combined to create an alternative reality through virtual media and to facilitate users’ perceptions and operation of virtual 3D objects in the real world [12].

The main point of AR is to integrate 3D virtual objects into real-world environments to strengthen perceived interactions with these environments and to enable users to rapidly acquire useful and relevant information. Virtual objects can reveal information that users cannot perceive with their physical senses, enabling them to smoothly complete tasks in the real world. Unlike VR, which is designed as a substitute for the real world, AR integrates virtual information with the real world. According to Azuma (1997) [14], AR is a variation on VR. VR places users in computer-simulated realities, whereas AR overlaps reality with virtual images presented to users. Therefore, rather than replacing reality, AR merely augments it. In addition, Azuma (1997) stated that there are three necessary characteristics for establishing AR: (1) a combination of virtual and real spaces; (2) instant interaction modes; and (3) 3D spaces involving X, Y, and Z axes.

3. Research Design

In this study, the proposed visualized system utilizes AR as the navigation interface to support VR, which enables elderly people to capture or scan specific images through mobile devices. The captured or scanned images are then synchronized to the AR cloud system. After the AR navigation interface is triggered, the VR scenery is presented to users.

The reminiscence therapy system developed in this study integrates VR and AR. A VR CASE headset was employed alongside a Samsung Note 5 Android phone. Regarding the software used, Photoshop was applied to paste the images, 3ds Max 2018 was used to create 3D scenes involving historic houses, and V-Ray plugins were employed to compute a 360° video (Figure 4). Subsequently, the AR cloud image recognition function on the Vuforia website was combined with the cloud VR system on the Fader website for the recognition of scene changes. Next, the VR/AR and 3ds Max models were combined, and a 360° video computed in V-Ray and captured using the aforementioned LG 360 CAM was employed for image recognition and program operation. The Java Development Kit and Android Software Development Kit were applied to write an Android program. The integrated development environment provided by Eclipse was used for debugging and implementing the program interface, enabling the mobile platform to operate on the Android system.

A historic house scene was imported into the Fader VR system using the above mentioned 360° video. This video was edited using Power Director, and visual effects were implemented using Adobe After Effects. Subsequently, the AR image cognition systems on Vuforia and MAKAR were incorporated with the Unity engine to develop the reminiscence therapy system.

Text, music, and images of scenery from the 1950s and 1960s were uploaded to the Fader cloud as VR trigger points for scene switching on the VR headset (Figure 5). The completed Fader VR files were transferred to the Samsung Note 5 Android phone, which was then placed in the VR CASE headset, allowing testers to explore the nostalgic virtual space (Figure 6).

Dynamic 360° images of the historic house can be switched between freely in the Fader 360° video and presented to a user wearing the VR CASE headset. Furthermore, the designed system combines an interactive visual–audio system with a motion guidance system that leads users to any scene or object inside the historic house. The diverse visual changes displayed by the system can trigger memories of past events among elderly users, thereby improving the effectiveness of the therapy through triggering more conversation topics.

4. Experimental Verification Instructions

To test the AR design of the reminiscence therapy system, the researchers recruited elderly users. Using the AR design, users previewed scenes in the VR system through a navigation interface. The tested AR/VR system contained an AR system that aimed to enable elderly people to capture or scan specific images using mobile devices, such as mobile phones or tablets. The captured or scanned images were synchronized to the aforementioned AR cloud systems on Vuforia and MAKAR and thus became cloud recognition pictures. After the users scanned and captured images using the designed mobile application, an AR navigation interface was triggered, which appeared and presented users with several choices of tours, namely a text guide, an audio guide, or three nostalgic stories (Figure 7). Using this AR system, users were able to preview the VR scene through the aforementioned navigation interface, allowing them to instantly be familiar with the VR environment they would later enter. This prevented users from becoming confused or lost, consequently rendering the later VR verification ineffective.

Regarding the VR design of the therapy system, users were able to experience a lifelike scene described by a text guide, and subsequently, three navigation interfaces were triggered. These interfaces comprised nine nostalgic songs sung by three singers, six nostalgic pictures, and three touching stories. The designed VR simulation system is available for mobile devices, such as Android phones and tablets. Users must wear a VR headset to use the system. The 3D historic house (i.e., a shiheyuan, which is a courtyard surrounded by buildings on all four sides) rendered in the system was modeled using 3ds Max 2018. Next, Photoshop CS6 was used to paste images. The AR cloud image recognition function in Vuforia, scene switching service in Fader VR, and the cloud system of MAKAR AR were incorporated into the 3D scenes for AR and VR through Unity 5.6.3p2 (64-bit). This allowed the 360° VR video to be generated, image recognition to be facilitated, and interactive programs to be created.

The navigation index was connected to three guide interfaces. Interface 1 played nine old songs by three famous Taiwanese singers, namely I Only Care About You, Tian mi mi and Farewell by Teresa Teng; One Plum Blossom, Thousand Miles Away and Only Love Moon on the Water by Fei Yu-Ching; and Please Follow Me, Any Empty Bottles for Sale, and Change by Su Rui. Interface 2 presented three videos related to family relationships, namely a story about raising a piglet, a story involving a grandpa and grandma, and a story involving grandparents and grandchildren. Interface 3 displayed six pictures of nostalgic items, namely an old bicycle, a train, a table and chair, a sewing machine, a house, and a well. These pictures aimed to stimulate users’ recollection of past events.

This study adopted the V-Ray 360 image computing technology in 3ds Max 2018, AR cloud image recognition systems in Vuforia and MAKAR, the VR cloud scene-switching system in Fader, and the Unity engine to develop its reminiscence therapy system. Through integrating the technologies of AR and VR text mining, video and audio playback, and 3D modeling, the system enabled participating elderly people wearing the VR headset (holding an Android smartphone with the application installed) to select and trigger any event by clicking the guiding images on the interface. The proposed system in this study is intended to be enjoyable for individual elderly people to promote the immersion experience compared with two-dimensional text and photos. Thus, the effectiveness of reminiscence therapy using the AR/VR system could be evaluated using group discussions with the elderly user, their family, and the caretaker. The findings could be applied to develop an integrated AR/VR interactive system for reminiscence therapy to increase the satisfaction and effectiveness of social interactions among elderly people in this era of an aging society.

5. Conclusions

Reminiscence therapy can be applied to alleviate weariness and meaninglessness related to aging in elderly people. This type of therapy stimulates elderly people to remember pleasant events. Through reviewing past events, they can reconstruct their overall awareness of their life value, philosophy, and stories, thereby developing self-care, cognition, and social interaction abilities and adopting strategies to solve current problems. Gradually, these elderly people can develop their self-identity, improve their psychological health, and enhance their life satisfaction.

The study results demonstrated that the 3D historic house model provided immersion experiences for participating elderly people using a Samsung Note 5 and VR CASE headset. In the developed AR/VR system, the users engaged in visual–audio interactions evoked by nostalgic elements, confirming the applicability of the designed system for reminiscence therapy. Reminiscence therapy was confirmed to enable sufficient memory recollection and cognition in elderly people, thereby being worthwhile as a therapy for dementia. The designed system led users to any scene or object inside the historic house through a motion guidance system and applied diverse visual changes to trigger memories of past events, which enhanced the therapy’s effectiveness through storytelling. This research can serve as a reference for the promotion of interactive AR systems that effectively improve social interaction skills and life satisfaction in elderly people.