Empirical Research on the Metaverse User Experience of Digital Natives

Abstract

The metaverse has been settled as a platform that is widely beloved by digital natives that are familiar with mobile devices and immersive contents. Thanks to the protocol enabling hedonic interaction, the user experience provides significant value from its communication, enabling learning experiences anytime and anywhere. However, the research topics are focused on the promotions of technology development, marketing effects, and relevant investment consensus. Surprisingly, the biggest problem was the lack of research from the perspective of the young generation, who mainly use the metaverse. This paper intends to examine the usability of digital native participants in detail and suggest how immersive contents, usage environment, and interface aspects should be designed from their point of view. As a result, the significant engagement factors and improvements, through heuristic usability evaluation considering content and user control, were discovered from individual interviews. Conversely, the elements to be supplemented in user experience were derived from information architecture and usage environment categories. In conclusion, the theoretical basis of the empirical usability evaluation on metaverse platforms and following recommendations with practical implications could gain more importance from this research.

1. Introduction

The metaverse industry has been growing significantly because of technological advancements in digital infrastructure and changes in the socio-cultural environment, particularly in the face of the COVID-19 pandemic [1,2]. According to global market research institutes, the metaverse market is expected to grow from USD 72.8 billion by IDC estimation [3] to USD 783.3 billion forecasted by Bloomberg Intelligence [4] by at least 2024.

Along with the growth of the metaverse market, the relevant ecosystems such as metaverse-related services, programming engines, hardware, and networks are also developing exponentially, and the market size of the ecosystem is expected to reach a USD 1.5 trillion by 2030 [5,6]. The social change into the non-face-to-face environment hastened the new normal lifestyle and consumption culture era. Followed by the social trend, a number of the global companies and brands are estimating the potential of the metaverse for their future business models, especially with the trend of the widespread use of smart mobile devices and applications [7]. All of the mentioned changes are happening because the metaverse, in which virtual and reality converge realistically, has the unique characteristic that leads to participation and communication between users beyond the temporal and spatial limits [8,9].

However, various research on the metaverse is mainly about some side of technology and investment consensus. This could only analyze how technologically advanced the current metaverse industry is [10,11,12], how helpful it is for e-commerce and brand promotion [13,14], and whether it is worth investing in connection with the value chain of the expanded ecosystem constitutes [6,15]. There is a particular gap in the research focus between the analysis of metaverse platforms and the user behavior in the platforms. What the core value of the metaverse is and how the platforms are used are still hidden behind a veil, especially the primary user, the digital natives, who are using metaverse platforms as a playful space and a place of active communication [16,17].

Additionally, there is no standardized criteria to measure, analyze, and evaluate the usability of the metaverse platform, which is universally used by the young generation that are sensitive to the buzzwords, behaviors, and tastes used by the reference group [9,16,18]. Moreover, there has also been an unquenchable thirst of the private sector to plan, design, operate, and develop the metaverse platform. Additionally, the policymakers promote industry and technology in cooperation with the private sector. Therefore, this study intends to narrow these academic and practical gaps.

1.1. Purpose of the Research

Recently, the global COVID-19 pandemic has been accelerating digital transformation, and it is essential to build the new processes and capabilities to meet the demand of the rapidly changing social structure [19]. Providing engaging digital experience became important to achieving remarkable innovation in the platform business. Therefore, the need for the research, which can satisfy users’ needs in using digital technology seems to be increasing daily [18,20]. To this end, it seems necessary to understand how many new platforms and services are emerging in the rapidly evolving fourth industrial revolution era, under which circumstances and for whom, and how much potential investment and technological advancement the new trend could bring [2,21]. The more the younger generation is affected by this trend, the more critical the user experience of new platforms and services there are [17]. The mentioned tendency could yield even more impact to the digital native generation that is greatly affected before they set clear individual values for their lives [10].

Schools, workplaces, meetings, and events changed their workspaces into various forms of contactless environments during the pandemic since they had been interested in the increased virtual and mixed reality metaverse platforms with high immersion and realism [2,15,22]. In particular, remote education and virtual classrooms based on the Internet and electronic technologies to deliver a broad array of solutions enhance the quality of teaching and learning activities in education fields during the pandemic situation [22,23]. Accordingly, many institutes, organizations, and companies use the metaverse to communicate in real-time and promote active participation [1,5]. The user experience could be the key success factor of the mentioned changes, and it can be inferred that it is necessary to secure a high level of understanding in terms of literacy for the metaverse from the user’s point of view [19,21]. Therefore, identifying the characteristics and level of the elements of the current user interface will be necessary, and further suggestions from the findings could make the excellent parts even better and quickly address the points for improvements. The research questions are the following:

Research Question (RQ) 1. What are the most important platform user experience factors from the perspective of major metaverse users, the digital natives?

Research Question (RQ) 2. What are the advantages and complements of the major metaverse platforms that are currently actively operating in terms of usability?

1.2. Structure of the Research Article

As we saw in the background and purpose of the study in the introduction so far, this paper intends to evaluate the usability of the major metaverse platform, which is the ‘extended real world’ from the digital native perspective. In the next chapter, the relationship between the metaverse and user experience will be explored by reviewing the theoretical background and previous studies such as virtual reality, extended reality, engagement, and usability. In particular, with the development of digital platforms and increasing interest in user experience, the scope of application of the concept of ‘usability’ is gradually expanding. In this way, we will carry out a research methodology for the effective evaluation of the evolving usability and application to actual cases.

In this paper, we specifically intend to examine the optimal metaverse research format from the digital native perspective, and the subjects of usability evaluation to be measured and to recruit participants. Based on the usability evaluation results derived through this, we will interpret the user experience for each platform and discuss the advantages and improvements of the usability evaluation system for each major item. This is because the general evaluation framework of the industry can be established only when the various requirements of users are broadly analyzed on the content and structure of subdivided content, users’ usage behavior and interaction, and the layered dimensions of the usage environment. Finally, the summary and significance of this study are presented theoretically and practically. The implications and key takeaways are also presented, along with the limitations of the research and suggestions for follow-up studies.

2. Theoretical Review

2.1. Metaverse and User Experience

According to the Metaverse Roadmap Report [24], the metaverse could be defined into four categories: Augmented Reality, Life Logging, Mirror Worlds, and Virtual Worlds. Among the four types, the area that is rapidly growing and creating new innovations is Virtual Worlds, which is metaverse-oriented and based on Virtual Reality, Mixed Reality, and Extended Reality because it is characterized as the space where an alternative world that is similar to the real world or completely different is digitally constructed [13,20,25]. It goes further from the concept of a virtual world with particular opportunities to act similarly to real life in the game and entertainment area [8,14,26].

Recently, the four classification systems of the metaverse are changing into a form of fusion since the boundaries between the classified systems are becoming less clear [16,18]. In addition, it is expanding to areas that are similar to economic and social activities in the real world beyond the previously focused fields on adventures and hedonic genres. This transformation is the ‘Expanded Virtual World’, which can realize various activity values in a paradigm where reality and virtual worlds are combined [6]. As such, it seems the era of the metaverse is in full swing. The metaverse platforms that conform to the new extended virtual world concept include ZEPETO, ROBLOX, GATHERTOWN, MINECRAFT, THE SANDBOX, and FORTNITE [4,5,10,26]. Among them, ROBLOX, GATHERTOWN, and ZEPETO have a fast-growing market size and are evolving as representative platforms that create content built on user participation [2,3,7]. Additionally, they absorb socio-cultural and commercial endeavors by enabling creators to engage in economic incentives [1,3,8].

In addition, multiple users can expect the memorable experience of sharing economic value through open communication and transaction activities, so that the platforms attract more and more people [10]. This phenomenon seems beyond the simple use of the metaverse as mission solving and consumption in the existing game and entertainment platforms, which are the services provided by the producers [11,26]. In other words, on the current metaverse platform, users can create their virtual assets using their own ideas, share the idea with other users, generate revenue, and engage in activities close to reality through various exchanges [5]. Thus, it seems obvious that the distinctive experience from the metaverse continuously provides users with high-quality content and opportunities for growth [12,14]. Furthermore, the experience makes the users share various values and will become the core point of the metaverse’s growth [18].

The studies related to user experience, however, have not been deeply researched among the metaverse-related topics, which includes an assessment of usage environment and content in terms of usability analysis or effectiveness. This is because prior research in the late 2000s only classified the types based on the content purpose and production intention of the existing metaverse platform, and the classification became predominant [15,27]. The mentioned trend could also be found from research from Google Trend, which was on the next-generation virtual world based on the case of the Second Life Service in the United States at the time [28]. Unfortunately, however, such research topics did not become an essential research topic until the late 2010s [13,18,22].

Since then, various content-based user experiences grew significantly because of the rapid development of immersive technology and the box office with future-oriented genres. The launch of the Oculus VR Go Headset by Meta (2017), the movie Ready Player One (2018) directed by Steven Spielberg, and the emergence of ZEPETO (2020) were significant milestones of metaverse history [2,12,29] like the breakthrough curves of ROBLOX in the late 2010s. Since then, media contents, games, immersive platforms, VR Techs, and the research in related fields have been revived; however, it is still difficult to find the studies that analyze such rising cases in depth and suggest relevant theoretical directions [26]. Unfortunately, many industry reports still seem to be focused on widely spread metaverse use cases by simply listing public campaigns, pilot educations, and cultural events [1,30]. It seems only possible to see limited contents for single visits and less-interesting experiences based on the form of celebrities and brands appearing in the case [5].

In spite of all the research bottlenecks of the rising industry, these bottlenecks could be a starting point to building the relevant research fundamentals of user behavior. A universal evaluation framework for the metaverse-relevant industry could be established on a matrix of usability factors based on user experience dynamics [20,25,27]. Like the interface components of created virtual maps, the structures of subdivided contents, segments by user interaction types, and the layered dimensions of the user environment would be the various requirements of analyzing in-depth research of the metaverse [11,18]. This study intends to reveal the immersive user experience of the general metaverse platform and how the major factors of the ‘Extended Real World’ are implemented, the world that shows high touch on mesmeric communication beyond the user’s point of view from conventional online games and cyber playgrounds.

2.2. Application and Extension of the Concept of Usability

2.2.1. Evolution of the Concept of Usability

Usability is an attribute that evaluates how a user interface can be easily used and refers to a method to improve the ease of use during the product or system design process [31,32]. Nielsen defines the five initial usability criteria: learnability, efficiency, memorability, error, and satisfaction. Discussing the criteria in detail, the ease of learning refers to whether a user can quickly complete an essential task when they encounter the system for the first time, and efficiency describes how well the user performs a learned task [31]. In addition, the ease of memory indicates whether the user can promptly recover proficiency when using the system again after long time, and error refers to the questions of how frequently the user generates errors, which of the errors are critical, and how quickly they can be corrected. Finally, satisfaction means whether the system is satisfactory to use [33].

In terms of usability, a well-designed user interface affects learning time, performance speed, prevention of errors, and user satisfaction, which are considered essential factors in determining the success of products and services in general [34]. The usability test is an activity that identifies problems to improve the user’s experience when designing products or services and discovers opportunities for improvement [35]. Although there are various methodologies for usability evaluation, general user tests and heuristic evaluation could be considered a major part of the evaluation method [36]. A general user test is conducted by requesting an actual user a task. Conversely, in a heuristic evaluation, an expert becomes the subject of the assessment and evaluates based on the usability criteria [37].

The heuristic evaluation is a method of usability engineering to find usability problems in user interface design, and a small number of experts apply heuristic principles to determine whether interface usability complies with the test criteria [33]. Therefore, the evaluation has the advantage of quickly deriving usability problems with a few people under a limited time or the difficult circumstances of recruiting participants in the initial design stage and general user evaluation [31,36]. However, since experts’ knowledge and competence greatly influence the evaluation result, it seems necessary to supplement general user evaluation methods to the evaluation made by experts based on the purpose and given situation [38].

The criteria for heuristic evaluation (in Figure 1) are defined by several experts, and among the criteria, the 10 heuristic principles of user interface evaluation, which Nielsen comprehensively summarized, are most commonly used: visibility of system status; match between system and the real world; user control and freedom; consistency and standards; error prevention; recognition rather than recall; flexibility and efficiency of use; aesthetic and minimalist design; recognize, diagnose, and recover from errors; help and documentation [31].

2.2.2. Extension of Usability Evaluation Method

In order to build a usability evaluation framework for the metaverse platform, there are interdisciplinary studies on the user experience of augmented games, hedonic mobile platforms, immersive media contents, and an expanded virtual world [39,40,41]. Baszucki, the CEO of ROBLOX, interestingly mentioned 8 characteristics to consider when creating a metaverse: emphasizing the need for an avatar, friends, immersion, access anywhere, low friction, various contents, circulating economic activity, and lastly, safety [42]. As a result, several aligned elements, as the usability measurement of the mobile interface and virtual space, enable users to experience continuous immersion and improve engagement in the metaverse platform.

Among the characteristics, avatars expressing one’s identity, immersive setting and features, interactions with other users, and easy-to-use accessibility are presented in general, and these four major elements are suggested as continuous immersion drivers in the virtual space in the context of insights from the metaverse case study [43]. Additionally, the above-mentioned trend seems to be supported by the recent media research by Palavicini, etc. [44], which shows the importance of spatial presence and flow concept discovered from recent media research, and by the study made by Jennifer and colleagues [45], which found that usability in a virtual reality environment had a high correlation with immersive user experience and presence.

As discussed above, it could be found that the elements that enable continuous usage and immersion experience are core concepts based on the literature reviews related to the usability of mobiles and metaverse [36]. In other words, although the usability evaluation factors vary for different purposes and subjects of the study, the basic procedure and technique of the usability evaluation method, which enables a continuous immersion experience, seem similar [38]. To understand in depth, it is necessary to consider exploratory research on the metaverse platform that seems to still be in an early stage of technology and service maturity, which is why it would be appropriate to collect experts’ opinions and re-combine the evaluation elements for the characteristics of the evaluation target [35,46].

For a better understanding, a designated evaluation framework of recent metaverse research would be applied [33,34,35] based on the prior studies reviewed above. Five detailed factors for each usability evaluation area are classified into the following categories: user perspective, information structure, graphical design, attribution of contents, and usage environment (

Table 1. Usability Test Framework for metaverse Platform (Source: Reorganized by the researcher referring to Literature Review [33,34,35]).

Category	Subcategory	Usability Test: Detail Check List
1. User Control	1.1. Responsiveness	Is it easy for users to know what tasks they can do here?
	1.2. Learnability	Can users naturally learn and work with the service? (e.g., Manual, Magnifying glass, Search box)
	1.3. Affordance	Is it providing a function that allows users to do what they want? (e.g., Movement control by button)
	1.4. Interactivity	How easy is it for users to interact with other users? (e.g., Conversation with voice, Encounter with screen)
2. Information Architecture (IA)	2.1. Logicality	Are menu choices ordered in the most logical way?
	2.2. Navigatability	Is it possible to use a method intuitively to navigate to the information what users need?
	2.3. Searchability	Is it easy to access when users need to search?
	2.4. Term Comprehension	Does it presented with easy terms (labels) that users could understand?
3. Graphic User Interface (GUI)	3.1. Congruence	Does the design concept fit the purpose of the content overall?
	3.2. Attractiveness	Does it have design points that can appeal to users?
	3.3. Aesthetic	Does it convey a unique atmosphere? (e.g., color, image, font, layout style)
	3.4. Immersiveness	Is dynamic design being used appropriately to help the user experience? (e.g., flash, multimedia)
4. Contents	4.1. Clarity	Is the purpose of the content, want to deliver to users clear?
	4.2. Consistency	Are the contents consistently organized by area?
	4.3. Differentiation	Does it provide the distinguished contents?
	4.4. Effectiveness	Does the content have a distinctly expected effect?
5. System Support and Settings	5.1. Preventionality	Does it prevent mistakes that users can make in advance? (e.g., guide presentation, deactivation, warning text)
	5.2. Error Detectionability	Is there an immediate signal when a service error occurs? (e.g., error popup, exclamation mark, alarm sound)
	5.3. Error Resilience	Can users correct errors? (e.g., Go back/Undo, return to a specific state/point in time, restart)
	5.4. Functionality	Is the system running smoothly? (e.g., play speed, network connection status)

). Based on the categories, an empirical study could be conducted to target actual digital natives using the major metaverse platforms in the market.

3. Research Method and Scope

3.1. Guided Participants Recruitment

In this study, the content related to climate change and energy education are selected and analyzed to examine the practical use of the usability evaluation system in the metaverse platform. When selecting content, the environment-related theme is considered as an integrated approach to holistic education rather than the math-and-writing (including the languages more than Korean)-focused curriculum method most widely used in elementary school [1,43]. Furthermore, climate change and energy programs are suitable themes where the educational effect could be increased when combined with the extended virtual world [47,48]. It overcomes the temporal and spatial limitations outside the classroom [49].

Regarding the metaverse platform selected for this study, the authors have already estimated the details in the research background section. The rankings measured by the market share, competitiveness, and popularity of the metaverse platform are different for each major market research institute around the world [1,2,3,4,5]. Nevertheless, the major metaverse platforms are selected, analyzed, and announced in consideration of the number of active members, content production volume, interaction size, and transaction scale in common.

In this study, too, based on these frames, we first compressed the major platform candidates to within ten. Among them, we selected three platforms that have the same or similar immersive content that is easy for digital natives to understand. GATHERTOWN, ROBLOX, and ZEPETO are those metaverse platforms. In other words, we found relative strengths by measuring and comparing the usability of each platform with common content related to climate change. Of course, we also found user experience elements with relatively low scores and suggested additional practical ways to improve them.

Meanwhile, it is hard to find the same content or world created by the same host within the three platforms as the scope of research. Furthermore, the structured search method is reviewed with 120 relevant keywords within the metaverse platform, Naver (Korean No.1 Portal), and Google for navigating the scope of research. The climate change and energy theme is selected as a topic with the most similar or identical content properties [1,30]. These are considered to be the understanding and feasibility of the participants, the digital natives [22,30].

Teenagers are generally seen as the most interested and the main user group of the metaverse, who participate in the conducted evaluation for the metaverse platform usability. Among the teenagers, two students from each elementary school system’s 4th, 5th, and 6th grade participated in the assessment voluntarily [16,49].

3.2. Ethical Considerations

The two core parts of conducting such an evaluation is keeping the voluntary willingness of the participants intact and following appropriate research ethics. Therefore, the interviews conducted for digital native participants are followed by the UNICEF guidelines [50], and the relevant prior consent and acceptance were given by the participants’ parents for the research process. All participants were informed about the details of the study in advance. Study participation was anonymous and voluntary, and students could withdraw from the study without any consequences. For data protection reasons, only the researchers had access to the research data. The procedures of this study complied with the provisions of the Declaration of Helsinki regarding research on human participants. Ethical Committees of the involved higher education institutions approved this study, as did the Korea Centers for Disease Control and Prevention (Ethical code: 22CAA08071). The specific contents and scope of the user evaluation are as follows.

3.3. Usability Test Scope of Metaverse Platform

3.3.1. Usability Test in GATHERTOWN

GATHERTOWN is a virtual space in the form of a digital office launched in 2020 by Gather Presence of the United States and is a video conferencing platform [2]. In the early days, it was mainly used for work to support corporate telecommuting but now is widely expanded to various social exchange activities such as university campuses, conferences, and festivals [51]. The number of accumulated subscribers surpassed 4 million within one year of service launch and 10 million within two years [16,52]. Basically, as a general-purpose metaverse platform, anyone can easily select their avatar without a separate sign-up process, and the fact that they can communicate through voice, video, and text has become the basis for many users to use [1,51]. In particular, users can share external data in various forms compared to the game-type metaverse platform [16], including key features such as video and presentation data transmission, joint document work, user participation voting, and open discussion by topic [2,52].

Taking advantage of these features of GATHERTOWN, the Korean National Forest Service (a government agency) created an urban forest town in the metaverse space under the theme of the ‘2021 Urban Forest Awareness Campaign’ to encourage citizen participation in November 2021 [53]. Since the platform was only capable of having 400 users at the same time, request for pre-registrations were sent out in October 2021, and the first event was conducted, and after the initial pilot event period, the campaign was fully open to anyone to participate. The activities consisted of various programs such as propensity tests, quizzes, treasure hunts, listening to eco-sound, and Do It Yourself experiences with the theme of urban forests so that participants can have an experience similar to a family picnic in the real world (Figure 2). It was the first campaign for users to experience the urban forest in a virtual space among the related green city campaigns running since 2008, and more than 3400 virtual avatars participated in November 2021 [53].

The experience method is to access the urban forest town in GATHERTOWN on a PC and perform four challenges. The challenging task for the usability evaluation was as shown in

Table 2. GATHERTOWN Usability Test Items ¹.

No.	Level	Duration	Mission 2	Criteria of Success	UT Dimension
1	Easy	1~2 min	Checking-in guestbook	Archiving 50-digit messages in Padlet guestbook	User, IA 3
2	Intermediate	2~3 min	Taking preference test	Submitting the 12 items to answer the questions	Content, GUI
3	Difficult	3~4 min	Hunting some Treasures	Finalizing outcome from the trailblazing in map	GUI, User
4	Difficult	4~5 min	Playing the Chrome-Cast	Completing 10 s melody file as a sharable link	IA, Content
Total	about 9~10 min		Total 4 Challenges	https://gather.town/app/cOov7zPALL7yqK97/KOREA%20FOREST%20SERVICE (accessed on 23 March 2022).

¹ Usability Test Items are reorganized by the researcher referring to Literature Review [16,51,52,53]. ² All of the missions in GATHERTOWN are open to all participants with no friction and no limit. ³ IA stands for information architecture, GUI stands for graphic user interface in usability test dimension.

, considering the detailed progress method, difficulty, and expected time. First, starting with the mission of leaving a guestbook in the virtual forest experience, based on a snail marathon with the character ‘Gru’, the main character of the city’s forest town (low difficulty level). Second, digital natives could participate in the eco-friendly activity preference test based on 12 items of personality type (intermediate difficulty level). Third, there are entertaining quiz treasure hunts in the metaverse forest map (choose one out of five types, the highest difficulty level). The last mission is to play a short eco-friendly song using the Chrome-Cast shared piano feature that contains the accessible melodies of the campaign users and then download the save link. These missions are highly designated activities related to the post-survey of user experience factors such as information architecture, graphic user interface, and contents.

3.3.2. Usability Test in ROBLOX

ROBLOX is a large-scale metaverse platform with more than 1 billion subscribers since it was launched in 2006 by ROBLOX Corporation of the United States and then overhauled in 2018 [2]. With about 18 million user-participated game maps and more than 43 million daily active users, affiliates in the global consumer goods, tourism, finance, education, and public sectors are continuously increasing their virtual space [1,42]. In ROBLOX, users can create their own games and play games created by other users, and the relevant statistics have been announced that more than 55% of elementary school students in the United States are using the platform [6,16]. In addition, it is highly scalable as ROBLOX has the strength to connect a player to various devices such as Xbox and Oculus VR headsets, as well as simultaneous support for PC and mobile apps [5,18]. Furthermore, there is a function in the platform to find and invite friends linked to social media such as Twitter, Facebook, and Discord, which provides an easy way to play together [4,7]. The distinctive feature is that users can purchase and trade items with game currency and points called Robux (about USD 0.0125 for 1 Robux as of 1 October 2022). Due to these factors, the industry is evaluated as one step closer to economic value creation following the socio-cultural aspect of the metaverse platform [15,17].

‘HMA’ is a virtual customer experience space implemented by Hyundai Motor Group to let users experience mobility in ROBLOX for the first time as a global automobile brand. The open beta was released in September 2021, and the official service started in October [54]. In about a 6-month timeframe until March 2022, there were more than 3.4 million visits from 100,000 people actively playing the adventure as a favorite. When users access the adventure, the journey starts from Festival Square and goes to five virtual spaces with various themes and experience materials. In particular, users can enjoy self-driving vehicles such as Casper, the latest Hyundai Motors utility vehicle, and the popular eco-friendly mobility IONIQ 5 [54].

Experience measurement in ROBLOX is conducted by accessing HMA with a desktop or tablet PC (pad) environment, which has the capacity for stable graphics and high RAM (Random Access Memory for computing devices) [55]. Specifically, the given task for the usability evaluation was composed of four items within 10 min, considering the high, medium, and low difficulty as shown in

Table 3. ROBLOX Usability Test Items ¹.

No.	Level	Duration	Mission 2	Criteria of Success	UT Dimension
1	Easy	1~2 min	Driving the Eco Vehicle	30 s+ Driving the designated Eco Vehicle	IA, User
2	Intermediate	2~3 min	Customizing my Mobility	Changing to the Eco interiors in special garage	GUI, Setting
3	Difficult	3~4 min	Getting Virtual Points	Participating 1 Event: Car Washing, Camping	User, GUI
4	Difficult	4~5 min	Attaining Special Badges	Visiting 1 Adventure: Eco Forest, Racing Park	Content, IA
Total	about 10~12 min		Total 4 Challenges	www.ROBLOX.com/games/7280776979/Hyundai-Mobility-Adventure (accessed on 23 March 2022).

¹ Usability Test Items are reorganized by the researcher referring to Literature Review [15,16,51,52,53]. ² All of the missions in ROBLOX are open to all participants with no friction and no limit.

. First, summon the desired car from the Adventure Plaza, test drive the vehicle for about 30 s, then move to the user’s own garage and customize the car with the colors, wheels, and other options per individual preference. These missions are designed to naturally increase the user experience by performing the activities to be measured in the post-survey (Figure 3). In addition, participate in one or more events in the adventure provided in real-time to receive H-Coins from the car wash, camping zone, concert, and maintenance. The last mission is to get at least one game master badge by visiting one of the remarkable adventures of Future Mobility City, Eco Forest, Racing Park, and Smart Tech Campuses (the highest difficulty level).

3.3.3. Usability Test in ZEPETO

ZEPETO is a three-dimensional avatar-based metaverse platform launched by Naver Zet inc. in August 2018 [1,56]. Users can create their own 3D avatars through augmented reality technology and communicate in real-time through voice, video, and text between acquaintances by actively using the friend invitation function in the metaverse [9,56]. Due to these features, it became popular with Generation Z worldwide, and as of March 2022, the cumulative number of its global subscribers had exceeded 300 million [16]. A great attraction is that anyone can download the app for free and customize their avatar as per their preference so many users worldwide can easily access and use it [26,55]. The high level of AR technology makes it possible to configure avatars in detail, such as their clothes, composition, and accessories, and to command various movements and emotional expressions, so the level of immersion is high [2]. Thanks to these features, it became a popular virtual hub space for global brands such as Gucci and Christian Dior, as well as various musicians (BTS, Blackpink) from entertainment companies such as Hive and JYP [1,16,56].

The ‘Eco Cube Experience’ is a virtual space where ‘Flowers and the Little Prince’, a non-profit foundation of the Kolon Group, implemented an eco-friendly energy exhibition on the ZEPETO platform starting in February 2022 [57]. It is an open space where anyone can participate. From the beginning to the exterior of the virtual convention hall built with eco-friendly materials despite its virtual exhibition concept, participants can feel the theme of peace of min. Furthermore, the full amenity of the atmosphere offers a substantial opportunity for mindfulness with the landscape space that is harmonized with nature, such as trees and lakes. In the energy-saving façade composition, a metaverse user could have the opportunity to directly watch the case of utilizing solar power to heat and cool the building. Eleven types of new and renewable energy for a sustainable future can be viewed in one place [57].

Specifically, three types in the new energy field are introduced as the fuel cell, hydrogen energy, and coal liquefied gas. Additionally, eight types of renewable energy fields, which are solar power, solar heat, biomass, small hydro, marine, wind power, geothermal, and energy source from wastes, are presented in the virtual exhibition. At the opening ceremony, an event was held to provide a Zem Point, which includes the activities such as completing all missions (i.e., quiz, game) and posting a certified shot on the ZEPETO feed (also known as the Bulletin Board), and the event successfully achieved 5000 cumulative visitors in March [56]. After the first pilot operation, it is expected to be widely used in after-school programs and visiting energy classes with which the foundation cooperates.

Meanwhile, the metaverse usability test through the eco-friendly energy convention zone within ZEPETO is to access the world with an individual mobile device and perform a given task. The four specific tasks for usability evaluation were composed as shown in Figure 3 and

Table 4. ZEPETO Usability Test Items ¹.

No.	Level	Duration	Mission 2	Criteria of Success	UT Dimension
1	Easy	1~2 min	Exploring the 3 Eco zones	Leaving certified photos after exploring the zone	User, GUI
2	Intermediate	2~3 min	Obtaining a soft currency	Experiencing 1 of the new/renewables energy booth	IA, Setting
3	Difficult	3~4 min	Climate Change OX Quiz	Submitting answers 3 quiz with avatar control	User, Content
4	Difficult	6~7 min	Playing a Sky Jump game	Taking the final stage shot, completed the 9 steps-up	GUI, IA
Total	about 10~12 min		Total 4 Challenges	www.zepeto.me/ecolonglongcube (accessed on 23 March 2022).

¹ Usability Test Items are reorganized by the researcher referring to Literature Review [15,51,55,56,57]. ² All of the missions in ZEPETO are open to all participants with no friction and no limit.

. First, after visiting the Eco-Long Long Cube green building and the surrounding amenities, leave a certification photo. Then, users could obtain soft currency, in terms of Zem points, by participating in the presentation of new and renewable energy virtual booths. After that, solve and pass all three stages of the OX quiz on climate change, prepared in the open studio of the annex building. If there is an incorrect answer, it automatically returns to the first stage and restarts. Finally, the mission can be completed by climbing the nine moving cubes step by step to reach the finish point [57]. The Cube Sky Jump game is the most difficult level ever of the metaverse experience mission, according to the comments made by most of the participants.

4. Results and Discussions

4.1. Results of User Experience by Platform

As a result of the usability evaluation for each metaverse platform, all six students succeeded in exploring the mission. The average mission challenge time was about 34.2 min, and the task performance rate reached about 94%. Looking at each platform, the average challenge time for the four missions in GATHERTOWN was about 12.0 min (standard deviation of 0.80), and the task performance rate was recorded at about 97%. The challenge time for the four missions in ROBLOX was measured to be about 11.4 min (standard deviation 0.93) on average, and the completion rate of the challenge reached about 96%, which can be seen as mostly not difficult.

Lastly, the average challenge time for the missions in ZEPETO was about 10.8 min (standard deviation of 1.66), and the task performance rate reached about 89%. Compared to the previous two platforms, it was estimated that the last mission itself was abandoned early or repeatedly challenged several times because of it having one of the highest difficulties. This mission was included as the cause of the wide variation in challenge time between participants compared to the previous two platforms. For reference, among the 12 task items in the three metaverses, the Eco Cube Sky Jump challenge in ZEPETO was recorded as the most difficult mission because not one participant passed.

Overall, it took 13.9% longer than the 30 min challenge time expected by the researcher in the initial design. Nevertheless, the challenge performance rate far exceeded the 70% estimate, indicating that sufficient platform experience was accumulated for the usability evaluation. This is because the time it takes to perform a specific action in the metaverse is indirectly an indicator of the difficulty and understanding of usability. In addition, measuring user behavior items reflecting the characteristics of user control and information structure can present a useful point of view in the usability evaluation. Each participant’s detailed platform experience progress results are summarized in

Table 5. Metaverse Platform Experience Mission Results by Participants ¹.

No.	User Code	Total Time	Mission Complete%	Gender	Grade	GATHER TOWN	Time (min)	ROBLOX	Time (min)	ZEPETO	Time (min)
1	KDH	34.5	97%	M	4	100%	11.3	100%	10.5	90%	12.7
2	CWY	32.8	92%	M	6	90%	12.6	100%	10.3	85%	9.9
3	LYW	36.1	93%	F	5	95%	12.9	95%	11.1	90%	12.1
4	WHE	35.2	98%	F	5	100%	12.3	100%	12.6	95%	10.3
5	PKS	34.6	95%	M	6	100%	10.8	95%	12.2	90%	11.6
6	JSB	31.8	88%	F	4	95%	11.9	85%	11.7	85%	8.2
-	Average minute 34.2		Avg. 94%	3:3	3:3	97%	11.6	96%	11.5	89%	10.8

¹ All of the research interviews conducted for digital native participants are followed by the UNICEF guidelines [50], and the relevant prior consent and acceptance were given by the participants’ parents for the research process.

.

Interestingly, the feedback of digital native participants was summarized through user interviews conducted to build a common user experience, and the following three things stood out. First, we encountered concerns that most of the terms of the metaverse platform were presented in English, and approximately 30% of the participants found the terms written in English difficult to understand. So, it might not be possible for local users to fully experience the service. In particular, a number of virtual campaigns and events are often held with terminologies that are not commonly or widely used by the public (e.g., professional jargon and abbreviations only familiar to particular group of people). This might be a substantial bottleneck of using the metaverse platform, which might limit the usage of the platform for the digital natives to just enjoy it as a simple game-type space (world/map). Certainly, it would be natural for participants of the platform to encounter unfamiliar new words because the entire experience in the platform would be new for them. Nevertheless, it would be necessary for the platform providers to optimize the users’ platform journey, so that users can enjoy the platform while facing less hurdles or none at all during their adventure.

The next interesting point is that digital natives have a fairly high level of understanding and utilization of digital device functions such as checking data usage before usability evaluation, setting Wi-Fi functions, and temporarily opening service access restrictions with parental consent. In this regard, the familiarity of digital natives with devices and contents is higher than expected, suggesting that usability evaluation studies need to be actively conducted in multiple dimensions [22,43]. Furthermore, elementary school children had more experience using pad-type tablet PCs (e.g., Galaxy Tab, iPad) instead of small smartphones compared to adults. Through this, considering the main users’ age of the platforms, it could be found that a vast majority of existing metaverse platforms are being used in a comparatively wider and better-performing environment than smartphones in terms of screen size and relevant immersiveness [16,17].

Particularly, the trend mentioned above has been accelerated in the last 2–3 years when distance learning from the classroom became the mainstream method due to the pandemic, which digital native participants and parents also experienced [49,58]. Noticing the change and seeing it as the new market demand, EduTech companies (e.g., Megastudy SmartAll, and Silicon Valley Unicorn Start-up Riiid) innovated in their elementary school curriculums along with the new circumstance [1,14,16]. Through the innovation, it seems that the companies, in line with the explosive demand, brought the impact made by Artificial Intelligence (AI), immersive contents, and a curriculum with real-time feedback and probably seemed like it would grow more in the future [2,3,59].

In this way, although children play games and entertainment for hedonic purposes through multimedia, it can be noticed that the metaverse-oriented education is becoming more familiar in the learning experience [9,22]. In this participatory observation and post-survey interview, digital natives also answered whether they preferred the user experience to digital learning content and public games. It was possible to reconfirm that digital services influence digital natives’ self-formation and communication with peer groups [17,22].

Based on this, it was found that a rich interdisciplinary study needs differences in perception, major values, and utilization methods with metaverse. Moreover, it could be varied for each life cycle, such as learning, living, social (building a relationship), and cultural enjoyment using the metaverse platform in the near future [1,14,16]. Finally, there are some suggestions that studies on the usability of the metaverse platform can be deeply discussed more academically, practically, and politically.

4.2. Results of Empirical Usability Test

The results of the actual metaverse usability evaluation, based on the similar user experiences built among participants through 12 missions, are as follows. First, in the case of GATHERTOWN, the usability evaluation results were counted in the order of contents area (average 5.450 points out of a total 7.000), user control (5.388), graphic user interface (4.763), system support and setting area (3.438), and information architecture (3.363). Next, the ROBLOX usability evaluation results were counted in the order of user control (average 5.463), contents (5.788), graphic user interface (4.888), information architecture (3.300), and system (2.975). Finally, the usability evaluation results were counted in the following order in ZEPETO: graphic user interface (average 5.875), contents (5.213), users (4.288), information architecture (3.488), and systems (3.363). Similarly, the average score of the usability evaluation of the user control and the contents category was high, while the usage environment (system) and information architecture were relatively low.

Comparing this with a radial graph (Figure 4), it is clear that the high and low scores for each of the three platform evaluation results can be relatively well found. In the case of the user control category, GATHERTOWN (5.388) got a high score, following ROBLOX (5.463), which seems to be because it was evaluated as an excellent element for interactivity and learnability between users. On the other hand, in the case of information architecture, the overall average score was low. Interestingly, compared to other platforms, it was found that factors related to user initiative may not receive usability evaluation in proportion to the systematic information structure. It could be interpreted as a difference in characteristics between function-oriented platforms and entertainment-oriented platforms. In other words, compared to practical platforms such as portals, shopping, and finance, they often have a vertical system of information and components according to the user’s intentions [9]. Meanwhile, the information composition method of the metaverse platform, which uses interaction-based social and hedonic flow, is more diverse and simultaneous [1,16,59].

Next, in the case of the graphic user interface category, ZEPETO received an overwhelmingly high score (5.875), which seems to appeal to the taste of digital natives overall with vivid colors and strong contrast compared to other metaverse platforms [9]. Furthermore, compared with the other two platforms, the tone and manner of the ZEPETO characters are large in size and have active movement. Additionally, the decorative elements of the virtual space are bold, forming the overall atmosphere. This contrasts with the user interface of GATHERTOWN, which is serviced mainly by small, retro-type, pixel-sized avatars, and ROBLOX, which consists of gaming-styled characters such as LEGO blocks [2,38]. However, it seems related to the main user group of each metaverse platform: ROBLOX and ZEPETO. It supposes that it can be interpreted in connection with the cultural code of ZEPETO that has many users from the regions where K-pop culture is relatively widespread, such as Asia and South America [1,59]. Vice versa, ROBLOX has many users from the US and Europe. In short, it can vary depending on cultural similarity, purpose and motive of usage, and preference of each generation [5,6], so a suggestion could be made as a part that can be revealed through follow-up research.

5. Discussions

As we highlighted in the research results, two research questions were answered. This study borrowed and verified the heuristic usability evaluation framework proposed by Jacob Nielsen [31,33] to analyze how effective the usability of the metaverse platform is for general users, especially digital natives. In order to evaluate the usability of the metaverse that has complex characteristics of games, virtual reality, and social media, it was possible to derive appropriate measures and build an evaluation framework through literature reviews and empirical research. This could be the answer to research question 1: what are the most important platform user experience factors from the perspective of major metaverse users, the digital natives?

The found measure and evaluation framework could be considered to be of noticeable academic significance because it seems to establish a basic framework for the usability evaluation of real users, a topic that has not been largely addressed in the metaverse-related research so far. Moreover, the significance could also be contributed by the fact that the detailed parts of the usability evaluation were made based on the digital natives’ life experiences and insights. As a result of conducting the evaluation, the major metaverse platforms were successfully evaluated from the user’s point of view in terms of user experience, information architecture, graphic user interface, contents, and usage environment.

The findings above lead to academic significance that includes the deep understanding of the actual usage behavior of the metaverse platform. Therefore, it goes one step further from the previous study on the intention to use the metaverse service and abstract use factors [30,43]. In other words, important determinants of users’ actual usage were fairly verified, not a theoretical model assumed to be cognitive factors related to perceived playfulness, ease of use, and usefulness [59]. In practice, major factors such as the user’s initiative in emerging new technologies and services could explain the acceptance of innovation and adaptation of new platforms [11,14].

Additionally, literacy competence of trailblazing digital service, visibility of user behavior, and interactions with other users in the metaverse could be explaining factors as well [5], with which the findings seem reasonably aligned. This could help to answer research question no.2: what are the advantages and complements of the major metaverse platforms that are currently actively operating in terms of usability? In addition, the framework can also suggest what the various factors could be of users’ being influenced to use the platforms continuously, which could indicate more than the superficial data such as consideration to visit, intention to use, and recommendation [60].

In the case of usability measurement, understanding and familiarity improve according to the usage time and frequency [31,32]. For a population who is curious and interested in new media and technologies more than the general scope, the potential for obsolescence and improvement of these measures will increase [34]. Accordingly, it is seen that there is a high need to develop the method and viewpoint of usability evaluation according to the degree of industrial maturity and technological involvement [35]. It is necessary to develop research on the future of mixed reality and extended reality by continuously checking the implications and trial and error insights of the research on the acceptance or adoption of existing new technologies and new media.

Regarding content among usability evaluation items, all three platforms received relatively high evaluations with an average score of above 5.2 out of 7. Out of that result, it could be well argued again that the metaverse platform, thanks to its immersiveness, effective message delivery, and content differentiation, served its objective well to convey the message of the research topic—the relevant cross-curricular theme regarding climate change [1,16]. As found above, the mentioned characteristics of the metaverse platform become substantially important criteria in terms of securing the metaverse’s excellent user experience [19]. Many kinds of intangible services and platforms could earn engagement and support from users depending on what kind of content they provide [18].

Accordingly, the value of the applied service is evaluated with this obvious value proposition for dedicated users. As discussed above, this is becoming an even more important factor in platforms providing digital services and advanced technologies [8,12,15].

Therefore, it is clear that user experience with the content ecosystem occupies quite an important weight in the usability evaluation of the metaverse platform [20,22]. The key is how valuable and differentiated the purpose of the content is to be delivered to users [19]. It is expected that this will eventually be accumulated into a satisfactory user experience and lead to the continuous usage of the platform [18]. Furthermore, the immersive technology and interactive infrastructure provided by the metaverse will ultimately make the effectiveness of the loaded content clear, which will deliver simultaneous value to users [10,46].

The usability evaluation related to the environment of the metaverse platform received the lowest overall score of 3 out of 7 points (43–49% when converted to 100 total). Compared to other usability factors, it can be seen that the factors for user error prevention, perception of change, and smooth operation are not yet sufficiently filled [33]. It was highlighted that the most urgent element is to supplement the function to prevent and recover user behavior errors in the metaverse platform in advance [25,35]. This will provide valid implications for designers, developers, and operational experts who plan, develop, and operate the metaverse platform in the real world.

In contrast to the fact that the existing literature focused on ‘user experience’ within the interface, this study will be meaningful in providing a more expanded perspective in terms of a complex digital platform. As we have already reviewed, the usability evaluation of video games also focused on GUI and contents [41,44,51], and user control and design were the main concerns in previous studies on usability evaluation on virtual reality learning [39,43,48]. This is fundamental to the fact that the heuristic evaluation is based on interaction and immersion in the user journey from the digital native perspective of experiencing the services provided by the metaverse platform. In other words, the evaluation areas related to user initiative, GUI, and immersive contents that were dealt with can be viewed as a visible interface level that evaluates the relatively close contact with the user.

On the basis of these previous studies, an attempt was made to expand the space–time axis by expanding the hidden aspects of user behavior and its context into the realm of usability evaluation [20,36]. This is related to the system level that is in contact with the expansion from UX design to the service design field, and it is the evaluation of information architecture or the user’s system environment [37,45]. As mentioned in Section 2, the concept of usability is expanding forward thanks to the development of the digital industry, and its value and usage are getting wider [19]. In this perspective, this study starts from a heuristic theoretical point of view and evaluates the interface, process, and system stages to provide implications that can lead to practical usability improvement strategies.

Currently, GATHERTOWN, ZEPETO, and ROBLOX do not provide the ability to correct user errors immediately, so there may be a potential risk of inconvenience to users in the future. Even four out of six digital native participants showed a usage pattern of restarting by pressing the “Exit” button when a problem occurred during usage. If these inconveniences are accumulated repetitively, a vicious cycle made from users’ subconsciousness is expected, which may eventually lead to a decrease in the service use frequency, decrease in re-visits, leaving the platform, and ultimately deletion of the metaverse application.

On top of what was discussed above, the need for an organic review of usability evaluation items would have to be emphasized in line with the rapid growth and expansion of the metaverse itself. In a situation where specialized metaverse platforms that reflect the various needs and tastes are continuously appearing, the usability evaluation verified by this study will need to be further explored. If we regularly receive opinions from experts through real-time Delphi surveys, measure them, and announce improvements, we will be able to maintain a highly appropriate evaluation item for future research.

6. Conclusions

In summary, user-friendly details are fulfilled, and users could experience a high usability and have a satisfactory experience with the rapidly developing metaverse platform. The findings from the framework could subsequently lead to a practical research outcome that might identify how user experience, an important element of the metaverse platform usage optimization, could be convenient and effective. Additionally, it could be sustained in the circumstance where the number of the convergence platforms, digital services, and their compatible IT devices are significantly increasing. Despite pioneering the empirical research of usability evaluation on the worldwide metaverse platform being used from a digital native point of view, it has some suggestions for future research. Firstly, regarding the methodological limitations of a small number of interview surveys, additional follow-up research with larger samples of participants could include more diverse perspectives. In addition, depending on the industrial and technological maturity of the metaverse, a richer level of understanding will be possible if a large-scale user survey is conducted in parallel and comparatively analyzed with the results. Finally, the method of evaluating usability would still have to be structured and assessed based on fully considered multi-dimensional factors coming from the complexity of actual users’’ usage circumstance. In this regard, it is expected that in-depth research tasks for the upcoming extended real world can be derived if additional studies by various user groups are conducted based on this study.