1 Introduction

In the recent years, the development of Internet has enabled a rapid expansion of ecommerce. According to the U.S. Census Bureau from 2007 to 2017, the e-commerce spending duplicated the spending on retail sales, reaching 10% of overall sales. For that reason, more and more companies and industries are making large investments in e-commerce websites or applications that allow companies to collect lot of useful information about their users.

The hospitality and tourism sector are one of the first industries to use Internet for this purpose. The online hotel bookings have had significant growth in the last years. In this sector, the user reviews are increasingly becoming a crucial factor in hotel selection. Users tend to trust online reviews to obtain information about goods and services (e.g. [2], [3], [18], [30], [35], [49]). Online reviews of travel websites like TripAdvisor, Booking.com, Airbnb or even Google Maps have become important information sources for users when they select a restaurant, decide which hotels to stay in, or visit tourist attractions (e.g. [9], [10], [32]). In this sense, Freeman (2020) highlights the strong reliance of users on opinions. In 2007, 52% of would-be users would not book a hotel if an online review was available.

Usually, online reviews contain users’ comments alongside assessments expressed through ordered qualitative scales formed by linguistic terms, stars or other items which are used by companies to evaluate the quality of their goods or services.

Rankings published on websites are a typical feature of e-WOM. Generally, overall scores of rankings are shown by means of numbers, items or stars that try to represent the average score of the users’ assessments and to summarize the proportion of positive, negative and neutral reviews (e.g. [17], [19], [33], [38]). The main advantages of rankings are that allow users to compare quickly different options. Likewise, they are an appropriate tool for the hotel industry for evaluating their competitiveness in the market in relation to their competitors [48]. Therefore, the development of reliable valuation systems for adequately processing and aggregating assessments is required.

Due to the fact that human beings are more comfortable expressing their opinions with words rather than numbers in situations of vagueness and uncertainty (e.g. [8], [45], [47], [51]) many travel sites gather qualitative information coming from Likert-type scales ([34]) to generate rankings. However, in the vast majority of qualitative scales, numerical values or symbols are assigned to the terms of the scales. For example, TripAdvisor uses an ordered qualitative scale formed by 5 linguistic terms, {terrible, poor average, very good, excellent}, and it assigns a numerical value from 1 to 5 to each linguistic term.

Another example is the website Booking.com. Initially, Booking.com used a 4-term qualitative scale: {poor, fair, good, excellent} in which each term was associated a numerical value: 2.5, 5, 7.5 and 10, respectively the website replaced the linguistic terms by smiley faces with the same numerical values (e.g. [36], [37], [39]).

Smiley face scales are used to make more pleasant questionnaires and surveys. These scales are very common in children surveys for measuring their pain levels or anxiety, but also smiley face scales are implemented in adult surveys for supplementing or replacing text response options in satisfaction scales (e.g. [11], [14], [42], [45]). Smiley face scales enable users to give their opinions without reading or understanding complicated linguistic terms, and they may speed processing of questions up, especially for low-literacy users. However, recently some studies have shown that visual design of surveys (symbols, numbers, and graphics) influences on response behavior (e.g. [1], [29], [50]). For instance, cultural differences in how smiley face scales are perceived or even how questions are formulated play a fundamental role in the interpretation of scales.

The effects of smiley faces, symbols and numbers on web surveys have been addressed in recent studies. For instance, some authors such as Emde and Fuchs (2012) or Gummer et al. (2019) have investigated the effects of the use of smiley face scales instead of rating scales formed by linguistic terms on response behavior in web surveys (e.g. [16]). In their experiments, they found few differences between the smiley face scales and linguistic terms. In addition, they did not find any evidence that the response burden decrease using smiley face scales.

Since items of smiley face scales can be interpreted in different ways, these scales can be considered as non-uniform. An ordered qualitative scale is non-uniform when individuals can perceive different psychological proximities between the consecutive items of the scale.

In this context of non-uniform qualitative scales, it is unadvisable to assign numerical values or scores to the linguistic terms, since these values may not adequately the individual’s perceptions of the proximities between the items of the scales. The use of numerical values converts ordinal information (linguistic terms, items or faces) into a cardinal scale producing a double negative effect. First, it assigns very often arbitrary numbers to users’ assessments, and on the other hand, it sets specific values for assessments, in way that items are provided with additional information (e.g. [20], [21], [41], [46]). As well, the outcomes coming from qualitative scales with numerical codifications can misrepresent the original ordinal information and generate meaningless and inaccurate results (e.g. [7], [21], [40], [44]). For instance, in the online customer satisfaction study presented in [22] it is shown that the choice of a certain numerical codification has a distorting effect on customer scores calculations.

Smiley face scales such as the scale of Booking.com should be treated from an ordinal point of view, taking into account how the smiley faces are perceived. However, the scoring system of Booking.com is not based on a logical ordinal approach, since it assigns numerical values to the smiley face scale for calculating final scores of accommodations.

In order to deal with ordered qualitative scales in a purely ordinal way and to avoid distorted results, we propose using the concept of ordinal proximity measure introduced by [25]. This concept takes into account psychological proximities between linguistic terms through a set of ordinal degrees. Ordinal proximity measures have been implemented to develop new decision-making procedures that employ non-uniform qualitative scales. In the group decision-making procedure introduced in [26] ordinal proximity measures play an essential role to rank order a set of alternatives that are assessed by a group of agents through a single ordered qualitative scale. In addition, ordinal proximity measures have been used in group decision-making procedures where agents can assess alternatives by means of one or two consecutive linguistic terms [23] and also in contexts where information from different ordered qualitative scales are combined [27]. Likewise, ordinal proximity measures have been recently applied in sociological contexts [28].

In this paper, we analyze the drawbacks and limitations of the scoring system of Booking.com and we propose handling its smiley face scale in an ordinal way by means of the notion of ordinal procedure introduced in [26]. This procedure avoids assigning numerical codifications to items of scales and it considers how users perceive the proximities between the items of the scale.

The rest of the paper is organized as follows. First, the paper recalls the scoring system used by Booking.com. It then presents the ordinal procedure. Subsequently, the ordinal procedure is applied to compare its results with those obtained by the scoring system of Booking.com. To do that, we have considered three different ways of understanding the proximities between the faces of the scale, that is, three different ordinal proximity measures. Finally, some concluding remarks and suggestions for future research are provided.

2 The Booking.com Scoring System

The scoring system of Booking.com begins when users receive an invitation via e-mail to rate their stay and write a review about their experiences. It is important to note that Booking.com only publishes reviews from verified customers who have booked through its website.

Figure 1: E-mail of Booking.com for collecting customers’ opinions. 

Figure 1 shows the qualitative scale that appears in the email. The scale is formed by four linguistic terms, poor, fair, good, and excellent, accompanied by a smiley face scale. When users select an option, they are redirected to the Booking.com satisfaction survey.

In the survey, users evaluate six different criteria: host(s), facilities, cleanliness, comfort, value for the money and location by means of the faces (see Figure 2). Traditionally, many studies assumed that Booking.com used a traditional scale of 0-10 or 1-10. Bjørkelund et al. (2012) detected something strange in the scale: It seems that Booking.com does not have any reviews with scores below 2.5… scores are somewhat inflated toward the higher end of the scale, but they could not to explain why the minimum score was 2.5. Lastly, Mellinas et al. (2015) showed that the rating scale used by Booking.com is actually a 2.5-10 scale (see Table 1).

Table 1: Codifications of the smiley face used by Booking.com. 

Booking.com gives the same importance to all criteria, so the accommodations’ final score is the average of the six criteria (see Figure 2).

Figure 2: Criteria rated by customers in the Booking.com satisfaction survey. 

Figure 3: Bonus questions used by Booking.com. 

As we can see in Figure 3, some additional criteria like the free WiFi access or the key collection are included in satisfaction surveys in recent years. In the case of the WiFi, this criterion appears in the information on the website (see Figure 4). However, the free WiFi access is not considered in the calculation of the overall score that is obtained taking into account only the six mentioned criteria (Figure 2).

Figure 4: Scores and criteria used by Booking.com. 

Figure 5: Qualitative scale used by Booking.com for every score range. 

Finally, Booking.com publishes the reviews of users through a different ordered qualitative scale formed by five linguistic terms, {very poor, poor, okay, good, wonderful}, for every score range. An example of this qualitative scale is shown in Figure 5.

As we have seen in Table 1, Booking.com assigns numerical values to the smiley face scale to calculate the final scores of accommodations. However, as explained in the Introduction, sometimes these numbers do not represent faithfully the perceptions of users about the proximities between the items of the scale and they can be also misleading. Concerning the scoring system of Booking.com, the score of 5 does not represent the middle point of the scale. In fact, taking into account the numerical scale {2.5, 5, 7.5, 10}, the middle score is 6.25 [30].

3 The Ordinal Procedure Versus the Scoring System of Booking.com

In this section, we propose ranking a set of hotels evaluated through the smiley face scale of Booking.com by the ordinal procedure introduced by [26] and we analyze and compare its results with those obtained using the scoring system of Booking.com. To do that, we have considered an example of seven hotels which have been assessed by a hypothetical user regarding the six criteria of Booking.com. The assessments obtained for each hotel are collected in Table 2.

Table 2: Assessments of the hotels. 

As can be seen, the assessments of the seven hotels add up 42.5 points when the numerical codifications of Booking.com are considered. Consequently, applying the Booking.com scoring system to the assessments of Table 2, we obtain the same final score for each hotel (7.08). However, the numerical codifications and the calculation of average scores seem to be inappropriate tools for ranking the hotels, since they neither consider the non-uniformity of the scale nor the heterogeneity and variety of the assessments. For example, Hotels 7 and 2 have the same final score, even though the former has received three

3.1 The ordinal Procedure

In order to avoid the arbitrary numerical codifications to the items of the smiley face scale used by Booking.com and to consider only the qualitative information provided by the scale, we now consider the above-mentioned ordinal procedure.

To apply the procedure, first, it is necessary to determine the proximities between the items on the scale. The information about the proximities is collected in an ordinal proximity measure [25] by means of ordinal degrees of proximity δ_1, δ_{2, … ,} δ_{h ,} whit δ₁ > ··· > δ_h, where δ₁ and δ_h are the maximum and minimum degrees of proximity, respectively. It is important to note that the ordinal degrees of proximity are not numbers. They only represent different levels of proximity.

Ordinal proximity measures can be determined by an expert on the field, but also asking users their perceptions about the items by means of an algorithm introduced by [24]. This algorithm provides the ordinal proximity measure for each user through an appropriate sequence of questions about the proximities between the items of the scale.

Figure 6: Non-uniform case 1. 

Since Booking.com does not provide any information about the proximities between the smiley faces, we have considered the ordinal proximity measure represented in Figure 6 to apply the procedure. As shown in Figure 6, in this ordinal proximity measure, the proximity between the first and second smiley faces is the same as the proximity between the second and third smiley faces (δ₂). This ordinal degree of proximity is greater than the proximity between the third and fourth smiley faces (δ₃), and it collects exactly the same proximity as that between the first and third smiley faces.

Once the ordinal proximity measure is determined (see Figure 6), the procedure calculates for each hotel the ordinal degrees of proximity between the assessments given by the user and the highest item of the scale,

. For instance, in the criterion host(s), hotel 1 has received the smiley face

. Considering the ordinal proximity measure represented in Figure 6, δ₃ is the corresponding ordinal degree of proximity between

and

. This way of obtaining the ordinal degrees of proximity is repeated for each hotel (see Table 3).

Table 3: Ordinal degrees of proximity. 

Next, the procedure orders the ordinal degrees of proximity in a decreasing fashion (from the highest to the lowest proximity) and selects the pair of medians (see Table 4). The idea of considering medians is also used in the Majority Judgment voting system ([4] [5] [6]

Table 4: Ordinal degrees of proximity ordered from the highest to the lowest proximity. 

Since ordinal degrees are not numbers and, therefore, it is not possible to perform algebraic operations with them, when the number of ordinal degrees of proximity is even, the median cannot be calculated as the mean of the two middlemost ordinal degrees. For this reason and in order to avoid the loss of information, the ordinal procedure considers the two medians of the ordinal degrees in the following way:

If the number of ordinal degrees of proximity is even, the procedure considers the two medians.

If the number of ordinal degrees of proximity is odd, the procedure replicates the median.

Table 5 contains the pairs of medians of the degrees of proximity for each hotel.

Table 5: Pairs of medians. 

Afterwards, the ordinal procedure ranks order the hotels taking into account the medians. To order the medians, and therefore the hotels, the procedure considers the following linear order:

(1) 

for all δ_r, δ_s, δ_t, δ_u such as r ≤ s and t ≤

This linear order is based on the idea of risk aversion:

(δ_1, δ₁) > (δ_1, δ₂) > (δ_2, δ₂) > (δ_1, δ₃) > (δ_2, δ₃) > (δ_1, δ₄) > (δ_3, δ₃) > ··· (2)

Nevertheless, it is important to note that this linear order is not the only possibility, and if deemed appropriate, it can be set other linear orders.

Taking into account the medians of the hotels collected in Table 5 and following Eq. (1), (δ_1, δ₃) > (δ_1, δ₄) > (δ_3, δ₃) > (δ_3, δ₄) > then:

Hotel 6 > Hotel 7 > Hotel 1 > Hotel 2 > Hotel 3 > Hotel 4 > Hotel 5.

Hotels 1, 2, 3 and 4 are in a tie. In these cases, the ordinal procedure applies a tie-breaking process based on [3, 4, 5]. This process consists of removing the pairs of medians that are in a tie and selecting the new medians of the remaining ordinal degrees of proximity. Then, the procedure applies again Eq. (1) to rank order the pairs of medians. The tie-breaking process is repeated as many times as necessary, until ties are broken.

After applying the tie-breaking process, the new pairs of medians for Hotels 1, 2, 3 and 4 are collected in Table 6.

Table 6: Pairs of medians of the ordinal degrees. Hotels 1, 2, 3 and 4. 

Following Eq. (1), (δ_1, δ₄) > (δ_3, δ₃) > (δ_3, δ₄), then Hotel 3 > Hotel 1 > Hotel 2 > Hotel 4.

After applying again, the tie-breaking process to Hotels 1 and 2, we obtain the new medians contained in Table 7. Since (δ_1, δ₅) > (δ_3, δ₄), then Hotel 1 > Hotel 2.

Table 7: Pairs of medians of the ordinal degrees. Hotels 1 and 2. 

Thus, the ranking obtained is:

Hotel 6 > Hotel 7 > Hotel 3 > Hotel 1 > Hotel 2 > Hotel 4 > Hotel 5.

The non-uniform case represented in Figure 6 is not the only possibility to perceive the proximities between the items of the smiley face scale of Booking.com. For instance, we can consider different non-uniform cases such as the one contained in Figure 7.

Figure 7: Non-uniform case 2. 

Figure 8: The uniform case. 

Another interesting case is the uniform one (see Figure 8). It is the most similar case to the scoring system of Booking.com, where the difference between consecutive smiley faces of the scale is the same, 2.5.

In Figures 7 and 8 are represented the proximities corresponding to these last two above mentioned cases, as well as the rankings obtained after applying the ordinal procedure.

4 Concluding Remarks

Travel websites like TripAdvisor or Booking.com are an important tool in user’s decision-making. These websites try to show the degree of satisfaction of users based on online reviews that usually display qualitative assessments alongside with the user’s comments.

The scoring system of Booking.com uses numerical codifications for calculating the final scores of accommodations. However, when users may perceive different proximities between the items of the scale, that is, when ordered qualitative scales can be perceived as non-uniform, numerical codifications may misrepresent the original ordinal information and could produce meaningless outcomes.

In this paper, we have analyzed some drawbacks and limitations of the scoring system of Booking.com and we have proposed the use of a recent ordinal procedure to deal with the smiley scale of the website. This procedure introduced by [26] ranks goods or services evaluated by means of ordered qualitative scales without assigning numerical codifications and respecting the ordinal information of the scales.

To analyze and compare both systems, we ranked seven hotels assessed through the smiley face scale of Booking.com whose scores add up the same value. The results evidenced that the scoring system of Booking.com does not always give accurate and reliable results, since sometimes it provides the same final score to all accommodations, despite the diversity and heterogeneity of the assessments. In addition, the different ways of ranking the hotels stressed the importance of considering the non-uniformity and considering the proximities between the smiley faces in assessment processes.

Considering how users perceive the proximities between the items of the scales is one of the main advantages of the ordinal procedure. This knowledge of users' perceptions enables the scoring systems to apply ways to adequately process and aggregate qualitative data improving analyses of the on-line reviews. One limitation of the procedure would be the determination of the ordinal proximity measure that properly captures users' perceptions. In [24] has been introduced an algorithm that generates a metrizable ordinal proximity measure through sequences of questions about the proximities between the terms of the scale.

On the other hand, it is important to take into account the importance of criteria in final scores. The scoring system of Booking.com uses the average of the scores to provide the overall score of accommodations, that is, it gives the same importance to the six criteria, but it does not consider if users attach different importance to each criterion influenced by their subjective attitudes and preferences. For instance, some users can consider location more important than facilities in their reviews (e.g. [12], [13], [15], [19], [31], [43]).

The different importance of the criteria might also address in future studies from an ordinal point of view using the ordinal procedure considered in this paper. To apply the procedure in this context, weights of the criteria are managed replicating the corresponding assessments according to the corresponding weights of each criterion [26].

Regarding future research, we intend to complete our analysis with more data obtained from different travel websites for examining which factors influence hotel selection. Likewise, it could be interesting to use ordinal proximity measures to development new procedures for evaluating the hotel competitiveness that allow them to identify their strengths and weaknesses.