Evaluating Usability of Academic Websites through a Fuzzy Analytical Hierarchical Process

Abstract

In the higher education sector, there is a growing trend to offer academic information to users through websites. Contemporarily, the users (i.e., students/teachers, parents, and administrative staff) greatly rely on these websites to perform various academic tasks, including admission, access to learning management systems (LMS), and links to other relevant resources. These users vary from each other in terms of their technological competence, objectives, and frequency of use. Therefore, academic websites should be designed considering different dimensions, so that everybody can be accommodated. Knowing the different dimensions with respect to the usability of academic websites is a multi-criteria decision-making (MCDM) problem. The fuzzy analytic hierarchy process (FAHP) approach has been considered to be a significant method to deal with the uncertainty that is involved in subjective judgment. Although a wide range of usability factors for academic websites have already been identified, most of them are based on the judgment of experts who have never used these websites. This study identified important factors through a detailed literature review, classified them, and prioritized the most critical among them through the FAHP methodology, involving relevant users to propose a usability evaluation framework for academic websites. To validate the proposed framework, five websites of renowned higher educational institutes (HEIs) were evaluated and ranked according to the usability criteria. As the proposed framework was created methodically, the authors believe that it would be helpful for detecting real usability issues that currently exist in academic websites.

1. Introduction

Educational practices have changed with contemporary technological advancement, which has improved the educational process in a positive way, especially in the higher education sector. Academic websites play an important role in promoting education to everybody. These websites enable higher educational institutes (HEIs; which are traditionally universities) to offer digital academic services to users in order to save time and resources [1]. Secondly, there is an international trend in HEIs to display themselves through their websites for research collaboration with other national and international universities and industries [2]. Currently, academic websites are not just platforms to display the programs, features, and facilities of a given HEI. In addition to this, such websites are extended with a number of different business processes and functions. For instance, the online admission process, access to the learning management system, enrolment, hostel management systems, and many other administrative operations are carried out through academic websites. As a result, academic websites have become web-based systems of a great size and complexity, operated by a variety of users. In the wake of coronavirus disease 2019 (COVID-19), online learning is even being planned by HEIs in developing countries, which has further greatly increased the importance of academic websites. As these websites have become an essential part of every HEI’s ability to perform different academic operations, the success of each educational organization—as well as the relevant users—depends on the user-friendliness, efficiency, and effectiveness of its academic website [3]. For the successful design of a website, it is important to understand the target users and meet their needs and expectations. Ironically, the users are the least likely group to be available and involved during the designing and development of the websites [4]. User diversity represents a hindrance in the effective and efficient use of academic websites that have not been designed as per the unique needs of the individuals who will use them. Consequently, the majority of users experience difficulty while attempting to work with these websites. Therefore, many users, after failing to perform the intended operation with such platforms, become frustrated and seek outside assistance. Thus, there is a great need to evaluate academic websites in order to pinpoint the usability problems that impede the user’s performance when using these websites [3]. To identify the usability problems and to improve the design of academic websites, an evaluation of academic websites is vital.

Mentes and Turan conducted an empirical evaluation of an academic website and pointed out that usability is key to allowing communication between the university and its users, which will eventually lead HEIs towards successful governance [3]. Lyla Hasan, after a heuristic evaluation of three university websites, indicated that the academic websites exhibited numerous usability problems related to content, navigation, language, inconsistent and incomplete information, and inappropriate design, which impeded the effective and efficient use of these websites [5]. It has been revealed through research that all HEI websites do not present academic operations and information in a consistent manner. For example, some websites present all of the required information online and provide students with the option to perform all operations electronically, whereas other websites only allow students to print forms and submit them in person or via postal mail [2]. The usability issues become worse when websites are aimed at offering each academic operation digitally. Generally, usability problems have a greater impact over the success and failure of software systems. It has been shown through several studies that 80% of the total maintenance costs are associated with the user’s problems with the software, rather than with technical defects. Furthermore, among these problems, 64% are usability problems [6]. Usability problems obstruct users in accomplishing their tasks/goals with software systems in an effective and satisfactory manner. Among software, websites are more vulnerable to usability problems, because websites are information-oriented and involve relatively more interactivity. The usability of websites is therefore crucial. It has been established that poor usability damages the website’s overall credibility, which results in the user losing confidence in the respective website. It has also been pointed out that usability is related to the part of the development process that focuses on developing a usable interface [7].

According to ISO 9241-11, usability is defined as the extent to which a product can be used by specified users in order to achieve specified goals with effectiveness, efficiency, and satisfaction. Applying such a definition to web usability, means that the design of website should be easy to use and understand [8]. The successful design of academic websites may involve multiple criteria that need to be identified by primary users of such websites, so that an evaluation framework can be proposed. Initially, websites of business organizations have been investigated to answer the questions: how they can evaluate the websites, and what are the basic determinants of these websites? Subsequent researchers have emphasized the need to evaluate academic websites to verify its effectiveness, accessibility, and usability [4,9]. To this end, many studies have been conducted to investigate whether the design of academic websites is effective at presenting academic information and operations. For example, some of the researchers have evaluated academic websites to assess their effectiveness using the Likert scale. Some developed guidelines to achieve usability, and others have identified usability problems through heuristic evaluation. However, they did not use any rigorous methods to evaluate the usability of academic websites; conversely, they employed simple evaluation mechanisms. Some researchers have also used automated tools and algorithms to efficiently evaluate the usability of a large number of academic websites. However, there is no specific decision-making model introduced considering users’ perspective to evaluate academic websites. Thus, the main scope of this work is to present a decision-making model for the usability evaluation of academic websites.

Considering that the evaluation of a website is a multi-dimensional problem that is based on several different criteria and the point that there is a paucity of usability evaluation models with a strong theoretical background for academic websites, this work has taken into account the multi-criteria decision-making (MCDM) method, followed by a detailed literature review, in order to propose a model for the usability evaluation of academic websites. In general, MCDM methods serve the process of making decisions when there are multiple-criteria or factors. The analytic hierarchy process (AHP) [10] is one of the decision-making methods of MCDM that is undoubtedly good at making decisions through pairwise comparisons of qualitative and quantitative factors. AHP presents an excellent procedure to compute the weight of the criteria utilized in experts’ reasoning processes [11]. However, the issue with traditional AHP is that it cannot completely reflect the vagueness that exists in human judgement. As a result, the criteria cannot be provided with precision. Moreover, AHP methodology is unable to remove decision-makers’ bias. Human decisions suffer from various inabilities like impreciseness, uncertainty, and inability to provide accurate deployment of Saaty’s scale while making the pairwise comparison. Furthermore, the decision may be influenced by decision-makers’ personal preferences. To overcome these limitations, AHP needs to be replaced with the fuzzy analytic hierarchy process (FAHP). FAHP is capable of overcoming such limitations. It can provide much-needed flexibility and freedom to DM while making a judgment. FAHP takes care of linguistic vagueness and uncertainty while making a decision. FAHP has been employed in many MCDM problem-solving applications. It is a powerful method for resolving imprecise information. It is also a convenient methodology where experts are more able to provide consistency in their judgement. When compared with other methods, FAHP involves simple steps to derive the weight as opposed to other methods like fuzzy technique for order performance by similarity to ideal solution (TOPSIS), fuzzy best worst method (FBWM), and fuzzy characteristic objects method (FCOMET). FTOPSIS needs more computational steps compared with FAHP for deriving the weights, whereas FBWM involves the formulation and solution of linear programming equations to derive the weights [12]. Moreover, FCOMET compares the characteristic objects instead of the alternatives to derive the weights. However, various methods based on fuzzy theory have been opted for in the literature. Here, considering the high analysis accuracy attainable by FAHP, as well as the time constraints and ease of the convenience for researchers and experts, the FAHP method was adopted.

The rest of the paper is organized as follows: Section 2 presets the relevant literature for this study. Section 3 is the research methodology. Section 4 presents the findings and discusses the results. Section 5 presents the ranking of websites. Section 6 gives the sensitivity analysis. Section 7 discusses the limitation of the present study. Finally, Section 8 provides the conclusion and future work.

2. Literature Review

For the last two decades, the Internet has grown tremendously and has given rise to powerful communication mechanisms that facilitate the flow of educational information and the digital processing of academic operations. Currently, HEIs mostly rely on academic websites, because these are a rapid and reliable medium to offer educational information and facilities to users. Academic websites can be defined as a platform that provides institutional information, opportunities for students, and educational facilities and industry cooperation [13]. Academic websites can only make HEIs efficient and practicable if their design supports usability aspects. Users will be satisfied if the websites fit the required functionality with desired usability characteristics [9]. So, measuring the usability of academic websites is crucial, and numerous researchers have made efforts in such a regard.

2.1. Usability Evaluation Studies

Astani and Elhindi evaluated the websites of the top 50 US-based universities in order to assess their effectiveness. The websites were evaluated using a list of predefined website characteristics that were rated based on a five-point Likert scale. However, they did not use any rigorous methods for evaluation; instead, a simple mechanism was used to evaluate academic websites [4]. Other researchers have utilized heuristic evaluations, such as Kostaras and Xenos, who applied heuristics to assess the website of the Hellenic Open University [14]. Similarly, the Lund University website was also evaluated through heuristics. Based on the feedback of experts, suggestions have been presented to improve the usability of the website [15]. Alotaibi research presented an empirical application of heuristic rules in order to test the usability of the websites of universities in the Kingdom of Saudi Arabia [16]. Another recent study conducted a heuristic evaluation of 24 academic websites through data mining techniques. A tool called Prometheus integrated ten heuristics in order to generate usability evaluations. The results collected in this process provided feedback for the industry to redesign and restructure academic websites so that they would comply with usability standards [17]. Another study used SortSite, an automated tool to evaluate the usability and accessibility conformance of ten randomly selected websites of public sector universities in Nigeria. The results showed that the evaluated websites did not conform to the implementation of the web content accessibility guidelines (WCAG) and usability guidelines, making it difficult to access and navigate the academic website; therefore, improving the usability by redesigning such websites was recommended [18]. Another study evaluated the website of the University of Putra Malaysia (UPM) using a questionnaire consisting of five factors of usability, namely attractiveness, controllability, helpfulness, efficiency, and learnability. Around 364 students responded to the questionnaire, and the findings highlighted the usability of the website good in terms of controllability, helpfulness, and efficiency, and poor in terms of attractiveness and learnability. However, the scope of the research was limited to a single website [19]. Caglar and Mentes investigated the usability of the European University of Lefke website using a website analysis and measurement inventory (WAMMI) consisting of 20 questions. The responses were collected from 293 students, and the findings revealed the dissatisfaction of users regarding the usage of the website [20]. Adepoju and Shehu conducted a study to figure out the usability level of websites of federal universities in Nigeria through accessibility evaluation. They utilized automated tools including HERA, WAVE, and Web accessibility checker to examine the conformity of the websites to WCAG. The results showed a poor level of usability, as well as the presence of numerous accessibility errors in these websites [21]. Another research evaluated the websites of Jordan’s universities from a usability perspective using two online automated tools and a questionnaire. The questionnaire responses were collected from 252 users of nine different Jordanian universities. The results showed that the overall usability level of these websites was acceptable, but there were many usability and interface design issues that needed to be addressed [22]. Rahman and Ahmed evaluated the website of the University of Dhaka using a survey method. A seven-point Likert scale questionnaire was designed considering items from various published instruments, such as the questionnaire for user interaction satisfaction (QUIS), WAMMI, and joint information systems committee (JISC) checklist for academic websites. A total of 864 students from different faculties of the university participated in the survey. The results revealed students’ dissatisfaction with the university websites, mainly owing to a lack of content and updates [23]. A statistical study was conducted to evaluate the usability and accessibility of three popular academic websites. Two evaluation techniques—questionnaire and performance-based evaluations—were employed in this study. The results showed a good level of usability [24]. Another study was conducted to analyze the navigational efficiency, organizational content, and user satisfaction from the website of the University of Hawaii. The feedback showed that the website’s user interface needed improvement [25]. Benaida and Namoun investigated the effect of four key factors, including the usefulness, interface quality, content, and satisfaction of users on the perceived usability of four Algerian academic websites. The IBM Computer System Usability Questionnaire (CSUQ) was used to collect data from 200 students of four Algerian universities. The results imply significant usability improvements are needed for the four factors explored in the study [26].

An empirical investigation of the Benue State University website from the students’ perspective was conducted using a standardized usability questionnaire known as WAMMI. The results revealed usability issues associated with the website, and recommendations were given to improve the website interface [27]. Another research stressed that the quality of websites is associated with MCDM issues. They investigated the performance of two algorithms, namely a linear weightage model (LWM) and simple additive weighting (SAW), in the ranking of university websites using five usability criteria—load time, page rank, traffic, stickiness, and backlink [28]. A recent study quantitatively evaluated the usability of 50 academic websites using a System Usability Scale (SUS) consisting of 10 questions. The data were collected from 600 participants, including both end-users (i.e., students) and experts (i.e., with software design expertise). The results indicated that end-users experienced more design and usability problems than experts [29].

The issue of website usability evaluation has been considered to be of great importance, as many researchers have attempted to contribute in this regard. However, academic websites are mainly evaluated using heuristic rules, questionnaires, and through some automated tools. In most of the cases, the evaluated websites were found to be poor in terms of usability, which indicates that while developing, designers put a lot more effort into the technology, organizational structure, and business objectives of the university rather than usability aspects [30]. Researchers have given recommendations to overcome usability issues and to improve the interface design of academic websites. However, the recommended guidelines are not general—they are specific to the academic websites evaluated by the different researchers. Thus, these guidelines cannot be presented as a framework or website design model to the software industry.

Although numerous evaluation studies have been conducted, there is still the need to fully comprehend the usability problem as having multiple-dimensions, as well as the factors that make them complex. The usability issue cannot be completely understood through human judgement or questionnaires, which are merely based on simple guidelines of usability. Usability is a multi-criteria decision-making (MCDM) problem, and to understand it, there is a need to investigate the criteria (dimensions) and sub-criteria (factors) upon which it depends. There are several MCDM approaches that have been presented in the literature including AHP, analytical network process (ANP), technique for order performance by similarity to ideal solution (TOPSIS), preference ranking organization METHod for enrichment evaluation (PROMETHEE), grey relation analysis (GRA), and data envelopment analysis (DEA) [31] [32]. To the best of our knowledge, there has been no study conducted to evaluate the usability of academic websites using FAHP. Our study utilized AHP to conduct an evaluation of academic websites, owing to the good strength of this technique for making decisions, using a pairwise comparison of qualitative, quantitative, and uncertain factors. AHP was integrated with a fuzzy method to further strengthen the AHP, as shown in previous research studies [33]. FAHP keeps the process simple, even when the number of alternative cases increases [34,35]. FAHP can help find more decisive results by replacing the membership scales with Saaty’s scale (1–9) and weighting them in the presence of uncertainties. Because of the fuzzy nature of the comparison process, it is useful for experts to show interval judgement rather than making fixed value judgement [12].

2.2. Factors Affecting the Usability of Academic Websites

The literature was reviewed in detail so as to identify the numerous key factors that suggest a successful design for academic websites. These factors can be considered significant as a mechanism to evaluate the usability of academic websites.

Table 1. Factors affecting usability of academic websites.

Authors/Researchers	Kokila Harshan Ramanayaka (2018)	Katerina Kabassi (2018)	Hmood Al-Dossari (2017)	Ratnakar Kumar (2017)	Sahar A. El Rahman (2016)	Sharmistha Roy (2016)	Deepti Mehrotra (2015)	Nagpal et al. (2015)	Renuka Nagpal (2013)	Layla Hasan (2013)	Ali H. Al-Badi (2013)	Saqib Saeed (2013)	Mifta Manzoor (2012)
References	[34]	[36]	[37]	[38]	[39]	[40]	[41]	[42]	[43]	[44]	[45]	[46]	[9]
Memorability			✓								✓
Learnability	✓		✓			✓			✓		✓	✓
Efficiency	✓		✓			✓					✓
Effectiveness	✓
User satisfaction	✓	✓	✓						✓
Ease of navigation	✓	✓		✓	✓	✓	✓	✓	✓	✓		✓	✓
Response time		✓				✓	✓	✓	✓
Content	✓	✓						✓		✓
Design		✓		✓								✓	✓
Ease of use		✓				✓	✓	✓	✓	✓		✓
Interactivity	✓	✓			✓
Informative	✓	✓					✓	✓	✓
Security		✓							✓
User friendliness		✓	✓		✓				✓
Accessibility	✓			✓	✓				✓				✓
Few errors			✓								✓
Reliability		✓
Loading speed		✓		✓				✓
Personalization		✓							✓
Accuracy		✓			✓	✓						✓
Attractiveness		✓				✓			✓		✓	✓
Readability				✓	✓

presents these usability factors identified through the literature review.

3. Research Methodology

This study used a multi-method and multi-step approach to achieve the main research objective, which is the evaluation and calculation of the local and global weights of the usability factors to rank academic websites on the basis of these factors. In the first step, a detailed literature review was conducted to identify and classify the most important factors that are frequently used by researchers to evaluate the usability of websites. Second, the identified usability factors were evaluated through a survey method to determine the importance of each factor. In the third step, the FAHP technique was used to calculate the local and global weights of the most important usability factors. In the fourth step, the academic websites were ranked and evaluated using the fuzzy extent analysis method. The global weights of the factors and the fuzzy weights of the pairwise comparison were calculated and multiplied finally to get the priority weight vector, which was used to rank the academic websites. Finally, a usability evaluation framework was proposed. The graphical representation of the research methodology is depicted in Figure 1.

A detailed discussion about the phases of the proposed research methodology is discussed below.

3.1. Identify Relevant Usability Factors (Phase 1)

To identify the factors affecting the usability of academic websites, a detailed literature review was conducted using different sources, such as Google Scholar (https://scholar.google.com.pk/ (accessed on 20 August 2020)), Science Direct (https://0-www-sciencedirect-com.brum.beds.ac.uk/ (accessed on 20 August 2020)), Research Gate (https://www.researchgate.net/ (accessed on 20 August 2020)), and IEEE Explore (https://ieeexplore.ieee.org (accessed on 20 August 2020)). As a result, a number of usability factors were identified, as shown in Table 1.

3.2. Determine the Importance of Usability Factors (Phase 2)

To further determine the importance of the identified usability factors with respect to academic websites, the opinions of the primary users of such websites needed to be elicited. To achieve this, a survey study was conducted, because it is an effective way to collect data from a huge population geographically located in different places. The data were collected through a Likert scale questionnaire consisting of a set of questions formulated to evaluate the importance of the usability factors identified during Phase 1. The Likert scale questionnaire had five choices, ranging from “very important” to “not important”. The numeric values ranging from 1 to 5 were assigned to each choice of the scale, where 5 was assigned to very important and 1 to not important. The participants of the survey were asked to express their opinions using five choices of scale against the relevant usability factors after visiting the selected academic websites. The participants’ response was based on their experience developed during the usage/exploring of the website. The SPSS 26.0 software was used to analyze the collected data.

The participants of the survey included students, faculty, and staff, as shown in

Table 2. Demographic profile of participants.

Demographic	Frequency	%
Male	106	60%
Female	70	40%
Qualification	Frequency	%
Undergraduate	98	56%
Graduate	27	15%
Postgraduate	21	12%
Profession	Frequency	%
Academic staff	14	8%
Faculty	16	16%

. The sample consists of 176 subjects including students, faculty, and staff. The demographic profile of participants was as follows.

The sample involved both male and female participants using academic websites to perform different academic activities. There were 106 (60%) male participants and 70 (40%) female participants. In terms of qualification, 98 (56%) participants were undergraduates, 27 (15%) were graduates, and 21 (12%) were postgraduates. Among them, 146 (83%) were students, 14 (8%) belonged to academic staff, and the remaining 16 (9%) were faculty.

To select an appropriate sample, a non-probability technique, also known as snowball sampling, was used. The technique starts with the selection of a subject possessing the desired characteristics. Then, the social network of the selected subject is utilized to select further participants. This technique is also called chain sampling as the correctly identified sample helps researchers find further like-minded participants. In this sampling technique, a sample size increases gradually, like falling snow [47,48].

3.3. Prioritizing Usability Factors (Phase 3)

During phase three, the FAHP was applied to calculate weights of the criteria (aka dimensions) and sub-criteria (aka factors) shortlisted during phase 2. Thus, another questionnaire was designed to make a pairwise comparison of the criteria and sub-criteria. The FAHP questionnaire was distributed among those 50 participants of original sample who had at least five years of experience using academic websites. To make a pairwise comparison between two usability factors, a fuzzy scale based on TFN (as shown in Figure 2) was used.

Table 3. Triangular fuzzy numbers.

Linguistic Variables	Triangular Fuzzy Numbers	Reciprocal Triangular Fuzzy Numbers
Equally important	(1,1,1)	$\left(\frac{1}{1},\frac{1}{1},\frac{1}{1}\right)$
Weakly important	(1,3,5)	$\left(\frac{1}{5},\frac{1}{3},\frac{1}{1}\right)$
Essentially important	(3,5,7)	$\left(\frac{1}{7},\frac{1}{5},\frac{1}{3}\right)$
Strongly important	(5,7,9)	$\left(\frac{1}{9},\frac{1}{7},\frac{1}{5}\right)$
Absolutely important	(7,9,9)	$\left(\frac{1}{9},\frac{1}{9},\frac{1}{7}\right)$
Intermediate values	(7,8,9), (5,6,7) (3,4,5), (1,2,3)	$\left(\frac{1}{9},\frac{1}{8},\frac{1}{7}\right),\left(\frac{1}{7},\frac{1}{6},\frac{1}{5}\right)$ $\left(\frac{1}{5},\frac{1}{4},\frac{1}{3}\right),\left(\frac{1}{3},\frac{1}{2},\frac{1}{1}\right)$

represents the TFN and reciprocal TFN. All of the participants responded to the questionnaire, but only 37 were complete and correct, so the rest of the responses were excluded from the process.

3.4. Fuzzy Set Theory and Fuzzy AHP Methodology

The triangular fuzzy number (TFN), as shown in Figure 3, was employed for various arithmetic operations [49]. S₁ and S₂ may be expressed as $\left(a_1,b_1, c_1\right)$ and $\left(a_2,b_2, c_2\right)$, respectively.

The various fuzzy operations using fuzzy set theory are as follows:

$${\tilde{S}}_1\oplus {\tilde{S}}_2=\left(a_1+a_2,b_1+b_2, c_1+c_2\right)$$

(1)

$${\tilde{S}}_1\ominus {\tilde{S}}_2=\left(a_1-a_2,b_1-b_2, c_1-c_2\right)$$

(2)

$${\tilde{S}}_1\otimes {\tilde{S}}_2=\left(a_1{a}_2,b_1{b}_2, c_1{c}_2\right)$$

(3)

$$\lambda \otimes S_1=\left({\lambda }_1{a}_1,{\lambda }_1{b}_1, {\lambda }_1{c}_1\right) \mathrm{where} \lambda >0, \lambda \in R$$

(4)

$${\tilde{S}}_1^{-1}=\left(\frac{1}{c_1},\frac{1}{b_1},\frac{1}{a_1}\right)$$

(5)

The principles of the extent analysis [50] for the pairwise comparison of two triangular fuzzy numbers (TFNs) were used in this research. We may consider the objective of a goal set as Y = {$y_1$,$y_2$,………,$y_n$} and X = {$x_1$,$x_2$,………,$x_m$}, respectively. It derives the m extent analysis values for each object as follows:

$$N_{gi}^1,N_{gi}^2\dots N_{gi}^m, i=1,2,\dots ,m$$

(6)

where $N_{gi}^j \left(j=1,2,\dots m\right)$ are TFNs and are represented as (a,b,c). Chang’s extent analysis procedure [50] is described below:

Step 1: Compute fuzzy extents synthetic value ($S_i$)

$$S_i= {\sum }_{j=1}^m{N}_{gi}^j\otimes {\left[{\sum }_{i=1}^n{\sum }_{j=1}^m{N}_{gi}^j\right]}^{-1}$$

(7)

$${\sum }_{j=1}^m{N}_{gi}^j=\left({\sum }_{j=1}^m{a}_j, {\sum }_{j=1}^m{b}_j, {\sum }_{j=1}^m{c}_j\right)$$

(8)

$${\sum }_{i=1}^n{\sum }_{j=1}^m{N}_{gi}^j=\left({\sum }_{i=1}^n{a}_j, {\sum }_{i=1}^n{b}_j, {\sum }_{i=1}^n{c}_j\right)$$

(9)

$${\left[{\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=1}^m{N}_{gi}^j\right]}^{-1}= \left(\frac{1}{{\sum }_{i=1}^n{a}_i}, \frac{1}{{\sum }_{i=1}^n{b}_i},\frac{1}{{\sum }_{i=1}^n{c}_i}\right)$$

(10)

where the lower limit value and upper limit value are denoted by c and a, respectively, whereas b provides the most promising value.

Step 2: Compute and compare fuzzy values

The degree of possibility of $S_2=\left(a_2,b_2, c_2\right)$ $\ge S_1=\left(a_1,b_1, c_1\right)$ which is defined as:

$$V\left(S_2\ge S_1\right)=sup\left[min({\mu }_{N_1}\left(x\right), {\mu }_{N_2}\left(y)\right)\right], y \ge x$$

(11)

where (x, y) represent the membership function value of each criterion. $S_1 \mathrm{and} S_2$ are fuzzy number, and are represented as follows:

$$V\left(S_2\ge S_1\right)=hgt \left(S_1{{\displaystyle \cap }}^{ }{S}_2\right)={\mu }_{S_2}\left(q\right)$$

(12)

where the highest intersection point (Q) between ${\mu }_{S_1} \mathrm{and} {\mu }_{S_2}$ gives the ordinate q (Refer Figure 4). Using the Equation (13), ${\mu }_{S_2}\left(q\right)$ may be calculated

$${\mu }_{S_2}\left(q\right)= \{\begin{array}{cc}0& if b_2\ge b_1\\ 1& if a_1\ge c_2\\ \frac{a_1-c_2}{(b_2-c_2)-\left(b_1-a_1\right)}& otherwise\end{array}$$

(13)

The values of $V\left(S_1)\ge V (S_2\right)$ and $V\left(S_2)\ge V (S_1\right)$ are compared to the values of $S_1 \mathrm{and} S_2$.

Step 3: Compute priority weight

Hence, the degree of possibility of a convex fuzzy number that is greater than the p convex fuzzy number is given by S_i = (i = 1,2,3,…,p)

$$\begin{gathered} \mathrm{V}\left(S\ge S_2,S_2,\dots ,S_p\right)=\mathrm{V}\left[\left(S\ge S_1\right),\left(S\ge S_2\right), \dots ,\left(S\ge S_p\right)\right] \\ =\mathrm{minV}(S\ge S_i) \mathrm{i} = 1,2,\dots ,p \end{gathered}$$

(14)

The equation of the weight vectors may be given as follows:

$$d^{\prime }\left(A_i\right)=\min V\left(S_i\ge S_p\right) for p=1,2,\dots ,m;p\ne i$$

(15)

Step 4: Compute normalized weight vector

$$W={\left(d\left(A_1\right),d\left(A_2\right),\dots \dots \dots , d\left(A_n\right)\right)}^T$$

(16)

Step 5: Sensitivity analysis of factors

A sensitivity analysis was performed to figure out the top ranked alternatives affected by the criteria. An important method was to observe the changes of the output by dropping one of the criteria at a time from the decision matrix [51].

3.5. Ranking Academic Websites (Phase 4)

Similar to the calculation made by the globally weigh the factors, the fuzzy weights of the pairwise comparisons of five websites were calculated. Same participants were requested to respond to the questionnaire so as to make a pairwise comparison of websites. The pairwise comparisons were aggregated and calculated, and the summed weights of websites were evaluated using the fuzzy extent analysis method. Furthermore, the global weights of the criteria and the fuzzy summed weights were multiplied with each other in order to get the priority weight vectors, and on that basis, the websites were ranked.

4. Findings and Discussion

The findings related to the above discussed phases are described below.

4.1. Identified Usability Factors

The detailed literature review of Phase 1 resulted in the extraction of a large number of pertinent usability factors which are key to evaluate the usability of academic websites. The identified factors were further filtered out using the threshold value. For example, the factors that appeared three or less than three times were discarded and considered to be less important, while the rest of the factors are shown in

Table 4. Identified usability factors.

Factors	Frequency	References
Ease of navigation	11	[9,34,36,38,39,40,41,42,43,44,46]
Ease of use	7	[36,40,41,42,43,44,46]
Learnability	6	[34,37,40,43,45,46]
Response time	5	[36,40,41,42,43]
Informative	5	[34,36,41,42,43]
Accessibility	5	[9,34,38,39,43]
Attractiveness	5	[36,40,43,45,46]
Efficiency	4	[34,37,40,45]
User satisfaction	4	[34,36,37,43]
Content	4	[34,36,42,44]
Design	4	[9,36,38,46]
User friendliness	4	[36,37,39,43]
Accuracy	4	[36,39,40,46]
Interactivity	3	[34,36,39]
Loading speed	3	[36,38,42]

. The remaining important usability factors were studied and categorized into three major dimensions, including usability, navigation and content.

The descriptions of the important usability factors are as follows:

Ease of Navigation: The academic websites should be easy to navigate so that students may swiftly locate the relevant information that interests them [23].

Ease of Use: The design of the website should be consistent so that it can be easily used by the students [43].

Learnability: The design of the academic website should allow students to quickly become familiar and easily learn how to perform different academic tasks through the website [37].

Response Time: The website that runs immediately after clicking on any control indicates that it has a fast average response time [43].

Informative: The academic website should contain the latest and detailed information [43].

Accessibility: The website should be accessible anytime from anywhere [34].

Attractiveness: The information on the academic website should be well organized [36].

Efficiency: How quickly a student can perform an academic operation after learning to operate the academic website [37]?

User Satisfaction: The interface design allows users to perform each academic activity with ease [37].

Content: The content relevant to the purpose of the academic activity in the form of text, image, and graphics should be available on the website [42].

Design: The overall interface, layout, and structure of the academic website should be well designed [9,43].

User Friendliness: The academic website should be user-friendly to the visitors; this means that the interface should not be difficult for the user to operate. [39,43].

Accuracy: The information should be updated and correct [36].

Interactivity: The academic website should offer a dialog with students in the form of appropriate messages, feedback, and hints to assist users when performing activities [34].

Loading Speed: There should be little time between web page requests and having pages presented on the computer screen [42].

4.2. Analysis of Usability Factors

During Phase 2, a survey was conducted to weigh the significance of each usability factor on a five-point Likert scale. The survey questionnaires were distributed among 176 participants (see Table 2) who were personally approached (via telephonic conversation and face to face meetings when possible). The objectives of the study and the value of their responses in formulating a framework to evaluate the usability of academic websites were illustrated. Along with a Google form of the survey, a separate, easy to read, and understandable tutorial was shared with them. The tutorial provided a clear and concise description of each factor so that participants could apprehend well before making a response. All of the participants responded and their responses were found to be complete and correct, so the entire data were employed in the further analysis. A total of 15 factors were evaluated, the data obtained using the questionnaires were analyzed using SPSS 26.0, and the results are shown in

Table 5. Analysis of data collected through the questionnaire.

Sr. No	Factors	Result (N = 176)
		Mean	Standard Deviation (SD)
1	Ease of navigation	4.6	0.8
2	Ease of use	4.6	0.8
3	Informative	4.6	0.9
4	Accessibility	3.5	1
5	Learnability	3.4	1.1
6	Accuracy	3.4	1
7	Efficiency	2.9	0.8
8	Interactivity	2.9	0.7
9	User satisfaction	2.8	0.8
10	User friendliness	2.4	0.9
11	Response time	2.3	1.7
12	Attractiveness	2.2	0.8
13	Content	1.8	1.2
14	Design	1.8	1.2
15	Loading speed	1.8	1.2

.

The results in Table 5 show that ease of navigation was recognized as the most important factor, holding a mean value 4.6. On the other hand, loading speed was considered the least important factor, bearing a mean value of 1.8. The MCDM methods compared limited factors (most important) with each other in order to determine their relative significance with respect to specified goals, whereas the detailed literature review produced a large number of factors that needed to be reduced. Thus, for further analysis, only those factors that had a mean value greater than or equal to 2.5 were considered, the rest of the factors were discarded. For that, the cutoff point needed to be selected on the basis of expert choice (in this case experts are those decision makers (DMs) who made a correct judgement based on the qualification, knowledge, and experience they had in MCDM techniques). Based on the expert opinion, a mean value of 2.5 was considered to be the most suitable threshold value, because if we set it as less than 2.5, very few factors were likely to be eliminated, and if the set cutoff point was greater than 2.5, then most of the important factors were believed to be eliminated. After filtering out using the threshold value, ten usability factors were qualified for a detailed analysis. Such factors included ease of navigation, ease of use, informative, accessibility, learnability, accuracy, efficiency, interactivity, and user satisfaction. A hierarchal structure was designed by considering factors such as criteria to measure the usability of university websites. The three dimensions of these usability measures were usability, navigation, and content.

4.3. Prioritization of Usability Factors

In Phase 3, the obtained fuzzy weights and key usability factors were ranked using the FAHP method. The FAHP activities were performed using MS Excel, and the implementation process and findings are discussed below. The first step in FAHP was to define the goal of the research. The goal of this research was to propose a hierarchical model to evaluate the usability of academic websites. The findings of Phases 1 and 2 are shown in Table 4 and Table 5, respectively, and revealed the key usability factors for the evaluation of academic websites. The major dimensions (aka criteria) of the usability evaluation for academic websites included usability, navigation, and content, which encompassed total 15 factors (aka sub-criteria). Out of all 15 factors, only nine factors were taken into account in Phase 3, and the remaining six factors, with mean values less than 2.5, were excluded.

The excluded factors were user friendliness, response time, attractiveness, content, design, and loading speed. The remaining nine factors along with their dimensions were used to develop the hierarchical model shown in Figure 5.

The next most important step was the pairwise comparison, where experts (i.e., those who have years of user experience in academic websites) had to agree on the index values entered into the pairwise comparison matrices. Experts were asked to fill out a pairwise comparison matrix that compared the significance of the three dimensions with respect to each other, as per the scale shown in Table 3. In the same way, experts had to complete pairwise comparison matrices that compared the significance of the factors of each dimension to each other. Towards this end, a questionnaire was developed and distributed among participants who had at minimum three years of experience using academic websites. The questionnaire was distributed among 50 participants, including academicians, senior students, and other academic staff. All participants reacted to the questionnaire, which was further processed to check the consistency ratio. Out of 50 responses, only 37 responses were found to be valid because they had a consistency ratio of less than 10%. Such responses were used to compute the fuzzy weights of each dimension and their related factors using Microsoft Excel.

4.4. Determining Local and Global Weights of Criteria

The responses given by experts were aggregated using the fuzzy extent analysis methodology, as described in the previous section. The normalized weights obtained were the local weights of the criteria and sub-criteria. Each local weight of the sub-criteria was multiplied by the weight of the criteria in order to obtain the global weight of that criteria. In this way, the global weights of the usability dimension were computed by multiplying the fuzzy weights of the factors corresponding to each dimension. The local and global weights of each criterion and sub-criteria are shown in

Table 6. Local and global weights for criteria using fuzzy analytic hierarchy process (FAHP).

Criteria	Weights of Criteria	Local Ranks of Criteria	Sub-Criteria	Local Weights of Sub-Criteria	Local Ranks of Sub-Criteria	Global Weights of Criteria	Global Ranks
Support	0.641	1	Ease of use	0.61	1	0.391	1
			Interactivity	0.177	3	0.114	4
			Learnability	0.213	2	0.137	2
Organization	0.194	2	Ease of navigation	0.649	1	0.126	3
			Accessibility	0.164	3	0.032	8
			Efficiency	0.187	2	0.036	6
Content	0.164	3	Informative	0.635	1	0.104	5
			Accuracy	0.158	3	0.026	9
			User Satisfaction	0.207	2	0.034	7

. Ease of use was ranked the most important, followed by learnability and then the rest of the criteria. These global fuzzy weights of the criteria were further used for alternative website comparisons.

On the basis of the usability evaluation criteria found after a rigorous process, the model below was proposed to assess the usability level of the academic websites.

To the best of our knowledge, this research is the very first that identified and prioritized the crucial usability dimensions and factors, taking into account the user perspective, through a robust approach. Previous research presented many studies that have been conducted to evaluate the usability of academic websites. The literature review section presents the details of previous studies, along with their research findings. In such studies, authors used simple techniques to investigate the usability problem and neglected the user perspective. In this research, we recognized that none of the previous researchers had taken into account the perspective of real users of academic websites, nor had they utilized robust methods such as FAHP to assess the academic web usability factors in order to determine the usability level of such websites.

A systematic framework (as shown in Figure 6) is prepared for the selection of the criteria and their evaluation. A comprehensive list of criteria was identified from the review of literature, out of which relevant criteria were chosen looking at the nature of the present case problem. The analysis was verified by experts.

With regard to the successful design of academic websites, the top-level concerns exhibited in the interface should be evident. The emphasis of the findings of this research was on user support. The support (i.e., basically usability or user support) was found to be the most important dimension, with a 0.641 contribution value. This dimension stresses that academic websites need to be designed to support ease of use, as these websites are enabling users by supporting them in the rest of their major commitments. The factor ease of use means that websites should be as easy as possible for the intended population, so that they may use them productively and pleasantly. The information must be presented in the most useful and usable formats, and should have an interface that characterizes consistency, standard, and predictability. Interactivity is also related to support, which means that information and academic objects should be presented in a way that productively engages visitors. The controls must be visible, and a meaningful feedback mechanism should be in place to encourage the user during their visit to the site [52]. Learnability is an important factor, especially for academic websites, as these websites are not frequently used by most users. So, their design should allow students to quickly become familiar and to easily learn how to perform different academic tasks through the website [37]. The use of familiar conventions (i.e., formatting and navigation schemes) makes it convenient for a user to learn the site with regard to its usage. The second critical dimension of academic websites is of organization, which indicates that the academic websites should be organized in such a manner that facilitates and encourages efficient and effective human–computer interactions [53]. This dimension highlights the idea that much thought is needed to structure the information in ways that will enable effective navigation. The information should be organized in a manner that would allow people to search, browse, and find different kinds of academic information in an efficient manner [52]. Moreover, the organization of information should ensure that everybody, including users who have difficulty seeing, hearing, and making precise movements, can use them. The third and last dimension of the proposed model is content, which indicates that the information provided on an academic website should be useful/interesting, accurate, and satisfactory to the audience.

5. Ranking Academic Websites

The proposed framework was used to evaluate the usability of five academic websites of renowned higher education institutes. To preserve confidentiality, websites were referred as A1, A2, A3, A4, and A5. A survey using a nine-point scale (1 = poor, 9 = excellent) questionnaire was performed to measure the usability of different academic websites. To determine the relative importance of alternative academic websites, an extent analysis method was employed. The fuzzy extent analysis was a de-fuzzification method that was proposed by the authors of [50]. The detailed calculation of the de-fuzzification process is shown below.

First, the results of the degree of possibility for $\widetilde{S_i}\ge {\tilde{S}}_j$ are computed from the fuzzy weighted summation using the fuzzy extent analysis method, as shown below:

V (SA1 ≥ SA2) = 0.80, V (SA1 ≥ SA3) = 0.73, V (SA1 ≥ SA4) = 0.98, V (SA1 ≥ SA5) = 0.96,
V (SA2 ≥ SA1) = 1.00, V (SA2 ≥ SA3) = 0.94, V (SA2 ≥ SA4) = 1.00, V (SA2 ≥ SA5) = 1.00,
V (SA3 ≥ SA1) = 1.00, V (SA3 ≥ SA2) = 1.00, V (SA3 ≥ SA4) = 1.00, V (SA3 ≥ SA5) = 1.00,
V (SA4 ≥ SA1) = 1.00, V (SA4 ≥ SA2) = 0.82, V (SA4 ≥ SA3) = 0.75, V (SA4 ≥ SA5) = 0.99,
V (SA5 ≥ SA1) = 1.00, V (SA5 ≥ SA2) = 0.81, V (SA5 ≥ SA3) = 0.73, V (SA5 ≥ SA4) = 1.00

Then, the degree of possibility of $\widetilde{S_i}$ is calculated, and it should be greater than the other convex fuzzy numbers ($\widetilde{S_j}$), as follows:

V ($\widetilde{S_1}\ge \widetilde{S_j}$| j = 1, 2, 3, …, n, j $\ne i)$ = min [($\widetilde{S_1}\ge \widetilde{S_2}),\left( \widetilde{S_1}\ge \widetilde{S_3}\right), \widetilde{(S_1}\ge \widetilde{S_4}),\widetilde{(S_1}\ge \widetilde{S_5})]$ = min(0.80, 0.73, 0.98, 0.96) = 0.73;

V ($\widetilde{S_2}\ge \widetilde{S_j}$| j = 1, 2, 3, …, n, j $\ne i)$ = min [($\widetilde{S_1}\ge \widetilde{S_2}),\left( \widetilde{S_1}\ge \widetilde{S_3}\right), \widetilde{(S_1}\ge \widetilde{S_4}),\widetilde{(S_1}\ge \widetilde{S_5})]$ = min(1.00, 0.94, 1.00, 1.00) = 0.94;

V ($\widetilde{S_3}\ge \widetilde{S_j}$ | j = 1, 2, 3, …, n, j $\ne i)$ = min [($\widetilde{S_1}\ge \widetilde{S_2}),\left( \widetilde{S_1}\ge \widetilde{S_3}\right), \widetilde{(S_1}\ge \widetilde{S_4}),\widetilde{(S_1}\ge \widetilde{S_5})]$ = min(1.00, 1.00, 1.00, 1.00) = 1.00;

V ($\widetilde{S_4}\ge \widetilde{S_j}$ | j = 1, 2, 3, …, n, j $\ne i)$ = min [($\widetilde{S_1}\ge \widetilde{S_2}),\left( \widetilde{S_1}\ge \widetilde{S_3}\right), \widetilde{(S_1}\ge \widetilde{S_4}),\widetilde{(S_1}\ge \widetilde{S_5})]$ = min(1.00, 0.82, 0.75, 0.99) = 0.75;

V ($\widetilde{S_5}\ge \widetilde{S_j}$| j = 1, 2, 3, …, n, j $\ne i)$ = min [($\widetilde{S_1}\ge \widetilde{S_2}),\left( \widetilde{S_1}\ge \widetilde{S_3}\right), \widetilde{(S_1}\ge \widetilde{S_4}),\widetilde{(S_1}\ge \widetilde{S_5})]$ = min(1.00, 0.81, 0.73, 1.00) = 0.73;

After obtaining the global weights of the factors, the fuzzy weights following pairwise comparisons of five websites, anonymously shown as websites A1–A5, were computed using the same steps used to compute the weights of the factors. To gather data regarding academic websites, the same previous sample was considered for the survey questionnaire, in which websites were compared with each other regarding the identified criteria. The result produced aggregated pairwise comparisons and calculated the summed weights of the websites evaluated using the fuzzy extent analysis method. Then, global weights of criteria and fuzzy summed weights were multiplied with each other to get the priority weight vectors, which were used to rank the academic websites. The results are shown in

Table 7. Weights vector of alternative websites and sub-criteria.

Weight Vector d’ w.r.t Criteria	C1	C2	C3	C4	C5	C6	C7	C8	C9
A1	0.7335	0.4557	0.3808	1.0000	0.6809	0.9542	0.7688	0.4105	0.8675
A2	0.9425	0.4326	0.7568	0.9616	0.9025	1.0000	0.9064	0.9497	1.0000
A3	1.0000	0.8305	0.8802	0.9555	1.0000	0.5782	1.0000	0.8037	0.8432
A4	0.7510	0.4399	0.5417	0.8250	0.7415	0.3680	0.4197	0.6548	0.4187
A5	0.7335	1.0000	1.0000	0.4288	0.8847	0.7577	0.7969	1.0000	0.5436

(C1) Ease of use; (C2) interactivity; (C3) learnability; (C4) ease of navigation; (C5) accessibility; (C6) efficiency; (C7) informative; (C8) accuracy; (C9) user satisfaction.

.

The weights obtained by multiplying the weight vectors of the alternatives and criteria were normalized, as shown in

Table 8. Normalized weights vector of alternatives and criteria.

Weight Vector d’ w.r.t Criteria	C1	C2	C3	C4	C5	C6	C7	C8	C9
A1	0.1763	0.1443	0.1070	0.2398	0.1618	0.2608	0.1975	0.1075	0.2362
A2	0.2265	0.1370	0.2126	0.2306	0.2144	0.2734	0.2329	0.2487	0.2723
A3	0.2404	0.2629	0.2473	0.2291	0.2376	0.1581	0.2570	0.2105	0.2296
A4	0.1805	0.1393	0.1522	0.1978	0.1761	0.1006	0.1078	0.1715	0.1140
A5	0.1763	0.3166	0.2809	0.1028	0.2102	0.2071	0.2048	0.2619	0.1480
Weights of criteria	0.3910	0.1140	0.1370	0.1260	0.0320	0.0360	0.1040	0.0260	0.0340

. After normalization of the weight vector of alternatives, the weights of each alternative were multiplied with the criteria weight and then summed to get the global performance of each alternative, using Equation (17):

$$D(A_{i)}=\left(d^{\prime }\left(A_{i_{C1}}\right)\times \left(W_{C1}\right)+d^{\prime }\left(A_{i_{C2}}\right)\times \left(W_{C2}\right)+\dots + d^{\prime }\left(A_{n_{Cn}}\right)\times \left(W_{Cn}\right)\right)$$

(17)

A comparison was made of the relative importance of university websites obtained from the global performance ranking using the de-fuzzification method used by the authors of [54]. The results are shown in

Table 9. Ranking of Ranking of Academic Websites.

Websites	Global Performance	Ranking
Website 1	0.1762	4th
Website 2	0.2190	2nd
Website 3	0.2400	1st
Website 4	0.1610	5th
Website 5	0.2038	3rd

.

The analysis shows that Website-3 was most effective and successful for operating HEIs academically and administratively, because this website followed critical factors of usability in comparison with the other four websites. Website-2 was found to be the second most usable academic website, followed by Website-5, Website-1, and Website-4.

The proposed fuzzy AHP methodology is validated through the usability evaluation of renowned academic websites in Pakistan. The usability of the selected academic websites was evaluated in a fuzzy environment, and it was hard to determine the usability issues because there were several uncertainties and ambiguities. So, this research applied the fuzzy AHP method to evaluate the usability of academic websites based on identified dimensions and factors.

6. Sensitivity Analysis

Sensitivity analysis is a tool to check the robustness of the results obtained. It helps to check the influence of different factors and the effect of each factor on the obtained results. In the present research, a sensitivity analysis was carried out to reveal the effect of the weight when it was dropped from the selection. Thus, the influence of each factor was revealed by neglecting the selection serially. The results obtained are depicted in

Table 10. Sensitivity analysis of academic websites.

Criteria	De-Fuzzified Weight of Alternative HEC Academic Websites					Ranking
Fuzzy Extent Analysis	A1	A2	A3	A4	A5
All criteria	0.1762	0.2190	0.2400	0.1610	0.2038	A3 > A2 > A5 > A1 > A4
Dropped ease of use	0.4065	0.4920	0.5419	0.3402	0.4879	A3 > A2 > A5 > A1 > A4
Dropped interactivity	0.6413	0.8112	0.8383	0.5837	0.6606	A3 > A2 > A5 > A1 > A4
Dropped learnability	0.6411	0.7568	0.8123	0.5596	0.6376	A3 > A2 > A5 > A1 > A4
Dropped ease of navgation	0.5673	0.7394	0.8125	0.5299	0.7206	A3 > A2 > A5 > A1 > A4
Dropped accessibility	0.6715	0.8316	0.9009	0.6101	0.7463	A3 > A2 > A5 > A1 > A4
Dropped efficiency	0.6589	0.8245	0.9121	0.6206	0.7474	A3 > A2 > A5 > A1 > A4
Dropped informative	0.6133	0.7663	0.8289	0.5902	0.6918	A3 > A2 > A5 > A1 > A4
Dropped accuracy	0.6826	0.8358	0.9120	0.6168	0.7486	A3 > A2 > A5 > A1 > A4
Dropped user satisfaction	0.6637	0.8265	0.9044	0.6196	0.7563	A3 > A2 > A5 > A1 > A4

> indicates preference over another.

.

When all of the weights of the criteria of website usability were considered, the website ranking A3 > A2 > A5 > A1 > A4 was obtained, where > shows the preference of one over another. When each criterion was dropped sequentially in the website selection, the same ranking prevailed. Figure 7 shows the ranking sequence under different scenarios. The rankings obtained under the “all criteria” and “one-criteria dropped” scenario the ranking did not differ, hence the ranking obtained using FAHP showed a good accuracy of results.

7. Limitations of the Present Study

In the present research, we employed TFN for obtaining the judgmental matrix. The high accuracy of the judgmental decisions was improved by adopting high transitivity and consistency in decision-making. FAHP facilitates the use of TFN, thus helping to reduce linguistic vagueness. However, the ranking obtained in the present research may not be generalized because of the limited number of criteria employed in the research. Furthermore, similar research was not found in the literature; hence, it is difficult to draw a parallel to the obtained ranking. The present research may help to pursue similar research in different application settings.

8. Conclusions and Future Work

Numerous research studies have been conducted to evaluate the usability of academic websites. However, previous research carried out usability evaluations through predefined checklists, heuristic rules, and using standard inventories. This study was conducted in a methodical way in order to propose a framework for the usability evaluation of academic websites. The usability factors with respect to academic websites were firstly identified through a literature review, which was further validated by the users of such websites using MCDM techniques. In this way, a usability evaluation framework was found that presents three criteria and nine sub-criteria for academic websites. This study found that the highly critical factors among the nine factors were ease of use, easy to learn, and easy to navigate. Other factors also have their own importance with respect to the usability of academic websites. The least significant factor of the key factors was accuracy. Although accuracy is ranked at a lower position in relation to other CSFs, it is not the lowest factor. It has its own special value, as it suggests that resources should not be wasted providing good usability to the wrong content. The results of the study imply that in order to deliver a good user experience to the users of academic websites, the interface should exhibit these properties. The proposed framework can help to diagnose weak areas of academic websites with respect to usability, so website developers may correspondingly improve the design to make them highly usable. The major strength of this work in relation to previous studies (where simple techniques were employed to evaluate usability) is that it has utilized a real strong approach that gives a meaningful and precise explanation regarding the academic websites’ usability dimensions and factors of the hierarchical structure. To further verify the soundness of this method, we are intended to explore it in other domains, such as online learning, e-commerce, and healthcare. It is also in the authors’ future plan to compare the results obtained through this method with other MCDM methods, such as fuzzy AHP and fuzzy TOPSIS, to discover the best suitable results. As a future work, this study will be extended by AHP integration with fuzzy simple additive weighting and fuzzy weighted product model to investigate their application and usefulness in the context of web usability. This study can also be extended further with the application of the fuzzy DEMATEL method to find the correlation that exists among the identified dimensions and factors.