Improving the Sustainability of Open Government Data

Abstract

In the context of big data, opening government data to society for development and utilization would improve the transparency of government work and promote harmonious social development. Improving the sustainability of open government data (OGD) is an important and challenging issue, and has received the attention of multiple governments. The goals of this study are to construct a new evaluation model for the sustainability of OGD that explores the main influential factors and amends the priorities. We used a combination of the Decision-Making Trial and Evaluation Laboratory (DEMATEL) method and the Analytical Network Process (ANP) method known as DANP to construct an influential network relationship map (INRM) of the evaluation system and calculate the criterion weights. The INRM and weights can help decision makers understand the complex assessment system and set priorities for improving OGD. The results demonstrate that the improvement priorities of dimensions have the following order: impact (D4), use (D3), data (D2), and environment (D1). Economic, social, and political values are the three critical factors for improvement of the sustainability of OGD. Some implications for management are also provided.

1. Introduction

Governmental departments generate, acquire, and preserve large amounts of data resources in the process of fulfilling their administrative duties. These data are public resources for the whole society. Open government data (OGD) refers to government data that anyone or any organization can access, use, and share freely [1,2]. Continuously opening government data to society for development and utilization will help improve the transparency of government work and promote public participation in governance [3]. OGD is easy for the public to reuse and redevelop for any purpose, to generate new services, stimulate innovation, and provide new economic opportunities, thus creating social and economic value [4,5]. OGD can stimulate the public to participate in government events and affairs, so that political value can be created by improving the transparency and public service quality of the government departments [6]. Hence, OGD is one of the key paths for sustainable development [7].

The US government launched the open data platform data.gov in 2009, and, since then, other countries have also established web portals to promote the sustainability of OGD, which has been progressing rapidly all around the world. As of 2016, 106 of the 193 member states of the United Nations have provided open data directories through “government websites” on topics such as the environment, transportation, and health [7]. Governments are actively boosting cooperation. As of April 2019, 79 countries have joined the Open Government Partnership (OGP), which was established in September 2011 [8]. Governments have thus aroused much attention toward the sustainability of OGD.

Local government OGD platforms have been put into use in areas such as Beijing, Shenzhen, and Guizhou, with the aim of helping people freely access government data. Individuals and businesses are developing applications by downloading government data, such as “I love health” and “Travel in Beijing”. These efforts are conducive to creating “smart” lifestyles and promoting sustainable social development. According to the 2016 Global Open data Barometer Report (Fourth Edition), China ranks 71 out of 114 in this respect [9]. Although the Chinese government actively promotes OGD, in general, this is insufficient. Therefore, it is important to improve the sustainability of OGD in China [10]. In other words, it is important to identify the factors influencing the sustainability of OGD and to propose improvement strategies.

Evaluation is an important way to increase the performance of the public sector [11]. Administrations can identify weaknesses and develop improvement strategies to promote the sustainability of OGD by assessing the capability and effect of OGD [12]. OGD is mainly evaluated by international organizations and scholars. The Global Open Data Barometer Report assessed the stage of open data readiness, execution, and influence, including economic, political, and social impacts [9]. The Open Global Data Index assessment is organized by the Open Knowledge Foundation of the United Kingdom, and has a citizen perspective on 15 first-level indicators [13]. The Open Data Readiness Assessment is implemented by the World Bank’s Open Data Working Group, and is evaluated based on eight dimensions and three levels [14]. The OGD Index is assessed by the Economic and Development Organization, and evaluates three aspects: communication infrastructure, capacity to promote and use communication technologies, and the possibilities for online services [15]. The Government Data Openness Survey evaluates 193 member countries of the United Nations by accessing government portals and data [7]. Based on the International Government Data Open Assessment Project, the World Wide Web Foundation and the New York University Governance Laboratory have proposed a common OGD assessment framework that includes four dimensions: environment, data, use, and impact [16]. The Digital and Mobile Governance Laboratory of Fudan University evaluates the China Open Forest Index for OGD for 46 provinces and cities in China, based on three dimensions: the data layer, platform, and readiness levels [17]. Some research has evaluated OGD from the perspective of service capabilities, data quality, and producing impact. The OGD service capabilities can be evaluated by open data policies, website usability, entrepreneurial data services, and user satisfaction [18,19,20]. Data quality has also been evaluated in terms of the data source, data integrity, and data service [21,22,23,24]. Other studies have focused on evaluation of the value and effectiveness of OGD, including organizational, economic, and social benefits [25,26,27,28,29].

In general, OGD evaluation methods include qualitative analyses, such as case analysis [9,22,23] and expert interviews [15], as well as quantitative methods, such as the analytic hierarchy process (AHP) [20,24], artificial neural networks (ANNs) [25], and structural equation modeling (SEM) [18,27]. The contents and indicators for evaluation are diverse. Previous studies have mainly focused on the evaluation of OGD projects, but little attention has been given to the sustainability of OGD.

In this paper, the term sustainability means to apply government data that has been continuously open to the public through the platform and to maximize the benefits of OGD. “Promote just, peaceful and inclusive societies” is one of the 17 sustainable development goals developed by the United Nations [30]. Sustainable OGD can continuously improve government performance and transparency, and encourage public participation in social governance [3]. Sustainability of OGD as an enabler of transparent, accountable, and effective public administration institutions can support sustainable development [7]. Additionally, sustainability of OGD can continuously generate new services, stimulate innovation, and provide new economic opportunities by reusing and redeveloping the opening data [4,5]. Sustainable OGD can contribute in many ways to the achievement of sustainable development goals, and can drive progress towards sustainable development [7]. Sustainable OGD is an important way to promote sustainable development. The ways to improve sustainability of OGD remain a very meaningful research issue. Thus, this study aims to reveal the influential factors and impact effects of sustainable OGD, as well as to propose strategies for improvement. In sum, the investigation aimed to:

(1)

Define the evaluating factors and establish the evaluation model for the sustainability of OGD.

(2)

Investigate the influence of degrees between factors.

(3)

Put forward suggestions to improve the sustainability of OGD.

The improvement of sustainable OGD is a multiple-attribute decision-making problem. Relevant variables usually have certain degrees of correlation, especially the interaction between important variables, such as dependence and feedback effects. The Decision-Making Trial and Evaluation Laboratory (DEMATEL) method combined with the Analytic Network Process (ANP) method can solve the problem by constructing an interaction between the evaluation dimensions and the criteria of sustainable OGD. This method has been widely used to examine emergency strategies, transportation, green buildings, network reputation, service quality, and other applications [31,32,33,34,35,36,37]. However, it is rarely used on OGD. The DANP is a new method to construct evaluation models of sustainability of OGD. Based on the DEMATEL and ANP methods, the relationships and weights of each influential factor can be calculated. The derived influential network relationship map (INRM) is used to represent the influence of relationships between dimensions and criteria in sustainable OGD. This will help government administrators understand the influencing factors in complex relationships and propose suggestions to improve the sustainability of OGD.

The remainder of this paper is organized as follows. Section 2 reviews the relevant literature to form the key factors for sustainability of OGD. Section 3 introduces the DANP method that was used to build the INRM and find the weights of the criteria. The data analysis and results are presented in Section 4. A discussion and the implications are discussed in Section 5, and Section 6 presents the conclusions.

2. Literature Review on Establishment Indicator Framework

This study adopts the OGD assessment draft framework proposed by ODI and New York University [17] and integrates OGD ecosystem theory [38]. We evaluate the sustainability of OGD from environment, data, use, and impact dimensions.

2.1. Definition of Sustainability of OGD

Sustainability refers to the ability to sustain or support objective things permanently or indefinitely. From a dynamic point of view, it means that healthy existence and development of the system can be continuously or indefinitely maintain or supported [38]. Sustainability was first proposed by ecologists, and this is now called ecological sustainability [39].

An ecosystem can be understood as “a system of people, practices, values and technologies in a particular local environment” [40]. There are various ecosystem analogies, such as the business ecosystem, digital ecosystem, and open government ecosystem [38,40,41,42]. The concept of a data ecosystem has seen growing use in research on data-driven government. In open data ecology, there are four archetypal sustainable value generating mechanisms. They are information transparency, collective impact, data-driven efficiency, and data-driven innovation. The sustainable value generation is influenced by: the incentives provided, the level of technical and legal openness of data, the maturity of resource (data) governance, the general data-related capabilities in society, and the technological maturity and prevalence [43]. In general, open data ecosystems can be seen as being composed of a number of elements, such as participants (actors), data resources, design, and interdependencies [41].

We also think of OGD as an ecosystem. The sustainability of OGD is a new area of research that links the concepts of sustainability and OGD. Generally, sustainable OGD emphasizes the realization of long-term continuous OGD and the reuse of data to generate political, economic, and social value [12,41]. Opening and using data encounters numerous impediments, such as availability and access, findability, usability, understandability, quality, metadata, and so on [44]. The OGD ecosystem includes stakeholders, processes of data, and benefits associated [6,38]. The components of the OGD ecosystem mainly include stakeholders, the process of utilization, benefit, and external environment [42]. An OGD ecosystem is defined as one driven by the realization of value [40]. The environment is the infrastructure of OGD ecosystem. The data is the basic and most important resource of the OGD ecosystem [6,38]. Impacts on politics, economy, and society will be created by using data resources [45]. The impacts are the sustainable value that sustainable OGD will generate [40,43]. This study adopts the OGD ecosystem and the OGD assessment draft framework proposed by ODI and New York University [16]. The research framework includes the environment, data, use, and impact dimensions with 18 criteria, which are discussed below.

2.2. Environment Dimension

The environment dimension refers to the background conditions for providing sustainable OGD. It is the premise for OGD implementation and provides protection and a technical foundation, including laws and regulations, political will and leadership, and technical capacity [28,37]. Laws and regulatory factors provide an implementation framework for sustainable OGD, to ensure that government data are continuously open for a long time, to improve the transparency of government information, and to stimulate the use and value creation of OGD [25,28].

OGD is open data that was generated by government management, and the open data is stored in various parts of the government [2,46]. Political will and leadership will strengthen collaboration among government departments to build a unified data set that can be sustainably opened to the public [45]. If leaders do not pay attention, sustainable OGD cannot be effectively developed [47]. As a basic policy to improve government transparency, sustainable OGD requires time and money to implement. OGD requires strong government support and a strong political will to mobilize government data and other related resources [48].

Public supervision, consultation, and participation in OGD will promote government transparency and social equity, as well as trust in society and social connectedness [3]. The more active the public is in OGD, the more sustainable OGD becomes [10]. Enthusiasm among the public for the access and reuse of open data is a prerequisite for OGD to create value [29].

Technical capacity provides technical support for improving the quality of sustainable OGD and promoting its effective use. Linked data technology realizes the association and integration between different datasets, data, and OGD platforms, as well as the semantic expression and integration of open data [49,50,51]. Ontology enhances the discovery capability of unstructured data and the reusability of mapping information [52]. Thus, this study uses laws and regulations, political will and leadership, general public, and technical capacity as indicators to assess the sustainability of OGD under the environmental dimension.

2.3. Data Quality Dimension

Data quality has significant direct effects on OGD adoption and sustainable value generation [43,53]. Data quality is a measure of the data content and reusability provided by sustainable OGD, which is the basis for the public reuse of data [22]. Moreover, data quality is critical in the implementation of sustainable value generation [40,43]. OGD should provide complete, accurate, timely, and easy-to-use data to improve the quality of OGD services and increase public satisfaction [17,54]. Open data may conflict with privacy, security, and transparency. When opening government data, the data should be densified and access controlled [55]. Completeness refers to the type of OGD and the richness of datasets. The richer the open data, the more value can be generated by reusing data, while protecting national security, trade secrets, and personal privacy [18,20,22,23,56,57]. The value of big data in social domain can only be created if the outcomes are correctly interpreted [58].

Accuracy is a measure of the correctness of data. Only suitably accurate data can be used to make decisions for the public. Correct data can help the public to make scientific decisions [20,22,23,54,57]. Timeliness refers to whether OGD provides timely data updates, whether the data is up-to-date, the frequency and cycle of data updates, and when the last data update has been made. Data timeliness is one of the important indicators for data quality evaluation [22,23,54,56,57]. Understandability means that keywords, titles, and descriptions in OGD are easy to understand not only by experts, but also by the general public [18,20,22,53]. The public can only use the data after understanding OGD.

Accessibility means that the public has equal access to open data without having to be questioned about the desired purpose, and the use of the data is not restricted [15,56]. Data usability refers to the basic data format provided to facilitate public access, downloading, and reuse [18,20]. Data usability can be divided into three levels: non-machine format (such as PDF), automatically processed structured formats (such as XLS), and inclusion elements, structured format descriptions, and semantic formats of data (such as XML and RDF) [57]. Non-machine-formatted data could potentially hamper the efficient reuse of open data [22]. This study uses completeness, accuracy, timeliness, understandability, accessibility, and usability as indicators to assess the openness of government data under the data quality dimension.

2.4. Use Dimension

The use dimension evaluates the convenience of public access to an OGD platform. Simple and convenient operation results in a good experience for the public. The interface refers to the user’s experience when visiting an OGD website, and is evaluated based on the design of the platform. If the interface experience is good, the public’s satisfaction with OGD will increase [18,53]. The data index is used to classify the OGD platform dataset by subject, institution, data format, or other qualities, as well as keyword search and advanced search operation, whether the evaluation data classification directory is reasonable, the convenience of retrieval [18], and whether the online help function will be user-friendly [20].

Data acquisition refers to whether the public requires complicated operations, such as registration or filling in personal information. The public’s use of OGD is affected by whether the platform’s operation steps are simple and whether obtaining the data is complicated [18]. Data display refers to whether the OGD is displayed to users in various forms, provides personalized previews and information displays, and provides multi-language support, which all affect the reuse of OGD [20]. Interactive communication refers to whether the OGD website allows providers to collect dataset ratings and comments. The OGD platform should provide forums and other interactive communication functions to encourage the public to use OGD and improve service satisfaction [18]. Thus, this study uses the interface, index, acquisition, presentation, and instructiveness factors as indicators of the openness of government data in the use dimension.

2.5. Impact Dimension

Impact refers to the effects and value generated after the data are opened and utilized. This is the goal of OGD [24,44] and promotes sustainable OGD [12]. OGD generates benefits in social, political, and economic aspects [20]. Social impact is generated through public participation and their use of OGD. The OGD platform should allow the public to query relevant information quickly, simplify government workflow, improve work efficiency, and improve quality of life [20,27]. OGD can change the public from passively communicating to actively participating in the model by letting them supervise, negotiate, and participate in governance [3,59].

The government is the provider of OGD. OGD can improve the transparency of public administration and its procedures, reduce the abuse of public power, and help fight corruption. OGD stimulates and improves the government’s serviceability and improves public satisfaction [27,59]. OGD could advance the democratic process, promote sustainable development, and thus could have a political impact [20]. An enterprise information system could directly access the OGD platform and process the data to create new products, provide new services, and create new jobs, economic growth, and enterprise innovation to generate economic value [60]. For example, using the government’s open traffic data, enterprises could develop a “mobile bus” application, and enterprises could gain economic benefits. This study uses social impact, political influence, and economic impact as indicators to assess the openness of government data under the impact dimension.

Table 1. The evaluation factors for sustainable OGD.

Dimension	Criteria	Descriptions	References
Environment (D1)	Legal & policy (C11)	Related laws and policy on OGD.	[25,28]
	Political will & leadership (C12)	Willingness and leadership of government agencies.	[47]
	Social public (C13)	The degree of public support and participation in OGD.	[3,28]
	Technical Capacity (C14)	Technological advancement in OGD.	[49,50,51]
Data (D2)	Completeness (C21)	The number of OGD sets.	[22,54,56]
	Accuracy (C22)	The data is accurate.	[22,54,56]
	Timeliness (C23)	Timely update of data.	[22,54,56]
	Understandability (C24)	The data description is easy for the public to understand.	[22,54,58]
	Accessibility (C25)	The data can be accessible to anyone for any purpose.	[22,54,58]
	Usable (C26)	Open data structure for easy to reuse.	[18,58]
Use (D3)	Interface (C31)	User experience of operating OGD platform.	[18,54]
	Index (C32)	Information systems or search engines.	[18,54]
	Acquisition (C33)	Whether the operation to obtain data is complex.	[18,20]
	Presentation (C34)	Data is presented in many forms.	[18,20]
	Interactive (C35)	OGD platform provides interaction with users.	[20,52]
Impact (D4)	Social (C41)	The impact of OGD on society.	[20,27]
	Political (C42)	The impact of OGD on politics.	[27,60]
	Economic (C43)	The impact of OGD on economics.	[18,60]

shows all of the discussed indicators from the relevant literature used to assess sustainable OGD.

3. Methodology

This study established an evaluation model of sustainable OGD by introducing the DANP model, which integrates DEMATEL with ANP. The DANP method has been applied to construct evaluation models in education, marketing, traffic, and other fields [34,35,36,37,61]. It is the new method for building an evaluation model for the sustainability of OGD. We calculated the influential weights (IWs) of DANP to find the causal relationships associated with each dimension and criterion. We used the results to derive the INRM. Both the weights and the INRM could help Chinese government administrators to improve their priorities for improving OGD. The details of this have been described in the literature [34,61,62,63,64,65,66].

3.1. DEMATEL Method

The Geneva Battelle Research Center [67,68] first proposed DEMATEL to construct structural analysis models to solve specific societal problems. DEMATEL is used to identify the correlation between factors. An INRM is constructed to clarify the sources of effect indexes, and can be used to improve system performance [69,70,71]. The details are as follows.

Step 1: Build the average direct influence relation matrix A = ${[a_{ij}]}_{m\times n}$. Questionnaires were used to collect data, and indicators were compared in pairs. Experts were invited to select the degree of influence, as well as positive and negative directions according to the scale. The results were recorded in the form of a matrix. Scales of 0–4 were used to represent the range from “absolutely no influence” (0) to “very high influence” (4). The number of experts is H, and the number of factors (the criteria) is n. The average scores of the H experts were $a_{ij}$, $x_{ij}$ is the degree of direct influence of the criterion i on the criterion j, and $x_{ij}^k$ is the score of the expert k. The direct influence–relation average matrix A is obtained by calculating the average of the same factor in N matrices of the respondents. A is expressed in Equation (1), where n is the number of criteria.

$$\mathit{A}=\left[\begin{array}{ccccc}{a}_{11}& \dots & a_{1j}& \dots & a_{1n}\\ ⋮& & ⋮& & ⋮\\ a_{i1}& \dots & a_{ij}& \dots & a_{in}\\ ⋮& & ⋮& & ⋮\\ a_{n1}& \dots & a_{nj}& \dots & a_{nn}\end{array}\right] \mathrm{where} a_{ij}=\frac{1}{H}{\sum }_{k=1}^H{x}_{ij}^k$$

(1)

Step 2: Formulate the normalizing direct influence–relation average matrix X. We obtained the normalized initial direct influence–relation matrix X by normalizing the average matrix A. X is the product of s and A, and is obtained from Equations (2) and (3).

$$\mathit{X}=s·\mathit{A}$$

(2)

$$s=\min \left[1/\underset{1\le i\le n}{\max }{\displaystyle \sum }_{j=1}^n{a}_{ij},1/\underset{1\le j\le n}{\max }{\displaystyle \sum }_{j=1}^n{a}_{ij}\right]$$

(3)

Step 3: Construct the total influence matrix T. In Equation (4), the total influence relation matrix $\mathit{T}$ is an $n\times n$ matrix formed as $\mathit{T}={\left[t_{ij}\right]}_{n\times n}$, and I is an $n\times n$ unit matrix. Element $t_{ij}$ is the degree of direct influence of the criterion i on criterion j.

$$\mathit{T}=\mathit{X}+{\mathit{X}}^2+{\mathit{X}}^3+\cdots +{\mathit{X}}^k=\mathit{X} {\left(\mathit{I}-\mathit{X}\right)}^{-1}, \mathrm{when} k\to \infty , {\mathit{X}}^k={\left[\mathit{O}\right]}_{n\times n}$$

(4)

Step 4: Obtain the INRM using the total matrix T. Using Equations (5) and (6), we can calculate each column sum (c_j) and row sum (r_i) from matrix T.

$$\mathit{c}={\left({\mathit{c}}_j\right)}_{n\times 1}={\left({\mathit{c}}_j\right)}_{1\times n}^{\prime }={\left[{\displaystyle \sum _{i=1}^n{\mathit{t}}_{ij}}\right]}^{\prime }$$

(5)

$$r={\left(r_i\right)}_{n\times 1}=\left[{\displaystyle \sum _{j=1}^n{\mathit{t}}_{ij}}\right]$$

(6)

The c_j is the sum of the direct and indirect influences of factor j on the other factors in matrix T, which is called the degree of influence. r_i is the sum of direct and indirect influences of other factors on factor i in matrix T, which is called the degree of effect. When i = j, r_i + c_i reflects the intensity affected given by and received from factor i; that is, the centrality of index i. r_i – c_i is called the degree of causality. Clearly, if r_i – c_i is greater than zero, the criterion i is an influencing element, but if r_i – c_i is less than zero, the criterion i is an effective element [33]. Consequently, based on r_i + c_i and r_i – c_i, we can draw the INRM.

Step 5: Derive the total influence relation matrices T_C, based on criteria, and T_D, based on dimensions. The matrix T can be transformed into T_C (as shown in Equation (7)) based on the criteria and T_D (as shown in Equation (8)) based on the dimensions. n is the number of dimensions, and m_i is the number of criteria in dimension i.

$$\begin{array}{ll}& \begin{array}{cccc}{D}_1& D_2& \cdots & D_n\\ C_{11}\cdots C_{1m_1}& C_{21}\cdots C_{2m_2}& \cdots & C_{n1}\cdots C_{n{m}_n}\end{array}\\ {\mathit{T}}_C=\begin{array}{cc}{D}_1& \begin{array}{c}{C}_{11}\\ C_{12}\\ ⋮\\ C_{1m_1}\end{array}\\ D_2& \begin{array}{c}{C}_{21}\\ C_{22}\\ ⋮\\ C_{2m_2}\end{array}\\ ⋮& ⋮\\ D_n& \begin{array}{c}{C}_{n1}\\ C_{n2}\\ ⋮\\ C_{n{m}_n}\end{array}\end{array}& \left[\begin{array}{cccc}\begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{11}\\ \\ \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{12}\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{1n}\\ \\ \end{array}& \end{array}\\ \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{21}\\ \\ \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{22}\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{2n}\\ \\ \end{array}& \end{array}\\ \begin{array}{ccc}& \begin{array}{c}\\ ⋮\\ \\ \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ ⋮\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ ⋮\\ \\ \end{array}& \end{array}\\ \begin{array}{ccc}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{n1}\\ \\ \end{array}& \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{n2}\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_{\mathrm{c}}^{nn}\\ \\ \end{array}& \end{array}\end{array}\right]\end{array}$$

(7)

$${\mathit{T}}_D=\left[\begin{array}{cccc}{t}_{11}& t_{12}& \cdots & t_{1n}\\ t_{21}& t_{22}& \cdots & t_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ t_{n1}& t_{n2}& \cdots & t_{nn}\end{array}\right] ={\left[t_{ij}\right]}_{n\times n} , \mathrm{where} t_{ij}=\frac{{\sum }_{l=1}^{m_j}{\sum }_{k=1}^{m_i}{T}_c{}^{m_{l }{m}_k}}{m_l\times m_k}$$

(8)

3.2. Obtaining the Influence Weights through the DANP

In 1996, Satty proposed ANP based on AHP. The ANP technique is used to build un-weighted and weighted super-matrices that consider the mutual influence of various factors or adjacent layers. Nevertheless, there are some weaknesses in the original ANP technique. First, we must assume the relationship structure of the evaluation system before using ANP. Second, the ANP questionnaire is difficult to understand, due to its complexity [72] and the time-consuming process of pairwise comparison. Thus, consistent results are difficult to obtain. Third, the hypothesis that every dimension has the same weight seems unreasonable because of conflicts of influence among the dimensions, or because the clusters are different in the real world [73]. Accordingly, the DEMATEL-based ANP (DANP) was proposed by Tzeng et al. [60,61,74] to solve these problems. The DANP method is used to solve real-world problems with feedback and interaction between criteria or dimensions, and to determine the affecting weights [61,62,63,75,76]. The DANP technique is summarized as follows.

Step 6: Construct the unweighted super-matrix W. The matrix ${\mathit{T}}_C^{\omega }$ can be obtained by normalizing matrix T_C, which is shown in Equation (7). For example, $T_C^{pq}$, which is a submatrix of T_C, can be normalized into ${\mathit{T}}_c^{\omega pq}$, as shown in Equations (9) and (10). The matrix ${\mathit{T}}_c^{\omega pq}$ can be transposed to an unweighted matrix super-matrix W_pq, as shown in Equation (12).

$$\begin{array}{ll}& \begin{array}{cccc}{D}_1& D_2& \cdots & D_n\\ C_{11}\cdots C_{1m_1}& C_{21}\cdots C_{2m_2}& \cdots & C_{n1}\cdots C_{n{m}_n}\end{array}\\ {\mathit{T}}_C^{\omega }=\begin{array}{cc}{D}_1& \begin{array}{c}{C}_{11}\\ C_{12}\\ ⋮\\ C_{1m_1}\end{array}\\ D_2& \begin{array}{c}{C}_{21}\\ C_{22}\\ ⋮\\ C_{2m_2}\end{array}\\ ⋮& ⋮\\ D_n& \begin{array}{c}{C}_{n1}\\ C_{n2}\\ ⋮\\ C_{n{m}_n}\end{array}\end{array}& \left[\begin{array}{cccc}\begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega 11}\\ \\ \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega 12}\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega 1n}\\ \\ \end{array}& \end{array}\\ \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega 21}\\ \\ \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega 22}\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega 2n}\\ \\ \end{array}& \end{array}\\ \begin{array}{ccc}& \begin{array}{c}\\ ⋮\\ \\ \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ ⋮\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ ⋮\\ \\ \end{array}& \end{array}\\ \begin{array}{ccc}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega n1}\\ \\ \end{array}& \end{array}& \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_c^{\omega n2}\\ \\ \end{array}& \end{array}& \begin{array}{c}\\ \cdots \\ \\ \end{array}& \begin{array}{ccc}& \begin{array}{c}\\ {\mathit{T}}_{\mathrm{c}}^{\omega nn}\\ \\ \end{array}& \end{array}\end{array}\right]\end{array}$$

(9)

For example, ${\mathit{T}}_C^{pq}$, which is a submatrix of T_C, can be normalized into ${\mathit{T}}_C^{\omega pq}$, as shown in Equation (9) and Equation (10).

(10)

$$\begin{array}{cc}& \begin{array}{ccccc}{c}_{q1}& \cdots & c_{qj}& \cdots & c_{q{m}_q}\end{array}\\ {\mathit{T}}_C^{\omega pq}=\begin{array}{c}{c}_{p1}\\ ⋮\\ c_{pi}\\ \begin{array}{c}⋮\\ c_{p{m}_p}\end{array}\end{array}& \left[\begin{array}{ccccc}\raisebox{1ex}{$t_{11}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_1^{pq}$}\right.& \cdots & \raisebox{1ex}{$t_{1j}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_1^{pq}$}\right.& \cdots & \raisebox{1ex}{$t_{1m_q}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_1^{pq}$}\right.\\ ⋮& & ⋮& & ⋮\\ \raisebox{1ex}{$t_{i1}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_i^{pq}$}\right.& \cdots & \raisebox{1ex}{$t_{ij}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_i^{pq}$}\right.& \cdots & \raisebox{1ex}{$t_{i{m}_q}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_i^{pq}$}\right.\\ ⋮& & ⋮& & ⋮\\ \raisebox{1ex}{$t_{m_{p}1}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_{m_p}^{pq}$}\right.& \cdots & \raisebox{1ex}{$t_{m_{p}j}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_{m_p}^{pq}$}\right.& \cdots & \raisebox{1ex}{$t_{m_p{m}_q}^{pq}$}\!\left/ \!\raisebox{-1ex}{$t_{m_p}^{pq}$}\right.\end{array}\right]\end{array}=\left[\begin{array}{cccc}{t}_{11}^{\omega pq}& \cdots & t_{1j}^{\omega pq}& \begin{array}{cc}\cdots & t_{1m_q}^{\omega pq}\end{array}\\ ⋮& & ⋮& \begin{array}{cc}& ⋮\end{array}\\ t_{i1}^{\omega pq}& \cdots & t_{ij}^{\omega pq}& \begin{array}{cc}\cdots & t_{i{m}_q}^{\omega pq}\end{array}\\ \begin{array}{c}⋮\\ t_{m_{p}1}^{\omega pq}\end{array}& \begin{array}{c}\\ \cdots \end{array}& \begin{array}{c}⋮\\ t_{m_{p}j}^{\omega pq}\end{array}& \begin{array}{cc}\begin{array}{c}\\ \cdots \end{array}& \begin{array}{c}⋮\\ t_{m_p{m}_q}^{\omega pq}\end{array}\end{array}\end{array}\right]$$

(11)

The matrix ${\mathit{T}}_c^{\omega pq}$ can be transposed to unweighted matrix super-matrix W_pq, as shown in Equation (12).

$$\begin{array}{ccc}& \begin{array}{ccccc}{c}_{p1}& \cdots & c_{pi}& \cdots & c_{p{m}_p}\end{array}& \\ {\mathit{W}}_{pq}=\begin{array}{c}{c}_{q1}\\ ⋮\\ c_{qj}\\ \begin{array}{c}⋮\\ c_{q{m}_q}\end{array}\end{array}& \left[\begin{array}{ccccc}{t}_{11}^{pq}& \cdots & t_{i1}^{pq}& \cdots & t_{m_{p}1}^{pq}\\ ⋮& & ⋮& & ⋮\\ t_{1j}^{pq}& \cdots & t_{ij}^{pq}& \cdots & t_{m_{p}j}^{pq}\\ ⋮& & ⋮& & ⋮\\ t_{1m_q}^{pq}& \cdots & t_{{\mathrm{im}}_q}^{pq}& \cdots & t_{m_p{m}_q}^{pq}\end{array}\right]& ={\left({\mathit{T}}_c^{\omega pq}\right)}^{\prime }\end{array}$$

(12)

Step 7: Calculate the weighted super-matrix ${\mathit{W}}_{\delta }$. Referring to step 5, we can calculate the total influence–relation matrix of the dimension’s matrix, T_D. Matrix ${\mathit{T}}_D^{\delta }$ can be obtained by dividing the matrix T_D by ${\delta }_i={\sum }_{j=1}^n{t}_D^{ij}$, i = 1, 2, …, n, as shown in Equations (13) and (14). The weighted super-matrix $W_{\delta }$ can be obtained from the normalized total influence–relation matrix ${\mathit{T}}_D^{\delta }$ and the un-weighted super-matrix W_pq, as shown in Equation (15).

(13)

$${\mathit{T}}_D^{\delta }=\left[\begin{array}{cccc}\raisebox{1ex}{$t_D^{11}$}\!\left/ \!\raisebox{-1ex}{${\delta }_1$}\right.& \raisebox{1ex}{$t_D^{12}$}\!\left/ \!\raisebox{-1ex}{${\delta }_1$}\right.& \cdots & \raisebox{1ex}{$t_D^{1\mathrm{n}}$}\!\left/ \!\raisebox{-1ex}{${\delta }_1$}\right.\\ \raisebox{1ex}{$t_D^{21}$}\!\left/ \!\raisebox{-1ex}{${\delta }_2$}\right.& \raisebox{1ex}{$t_D^{22}$}\!\left/ \!\raisebox{-1ex}{${\delta }_2$}\right.& \cdots & \raisebox{1ex}{$t_D^{2\mathrm{n}}$}\!\left/ \!\raisebox{-1ex}{${\delta }_2$}\right.\\ ⋮& ⋮& & ⋮\\ \raisebox{1ex}{$t_D^{\mathrm{n}1}$}\!\left/ \!\raisebox{-1ex}{${\delta }_n$}\right.& \raisebox{1ex}{$t_D^{n2}$}\!\left/ \!\raisebox{-1ex}{${\delta }_n$}\right.& \raisebox{1ex}{$t_D^{\mathrm{n}3}$}\!\left/ \!\raisebox{-1ex}{${\delta }_n$}\right.& \raisebox{1ex}{$t_D^{\mathrm{nn}}$}\!\left/ \!\raisebox{-1ex}{${\delta }_1$}\right.\end{array}\right]=\left[\begin{array}{cccc}{t}_D^{\delta 11}& t_D^{\delta 12}& \cdots & t_D^{\delta 1n}\\ t_D^{\delta 21}& t_D^{\delta 22}& \cdots & t_D^{\delta 2n}\\ ⋮& ⋮& & ⋮\\ t_D^{\delta n1}& t_D^{\delta n2}& \cdots & t_D^{\delta nn}\end{array}\right]$$

(14)

The weighted super-matrix $W_{\delta }$ can be gained by the normalized total influence–relation matrix $T_D^{\delta }$ and the un-weighted super-matrix W_pq, as described in Equation (15).

$$W_{\delta }=T_D^{\delta }{W}_{pq}=jn\left[\begin{array}{ccccc}{t}_{11}^{\delta D}\times w_{11}& \dots & t_{j1}^{\delta D}\times w_{1j}& \dots & t_{n1}^{\delta D}\times w_{1n}\\ ⋮& & ⋮& & ⋮\\ t_{1i}^{\delta D}\times w_{i1}& \dots & t_{ji}^{\delta D}\times w_{ij}& \dots & t_{ji}^{\delta D}\times w_{ij}\\ ⋮& & ⋮& & ⋮\\ t_{1n}^{\delta D}\times w_{n1}& \dots & t_{jn}^{\delta D}\times w_{nj}& \dots & t_{nn}^{\delta D}\times w_{nn}\end{array}\right]$$

(15)

Step 8: Limit weighted super-matrix $W_w$. If the weighted super-matrix is multiplied by itself many times, we can obtain the limit super-matrix, which converges and becomes stable. Therefore, we can find the IWs (also called the DANP weights) by using Equation (16), where λ is the number of multiplication iterations.

$${\mathit{W}}_W=\underset{\lambda \to \infty }{\lim }{(\mathit{W})}^{\lambda }$$

(16)

4. Data Analysis and Results

Data were collected from 10 experts to obtain the average matrix (

Table A1. The average direct influence relation matrix A.

	C11	C12	C13	C14	C21	C22	C23	C24	C25	C26	C31	C32	C33	C34	C35	C41	C42	C43
C11	0.000	3.700	3.500	3.600	1.300	2.600	0.600	1.500	1.400	1.100	0.900	0.900	0.900	0.000	1.000	0.800	0.500	0.700
C12	3.000	0.000	3.000	3.800	1.200	1.500	1.500	1.400	1.500	1.000	1.700	1.100	1.200	0.000	0.000	0.700	0.000	0.500
C13	3.500	2.600	0.000	2.800	1.500	1.400	1.100	0.800	1.200	1.000	0.900	1.300	1.200	0.800	0.700	0.400	0.400	1.500
C14	3.000	2.600	3.200	0.000	2.400	1.200	1.300	1.500	1.400	1.200	1.100	1.300	1.300	1.000	0.400	0.700	0.400	0.900
C21	0.000	0.300	1.000	0.000	0.000	2.800	1.900	2.700	1.800	3.200	2.000	2.100	2.200	1.000	0.500	1.100	0.500	0.900
C22	0.300	0.000	1.000	0.200	1.700	0.000	1.700	2.400	1.700	2.700	1.800	1.600	1.600	0.600	1.200	1.300	1.100	1.600
C23	0.000	0.000	1.000	0.400	1.800	1.600	0.000	2.000	2.400	2.000	2.300	1.600	1.700	0.400	0.400	0.700	0.200	1.900
C24	0.000	0.400	0.700	0.000	2.400	2.500	2.400	0.000	2.400	2.100	3.400	2.700	2.900	2.800	1.000	2.200	0.800	2.300
C25	0.300	0.400	1.000	0.000	2.100	2.000	2.300	2.800	0.000	2.500	2.900	1.700	1.700	2.500	1.400	0.900	0.600	1.700
C26	0.200	0.000	1.000	0.800	1.500	1.800	2.100	1.700	1.600	0.000	3.600	2.800	2.100	2.500	1.000	2.100	1.000	1.800
C31	0.100	0.000	0.800	0.000	0.500	1.300	1.400	0.500	0.900	1.500	1.800	1.600	2.200	0.000	1.800	3.000	2.300	2.900
C32	0.100	0.000	0.900	0.000	0.600	1.100	1.100	0.500	0.200	1.500	1.600	1.500	2.300	2.900	0.000	3.200	2.100	3.200
C33	0.000	0.000	0.600	0.200	0.800	0.600	0.700	0.300	0.000	1.300	1.700	1.600	2.100	2.600	1.700	0.000	1.200	2.400
C34	0.200	0.000	3.200	0.400	0.800	0.500	0.400	0.500	0.300	1.400	1.500	1.100	2.000	2.800	1.900	2.200	0.000	2.500
C35	0.900	0.000	3.100	0.200	0.400	0.700	0.200	0.400	0.000	1.000	1.800	1.300	1.600	1.600	2.000	2.300	1.600	0.000
C41	0.000	0.200	1.100	0.000	1.200	1.300	1.000	1.100	1.400	1.300	0.000	2.300	2.700	1.400	0.800	1.400	0.700	1.500
C42	0.400	0.000	1.300	0.600	0.800	1.600	1.600	0.700	0.600	0.100	1.500	0.000	2.800	2.300	1.100	1.000	0.700	1.300
C43	0.500	0.500	0.900	0.900	0.600	1.500	1.500	0.900	0.600	0.500	1.200	2.000	0.000	2.400	0.900	1.000	0.700	1.700

Note: The average gap ratio in consensus (%) = $\frac{1}{k\left(k-1\right)}{\sum }_{i = 1}^k{\sum }_{j = 1}^k\left(\frac{\left|d_{ij }^s- d_{ij}^{s - 1}\right|}{d_{ij}^s}\right)$ *100% = 4.7 < 5%, where k is the number of criteria (m = 18), s is the number of experts (s = 10), and significant confidence reached 95.3% (more than 95%).

of Appendix A). The details of the questionnaire for the data collection are illustrated in Appendix B. The DEMATEL method was then used to calculate the influential relationships among dimensions and criteria based on pairwise comparisons (Table A1), and the INRM was constructed. Finally, the influential weights of the dimensions and criteria were determined by DANP.

4.1. Descriptive Analysis of Data Collection

We collected data from 10 experts who have more than 10 years of experience in the OGD industry. Four of the experts were government officials engaged in OGD management, three were scholars engaged in OGD-related research, and three were senior managers who use OGD for application-development enterprises. We collected all standard data through an expert questionnaire in July 2018. Every expert took 40–60 min to complete the survey. The average matrix A was calculated from the means of all experts’ responses to the questionnaires, and is presented in Table A1 in Appendix A. The confidence level of the 10 questionnaires was 95.3% (a little more than 95%), and consistency gaps were 4.7% (little smaller than 5%). The results show that there is a good consistency, and the 10 questionnaires can reflect most real situations.

4.2. Assembling the Influence Network via the DEMATEL Method

Using Equations (2) and (3), we can obtain the normalized directed-relation matrix. Consequently, the total influence matrix T can be calculated with Equation (4).

Table A2. Total influence matrix of criteria T.

	C11	C12	C13	C14	C21	C22	C23	C24	C25	C26	C31	C32	C33	C34	C35	C41	C42	C43
C11	0.068	0.161	0.218	0.172	0.137	0.190	0.122	0.142	0.130	0.143	0.163	0.153	0.166	0.116	0.105	0.068	0.161	0.218
C12	0.146	0.054	0.195	0.172	0.128	0.153	0.139	0.133	0.128	0.131	0.175	0.150	0.164	0.107	0.071	0.146	0.054	0.195
C13	0.157	0.126	0.110	0.143	0.132	0.148	0.125	0.113	0.116	0.129	0.150	0.152	0.163	0.129	0.092	0.157	0.126	0.110
C14	0.146	0.127	0.205	0.065	0.164	0.152	0.140	0.140	0.129	0.144	0.168	0.163	0.177	0.144	0.089	0.146	0.127	0.205
C21	0.035	0.035	0.120	0.038	0.087	0.187	0.155	0.168	0.134	0.197	0.196	0.188	0.205	0.154	0.093	0.035	0.035	0.120
C22	0.042	0.026	0.117	0.042	0.130	0.099	0.141	0.151	0.123	0.176	0.180	0.164	0.179	0.136	0.110	0.042	0.026	0.117
C23	0.031	0.024	0.109	0.043	0.127	0.138	0.084	0.135	0.138	0.148	0.183	0.154	0.170	0.118	0.079	0.031	0.024	0.109
C24	0.042	0.042	0.135	0.045	0.171	0.200	0.188	0.106	0.164	0.189	0.262	0.231	0.257	0.233	0.129	0.042	0.042	0.135
C25	0.047	0.041	0.131	0.042	0.154	0.175	0.174	0.177	0.088	0.188	0.233	0.188	0.206	0.207	0.128	0.047	0.041	0.131
C26	0.046	0.030	0.133	0.064	0.135	0.165	0.165	0.142	0.131	0.112	0.248	0.216	0.216	0.208	0.118	0.046	0.030	0.133
C31	0.030	0.027	0.106	0.031	0.099	0.119	0.105	0.098	0.099	0.117	0.103	0.163	0.188	0.139	0.087	0.030	0.027	0.106
C32	0.042	0.024	0.109	0.048	0.084	0.122	0.115	0.082	0.073	0.078	0.137	0.089	0.182	0.155	0.091	0.042	0.024	0.109
C33	0.046	0.038	0.101	0.058	0.079	0.120	0.113	0.088	0.074	0.090	0.131	0.147	0.102	0.158	0.086	0.046	0.038	0.101
C34	0.030	0.027	0.106	0.031	0.099	0.119	0.105	0.098	0.099	0.117	0.103	0.163	0.188	0.139	0.087	0.030	0.027	0.106
C35	0.042	0.024	0.109	0.048	0.084	0.122	0.115	0.082	0.073	0.078	0.137	0.089	0.182	0.155	0.091	0.042	0.024	0.109
C41	0.030	0.027	0.106	0.031	0.099	0.119	0.105	0.098	0.099	0.117	0.103	0.163	0.188	0.139	0.087	0.030	0.027	0.106
C42	0.042	0.024	0.109	0.048	0.084	0.122	0.115	0.082	0.073	0.078	0.137	0.089	0.182	0.155	0.091	0.042	0.024	0.109
C43	0.046	0.038	0.101	0.058	0.079	0.120	0.113	0.088	0.074	0.090	0.131	0.147	0.102	0.158	0.086	0.046	0.038	0.101

in Appendix A presents the equations, which show that all of the criteria of the relationships have complex interrelationships. We can calculate the matrix of overall influence dimensions by averaging the influence matrix T within each dimension.

Table A3. Total influence matrix of dimensions T_D.

	D1	D2	D3	D4
D1	0.142	0.138	0.109	0.162
D2	0.061	0.148	0.142	0.204
D3	0.059	0.090	0.140	0.155
D4	0.055	0.098	0.113	0.138

in Appendix A shows the detailed steps.

Table A4. Sum of influences given and received on criteria and dimensions.

Dimensions	ri	ci	ri + ci	ri – ci	Criteria	ri	ci	ri + ci	ri – ci
D1	0.550	0.316	0.867	0.234	C11	2.532	1.081	3.613	1.451
					C12	2.345	0.876	3.221	1.469
					C13	2.320	2.444	4.764	−0.124
					C14	2.501	1.159	3.660	1.343
D2	0.613	0.516	1.129	0.097	C21	2.365	2.035	4.400	0.330
					C22	2.217	2.490	4.707	−0.273
					C23	2.018	2.241	4.259	−0.223
					C24	2.920	2.058	4.978	0.863
					C25	2.611	1.854	4.465	0.758
					C26	2.607	2.438	5.046	0.169
D3	0.454	0.541	0.995	−0.087	C31	2.094	2.807	4.901	−0.713
					C32	2.099	1.838	3.937	0.262
					C33	1.663	2.509	4.172	−0.846
					C34	2.049	1.558	3.607	0.491
					C35	1.810	2.934	4.744	−1.124
D4	0.409	0.715	1.124	−0.306	C41	1.854	3.094	4.948	−1.239
					C42	1.746	2.858	4.604	−1.112
					C43	1.759	3.240	4.999	−1.482

indicates the sum of influences given by and received from the criteria and dimensions. We obtained the sum using Equations (5) and (6) in step 4.

Table A4 shows that “environment” (D₁) (r_i – c_i) had the largest positive value (0.234), which indicates the most influential dimension in the OGD evaluation system. Furthermore, the environment has a deep influence on the other dimensions. However, “impact” (D₄) (r_i – c_i) had the smallest negative value (−0.306), which indicates it is the most easily influenced by other dimensions. Accordingly, “environment” should be considered a crucial factor in OGD improvement, and “impact” should be considered as the affected dimension. In addition, “data” (D₂) (r_i + c_i) had the highest positive value (1.129). “Data” should be considered the most interactive relationship with other dimensions, and had the most interactive dimension from the perspective of experts.

It is well acknowledged that the environment of laws, policies, political will, and leadership influence the completeness of opening data, the usability of data, and other factors [29]. By reusing government data, stakeholders create new value that is impacted by data attributes, such as completeness, accuracy, timeliness, understandability, accessibility, and usability of data [58]. Therefore, data affects the other dimensions, such as use and impact, and is also affected by the environment. In contrast, “environment” (D₁) (r_i + c_i) had the lowest relation (0.867) to other dimensions, meaning that it has the smallest total degree of influence among the dimensions. From Table A4, the INRM can be drawn by illustrating the influence network relationship of the four dimensions and 18 criteria, as shown in Figure 1.

4.3. Discovering the Influential Weights in DANP

The influential weights can be calculated from Equation (16), as shown in

Table 2. Influence weights of the OGD evaluation system.

Dimensions	Local Weight	Ranking	Criteria	Local Weight	Ranking	Global Weight
D1	0.1465	4	C11	0.1925	17	0.0282
			C12	0.1490	18	0.0218
			C13	0.4546	5	0.0666
			C14	0.2039	16	0.0299
D2	0.2385	3	C21	0.1534	14	0.0366
			C22	0.1939	8	0.0462
			C23	0.1762	11	0.0420
			C24	0.1544	13	0.0368
			C25	0.1393	15	0.0332
			C26	0.1829	9	0.0436
D3	0.2720	2	C31	0.2442	6	0.0664
			C32	0.1598	10	0.0435
			C33	0.2114	7	0.0575
			C34	0.1360	12	0.0370
			C35	0.2485	4	0.0676
D4	0.3430	1	C41	0.3272	2	0.1122
			C42	0.3141	3	0.1077
			C43	0.3587	1	0.1230

. The results indicate that “impact” (D₄) has the largest IW, followed by “use” (D₃), “data” (D₂), and “environment” (D₁). The “economic” (C₄₃) (12.3%), “social” (C₄₁) (11.22%), and “political” (C₄₂) (10.77%) criteria are the three most significant, followed by “interactive” (C₃₅), “general public” (C₁₃), and “interface” (C₃₁). On the other hand, “political will and leadership” (C₁₂) (2.18%), “laws and policy” (C₁₁) (2.82%), and “technical capacity” (C₁₄) (2.99%) are the least important criteria in the OGD sustainability evaluation system.

5. Discussion and Implications

The evaluation system for OGD sustainability is illustrated in Figure 1, with “environment” (D₁) on the top and “impact” (D₄) on the bottom. “Environment” (D₁) influences other dimensions, such as “data” (D₂), “use” (D₃), and “impact” (D₄). This shows that “environment” (D₁) is the main influential factor that affects the other dimensions, such as “data” (D₂), “use” (D₃), and “impact (D₄).” “Impact” (D₄) is an important result factor that is influenced by “environment” (D₁), “data” (D₂), and “use” (D₃). “Data” (D₂) is on the left, and is a mixed factor that influences “impact” (D₄) and is influenced by “environment” (D₁) and “use” (D₃).

The major influence route between dimensions can also be formed, as shown in Figure 2, where the primary cause is “environment” (D₁), and “impact” (D₄) is located in the center of the OGD sustainability evaluation system. This means that “environment” (D₁) is the most important influential factor. Therefore, the environment is the basis of OGD that provides useful conditions. To improve the sustainability of OGD, the government should strengthen environmental development. Furthermore, “impact” (D₄) was affected by other dimensions, which means that it is an important element. Using the DANP method, we can calculate the IWs shown Table 2. “Impact” (D₄) and “use” (D₃) are the two most important dimensions, and the cumulative total proportion is 61.5% in the OGD sustainability evaluation system. Therefore, the government should consider the usability of the OGD platform, and make it easy to use for public operation and data reuse. Improving the impact of OGD will create social value, political value, and economic value, and will promote sustainable OGD.

Figure 1 shows that the factors that affect other indicators in each dimension criterion are “political will and leadership” (C₁₂), “understandability” (C₂₄), “presentation” (C₃₄), and “political” (C₄₂). Notably, “political will and leadership” (C₁₂) has the highest priority, because OGD requires strong government political will and leadership [8,35]. In the “environment” (D₁) dimension, “political will and leadership” (C₁₂) influences “laws and policy” (C₁₁), “social public” (C₁₃), and “technical capacity” (C₁₄), while “general public” (C₁₃) is affected by other criteria. Thus, to improve the sustainability of OGD, government administrators must have a strong will and political leadership.

Moreover, “understandability” (C₂₄) also deserves further attention, because users can only utilize data if they understand it. Therefore, the data description should be simple and easy for the public to understand. Government offices should also consider “presentation” (C₃₄) because the data format is very important for the reuse of data. The OGD platform should provide a machine-readable format such as XLS, CSV, JSON, or XML. Citizens and organizations that access the data interface could develop applications to create value and to achieve the goals of OGD [3]. Furthermore, additional attention should be given to the “political” factor (C₄₂). Political values, such as government transparency and public participation, are conducive to creating social and economic value.

The IWs between dimensions were calculated by DANP to evaluate OGD sustainability (Table 2), and the dimensions have the following ranking in terms of priority for making improvements: “impact” (D₄), “use” (D₃), “data” (D₂), and “environment” (D₁). For government officials, the result is clear and simple, and provides improved prioritization for many complex criteria. “Economic” (C₄₃), “social” (C₄₁), and “political” (C₄₂) are the three most significant criteria, with weights of 0.123, 0.1122, and 0.1077, respectively. These indicators constitute the “impact” (D₄) dimension. The format and timeliness of OGD are the basis of data utilization. The government should provide a variety of machine-readable data formats and update the data promptly to eliminate “zombie” data. Citizens and organizations should be able to reuse the open data easily to create economic value by downloading or directly importing it to their computer systems. The government should open as much data as possible in all areas to improve government transparency and generate political value.

Furthermore, the following three criteria are important: “interactive” (C₃₅), “general public” (C₁₃), and “interface” (C₃₁)”, which have weights of 0.0676, 0.0666, and 0.0664, respectively. Hence, government officials should pay attention to the interactive functions and improve the user-friendliness of the OGD platform’s interface operations. Strengthening personal privacy protection in the process of OGD could enhance users’ trust and increase the degree of public support and participation in OGD. The three indicators with the weakest weights were “political will and leadership” (C₁₂), “laws and policy” (C₁₁), and “technical capacity” (C₁₄), which had weights of 0.0218, 0.0282, and 0.0299, respectively. Thus, these three indicators play relatively small roles in the OGD sustainability evaluation system.

For long-term enhancement, the government should play a leading role in the continuous opening of government data, which requires strong political will and leadership in the government. Therefore, the government should pay full attention to OGD, actively promote it, and provide corresponding policy and technical support. This study shows that social, political, and economic value are important indicators for the evaluation of OGD sustainability. Therefore, government officers should promote the value generated by data.

6. Conclusions

This study used 18 criteria to establish an evaluation system for OGD sustainability. The results indicated that “environment” (D₁) is the main factor that influences other dimensions, and “impact” (D₄) is an important result. The three most significant criteria were the “economic” (C₄₃),” “social” (C₄₁),” and “political” (C₄₂) criteria, which had about 34% of the total evaluation weight. A new mixed model was constructed as an evaluation system of OGD sustainability by integrating the DEMATEL and DANP methods. We identified complex relationships among criteria dimensions, and the INRM was constructed using the DEMATEL method. The INRM can reflect the causal relationships between the dimensions and criteria. The IWs of the dimensions and criteria can be calculated by applying the DANP method. Some limitations remain that require further study. First, this study only established an evaluation system for OGD sustainability, but the model was not applied to evaluate an OGD project. Therefore, in the future, to validate the model and improve the sustainability of OGD, we will apply this system to evaluate existing cases of OGD. Second, the assessment model included environment, data, use, and impact dimensions. The other possible dimensions of sustainability of OGD were not considered, such as data security or privacy protection. The other dimensions could be included to improve the evaluation model in the future work. Third, to help government managers identify the key factors and priorities for improving the sustainability of OGD, we calculated the weights of dimensions and criteria using the DANP method. Future studies could use additional multi-criteria approaches, such as data envelopment analysis (DEA), to estimate the relative IWs for the sustainability of OGD. The results of these studies could then be compared with those presented here.