Next Article in Journal
Review of Transition from Mining 4.0 to Mining 5.0 Innovative Technologies
Next Article in Special Issue
Geosystemic Impacts of the Extreme Rainfall Linked to the El Niño 2015/2016 Event in Northern Paraná, Brazil
Previous Article in Journal
Vibration Signal Diagnostic Information of Reinforced Masonry Elements Destruction
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Assessing Vulnerability in Flood Prone Areas Using Analytic Hierarchy Process—Group Decision Making and Geographic Information System: A Case Study in Portugal

by
Sandra Mourato
1,2,*,
Paulo Fernandez
2,3,
Luísa Gomes Pereira
4,5 and
Madalena Moreira
2,6
1
School of Technology and Management, Polytechnic Institute of Leiria, 2411-901 Leiria, Portugal
2
MED&CHANGE—Mediterranean Institute for Agriculture, Environment and Development & CHANGE—Global Change and Sustainability Institute, Universidade de Évora, 7006-554 Évora, Portugal
3
Polytechnic Institute of Castelo Branco, 6001-909 Castelo Branco, Portugal
4
Águeda School of Technology and Management, Aveiro University, Rua Comandante Pinho e Freitas, Nº 28, 3750-127 Águeda, Portugal
5
Research Centre of Geospatial Sciences, Porto University, 4099-002 Porto, Portugal
6
Architecture Department, Universidade de Évora, Ap. 94, 7006-554 Évora, Portugal
*
Author to whom correspondence should be addressed.
Submission received: 7 March 2023 / Revised: 10 April 2023 / Accepted: 12 April 2023 / Published: 13 April 2023

Abstract

:
A flood vulnerability index was constructed by coupling Geographic Information System (GIS) mapping capabilities with an Analytic Hierarchy Process (AHP) Group Decision-Making (GDM) resulting from a paired comparison matrix of expert groups to assign weights to each of the standardised criteria. A survey was sent to 25 flood experts from government organisations, universities, research institutes, NGOs, and the private sector (56% academics and 44% non-academics). Respondents made pairwise comparisons for several criteria (population, socio-economic, buildings, and exposed elements) and sub-criteria. The group priorities were obtained by combining the Consistency Ratio (CR) and Euclidean Distance (ED) measures to assess the weight of each expert and obtain a final weight for each criterion and sub-criteria. In Portugal, 23 flood-prone areas were considered, and this work contributes with a tool to assess the flood vulnerability and consequently the flood risk. The flood vulnerability index was calculated, and the relevance of the proposed framework is demonstrated for flood-prone areas, in mainland Portugal. The results showed that in all five hydrographic regions, flood-prone areas with very high vulnerability were found, corresponding to areas with a high probability of flooding. The most vulnerable areas are Ponte de Lima in the North, Coimbra, and Pombal in the Centre; Loures in the Tagus and West Region; Setúbal and Alcácer do Sal in the Alentejo Region and Monchique in the Algarve Region. This methodology has the potential to be successfully applied to other flood-prone areas, combining the opinions of stakeholders validated by a mathematical model, which allows the vulnerability of the site to be assessed.

1. Introduction

Flood vulnerability is part of the flood risk concept [1]. Vulnerability can be divided into three components: (i) exposure to a natural event, (ii) susceptibility or resistance, and (iii) adaptive capacity, also known as resilience.
In recent years, there has been rapid methodological development in flood vulnerability assessment studies [2].
The integration of multicriteria decision analysis (MCDA) into a geographic information system (GIS) allows the development of a GIS multicriteria decision support framework for flood vulnerability assessment [3,4,5]. Some of these studies use flood vulnerability indices. The design of a flood vulnerability index can be divided into several stages: conceptual framework, structural design, scale of analysis, indicator selection, normalisation, weighting, and aggregation [6]. Flood vulnerability is multidimensional and has spatial indicators with a complex variety of sources, types, and scales. The hierarchical model used criteria divided into sub-criteria that share the same core dimension of vulnerability [6]. The relative importance of vulnerability indices was calculated using weighting methods, such as principal component analysis (PCA) or analytic hierarchy process (AHP) [2,7,8,9,10,11,12,13,14]. The AHP is a multi-criteria decision-making tool and has three underlying concepts [15]: (i) structuring the complex decision problem as a hierarchy of objective, criteria and sub-criteria; (ii) pairwise comparison (i.e., judgement) of elements at each level of the hierarchy; and (iii) vertical synthesis of the judgements across the different levels of the hierarchy. The AHP method was used in most studies on flood risk management through MCDA with application to the risk, hazard and vulnerability assessment [16]. AHP has been used in several studies to assess either flood hazard [17,18], flood risk [2,19,20,21,22,23], and flood vulnerability [10,24].
Although AHP was initially developed for an Individual Decision Making (IDM) context [2,25], other authors [4,12,26] applied a methodological extension for coping with multi-person Group Decision Making (GDM) context. Compared to individual decision-making, group research has the potential to gather more complete information. Since the group has several members with different points of view and different experiences on a particular issue, it will improve the decision-making. A GDM with experts in the field of hydrology was used for potential flood-prone areas mapping [27], which is following the implementation of the EU Flood Risk Directive [28]. The assessment of vulnerability addressed in the Flood Risk Management Plans (FRMP) at the river basin district level issued by the Portuguese Environment Agency, adopts the GIS-MCDA methodology developed by Fernandez et al. [29].
However, to the best of our knowledge, except for the study by de Brito et al. [12], no other study has considered group decision-making for the assessment of flood vulnerability. In that study, the experts were asked to construct the problem structure and establish the structure of the vulnerability index, and the questionnaire was answered in workshops. In this approach, the experts were actively involved in all steps of the vulnerability modelling process and thus had a great influence on the final index, but it was very time-consuming for all the participants. When dealing with AHP-GDM, there is a need to integrate the results. Two of the most traditional aggregation approaches are Aggregation of Individual Judgements (AIJ) and Aggregation of Individual Priorities (AIP): (i) In AIJ, the individual pairwise comparison matrices are first aggregated to obtain a group matrix. The priority vector is then derived from this new matrix. (ii) In AIP, the priority vectors are first obtained for each individual, then aggregated to obtain the group priorities. When using GDM, it is necessary to ensure the consistency measure of each expert itself and the consistency measure of the whole group [30]. Aguarón et al. [31] identify the following differences: (1) Individual consistency is related to the (internal) coherence of the decision maker when considering his/her judgement in the pairwise comparison matrices. Therefore, consistency is usually assessed as the ‘representativeness’ of the local priority vector derived from the pairwise comparison matrices, and (2) group consistency refers to the (internal) coherence of the group in selecting its priority vector, i.e., its representativeness of the individual positions. The AHP-GDM methodology of Aguarón et al. [31], although quite complete, would be very computationally time-consuming in the case of a large number of criteria and sub-criteria involved. Srdevic et al. [32] presented a methodology that minimises the risk of negligent, and incompetent or irresponsible group decisions by assigning weights to experts according to their demonstrated individual inconsistencies. This methodology considered two measures of consistency that are commonly used as indicators of decision quality. The group decision is derived by assuming that the importance of the experts is correlated with their demonstrated consistency measures of Consistency Ratio (CR) and Euclidean Distance (ED) at each level of the hierarchy tree. This method was well accepted by the stakeholders participating in the study.
The main objective of this paper is to propose a methodology for flood vulnerability assessment with multiple criteria (e.g., population, buildings, socio-economic, and exposed elements) and sub-criteria, integrating GIS in a multicriteria decision support framework with an AHP-GDM supported by expert opinions (GIS-AHP-GDM). The proposed methodology allies the different expert opinions obtained through an online survey and validated by a mathematical model. The individual consistency of the experts can be evaluated by the CR and the ED when judging the pairwise comparison matrices. The judgments are subject to aggregation of individual priorities, and the rationale for criteria selection, weighting and aggregation is justified based on the choices made in previous studies [29]. The GIS representation of the flood vulnerability index at the neighbourhood level can be used for decision-making since high-resolution spatial data are required to improve the flood risk assessment.

2. Materials and Methods

2.1. Study Area

The study area is the Portuguese mainland, with a focus on the flood-prone areas identified by the Portuguese Environment Agency following the EU Flood Risk Directive. The Portuguese Environment Agency prepared a preliminary flood risk assessment for each hydrographic region to identify areas of significant potential risk from fluvial flooding, known as critical zones, based on recorded historical events (Figure 1) and a return period of 100 years.

2.2. Data

Two spatial data sets support the study: (i) Portuguese Statistics Geographical Information (Census 2021) for small territorial units (neighbourhoods). The neighbourhood is the territorial unit that identifies the smallest homogeneous area, representing a block in urban areas, and (ii) the Land Use and Land Cover (LULC), named as COS 2018 (minimum mapping unit of 1 ha), provided by the Direção-Geral do Território, available on the Web Feature Services [33].
The variables are presented in Table 1, distributed according to the criteria used in the group based AHP built on the hierarchical structure. From the Census (2021) data, it was possible to characterise the population, buildings, and socio-economy. COS 2018 data were used to characterise the exposed elements.
The rationale behind the choice of these variables can be found in Fernandez et al. [29] and Fernandez et al. [9] and were summarized here.
Population variables take into account age structure, gender, and family size. Age structure was divided into three classes to allow for the classification of vulnerability by age. Children and older people are thought to be more vulnerable to flooding [34,35,36]. Gender is used to differentiate vulnerability between women and men. Women tend to be more risk-aware and prepared to act [35]. However, women may face more difficulties during the recovery phase, often due to their lower wages and family care responsibilities [34,35]. The number of family members is used because larger families often have to share sources of income and have more dependents to evacuate, such as children and the elderly [34,35,37,38].
Building variables include the year of construction, number of storeys, building function and use. The year of construction was divided into buildings constructed before and after 1980. In 1980, Portugal introduced safety regulations for building structures. Older neighbourhoods also have older drainage systems. These may be more flood-prone [39]. The number of storeys was divided into buildings with one or two floors and with more than three floors. [40]. People and their property can be protected when on the upper floors of buildings. Collective buildings (e.g., nursing homes, hotels, hospitals, and prisons) are more vulnerable, because they present additional difficulties in the evacuation and resettlement process.
Socio-economy variables include the level of education, housing occupancy, unemployment rate, and illiteracy rate. The level of education and the illiteracy rate are evident factors of socio-economic vulnerability because there is a relationship between these factors and economic capacity, social status, and job opportunities [34,35]. The level of education was divided into three hierarchical levels: junior high school, high school, and university. Property owners are more likely than tenants to make structural changes to their buildings and to take out or improve their existing insurance policies [41]. People who rent a house do not have the financial resources for home ownership. They often do not have access to information about financial support for rebuilding [34,35]. Many property owners have a mortgage with a bank and are therefore insured. The unemployment rate is related to individual vulnerability, as lack of employment leads to lower income. This socio-economic deprivation reduces an individual’s likelihood of coping with the consequences of an adverse event and their capacity to recover [41].
Exposed element variables include land use, population, and building density. Floods in urban areas affect economic activities and a large number of people due to the temporary covering of water, causing greater difficulties in emergency management and greater economic damage [42]. There is also economic damage in agricultural areas due to the temporary covering of crops by floodwaters, but the seasonality of agricultural practice coincides with periods of the year that have a lower probability of flooding; however, climate change and the increased frequency of extreme events may contribute to a change in this relationship [43]. Forest areas may be covered by water, but the damage will be extremely low. Some species are more tolerant of flooding, and slightly tolerant species can survive up to 50 days of flooding [44]. Population and building density are factors that influence social vulnerability in risk areas [41,45]. In urban areas, the high density of the population makes the rescue process more difficult. In some cases, higher population density indicates a higher number of poor people [46].

2.3. Methodology

To assess the flood vulnerability of the Portuguese flood-prone areas, the methodology includes five steps: (1) database creation; (2) vulnerability model hierarchical structure; (3) online expert survey; (4) criteria weighting (AHP-GDM); and (5) vulnerability mapping. Figure 2 shows the methodology flowchart.

2.4. Hierarchy Structure of the Vulnerability Method

A conceptual model must be created concerning the selected criteria. The AHP method requires the decomposition of the decision problem into a hierarchy tree with sub-criteria. AHP requires a goal to be specified at the top level, followed by criteria and sub-criteria. A schematic representation of these levels is given in Figure 3, corresponding to the criteria and the associated code.
The first level of the hierarchical structure represents the objective of the analysis, in this case, flood vulnerability. The second level comprises the four criteria for assessing vulnerability: Population, Buildings, Socio-economic, and Exposed Elements. Finally, the sub-criteria evaluate the flood vulnerability criteria (fourteen sub-criteria at the third level and twenty-one sub-criteria at the fourth level).

2.5. Online Expert Survey

Decision-making should involve multiple expert opinions. In this study, an expert refers to a person with extensive knowledge of flood vulnerability analysis, acquired through field experience or education [47].
Therefore, an online survey was structured to include the perceptions and priorities of a group of experts concerning flooding in the vulnerability assessment. Experts usually have different attitudes, interests, and knowledge about the problem, which can be reflected in a certain inconsistency in the decision-making process. Therefore, the participation of several experts in the weighting process is considered essential. The online survey was carried out with 25 people (56% from the academic world and 44% from the non-academic world) selected from among flood experts from governmental organisations (8% national laboratories, 12% civil protection authorities, and 8% municipalities), universities (37% professors and 19% researchers from the fields of natural hazards, geographic information, hydrology and hydraulics, and social sciences), NGOs (4%) and private companies (12%), distributed all over Portugal. The composition of this panel allows for different points of view, rather than deriving a single solution based on a single opinion. The number of members of the panel can be considered sufficient, as the experts have to be part of a comprehensive questionnaire. Just as important as the number of members of the group is the variability of the sectors from which they come and their geographical distribution. The survey was conducted online, and the respondents do not know each other. Their participation is limited to the first round of responses. There are three reasons why we did not encourage a meeting between the participants in our methodology: (i) the gathering of group members takes a lot of time; (ii) if the experts are together in a group, this can give some bias to the results. Group dynamics can have some pitfalls, such as members often follow the opinion of the majority or of a more persuasive group member, or they may be concerned with their reputation within the group; and (iii) the method relies on a mathematical validation that, for each criterion, gives higher weights to the most consistent answers. Therefore, it is not possible to follow a group consensus approach and the weighting of each expert in the final priority vector is completed without the intervention of the experts.
Throughout the questionnaire, the AHP pairwise comparisons were made by asking the experts: “Which of the two criteria contributes more to the vulnerability? The questionnaire contains fourteen comparison questions (nine questions at the fourth level, four questions at the third level and one question at the second level). The response options were the nine-point numerical scale of Saaty [15] (Table 2).
As an example, the second level questions are shown in Figure 4.

2.6. Analytic Hierarchy Process—Group Decision-Making

This work uses the AIP aggregation method: Priority vectors (including criteria weights) are first obtained for each individual, then aggregated to obtain the group priorities.
The AHP determines the preferences among the set of items at a given level of a hierarchy by making pairwise comparisons of these items with respect to the items at the higher level, using Saaty’s scale. As only two criteria are compared at a time, this method reduces the complexity of the problem. Once these comparisons have been made, the weights of the criteria are obtained using the principal eigenvector of the matrix [15].
The consistency ratio (CR) and Euclidean Distance (ED) are used to weigh the responses of the experts involved. Assuming that the weight of the experts is correlated with their demonstrated consistency measures of CR and ED at each level of the hierarchy tree, the group decision is derived by defining the weight of each criterion by each expert in the group. Finally, the aggregation of individual priorities is obtained through a weighted geometric mean. This is based on the final weight of each expert for each vulnerability criterion.
The CR, is defined as Equation (1):
C R = C I C R
where the random index (RI) is a tabulated value for the number of criteria (n), and CI is the consistency index (2).
C I = λ m a x n n 1
where λmax is the eigenvalue and n is the size of the comparison matrix.
The CR defines the probability that the matrix scores are randomly generated. Saaty [15] suggests that matrices with CR values greater than 0.10 should be re-evaluated. The CR depends on the size of the matrix and the analysis: for individual experts, the CR is limited to 0.10 or 0.20, while for group respondents the CR could be relaxed to a higher value to allow for non-expert responses [48,49].
The other consistency measure used was the ED. This measure compares the entries of the comparison matrix (aij) as entered by each expert and the corresponding ratios wi/wj of the computed weights (which should ultimately approximate the elicited entries, i.e., aij = wi/wj) [32]. Thus, as given in Equation (3), the ED measures the total distance between all judgment elements in the comparison matrix and the corresponding priority ratios contained in the derived vector w. Thus, the lower the ED, the higher the expert consistency.
E D = [ i = 1 n j = 1 n ( a j w i / w j ) 2 ] 1 / 2
where aj are the entries of the comparison matrix and wi/wj are the corresponding ratios of the calculated weights.
This method screens the hierarchy tree and adds local CRs and EDs. It takes into account the expert opinions. Using both CRs and EDs is a simpler way of assessing consistency within the group than using CRs alone. Each expert’s weight in the final group weight is the combination of these consistency measures. Following the methodology of Srdevic et al. [32], the procedure for calculating each expert’s final weight is briefly outlined as follows:
1. For each expert, the CR and ED consistency scores are calculated for all pairwise matrices.
2. To estimate the level of overall consistency of the expert responses, the CR values for each expert are summed. The higher the sum, the lower the quality of his answers.
3. The ED scores for each expert are summed to estimate the consistency of their answers compared to the other experts. The higher the sum, the lower the quality of his answers.
4. Calculate the inverse of the sum of the CRs obtained in step 2, computed for each of the experts.
5. Calculate the inverse of the sum of the EDs obtained in step 3, computed for each of the experts.
6. Normalise the values of CRs and EDs obtained in step 4 and step 5, respectively, with the sum of all values obtained for each expert.
7. The final weight of each expert (α_k) is the mean of the normalised values of CRs and EDs obtained in step 6.
8. The final group priority vector is the geometric mean of the individual priority vectors of each expert, according to Equation (4):
z i G = k = 1 k [ z i ( k ) α k ]
where k stands for the number of experts, z i ( k ) stands for the priority of the i-th alternative for the k-th expert, α k is the weight of the k-th expert, and z i G is the aggregated group priority value;
9. The final additive normalisation of priorities ziG is calculated.
The criteria weights are used for the respondents’ partial weights for the evaluation of the overall objective, e.g., to obtain their final ranks based on the calculated final weights synthesised top-down through the hierarchy tree.

2.7. Flood Vulnerability Index Mapping

Before collecting the criterion maps in a GIS environment, the variables need to be transformed into standard units and integrated into a spatial database. This is because they are represented by different dimensional scales (e.g., percentages and density). The criteria of the vulnerability model were standardised using a linear fuzzy set membership function. In this function, vulnerability varies linearly between values ranging from 0 (no vulnerability) to 1 (total vulnerability). The standardisation functions under consideration are presented in Fernandez et al. [29]. For example, a decreasing function was applied to the percentage of residents aged between 14 and 65 years. These residents have a lower vulnerability than the residents who are less than 14 years old or more than 65 years old, and the higher vulnerability applies when the group has a lower percentage of people between 14 and 64 years old. For residents over 65 and over 14, who are theoretically the most vulnerable, the function increases.
Several sub-criteria and criteria maps have been calculated according to the hierarchical structure: (i) at level 4, 21 sub-criteria (P11, P12, P13, P21, P22, P31, P32, S11, S12, S13, S21, S22, B11, B12, B21, B22, B31, B32, E11, E12, and E13), (ii) at level 3, 14 sub-criteria (P1, P2, P3, S1, S2, S3, S4, B1, B2, B3, B4, E1, E2, and E3); (iii) at level 2, 4 criteria (P, B, S, and E) and the vulnerability map at the target level.
The flood vulnerability maps are a planning tool. They are essential for flood risk assessment and management by decision-makers. The map algebra with weighted overlay was used to flood vulnerability mapping. The standardised criteria maps were multiplied by the final weights from the AHP-GDM (Section 2.6) and successively summed bottom-up in the hierarchical structure to produce flood vulnerability maps using GIS-AHP-GDM. All spatial data geoprocessing tasks for map production were developed with the ArcGIS software 10.5.1 package.
Finally, the standard deviation classification method shows how much a neighbourhood’s flood vulnerability value varies from the mean. This method forms each class by adding and subtracting the standard deviation from the mean of the flood vulnerability values dataset. The resulting vulnerability values at the neighbourhood scale were classified, respectively, into five flood vulnerability categories: (i) very low (mean − 2 std dev); (ii) low (mean − 1 std dev); (iii) medium (mean); (iv) high (mean + 1 std dev); and (v) very high (mean + 2 std dev).

3. Results and Discussion

3.1. Individual Criteria Weights

The hierarchical structure of the vulnerability index shown in Figure 3 was translated in the online survey into a total of 14 pair-wise matrices to which each of the experts responded. The CR and ED statistics of the expert survey responses to all the criteria are shown in Table 3. No expert response was discarded based on CR and ED criteria. Instead, if an expert’s individual criteria weight results from a less consistent matrix, the weight of that expert’s response will be very low, resulting in a small contribution to the final group weight.
Table 3 shows that although some of the indicators of consistency have higher values, the median for both is relatively low. In the case of CR, it is lower than the standard threshold of 0.1 and more than 75% of the results are below the 0.2 threshold. Considering a CR threshold of 0.2 [48,49], three experts had all consistent answers, fourteen had only one inconsistent answer, five had two inconsistent answers, two had three inconsistent answers, and only one had eight inconsistent answers.

3.2. Group Criteria Weights

The individual responses were then aggregated using the approach described in Section 2.6. This resulted in a group weight for each criterion. Figure 5 shows the dispersion of the individual expert weights and the final weight for each criterion.
The variable weights show a considerable spread among the experts, with the smaller differences (considering the 25th–75th percentile interval) found in the criteria S, B, P1, P2, P3, E3, S2, S3, P12, S12, E11, E12, and E13. The black dot (final weight of the criteria group) is included in the 25th and 75th percentile of the individual weights, except for the criteria S12 (number of inhabitants with high school education) and E13 (percentage of forest land use), and for some criteria it almost corresponds to the median. These results support the use of the proposed approach, as all the answers of the experts have been taken into account, but the final weight for each criterion has a mathematical analysis instead of a simple average of the individual weights.

3.3. Final AHP-GDM Weights

The final AHP-GDM weights for the flood vulnerability criteria and sub-criteria (Figure 6) were obtained by aggregating the individual weights.
The population criterion is the one that contributes more to the final vulnerability, followed by the building and exposed elements, the socio-economic criterion had the lower weight in the final vulnerability.
Within the population criterion, the age structure sub-criterion contributes significantly to vulnerability, while gender contributes the least. The percentage of the population under 14, the percentage of the population between 14 and 65, and the percentage of households with more than five persons were considered the most relevant sub-criteria. Several studies indicate that the elderly have limited mobility and physical difficulties during an evacuation, are more reluctant to leave their homes, have health problems, and have a longer recovery time after the flood event [50]. Households with many relatives are also more likely to share income and have more dependents, such as children and the elderly, to evacuate [35], making them more vulnerable. The youngest also have high levels of physical vulnerability and dependency [34,35,36,38].
For the building’s criterion, the function and percentage of collective buildings are the sub-criteria with the higher contribution. Collective buildings can contribute to increased vulnerability because they present additional difficulties in the evacuation and resettlement process [51]. The percentage of buildings built before 1980, the percentage of buildings with one or two floors, and the percentage of buildings that are exclusively or mainly residential were considered to be the most relevant sub-criteria. Newer buildings are built according to stricter building safety regulations and should therefore be more resistant. Older neighbourhoods (usually in city centres and/or close to rivers) also have older sewerage systems, which contribute to higher flood vulnerability [39]. As the number of storeys increases, flood vulnerability decreases as upper storeys can protect people and their property [40]. The results showed that residential buildings contribute more to the vulnerability than non-residential buildings. A plausible explanation, which may vary from country to country depending on the insurance practices, relates to the fact that non-residential buildings are, in general, insured.
For the socio-economic criterion, the four sub-criteria have a similar weight to the final social vulnerability. In the socio-economic criterion, the percentage of persons having completed junior high school and the percentage of rented dwellings were considered the most relevant sub-criteria. Educational attainment and illiteracy rates can be considered socio-economic vulnerability factors because of their direct relationship with economic capacity and employment opportunities [34,35]. Landlords make improvements to their buildings and have insurance coverage, unlike tenants [41].
In terms of the exposed elements criterion, the land use and population density sub-criteria have a slightly higher contribution than building density. Urban areas were considered to be the most relevant sub-criteria. In urban areas with high population densities, the rescue process can be challenging. There is also a correlation between high population density and the presence of low-income families [46]. Flooding in urban areas puts a strain on emergency response and can result in significant economic losses as the flood affects economic activities and a larger number of people [42].

3.4. Flood Vulnerability Index for Flood-Prone Areas

The five classes of the flood vulnerability index were obtained by applying the AHP-GDM weights to the characteristics of the flood-prone sites at the neighbourhood level. For each of the 23 flood-prone sites in mainland Portugal, the percentage of each vulnerability class is shown in Figure 7a. The most vulnerable sites are as follows: Ponte de Lima in the North; Coimbra and Pombal in the Centre; Loures in the Tagus and West Region; Setúbal and Alcácer do Sal in the Alentejo Region; and Monchique in the Algarve Region. Sites of high or very high vulnerability have been identified in all areas. For example, the city of Águeda, in the central region, is frequently flooded and there is considerable damage in the urban area. Nevertheless, a large area is classified as low vulnerability. This is because the flooded area consists mainly of agricultural land.
This index, in addition to the final vulnerability class, allows stakeholders to assess which criterion is the main contributor to the final vulnerability (Figure 7b–e). For all 23 flood-prone areas, Figure 7c shows that the socio-economic criterion is generally the one that contributes the least to the flood vulnerability index, and the exposed elements are the criteria that contribute the most. This is mainly because these areas are usually urban centres where urban land use predominates.
As the flood vulnerability index is supported in GIS, it is possible to spatially display the index for each flood-prone area studied. As an example, five flood-prone sites have been selected to illustrate the spatial distribution of the flood vulnerability index categories at the neighbourhood scale (Figure 8).
As we can see, the 100-year return period flood area of the Águeda site (Figure 8d) includes a large area of agricultural land that is not considered to be highly vulnerable. However, in the city centre, the vulnerability is very high; several small commercial businesses are severely affected by flooding almost every year. In Coimbra (Figure 8c), the area considered to be very highly vulnerable also includes a national monument (Santa Clara-a-Velha Monastery). This area is frequently flooded, even after several major economic investments in flood defences. This approach makes it possible to validate the results for some areas with prior knowledge.
The results seem to be in line with what is already known from past events, and since the analysis is very detailed (small territorial units), the authorities can make better decisions in terms of preparedness or intervention when a flood occurs.
These maps can be used by the authorities only with the final vulnerability (Figure 8), but the spatial representability of the flood vulnerability index calculated in this work can be extended to each criterion and, as an example, Figure 9 shows the spatial representation of the four criteria for the flood-prone area of Coimbra: Population, Socioeconomics, Buildings, and Exposed elements. This information at the second level is important for decision-makers to take supported actions to reduce vulnerability in the hotspots of the most vulnerable criterion. If needed by the decision makers, it is also possible to zoom in to the third-level maps.
The socio-economic criterion (Figure 9b) is the one that contributes less to the vulnerability. In addition, it is a criterion for which the authorities can apply very few measures to reduce vulnerability. The criterion that contributes most to the vulnerability to flooding is the Exposed Elements (Figure 9d). The city centre of Coimbra is a highly urbanised area, so the authorities need to be aware of flood events to take the necessary operational measures to deal with frequent flooding. For the population criterion (Figure 9a) and buildings (Figure 9c), the proposed methodology was able to identify the areas with more elderly people and older buildings within traditional narrow streets.

4. Conclusions

The vulnerability index has the potential to be a powerful tool as it summarises the complexity of vulnerability and provides a quantitative metric to compare the vulnerability of flood-prone areas at the neighbourhood level. Analysing flood vulnerability at this local scale is important because flood risk assessment should also use high-resolution spatial data. It is also easy for decision-makers to understand it because it can be mapped.
The methodology proposed has the advantage to include in the decision multiple expert opinions, consulted by online survey, which can be a less time-consuming activity rather than attend to several meetings to achieve consensus. In addition, the expert responses are weighted in the final score by a mathematical method, allowing the decision to be more realistic valuing the experts who demonstrated to be more consistent.
The GIS-AHP-GDM was useful for assessing flood vulnerability, considering either the community’s resilience or exposed elements. The AHP, based on expert knowledge of flood vulnerability analysis, was suitable for weighting the selected multi-criteria and decision-making to access vulnerability in flood-prone areas. These flood vulnerability maps can be used in combination with predicted flood hazard maps to support decision-making by authorities to improve flood preparedness and response.
Despite the uncertainties related to the difficulties that some experts may have experienced in answering the questionnaire, which are limitations (since validation was limited to areas where prior knowledge exists) and constraints related to gathering a heterogeneous group, this approach may encourage further conceptual and applied development of GIS-AHP-GDM for flood vulnerability assessment.

Author Contributions

Conceptualization, S.M.; methodology, S.M., P.F. and M.M.; data curation, S.M. and P.F.; formal analysis, writing—original draft preparation, S.M., P.F., L.G.P. and M.M.; writing—review and editing, S.M., P.F., L.G.P. and M.M. All authors have read and agreed to the published version of the manuscript.

Funding

This work was funded by Operational Programme of the Centre Region, in its FEDER component, under the framework of the project CLIMRisk—Climate change adaptation measures in the management of natural and environmental risks (Centro-01-0145-FEDER-023566)—and by National Funds through FCT—Foundation for Science and Technology under the Project UIDB/05183/2020.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The online survey answers are not available. The respondents was just authorization was just for results publication.

Acknowledgments

The authors are grateful to experts for participating in the online survey contributing to the criteria ranking process.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Santos, P.P.; Pereira, S.; Zêzere, J.L.; Tavares, A.O.; Reis, E.; Garcia, R.A.C.; Oliveira, S.C. A comprehensive approach to understanding flood risk drivers at the municipal level. J. Environ. Manag. 2020, 260, 110127. [Google Scholar] [CrossRef] [PubMed]
  2. Chakraborty, S.; Mukhopadhyay, S. Assessing flood risk using analytical hierarchy process (AHP) and geographical information system (GIS): Application in Coochbehar district of West Bengal, India. Nat. Hazards 2019, 99, 247–274. [Google Scholar] [CrossRef]
  3. Wang, Y.; Li, Z.; Tang, Z.; Zeng, G. A GIS-Based Spatial Multi-Criteria Approach for Flood Risk Assessment in the Dongting Lake Region, Hunan, Central China. Water Resour. Manag. 2011, 25, 3465–3484. [Google Scholar] [CrossRef]
  4. Roy, D.C.; Blaschke, T. Spatial vulnerability assessment of floods in the coastal regions of Bangladesh. Geomat. Nat. Hazards Risk 2015, 6, 21–44. [Google Scholar] [CrossRef]
  5. Hoque, M.A.; Tasfia, S.; Ahmed, N.; Pradhan, B. Assessing Spatial Flood Vulnerability at Kalapara Upazila in Bangladesh Using an Analytic Hierarchy Process. Sensors 2019, 19, 1302. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Tate, E. Social vulnerability indices: A comparative assessment using uncertainty and sensitivity analysis. Nat. Hazards 2012, 63, 325–347. [Google Scholar] [CrossRef]
  7. Remo, J.W.F.; Pinter, N.; Mahgoub, M. Assessing Illinois’s flood vulnerability using Hazus-MH. Nat. Hazards 2015, 81, 265–287. [Google Scholar] [CrossRef]
  8. Tavares, A.O.; Santos, P.P.d.; Freire, P.; Fortunato, A.B.; Rilo, A.; Sá, L. Flooding hazard in the Tagus estuarine area: The challenge of scale in vulnerability assessments. Environ. Sci. Policy 2015, 51, 238–255. [Google Scholar] [CrossRef]
  9. Fernandez, P.; Mourato, S.; Moreira, M.; Pereira, L. A new approach for computing a flood vulnerability index using cluster analysis. Phys. Chem. Earth 2016, 94, 47–55. [Google Scholar] [CrossRef]
  10. Hu, S.; Cheng, X.; Zhou, D.; Zhang, H. GIS-based flood risk assessment in suburban areas: A case study of the Fangshan District, Beijing. Nat. Hazards 2017, 87, 1525–1543. [Google Scholar] [CrossRef]
  11. Roder, G.; Sofia, G.; Wu, Z.; Tarolli, P. Assessment of Social Vulnerability to Floods in the Floodplain of Northern Italy. Weather Clim. Soc. 2017, 9, 717–737. [Google Scholar] [CrossRef]
  12. de Brito, M.M.; Evers, M.; Almoradie, A.D.S. Participatory flood vulnerability assessment: A multi-criteria approach. Hydrol. Earth Syst. Sci. 2018, 22, 373–390. [Google Scholar] [CrossRef] [Green Version]
  13. Fekete, A. Social vulnerability change assessment: Monitoring longitudinal demographic indicators of disaster risk in Germany from 2005 to 2015. Nat. Hazards 2019, 95, 585–614. [Google Scholar] [CrossRef]
  14. Xie, W.; Meng, Q. An Integrated PCA–AHP Method to Assess Urban Social Vulnerability to Sea Level Rise Risks in Tampa, Florida. Sustainability 2023, 15, 2400. [Google Scholar] [CrossRef]
  15. Saaty, T. The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation; McGraw-Hill: New York, NY, USA, 1980. [Google Scholar]
  16. De Brito, M.M.; Evers, M. Multi-criteria decision-making for flood risk management: A survey of the current state of the art. Nat. Hazards Earth Syst. Sci. 2016, 16, 1019–1033. [Google Scholar] [CrossRef] [Green Version]
  17. Kourgialas, N.N.; Karatzas, G.P. Flood management and a GIS modelling method to assess flood-hazard areas—A case study. Hydrol. Sci. J. 2011, 56, 212–225. [Google Scholar] [CrossRef]
  18. Stefanidis, S.; Stathis, D. Assessment of flood hazard based on natural and anthropogenic factors using analytic hierarchy process (AHP). Nat. Hazards 2013, 68, 569–585. [Google Scholar] [CrossRef]
  19. Lyu, H.-M.; Zhou, W.-H.; Shen, S.-L.; Zhou, A.-N. Inundation risk assessment of metro system using AHP and TFN-AHP in Shenzhen. Sustain. Cities Soc. 2020, 56, 102103. [Google Scholar] [CrossRef]
  20. Gacu, J.G.; Monjardin, C.E.F.; Senoro, D.B.; Tan, F.J. Flood Risk Assessment Using GIS-Based Analytical Hierarchy Process in the Municipality of Odiongan, Romblon, Philippines. Appl. Sci. 2022, 12, 9456. [Google Scholar] [CrossRef]
  21. Pathan, A.I.; Girish Agnihotri, P.; Said, S.; Patel, D. AHP and TOPSIS based flood risk assessment—A case study of the Navsari City, Gujarat, India. Environ. Monit. Assess. 2022, 194, 509. [Google Scholar] [CrossRef] [PubMed]
  22. Chen, Y.-R.; Yeh, C.-H.; Yu, B. Integrated application of the analytic hierarchy process and the geographic information system for flood risk assessment and flood plain management in Taiwan. Nat. Hazards 2011, 59, 1261–1276. [Google Scholar] [CrossRef] [Green Version]
  23. Zou, Q.; Zhou, J.; Zhou, C.; Song, L.; Guo, J. Comprehensive flood risk assessment based on set pair analysis-variable fuzzy sets model and fuzzy AHP. Stoch. Environ. Res. Risk Assess. 2013, 27, 525–546. [Google Scholar] [CrossRef]
  24. Ganji, K.; Gharechelou, S.; Ahmadi, A.; Johnson, B.A. Riverine flood vulnerability assessment and zoning using geospatial data and MCDA method in Aq’Qala. Int. J. Disaster Risk Reduct. 2022, 82, 103345. [Google Scholar] [CrossRef]
  25. Mishra, K.; Sinha, R. Flood risk assessment in the Kosi megafan using multi-criteria decision analysis: A hydro-geomorphic approach. Geomorphology 2020, 350, 106861. [Google Scholar] [CrossRef]
  26. Renard, F. Flood risk management centred on clusters of territorial vulnerability. Geomat. Nat. Hazards Risk 2016, 8, 525–543. [Google Scholar] [CrossRef] [Green Version]
  27. Papaioannou, G.; Vasiliades, L.; Loukas, A. Multi-Criteria Analysis Framework for Potential Flood Prone Areas Mapping. Water Resour. Manag. 2015, 29, 399–418. [Google Scholar] [CrossRef]
  28. European Parliament and the Council of the European Union. Directive 2007/60/EC on the Assessment and Management of Flood Risks; Official Journal of the European Union: Brussels, Belgium, 2007. [Google Scholar]
  29. Fernandez, P.; Mourato, S.; Moreira, M. Social vulnerability assessment of flood risk using GIS-based multicriteria decision analysis. A case study of Vila Nova de Gaia (Portugal). Geomat. Nat. Hazards Risk 2015, 7, 1367–1389. [Google Scholar] [CrossRef] [Green Version]
  30. Dong, Y.; Zhang, G.; Hong, W.-C.; Xu, Y. Consensus models for AHP group decision making under row geometric mean prioritization method. Decis. Support Syst. 2010, 49, 281–289. [Google Scholar] [CrossRef]
  31. Aguarón, J.; Escobar, M.T.; Moreno-Jiménez, J.M.; Turón, A. AHP-Group Decision Making Based on Consistency. Mathematics 2019, 7, 242. [Google Scholar] [CrossRef] [Green Version]
  32. Srdevic, Z.; Blagojevic, B.; Srdjevic, B. AHP based group decision making in ranking loan applicants for purchasing irrigation equipment: A case study. Bulg. J. Agric. Sci. 2011, 17, 531–543. [Google Scholar]
  33. DGT. Carta de Uso e Ocupação do Solo. Registo Nacional de Dados Geográficos. SNIG; Direção-Geral do Território: Lisboa, Portugal, 2018. Available online: https://geo2.dgterritorio.gov.pt/geoserver/COS2018/wms?service=wms&version=1.3.0&request=GetCapabilities (accessed on 5 September 2022).
  34. Cutter, S.L.; Boruff, B.J.; Shirley, W.L. Social Vulnerability to Environmental Hazards. Soc. Sci. Q. 2003, 84, 242–261. [Google Scholar] [CrossRef]
  35. Fekete, A. Validation of a social vulnerability index in context to river-floods in Germany. Nat. Hazards Earth Syst. Sci. 2009, 9, 393–403. [Google Scholar] [CrossRef] [Green Version]
  36. Kuhlicke, C.; Scolobig, A.; Tapsell, S.; Steinführer, A.; Marchi, B. Contextualizing social vulnerability: Findings from case studies across Europe. Nat. Hazards 2011, 58, 789–810. [Google Scholar] [CrossRef]
  37. Martins, V.N.; Silva, D.S.; Cabral, P. Social vulnerability assessment to seismic risk using multicriteria analysis: The case study of Vila Franca do Campo (São Miguel Island, Azores, Portugal). Nat. Hazards 2012, 62, 385–404. [Google Scholar] [CrossRef]
  38. Hewitt, K. Regions of Risk: A Geographical Introduction to Disasters; Themes in Resource Management Addison Wesley Longman: Essex, UK, 1997; p. 416. [Google Scholar]
  39. Simonovic, S.P.; McBean, G.; Prodanovic, P.; Burn, D.H.; Wey, K.; Kay, P.; Emerson, A.; Mortsch, L.; Hebb, A.; Goldt, R.; et al. Floods: Mapping Vulnerability in the Upper Thames Watershed under a Changing Climate; University of Waterloo, Environment Canada and Upper Thames River Conservation Authority: Waterloo, ON, Canada, 2007. [Google Scholar]
  40. Schneiderbauer, S. Risk and Vulnerability to Natural Disasters—From Broad View to Focused Perspective; University Berlin: Berlin, Germany, 2007. [Google Scholar]
  41. Tapsell, S.M.; Penning-Rowsell, E.C.; Tunstall, S.M.; Wilson, T.L. Vulnerability to flooding: Health and social dimensions. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 2002, 360, 1511–1525. [Google Scholar] [CrossRef]
  42. Balica, S.F.; Wright, N.G.; Meulen, F. A flood vulnerability index for coastal cities and its use in assessing climate change impacts. Nat. Hazards 2012, 64, 73–105. [Google Scholar] [CrossRef] [Green Version]
  43. Morris, J.; Brewin, P. The impact of seasonal flooding on agriculture: The spring 2012 floods in Somerset, England. J. Flood Risk Manag. 2013, 7, 128–140. [Google Scholar] [CrossRef]
  44. Whitlow, T.H.; Harris, R.W. Flood Tolerance in Plants: A State-of-the-Art Review; Tech. Report E-92; U.S. Army Engineer Waterways: Vicksburg, MS, USA, 1979.
  45. Cardona, O.D. Indicators of Disaster Risk and Risk Management: Summary Report; Inter-American Development Bank: Washington, DC, USA, 2005. [Google Scholar]
  46. Masozera, M.; Bailey, M.; Kerchner, C. Distribution of impacts of natural disasters across income groups: A case study of New Orleans. Ecol. Econ. 2007, 63, 299–306. [Google Scholar] [CrossRef]
  47. Krueger, T.; Page, T.; Hubacek, K.; Smith, L.; Hiscock, K. The role of expert opinion in environmental modelling. Environ. Model. Softw. 2012, 36, 4–18. [Google Scholar] [CrossRef]
  48. Wedley, W.C. Consistency prediction for incomplete AHP matrices. Math. Comput. Model. 1993, 17, 151–161. [Google Scholar] [CrossRef]
  49. Ho, D.; Newell, G.; Walker, A. The importance of property-specific attributes in assessing CBD office building quality. J. Prop. Invest. Financ. 2005, 23, 424–444. [Google Scholar] [CrossRef]
  50. Rygel, L.; O’sullivan, D.; Yarnal, B. A Method for Constructing a Social Vulnerability Index: An Application to Hurricane Storm Surges in a Developed Country. Mitig. Adapt. Strateg. Glob. Chang. 2006, 11, 741–764. [Google Scholar] [CrossRef]
  51. Flanagan, B.E.; Gregory, E.W.; Hallisey, E.J.; Heitgerd, J.L.; Lewis, B. A Social Vulnerability Index for Disaster Management. J. Homel. Secur. Emerg. Manag. 2011, 8. [Google Scholar] [CrossRef]
Figure 1. (a) Flood-prone area’s location; (b) Portuguese Hydrographic Regions and flood-prone areas code.
Figure 1. (a) Flood-prone area’s location; (b) Portuguese Hydrographic Regions and flood-prone areas code.
Applsci 13 04915 g001
Figure 2. Methodology flowchart.
Figure 2. Methodology flowchart.
Applsci 13 04915 g002
Figure 3. The hierarchical structure of the flood vulnerability criteria used in AHP.
Figure 3. The hierarchical structure of the flood vulnerability criteria used in AHP.
Applsci 13 04915 g003
Figure 4. Example of a Google forms second level question (translated from the original questionnaire in Portuguese).
Figure 4. Example of a Google forms second level question (translated from the original questionnaire in Portuguese).
Applsci 13 04915 g004
Figure 5. Dispersion of individual expert’s weights for each criterion. The red line indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to the most extreme data points not considered outliers, and the outliers are plotted individually using the plus red symbol. A black dot marks the final group weight for each criterion.
Figure 5. Dispersion of individual expert’s weights for each criterion. The red line indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to the most extreme data points not considered outliers, and the outliers are plotted individually using the plus red symbol. A black dot marks the final group weight for each criterion.
Applsci 13 04915 g005
Figure 6. Final criteria and sub-criteria AHP-GDM weights.
Figure 6. Final criteria and sub-criteria AHP-GDM weights.
Applsci 13 04915 g006
Figure 7. Vulnerability classes distribution in the flood-prone sites: (a) Flood vulnerability; (b) Population criterion; (c) Socioeconomics criterion; (d) Buildings criterion; (e) Exposed Elements criterion.
Figure 7. Vulnerability classes distribution in the flood-prone sites: (a) Flood vulnerability; (b) Population criterion; (c) Socioeconomics criterion; (d) Buildings criterion; (e) Exposed Elements criterion.
Applsci 13 04915 g007
Figure 8. Flood vulnerability in five of the flood-prone areas: (a) Régua (Douro River); (b) Porto/Vila Nova Gaia (Douro River); (c) Coimbra (Mondego River); (d) Águeda (Águeda River); (e) Abrantes/Santarém/Vila Franca Xira (Tagus River).
Figure 8. Flood vulnerability in five of the flood-prone areas: (a) Régua (Douro River); (b) Porto/Vila Nova Gaia (Douro River); (c) Coimbra (Mondego River); (d) Águeda (Águeda River); (e) Abrantes/Santarém/Vila Franca Xira (Tagus River).
Applsci 13 04915 g008
Figure 9. Detailed criteria map for the flood-prone area of Coimbra (Mondego River): (a) Population criterion of flood vulnerability; (b) Socio-economic criterion of flood vulnerability; (c) Buildings criterion of flood vulnerability; (d) Exposed elements criterion of flood vulnerability.
Figure 9. Detailed criteria map for the flood-prone area of Coimbra (Mondego River): (a) Population criterion of flood vulnerability; (b) Socio-economic criterion of flood vulnerability; (c) Buildings criterion of flood vulnerability; (d) Exposed elements criterion of flood vulnerability.
Applsci 13 04915 g009
Table 1. Description of the data used in the AHP-GDM.
Table 1. Description of the data used in the AHP-GDM.
PPopulationBBuildings
P1Age structureB1Building year
P11Percent of residents under 14 years oldB11Percent of constructions built before 1980
P12Percent of residents over 65 years oldB12Percent of constructions built after 1980
P13Percent of residents between 14 and 65 years oldB2Floors
P2GenderB21Percent of buildings with one or two floors
P21Percent of male residentsB22Percent of buildings with three or more floors
P22Percent of female residentsB3Function
P3Family NumberB31Percent of buildings that are exclusively or mainly houses
P31Percent of households over 5 individualsB32Percent of buildings that have a non-residential function
P32Percent of households under 4 individualsB4Percent of buildings that are a collective accommodation
SSocio EconomyEExposed elements
S1Level of educationE1Land use
S11Percent of individuals who completed junior high schoolE11Percent of urban area
S12Percent of individuals who completed high schoolE12Percent of agricultural area
S13Percent of individuals who completed universityE13Percent of forestry area
S2Housing occupancyE2Population density (Inhab/km2)
S21Percent of homes ownedE3Building density (buildings/km2)
S22Percent of homes rented
S3Percent of unemployment rate
S4Percent of illiteracy rate
Table 2. Saaty’s scale for weight assignment adapted from [15].
Table 2. Saaty’s scale for weight assignment adapted from [15].
Less Important More Important
ExtremelyVery
Strongly
StronglyModeratelyEquallyModeratelyStronglyVery
Strongly
Extremely
1/91/71/51/313579
Table 3. Descriptive statistics of the pair-wise matrix consistency criteria CR and ED.
Table 3. Descriptive statistics of the pair-wise matrix consistency criteria CR and ED.
StatisticsCRED
Minimum0.000.00
Maximum0.7320.79
Median0.082.63
Quartile 750.164.44
Percentile 900.287.69
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Mourato, S.; Fernandez, P.; Pereira, L.G.; Moreira, M. Assessing Vulnerability in Flood Prone Areas Using Analytic Hierarchy Process—Group Decision Making and Geographic Information System: A Case Study in Portugal. Appl. Sci. 2023, 13, 4915. https://0-doi-org.brum.beds.ac.uk/10.3390/app13084915

AMA Style

Mourato S, Fernandez P, Pereira LG, Moreira M. Assessing Vulnerability in Flood Prone Areas Using Analytic Hierarchy Process—Group Decision Making and Geographic Information System: A Case Study in Portugal. Applied Sciences. 2023; 13(8):4915. https://0-doi-org.brum.beds.ac.uk/10.3390/app13084915

Chicago/Turabian Style

Mourato, Sandra, Paulo Fernandez, Luísa Gomes Pereira, and Madalena Moreira. 2023. "Assessing Vulnerability in Flood Prone Areas Using Analytic Hierarchy Process—Group Decision Making and Geographic Information System: A Case Study in Portugal" Applied Sciences 13, no. 8: 4915. https://0-doi-org.brum.beds.ac.uk/10.3390/app13084915

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop