The Contribution of Higher Education to Sustainability: The Development and Assessment of Sustainability Competences in a University Case Study

Abstract

Universities can make a significant contribution to sustainability, and the development of sustainability competences in their graduates should be a key outcome of their courses. We propose an assessment framework for enabling and evaluating the attainment of sustainability competences in University students. We outline its six steps, offering tools on how to assess the alignment of University programs’ Learning Outcomes (LOs) to sustainability and how translate them into competences for sustainability. We provide approaches to evaluate existing assessment methods in terms of enabling students to develop and apply their competences, guidance on how to conduct the assessments to collect data on student performance and eventually how to use the data, and evidence collected to evaluate if the students are developing the intended competences. We illustrate the application of the assessment tool in a University case study and we draw conclusions on the evidence it offers to how higher education practitioners can benefit from its use.

1. Introduction

Education can play a crucial role in the realization of Sustainable Development Goals (SDGs) and the transformational transition to sustainability. This is reflected in SDG 4 and mainly in target 4.7, which explicitly suggests that “by 2030 ensure all learners acquire knowledge and skills needed to promote sustainable development, including among others through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship, and appreciation of cultural diversity and of culture’s contribution to sustainable development” [1]. Educational institutions have a role to play in the transformation required for sustainability to emerge by equipping their graduates with the competences required to become citizens of a sustainable future [2]. The role of higher education institutions is particularly crucial, as they prepare the future professionals with the knowledge and skills, i.e., the competences needed to address pressing challenges such as climate change, violent conflict, and the health emergencies that society is facing today [3,4]. Thus, the integration of sustainability principles in education curricula is an important endeavor for higher education [5].

This endeavor requires a Whole Systems Approach, starting by creating a shared vision of sustainability for all HE stakeholders. It continues by establishing the curricula, pedagogies, educator training programs, and learning environments that will enable learners to develop the competences for such a vision to realize, while making interconnections between all aspects of the organizational and operational structure of the institution [6]. An example of a systemic vision for sustainability in HE based on the SDGs would include establishing the following sustainability attributes [2,7]:

• achieving the Safe Operating Space by respecting planetary boundaries such as freshwater use, climate change and biogeochemical flows (SOS),
• achieving the Just Operating Place by establishing the required social foundation for all people to prosper (JOS),
• enabling Resilient Sustainable Behaviors that make learners critical thinkers and adept decision-makers in complexity and uncertainty (RSB),
• engaging with Alternative Economic Models (AEM) that respect planetary and human Health and Wellbeing (HW),
• practicing Transparent Governance (TG),
• enhancing inter and trans-disciplinary Collaboration (COL) and promoting Diversity and Inclusion (DI).

The ongoing discourse about sustainability and the realization of the SDGs makes research into sustainability competences all the more pertinent. Competences represent an integrated set of knowledge, skills, attitudes, and values that people bring into play in different contexts (society, education, work, and family) to address situations involving complex challenges [8,9,10]. Moreover, competences refer to both the performance ability to deliver a task and the willingness to engage in the task, and therefore have direct links with motivation, worldview, and values [11]. Over the past few years, sustainability education programs, reflecting the interdisciplinary and collaborative nature of the new science of sustainability, have made significant progress in conceptualizing key competencies for sustainable development [12,13]. Foresighted or anticipatory thinking, systems thinking, interdisciplinary work, and participation are examples of some of the competencies targeted by higher education sustainability programs [3,14,15].

The integration of competences in HE curricula, in turn, has implications for the educational process as the curriculum content, pedagogy, and assessment should ensure the defined educational outcomes are met. A curriculum for sustainability should provide space for learners to explore, analyze, and engage with the world around them holistically, and to develop the competences that will enable them tackle its complexities and realize the vision of the 2030 agenda [16]. Research relating sustainability competences to appropriate pedagogies for their development concludes that among the most effective approaches are problem- and project-oriented learning [17,18,19], as they offer opportunities for active, collaborative, and action-oriented learning and foster research skills [20].

Authentic assessment is framed in the form of learning experiences that progress from simple to complex and ultimately prepare students to apply their competences to real-world situations with teachers acting as facilitators [21]. Taking into account the aspirational component of sustainability competences, relating them to the willingness to act and showcase application of the capability embedded in knowledge and skills [22], assessments need to be designed in ways to enable students to demonstrate the intended competences.

The consistency between competences, defined learning outcomes, and ways to teach and assess them are significant indicators that the curriculum engages students in authentic learning about sustainability [23,24]. While progress has been clearly made in incorporating sustainability in university educational offerings, there is little available research on the extent to which HE institutions are effective in equipping students with sustainability competences [25]. In fact, there is a clear need for the development and application of evaluation tools that can support universities to monitor and manage their contribution to sustainability. The aim of this study is to present the development and application of an assessment framework for developing and evaluating sustainability competence in university students. The framework enables translation of existing programs’ learning outcomes into competences for sustainability, considers and evaluates the capacity of existing assessments of learning to enable students to apply their competences, and provides evidence on the effectiveness of HE programs.

2. Materials and Methods

2.1. Assessment Framework Methodology

The assessment framework evaluates a program’s alignment of learning outcomes (LOs) to sustainability, the capacity of existing and new assessments to measure them, and provides evidence of competence development in learners through a collection of tools and approaches in a six step methodology. It requires a sustainability vision to be developed by the program of study, clear learning outcomes that describe the competences the students will have developed after its completion and assessments, and pedagogies designed to allow students to mobilize and apply these competences to respond to a number of challenges in a way that students develop the associated intended competences. The interconnected nature of competence selection, effectiveness of pedagogies, curriculum content, and assessment for the development of competence in learners forms the basis of the framework that enables HE institutions to evaluate the development of their learners’ sustainability competences (Figure 1).

The assessment framework comprises six steps (Figure 2). To begin the assessment (step 1), teaching staff and program coordinators are encouraged to assess the alignment of their program’s LOs to their sustainability vision. This can be done by comparing their learning outcomes to their stated vision and identifying gaps to address or comparing them, in absence of a stated vision, to the sustainability attributes of the SDGs guided vision mentioned in the introduction (SOS, JOS, AEM, RSB, COL, TG, DI, and HW). The tool for this, which is based on a set of word codes for each sustainability attribute that the LOs are compared against, provides insight on the areas of sustainability that are included or excluded in the LOs and helps university program developers to understand what their educational offer supports in terms of the SDGs and what gaps they should address in terms of holistic representation of sustainability [7].

The next step includes the translation of the aligned LOs into a set of competences (step 2). Consultation with the academic staff, students participating in the program, and other relevant stakeholders, such as alumni and professional accreditation bodies, can further facilitate this step. The process of translation examines the following curricular concepts to ensure constructive alignment between LOs, content, pedagogy, and assessment: (1) ‘what is the student expected to know and do for sustainability?’; (2) ‘what is to be taught and learnt for sustainability?’; (3) ‘how is it to be taught and learnt?’; and (4) ‘how is it to be assessed?’ [26].

Then (step 3), competences are defined using clear statements on what the students need to master and their cognitive, affective, behavioral and metacognitive dimensions are described [27]. Performance indicators for each dimension state what the learner is expected to know and be able to do and should be appropriate for the level of study they target. The indicators of competence performance reflect learners’ cognitive abilities such as knowledge [28], understanding [29], and applied skills [30,31]. In addition, indicators include socio-emotional skills, which are attitudinal and behavioral [23,30,32], and metacognitive abilities related with the evaluation of intentions and actions [33,34].

The assessment framework then (step 4) requires the evaluation of the course’s own assessment methods to establish how well these methods assess students’ competence development and, if necessary, the development of new ones. These assessments can be formative, used during the course to motivate learning and/or summative at course completion to evaluate learning. To facilitate this evaluation, a typology of indicative assessment methods is provided for the assessment of sustainability competences sourced from the literature in

Table 1. Sustainability competences and some appropriate tools for their assessment based on literature review [2,3,8,15,46].

• Competence: Systems Thinking

Assessment tools: Concept maps (conceptual diagrams that represent the relationships between concepts) [47], computer simulations of complex systems and qualitative modelling of systems (elements, interactions and impact analysis) [48,49], self-assessment surveys [50] and problem scenarios where students are asked to bridge the gap between the current state and a goal or desired state [51].

• Competence: Future thinking

Assessment tools: Scenario construction (defining goals, objectives, processes, exploring what will happen, can happen or should happen), visioning exercises (exploring various desirable futures), foresight (identification of emerging trends and uncertainties), back-casting (exploring the feasibility of scenarios and visions) [8,52,53,54].

• Competence: Collaboration

Assessment tools: Collaborative problem-solving activities, such as projects and case studies (working together to form aim, objectives, goals and outcomes for a specific problem or case), transdisciplinary work (working with academic and community stakeholders to define and address a problem) [8,37]; collaborative computer assessments and games [55,56], focus groups and interviews [57], self and peer assessments [58,59].

• Competence: Strategic thinking

Assessment tools: Case study analysis, stakeholder analysis (who has power and interest over a plan), SWOT analysis (strengths, weaknesses, opportunities and threats of an action), devising strategies (identify short and long term goals and objectives and map actions), force field analysis (explore drivers and barriers to change and plan action accordingly) [60,61,62].

• Competence: Normative thinking

Assessment tools: Argument mapping (diagrammatic analysis of arguments, reasoning and evidence), six hats thinking (seeing a problem through different perspectives), debates (supporting opposing views on a statement), normative scenarios (how things should be) [53,63,64,65].

• Competence: Effective communication (oral and written)

Assessment tools: Oral presentation [66], written reports [67], essays, portfolios [68], lab or course diaries, role play [69]

• Competence: Modelling sustainable behavior

Assessment tools: Student Conference (students organise, submit abstracts, papers, peer-review, hold roundtable discussions and present) [70], reasoning exercises, observations of students performance and completion of assessment rubrics [59], SuliTest (survey that measures sustainability knowledge and skills) [71], Sustainability values test [72].

• Competence: Critical thinking

Assessment tools: Argument mapping, debates, critical essay analysis [73,74], critical writing [63,65]; critical thinking questionnaires, reflective writing [75].

• Competence: Self-awareness and self-regulation

Assessment tools: Self-assessments and focus groups [59], computer based self-evaluations [76], the 4Cs framework (conviction, convincing, compelling, conforming) [77], reflective writing [75], rubrics.

• Competence: Emotional intelligence and management

Assessment tools: Six hats thinking, Emotional intelligence appraisal (a performance-based assessment) [78], MSCEIT test (picture-based test) [79].

• Competence: Ability to use media (media literacy is about the use, critical understanding and communication of information through media)

Assessment tools: presentations, videos [80], podcasts [81], blogs, social media posts [82], self-assessment and independent assessment rubrics serve to assess the use of media, evaluating message, content, target audience, motives in a media piece as essay [83], interaction/collaboration/communication analytics through digital platforms [84].

• Competence: Decision making

Assessment tools: Complex problem scenarios where the student is physically or through computer simulations asked to solve [85], socio-ecological dilemmas [86], serious games [87] and social simulations [88].

• Competence: Sustainability literacy

Assessment tools: Tests, exams and essays/reports (for checking knowledge and understanding, written communication, analysis and synthesis) [59]; New Environmental Paradigm scale (survey that measures pro-ecological worldview) [89], SuliTest [71], multiple choice test (systems, action and effectiveness knowledge) [90].

; instead, this reflects current understanding of important competences for sustainability and is intended to give an overview of the domain to HE practitioners but is not an exhaustive list neither of competences nor of assessment tools. These can assist program coordinators in developing new assessment methods when existing ones are found to be inadequate, as well as incentivizing curricula that encourage competence development while giving students the opportunity to reflect on them [22].

Assessments should offer students opportunities to develop agency by engaging in authentic learning [35,36]. Research considers active learning tasks, such as case studies, complex, real-world project and problem-based tasks related to sustainability (comprising environmental, social and economic challenges) to contribute to sustainability competence development [24,37]. This is because students are enabled to act upon their knowledge, skills, and attitudes, experience them, and be in the position not only to understand what they entail but to use them as well [21,23,28]. Furthermore, these tasks should assist not only the development of students’ content knowledge but also the further transformation of their abilities through cognitive dissonance [38], collaboration [39], and active contribution [21,40].

The students faced with complex tasks will utilize their previous knowledge and skills and act from certain worldview, value-orientation, and perspective as starting point [31,41]. As they uncover more information on the task, link different concepts, engage in discussions with others, test their own ideas and compare them with others, they stretch their zone of proximal development [42]. This mobilizes them to develop a plan of action to control their learning and apply it in the real world, thus mastering it as a consequence [43,44,45]. When assessments have been designed to require learners to demonstrate how they have developed the program’s competences and how they can apply them to respond to a number of challenges, the formal assessment process can begin.

The formal assessment process (step 5) aims to both evaluate student performance or progress and provide evidence on the efficacy of the learning and teaching process. The formal student assessment starts with the application of the assessment methods found appropriate, or were developed from scratch in the previous step by the program leaders and the collection of their data. Ideally, assessments in conjunction with generating data of student performance should help students monitor their own progress and reflect on their learning experience while also providing the opportunity for staff to reflect on the students’ level of achievement and revise teaching methods accordingly. The process can be facilitated by the development and application of rubrics that consist of fixed scales with points corresponding to the performance indicators descriptions of the competences assessed, making it easier to measure and communicate students’ performance of these [91]. These indicators consist of levels, such as below basic, basic, intermediate, advanced, and expert and assist the educators to score the ability of the students to perform the task described [92]. The rubrics can be easily adapted for student reflection on their developed knowledge, skills, and attitudes. Apart from educator and student rubrics, assessment data can be collected from peer-assessment surveys, problem-solving tasks, observation of student work, interviews, and focus groups with the students, project-work, and other tools (Table 1). Data from these assessment tools in combination with the data from the formal assessment methods can be used in order to triangulate the assessment results (using two or more assessment measures to deduce the performance of a student or a team) for validity reasons [93].

Once the competence assessment is concluded, the contribution of a program to developing sustainability competences in its learners can be identified (step 6). This can be done by analyzing the data collected and making informed decisions about changes in the teaching and learning methods used, enabling students to be actively involved in teaching and assessment, and reflecting on lessons learnt and how things can be done differently in the future. The tool was applied in the context of a well-established MSc program in the environmental field at UK University, and findings are discussed in the context of additional data collected, as well as an evaluation of the tool through direct input from students via self-reflective questionnaires.

2.2. Assessment Framework Application in a University Case Study

The Master’s program in “Environmental Technology” at Imperial College London has a general orientation towards sustainability as is stated in its mission and practices (curriculum development, teaching and learning methodologies) that allows for this kind of experimentation [94,95].

The course aims to equip students with the necessary knowledge and skills to pursue a career in the environmental sector. It also aims to develop in learners the ability to solve sustainability problems through an interdisciplinary and systems approach, using critical enquiry, developing their ability to communicate and manage self and resources independently and as part of a group, and applying analytical, research, and ICT tools appropriately [94]. The course spans three terms, the core course term, the option term, and the independent research term. The pedagogical approaches used in teaching and learning include both teacher-centered techniques (lectures and demonstrations aided by audio-visual tools) and learner-centered strategies (discussions, collaborative projects, independent research essays, small group seminars, policy seminars, practical exercises, case studies, and computer-based activities). The students are mostly assessed on the knowledge they gain through exams and on the skills they develop through project work reports, collaborative and independent essay writing, and practical exercises results/reporting.

The assessment framework case study was implemented during the academic year, 2018–2019. For this study, the focus was on the option term as it touches on all the intended sustainability LOs and students of the Water Management, Pollution Management, and Environmental Analysis and Assessment options of the program were recruited. Those students were selected as they constitute three different and self-contained subgroups showing some variation in intended competences development, learning methods used, and assessments applied while constituting parts of the same course, but share important similarities in their knowledge and skills orientation as they are all part of the Natural Sciences option. The gender, student status, and ethnicity data for the 52 students recruited are summarized per group in

Table 2. Gender, student status and ethnicity data for the sample of students recruited from the Water Management, Pollution Management and Environmental Analysis and Assessment options for the purpose of this study.

Natural Sciences Option Data (N = 52)
Gender	Student Status	Ethnicity
Water Management Option (WM) (N = 19)
60% Women, 40% Men	20% Home/UK, 80% Overseas	65% Chinese, 15% White, 15% Asian, 5% Black
Pollution Management Option (PM) (N = 16)
81.25% Women, 18.75% Men	18.75% Home/UK, 12.5% EU, 68.75% Overseas	56.25% Chinese, 31.15% White, 6.25% Black Caribbean, 6.25% prefer not to say
Environmental Analysis and Assessment (EAA) (N = 17)
64.3% Women, 35.7% Men	7.1% Home/UK, 7.1% EU, 85.8% Overseas	78.6% Chinese, 14.3% Asian, 7.1% White

.

The application of the six steps of the assessment framework in the Master’s program in Environmental Technology is described below.

The Master’s program had already established learning outcomes in its curriculum (

Table 3. MSc Environmental Technology Learning Outcomes (Academic year 2018–2019) and translated competences.

MSc Program LOs 2018–2019	Translated Competences
1. Use a systems approach to understanding the present and past interactions between the processes and the perturbations of these systems by human influences	Systems thinking and dealing with complexity
2. Be skilled in interdisciplinary analysis, assessment and solution of sustainability problems anticipating future implications and managing uncertainty	Future thinking and dealing with uncertainty
3. Be able to communicate and collaborate with specialist experts across a range of disciplines and various stakeholders	Collaboration
4. Develop knowledge and understanding of environmental science, technology and policy concepts and principles	Environmental Science, Technology and Policy literacy (knowledge and understanding)
5. Integrate and evaluate information from various sources, apply professional judgement, reflection and balance costs and benefits	Critical thinking, reasoning and reflection
6. Be able to apply natural and social science research methodologies, techniques and tools for experimentation, data collection and analysis	Research competence
7. Plan, conduct and write-up a program of original research
8. Analyse and assess the natural/ social science literature effectively
9. Communicate research, strategies and policy implications effectively through presentations and professional reports use of Information and Communications Technology	Effective communication
10. Formulate strategy to address sustainability problem (prevention, mitigation, remediation)	Strategic thinking
11. Assess different options and weigh trade-offs to reach decision	Decision making
12. Learn independently with open-mindedness and critical enquiry	Self-regulation, self-awareness and management skills
13. Learn from the shared experiences with others
14. Develop self-confidence, efficiency and resilience

) but did not include a sustainability vision, and thus the LOs were evaluated on their alignment to the sustainability attributes using the respective tool as per step 1 of the assessment framework methodology. The LOs alignment to sustainability attributes was high in terms of the SOS, AEM, RSB, COL, and TG [7]. However, there were areas of low coverage for JOS, HW, and DI, which are the social and institutional aspects of sustainability. These were brought to the attention of the academic staff and program director for consideration in the upcoming curriculum review.

Next, the translation of LOs into competences (step 2) was performed in consultation with the program director, teaching staff (Lecturers and Teaching Fellows), program developers (Senior Strategic Teaching Fellow), students, and alumni. The competences that the program aims to develop and assess in the students of the options recruited can be found in Table 3.

To evaluate competence performance in those students, competence statements and indicators covering the cognitive, affective, and behavioral aspects of competence were developed by applying the procedure outlined in the methodology section (step 3) and can be found in Supplementary Material. It must be noted that Collaboration and Effective Communication competences were grouped together as it made sense to evaluate them at the same time because of the specific learning activities that students participated in that required effective communication to achieve working as a group and/or collaborate as a group to achieve effective written communication through group reports. Performance levels for each indicator of competence were structured in consultation with staff, with five levels of performance ranging from below basic, basic, and intermediate, to advanced and expert. The accepted level of performance is basic, which corresponds to 50–59% performance score, deemed as satisfactory for the MSc student to pass. The below basic level corresponds to poor/limited ability to perform the task considering the MSc level (<50%). Intermediate ability shows good (above satisfactory) performance in the task with occasional shortcomings (60–69%), advanced level shows very good performance showing novel insights into the problem (70–79%) and expert performance shows high degree creativity and innovative thinking (≥80%). The target threshold for the performance of the students of this program in the selected competences was set at Intermediate (60–69%).

These performance levels were further divided into qualitative categories. The 50–54% refers to “satisfactory, with a reasonable grasp of the relevant concepts and facts, but little evidence of the ability to synthesize and or evaluate, or with significant lapses”. The 55–59% refers to “satisfactory, with a good grasp of the relevant concepts and facts, but little evidence of the ability to both synthesize and evaluate, or with marked lapses”. The 60–64% refers to “a good grasp of the subject and some evidence of ability to synthesize and criticize”. and the 65–69% to “A very good grasp of the subject and evidence of ability to synthesize and criticize including use of supplementary reading, but falling short of excellence in one or more of these aspects”. The qualitative description for 70–79% refers to “Showing a thorough grasp of the subject, and ability to synthesize and criticize, with critical use of supplementary reading, occasionally falling below a general level of excellence (i.e., original insights and innovative thinking)”. The 80–89% refers to “excellent throughout, demonstrating a detailed knowledge and systematic understanding of key aspects of the subject, with strong evidence of independent thinking and original insights to the subject”. Finally, the 90–100% refers to “outstanding–making an original contribution, by questioning or challenging prevailing paradigms, offering new insights that are informed by critical evaluation of current research/practice, clearly demonstrating innovative/creative thinking”.

As per step 4, the formal assessment methods used in the options where assessed. They included oral presentations (formative and summative) as well as written reports of student work (summative), either individual or group-based with some variation across the three groups. In addition, exams played an important role in assessing knowledge and understanding of scientific concepts, environmental management, and assessment practices. These assessment methods offered opportunities for data collection regarding the selected competences (Table 3) according to the typology presented in Table 1. The oral presentations and written reports aimed to assess effective communication. The project reports required the students to develop system models and thus use systems thinking; short and long-term strategies to address problems and thus strategic thinking; and cope with future uncertainties related to environmental, political, and financial changes, and thus future thinking. They also had to consider the values and needs of stakeholders, manage trade-offs and make decisions, thus use decision making and collect, analyze and synthesize appropriate data to support their decisions and conclusions, thereby using their research and critical thinking skills. However, collaboration, self-regulation, team monitoring, and leadership were only assessed through informal discussions between the educators and the students during meetings regarding project progression.

Considering the above, the formal assessments were used to capture competence development in students. To triangulate the results, they were supplemented by educator assessment rubrics and student self-assessment surveys (step 5) (Supplementary Material). In addition, this was performed to provide educators with the opportunity to assess and reflect on students’ individual and group work and to give voice to the students regarding their assessment, as traditionally, only the educators evaluated student learning for this course module. The assessment tools that were used as part of the assessment framework to assess the option competences in students can be found in

Table 4. Description of option modules, assessment methods used and competences assessed in the MSc Environmental Technology.

Option (Module)	Assessment Tools Used	Competences Assessed
Water Management	Exam and course work	Knowledge and understanding of water systems and water management
Anglian Water Project (AWP) (work in small groups *)	Group report and course work	Systems thinking and dealing with complexity Future thinking and dealing with uncertainty Critical thinking, reasoning and reflection Research competence Strategic thinking Collaboration and effective communication Decision-making Self-regulation, self-awareness and management skills
Hounslow Heath Project (HHP) (work in two big groups *)	Group presentation and Individual report
Environmental Assessment and Analysis	Exam and course work	Knowledge and understanding of resource depletion and contamination assessment and management
Hounslow Heath Project (HHP) (work in two big groups *)	Group presentation and Individual report	Systems thinking and dealing with complexity Future thinking and dealing with uncertainty Critical thinking, reasoning and reflection Research competence Strategic thinking Collaboration and effective communication Decision-making Self-regulation, self-awareness and management skills
Waste Management Project (WMP) (Phase 1: work in small groups *)	Phase 1: Group report and individual presentation
(Phase 2: work in two big groups *)	Phase 2: Individual report and group presentation
Pollution Management	Exam and course work	Knowledge and understanding of pollution problems and pollution assessment and management
Waste Management Project (WMP) (Phase 1: work in small groups *)	Phase 1: Group report and individual presentation (assessed all competences in column 3)	Systems thinking and dealing with complexity Future thinking and dealing with uncertainty Critical thinking, reasoning and reflection Research competence Strategic thinking Collaboration and effective communication Decision-making Self-regulation, self-awareness and management skills
(Phase 2: work in two big groups *)	Phase 2: Individual report and group presentation
Pollution management Case Studies	Group presentation and Individual report

* The students who participated in the AWP worked consistently in teams of 4 to 5 people throughout the duration of the project. The students who worked in the WMP started in groups of 3 people and half-way through the project merged into two big groups consisting of 15 students. Lastly, HHP students worked in two big groups throughout the project.

.

3. Results

Both the educator rubrics and the self-assessment survey (Supplementary Material) consisted of statements, corresponding to the competences assessed. The rubrics were given to the educators to assess student reports and the surveys were given to the students to complete at the end of their project work. Twelve (12) educator assessment rubrics and 81 student self-assessment questionnaires were collected in total. The self-assessment survey was administered to the AWP, WMP, and HHP students and, due to access limitations, was not given to the PM case studies students.

The results of the formal assessment for the three groups of students are summarized in

Table 5. Average marks for each option module per assessment method.

Option (Module)	Assessment	Competences Assessed	Average Mark
Water Management (N = 19)	Exam and Course Work (Total)	All the Below	69
Water systems and Water management	Exam	Knowledge and understanding	66
Anglian Water Project (AWP) coursework (work in small groups *)	Group report	Systems thinking/Complexity	74
		Future thinking/Uncertainty
		Critical thinking/Reasoning/Reflection
		Research competence
		Strategic thinking
		Decision-making
	Individual presentation	Collaboration & Effective communication	72
		Self-regulation/self-awareness/management
Total for AWP coursework			74
Hounslow Heath Project (HHP) coursework (work in two big groups *)	Individual report	Systems thinking/Complexity	72
		Future thinking/Uncertainty
		Critical thinking/Reasoning/Reflection
		Research competence
		Strategic thinking
		Decision-making
		Collaboration and Effective communication
Environmental Assessment and Analysis (N = 17)	Exam and course work	All the below	67
Resource depletion and contamination assessment and management	Exam	Knowledge and understanding	64
Hounslow Heath Project (HHP) coursework (work in two big groups )	Individual report	Systems thinking/Complexity	73
		Future thinking/Uncertainty
		Critical thinking/Reasoning/Reflection
		Research competence
		Strategic thinking
		Decision-making
		Collaboration and Effective communication
Waste Management Project (WMP) coursework	Phase 1: Group report	Systems thinking/Complexity	69
(Phase 1: work in small groups )		Future thinking/Uncertainty
		Critical thinking/Reasoning/Reflection
		Research competence
		Strategic thinking
		Decision-making
	Individual presentation	Collaboration & Effective communication	71
		Self-regulation/self-awareness/management
Waste Management Project (WMP) coursework	Phase 2: Individual report	Systems thinking/Complexity	65
(Phase 2: work in two big groups )		Future thinking/Uncertainty
		Critical thinking/Reasoning/reflection
		Research competence
		Strategic thinking
		Decision-making
		Collaboration and Effective communication
Total for WMP coursework			68
Pollution Management (N = 16)	Exam and course work	All the below	67
Pollution problems and pollution assessment and management	Exam	Knowledge and understanding	65
Waste Management Project (WMP) coursework (Phase 1: work in small groups )	Phase 1: Group report	Systems thinking/Complexity	69
		Future thinking/Uncertainty
		Critical thinking/Reasoning/Reflection
		Resear competence
		Strategic thinking
		Decision-making
	Individual presentation	Collaboration & Effective communication	72
		Self-regulation/self-awareness/management
Waste Management Project (WMP) coursework	Phase 2: Individual report	Systems thinking/Complexity	67
(Phase 2: work in two big groups )		Future thinking/Uncertainty
		Critical thinking/Reasoning/Reflection
		Research competence
		Strategic thinking
		Decision-making
		Collaboration and Effective communication
Total for WMP coursework			69
Pollution management Case Studies (PMCS) coursework	Group report	Systems thinking/Complexity	69
		Future thinking/Uncertainty
		Critical thinking/Reasoning/Reflection
		Research competence
		Strategic thinking
		Decision-making
	Individual presentation	Collaboration & Effective communication	69
		Self-regulation/self-awareness/management
Total for PMCS coursework			69

, showing the average marks for the students of the Water Management, Pollution Management, and Environmental Assessment and Analysis collected through the various formal assessment methods used for the needs of the program per option.

In terms of knowledge and understanding, the students of the WM option received 66 (B merit upper), the students of EAA options received 64 (B merit lower), and the students of the PM option received 65 (B merit upper).

In terms of the other competences evaluated, the students of the WM option received marks between 72 and 74, which represent the A distinction category. The students of the EAA option received marks between 65 (B merit upper) and 73 (A distinction) in terms of systems, future, critical and strategic thinking, decision-making, and research skills for both the individual and group coursework, and 71 (A distinction) for collaboration, effective communication, and self-regulation and in their small group work. The students of the PM option received marks between 67 and 69 (B merit upper) for the competences systems, future, critical and strategic thinking, decision-making and research skills, and for collaboration, effective communication, and self-regulation received marks between 69 (B merit upper) and 72 (A distinction). In all cases, group competences received higher average scores than individual ones.

The educator assessment rubrics results showed, for the students of the WM option, intermediate (60–69%) to advanced (70–79%) levels in the competences systems, future, strategic, decision making, critical thinking and research skills, and an intermediate level for collaboration, effective communication, and self-regulation. For the students of the EAA option, the results showed an intermediate level for all competences, apart from collaboration, effective communication, and self-regulation for which the results showed an intermediate to advanced level. For the students of the PM option, the results showed intermediate level for all competences. The rubrics mainly assisted the educators in assessing more easily and clearly the level of student competence, as the formal assessment criteria were only focused on assessing the coursework produced (e.g., reports and presentations), and giving rich and targeted feedback for each competence examined in the feedback report given to the students. The students, on the other hand, received feedback on their individual as well as group work. An example of the feedback given to a group of students of the WM option based on the formal assessment criteria and the educator rubrics is provided in Supplementary Material.

Students were given a self-assessment survey to reflect on their developed competences through teamwork. The same survey allowed educators to gain understanding about the performance of the students as a team and thus assess their collaboration competence. The self-assessment results per project are shown in

Table 6. (a) Descriptive Statistics of the self-assessment survey results of the WM students. (b) Descriptive Statistics of the self-assessment survey results of the EAA students. (c) Descriptive Statistics of the self-assessment survey results of the PM students.

(a)
Competence	N	Minimum	Maximum	Mean	Std. Deviation	Normalised Scores
Systems thinking and…	19	3.00	5.00	4.16	0.602	78.95
Future thinking and …	19	2.00	5.00	3.68	0.885	67.11
Decision making	19	2.00	5.00	3.84	0.602	71.05
Critical thinking	19	3.00	5.00	3.68	0.582	67.11
Collaboration	19	3.00	5.00	3.84	0.501	71.05
Research competence	19	1.00	5.00	3.63	1.012	65.79
Self-regulation, …	19	2.50	5.00	3.76	0.586	69.08
Strategic thinking	19	3.00	5.00	4.00	0.667	75.00
Valid N (listwise)	19				Average	70.64
(b)
Competence	N	Minimum	Maximum	Mean	Std. Deviation	Normalised Scores
Systems thinking and...	32	1.00	5.00	3.41	1.012	60.16
Future thinking and…	32	1.00	5.00	3.09	1.174	52.35
Decision making	32	1.00	5.00	3.78	0.870	69.53
Critical thinking, …	32	1.00	5.00	3.56	0.948	64.06
Collaboration and..	32	1.00	5.00	3.41	1.043	60.16
Research competence	32	2.00	5.00	3.14	0.961	53.52
Self-regulation, …	32	1.00	5.00	3.69	0.896	67.19
Strategic thinking	32	1.00	5.00	3.55	0.910	63.67
Valid N (listwise)	32				Average	61.33
(c)
Competence	N	Minimum	Maximum	Mean	Std. Deviation	Normalised Scores
Systems thinking	30	3.00	5.00	3.77	0.679	69.17
Future thinking	30	2.00	5.00	3.43	0.817	60.83
Decision making	30	2.00	5.00	3.50	0.777	62.50
Critical thinking	30	2.00	5.00	3.37	0.669	59.17
Collaboration	30	2.00	5.00	3.23	0.935	55.83
Research competence	30	2.00	5.00	3.43	0.898	60.83
Self-regulation, …	30	2.00	5.00	3.57	0.774	64.17
Strategic thinking	30	2.00	5.00	3.50	0.820	62.50
Valid N (listwise)	30				Average	61.88

a–c. Notably, WM students self-assessed their competences higher than PM and EAA students. Systems thinking for WM and PM and decision making for EAA students were the strongest competences, whereas research competence for WM, collaboration for PM, and future thinking for EAA were the weakest competences reported by the students.

Overall, the lowest scored competences future thinking and dealing with uncertainty and research competence for EAA option students and collaboration and critical thinking for PM students seemed to belong to the 50–59% “basic qualitative category” which is not aligned with the expected level of student competence at the Master’s level—which is Intermediate (60–69%)—and should be given more attention by the program coordinators in a curriculum review.

When comparing student self-assessment scores with formal assessment marks, the results show that in all cases students self-assessed much lower. However, there is compatibility between the formal assessment marks and student self-assessment scores for WM and EAA students in terms of level of performance. In the first case, both assessments show A Distinction (advanced competence), and in the latter case both assessments show B Merit (intermediate competence). On the other hand, the PM students gave lower scores to themselves than the educators did (B Merit/intermediate from the students and A Distinction/advanced from the educators respectively). A reason why the students may self-assess research skills differently could be that they have been exposed to different research experiences in their undergraduate studies [96]. In addition, students may have a limited ability to self-assess their skills if not adequately trained, and this may be reflected in the overall lower scores reported [97].

The data show that for most competences, the students self-assess in the intermediate to advanced level depending on the option they attended and also received marks from B Merit (intermediate competence) to A Distinction (advanced) competence according to the formal departmental assessments and educator rubrics. When compared to the threshold set by the department (students to achieve at least intermediate level, 60–70%), the results show satisfactory attainment (step 6), with the exception already mentioned in the results from student self-assessment surveys. The competences flagged by students as their weakest should be targeted by the curriculum developers and educators in the next curriculum review. Lastly, but importantly, there will be benefits for the assessment of competences in case the Master’s program management and teaching team consider inclusion and refinement of the educator rubrics and student self-assessment surveys developed for the program.

4. Discussion

The tool was designed for use by higher education practitioners to evaluate student competence development. The ultimate aim is to help HE institutions evaluate their contribution to empowering graduates with sustainability competences. Its application shows that it is crucial to not only have the LOs aligned to sustainability, but to generate evidence that the translated competences are actually being developed in learners as it will help curriculum planers to develop appropriate programs. Although we provide a tool to check the alignment of LOs to sustainability attributes in the absence of a sustainability vision by the HEI, such a vision needs to be developed by the HEIs engaging all stakeholders in a participatory way, as that will guide the definition of competences to achieve it [2]. A holistic vision for the HEI will drive change in other institutional aspects that are crucial for enabling the development of sustainability competences by all. Furthermore, the tool could help HE practitioners improve/modify their assessment methods to enable active experience and appropriate assessment of the defined sustainability competences, using guidance in Table 1. This will generate evidence on the effectiveness of their teaching and learning approaches to develop those competences in students. In conclusion, assessments should be generating data and insights that help educators and students to make evidence-based decisions in terms of their teaching and learning respectively, and to identify and remediate barriers in achieving their goals.

This study addresses an important need in the academic community in relation to the multiple perspectives on sustainability on the one hand, and the diversity of existing competence frameworks and assessment tools on the other. A focus on participatory approaches in formulating competences for sustainability that fit an institution’s vision, mission, aims, and needs, instead of a prescriptive approach of applying predefined competence frameworks, should be prioritized by all HE institutions. Moreover, this approach focuses on designing the selected sustainability competences into the learning and assessments activities. Thus, both leaners and educators benefit from the clarity/transparency of educational aims, effectiveness of pedagogies, and accountability/ownership of outcomes. In addition, the data generated from the assessment tools enable HE practitioners to identify gaps in terms of sustainability alignment and barriers that prevent students from developing sustainability competences.

Two recent systematic literature reviews on the assessment tools for sustainability competences [59,98] place emphasis on the fact that universities put a lot of effort in compiling pedagogies that will enable sustainability competences in learners, rather than thought on which assessments are appropriate for them. The tool, placing emphasis on competences translated from LOs based on the program’s mission and aim, offers a methodology for education practitioners to consider which tools to use for targeted competence assessment. Furthermore, the studies show that the most frequently used assessment tools are self-assessment questionnaires and surveys, followed by reflective writing (essays, reports, and diaries) and focus groups/interviews. The least used and maybe more refined tools are concept maps, coursework assignments, and rubrics. Therefore, a combination of the above tools is needed to capture competence development in students and application in appropriate teaching and learning activities.

The case study yields some useful suggestions to HE practitioners to assist them when applying the competence assessment tool. The program coordinators need to make sure the competences represent not only the main curricular, pedagogical, and assessment aspects collectively the educational ecosystem, but reflect an awareness of the diversity of perspectives, voices, and cultures comprising the staff and student bodies as well as relevant societal aspects, such as ones related to professional life and emergent social transitions. This can happen by setting and agreeing values to guide their participatory process of competence definition [99]. These values will clarify the sustainability vision pursued and thus frame the competences as enablers to achieve that vision, as otherwise traditional competences that promote unsustainability can be selected.

As far as the assessments methods are concerned, the education practitioners should focus on the ones that enable students actively experience the competences they are expected to develop and reflect on them. This can be done by using holistic assessments of the cognitive, affective and behavioral aspects of competence with its metacognitive aspects, which have been found effective in enabling intrinsic motivation and longer-term engagement [100]. It is also important to offer the students a variety of those assessment tools to both cater to the diversity of student learning styles and to capturing the complex aspects of the competence constructs [26]. This can be challenging for educators to achieve and thus consultations, meetings, and mentoring, as well as training sessions, would be beneficial for clarifying how to proceed with assessment.

One important decision-making point regarding the use of the tool is group work. Although students benefit from working in groups as they are challenged to develop their competences further, the size of a group can greatly influence the decision making processes performed by the group [101]. For example, the bigger the group, the less each member will be able to say. A larger group may inhibit certain individuals from contributing due to peer pressure. In addition, systems thinking is a competence that relates to systems analysis, and applying the necessary modelling and mapping of stakeholders and interactions may be more difficult in big groups as there may be many points of view, higher complexity and more conflict [101]. This means that when engaging the students in collaborative project work, there should be a careful selection of the size of the group and of the roles the students will play within the group. This may be correlated with their educational and professional background and personality.

While the tool presented here focuses on competences for evaluating a program’s contribution to sustainability, it should be noted that competence-based assessments have also been the subject of criticism. This is attributable to the complexity of assessing poorly understood concepts, resulting in the potential to narrow the curriculum because of the increased focus on what is assessed at the expense of non-tested skills, which receive decreased attention. This entails the danger of overlooking important aspects of the student’s personality as there may not be appropriate assessment methods to capture them and the caveat that using performance levels can negatively label teachers and students, thus influencing their attitudes [102]. Competence assessments alone cannot benefit educators and students if it not coupled with systemic interventions such as teacher training sessions, involving time and cost requirements for developing relevant assessment material that is sensitive to class or cohort size and norms and behaviors that create resistance to change.

Despite these reservations, competence-based assessments place the importance of assessment not only on the outcomes of learning but equally on the process and experiences that led to those outcomes [103]. Their approach further provides specific, targeted and actionable feedback to the educator and student [104]. Although competences are complex as constructs and difficult to assess, they reflect the multidimensional, integrated, and performance-based nature of assessment and in terms of their sustainability definition, they can act as indicators of closing the gap towards a sustainability vision. Lastly and importantly, competences are important for entering and progressing in a professional environment as they are sought after by employers and used in applicant screening tests [105].

Further considerations for the education practitioners viewing this case study include the fact that the assessment framework was applied in a master’s course that already had strong links with sustainability. For programs that have weaker links to sustainability as well as other types of courses, for example undergraduate university courses, doctoral training programs, or school education contexts, it should be applied in future case studies to serve the needs of diverse education stakeholders. Its effectiveness as a decision-making tool can be validated in specific case studies that aim to collect and analyze data and make judgements to guide curriculum reviews. The Whole Institution Approach advocated by recent policy developments [106] emphasizes that institutional and contextual aspects play an important role in the university’s contribution to sustainability. Thus, sustainability competence development should be seen holistically i.e., from an educational, institutional and contextual perspective [21] and become aligned with whole system effectiveness in promoting competences for sustainability.