Can Pedagogical Innovations Be Sustainable? One Evaluation Outlook for Research Developed in Portuguese Higher Education

Abstract

Across higher education, teachers have been inspiring themselves (and others) to apply pedagogical innovations grounded in a unifying intention: to prepare students for labour markets and societal challenges. Research-based education has been funded to promote pedagogical innovations with valuable impact on the students’ academic success and/or the teachers’ academic growth. However, there is still few documented research-based evidence that highlight how long-lasting and/or how embedded are the pedagogical innovations in academic practices, particularly when the funding period comes to an end. The purpose of this article is to discuss the extent to which funded national research-based education projects, developed in public Portuguese higher education institutions (universities and polytechnic institutes), have considered the sustainability of research results (e.g., pedagogical innovations), after funding ends. Based on a qualitative research approach, data collection included: document analysis of 39 projects’ materials and conceptions collected with 9 coordinators and 17 key participants from selected projects, through interviews and questionnaires, respectively. Content analysis of data collected showed that there are key factors that contribute to enhance and/or limit the sustainability of research results (e.g., funding and institutional support to maintain innovation). Results and recommendations are put forward to inform educational policies, funding agencies and involved actors (e.g., institution leaders, researchers, and teaching staff) to increase the sustainability of research results.

1. Introduction

Around the academic world, the development of pedagogical innovations is increasingly a focus of institutional leaders (e.g., Rectors) and teachers [1,2,3,4], as a response to specific problems (e.g., lack of students’ learning) [5]. A pedagogical innovation can be the introduction of a resource and/or strategy that, when implemented and evaluated by teachers, leads to student learning [6,7,8,9,10,11,12].

Teachers should be knowledgeable regarding innovative strategies and resources to promote a student-centred approach [13,14] and create a supportive intellectual and emotional environment that can encourage students to learn actively [15,16]. Some innovative strategies are collaborative learning, mentoring and tutoring, debates, role-playing, peer teaching, problem-solving [17] and formative feedback [18,19].

However, some studies [18,19] highlighted teachers’ difficulties in developing innovative strategies in academic practices, such as implementing peer-written feedback in large classes (lectures), or lacking needed resources (e.g., materials and equipment). Therefore, it is also important to reflect upon what works, what does not, and options to integrate a pedagogical innovation into academic practices [20].

One possible way to innovate pedagogically is linking two of the important academic dimensions—research and teaching—in teachers’ daily professional routines [21,22,23]. This challenge could be better achieved, as some authors [24,25] point out, through teachers’ participation in research-based education projects in a scenario that also includes financial support. According to [25], the existence of financial incentives and regulations are identified as factors that promote and sustain pedagogical innovations developed during funded re-search-based education projects. There are different ways of financially supporting re-search, including loans, equity investments, award schemes and grants [26], although a grant is the known best source of funding among researchers.

Nevertheless, according to different studies [27,28], obtaining a grant to develop a research-based project, particularly in higher education, requires a considerable amount of effort and time on the part of researchers and teachers and is, in most cases, difficult to obtain [29], particularly in research-based education [27].

We find the preceding observation particularly relevant when exploring the context of Portuguese higher education. Despite the European crisis that affected Portugal during the first decades of the 21st century [30], higher education “key actors” (e.g., teachers) had several funding opportunities, launched by different Portuguese governmental entities (e.g., the Ministry of Science and Higher Education) and national sponsoring research agencies (e.g., the Calouste Gulbenkian Foundation) to apply for grants to develop research-based education projects [31]. Some funded research-based education projects have focused on the development of pedagogical innovations in universities and polytechnic institutes, revealing research evidence regarding the promotion of the teachers’ academic development and/or on the students’ learning development [19,31,32,33].

However, when the funding period comes to an end, favourable conditions are necessary to continue to implement the research-based education project and ensure the impact and sustainability of results (e.g., pedagogical innovations). The call for evidence regarding the impact and sustainability of funded research comes from sponsoring agencies (e.g., European Commission) and has been studied by several authors in different areas such as [34,35,36]. According to Luukkonen [37], impact is related to the effect that the activity has and its results for people, practices, organisations and systems; sustainability is the capacity of the project results and products to persist and remain in use beyond the duration of a funding period. This is what is referred to as the sustainability of pedagogical innovations developed through funded research-based projects [28].

The study of sustainability of research is a significant implementation challenge, particularly in educational research. Some authors, such as [26,38,39,40], have reflected on the importance of designing methodologies to assess the impact of research grant funding. Additionally, there are already several studies about the sustainability of research (e.g., [41,42,43]), which demonstrates the need to better understand it, namely, the importance of pursuing research-based education projects focused on the development of pedagogical innovations in higher education (HE) [44]. Nonetheless, much of the literature on sustainability remains theoretical, with little practical guidance on how to sustain research-based education project delivery, implementation strategies, and research outcomes, particularly after the end of a funding period [28,44].

The purpose of this article is to present and discuss a study focused on the understanding of the extent to which research-based education projects, sponsored by Portuguese research agencies between 2004 and 2013 and implemented in Portuguese public higher education institutions (universities and polytechnic institutes), considered the sustainability of research throughout their proposals and development process, and, if so, how.

The following research aims were defined: (1) to characterise which “sustainability actions” were included in the design and development process in the national research-based education projects selected for the study; (2) to determine which “actions” could hinder or promote the sustainability of pedagogical innovations developed through funded projects; (3) to propose recommendations for the future at the political (for politicians and research sponsors), institutional (for leaders) and individual (for teachers) levels, such that the sustainability of the research, throughout their proposals and development process, is considered.

One considers that the answer to this question will enable the comprehension of the “sustainability actions” that could influence the design of research-based education in an HE context, and therefore give recommendations for the future at political (to politicians and sponsors), institutional (to leaders) and individual levels (to teachers).

2. Rationale

Regarding the creation process of an innovation, there is no single “true” path to develop an innovation in academic practice [45,46,47,48]. For instance, the Centre for Educational Research and Innovation proposed, some years ago, that an innovation should have two main subcomponents: what has been introduced is original to a particular individual or group, and the successful change that results from its adoption [47].

Independent of the delivery context, Serdyukov [20] explains that an innovation is understood as the successful introduction of a method, strategy or resource into someone’s practice. The author also explains that it requires three major steps: an idea, its implementation, and the outcome that results from the execution of the idea and produces a change [20]. In accordance with this, [48] explained that an innovation is an idea, practice or object perceived as new by an individual, and if the idea seems new to the individual, it is an innovation. It is, then, the perceived novelty of the idea introduced, for the individual, that determines their reaction to it.

However, some obstacles could emerge during the development of pedagogical innovations in HE, such as: the powerful influence of the teachers’ discomfort and/or apprehension regarding change from traditional practices to more student-centred practices [49]; an overload of time spent in preparing the new/innovative lessons [50]; the teachers’ difficulties innovating in the context of large classes [19]; and the lack of resources (e.g., curricular materials, equipment, human resources) [31]. Therefore, the development of pedagogical innovations could involve a twofold challenge: on the one hand, that students will not participate in activities and/or will not learn sufficient content; on the other hand, that teachers will feel a loss of control, that they lack necessary competencies, and/or that they will be criticised for teaching in unconventional ways.

Innovations, and innovative processes, are intended to increase productivity and learning efficiency, and/or to improve learning quality [24]. Fishman et al. [45] described the process of innovation as viable, by considering the combination of three elements: adaptation to institutional context; successful enactment; sustaining innovation. According to this, one can say that, in order to innovate pedagogically in HE, teachers should look beyond what they are currently doing in their academic practice and create something different, be it in quality, quantity or both [51,52,53,54].

The maintenance, adaptation, and/or innovation of an academic practice, emerging from research-based education projects, may involve ‘new forms’ of work/practices/strategies that should become normal practices for the project participants (e.g., teachers, students, and others). As Costa et al. [24] points out, teachers also need to guarantee the sustainability of the innovations produced. In this context, it is important to reflect on challenges related to the development and sustainability of pedagogical innovations developed through funded research-based education projects (e.g., after the end of financial support).

Studying the sustainability of research has been a persistent challenge for the scientific community, and across a range of settings [55,56,57,58,59,60,61,62,63]. The term ‘sustainability’ is still commonly associated with environmental situations addressing subjects that are, for instance, related to the preservation of the planet (e.g., the use of natural resources by human beings). For instances, in the Brundtland Report [57], sustainability can be seen as both an effort to preserve or improve the natural environment, while also providing the means to improve the conditions of the socially and economically disadvantaged people in the world. As De Castell et al. [58] points out, a relevant assumption within the term sustainability is the preservation or renewal of a set of resources on which social and economic development rests.

A literature review conducted by [59] categorised existing definitions of “sustainability of research” into three categories: maintaining the benefits of a project (e.g., the ability to continue the outcomes that emerge from research); project institutionalisation (e.g., ensuring that activities designed during research are formally incorporated into existing organisational practices); capacity building (e.g., to build on the capacity of the organisation’s community to continue implementing the research results).

Shediac-Rizkallah et al. [59] divided the sustainability of research into three categories: maintenance of the research benefits achieved through the project; continuation of the project within an organisation; and the ability to continue to develop a project. Later, [55] considered research sustainability to be related to a process of change with the purpose of strengthening and/or sustaining a given innovation in a specific context. The author also explains that sustainable innovation can be integrated into ongoing operations to benefit diverse stakeholders.

In this vein, Southwell et al. [60] identified three interrelated tendencies of the sustainability of research, namely: research outcomes becoming standard practice of individuals and/or the organisation (e.g., an innovation is sustained); local policies, procedures, and/or structures accommodate and/or provide incentives (e.g., financial support) to adopt the practices (e.g., an innovation is embedded); and normal practice is influenced beyond its initial site of introduction in a beneficial way (e.g., scaling up an innovation into other contexts).

Sustainability requires that ‘new forms of work’ and the ‘improvement of research results’ become normal practice for the participants and/or host institutions. According to Fixsen et al. [39] (p. 17), sustainability means that “after the intensity of establishing a fully implemented evidence-based implementation program in a new community (often requiring 2 to 4 years), the implementation site needs to be sustained in subsequent years”.

The existence of “program champions” in host institutions in which projects have been implemented could play a key role in the promotion of sustainable actions to promote and sustain results [59,61], even after the end of the funding period. [39] (p. 17) notes that “through it all, implementation site leaders and staff, together with the community, must be aware of the shifting ecology of influence factors and adjust without losing the functional components of the evidence-based program or the program dying due to a lack of essential financial and political support.”

Stirman et al. [62] highlighted a relationship between project design, organisational systems, connection with the community, and funding support. Based on a reviewing article identified through the four knowledge syntheses of sustainability, Moore et al. [56] (p. 7) developed a comprehensive definition of sustainability that included five constructs: “(1) after a defined period of time, (2) the program, clinical intervention, and/or implementation strategies continue to be delivered and/or (3) individual behaviour change (i.e., clinician, patient) is maintained; (4) the program and individual behaviour change may evolve or adapt, while (5) continuing to produce benefits for individuals/systems”.

Upon critically analysing different authors’ perspectives of the concept of research sustainability, and subsequent reflection, Costa et al. [24] (p. 117) explained that sustainability could be related to “the continuation of projects’ components (e.g., the innovative practice developed), capacity building (e.g., human and financial resources), and continued benefits or outcomes in the same and/or new settings (e.g., new educational scenarios and participants)”.

Notably, the concept of sustainability of research aids understanding how a particular instance of pedagogical innovation developed though a funded research-based education project could continue and/or evolve, over a certain period, after the end of the funding period. Thus, one can see the potential of these definitions to reflect on the sustainability of research-based education projects. However, much of the scientific literature remains theoretical, with little practical guidance on how to sustain research-based education project delivery, implementation strategies, and outcomes (e.g., pedagogical innovations in academic practice).

In this context, some authors [24,28,44,63] have borrowed the idea of sustainability to understand how pedagogical innovation is developed and sustained after the end of the funded period of research-based education projects. This option could be justified mainly given its focus on the relational aspects of the development process of pedagogical innovations (application, implementation, and the post-funding period).

In this context, Guerra [28] (p. 10) puts forth a definition of the sustainability of pedagogical innovations developed through funded, research-based education projects in HE: “The sustainable use of pedagogical innovations […] is considered to be achieved when after a defined period of time, they continue to be delivered in higher education, with or without adaptations, in the same academic scenery (e.g., course or curricular unit), with the aim of producing benefits for individuals (e.g., students’ academic success and/or teachers’ academic growth) and/or systems (e.g., institutions, courses, curricular units)”.

In research-based education projects, sustainability can mean, for example, the use of a particular infrastructure, continuation of services, mobilisation of participant competencies, development and/or continuation of partnerships and collaborations, and the integration of ideas and/or perspectives by the host institution (e.g., universities). The literature highlights different levels of effort by diverse key actors in the HE context, which may or may not guarantee the sustainability of research results.

Research-based education should be developed, disseminated, and exploited in such a way that: results can be tailored to the needs of others (e.g., teachers); transferred to new areas (e.g., curricular units); sustained after the funding period has finished; and/or used to influence future policy and practice. Therefore, the sustainability of funded research-based education projects require the recognition of different levels of key actors’ actions in HE spheres, namely: at the political level (macro level, e.g., the politicians that define the funding research agendas and/or regulations to empower innovation through funded research); at the institutional level (meso level, e.g., the institutional support of HE leaders to embed research output in collective dynamics); and at the individual level (micro level, e.g., personal and academic teachers’ motivations and the resilience to sustain, embed and/or upscale outcomes in practices).

3. Methods

A qualitative research paradigm [64,65,66,67,68], adapting principles of a case study approach [67], was adopted during two research phases: the first aimed to characterise the national funded research-based education projects developed in Portuguese public HE institutions (PPHEIs), from 2004 to 2013; the second aimed to determine the macro, meso and micro actions (e.g., institutional structures, personal dynamics) that hinder or promote the sustainability of effective pedagogical innovations (before, during, and after the end of the funding period) in science and engineering fields.

Table 1. Phases, data collection, and aims of the study.

	Data Collection	Aims
1st phase	Conducting a document analysis of 39 projects	To characterise the national funded research-based education projects developed in Portuguese public HE institutions (PPHEIs), from 2004 to 2013
2nd phase	Performing individual interviews with 9 coordinators of the projects	To determine the macro, meso and micro actions (e.g., institutional structures, personal dynamics) that hinder or promote the sustainability of effective pedagogical innovations (before, during and after the end of the funding period) in science and engineering fields
	Inquiring 17 key participants involved in projects through an online questionnaire

presents the data collection procedures, purposes, and aims applied in the two research phases of the study.

3.1. Document Analysis of National Funded Research-Based Education Projects (First Phase)

In the first phase, a qualitative document analysis [69] was adopted to characterise the national funded research-based education projects developed in Portuguese public HE institutions (PPHEIs) from 2004 to 2013. This technique enabled a rigorous, impartial, and systematic analysis of the contents of collected written documents. This process of document analysis occurred in the following steps:

1.

Collection of documents: the definition of the criteria for establishing the corpus of analysis—the funded research projects—considered: the research setting (studies should be implemented in Portuguese public higher education institutions (PPHEIs), i.e., universities and/or polytechnic institutes); the research object (studies should be focused on the development of pedagogical innovations in HE courses, e.g., science and engineering courses); the research period (national projects funded between 2004 and 2013); financial support (three main research sponsors in Portugal, namely, the Foundation for Science and Technology (FCT), the Ministry of Science, Technology and Higher Education (MSTHE), and the Calouste Gulbenkian Foundation (CGF)).

Regarding the identification of studies found on each research sponsor’s website, it was important to clarify that each national sponsor has a “funded projects database of all scientific domains”. All collected documents were in the public domain, although not all appropriate documents were available online. The team requested such documents from its contacts within the organisations.

Only 70 documents that were disclosed in the public domain were collected: 33 projects funded by FCT, specifically in the field of “Education and Educational Policies”; 21 projects funded by the MCTES (the Science and Innovation Operational Programme (POCI 2010), under the Axis IV-Science, higher education, As IV. 1-higher education Qualification, action IV. 1.2-innovative projects in higher education); 16 projects funded by MSTHE in the framework of “Innovative Educational Projects (2009/2015)”.

Table 2. Search queries applied in the three national research sponsors databases.

Queries	Sponsors	Studies
“higher education” OR “university” OR “polytechnics” AND year > 2004 < 2013	FCT	33
	MSTHE	21
	CGF	16
Total		70

presents a summary of search query techniques adopted on each research sponsor’s websites (FCT, CGF, and MSTHE), as well as the identified funded projects.

2.

Eligibility of documents: after the selection of 70 projects, 64 coordinators were contacted by email (November 2015), asking for their collaboration in this study and requesting their authorisation for data collection, such as reports of activities and scientific publications of the projects.

A total of 36 coordinators agreed to collaborate in this study and sent a total of 39 project documents (e.g., funding applications, scientific reports and/or books and articles). In this way, 39 studies were eligible for data collection in the first phase.

Researchers preserved anonymity by referring to the projects and project coordinators as follows: [Coordinator POSITION and Proj. NUMBER]. For example, [Proj. 1.A] refers to Coordinator A and Project 1.

Each project’s document was screened (by reading titles, research team, host institution, keywords, and abstracts) and analysed through a content analysis technique [69,70,71], using qualitative analysis software (WebQDA: https://www.webqda.net/?lang=en, accessed on 1 July 2021). Five queries (who, what, where, who, when) were defined to characterise the national funded research-based education projects developed in Portuguese public HE institutions (PPHEIs), from 2004 to 2013 (

Table 3. Inquiries applied to data collected in the 1st phase.

Document Analysis
Queries	Indicators of Analysis
WHO has afforded the development ofpedagogical innovations?	(political and research sponsors’ agendas)
WHAT has developed with this research?	(e.g., activities, the resources)
WHERE did the research take place?	(e.g., the host institution support)
WHO has conducted these projects?	(e.g., team elements)
WHEN did the project start and finish?	(e.g., the length of the financial support)

).

Each document was analysed to determine the extent to which the project it described addressed or considered each of the identified indicators for characterising the projects (Table 3). Text relevant to each indicator was highlighted and coded using qualitative data analysis software (WebQDA), based on the meaning, relevance and context for each indicator, as ‘yes’, ‘no’, or ‘unclear’. Scores of ‘yes’, ‘no’, or ‘unclear’ were assigned numerical values (1 to 3) to assist in aggregation and data presentation. The documents served as the individual source of information for the scoring, which facilitated research objectivity.

Each coordinator of the selected projects (n = 39) was involved in order to verify our coding of each project’s document. In addition, a third person (researcher in education) provided ad hoc verification and also served as an external mediator for any inconsistencies between the two primary coders. This ensured robust interpretative analysis and conclusions [64].

This first phase served as a baseline, reflecting on the theory and practice of funded research-based education projects in PPHEI, rather than an evaluation of a project. Results are presented in the following section.

3.2. Perceptions of Coordinators and Key Participants of Selected Projects (Second Phase)

In the second phase, the focus was to determine the macro, meso and micro actions (e.g., institutional structures and personal dynamics) that hinder or promote the sustainability of effective pedagogical innovations (before, during, and after the end of the funding period) in science and engineering fields.

In this way, conceptions of 9 coordinators of 12 funded projects were collected, through individual interviews (from January 2017 to February 2018). The group of coordinators was organised considering the following criteria: include coordinators with more than one funded project (e.g., different research sponsoring agencies); incorporate coordinators of projects that finished in different periods, specifically, 1–5 years after the end of the project (medium term); more than 5 years after the end of the project (long term); involve coordinators with projects implemented in the scientific areas of science (biology) and engineering; include coordinators with management responsibilities in the host institutions of the projects (e.g., vice-chancellors of universities).

Two open questions were presented to coordinators (interviews): To what extent were the pedagogical innovations developed during the funded research-based education projects adopted, adapted and/or reinvented, after the end of the funded period?; and What were the main factors that have contributed to the sustainability of pedagogical innovations, mainly at the macro (political and research), meso (institutional) and micro levels (projects’ dynamics)?

During the interviews with the 9 coordinators, we asked them to provide us with the contacts of some of their team members—the key participants—to deepen our understanding of the “sustainability actions” applied at individual level (MICRO level). An online questionnaire was designed with a set of questions that all key participants were asked to complete. The aims of the questionnaire were to: check the type of involvement of the key participant in the project; characterise the pedagogical innovation(s) developed and its operationalisation; determine the type of participation, the level of autonomy, the importance and effectiveness (impact) of participation in the context of the selected project(s); and understand the dynamics that occur after the end of the projects, with regard to the implementation, adaptation and/or reinvention of pedagogical innovations, as well as the reasons underlying them (e.g., where were they implemented, what were barriers, lessons learned and facilitators). A total of 40 key participants were contacted (40 key participants), but only 17 key participants responded to the online questionnaire (

Table 4. Corpus collected in the 2nd phase (interviews and questionnaires).

Host	U_1		U_2	U_3			U_4		U_5		I_1	I_2	Total
Project	Proj. 1A	Proj. 2A	Proj. 3B	Proj. 4C	Proj. 5C	Proj. 6D	Proj. 7F	Proj. 8F	Proj. 9G	Proj. 10G	Proj. 11H	Proj.12I	12 P
Year	2004	2010	2009	2010	2013	2012	2010	2012	2008	2010	2010	2011
Sponsor	FCT	FCT	FCT	MSTHE	CFG	CFG	CFG	CFG	FCT	FCT	MSTHE	CFG
Coorddinator (expertise)	A (Edu)		B (Edu)	C (Eng)	D (Eng)	E (Eng)	F (Eng)		G (Eng)		H (Psy)	I (Eng)	9 I
Key particpants	3		2	7			1		2		2	0	17 Q
Legend: University 1 (U_1); Institute 1 (I_1); Education (Edu); Engineering (Eng.); Psychology (Psy); Interviews (I); Online questionnaire (Q)

).

Notably, seven key participants participated in more than one funded research-based education project [Q1, Q2, Q3, Q7, Q10, Q13, Q15] (

Table 5. Corpus collected in the 2nd phase (questionnaires).

Host	U_1		U_2	U_3			U_4		U_5		I_1	Total
Q	Proj. 1A	Proj. 2A	Proj. 3B	Proj. 4C	Proj. 5C	Proj. 6D	Proj. 7F	Proj. 8F	Proj. 9G	Proj. 10G	Proj. 11H
Q1	x	x										2
Q2	x	x										2
Q3	x	x										2
Q4			x									1
Q5			x									1
Q6				x								1
Q7				x	x							2
Q8					x							1
Q9					x							1
Q10					x	x						2
Q11					x							1
Q12					x							1
Q13							x	x				2
Q14										x		1
Q15									x	x		2
Q16											x	1
Q17											x	1

).

The analysis of the perceptions of coordinators and other key participants (e.g., teaching staff and students involved in these innovations) helped to explore specific “sustainability actions” applied before the implementation of studies (e.g., in proposals), during the implementation of studies (e.g., during funding), and at the end of studies (e.g., after the financial support ends).

A content analysis technique [69,70,71], using WebQDA software, was applied to the data collected. Qualitative analysis involved specifying characteristics of a participant’s statement (coordinators and key participants), coding them, counting occurrences of the coded categories, and subsequently using descriptive techniques to analyse the data. In effect, the coded categories were treated as variables. The content analysis procedure was similar to the first phase, the distinction being whether the data were used to attempt to interpret meanings or focus on identifying dominant tendencies of “research sustainability actions”. Indicators of analysis have emerged from the authors’ conceptualisations of pedagogical innovations [24], the sustainability of research [28,42], and from the content analysis process [69,70,71].

Three main dimensions were considered in this second phase: the Portuguese political and research agendas to produce pedagogical innovations in HE and their sustainability (MACRO level); the host institution’s contribution to support pedagogical innovations in HE and their sustainability (MESO level); the project dynamics, through individuals promoting and sustaining pedagogical innovations in HE (MICRO level).

4. Results

The goal for the first phase was to characterise the national funded research-based education projects developed in Portuguese public HE institutions (PPHEIs), from 2004 to 2013; the goal of the second phase was to determine the macro, meso and micro actions (e.g., institutional structures and personal dynamics) that hinder or promote the sustainability of effective pedagogical innovations (before, during, and after the end of the funding period) in science and engineering fields.

Information from the three main sources of data (projects documents, interviews with project coordinators, and online questionnaires from key participants in projects) is given in the following sections, and thus triangulated in order to enhance the credibility of findings [65]. Results are presented and discussed below, considering each research phase.

4.1. Document Analysis of the Research-Based Education Projects

Table 6. Projects categorised by year and funding source.

	Research Sponsor
Funding Year	FCT	MSTHE	CGF	Total
[2004- ...]	3	0	0	3
[2005- ...]	0	0	0	0
[2006- ...]	9	0	0	9
[2007- ...]	0	0	0	0
[2008- ...]	3	0	0	3
[2009- ...]	3	0	0	3
[2010- ...]	3	10	3	16
[2011- ...]	0	0	1	1
[2012- ...]	0	0	2	2
[2013- ...]	0	0	2	2
Included studies	21	10	8	39

presents the funded research-based education projects promoted in the Portuguese public higher education context by year and funding source.

In the defined period of analysis (2004 to 2013), HE institutions and academics (e.g., teachers) had several opportunities, launched by different Portuguese Governments, to apply for grants to develop research-based education projects in the Portuguese context. However, each research sponsoring agency has its specific research purposes in its “open call applications”.

Both the MSTHE and the CGF had specific calls to grant research-based education focused on the promotion of pedagogical innovations in Portuguese HE. For example, the Portuguese Ministry of Higher Education (MSTHE) has funded ten research-based education projects, particularly in 2010. From 2010 onward, the FCG has been the main research sponsor of promoting pedagogical innovations, through eight research-based education projects in PPHEIs.

If we look to the kind of calls promoted by each research sponsor, we can highlight a more assertive concern, especially from 2010 onward, of supporting research-based education projects, intentionally focused on the promotion of students’ academic success and/or teachers’ academic growth. Both the FCG and the MSTHE included “Innovation in Higher Education” as keywords in their open research calls. We believe that this certainly affected the coordinators’ options regarding the kind of projects for which they applied for financial support during the selected period.

One reason for the Portuguese political and research agenda’s alignment to fund research-based education projects in HE can be explained, for example, by the political scenario of HE during the analysed period (e.g., during the Bologna Process’ implementation). In fact, Tavares et al. [72] explained that Portuguese HE academics have combined to change their pedagogical practices to a more student-centred approach, and this was a huge academic challenge, despite the European crisis that affected Portugal, especially from 2008 to 2010 and then going forward [30].

Results also reveal that 2005 and 2007 were weak years, in terms of the financial support of research-based education projects in HE. The “Bologna Process influence” emerged, in the document analysis, as a contextualisation reason for academics to apply for research funds to develop pedagogical innovations in HE, which uncovers an alignment between political and research agendas (in the FCT, CGF, and MSTHE).

MSTHE-funded projects (10) and CFG-funded projects (8) were considered short empirical studies (one-year period of funding), whereas the FCT-funded projects (21) were considered long empirical studies (two- or three-year periods of funding). However, research-based education projects need time to identify a specific educational problem, to look for solutions, to implement and evaluate them, and assure their sustainability, after the end of the funding period. Consequently, funded research-based education projects aiming to develop pedagogical innovation in academia will require more than one year.

Ref. [31] explains that one possible explanation for difficulties in promoting the “sustainability of research” could be related to the length of funding projects. Certainly, to open calls for short periods of time does not allow much more than to understand a particular phenomenon in HE (e.g., to identify factors for the students’ academic difficulties).

Results show that the 39 projects were developed in 15 PPHEIs, of two types: universities (9) and polytechnic institutes (7). Despite the tendency of growing convergence, partially due to the Bologna implementation process, polytechnic institutes and universities have specific missions and distinct purposes. Universities have a strong conceptual and research-oriented focus, and polytechnics are known for offering a more practical and vocational-oriented education [73].

Figure 1 systematises the “scientific areas” of curricular units and courses in which the included funded research-based education projects were implemented. This categorisation considered the “year of open call for research”.

A total of 19 projects were categorised in “HE”, with the majority funded by the FCT, and with most funded in 2006 and 2010. These funded projects mainly focused on the comprehension of a broad spectrum of PPHE subjects, such as the study of: professional fulfilment and motivation of teachers (Proj. 20); factors for the students’ academic success (Proj. 21); and women’s roles in science and engineering courses (Proj. 22).

The remaining projects (20) highlighted pedagogical innovations implemented in engineering courses (16), biology courses (3) and an accounting course (1). Some funded projects were, for instance, focused on the promotion of teachers’ academic growth (e.g., Proj. 2.A) and/or on students’ active learning (e.g., Proj. 11.H).

To sum up, document analysis of the funded projects suggested awareness in collating key participants of the host institutions (e.g., teachers and students) in order to assure the success of the project, namely, the promotion of an innovative strategy, even after the end of funding. This can also be a consequence of the kind of funding call open at the time (e.g., FCG and MSTHE, as already mentioned), and/or the overall conditions for the host institution’s support (e.g., involving “program champions”).

4.2. Coordinators and Key Participants of Selected Projects

Concerning the Portuguese political and research agendas to afford pedagogical innovations in HE and their sustainability (MACRO level), results show that all coordinators explained the importance of having funding to follow up projects.

Additionally, all coordinators mentioned the “length and extent of the financial support” as a “sustainability action”, which could compromise the dissemination and exploitation of the results, particularly the pedagogical innovations’ applicability, after the ending of the funded period.

For instance, Coordinator A had developed two sequential projects (Proj. 1.A and Proj. 2.A), specifically focused on the academic development process of biology teachers. This coordinator applied for new research funds to promote the continuation of the previously developed pedagogical innovations (e.g., questioning strategy in genetic curricular units) (Proj. 1.A) but involving other biology teachers (Proj. 2.A), as expressed by her: “In general, we are more successful [in sustaining pedagogical innovation] when the projects are interconnected […] where there is an articulation between approved proposals […] for instance, my project’s financial period ends, I win another one but with different goals” (Proj. 1.A and Proj. 2.A). Another example was presented by Coordinator 7, who had two funded follow-up projects (Proj. 7.F and Proj. 8.F), which allowed the research team to use and sustain different virtual laboratories in the engineering courses.

Probably, the coordinators had accomplished this articulation to be successful in their application for research funds, although they did not assume it in the interviews. This inconsistency may suggest a culture of disarticulation in the alignment between political and research agendas. [74] (p. 427) infers that teachers tend to assume “technically unarguable, and socially and politically neutral” positions. This reflection aligns with those coordinators’ statements, which could lead to the development of pedagogical interventions in HE that do not actually include the challenges and principles that emerge from policies, namely, the Bologna Process.

In what concerns “the host institutions’ contribution to supporting pedagogical innovations in HE and their sustainability” (MESO level), all the interviewed coordinators highlighted the importance of coordination between the research design (e.g., aims and methods), and the “institutional strategy of the host institution” to promote pedagogical innovations. The coordinator’s leadership in benefiting the host institution leaders contributed to the promotion, and even sustainability, of research, as expressed by this coordinator: “The first contact we did was with the Head of the Departments the President of the Pedagogic Assembly/PA […] therefore they have been always involved […] this dynamic causes decisions made by the PA President to nominate, for example, a person responsible to collaborate with us in the Departments […]” (Proj. 3.C).

Despite the HE organisational institutions’ differences (universities and polytechnic institutes), already discussed, all coordinators agreed on the existence of “special elements” in the host institution, because they had the authority and the capacity to meet the commitments in order to achieve the project goals. Those elements had different roles in the host institutions (e.g., directors of HE courses), and were able to motivate their peers (e.g., directors, teachers, etc.) of the courses to be involved in the development process for innovation. Authors call these elements the “program champions” [42,61], highlighting their relevant role in project goals’ achievement and sustainability.

The results revealed that two host institutions, both universities, have created “staff development offices” to support the development of pedagogical innovations, after the end of relevant funding periods. One coordinator explained that, when the funding period of a project finished, “the development of tutorial sessions with teachers and students of the host institution” was maintained. Another coordinator explained that the staff development office, installed in an engineering faculty, has enabled: (a) the maintenance of educational resources; and (b) retaining human resources (e.g., research fellows) to support teachers during impact evaluation of the implemented pedagogical innovations. Thus, outcomes emerged from the funded research-based education projects that have contributed to sustainability for pedagogical innovations.

Regarding “the influence of project dynamics to promote and sustain pedagogical innovations” (MICRO level), with respect to the choice of the research team, partnership between different profiles of team members also seemed to be important. For example, all coordinators have highlighted the importance of involving multidisciplinary teams with educational researchers and engineering teachers at the same university.

According to the opinion of the six coordinators (Proj. A.1, Proj. A.2; Proj. B.3; Proj. C.4; Proj. D.5; Proj. E.6; Proj. H.11), during this interdisciplinary collaboration (a) to prepare and/or reflect on the implementation (e.g., the impact of pedagogical innovations in teachers’ academic growth and/or students’ learning success); (b) the teachers were more focused on their academic practices (e.g., to design and implement pedagogical innovation in curricular units); (c) the educational researchers were responsible to collect and analyse data to monitor and evaluate the impact of the implemented pedagogical innovations.

As Pedrosa-De-Jesus et al. [19] highlighted, it is important to create institutional opportunities to enhance collaboration between teachers and educational researchers, particularly through involvement in research-based education projects. For that to happen, these “key actors” should become “peers” in conceiving and evaluating pedagogical innovations [31].

All coordinators clarified that they published the research results in journals, mostly in English (the common language of the scientific community). This certainly could constitute a constraint on that sustainable channel—science-related communication. Knowledge mobilisation in the social sciences, such as in educational sciences, could be an action that opens results to a wider audience and contributes to the sustainability of research outputs [75].

As presented in our rationale, currently, the research sponsors (e.g., European Commission) have been asking for the inclusion of “dissemination and exploitation plans for funded research” in order to guarantee: the impact of research on people, practices, organisations, and systems (research impact); and the use of research results beyond the term of the funding period (sustainability of research).

However, all coordinators emphasised that they are usually evaluated concerning scientific production in their areas of expertise (i.e., in biology or engineering), and not so much by the pedagogical innovations developed throughout research-based education projects.

The 17 key participants explicitly confirmed, in the online questionnaire, that the pedagogical innovations had been sustained when the initial funded research came to an end. For instance, key participants 1 and 2 participated in both Proj. 1.A and Proj. 2.A.

As we noted above, Coordinator A’s leadership in promoting two funded research-based education projects has contributed to the promotion and the sustainability of research results. Thus, this group of biology teachers was engaged in the development of pedagogical innovations in their academic practice.

In terms of the benefits, the key participants said that the main gains of their participation in the two funded projects were at two levels. First, at the research level, the key participants from biology [Q1, Q2] highlighted two important benefits: implementing pedagogical innovations in their classes as a result of research-based education (e.g., written formative feedback); contact with other scientific cultures, namely, educational research; and reflecting on their academic growth.

Second, at the academic level, other key participants from engineering [Q7, Q10, Q13, Q15] highlighted other potentialities, such as: designing and integrating various web-related technologies to teach and learn in engineering courses (e.g., virtual laboratories in practical lessons); creating supporting pedagogical documentation (e.g., online tutorial videos in theoretical lessons); and involvement in communities of practice of education of engineering.

Beyond the funding of those projects, one key participant from the engineering area [Q15] said that belonging to a community of practice of education in engineering allowed them to sustain the pedagogical innovations developed through previous funded research (e.g., remote laboratories) in other higher education institutions (e.g., in Brazilian universities).

Nevertheless, the key participants identified constraints to sustain the innovations after the end of the funding period, which emerged within the borders of the host institution where those projects were implemented, such as: the existence of leaders’ misconceptions of the conceptual matrix of the project [all Q]; the difficulty of reaching and convincing more colleagues to (re)use pedagogical innovations in their academic practices [Q4]; and the difficulty of maintaining technical and operational remote laboratories, without human resources capacitated for that [Q15 and Q16].

5. Conclusions

This study helped to characterise which “sustainability actions” have been included in the design and development process in national research-based education projects selected in the study and to identify which actions could hinder or promote the sustainability of pedagogical innovations developed through funded projects. The development process of sustaining pedagogical innovations emerging from funded research-based education projects in HE is complex, due to the multiple factors involved (political, institutional, and individual).

Results also demonstrated that, in some cases, the pedagogical innovations developed during funded research-based education projects were sustained by the key participants after the end of the funding period. In those cases, key factors resulted in promoting innovations and sustaining pedagogical innovations in academic practice over the years. Results show how “sustainability actions” were applied in projects, namely: in applying for project funding (before the funding period); in the project’s implementation (during the funding period); and after the project’s implementation (after the funding period).

Although we cannot generalise these outcomes to PPHEIs, due to the qualitative and non-representative sample of funded research in the analysed period (2004 to 2013), the study allowed us to propose some recommendations for the future at the political (recommendations aimed at politicians and sponsors), institutional (aimed at leaders) and individual levels (aimed at teachers); these recommendations impact how the sustainability of research is considered, throughout their proposals and development processes.

The political agendas should not only afford the development of pedagogical innovations in HE, but also guarantee that the project design proposal is integrated from the beginning with “sustainability actions” to disseminate and/or exploit research results, even without funding. A research funding agenda should open calls that explicitly define the goals to be achieved—produce innovations in HE (such as those of CGF) and require plans for sustainability (such as those from the European Commission). However, the funding period of the research needs to be adequate for such an endeavour. Short periods (e.g., one year) do not seem to enable conceiving, implementing, and assessing innovation in HE.

Decisions at the institutional level should be aligned with national and international political and research agendas, particularly those that are specifically related to the importance of promoting pedagogical innovations. For that, it is important that host institutions (e.g., universities) continue to provide “internal support”, not necessarily just funds, to teams that have already produced knowledge from previous funded research-based education projects. Different institutional initiatives could be implemented, such as: making diverse institutional resources (e.g., the creation of “staff development offices”) available to these research teams; giving internal extra funding for the development of follow-up research-based education projects; and valuing the involvement of research-based education projects in teachers’ professional evaluation process (e.g., investing in “staff development of teachers”).

Finally, the projects’ internal dynamics are closely related to the coordinator’s leadership, who may apply different “sustainability actions” during the development process of the project, namely: in the proposal project, where sustainability should be incorporated (e.g., by empowering the participants teachers, by planning effective dissemination of the project results); and during the project implementation, project coordinators should create an environment that ensures the participation of teachers in critically reflecting on their practices, prepare the conditions for the continuation of innovation after the funding period (e.g., negotiate such conditions at an institutional level), including “program champions” (e.g., directors of courses where the innovation will be implemented) and members with different profiles (e.g., educational researchers and teachers in line with the specialties of the courses where the innovation will occur). Nevertheless, projects should be developed in such a way that results/outcomes (e.g., pedagogical innovations) can be tailored to the needs of others, transferred to new areas, sustained after the funding period has ended and/or be used to influence future policy and practice.

The “sustainable actions” identified in our study can provide guidelines for the conditions necessary for the sustainability of research-based education projects, and expand their scope to other HE scenarios (course units, courses and/or institutions). In particular, this study reveals that different “key actors” are involved in the creation and sustainability of pedagogical innovations.

Although results cannot be generalised statistically—because generalisations are made for theory and not for a population [67]—the identified “sustainability actions” could be performed, as [24,25,28,31,44] highlight, by key actors of three higher education spheres—macro (policy and decision-makers in research-sponsoring agencies), meso (institutional leaders), and micro (teamwork dynamics). Each “key-actor” was linked to sustainability factors that have influenced (or not influenced) the sustainability of the research, namely:

• In the national research agencies (e.g., funding lines focused on pedagogical innovations in higher education);
• The host institutions (e.g., institutional leaders who have responsibilities in curricular development);
• The projects (e.g., researchers and teachers who, in the light of the policy guidelines, aim to introduce pedagogical innovations in their academic practices).

At an upper level of sustainability of research, upscaling it is necessary, meaning, for example, dialogue with diverse “key actors” to reinforce definitions from political and research agendas, so that research can be developed in a way so as to promote pedagogical innovation developed through research-based education projects in HE, and requiring its sustainability.

Ultimately, what is important is the presence of leadership competencies of “key actors” in each higher education sphere, because each “key-actor” could play a different, but relevant, role in influencing, promoting, and sustaining pedagogical innovations through educational research.