A Conceptual Framework for the Evaluation of Social Agriculture: An Application to a Project Aimed at the Employability of Young People NEET

Abstract

Agriculture can be a possible provider of social services of relevant importance for the whole society. In order to generate a valuable social service, a multi-actor approach is often applied, based on an active collaboration among public institutions, non-profit organizations, and private firms, and capable of generating multiple positive impacts. This new approach may both favorite agricultural diversification and enhance the quality of life of rural communities. However, in order to enable policymakers in motivating the public support to these types of initiatives, an evaluation method capable of disentangling the multiple benefits generated by social agricultural projects is required. In this paper, we adapted the evaluation method previously developed by the SIMRA consortium for Social Innovation initiatives, to a project aimed at the employability of NEETs in the south of Italy. A selection grid, framed by cross-referencing the national policy objectives of social agriculture and the criteria of eligibility adopted in public calls is proposed, to choose the suitable indicators for the evaluation. The evaluation experience allowed the measurement of 34 indicators of performance. The results prove that 12 indicators are positive, while 12 are moderate, and 10 are low and are mainly related to the enhancement of social inclusion. The evaluation exercise may be useful to disentangling the multiple outcomes generated by initiatives based on social innovation, which are highly based on intangible assets, and exert a positive effect on the internal cohesion and the engagement of the civil society.

1. Introduction

In the last decades, the agricultural sector has been attracting growing attention in the EU as a possible provider of social services of relevant importance for the whole society [1,2]. Beyond the multifunctional role of agriculture in the provision of ecosystem services, the domain of social services is comprehensively described in Di Iacovo and O’Connor [3]. More recently, several EU member states have enacted specific laws and regulations, to provide a clear definition of social agriculture and to activate some policy intervention to its promotion and deployment. In this context, Italy enacted the 141/2015 law, which defines social agriculture as an aspect of the multifunctionality of agricultural enterprises, aimed at the development of social and socio-sanitary services, educational, and socio-occupational placement. The aim is to facilitate plain and proper access to basic services to disadvantaged individuals, families, and local communities in all of the national territory, particularly in rural or lagging behind regions [4,5,6,7,8,9]. In order to generate a valuable social service, a multi-actor approach is required, where local authorities may establish a clear and flexible regulatory framework promoting an active collaboration among public institutions, non-profit organizations, and private firms. In this regard, social innovation (SI) may represent a promising approach, as the key players may experiment with novel interaction patterns and new informal institutions, based on co-creation, capable of valorizing local resources, and pursuing the co-production of multiple goods and services [10]. This new approach may be promising in terms of agricultural diversification and in terms of the viability of rural communities, due to the provision of social services, which are essential to ensure an adequate quality of life.

However, there is a high degree of complexity in promoting endogenous models of development, as there is no clear and straightforward relationship between the resources needed for the promotion of social innovation (e.g., voluntary workers, natural and financial resources) and the desired outcomes (e.g., services provided to marginalized individuals, marketed goods to sustain the project). In other words, policy support will be aimed at funding initiatives for the creations of immaterial capital, which will be capable of activating the generation of both social services for the targeted individuals, and other goods and services which may contribute to the sustainability of the SI initiative (e.g., sells of agricultural products).

Therefore, policymakers should be able to evaluate whether the resources allocated for supporting IS initiative are effective in achieving the objectives of social agriculture. The evaluation process should be capable of disentangling the multiple benefits generated by SI initiatives, and isolating the specific outcomes. Definitely, this is primarily important for public institutions (i.e., public managers, policymakers), in order to perform a cost-benefit analysis and to motivate the public support to social agriculture, from a political perspective. In addition, the evaluation is also important for the actors involved (i.e., innovators, consultants) in a specific initiative, in order to achieve an adequate level of self-awareness of the efforts and achievements during the evolution of the initiative [11,12].

In this paper, we present a case study that referred to a project for the employability of NEETs. In fact, in the context of Southern Italy, it is perceived that agriculture is the most relevant sector with the highest employability, due to the fact that farmers face enormous difficulties in finding young and professional workers for specialized jobs. In specific terms, organic farming is particularly emerging in this area and, therefore, the project was aimed at bridging the gap between NEETs and farms. It is an example of initiatives based on the collaboration of multiple heterogeneous actors and aimed at generating multiple relevant impacts for the advancement of the local agro-food system. However, we envisage the problem of identifying and discriminating the impacts of this sort of SI initiative, which we try to overcome by proposing a selection grid formed by cross-referencing the policy objectives pursued by means of social agriculture, and the selection criteria applied for funding a specific category of SI initiatives. The question we try to answer is the following: “which indicator best measures the impact of the SI initiative in terms of specific social agriculture objectives?”.

In our contribution, we borrowed the indicators which had already been developed in previous research, and that have already been tested in the evaluation of several types of SI initiatives, within the EU Horizon project “Social Innovation in Marginalised Rural Areas” (SIMRA) [13]. During four years of activity (from 2016 to 2020), the project contributed to the advancement and the understanding of SI in the agricultural sector, with special attention to forestry and rural domains (http://www.simra-h2020.eu/, accessed on 20 June 2021).

As a study case, we performed an evaluation exercise related to the promotion of a social agricultural project aimed at young people who are not in employment, education, or training (NEETs), in a rural area located in Southern Italy. In particular, we selected a project which is still at the early stage of advancement, whose promoters are keen to perform an evaluation exercise, to understand how to steer the activities to improve project effectiveness and sustainability, in the long run.

This paper faces two main issues. The first relates to the need to understand whether a specific project can effectively generate valuable social services for a rural community, which are consistent with the objectives of social agricultural policy. The second relates to the need to identify a set of pre-defined indicators, which are suitable to check some project critical aspects, and support the project managers in the adoption of some specific improvements.

In order to achieve these goals, we propose a methodology to adopt the evaluation indicators which are either consistent with the “social innovation” theoretical framework and are also to measure the outcomes of “social agricultural” policies. The evaluation exercise has been conducted by using a set of indicators, already tested in several case studies within the SIMRA project, and that are specifically addressed at SI projects or initiatives [14,15].

The further development and fine-tuning of the methodology proposed in this paper may be addressed at different types of audience: (i) actors who are involved in policy evaluation, such as policymakers and analysts, as well as evaluation agencies, (ii) project designers and managers, (iii) experts of communication activities, to disseminate the achievements of specific initiatives and facilitate the public engagement and involvement.

The structure of the paper is the following. In the next section, we present an introduction to the evaluation of social initiatives. In the third section, we describe the adaptation of the SI evaluation methodology developed by the SIMRA consortium, in order to perform an early-stage evaluation of an initiative aimed at improving the NEETs employability. The study case is reported in the fourth section, while the conclusions and implications for project managers and policymakers are drawn in the last section.

2. Evaluation of SI Initiatives

There are already several methodologies for the evaluation of third sector organizations [16,17,18,19,20,21], which may be adapted to evaluate the structural part of social innovation (i.e., resources involved, activated processes, etc.). However, to the best of our knowledge, there is still a gap in evaluation tools suitable for the analysis of early-stage initiatives stemming from multi-objective policies (e.g., creation of job opportunities for the youths, favoring women entrepreneurship) and based on multi-stakeholders interplay. Here, the perspective is that multiple policy measures are synergically designed to boost cooperation and collaboration between diversified groups of private and public institutions, and non-profit organizations in order to achieve multiple social goals, through social innovation. Consequently, the difficulty in performing the evaluation relies on the comparison between the resources mobilised (most of which are immaterial), and the bundle of benefits generated by the initiative. In addition, considering the fact that the arrangement of new relationships, or the re-definition of already existing ones, will likely generate some impacts only in the long terms, some indications are needed, particularly at the early stages of the initiative, where some relevant adjustments and improvements may be crucial, for the sustainability of the project.

In this context, SI is conceived as a novel approach to stimulate the synergy between different actors, aiming at satisfying common social needs. To this regard, the EU promotes SI for the development of the marginalised rural areas across Europe, considering that, as claimed by [22], along with the progressive reduction of public and private sources allocated for social interventions in the third sector, the politics of resources allocation has been changing, requiring objectivity, transparency, and rationality.

In this regard, one of the main achievements of the SIMRA project has been the creation of new knowledge to foster the scientific progress regarding the evaluation tools of SI initiatives in the agricultural sector, with special attention to those realized in marginalized rural areas (http://www.simra-h2020.eu/, accessed on 20 June 2021). The research consortium identified and analysed 54 examples of SI across Europe and Mediterranean Areas, which contributed to forming a robust theoretical and methodological framework, enriched with a wide empirical investigation of the SI. One of the main project’s outputs is represented by a comprehensive evaluation manual, embracing all the relevant aspects of the SI [13]. The manual covers all the essential aspects of the evaluation, starting from the definition of SI, which was previously provided by [23] and is stated as follows: “the reconfiguring of social practices, in response to societal challenges, which seeks to enhance outcomes on societal well-being and necessarily includes the engagement of civil society actors” (p.12). Next, it describes the comprehensive evaluation framework, providing specific concepts and mechanisms useful to explore the most important aspects of SI, which are subsequently covered by the evaluation criteria and the specification of indicators. In addition, instructions for the preparation of the evaluation, facilitating the application of the methodology to practical cases are provided. Another relevant contribution of the manual is the collection of tools, a set of 11 questionnaires addressed at the different types of actors of SI (e.g. people who conceived the idea and started the process, individuals who joined afterwards, beneficiaries of the activities, etc.). Within each tool, an ordered sequence of questions is listed, which are finalised for data collection needed to calculate the indicators. More than one hundred indicators have been elaborated, in order to capture the multifaceted aspects of SI, and they are grouped in 34 composite indicators, corresponding to aggregated concepts.

Certainly, the all-encompassing nature of the indicators, requires an accurate selection of the suitable indicators needed for the specific evaluation purpose (i.e., policy analysis), consistently with the local specificity of the study case which, in the case of SI, do play a relevant role. The selection of indicators presented in this paper has been performed by a selection grid, resulting from the cross-referencing of the different policy objectives of social agriculture and the selection criteria adopted by diverse administrative levels (e.g. central government, regional authorities, local agencies) in public calls.

3. Methodology

The starting point to explain the evaluation approach adopted in this paper, is represented by the introduction of the legal framework defining social agriculture in Italy, which involves at least the national and the regional administration government levels. This multilevel approach augments the complexity of the analysis, due to the interaction of different public bodies, which are responsible for the design and selection of public calls, through which local groups of actors are empowered and financially supported, to activate the specific SI initiative aimed at NEETs social inclusion and their employability. In this context, it is important to verify whether the objectives defined by the different policy measures are actually and adequately achieved by the SI initiative.

The methodology applied in this work can be summarized as follows:

• step 1: identification of criteria for the selection of indicators;
• step 2: selection of indicators for the evaluation of SI;
• step 3: data collection, by using the tools defined by the manual of evaluation;
• step 4: data elaboration.

Step 1. The Italian legal framework defining social agriculture is defined by the national law 141/2015. The regional governments are supposed to enact specific regulations, providing the rules for the elaboration of specific policy measures (e.g. calls for financial support, tenders for public procurement). Among the regional calls specifically addressed at social agriculture, three of them are relevant, as they provide financial support on an annual basis: PuLSA (Apulia Region), “Welfare, che Impresa!” (promoted by various Italian public fundations), and “Coltiviamo Agricoltura Sociale” (promoted by national agricultural associations). Besides some specificity, the calls recently opened have a similar structure, regarding the procedure for the empowerment and funding of local multi-agents groups, the eligibility criteria for the project application, and the selection criteria, according to the relevance and the target groups. The eligibility of projects depends on the consistency of the selection criteria of local calls with the national policy.

Concerning the criteria for project selection provided by the calls, the most important are the following:

-

activities and strategies to strengthen social farming practices: definition of precise strategies, activities and concrete commitment in the generation and reinforcement of local actors’ networks, both formal (i.e., contracts, agreements), and informal (i.e., practices, routines) relationships, for the achievement of the project objectives and the strengthening of the local social agriculture practices;

-

involvement of traditional farms: strong involvement of the traditional agricultural enterprise;

-

new networks: potential to create connections between the different actors at bonding (i.e., horizontal links between practitioners or beneficiaries), bridging (i.e., links between firms of beneficiaries with research and professionals), and linking (i.e., links between local actors and institutions) levels. Within these criteria, a specific focus is put on the social involvement of under 35s and women, which can be formal (i.e., hired labor) or informal (i.e., temporary collaboration), and should be directed at favouring the creative contribution to the project implementation;

-

innovation potential: expected innovative impact of the project in the specific context it will be implemented;

-

replicability of the project: formulation of the project based on standardised protocols, practices, and experiences, which represents the pre-conditions for possible replication by other subjects and in other contexts;

-

link with the context of action and with third parties: capacity to match local needs and a good potential to involve local actors;

-

presence of the technological component to facilitate scalability: partnership with technological partners and technical elements, favouring the scalability of the project, its best practices, and/or its format and protocols;

-

organization capacity and management skills of the project team: project teams endowed with better planning skills, strategic abilities, and competencies;

-

facilitation of the emergence of new projects: potential to foster co-design experiences (participation process) and possess concrete tools to implement social agriculture.

-

Concerning the second dimension of the evaluation framework (i.e., the relevance and target groups), we consider the national framework objectives, which can be summarised as follows:

-

social inclusion: integration of disabled workers and marginalised or disadvantaged peoples within the local agricultural sector;

-

creating conditions for social inclusion: provision of social services to local communities to promote, accompany and carry out actions aimed at developing skills and capacity, social and labour integration, and inclusion of recreational and services for daily life;

-

goods and services to improve social conditions: the creation of new health services supporting medical, psychological, and rehabilitation therapies aimed at improving the health conditions and social, emotional, and cognitive functions of the person, also through the help of farmed animals and the cultivation of plants;

-

education in nutrition and environmental awareness: capacity of the project to promote environmental and food education, and biodiversity safeguard. It is a broad objective aiming at the spread of local knowledge, by the organization of social and educational farms, such as initiatives for the reception and care of preschool children and marginalised people.

The crosscheck of the consistency of the regional criteria and the national objectives forms the basis for the selection of the indicators actually consistent with the nature of the specific project.

Table 1. Selection criteria of the composite indicators to evaluate SI initiatives concerning social agriculture.

	Objectives
Indicators of Relevance	A. Social Inclusion	B. Enabling Conditions to Social Inclusion (Social Activities)	C. Goods and Services to Improve Social Conditions	D. Education in Nutrition and Environmental Awareness
activities and strategies to strengthen social farming practices	X4—Engagement of civil society
involvement of traditional farms			X7.2—Leadership
creation of new networks (e.g. social involvement of under 35s and women)	X8.1—New networks	X10.1—Feedback loops and multiplier effects
innovation potential	X7.1—Social Innovation Idea	X8.2—New attitudes		X2—Response to societal challenges
replicability of the project/ best practices		X7.4—Capabilities
link with the context of action and with third parties		X6.2—Perceived Opportunities and Threats
organization capacity and management skills of the project team		X7.5—Endogenous versus exogenous drivers of the SI process	X11.3—Relevance of the SI initiative
facilitation of the emergence of new projects		X11.1—Relevance of the SI process

Source: Our elaboration. (Labels of composite indicators corresponds to those proposed by [13]).

reports regional selection criteria (in rows) and national objectives (in columns). The labels reported within some cells identify the composite indicators elaborated in the SIMRA evaluation manual [13].

Step 2. After the identification of the composite indicators, the following step consisted in the selection of indicators reported in the evaluation manual produced by the SIMRA project [13], which are consistent with social agricultural projects aimed at NEETs employability. Each indicator listed in the manual is well-rooted on the theoretical framework of the SI, and can be interpreted accordingly.

Table 2. List of selected composite indicators and indicators suitable for the evaluation of a SI aimed at social agricultural initiatives.

X4—ENGAGEMENT OF CIVIL SOCIETY (p. 126)

SIE1. Local community’s contribution to the results of the SI initiative Q: “To what extent has the local community contributed to the results of the Social Innovation initiative?” Measure of the contribution of the local community in determining the success of the SI initiative. The higher the local community’s contribution to the results of the SI initiative with respect to external actors, the better its capacity to produce effects on multiple actors Likert scale, 1–10, referred to 6 categories of actors: 1. Supportive policies; 2. External helpers as advisors, brokers, and animators; 3. Wider local community 4. Core group (innovators and followers); 5. Members of the network; 6. Project partners Ratio between two average scores expressing the role of local community and the role of external actors Range of values: [0.1–10]; Acquisition data Tools: 3, 4

SIE2. Actors’ motivation for engaging in the SI initiative Q: “To what extent has the motivation to serve a good cause inspired the actors in the Social Innovation network?” Measure of the number of motivations exhibited by the actors engaged in the SI initiative. The higher the number of actors’ motivation, the greater the likelihood of the SI initiative to produce its expected results and to determine the desired effects in the long run Selection of motivations: 1. I liked the idea and it made sense; 2. I wanted to serve a good cause; 3. I like the leadership and charisma of the innovator(s) and followers; 4. I wanted to share my expertise for the project; 5. I wanted to feel personally fulfilled; 6. I wanted to receive economic benefits; 7. It was part of the duties of my job; 8. For previous relationships I had with people involved; 9. others, to be specified). 100*(n. of selected items)/total respondents Range of values: [0–100]; Acquisition data Tool: 4

SIE3. Participation of actors in network meetings Q: “To what extent have the actors been participating in network meetings?” Measure of the frequency of participation of Transformers in the meetings of the network. The higher the measure, the greater the commitment of the actors in the SI initiative Selection of 4 levels of participation: 1 (a few of them); 2 (some of them), 3. (many of them), 4. (almost all of them). N. of respondents who scored 3 or 4 along a four degree likert scale (1 = few meetings; 4 = almost all meetings) over the total number of respondents * 100 Range of values: [0–100]; Acquisition data Tool: 4

SIE4. Civic society engagement in the SI network Q: “To what extent has the Social Innovation network engaged civil society?” Share of representatives of civic society with respect to other types of participants to SI network. The higher the value, the greater the likelihood of achieving the expected results Selection of the The categories of participants to the Social Innovation network are: 1. Business entrepreneur (for profit, business); 2. Social entrepreneur (not for profit); 3. Member of a civil association; 4. Public sector officials (different levels of administration); 5. Citizen; 6. Other (to be specified case by case). Percentage: 100* (Number of options “3” + “5”/Total options) Range of values: [0–100]; Acquisition data Tools: 3, 4

X2—RESPONSE TO SOCIETAL CHALLENGES (p. 123)

SIS1. Capacity of the SI to tackle multiple European societal challenges Q: “To what extent has the Social Innovation initiative dealt with European societal challenges?” Measure of the capacity of the SI idea to tackle at the same time multiple European societal challenges as identified in the Europe 2020 strategy. The higher this capacity, the higher is the initiative likelihood to spread effects in different domains The European societal challenges are: (i) health; (ii) ageing of population; (iii) income, jobs, education; (iv) sustainable agriculture and food security; (v) water use and quality; (vi) secure, clean and efficient energy; (vii) smart, green and integrated transport; (viii) environment and climate change; (ix) inclusive societies; innovative societies; (x) secure societies. The computation is based on the identification by Innovator(s), Follower(s) and project partners of the different challenges addressed by the Social Innovation idea, i.e. (i) Health and wellbeing; (ii) Demographic change (e.g. aging of the population); (iii) Income, jobs, education; (iv) Sustainable agriculture and forestry and food security; (v) Water use and quality; (vi) Secure, clean and efficient energy; (vii) Smart, green and integrated cities and mobility; (viii) Environment and climate change; (ix) Social inclusion and cohesion; (x) Innovation and modernisation; (xi) Security and freedom. Count of items selected by respondents, divided the total number of items * 100 Range of values: [0–100]; Acquisition data Tools: 3, 4, 5

SIS2. Actors’ perception of the improvement in European societal challenges thanks to SI initiative Q: “To what extent has the Social Innovation initiative improved the European societal challenges in the territory, according to the Social Innovation actors?” Measure of actors’ perception regarding the advancements achieved by the SI initiative in different European societal challenges. The higher the value, the higher is its likelihood to determine positive effects in the local context. Based upon the perceptions of the actors in the Social Innovation, the indicator measures the extent to which the European societal challenges have been improved in the territory due to the Social Innovation initiative. The evaluation is done on an ordinal scale, i.e. 0 (not at all), 1 (to some extent), 2 (to a great extent). The answers of Innovator(s), Follower(s), Transformer(s) and project partners are considered to assess the perceptions of actors involved both in the Social Innovation process and project. Sum of 0–2 score attributed by respondents for each challenge, divided maximum score * 100 Range of values: [0–100]; Acquisition data Tools: 3, 4, 5

X6.2—PERCEIVED OPPORTUNITIES AND THREATS (p. 140)

Ba1. Balance between opportunities and threats Q: “To what extent local conditions have enabled the Social Innovation’s emergence?” Measures of the balance between enabling and constraining conditions for development of SI initiative during its initial steps, as perceived by actors. The higher the value, the more likely the SI initiative will emerge Identification of maximum three types of enabling factors and three types of constraining conditions), for each domain: 1. economic, 2. social, 3. environmental, 4. institutional.Count of items selected by respondents, divided the total number of items * 100 Range of values: [0–12]; Acquisition data Tools: 3, 4, 5

Ba2. Supportive policies’ role on sustaining the results of the SI initiative Q: “To what extent have supportive policies sustained the results of the Social Innovation initiative?” Measure of the contribution of supportive policies to SI initiative The indicator measures (1–10 Likert scale) how much of the success of the Social Innovation initiative can be attributed to supportive policies, according to the perceptions of Innovator(s), Follower(s), and Transformer(s). The importance of supportive policies for sustaining the results of the Social Innovation initiative is measured on a Likert Scale from 1 to 10, where 1 is “not at all” (i.e., the results of the Social Innovation initiative cannot be attributed to supportive policies at all) and 10 is “to a great extent” (i.e., the results of the Social Innovation initiative can be attributed to supportive policies to a great extent). Mean of all answers for “Supportive policies” in Tools 3 and 4 Range of values: [0–10]; Acquisition data Tools: 3, 4

Ba3. Contribution of the Social Innovation initiative to local governance (democracy) Q: “To what extent has the Social Innovation initiative dealt with issues of governance?” The indicator describes the extent to which the SI initiative ameliorates (or worsen) the quality of local territorial governance. The higher the value, the higher is the SI contribution to enhance the quality of local governance Identification (dichotomous choice) of governance elements: (1) Options for citizens engagement, (2) Stakeholders consultation, (3) Voice of minorities, (4) Gender balance, (5) Transparency, (6) Bureaucracy, (7) Capacity of public administrations, (8) Policy initiatives, (9) Legal framework, (10) Conflict of interests and corruption, (11) Quality of public services, (12) Market and economy. Average score obtained between the counts of selected item (3) and item (4), with respects to counts of item (5) Range of values: [0–100]; Acquisition data Tools: 3, 4, 5

X7.1—SOCIAL INNOVATION IDEA (p. 142)

Ca1. Attractiveness of the SI idea Q: “To what extent has the Social Innovation idea attracted the Transformer(s)?” The indicator measures whether the SI idea was amongst the motivations that drove the transformers to get involved in the initiative. The higher the value, the more likely the transformers considered the SI initiative interesting and valid. This is a proxy of SI capability to attract others Dichotomous choice among following options: they liked the idea and it made sense, they wanted to serve a good cause, they liked the leadership and charisma of the innovator(s) and followers, they wanted to share their expertise for the project, they wanted to feel personally fulfilled, they wanted to receive economic benefits, they was part of the duties of their job, for previous relationships they had with people involved, other motivations % “yes” in options options (they liked the idea, and it made sense) and (they wanted to serve a good cause) Range of values: [0–100]; Acquisition data Tool: 4

Ca2. Innovativeness of the SI idea in the territory Q: “To what extent do the actors in the Social Innovation process perceive the idea as innovative in their territory?” The indicator measures to what extent the actors in the whole SI network perceive their SI idea to be innovative in their territory. The higher the indicator value, the greater the perceived innovativeness of the SI idea in the territory The perception is expressed on a Likert Scale from 1 (=not at all) to 10 (=to a great extent). Compute the mean of all answers from Tools 3 and 4 Range of values: [1–10]; Acquisition data Tool: 3, 4

X7.4—CAPABILITIES (p. 146)

Cd2. Previous experience of actors who contributed to the SI process Q: “To what extent have the previous experiences of actors contributed to the development of the Social Innovation process?” The indicator measures whether the innovators, followers and transformers have had previous working experience in fields related to the SI initiative. It is assumed that, having had previous experiences in similar fields, the actors have higher levels of capacity to contribute to the development of the Social Innovation initiative. Percentage of respondents who have previous experience in agriculture sector Range of values: [0–100]; Acquisition data Tool: 3, 4

X7.5—ENDOGENOUS VERSUS EXOGENOUS DRIVERS OF THE SI PROCESS (p. 142)

Da1. Role of newcomers in the SI process Q: “To what extent have newcomers contributed to the development of the Social Innovation process?” The indicator measures if innovators, followers and transformers have a specific relationship with the territory where the SI has been implemented. The higher the indicator value, the greater the role of newcomers (i.e., someone who comes from other provinces) in the development of the SI process It specifies the percentage of actors in the core group (Innovator(s) and Follower(s)) and in the network (Transformer(s)) that are newcomers for the territory. Respondents have 1 option for which to choose for describing their own relationship to the local territory: (i) I have always lived here (endogenous); (ii) I have lived here, but I studied or worked away (neo-endogenous); (iii) I come from outside, but I have been living here for a while (neo-endogenous); (iv) I come from outside and I consider myself a newcomer (exogenous). Percentage: 100*(Number of “yes” in option 4/Total number of respondents) Range of values: [0–100]; Acquisition data Tools: 3, 4

Da2. SI actors’ perception of external helpers’ contribution to the results of the SI initiative Q: “To what extent have external helpers contributed to the results achieved by the Social Innovation initiative?” The indicator measures to what extent the results of the SI initiative can be attributed to external helpers, such as advisors, brokers, animators, politicians, etc. The higher the indicator value, the greater the perceived contribution of external helpers to the results of the SI initiative The contribution of external helpers to the results of the Social Innovation initiative is measured on the basis of the perceptions of respondents (Innovator(s), Follower(s), Transformer(s) and project partner(s)), and expressed on a Likert Scale from 1 (not at all) to 10 (to a great extent). Step 1: Mean of scores for all respondents in Tool 3 and Tool 4; Step 2: Mean of scores for Tool 5; Step 3: Mean of the means in Step 1 and Step 2 Range of values: [1–10]; Acquisition data Tools: 3, 4, 5

X8.1—NEW NETWORKS (p. 152)

Ea1. Attendance level at meetings in the SI process Q: “To what extent have the Social Innovation members attended the process meetings?” The indicator measures the level of attendance of the transformers at the meetings of the Social Innovation process. The higher the indicator value, the more the members of the Social Innovation process attended the meetings. The respondents have to self-evaluate the level of attendance at the meetings on a Likert Scale: (1) [I have attended] a few of them; (2) [I have attended] some of them; (3) [I have attended] many of them; (4) [I have attended] almost all of them. Average score along a 4 degree likert scale Range of values: [1–4]; Acquisition data Tool: 4

Ea2. Balance between public and private sector of the members of the SI network Q: “To what extent have members of the Social Innovation network been equally distributed amongst the public and private sector?” The indicator measures the distribution of network members of the SI process amongst the private and public sector. The higher the indicator value, the more the members of the SI process are equally distributed amongst the public and private sectors. The Innovator(s), Follower(s), and Transformer(s) have to specify which of the following options they predominantly represent: (i) Business entrepreneur; (ii) Social entrepreneur; (iii) Civil society organisation; (iv) Public institution; (v) Yourself; (vi) Other. 1- absolute value of: [2*(n. public officials/total respondents) -1] Range of values: [0–100]; Acquisition data Tools: 3, 4

Ea3. Contribution of the members of the SI network to the results of the SI initiative Q: “To what extent have members of the network contributed to the results of the Social Innovation initiative?” The indicator shows to what extent the results of the SI initiative can be attributed to the members of the network. The higher the indicator value, the more the members of the SI network contributed to the results of the SI initiative. The indicator is expressed on a Likert Scale, from 1 (not at all) to 10 (to a great extent), based on the perception of Innovator(s), Follower(s) and Transformer(s). Average score for option Members of the network” Range of values: [1–10]; Acquisition data Tools: 3, 4

Ea5. Female inclusion in the SI network Q: “To what extent have female members been included in the Social Innovation network?” The indicator measures the participation of females within the Social Innovation network in the reconfiguring process. The higher the indicator value, the greater the proportion of female members in the SI network Proportion of female participation within the Social Innovation network.Innovator(s), Follower(s), and Transformer(s) are considered. Percentage of female respondents Range of values: [0–100]; Acquisition data Tools: 3, 4

Ea6. Young people’s participation in the Social Innovation network Q: “To what extent have young people participated in the Social Innovation network?” The indicator measures the proportion of young people participating in the SI network. The higher the indicator value, the more young people have participated in the SI network. This indicator is useful for verifying whether young people are active in the promotion of a Social Innovation process. Proportion of young people participating in the Social Innovation network. Innovator(s), Follower(s) and Transformer(s) are considered. Percentage of young people (under 40) respondents Range of values: [0–100]; Acquisition data Tools: 1, 2

Ea7. Education level within the SI network Q: “To what extent has the Social Innovation process been promoted by actors with university level qualifications?” The indicator measures the proportion of members of SI network who have university level qualifications. The higher the indicator value, the greater the proportion of members of SI network with university level qualifications involved in the SI process. The indicator is useful for verifying whether people with university level qualifications are active in the Social Innovation process. Proportion of Innovator(s), Follower(s) and Transformer(s) (i.e., all actors involved in the Social Innovation process) who have university level qualifications. Percentage of people involved in the SI process with university degrees Range of values: [0–100]; Acquisition data Tools: 3, 4

Ea10. New relationships within the SI network Q: “To what extent have new relationships been created within the Social Innovation network?” The indicator shows whether the relationships established within the SI network already existed or have been newly created. The higher the indicator value, the greater the proportion of newly created relations The respondents (Innovator(s), Follower(s), Transformer(s)) were asked to state how many of the contacts they had within the Social Innovation network were: (i) Close contacts (i.e., already existing before the creation of the Social Innovation network); (ii) Already known by name; (iii) Completely new contacts. The indicator is expressed as a percentage of new contacts (the sum of types ii and iii) of the total number of contacts. Average percentage per respondent of newly created relations Range of values: [0–100]; Acquisition data Tools: 3, 4

Ea12. Level of internal trust in the SI network Q: “To what extent has trust been spread amongst the actors within the Social Innovation network?” The indicator shows to what extent the members of the SI network trust each other. The higher the indicator value, the greater the level of trust amongst the actors within the SI network It is based on the perceptions of Innovator(s), Follower(s) and Transformer(s). The indicator is expressed on a Likert Scale, from 1 (not at all) to 10 (to a great extent). Average score along a 10 degree likert scale Range of values: [1–10]; Acquisition data Tools: 3, 4

Ea13. Level of representativeness of the actors involved in the SI network in relation to the categories of the organisations Q: “To what extent were actors in the network representative of the categories of organisations involved in the Social Innovation network?” The indicator shows the level (0–10 Likert scale) of representativeness of different types of organisations. The higher the indicator value, the greater the perceived level of general representativeness of the actors in the Social Innovation network in relation to the categories of the organisations. The indicator shows the level of representativeness of the actors involved in the network in relation to the category of organisations. Respondents have to state their perception of the extent to which the actors in the network are representative of: (i) Public administrations; (ii) Public enterprises; (iii) Civil organisations (e.g. associations, not-for-profit); (iv) Private enterprises (e.g. for profit). The level of representativeness is measured on a Likert Scale from 1 (not at all)) to 10 (to a great extent). The evaluation question is posed to Innovator(s), Follower(s) and Transformer(s). Average of the mean score along a 10 degree likert scale obtained for each category for all respondents Range of values: [0–10]; Acquisition data Tools: 3, 4

X8.2—NEW ATTITUDES (p. 159)

Eb1. Level of pro-action of transformers during the SI process Q: “To what extent have the network members been proactive during the Social Innovation process?” The indicator describes at which moment in the SI initiative [(i) From the beginning; (ii) During the development of the Social Innovation process; (iii) After the first results of the implementation of the Social Innovation project; (iv) It is not yet pro-active] the network members changed their attitudes and became pro-active. The higher the indicator value, the greater the proportion of network members who were or became proactive during the SI process. The basic assumption is that the members of the Social Innovation network have been convinced of the validity of the initiative from the beginning, or have changed their attitudes towards the Social Innovation initiative during its development.Four moments were considered by the respondents: “My attitude towards the Social Innovation initiative became pro-active …” (i) From the beginning; (ii) During the development of the Social Innovation process; (iii) After the first results of the implementation of the Social Innovation project; (iv) It is not yet pro-active. Count of answers to opinion 1 and 2, divided the total number of items * 100 Range of values: [0–100]; Acquisition data Tool: 4

Eb2. Perception of the actors of their level of empowerment during the SI process Q: “To what extent have the actors felt empowered during the Social Innovation process? The indicator shows the level of empowerment felt by members of SI network during the SI process. The higher the indicator value, the more the actors felt empowered during the SI process Respondents must indicate the level of empowerment felt by themselves, as Innovator(s), Follower(s), and Transformer(s) during the Social Innovation process. The level of empowerment is measured on a Likert Scale from 1 (not at all) to 10 (to a great extent), and it is based on the perceptions of the respondents. Average score along a 10 degree likert scale Range of values: [1–10]; Acquisition data Tools: 3, 4

X7.2—LEADERSHIP (p. 144)

Cb2. Innovators and Followers’ contribution to the results of the Social Innovation initiative Q: “To what extent have the Innovator(s) and Follower(s) contributed to the results of the Social Innovation initiative?” The indicator measures to what extent the results of the SI initiative is the result of the efforts made by the project partners, in relation with other factors. The higher the indicator value, the greater the contribution of the project partners The respondent must specify to what extent the results of the Social Innovation initiative can be attributed to the action of the core group (i.e. Innovator(s) and Follower(s)), compared to the contribution of other factors. The indicator is based on perception of the core group itself (Innovator(s) and Follower(s)) and members of the network (Transformer(s)). Average score of individual responses on a 10 degree likert scale Range of values: [1–10]; Acquisition data Tools: 3, 4

X10.1—FEEDBACK LOOPS AND MULTIPLIER EFFECTS (p. 171)

Ha1. Likelihood of feedback loops thanks to dissemination activities Q: “To what extent has the Social Innovation initiative been disseminated in order to increase the likelihood to generate feedback loops?” The indicator measures the use of dissemination channels The higher the number of dissemination channels used by the actors of the Social Innovation, the greater the likelihood of the Social Innovation initiative to generate feedback loops. The respondent must select the dissemination channels: (i) events; (ii) printed material; (iii) press; (iv) websites; (v) social media; (vi) newsletter; (vii) broadcasting; (viii) meetings with donors; (ix) meetings with politicians; (x) meetings with enterprises; and (xi) communications to other networks. Percentage of communication channels activated by the SI initiative and aimed at local actors Range of values: [0–100]; Acquisition data Tools: 1, 3

Ha2. Upscaling of the SI initiative Q: “To what extent has the Social Innovation initiative been upscaled to higher levels?” Diffusion of the SI initiative beyond the local level. The indicator measures whether the Social Innovation initiative has had: [1] effects beyond the locality; [2] has contributed to the development of national or international laws or standards; [3] or if it was aggregated in national and international groups representing similar Social Innovation initiatives. The higher the influence of the Social Innovation initiative at different levels, the greater the likelihood it will diffuse at higher levels The respondent must select the most suitable choice. Percentage of dissemination actions targeted at upper administrative levels (e.g. province, region) Range of values: [0–100]; Acquisition data Tools: 1, 3

Ha3. Out-scaling of the SI initiative To what extent have people in different contexts come to learn about the Social Innovation initiative and then did something similar themselves?” Diffusion of the SI initiative to other surroundings. The indicator measures the potential of the Social Innovation initiative to attract people from different contexts who want to learn about the initiative and then do something similar. The greater the number of similar initiatives that have come to learn from the SI, the greater the likelihood it will diffuse its results to other surroundings. The respondent must specify the if the SI initiative has been understood and was studied by people coming from external contexts Percentage of similar initiatives to whom the SI initiative had established some connections Range of values: [0–100]; Acquisition data Tools: 1, 3

Ha4. Replication of the SI initiativeQ: “To what extent were the actors in the Social Innovation initiative capable of identifying elements that would enable its replication?” The indicator measures the capability of the actors in the Social Innovation initiative of identifying the elements that enable its replication. The higher the number of elements possessed by the actors of the SI initiative, the greater the likelihood that it can be replicated. The respondent must indicate the elements that enable the SI initiative replication. Percentage of capabilities identified by the actors, which are suitable for SI initiative replication Range of values: [0–100]; Acquisition data Tools: 1, 3, 6, 7

X11.1—RELEVANCE OF THE SI PROCESS (p. 184)

R1. Shared individual and collective needs within the whole SI network Q: “To what extent have the individual and collective needs of the actors been shared within the whole Social Innovation network?” The indicator measures the congruence of needs identified by Innovator(s) and Follower(s) during the idea formulation with those identified by Transformer(s) during the Social Innovation process. The indicator considers individual and collective needs of Innovator(s) and Follower(s) together with needs that the Transformer(s) think the Social Innovation process was attempting to address. The higher the number of needs individually and collectively shared by Innovator(s), Follower(s) and Transformer(s) of the total number of needs identified, the greater the relevance of the Social Innovation process. The respondent must identify which individual needs have been addressed by the SI initiative by Innovators and Followers and whether they are coherent with those identified by Transformer(s) during the Social Innovation process. Percentage of individual and collective needs which have been covered by the SI initiative, with respect to the total number identified needs Range of values: [0–100]; Acquisition data Tools: 3, 4

R2. Shared vision regarding collective needs within the whole SI network Q: “To what extent has the vision of collective needs been shared by actors of the Social Innovation process?” The indicator measures the proportion of shared needs which are included in the vision elaborated by the by actors of the Social Innovation process. Innovator(s), Follower(s) and Transformer(s) share the same vision of collective needs to be satisfied by the Social Innovation process. The higher the number of actors in the network who identify at least one need which has also been identified by Innovator(s) and Follower(s), the greater the relevance of the SI process. The respondent must identify which shared needs have been addressed by the SI initiative by Innovators and Followers and whether they are coherent with those identified by Transformer(s) during the Social Innovation process. Percentage of shared needs covered by the vision elaborated by the SI actors Range of values: [0–100]; Acquisition data Tools: 3, 4

X11.3—RELEVANCE OF THE SI INITIATIVE (p. 186)

R5. Level of satisfaction of the SI initiative actors with territorial needs Q: “To what extent have the products and/or services provided by the Social Innovation initiative satisfied the territorial needs of actors?” The indicator measures the extent to which the products and/or services provided by the SI initiatives satisfy the needs of the territory. The indicator is based upon the perception of all categories of actors involved in the Social Innovation initiative within a certain territory (excluding the beneficiaries), i.e. Innovator(s), Follower(s), Transformer(s) and project partners. The higher the level of satisfaction of the actors with respect to the territorial needs, the greater the relevance of the Social Innovation initiative. The respondent must specify whether the products or services provided by the SI initiative were aimed at satisfying the territorial needs Average of the score expressing the level of satisfaction, within range 1–10 Range of values: [1–10]; Acquisition data Tools: 3, 4, 5

R6. Shared needs within the SI initiative Q: “To what extent were the needs of the actors of the Social Innovation initiative consistent with those identified by the beneficiaries?” The indicator measures the congruence of the needs identified by the actors of the SI initiative (Innovator(s), Follower(s) and Transformer(s)) with those of the beneficiaries.The higher the congruence of the needs identified by the actors of the Social Innovation initiative with those of the beneficiaries, the greater the relevance of the SI initiative. The respondent must specify whether the products or services provided by the SI initiative were consistent with those identified by the beneficiaries Percentage of the needs identified by the SI actors, with respect to those identified by the beneficiaries Range of values: [0–100]; Acquisition data Tools: 3, 4, 6, 7

Source: Description of the indicators, labels and page references are referred to the SIMRA evaluation manual [13].

reports, for each indicator, the description, the question posed to the respondent, the calculation method, and the tool required for the data collection, as extensively explained in the SIMRA manual.

Step 3. The data collection requires particular attention to the correct specification of the individuals to be interviewed, as specified by [13], who are classified according to their roles within the SI initiative.

The first group of agents is represented by the key leaders and initial drivers of innovation. These are labelled as innovators and are the individuals who first had the idea and elaborated on it. They form the first nucleus of the SI initiative.

The second group of agents joining the innovators, are called followers. They provide the initial support as they believe in the initiative and firstly adopt it. Together with the innovators, they form the core group of the initiative.

As the core group starts to implement the idea, new agents join the initiative. These early adopters are referred to as transformers, who support the core group by adopting the idea, and further by spreading it to other people.

Innovators, followers, and transformers together form the SI network. While the SI network relates only to the agents directly involved in the stages of creation, implementation, and development of the SI initiative, other individuals are involved in the later stages of the initiative, as they are affected by its outcomes and impacts. These agents are identified as beneficiaries.

Step 4. Most of the collected information are the result of a verbal judgement and, therefore, are qualitative data, which can be elaborated in terms of counts and scores on the likert scale. Subsequently, each indicator has been elaborated according to the formulae specified in [13]. In order to achieve an overall evaluation, results of the elaborations have been coded according to tertiles, expressing a low, medium, and high performance. Finally, we obtained a synthetic evaluation of the overall SI initiative, based on the distribution of the measure of performance indicators, according to three tertiles.

4. Empirical Application

4.1. Description of the Case Study

The evaluation exercise refers to an overall project named “SeminaMenti—L’Orto dei Principi”, which literally means “Mind Seeding—The Princes’ Vegetable Garden”, whose vision consists in educating consumers to high quality vegetables, locally produced by farmers who provide some working positions reserved to include socially marginalised people. It is a relatively small project, promoted by a partnership of three actors, a social cooperative, a professional training agency, and a charity organization, and the endorsement of the municipal government of San Severo town. In this context, the SI initiative object of evaluation is aimed at the improvement of the employability of 10 young people NEET who will likely be hired by the organic farms involved in the whole SI initiative, for producing, processing, distributing, and selling agricultural products to the local community. The municipal government of San Severo town (Apulia region, Italy) allocated about 30,000 EUR budget to incentivize the participation of the youngsters, while the Apulia region supported the cost for the training activities. The core of the initiative is represented by the training activities of NEETs, whose purpose is twofold: to provide and reinforce professional and soft skills, and to favorite the creation of relationships among all people involved in the project, who should boost the potential development of organic farming and the provision of high-quality food for the local community.

In order to apply the SIMRA evaluation method, the basic question to be answered is whether this project can be considered as a “social innovation”. In order to perform this first check, we followed the SIMRA approach to verify the presence of the fundamental prerequisite of the SI:

i)

the trigger, or the prime-mover of the SI, that is a happening or condition no longer acceptable by the community (e.g., marginalisation of social groups), or that represents unexpected chances for the development of the area (e.g. normative change turning the attractiveness of the area). It can be a single event (e.g. an environmental disaster) or the accumulation of happenings (e.g. consistent migration flows due to unemployment);

ii)

the unmet social needs, which are challenges referred to the societal, economic, environmental, and/or institutional domains;

iii)

the perceived context, that is the conditions that influence the actors’ behaviour;

iv)

the agency, that is a group of actors (with their ideas, values, willingness, and capacity) preparing and implementing the transformation;

v)

the preparatory actions, including objects, activities, discourses, and narratives of change used by the agency for preparing and starting the reconfiguration of social practices and networks since the early stage of the process.

Table 3. Characterization of the SI initiative, within the “SeminaMenti” project.

Elements of SI	Specific Features of the SI
Trigger	The municipality of S. Severo needed to provide social support to NEET, who are not currently targeted by any specific social policy; the presence of social cooperatives able to provide social services, with a new “inclusive” vision; the availability of land to be used as experimental fields for organic farming
Social needs	High unemployment rate; young people with poor education and skills, unable to catch regular job opportunities and exposed to engagement to illegal activities; agricultural products are not valorised; local community is not educated in food quality and seasonality
Perceived context	Agriculture is the economic sector with highest employability; local consumers are willing to buy high quality products
Agency	Institutions: municipality of S. Severo, Social Cooperative “Attivamente”, education and training agency “SMILE”, non-profit charity organisation “Caritas”; about 10 people belonging to the institutional agents, who are directly involved in the initiative, one external consultant (agronomist)
Preparatory actions	Identification of external actors to activate possible collaboration for further development of the SI; market observation of organic food

Source: own elaboration, based on interviews of actors involved in the project.

reports the arguments supporting the hypothesis that the case study characteristics comply with the fundamental requirements for SI definition.

The project activities started in October 2019, while the data collection for the evaluation occurred in March 2020, that is, the mid-term of the one-year full duration of the training of NEETs. The group of people involved in the project management was aware of the importance of getting an intermediate evaluation, to get some advice and recommendations, and, therefore, exhibited full cooperation to conduct the direct interviews.

4.2. Data Collection and Elaboration

A preliminary introductory focus group was conducted in February 2020, with all representatives of institutional actors promoting the “SeminaMenti” project (i.e., the social cooperative, the professional training agency, the charity organization, and the municipal councilor for social affairs). Subsequently, due to the containment measures of the Covid-19 adopted in Italy (Decree of the Council of Ministers DPCM 26 April 2020, starting 4 May, and up to 17 May 2020), we conducted a series of in-depth interviews via online web conferencing platforms. We adopted the same interview protocols used in the SIMRA case study analysis, namely Tools 3, 4, 5, and 7.

After the survey, we calculated the indicators for the evaluation of SI selected for this specific initiative (Table 2), by applying the formulae specified in the evaluation manual [13]. The results of the data analysis are reported in

Table 4. Results of the evaluation exercise.

Indicators and Sub-Indicators	Value	Eval. Range	Perform.*)
X4—Engagement of civil society (linked with Obj. A—Soc. Inclusion)
(01) SIE1—Local community’s contribution to the results of the SI initiative	1.1	0.1–10	L
(02) SIE2—Actors’ motivation for engaging in the SI initiative	0	0–100	L
(03) SIE3—Participation of actors in network meetings	0	0–100	L
(04) SIE4—Civic society engagement in the SI network	0	0–100	L
X2—Response to societal challenges (linked with Obj. D—Education in nutrition and envir.)
(05) SIS1—Capacity of the SI to tackle multiple European societal challenges	58.0	0–100	M
(06) SIS2—Improvement in European societal challenges	45.5	0–100	M
X6.2—Perceived Opportunities and Threats (linked with Obj. B—Enabling conditions)
(07) Ba1—Balance between opportunities and threats	1.6	0–12	L
(08) Ba2—Supportive policies’ role on sustaining the results of the SI initiative	7.0	0–10	H
(09) Ba3—Contribution of the Social Innovation initiative to local governance	54.0	0–100	M
X7.1—Social Innovation Idea (linked with Obj. A—Soc. Inclusion)
(10) Ca1—Attractiveness of the SI idea	100	0–100	H
(11) Ca2—Innovativeness of the SI idea in the territory	9	1–10	H
X7.4—Capabilities (linked with Obj. B—Enabling conditions)
(12) Cd2—Previous experience of actors who contributed to the SI process	66.7	0–100	M
X7.5—Endogenous versus exogenous drivers of the SI process (linked withObj. B—Enabling conditions)
(13) Da1—Role of newcomers in the SI process	50.0	0–100	M
(14) Da2—SI actors’ perception of external helpers’ contribution to the results	4.7	1–10	M
X8.1—New networks (linked with Obj. A—Soc. Inclusion)
(15) Ea1—Attendance level at SI process meetings	1.3	1–4	L
(16) Ea2—Balance between public and private members of the SI network	25.0	0–100	L
(17) Ea3—Contribution of the members of the SI network to the results	8.15	1–10	H
(18) Ea5—Female inclusion in the SI network	15.0	0–100	L
(19) Ea6—Young people’s participation in the Social Innovation network	85.0	0–100	H
(20) Ea7—Education level within the SI network	20.0	0–100	L
(21) Ea10—New relationships within the SI network	51.0	0–100	M
(22) Ea12—Level of internal trust in the SI network	8.0	1–10	H
(23) Ea13—Level of representativeness of the actors involved in the SI network	5.0	1–10	M
X8.2—New attitudes (linked with Obj. B—Enabling conditions)
(24) Eb1—Level of pro-action of transformers during the SI process	50.0	0–100	M
(25) Eb2—Perception of the actors of their level of empowerment	7.1	1–10	H
X7.2—Leadership (linked with Obj. C—Goods and Services)
(26) Cb2—Project partners’ contribution to the results of the SI initiative	1.2	1–10	L
X10.1—Feedback loops and multiplier effects (linked with Obj. B—Enabling conditions)
(27) Ha1—Likelihood of feedback loops thanks to dissemination activities	87.5	0–100	H
(28) Ha2—Upscaling of the SI initiative	50.0	0–100	M
(29) Ha3—Out-scaling of the SI initiative	50.0	0–100	M
(30) Ha4—Replication of the SI initiativ	75.0	0–100	H
X11.1—Relevance of the SI process (linked with Obj. B—Enabling conditions)
(31) R1—Shared individual and collective needs within the whole SI network	50.0	0–100	M
(32) R2—Shared vision regarding collective needs within the whole SI network	100.0	0–100	H
X11.3—Relevance of the Social Innovation Initiative (linked with Obj. C—Goods and Services)
(33) R5—Level of satisfaction of the SI initiative actors with territorial needs	7.3	1–10	H
(34) R6—Shared needs within the SI initiative	83.3	0–100	H

Notes: *) Performance expressed by the indicator: L = low, M = medium, H = high (best tertile). Source: Own elaborations.

.

The first evidence that emerged from results, is that 34 indicators have been evaluated and the majority of them are satisfactory (i.e., 70% belong to the high and medium category), while only one 30% are poorly performant. A critical issue of this evaluation is to understand the mechanism underlying the performance of the different criteria. We will proceed with the interpretation of results for each composite indicator, explained in terms of the basic elements characterizing the SI according to the theoretical framework, as described in Table 4.

X4—Engagement of civil society: all the indicators are very poor, probably due to the fact that the project is still at the early stage of development, and the main role is still played by some local organizations (i.e., the non-profit organization, the training agency, and the municipality), while the local community has not been extensively involved;

X2—Response to societal challenges: the initiative has been oriented at the training of NEETs towards organic agriculture and, therefore, emphasizing the relevance of environmental sustainability; the score is quite satisfying, as the design of the project is adequate to respond to some relevant social challenges.

X6.2—Perceived Opportunities and Threats: the scores of the three indicators are heterogeneous, due to contrasting forces. On the one hand, the strong cohesion among the institutional actors, endorsing and funding the initiative, provides a positive perception towards the opportunities of developing an innovative type of agriculture, capable of redeeming NEETs and promoting organic farming. On the other hand, there is some sort of discomfort regarding the financial sustainability of the SI process, in the mid and long terms, especially in case other actors (e.g., farms willing to recruit the trained NEETs) may not collaborate;

X7.1—Social Innovation Idea: the indicators of this group reveal that the idea behind the project is original and well suited to the perceived context, as it is able to valorise some local strengths (i.e., local consumers are willing to buy high-quality products), exploiting the capacity of the primary sector to provide high employability.

X7.4—Capabilities: all actors forming the agency already have some experience in activities which were closely related to the provision of social services; however, the current experience was the first multi-actor initiative they conceived and planned together;

X7.5—Endogenous versus exogenous drivers of the SI process: the performance of these indicators is moderate, demonstrating that, despite the need to valorise the role of external members, it may contrast with the internal cohesion of the core group (i.e., agency);

X8.1—New networks: the performance in this group of indicators is polarized as low values in some indicators alternate with high performance in other indicators. This is probably due to the fact that the preparatory actions of this project have been effectively targeted at the involvement of young people, who are the direct beneficiaries of the intervention. However, less attention has been paid towards the balance of the partnership and the female inclusion, as the action mainly engaged male farmers.

X8.2—New attitude: the initiative seems effective in favoring the collaboration of actors actively involved in the training of NEETs (the so-called transformers), who exhibited a pro-active attitude and foresee some further development of the SI process;

X7.2—Leadership: the value of the indicator is moderate, which demonstrates that there is not a prevalence of any actor on others. This can be positive, as all actors may act responsibly and aware of their role within the project, and will exhibit a more pro-active role;

X10.1—Feedback loops and multiplier effects: the project is able of providing high symbolic value as it addresses one of the most relevant social challenges, that is to provide job opportunities for NEETs, and this reflects on high performance for this composite indicator which assesses the replication potential of the idea.

X11.1—Relevance of the SI process: the performance of these indicators are crucial, as they demonstrate the adequacy of the SI in meeting the social needs, which is one of the main pre-requisite for the activation of a SI process, and its future development;

X11.3—Relevance of the Social Innovation Initiative: the good performance of these indicators are consistent with the previous one. In this case, the evaluation refers more specifically to the NEETs training initiative.

These figures reveal the need for some improvement, which could be achieved if more efforts are focused on about one-third of aspects affecting the indicators with lower scores. In order to identify possible corrective actions, the following strategy may be considered: (i) to assign the higher priority to actions addressed at critical issues, identified by poor indicator performance; (ii) consolidate and reinforce the activities underlying the indicators falling in the medium category; (iii) keeping a close monitoring of the well performing indicators, by considering them as strengths of the SI initiative.

Certainly, most of the actions must be planned and implemented by the agents directly involved in the SI. However, external partners and policymakers may also play an important role, in the endorsement of the SI initiative and the activation of parallel initiatives exerting a synergistic effect.

In order to meet the evaluation needs of policymakers, we rearrange the indicators, according to the cross-reference between the composite indicators and the policy objectives presented in Table 1. Accordingly, we are now able to evaluate the SI initiative aimed at NEETs employability, in terms of the objectives of social agriculture (

Table 5. Summary of results.

	Indicators “Low”	Indicators “Medium”	Indicators “High”	Tot. Counts of Indicators
Obj.A—Social Inclusion	8 (54%) SIE1, SIE2, SIE3, SIE4, Ea1, Ea2, Ea5, Ea7	2 (13%) Ea10, Ea13,	5 (33%) Ca1, Ca2, Ea3, Ea6, Ea12	15 (100%)
Obj.B—Enabling conditions to social inclusion	1 (7%) Ba1	8 (57%) Ba2, Cd2, Da1, Da2, Eb1, Ha2, Ha3, R2	5 (36%) Ba3, Eb1, Ha1, Ha4, R1	14 (100%)
Obj.C—Goods and Services	1 (33%) Cb2		2 (67%) R5, R6	3 (100%)
Obj.D—Education in Nutrition and Environment		2 (100%) SIS1, SIS2		2 (100%)
Tot.	10 (30%)	12 (35%)	12 (35%)	34 (100%)

Source: Own elaborations.

).

The distribution of the indicators shows that the SI initiative was mainly evaluated with regards to its contribution to the two first objectives of social agriculture, such as social inclusion and the enabling conditions to favor social inclusion. On the contrary, only a few indicators were suitable to evaluate the contribution to the other two objectives, regarding the creation of goods and services and the education of society in nutrition and environment. This distribution can be explained by the fact that the SeminaMenti project is still at its early stage of development, which is mainly focused on the training of NEETs in organic farming; afterwards, they are expected to be recruited by organic farms and continue to collaborate with the SI agency to promote organic food to the local community (e.g. schools, open market, specialized shops).

Regarding the performance of the first objectives, despite the overall performance seems quite satisfactory, some weaknesses emerged regarding the gender balance (NEETs were only male), and the active involvement of local people in the initiative. This can be explained by the specific cultural context of Southern Italy, where agriculture has always been considered a male domain while, on the contrary, women have played a main role in food preparation, and in restaurants. Probably, the SI agency has not been sufficiently effective in advertising that the training activities for NEETs were based on the integration of cultivation of horticultural crops with post-harvest operations and marketing; if this were the case, some women would have been interested in the project.

5. Conclusions

This paper focuses on the importance of an evaluation methodology aimed at initiatives for the promotion of social agriculture, based on SI. The main feature of SI is that the most relevant outcomes are represented by intangible assets, such as the strengthening of social capital, the reinforcement of the sense of belonging to the local community, and a more pro-active attitude towards new opportunities offered by the market (e.g., provision of organic products to the local market), and the social services provided by the agricultural sector (e.g., the social inclusion of NEETs and their employability). The complexity involved in SI initiatives, arising from the multi-agent collaboration (e.g., non-profit organizations, municipalities, private firms, training institutions), and the multi-faceted nature of benefits which may be created by these activities (most of which are immaterial), poses a relevant challenge for the evaluation of the outcomes and, consequently, for grasping the whole relevance in terms of impacts for the whole society. It is worth mentioning that SI initiatives are highly dependent on intangible assets, typically included in the concept of social capital, as shared values, perceptions, expectations, trust, reciprocity [24,25,26,27], which deserves some form of representation (e.g., by means of evaluation), in order to exert a positive effect on the internal cohesion and external engagement (i.e., civil society).

Despite the difficulty of performing an evaluation exercise at the early stages of development of a SI initiative, we claim that even by collecting some (unavoidably) approximate qualitative data, it is possible to identify some critical aspects of the project, which may provide some valuable insights to project managers, policymakers and the local community, either for the steering and the improvement of the project, but also for the strengthening of mutual relationships among the participants and the civil society.

The difficulty of disentangling the multiple benefits of SI initiatives in social agriculture is particularly relevant for policymakers, who are responsible for policy measures that must comply with the “best value for money” principle. In this paper, we proposed a methodology based on an evaluation grid, through that we selected a set of indicators which have already been applied for the evaluation of SI initiatives. In this way, we have been able to perform a consistent evaluation exercise referring to a SI aimed at training and favoring the employability of NEETs, within the frame of an agricultural policy oriented at supporting social agriculture.

We have been able to demonstrate that the SeminaMenti project performed quite well, mainly in terms of two objectives of the national policy of social agriculture, such as social inclusion and the social inclusion and the enhancement of enabling conditions for social inclusion. On the contrary, only a few indicators for SI evaluation were suitable for the evaluation of the other two objectives, such as the provision of goods and services and the improvements in education in nutrition and environmental awareness. This was mainly due to the fact that the SeminaMenti project is still at an early stage of development, and it will take a long time in order to produce some relevant results in these domains.

The evaluation experience presented in this contribution demonstrates that, despite the valuable efforts made by civil society in promoting some virtuous initiatives of social inclusion (e.g., the enhancement of the employability of NEETs), the role of public funding is indispensable, in order to support these initiatives in the short and long rum, and to replicate them. For this reason, an evaluation exercise is needed in order to provide a robust analysis of benefits arising from a specific initiative. This is a new domain of investigation, as the available evaluation frameworks are suitable for single project assessment and for the single institution (private, public, or non-profit), while only a few approaches and experiences are available to evaluate multi-actor and multi-goal interventions (or process), typical of holistic and bottom-up frameworks.

The adaptation of the methodological framework developed by the SIMRA consortium, to a social agricultural project aimed at fostering the employability of NEETs, needs some further efforts, especially with regards to the robustness of the selection grid, formed by the cross-reference between national policy and criteria of relevance used project selection in public calls. Though we could have applied some consolidated methodologies to obtain a robust selection grid of indicators, we were not able to interact with policymakers and experts who are fully familiar with concepts such as social agriculture, social innovation, selection criteria of public calls, as they refer to a domain which is still at a niche level. Therefore, the limitation of this study is that it is still valid as a conceptual framework at an academic level, while some time will still be required until its application at an operational level.