Spatial Targeting of Agricultural Support Measures: Indicator-Based Assessment of Coverages and Leakages

Abstract

Agricultural support programs distribute payments to farms based on a diverse set of policy objectives. Adequate targeting of this support to priority areas is key to efficient and effective policy. We evaluated the targeting strategy of a national-level program in Mexico that distributed support based on seven criteria that prioritized poor smallholder farming communities at high risk of cropland failure. We used a series of logistic models to assess the coverage and leakage rates of the program’s targeting strategy and found rates of about 80 and 20 percent, respectively. We also found significant differences between the targeting priorities specified in program rules and the observed distribution of support measures. In general, the program favored arid and semi-arid regions at high risk of soil erosion but neglected smallholder farms in high-poverty regions with elevated rates of cropland failure. Our findings highlight the continued lack of financial support for smallholder agriculture in Mexico, despite program rules and priority statements that stress the vulnerability of this sector. This study also illustrates the important role of spatial targeting in better aligning agricultural support payments with stated policy priorities. This alignment is often overlooked in ex-post assessment, but it is critical for improving targeting precision, equity, and overall policy effectiveness.

1. Introduction

Agricultural policies have expanded in recent decades beyond the traditional emphasis on crop productivity to address a range of socioeconomic, rural development, and environmental objectives [1]. Research into this policy diversification has focused on the Common Agricultural Policy of the European Union, where the term agri-environmental has come to describe a wide variety of policies, schemes, programs, and measures. Generally, findings show that better targeting of agri-environmental support leads to more effective and cost-efficient policy [2,3], so long as transaction costs are low [4] or are offset by enhanced provision of services [5]. Studies of targeting effectiveness tend to examine single support measures [6] and provide insights into specific policy actions and outcomes (i.e., levels of farmer participation, promotion of green farming practices, provision of ecosystem services, rural development, and equity) [7,8,9,10,11,12]. Few studies focus on the initial distributional aspects of targeting and if the distribution of support actually reflects program rules and stated priorities. Evaluating this (mis)alignment is a critical though often overlooked part of assessing overall targeting effectiveness.

In theory, targeting decisions are based on carefully defined objectives and priority criteria [13,14] and quantitative indicators of those criteria [15,16,17]. In practice, poorly defined criteria and insufficient indicator data often result in ineffective targeting and suboptimal outcomes [18,19,20,21]. Collecting indicator data that reflect program criteria can be challenging due to the heterogeneity of data types and the complexities of monitoring multifunctional (agri-environmental) landscapes [22,23,24,25]. Data challenges can be severe in lower- and middle-income countries, which often lack the institutional resources to collect data and to implement, monitor, and evaluate targeting strategies [26,27,28,29,30].

Targeting leakages occur when gaps exist between stated coverage rules (priority criteria) and actual coverage (observed distribution of measures) [31]. In a statistical context, ineffective targeting can lead to two types of coverage error. Type I errors occur when priority areas are under-covered (errors of omission), and Type II errors occur when non-priority areas are covered (errors of inclusion or over-coverage) [32,33]. In a general policy context, most targeting studies seek ways to maximize coverage and minimize leakage through optimization modeling [34,35,36]. For example, in humanitarian and development contexts, optimization models help policymakers target limited support to reduce poverty in vulnerable areas experiencing violent conflict or famine [37,38]. In environmental contexts, targeting studies often explore how to maximize coverages and minimize leakages of conservation investments in protected area management [39]. In agricultural (agri-environmental) contexts, quantitative assessment of targeting coverages and leakages is less common [40], though improved quantitative understanding is key to optimizing distributional outcomes under budget constraints [41,42]. This is especially true in lower- and middle-income countries, where agri-environmental policies are relatively new [43] and where few critical analyses have been conducted [1].

To address this research gap, the objective of this study was to quantify and assess the targeting effectiveness, coverage, and leakage rates of one of Mexico’s largest agricultural (agri-environmental) programs, the Comprehensive Program for Sustainable Agriculture and Productivity Growth on Vulnerable Lands. We used a series of logistic models, receiver operator characteristic curves, and odds ratios to examine the degree to which the program distributed agri-environmental support measures (AEM) based on official program rules and targeting priorities. We examined targeting effectiveness on two levels. First, we assessed the overall coverage and leakage rates based on whole model effects, whereby priority criteria were used to explain the observed distribution of AEM. Second, we explored the partial effects (simple and adjusted) of each criterion as a determinant of this distribution. We used these results to identify criteria that played significant roles in AEM distribution and contrast these with criteria that played less significant roles. We discuss our findings in the context of Mexican agricultural policy, support for smallholder farms, and the role of spatial targeting in agri-environmental policy more broadly.

2. Materials and Methods

2.1. Study Background: Agri-Environmental Policy in Mexico

Agricultural policy in Mexico is characterized by an extensive rural support system managed by the Ministry of Agriculture and Rural Development (SADER) (formerly SAGARPA). SADER began merging agricultural and rural development policy in the early to mid-20th century, but greater attention to rural development and poverty came with passage of the North American Free Trade Agreement (NAFTA) in 1994 [44]. These early support programs aimed to boost crop productivity as the primary means of addressing poverty, a policy emphasis that largely remained through the first two decades of the 21st century [45].

Broad integration of environmental concerns into SADER policy came after passage of the Sustainable Rural Development Law in 2001, which required agricultural policy to explicitly address issues of environmental sustainability and equity [46,47]. Policies began addressing these concerns in rural extension services, micro-credit lending, and an emerging suite of agri-environmental programs [48,49,50]. Although the emphasis on boosting productivity remained, SADER took an increasingly integrative approach to policy design, providing payments for conservation-based approaches to agricultural land management [51].

From 2003 to 2007, SADER implemented one of its largest initiatives, the Comprehensive Program for Sustainable Agriculture and Productivity Growth on Vulnerable Lands (hereafter PIASRE). The program had several objectives: (i) increase productivity on small farms in regions with high levels of cropland failure; (ii) decrease vulnerability in regions susceptible to soil erosion and climatic stress; (iii) repair and enhance rainfed agroecosystems; and (iv) provide support in rural areas with high or very high levels of human marginality [52,53]. The program funded AEM across the country in the form of payments for small-scale soil and water conservation projects. A total of 552 of Mexico’s 2455 municipalities received support, at a total cost of over two billion pesos (~200 million US dollars) [54]. The program distributed support based on seven priority selection criteria, which SADER issued through federal directives and program-level Rules of Operation [52,53].

An evaluation of the program by the Food and Agriculture Organization of the United Nations (FAO) found that the strategy to distribute AEM across the country was inadequate. Specifically, FAO indicated that the program did not use a method to quantify and delimit target areas and populations owing to the lack of sufficient data on priority criteria and their indicators [52]. The program also did not provide a value ordering or ranking of the criteria, which further limited FAO assessment. Financial data on the program were unavailable, and the potential role of budgetary factors on the targeting strategy was not assessed. The report recommended that future programs design targeting strategies based, in part, on Mexico’s National Agricultural Census (CAP) database, which included data aligned with several program criteria [55]. The report did not provide evidence of poor targeting beyond finding that inadequate data and methods had been used. In this study, we developed quantitative indicators for the seven criteria, assessed their roles as determinants of AEM distribution, and evaluated the FAO finding.

2.2. Data Sources and Variables

Agri-environmental measures. We first downloaded the complete record of official registries for the PIASRE program. These data were published online through its System of Transparency (SITRAN), an open-access database available from ~2006 to ~2013 [54]. From the registries, we identified each AEM (N = 12,416) and the municipality where it was implemented, classifying all municipalities as either receiving support (AEM[Y]) or not (AEM[N]). The municipality was the unit of observation for the study, which was the smallest program administrative unit common to all variables. The program rules specified the following targets for AEM.

Failed cropland. Rainfed farming regions prone to cropland failure (FC). Although little guidance on potential data sources was provided, rules specified that FC should be calculated as the difference between the total rainfed surface area sown and harvested (sown [ha] − harvested [ha] = FC), following established accounting practice in Mexico [53]. Using this formula, we calculated the total FC for each municipality from 2002 to 2006 using data from the Agriculture and Fisheries Service (SIAP), a branch of SADER that provides official yearly totals of municipal-level productivity across the country [56].

Marginality. Regions with high and very high levels of marginality [52,57]. We classified the marginality level of each municipality using the 2005 Marginality Index, a broadly used indicator of poverty and deprivation developed by the Mexican government. The index includes data on ethnic identity, income level, educational access, and other needs-based components unique to Mexico [58,59,60]. Each composite index score is comprised of 10 sociocultural and economic indicators assessed by the National Population Council. Standardized scores are used to classify each municipality according to five levels of marginality (very low, low, medium, high, and very high) [61].

Rural areas. Rules did not specify how rural areas were to be determined for targeting [52]. We used the National Institute for Federalism and Municipal Development 2005 dataset to classify each municipality as either “rural” or “other” based on six categories of population distribution, from rural to metropolis [62].

Farmland size. Farms <40 ha in size [53]. We derived farmland size indicators using the National Agricultural Census (CAP) dataset (2008) following the earlier recommendations [52]. Matching individual census plots with farm-level data on AEM recipients was not possible due to data misalignment. Therefore, we calculated the mean farmland area (MFA) of each municipality by dividing the total cultivated area by the total number of farms following previous studies [63,64]. We then classified each municipality as having MFA greater than or less than 40 ha.

Rainfed surface area. Regions with large rainfed surface areas [52]. We again used the CAP dataset to calculate the total surface area per municipality under rainfed cultivation [52,55].

Climate region. Regions repeatedly impacted by adverse climate phenomena. Rules specified that climate regions should be determined based on the CONAZA and UACH (2003) study, which classified regions using a series of eight-part modified Thornthwaite projections [65]. Using this classification, we assigned the centroid of each municipality to one of the eight corresponding climate zones.

Erosion risk. Regions at high risk of soil erosion. Rules specified that risk determination should follow the national soil survey and erosion risk classifications of the Secretary of the Environment and Natural Resources and the Colegio de Postgraduados [66]. Using this classification, we assigned the centroid of each municipality to one of four corresponding erosion risk zones from “negligible risk” to “high risk”.

2.3. Statistical Approach

We use three logistic regression models to examine the seven targeting criteria as determinants of AEM distribution. Logistic regression is widely used in land-use classification and agri-environmental policy studies to model relationships between one or more independent variables and one or more binary dependent variables [67,68,69]. Here, we model the seven priority criteria as independent variables and AEM distribution as the binary outcome, receiving AEM (AEM[Y] = 1) or not (AEM[N] = 0). The logistic model has the form

$$\log \left(\frac{P}{1-P}\right)=b_0+b_1{X}_1+b_2{X}_2+\cdots +b_n{X}_n$$

(1)

where P is the response probability that the expected outcome of AEM(Y) or AEM(N) is present; X₁ through X_n are independent variables; and b₀ though b_n are the coefficients. In logistic regression the coefficients for continuous variables show the change in the expected log odds per one-unit increase in X, holding other variables constant. To facilitate interpretation, coefficients are often expressed as odds ratios, which are the exponentiated (inverse or undone) versions of the logistic coefficients. Typically, the odds of the target outcome are divided by those of the non-target outcome in the form

$$\mathrm{Odds}\mathrm{Ratio}=\frac{odd{s}_{AEM\left(Y\right)}}{odd{s}_{AEM\left(N\right)}}=\frac{p_{AEM\left(Y\right)}/\left(1-p_{AEM\left(Y\right)}\right)}{p_{AEM\left(N\right)}/\left(1-p_{AEM\left(N\right)}\right)}$$

(2)

The odds ratios for each variable shows the change in the relative odds of the target outcome occurring for every one-unit increase in X. When odds ratio = 1, there is no effect on the odds of AEM(Y); when odds ratio >1, the effect on the odds of AEM(Y) is greater than 1; and when odds ratio <1, the effect on the odds of AEM(Y) is less than 1. For categorical variables, the odds are calculated in relation to a reference category using either dummy or effects coding, rather than in relation to unit increases.

Model 1: Simple logistic (unadjusted). We first used seven simple logistic regression models to estimate the singular effects of each priority criterion on the likelihood of AEM(Y). The univariate models served to establish baseline relationships between each criterion and AEM distribution and as a reference point from which to quantify the adjustment effects of the other six criteria in Model 2. Key assumptions of simple logistic regression were met. Basic linear relationships between independent variables and the log odds were identified using scatterplots for the two continuous variables and correspondence analyses for the five categorical variables.

Model 2: Multiple logistic (adjusted). A multiple logistic regression model was used to examine the combined effects of the seven priority criteria on AEM distribution. Key assumptions of multiple logistic regression were met. Independence of parameters was established with all variable inflation factors ≤2.34. Absence of collinearity was established based on diagonal values in a covariance matrix (all between −0.163 and 0.139), and the standard errors of parameter coefficients were low (i.e., ≤0.43). However, a test for spatial autocorrelation (SAC) was performed on the residuals of the dependent variable yielding a Moran’s Index of 0.033, a Z-score of 13.78, and a p-value of 0.001, indicating the presence of SAC.

Model 3: Multiple logistic (adjusted for SAC). Model 3 was designed to control for SAC. Methods of controlling SAC in logistic regression include: developing autocovariate models in which a spatial control variable is incorporated based on the weighted average distances between neighbors of the same unit response variable [70] or SAC residuals [71]; using random sampling to minimize the effects of SAC [72]; incorporating geographic coordinates as continuous predictor variables [73]; and incorporating other spatially explicit lag variables that control for locational effects [74].

To control for SAC, we included a physiographic region variable in Model 3 by assigning the centroid of each municipality to one of nine distinct regions based on the National Institute of Geography and Statistics classification [75]. Tests for SAC in Model 3 yielded a Moran’s Index of −0.001, a Z-score of −0.31, and a p-value of 0.755, indicating the absence of SAC.

2.4. Assessing Targeting: ROC Curves and Odds Ratios

Receiver operator characteristic (ROC) curves illustrate the ability of a diagnostic test to distinguish between two conditions [33]. ROC curves have been used in humanitarian and development contexts to test the ability of a model to distinguish poverty thresholds or the accuracy of existing poverty indicators [76]. In these contexts, ROC curves often examine the probability that a model will correctly classify a poor person as “poor” (true positive), a measure of model “sensitivity” usually plotted on the y-axis. The same curves also illustrate the probability a model will correctly classify a non-poor person as “non-poor” (true negative). This is known as model “specificity” and is usually plotted on the x-axis as 1 - specificity. The ROC curve itself represents all possible combinations of sensitivity and 1 - specificity that result from different probability threshold cutoff points [41]. The area under the curve (AUC) represents the overall ability of the model to correctly distinguish between the binary outcomes (accuracy) and is often used as a measure of model fit [77]. For example, an AUC of 0.70 correctly predicts outcomes 70 percent of the time. Often, a diagonal line from the point of origin (0, 0) to (1, 1) is drawn to represent a model with no discrimination (chance), with an AUC of 0.50. The difference between the two (AUC = 0.20) represents a model accuracy greater than random chance. Because the point (0, 1) represents perfect specificity and sensitivity, the point on the ROC curve closest to (0, 1) represents the optimal probability cutoff point where sensitivity and specificity are maximized. This point can be determined using the Youden Index, which calculates the maximum vertical distance between the diagonal (AUC = 0.50) and the ROC curve for each single decision threshold [78].

In the context of poverty targeting, where “poor” = 1 and “non-poor” = 0, model specificity represents targeting coverage and 1—sensitivity represents targeting leakage. The Youden Index in this case identifies the cutoff point where coverage is maximized and leakage is minimized, assuming the costs of over-coverage (false positive) and under-coverage (false negative) are equal. Verme and Gigliarano (2019) provide extended explanation of the use of ROC curves for this purpose as a diagnostic tool for optimizing targeting and increasing overall distributive efficiency under different scenarios. Here we used a modified version of this method, which is to our knowledge the first use of ROC curves in assessing the targeting effectiveness of agri-environmental policy.

For each of the three models, we first assessed overall model explanatory power with AUC and pseudo-R-squared (McFadden and Nagelkerke) measures, lowest information criteria (Akaike and Bayesian), and likelihood ratio tests. We then developed the ROC curve for each of the models by plotting all possible combinations of targeting leakages on the horizontal axis and targeting coverages on the vertical axis, based on all combinations of predicted and observed coverages, respectively. These combinations were plotted at each possible threshold cutoff point for receiving AEM (Y = 1; N = 0) as determined by the maximum likelihood function of the logistic curve. We identified the cutoff point (Youden Index) as the optimization point between coverage and leakage rates assuming the distributive costs of model misclassification due to false negative and false positive were equal. We then illustrated the two error rates (Type I and Type II) and coverage rates in 2 × 2 confusion matrices based on the predicted and observed outcomes at the cutoff value. We developed matrices only for Models 2 and 3 as they provided the greatest explanatory power. Using the matrices, overall coverage rates were identified based on true positive and true negative results. Under-coverage (exclusion or Type I error) and over-coverage (inclusion or Type II error) rates were based on false negative and false positive results, respectively.

We used odds ratios to examine the partial effects of each targeting criterion on AEM distribution in Models 2 and 3. We used forest plots to illustrate the odds ratios for Models 2 and 3 and the adjustment effects of controlling for SAC (Model 3). All spatial analyses were performed with ArcGIS Pro (ESRI) and other statistics with JMP Pro 14.2.0 (SAS Institute, Cary, NC, USA).

3. Results

3.1. Descriptive Statistics of Variables

About 23 percent of municipalities (552/2455) received AEM through the PIASRE program (

Table 1. Targeting indicators for municipalities with and without receiving support measures through Mexico’s PIASRE program.

	Receipt of Agri-Environmental Measures among Mexican Municipalities
Variable (Independent)	Yes (n = 552)		No (n = 1903)		Total (n = 2455)
	Mean	SD	Mean	SD	Mean	SD
Failed cropland (10 k ha)	1.646	2.148	0.150	0.400	0.252	0.733
Rainfed cropland (10 k ha)	0.603	1.296	0.815	1.578	1.002	1.757
	n	%	n	%	n	%
Marginalization level	552	100	1903	100	2455	100
very low	74	13	205	11	279	11
low	130	24	294	15	424	17
medium	127	23	374	20	501	20
high	175	32	711	37	886	36
very high	46	8	319	17	365	15
Mean farm area	552	100	1903	100	2455	100
MFA < 40 ha	461	84	1799	95	2260	92
MFA > 40 ha	91	16	104	5	195	8
Rural classification	552	100	1903	100	2455	100
rural	364	66	1062	56	1426	58
other	188	34	841	44	1029	42
Climate region	552	100	1903	100	2455	100
Perhumid (A)	19	3	315	17	334	14
Humid (B)	23	4	310	16	333	14
Moist subhumid (C2)	23	4	223	12	246	10
Dry subhumid (C1)	140	25	564	30	704	29
Semiarid light (D3)	114	21	274	14	388	16
Semiarid mod. (D2)	124	22	134	7	258	11
Semiarid dry (D1)	93	17	71	4	164	7
Arid (E)	16	3	12	1	28	1
Erosion risk	552	100	1903	100	2455	100
negligible	6	1	181	10	187	8
low	195	35	931	49	1126	46
moderate	299	54	425	22	724	29
high	52	9	366	19	418	17
Physiographic region	552	100	1903	100	2455	100
Southern mts.	91	16	803	42	894	36
Yucatán-LGCP	14	3	217	11	231	9
Volcanic axis	95	17	595	31	690	28
Central tablelands	88	16	17	1	105	4
Gulf coastal plains	46	8	74	4	120	5
Pacific coast plains	25	5	22	1	47	2
Eastern mts.	61	11	132	7	193	8
Western mts.	124	22	42	2	166	7
Baja peninsula	8	1	1	0	9	0

). We compared priority criteria indicator values between these municipalities (AEM[Y]) and those that did not receive support (AEM[N]). The total mean area of failed cropland was 11 times larger for AEM(Y) municipalities (16,460 ha) than for AEM(N) (1500 ha), but the total mean areas of rainfed cropland differed little (26 percent). Marginalization levels between AEM(Y) and AEM(N) municipalities showed that as a percent of the total, municipalities with very low, low, and medium marginality were two, nine, and three percent higher in AEM(Y) than AEM(N), respectively. Municipalities with high or very high marginality were five and nine percent lower in AEM(Y) than AEM(N). Among AEM(Y) municipalities, 84 percent had mean farm areas (MFA) less than 40 ha, while among AEM(N) municipalities, 95 percent had MFA less than 40 ha. Rural municipalities comprised 64 percent of AEM(Y) and 56 percent of AEM(N). Regarding climate region, each of the four most arid zones (E, D1, D2, D3) comprised a greater percent of AEM(Y) than AEM(N) municipalities, whereas each of the four most humid zones (A, B, C2, C1) comprised a smaller percent of AEM(Y) than AEM(N) municipalities. Almost 90 percent of AEM(Y) municipalities exhibited low or moderate risk of soil erosion, with negligible and high erosion risk comprising only one and nine percent, respectively. For AEM(N) municipalities, low and moderate erosion risk was almost 20 percent lower than in AEM(Y) municipalities, while negligible and high erosion risk were 9 and 10 percent higher. For physiographic region, the highest percent of AEM(Y) was in the Western mountains regions (22 percent), while for AEM(N), the highest percent of municipalities was in the Southern Mountains region (42 percent).

3.2. Model Results

3.2.1. Targeting Coverage and Leakage

Each priority criterion’s main effects (unadjusted) on AEM distribution in Model 1 were statistically significant (α = 0.05) (

Table 2. Fit statistics for simple (Model 1) and multiple (Models 2 and 3) logistic regression models.

	AUC	R2 (M)	R2 (N)	LRT	p-Value	AICc	BIC
Model 3 (multiple)	0.87	0.33	0.45	864	<0.0001	1808	1964
Model 2 (multiple)	0.83	0.26	0.37	673	<0.0001	1982	2092
Model 1 (simple)
Climate region (CR)	0.75	0.14	0.21	359	<0.0001	2274	2321
Erosion risk (ER)	0.69	0.09	0.14	230	<0.0001	2395	2418
Rainfed cropland (RC)	0.69	0.03	0.05	83	<0.0001	2538	2550
Failed cropland (FC)	0.68	0.06	0.10	160	<0.0001	2461	2473
Marginalization (MI)	0.59	0.02	0.03	47	<0.0001	2580	2609
Mean farm area (MFA)	0.56	0.02	0.04	61	<0.0001	2560	2572
Rural classification (R)	0.55	0.01	0.01	18	<0.0001	2603	2614

). Climate region and erosion risk had larger whole effects than other criteria, the highest likelihood ratio tests (LRTs), highest pseudo-R-squared (M and N), and lowest information criteria scores (AICc and BIC). Climate region and erosion risk also produced the highest singular AUC values, correctly explaining AEM distribution 75 and 69 percent of the time.

Overall, Models 2 and 3 explained AEM distribution about 83 and 87 percent of the time, respectively, which indicates good to excellent discriminatory capacity [77]. The controls imposed on Model 3 improved the success rate from Model 2 by about three percent (AUC = 0.866–0.834). Pseudo-r-squared measures were higher in Model 3 (McFadden = 0.33, Nagelkerke = 0.45) than in Model 2 (McFadden = 0.26, Nagelkerke = 0.37); likelihood ratio tests also were higher (Model 3 = 864, Model 2 = 673); and Model 3 had lower AICc (Model 3 = 1808, Model 2 = 1964) and BIC (Model 3 = 1982, Model 2 = 2092) scores (Table 2).

The probability threshold cutoff points for Models 2 and 3 as determined by the Youden Index were 0.201 and 0.198, respectively. At these optimization points, Models 2 and 3 correctly explained AEM(Y) about 77 and 80 percent of the time, respectively (Figure 1). The under-coverage rates of Model 2 and Model 3 were 23 and 20 percent, respectively, while over-coverage rates were 24 and 19 percent, respectively (

Table 3. Coverage and leakage rates for Models 2 and 3. Bold values refer to coverage rates, bold italicized values to over-coverage rates, and italicized values to under-coverage rates.

			Observed AEM
	Model		No	Yes
Predicted	2	No	75.72%	23.01%
AEM		Yes	24.28%	76.99%
	3	No	81.14%	20.29%
		Yes	18.86%	79.71%

).

The overall coverage rate of Model 3 was 79–80 percent, and the leakage rate was 19–20 percent. The targeting errors of Model 3 showed AEM were misallocated to non-priority municipalities 19 percent of the time (inclusion error), while priority municipalities were excluded from receiving AEM 20 percent of the time (exclusion error).

3.2.2. Positive Odds of Receiving AEM per Priority Criterion

Within these overall measures of targeting effectiveness, the roles of each targeting criterion in determining AEM(Y) varied widely. In other words, the coverage–leakage rates above were determined only after the models assigned very different effects to each of the explanatory variables.

Failed cropland was a strong determinant of AEM(Y) in Model 2 but a weak determinant in Model 3. The odds of AEM(Y) increased 73 percent with every one-unit (10,000 ha) increase in failed cropland in Model 2 but fell to a statistically insignificant 15 percent once adjusted for SAC and physiographic region (Model 3). Rainfed cropland was a statistically significant determinant in both models. The odds of AEM(Y) increased by 19 and 20 percent with every one-unit increase in rainfed cropland in Models 2 and 3, respectively (Figure 2).

Marginalization level was negatively associated with AEM(Y) in Model 2. Compared to municipalities with very low marginalization, those with medium, high, and very high were 3, 20, and 31 percent less likely to receive AEM, respectively, although these effects were not statistically significant. The association reversed after the adjustments of Model 3, where the odds of AEM(Y) in municipalities with low, medium, high, and very high marginality were 22, 54, 91, and 70 percent higher, respectively. However, even in Model 3, only the high-marginality classification effects were statistically significant.

Mean farm area of municipalities had little effect on AEM distribution. Municipalities with mean farm areas less than 40 ha (target classification) were 17 percent less likely to receive AEM in Model 2. In Model 3 the effect was negligible and statistically insignificant. Contrasting with this, the rural municipality classification was a significant determinant of AEM(Y). Rural municipalities were 146 percent more likely to receive AEM than non-rural municipalities in Model 2 and 75 percent more likely in Model 3.

Climate region was a strong determinant of AEM distribution. In Model 2, the odds of AEM(Y) in the most arid regions (E, D1, and D2) were 35, 27, and 11 times higher, respectively, than in the perhumid region (A). These odds fell to 21, 20, and 9 times higher in Model 3. In humid regions the adjustment effects of Model 3 were smaller. Overall, the positive relationship between aridity and the odds of AEM(Y) was strong, statistically significant, and relatively consistent between Models 2 and 3.

Erosion risk also was a strong determinant of AEM distribution. Compared to municipalities with the lowest erosion risk (negligible), the odds of AEM(Y) in municipalities with low, moderate, and high risk were 18, 33, and 23 times higher, respectively, in Model 2. The positive (ordered) association between erosion risk and the odds of AEM(Y) was clearer in Model 3. Here, the odds of AEM(Y) in municipalities with low, moderate, and high risk increased sequentially at 21, 22, and 29 times higher, respectively.

Physiographic region, like climate and erosion risk, was a strong and statistically significant determinant of AEM distribution. Unlike the other variables, however, the different physiographic regions (PRs) did not reflect a logical value ordering based on program criteria. However, for illustrative purposes we ordered the PR levels in a general north-trending sequence. This showed that municipalities in the northern regions of the Eastern Mountains, Central Tablelands, and Baja Peninsula were about 17, 21, and 74 times more likely, respectively, to receive AEM than municipalities in the southern most region (Southern Mountains). Inclusion of PR satisfied model assumptions and controlled the effects of SAC, and moderately attenuated the effects of climate regions and erosion risk on AEM distribution. However, the largest attenuating effects of PR were on the relationship between marginality and AEM distribution.

Overall, the effects of climate zone, erosion risk, and physiographic region were the strongest determinants of AEM distribution in both models. The partial effects of climate region showed strong positive association between aridity and the odds of receiving AEM. The other priority criteria were less significant, despite being described in program rules as important determinants of AEM distribution (Figure 2).

4. Discussion

Our logistic models explained the distribution of agri-environmental measures in one of Mexico’s largest support programs about 83 to 87 percent of the time. The targeting strategy had coverage and leakage rates of about 80 and 20 percent, respectively. These findings suggest a relatively effective targeting strategy [41,77,79]. However, closer examination reveals significant differences in the whole and partial effects of the seven priority criteria on AEM distribution. Climate region and erosion risk indicators were far better determinants of AEM distribution than were a municipality’s mean farmland size, rainfed land area, failed cropland area, or marginalization level.

4.1. AEM Targeting Favors Arid Regions at Risk of Soil Erosion

The observed distribution of AEM reflects Mexico’s strong policy emphasis on addressing hydroclimatic vulnerability and the effects of desertification on agriculture [80]. Municipalities in arid and semi-arid regions at high risk of soil erosion were about 20 times more likely to receive AEM than wetter municipalities with little erosion risk. About 65 percent of Mexican territory is dry (hyper-arid, arid, semi-arid, and sub-humid) [81], and about 60 percent of agriculture is rainfed and thus susceptible to climatic variability [82]. This vulnerability of rainfed agriculture to drought has played a major role in institutional responses to cropland failure over the last several decades [83,84], which come in the form of emergency relief, subsidies, and agri-environmental support measures [48,49,50]. Unlike the differences we found among climate regions, we found no significant differences among the low, medium, and high erosion risk classifications in the likelihood of receiving AEM (Model 3). This suggests that the program did not target AEM based on erosion risk distinctions to the same degree it did based on climate region distinctions. In general, aridity was a stronger determinant than erosion risk.

While the PIASRE program successfully targeted AEM to dry regions with some erosion risk, the conceptual basis for prioritizing these factors over regions with high levels of failed cropland is unclear. Furthermore, the lack of specificity in program rules in how to define and constrain the climate region, erosion risk, and failed cropland criteria makes assessing these relationships problematic. Ultimately, this lack of clarity serves as a major impediment to any meaningful ex-post assessment of policy effectiveness beyond the immediate, indicator-based targeting strategy.

4.2. AEM Targeting Neglects Marginalized Smallholder Farms

A 2009 report by the World Bank characterized the PIASRE program as specifically targeted to the rural poor, although the reasoning behind the characterization is unclear [50,85]. Our assessment found weak evidence that the program prioritized poor rural areas for AEM distribution. Instead, we found a negative relationship between marginalization and the likelihood of receiving AEM in Model 2, and only a slightly higher likelihood of receiving AEM among municipalities with high levels of marginalization in Model 3.

The discrepancy between the above characterization and our findings has at least two possible explanations: definitional ambiguity and characterization based on the targeting rules, rather than the actual distribution of AEM. First, there are many different methods for defining and measuring poverty in Mexico, and as the report notes, methods often vary across policies [50]. In the current case, PIASRE program rules specify using marginalization level as the priority indicator, marginalization being a common metric of multidimensional poverty in Mexico [58,59,60]. Therefore, definitional ambiguity does not explain the different characterizations.

A more likely explanation is that the characterization was based on the program’s targeting rules [52,53,57] rather than on observed coverages. The report acknowledges that leakages typically prevent some targeted support from reaching poorer regions but does not specify the nature or scale of leakages in the PIASRE program [50]. Our study confirms that the program was targeted to the rural poor, but only at the level of program design. The observed distribution of AEM was minimally aligned with the poverty targeting priorities in program rules. That is, the targeting strategy effectively failed to prioritize Mexico’s poorest regions, resulting instead in targeting under-coverages and more general policy leakages. Other studies confirm significant leakages in Mexican agricultural [49] and development policy [79], though these leakages are seldom quantified. This is due, in part, to transparency issues and the lack of available program data [86].

The leakages identified in this study reflect a broader institutional neglect of smallholder agriculture in Mexico. Despite rhetoric, policies have long failed to provide support to small farms, tending instead to distribute support to larger and wealthier farms [87,88]. In Mexico, close associations exist between smallholder agriculture, rural poverty, lack of access to irrigation, degraded or marginal farmland, and agricultural vulnerability more generally [89,90,91]. Commonly classified as having fewer than five hectares, smallholder farms in Mexico comprise about 72 percent of all farms but hold only about 18 percent of all cultivated land [49]. PIASRE program rules specified that farms smaller than 40 ha should be prioritized for AEM—a range that would include very small, small, medium, and many large farms [63]. Yet we found no statistically significant relationship between mean farm area and AEM distribution—municipalities with mean farm areas smaller than 40 ha were no more likely to receive AEM than municipalities with areas greater than 40 ha. Although Mexican agricultural policy has expanded to address a broader range of socioeconomic and environmental concerns, our study highlights the persistent neglect of its poorest farming regions. This has occurred despite enhanced legal requirements and some outward improvements in transparency over the last several decades [49,86].

4.3. Targeting Gaps in Agri-Environmental Policy Implementation

Gaps between stated program rules and the observed distribution of support are largely attributable to the absence of value ordering among the targeting criteria. In place of an explicit value order, our study reveals the implicit ordering of priorities based on ex-post assessment of the observed distribution. Value ordering of priority criteria is key to policy effectiveness when multiple objectives are involved in targeting [4,19]. Conflicts can emerge among objectives, e.g., optimizing equity verses efficiency [92,93], which may require tradeoffs and careful analyses of interaction or spillover effects [6]. These effects can be highly variable in space and time. Recognition of this variability has driven calls for indicator-based approaches to targeting assessment and the selection of priority criteria [2,19,94]. In the case of the PIASRE program, similar indicator-based approaches to assessment could be used to provide the framework for a more empirically grounded value ordering of criteria.

It is beyond the scope of this study to explore whether the PIASRE program’s targeting strategy would have achieved beneficial results if properly implemented. A two-year experimental case study found that the program produced mixed benefits [95]. Support measures were targeted to address a specific soil and water conservation objective, although the incentive structure produced only short-term conservation benefits. In the medium to longer terms, support measure actions actually resulted in landscape degradation. In addition, the empirical results needed to make this assessment required a level of intensive and sustained field monitoring that is seldom practical over larger spatial scales and timeframes.

Such site-level experimental approaches constitute the greatest share of ex-post assessment of the environmental effects of AEM. The spatiotemporal limitations associated with these approaches often result in a general lack of understanding of targeting effects on larger-scale processes [9]. Additional challenges to assessing larger-scale processes include the high costs of monitoring and measurement and the lack of linearity and immediacy of AEM effects [96]. Indeed, misalignment between institutional and ecosystem timeframes is a complicating scale effect that is often encountered when working at the intersection of social and ecological systems [97]. The design and evaluation of multi-objective agri-environmental policy requires similarly synthetic approaches to socioenvironmental systems science [98]. As such, improved integration of agri-environmental policy approaches remains a central challenge to targeting design, implementation, and evaluation [92].

We cannot say with certainty why smallholders in poor regions prone to cropland failure failed to receive a greater share of support, as required by program rules. The program’s over-coverage of arid and semi-arid regions may reflect legitimate concerns or policy changes that remain unexplained. Furthermore, we recognize that budgetary factors may have played a role in the targeting strategy, though we were unable to assess this as these data were unavailable. We also did not investigate the willingness or unwillingness of farmers to participate as a determinant of AEM distribution though we recognize its potential relevance [7,8]. Previous research shows that support programs involving little actual change in farming practices often generate higher farmer participation [12]. One study shows the land management practices supported by the PIASRE program were already in widespread use in marginalized smallholder regions [99], which would suggest a higher willingness to participate. Further research into the specific drivers of the targeting leakages identified in this study is needed. The methods developed in this study also could be applied to examine targeting strategies in rural humanitarian and development contexts, helping policymakers better target support to reduce poverty and vulnerability, conflict, and famine [37,38].

The push for greater policy focus on results-based targeting of agri-environmental measures is growing in Europe [8,100,101]. In Mexico, results-based targeting is also discussed, but in the context of payments for hydrological services and biodiversity conservation [102,103]. Mexico has one of the largest payments for ecosystem services programs in the world, which the country manages through the Ministry of the Environment and Natural Resources (SEMARNAT) and the National Forestry Commission (CONAFOR) [104,105]. Although separate ministries, some integration of these efforts with SADER policy could help generate the results- and indicator-based assessments of AEM that are needed in Mexico [88,94]. In turn, findings from these combined efforts could better inform the selection and value ordering of priority criteria—a factor critical not only to the design of targeting strategies but also to their effective implementation.

5. Conclusions

We used logistic models, ROC curves, and odds ratios to assess the targeting coverages and leakages of Mexico’s PIASRE program, which distributed agricultural support measures to farms across the country based on seven priority criteria. Our results show that despite coverage and leakage rates of 80 and 20 percent, respectively, large differences exist in priority criteria effects on AEM distribution. While the program favored arid and semi-arid regions at risk of soil erosion, it neglected poor smallholder farming regions with high levels of cropland failure. These targeting gaps reveal that despite new legislation and some outward improvements in transparency, actual policy support for smallholder agriculture remains inadequate. Better structural alignment between the design and implementation of targeting strategies is needed. This alignment is a critical though often overlooked determinant of targeting precision and overall policy effectiveness.