http://orcid.org/0000-0002-7621-8319
Figures
Abstract
Supported employment is a treatment whereby those with severe mental illness (or other disabilities) receive aid searching for competitive employment and mental health (or other) treatments concurrently. The most popular implementation of supported employment is individual placement and support (IPS). We conducted meta-analytic analyses of the randomized controlled trials of IPS. We found that subjects in IPS, compared to usual treatment conditions, had better vocational outcomes (obtained any competitive employment: RR = 1.63, 95%CI = [1.46, 1.82]; job tenure: d = 0.55, 95%CI = [0.33, 0.79]; job length: d = 0.46, 95%CI = [0.35, 0.57]; income: d = 0.48, 95%CI = [0.36, 0.59]) Non-vocational outcomes estimates, while favoring IPS, included the null (quality of life: d = 0.30, 95%CI = [-0.07, 0.67]; global functioning: d = 0.09, 95%CI = [-0.09, 0.27]; mental health: d = 0.03, 95%CI = [-0.15, 0.21]). Analysis of the expected proportion of studies with a true effect on non-vocational outcomes with d>0.2 showed some reason to expect a possible improvement for quality of life for at least some settings (Prop = 0.57, 95%CI = [0.30, 0.84]).
Citation: Frederick DE, VanderWeele TJ (2019) Supported employment: Meta-analysis and review of randomized controlled trials of individual placement and support. PLoS ONE 14(2): e0212208. https://doi.org/10.1371/journal.pone.0212208
Editor: Maurizio Pompili, Sapienza University fo Rome, ITALY
Received: August 26, 2018; Accepted: January 29, 2019; Published: February 20, 2019
Copyright: © 2019 Frederick, VanderWeele. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In this article we present a series of meta-analyses of randomized controlled trials from the supported employment literature focused on the individual placement and support (IPS) treatment. We examine both vocational (e.g., competitive employment rates) and non-vocational (e.g., quality of life) outcomes. We hypothesized that IPS would lead to better vocational outcomes. We also hypothesized that IPS would lead to better non-vocational outcomes, as work may be a pathway to overall flourishing [1].
In our society, it is generally thought that working is better than not working. There is the obvious necessity for the vast majority of persons to work in order to sustain life in our current economic system. Beyond that, persons often report that they derive a great sense of meaning from their work. In his seminal book Working, Studs Terkel takes us into the lives of Americans and relates their stories and shows us just how much meaning people derive from and through their work [2].
Beyond narrative accounts, there is evidence that working may be associated with improved aspects of wellbeing such as mental health [3, 4]. How we go about and think about work (e.g., job crafting) may also impact the level of work engagement [5], which has been associated with increased levels of meaning [6]. Job crafting may also impact meaning via person-job fit [7].
There are also known associations between work, or more properly the lack of work, and negative outcomes. The most well-cited is the link between unemployment and suicide [8]. In that study, Nordt et al. estimated that one in five suicides might be attributable to unemployment for the period of 2000–2011 within sixty-three countries sampled [8].
All of this is by way of stating the obvious–work is a central part of most of our lives. Either directly or indirectly work appears to be significantly related to the modification of individual’s wellbeing. Unemployment may be associated with negative outcomes [9, 10], such as suicide, in the most extreme. Employment may be associated with positive outcomes such as increased physical and mental health [11–13]. These effects could arise because work is expected within our societies and therefore having and maintaining work is necessary for ‘fitting’ in to expected roles. It may also be an issue of time, activity, and relationships. We spend a large amount of lives in work and as such the workplace may provide the locus in which we experience our lives and a forum for developing relationships. Alternatively, it could be that there is something about work itself, something about the act of making and contributing, that is a mechanism that contributes to well-being [1].
Not all who wish to work can work or are able to find work. There are a substantial number of persons who are unemployed. The current unemployment rate in the United States stands at 4.0% as of June 2018 (https://data.bls.gov/timeseries/LNS14000000). Just nine years ago at the height of the recession unemployment reached its peak at 10% in October 2009 (https://data.bls.gov/timeseries/LNS14000000). These unemployment statistics include those who are looking for work, but who are not currently working. The statistics do not include those who have stopped looking for work and are assumed to be effectively out of the workforce. In 2014, the National Alliance on Mental Illness (NAMI) reported that unemployment levels for persons “receiving public mental health services” was at 80% nationally (https://www.nami.org/Press-Media/Press-Releases/2014/Mental-Illness-NAMI-Report-Deplores-80-Percent-Une). The NAMI report also stated “approximately 60% of the 7.1 million receiving public mental health services want to work”. We can see that there is a significant gap within this group between the number who say that they want to work and the number who do work. An obvious question then is how might we go about helping those who wish to find work do so.
One intervention that appears to be quite effective at helping those with SMI find work is the individual placement and support (IPS) treatment that is derived from supported employment (SE) theory. Past meta-analyses have estimated that supported employment increases the likelihood of achieving competitive employment by 2.4-fold (RR = 2.40 95%CI = [1.99, 2.90]) [14] and of finding any employment by 3.24-fold (RR = 3.24 95%CI = [2.17, 4.82]) [15]. What is more uncertain is to what degree does supported employment, in particular IPS, help decrease symptoms and provide for an overall better quality of life and wellbeing. We explore all of these issues in this article.
What is supported employment?
Supported employment (SE) is a theory of work rehabilitation that argues that individuals–often called consumers–with some identified disability should be helped to find competitive employment (CE) at the beginning and not made to wait until after finishing some treatment or training program. Traditionally, the disability was severe mental illness (SMI), but has been expanded in recent years to cover a range of mental and physical conditions such as spinal cord injuries in veterans [16] as well as affective disorders [17]. SE may then be broadly construed so as to be part of the broader vocational rehabilitation literature (or perhaps even the broader vocational literature) and not just solely focused on SMI populations.
A bit confusingly, the term “supportive employment” is often used to refer to the type of treatment as well as the achievement of the main outcome, which is competitive employment [18]. Supported employment encompasses a fairly broad category that ought to be seen as a general theory of treatment and not a specific treatment in and of itself. Instead, the most popular implementation is individual placement and support (IPS). The essential point of SE and IPS is that treatment ought to offer subjects a concurrent means of receiving care and at the same time receive help in finding employment. This is in contrast to traditional programs, which we will often refer to as treatment as usual (TAU), in which subjects first receive treatment for their disability and then they would be helped to find employment. The difference can be succinctly stated: SE/IPS is ‘place-and-treat/train’ whereas TAU is ‘train/treat-then-place’.
Within the literature, authors enumerate different numbers of essential points to SE ranging from 6 to 8. Even the same authors may differ in the total number of essential points. For example, Bond lays out six key points in the form of headings in his paper [18]: “[1] Services focused on competitive employment… [2] Eligibility based upon choice; no exclusions … [3] Rapid job search … [4] Rehabilitation and mental health care integration … [5] Attention to patient's preferences … [6] Unlimited time and individual support”. In a later paper, the number of elements is listed as eight [3]: “[1] eligibility based on … choice, [2] focus on competitive employment …, [3] integration of mental health and employment services, [4] attention to client preferences, [5] work incentives planning, [6] rapid job search, [7] systematic job development, and [8] individualized job support ….”
The crucial point is not the exact enumeration of points, but as Bond [18] writes, it is the fact that SE casts aside three prior general assumptions. First that “people with SMI need an extended period of time in vocational preparation before entering a competitive job in order to become work ready and identify career goals", second that “rehabilitation services should be provided separately from mental health treatment services", and finally that programs should use transitional employment that "consists of time-limited job placements developed by a rehabilitation agency that [patients] work in preparation for competitive jobs."
To reiterate, the core of SE is the focus on individualized care oriented toward improving mental health and getting the subject employed (preferably in competitive employment) as soon as possible, which are done concurrently. Previous programs generally had a stepwise implementation wherein patients would receive clinical and pre-vocational training prior to seeking employment. Supported employment does away with this stepwise technique and replaces it with a concurrent approach.
Treatments based upon SE have been deployed in at least 38 of the 50 states in the United States [19]. Of those states, Johnson-Kwochka et al. report that there was an average of 13.7 programs per state (SD = 16) and there were a total of 523 programs [19]. There are a number of SE implementations that have been designed and tested internationally (See Results; See [14] for a meta-analysis focused on this topic). Clearly, the use of SE is not limited to a few centers or isolated to a single country. There are also many resources for finding how to do SE/IPS available online. For example, the IPS Employment Center (https://www.ipsworks.org/), which also provides online training courses (https://www.ipsworks.org/training-consultation-services/), has a large collection of published books and manuals (https://www.ipsworks.org/order/books/.
In this present work, we searched the literature for randomized controlled trials of supported employment that employed individual placement and support. We then performed random effects meta-analyses on a number of outcomes, including the most widely used one–competitive employment. We also used a new statistical measure to estimate the proportion of settings in which we would expect to find a scientifically meaningful effect [20].
Methods
Literature search and data extraction methods
Our search goal was to identify the extent literature of randomized controlled trials that used individual placement and support to test supported employment against treatment as usual controls. To this end, we searched PSYCHinfo for articles. Exact terms and results are detailed below in the Results section. Relevant data were extracted from the articles and entered into a PostgreSQL database (v.9.6). The first author did the search and data extraction. We have included the PLoS One checklist as supporting information (S1 File).
Statistical methods
We performed random effects meta-analyses in R (v.3.2.1) using the metafor package [21]. Data from the PostgresSQL database were queried and imported into R using r-postgresql library for analysis. For reproducibility, we saved the relevant data as R data files and have included them as supplemental materials along with the needed R script file (S2 and S3 Files, respectively).
For binary outcomes (e.g., subjects attaining competitive employment) we computed the log relative risk. We convert from log(RR) to RR when needed for comparisons to previous research. For continuous data (e.g., number of weeks worked), we computed the standardized mean difference from the given means and standard deviations. In some cases standard deviations were not directly provided in the articles. In these cases, we computed the needed standard deviations via established methods (e.g., estimating standard deviations standard errors or confidence intervals).
In addition to the random effects meta-analyses, we also estimated the proportion of scientific meaningful effect sizes across various settings examined in the meta-analysis [20]. For risk ratios we set the scientific meaningful values at 1.2 and 0.8. For standardized mean differences we set the scientific values of interest at 0.2 and -0.2. We performed these analyses in R using the EValue library on samples that had five or more articles. Such proportion metrics can be especially helpful when there is substantial heterogeneity in effect sizes across studies and settings. For example, a pooled estimate close to the null may either be due to most of the effect sizes being close to null, or because several are large and positive while others are large and negative. The proportion metrics help to distinguish these scenarios.
Results
Literature search results and data extraction
Search was conducted using PSYCHInfo database by the first author (See Fig 1 for PRISMA flowchart and Table 1 for queries). The primary query was “supported employment AND randomized controlled trial”, which returned 111 results. A larger search with “supported employment” with methodology filters (clinical trial, follow-up study, treatment outcome, prospective study, clinical case study, focus group, retrospective study, and nonclinical case study) was also done in order to find any studies that may have been misclassified or had not contained the keywords “randomized controlled trial”. This larger search returned 313 results. After de-duplication, we were left with 368 publications.
Our literature search found 368 unique articles and we found another 5 that were referenced in articles. We initially screened articles to check that they used an RCT design, extended the duration of a previous RCT experiment, or performed secondary analysis on a previous RCT. We found that 49 articles met this criterion plus the 5 that we found referenced in other articles. We then read those articles and excluded ones that used a mixed treatment (e.g., IPS plus some other component) or did not report an outcome variable of interest. We excluded 24 articles for these reasons. This left 30 articles in total that we used for analysis.
In order to retain an article it had to have performed or done analysis on a RCT of IPS vs. a treatment as usual control. The outcome variables had to have been either the usual work outcomes (e.g., competitive employment achieved, about of time in work) and/or non-work outcomes (e.g., quality of life).
After deduplication, we screened the articles for methodology by reading abstracts, keywords, and subject headers. We retained only articles that appeared to have done RCT trials, did a follow-up to or extension of a previous RCT (i.e., included additional follow-up time), or performed secondary analysis on a previous RCT. We did not retain articles that were cross-sectional, prospective, retrospective, or case studies. We did not retain articles that reported results from just a single arm of RCT studies. We did not retain studies that compared IPS to a better/different version of SE unless there was a third control arm of the study that was treatment as usual. We did retain articles that appeared to be secondary analyses on previous RCT studies in order to see if new outcomes of interest were reported. After screening we were left with 49 articles.
We then read the papers to ensure that they had the needed outcome measures and were in fact RCT studies between IPS and a treatment as usual control group. We removed 26 articles. Of those, we removed studies that used IPS plus some other treatment (n = 8). The remaining articles were removed because there was not the required outcome variable present.
While reading the articles, we noted any references to other RCT studies that used IPS as a treatment. We made sure that any RCT studies that were referenced were cross-referenced to our list. If the study was missing, we sought it out and checked it. If it met our inclusion criteria, we retained the study. We found seven additional articles by this method of which four met our criteria [16, 22–24]. Finally, we also found one additional article [25] from our reading of a previous meta-analyses [26].
In total, we found 30 articles [16, 22–25, 27–50] for our meta-analysis that matched our criteria (Tables 1–3 and Fig 1). Of those, 25 were original RCTs (Table 2), two were follow-ups that extended the period of observation of a previous RCT (Table 3), and three were secondary analyses on a previous RCT (Table 4). All articles reported the results of randomized controlled trials (RCT) that focused on comparing supported employment (SE) as implemented via individual placement and support (IPS) to a control condition, which was usually the locally available best usual practice.
In general, research articles varied greatly in quality, as has been previously reported [15]. Researchers measured the same outcome variable in a number of ways. For example, competitive employment was often measured as having had any competitive employment during the study period and/or as having been employed at the end of the study.
A number of researchers did calculate the number of needed subjects based on expected effect sizes and attrition rates. However, the effect size estimates used were generally for the main outcome, which was usually competitive employment. This effect (as shown below) is much larger than other possible effects (e.g., quality of life improvement). Because power was calculated only for the main outcome variable, researchers may have lacked sufficient power to detect secondary outcomes of interest. This provides a strong motivation for meta-analysis. We return to this point in the discussion.
Articles also varied by whether or not they reported measures of fidelity to the study protocol. A number of fidelity scales have been developed that include a shorter 15-question [51] and longer 25-question survey [52].
We performed an additional search for meta-analyses and found nine previous meta-analyses (query: “supported employment” with methodology filter ‘meta analysis’ in PsycINFO). We briefly detail some results from these previous articles. One meta-analysis used 14 studies and found significant effects on competitive employment over controls [15]. These authors note that in general the level of studies were low to very low quality. Still, the authors concluded that IPS increased levels of any employment (k = 7, RR = 3.24, 95%CI = [2.17, 4.82]), length of competitive employment (k = 1, MD = 70.63, 95%CI = [43.22, 94.04]), length of days in any employment (k = 2, MD = 9.86, 95%CI = [5.36, 14.36]), length of job tenure (k = 1, MD = 9.86, 95%CI = [5.36, 14.36]), increase in paid employment (k = 2, MD = 84.94, 95%CI = [51.99, 117.89]), and decreased time to first employment (k = 1, MD = -161.60, 95%CI = [-225.73, -97.47]). However, this study did not look at non-vocational outcomes.
A more recent meta-analysis looked at the possible moderation by country of treatment [14]. The authors found that there did not appear to be substantial variation by country and that IPS consistently outperformed controls in helping subjects find competitive employment (RR = 2.40, 95%CI = [1.99, 2.90]).
A third meta-analysis looked at supported employment effects on non-vocational outcomes [53]. The researchers found a significant effect on quality of life with those in IPS having a higher level (d = 0.28, 95%CI = [0.04, 0.52]), but no significant effects on global functioning (d = -0.01, 95%CI = [-0.13, -0.11]) or mental health symptoms (d = -0.20, 95%CI = [-0.62, 0.23]).
Although there have been previous meta-analyses, our current paper contributes in ways beyond prior meta-analyses in including more studies, a broader range of outcomes, examining length of experiment as a moderator, and using a novel statistic technique to estimate the proportion of settings with scientific meaningful effect sizes, which is able to help interpretation of meta-analytic researches with significant heterogeneity.
Competitive employment status
Researchers investigated competitive employment outcomes in one of two general ways (Table 5). First, some researchers investigated if subjects obtained any competitive employment at any time during the study period. Second, other researchers looked at employment at the end of follow-up or repeatedly at follow-up time points.
We performed two different analyses. First, we looked at the most inclusive definition that counted an individual as competitively employed if they had been so at any point during the study. Second, we looked at the outcomes at only the end of the studies. When there was a follow-up study that added additional years of observation, we used that for analysis instead of the original study. Finally, given that we had access to the total number of subjects randomized, we performed the analyses following the intent-to-treat paradigm. In other words, for the Ns needed to be specified we used the total number of subjects randomized and not the values that the authors may themselves have used in their analyses.
We estimated that those in IPS conditions were 1.6x more likely to have found CE at any point in the study compared with those in control conditions (Fig 2A, k = 23, log(RR) = 0.49, SE = 0.06, Z = 8.70, p<0.0001, 95%CI = [0.38, 0.60], τ2(SE) = 0.01(0.02), I2 = 22.59%, H2 = 1.29, Q(22) = 29.64, p = 0.1277). We also estimated that those in IPS conditions, compared to those in control conditions, were 1.8x more likely to have been competitively employed at the end of studies (Fig 2B, k = 13, log(RR) = 0.58, SE = 0.08, Z = 7.23, p<0.0001, 95%CI = [0.42, 0.74], τ2 (SE) = 0.01(0.03), I2 = 10.10%, H2 = 1.11, Q(12) = 14.93, p = 0.2451).
A) For having held a competitive employment job for any portion of the experimental period B) For having held a competitive employment job at end of the experiment.
Given the variety of study lengths (Tables 2 and 3), we reran the above two analyses including study length as a moderator. Estimates for length as a moderator for finding CE at any point during the study or having been in CE at the end of the study both were close to zero and included the null (Any: Length = -0.0001, 95%CI = [-0.01, 0.01]; End: Length = 0.0005, 95%CI = [-0.02, 0.02]).
Our estimates showed that there was significant heterogeneity in the underlying study populations. In order to aid in interpretation we estimated the proportion of settings that the effects would exceed a scientifically meaningful threshold of RR = 1.2 or below the threshold of RR = 0.8. For finding CE during any part of a study, the proportion of studies with true RR>1.2 is estimated to be >0.99 (95%CI = [0.95, 1]), with the proportion of studies with true RR<0.8 expected to be <0.001 (95%CI = [0, <0.01]). Looking at CE at the end of follow-up, our estimates were very similar (RR = >1.2, Proportion>0.999, 95%CI = [>0.99, 1]; RR<0.8, Proportion = <0.001, 95%CI = [0, <0.01]).
Secondary vocational outcomes
In addition to the main outcome of interest—competitive employment status—studies often measured and reported secondary vocational outcomes (Tables 6–10 and Figs 3–5). We performed meta-analyses on four of these (time to first competitive employment job, job tenure, total time worked, and income). We only performed meta-analysis from papers that only used competitive employment for these statistics. We omit from analysis statistics that combined all types of employment (i.e., competitive and not).
A) Job tenure, which is the duration of the longest held competitive employment job. B) Job duration, which is the total length of competitive employment.
There were generally two methods in which estimates of the treatment for secondary vocational outcomes were derived. First, researchers estimated the effect using only those subjects that found a competitive employment job. Second, researchers used an intent-to-treat method whereby they imputed data with zeros of those that did not achieve competitive employment status. We include the estimates as given by the authors, which is a mix of the above two methods. If a study reported both intent-to-treat and conditional statistics, we selected to use the intent-to-treat values.
Time to first competitive employment
Time to first competitive employment job was reported by seven studies (Table 6). We estimated that those in IPS would have a shorter time to their first competitive employment job (Fig 3, k = 9; d = -0.31, SE = 0.08, Z = -3.93, p<0.0001, 95%CI = [-0.46, -0.15], τ2 (SE) = 0(0.03), I2 = 0.00%, H2 = 1.00, Q(8) = 5.8994, p = 0.6585). Because there was not significant heterogeneity, we were unable to estimate a proportion. Experiment length was not a significant moderator (Length = 0.02, SE = 0.01, Z = 1.42, p = 0.16, 95%CI = [-0.01, 0.05]).
Job tenure & length of competitive employment
The amount of time that a subject spent in or held a competitive employment job was measured in two ways. First, researchers reported job tenure or length of the longest held CE job per subject (Table 7). Second, researchers reported total time worked as the sum of all time in all CE jobs per subject (Table 8). Researchers reported total time in different units (e.g., hours, weeks). For our analysis, we selected to use the smallest unit that a researcher reported (i.e., we include hours if researchers reported both hours and weeks).
For job tenure, we estimated that persons in IPS held a CE job longer than those in control conditions (Fig 4A, k = 5, d = 0.56, SE = 0.12, Z = 4.72, p<0.0001, 95%CI = [0.33, 0.79], τ2 (SE) = 0.08(0.06), I2 = 68.68%, H2 = 3.19, Q(7) = 21.4770, p = 0.0031). Running the analysis with experiment duration as a moderator was not significant (Length: d = 0.005, 95%CI = [-0.01, 0.02]). We estimated that scientific meaningful effects that we expect to be observed with d>0.2 was 0.90 (95%CI = [0.68, 1]) and that the proportion of studies with d<-0.2 to be tiny with a confidence interval that included the null (Prop = 0.003, 95% = [0, 0.02]).
For total time worked in CE, we estimated that those in the IPS condition worked significantly more than controls (Fig 4B, k = 17, d = 0.46, SE = 0.06, Z = 8.23, p<0.0001, 95%CI = [0.35, 0.57], τ2 (SE) = 0.02(0.02), I2 = 31.03%, H2 = 1.45). Experiment length did not appear to be a significant moderator (Length: d = 0.005, 95%CI = [-0.004, 0.015]). We estimated the proportion of effect sizes of d>0.2 as 0.98, (95%CI = [0.86, 1]) and the proportion with d<-0.2 to be essentially zero (Prop = <0.001, 95%CI = [0, <0.001]).
Income from competitive employment
In addition to the question of if a subject found a CE job and how much time they spent in the job, researchers also measured income (Table 9). For the following analyses, as above, we used only values that were derived from competitive employment (i.e., if the reported statistic was from income of competitive and non-competitive employment sources, we did not use it).
We estimated that those in IPS conditions would have significantly more income (Fig 5, k = 12; d = 0.48, SE = 0.06, Z = 8.31, p<0.0001, 95%CI = [0.36, 0.59], τ2 (SE) = 0.0004(0.02), I2 = 0.90%, H2 = 1.01, Q(11) = 8.3404, p = 0.6825). We estimated that we would expect all settings to have an effect size d>0.2 (Prop = 1, 95%CI = [1, 1]) and none to be in the opposite direction with d<-0.2 (Prop = <0.001, 95%CI = [0, <0.001]). Length was not a significant moderator (Length = -0.004, 95%CI = [-0.01, 0.005]).
Non-vocational outcomes
In addition to the vocational outcomes reported above, we performed analyses on three non-vocational outcomes (quality of life, global functioning, and mental health).
Quality of life
Researchers used various measures for quality of life, such as Manchester Short Assessment (Table 10). For analysis, we pooled all measures of quality of life together. We estimated that those in IPS conditions had a higher levels of QOL, but that the confidence intervals for the estimate included the null (Fig 6, k = 9, d = 0.30, SE = 0.19, Z = 1.60, p = 0.11, 95%CI = [-0.07 0.67], τ2 (SE) = 0.2955(0.1602), I2 = 94.90%, H2 = 19.59, Q(8) = 62.1751, p<0.0001). We estimated that the proportion of settings that we expect the effect size to be d>0.2 was relatively large (Prop = 0.57, 95%CI = [0.30, 0.84]) but that those that would be expected to be d<-0.2 was relatively small with a confidence interval that included zero (Prop = 0.18, 95%CI = [0, 0.40]). Length of study was not a significant moderator (Length = -0.0005, 95%CI = [-0.03, 0.03]). Given the heterogeneity and the different scales used, we also tested to see if scale was a significant moderator and we estimated that it may be with the Personal Wellbeing Index having the highest estimate (Table 11).
Global functioning
Several studies reported some measure of global functioning (Table 12). Global functioning measures how severe a person’s symptoms are (i.e., how big of an impact on daily life are symptoms). We estimated that those in IPS had a slightly higher global functioning, but that the confidence intervals of the estimate include the null (Fig 7A, k = 5, d = 0.09, SE = 0.09, Z = 0.9957, p = 0.3194, 95%CI = [-0.0891, 0.2731]). Our estimates of scientifically meaningful proportions were both small and the confidence intervals included the null for both directions (d>0.2, Prop = 0.21, 95%CI = [0, 76]; d<-0.2, Prop = 0.01, 95%CI = [0, 0.15]).
A) Global functioning. B) Mental health.
General mental health
In addition to the above measures, researchers also recorded values for several other broadly mental health related measures. Here, we pooled reported outcomes on the Brief Psychiatric Rating Scale, the Positive and Negative Syndrome Scale, and the mental component of the Short-Form Health Survey (Table 13).
We estimated a very small effect and that the confidence intervals of the estimate include the null (Fig 7B, k = 6, d = 0.03, SE = 0.09, Z = 0.3320, p = 0.7399, 95%CI-0.15, 0.21], τ2 (SE) = 0.03(0.03), I2 = 64.55%, H2 = 2.82, Q(5) = 15.67, p = 0.0078). Our estimates of scientifically meaningful proportions were both small and the confidence intervals included the null for both directions (d>0.2, Prop = 0.15, 95%CI = [0, 0.51]; d<-0.2, Prop = 0.08, 95%CI = [0, 0.36]). Experiment length was not a significant moderator (Length = 0.03, SE = 0.02, Z = 1.46, p = 0.14, 95%CI = [-0.01, 0.08]).
Discussion
From the above results, it is clear that supported employment (SE) as deployed via individual placement and support (IPS) treatments is significantly better than treatment as usual (TAU) on all vocational outcomes. Furthermore, there is some evidence that IPS may also improve quality of life and maybe global functioning, but more research with larger sample sizes are needed to know for sure.
For all vocational outcomes, those in IPS treatments had higher rates of competitive employment at any point during the study (log(RR) = 0.49, 95%CI = [0.38, 0.60], Fig 2A) and at the end of the study (log(RR) = 0.58, 95%CI = [0.42, 0.74], Fig 2B), shorter time to first competitive employed job (d = -0.31, 95%CI = [-0.46, -0.16], Fig 3), longer job tenure (d = 0.56, 95%CI = [0.33, 0.79], Fig 4A), longer durations of work (d = 0.46, 95%CI = [0.35, 0.57], Fig 4B), and higher total income (d = 0.48, 95%CI = [0.36, 0.59], Fig 5).
In addition to vocational outcomes, we also examined non-vocational outcomes. All our estimates were small and included the null (quality of life d = 0.30, 95%CI = [-0.07, 0.67], Fig 6; global functioning d = 0.09, 95%CI = [-0.09, 0.27], Fig 7A; mental health d = 0.03, 95%CI = [-0.15, 0.21], Fig 7B). For quality of life, additional analysis estimating the proportion of settings did provide some additional insight pointing to possible positive effects. We estimated that the proportion of settings that would have an effect of d>0.2 was 0.57 (95%CI = [0.30, 0.84]). This latter result is suggestive that there may be a significant effect of IPS on these outcomes, but that the effect is sensitive to the setting. If this is a true effect, a possible route of causation for increased quality of life and other non-vocational outcomes could be via the competitive employment. A recent study using mediation analysis suggest improvement in quality of life may be fully mediated via competitive employment [54].
Methodological interpretation and critique
In the research reviewed here, researchers often reported that a result was not significant, but gave no estimate of the result (e.g., means and standard deviations). Because of this, we were unable to include all outcomes that researchers said that they had measured. We urge all researchers going forward to report all estimated values, even for non-significant results based upon null hypothesis testing. Although a result may not rise to the level of statistical significance in a single study, through the use of meta-analysis future researchers might be able to tease out that there is a significant population level effect present.
We also suggest that researchers base sample sizes upon the smallest effect that they wish to detect, not the main outcome, especially if secondary outcomes are expected to have considerably smaller effect sizes than the main outcome.
In subsequent meta-analyses we would also encourage researchers to report not only the pooled effect estimate but also the estimated proportion of settings in which there are scientifically meaningful effect sizes, both positive and negative. In some cases, these proportion measures may not be especially additionally informative. For example, with achieving competitive employment, the pooled estimates and the proportion measures gave very similar information of consistent beneficial effects. But in other settings, a wide confidence interval that includes the null or a near-null pooled estimate may be principally driven by effect heterogeneity. Thus with quality of life, although the point estimate was small and its confidence intervals include the null (d = 0.30, 95% CI = [-0.07, 0.67]), the estimated proportion of studies for which the effect size was d>0.2 was substantial (Prop = 0.57, 95% CI = [0.30, 0.84]), indicating that for a number of studies there was substantial improvement. Furthermore, the estimated proportion of studies that would be in the opposite direction with d<-0.2 included was not entirely negligible but confidence interval did include the null (Prop = 0.18, 95%CI = [0, 0.40]). We believe more routine reporting of such metrics, especially in contexts like SE where effects can be quite variable, would help provide a more nuanced understanding of results.
Can we expand supportive employment to more groups?
SE and IPS were originally developed for and used with populations that had severe mental illness. Given the effectiveness of IPS, are there additional groups that could benefit. The answer appears to be a ‘yes’ from our above results and past literature.
A number of studies have looked IPS in the US veteran population. The majority of studies looked at veterans with spinal cord injuries. Ottomanelli and colleagues show that veterans with SCI have improved CE outcomes when placed in SE/IPS treatments compared to standard ones [16].
There is also work done on PTSD and veterans, which also found that SE/IPS was better than TAU [55]. One study looked at moderating effect of PTSD on outcomes for SE [56] and found that PTSD groups were less likely to be employed in the SE and other conditions. The difference seemed to be most evident in the SE treatment group.
Previous research reported that for populations with schizophrenia or schizoaffective disorder, IPS proved more effective than TAU as measured by subjects finding competitive employment (57% vs. 29%)[44].
Although research in SE first looked at those with severe mental illness, it now includes other disabilities such as spinal cord injury [16, 24] and affective disorders [17]. There were a number of other subject populations that were not explicitly detailed in the results because they were limited to a single study per population. These include those on social security disability insurance [57], young adults [58], emerging adults ages 17–20 [59], those with general affective disorders [17], homeless populations [42], those receiving methadone treatment [60], minority groups [61], those with a criminal justice history [29, 62], and those with musculoskeletal injuries [38]. All of these studies showed that those in IPS faired better than those in the TAU control on the main measured outcome of competitive employment rates.
That there is the possibility for expanding the reach of SE via IPS raises some interesting possibilities as well as serious questions. The most obvious question is “What is the limit of IPS?” Could IPS be provided to those without any noticeable or known disability? The answer appears to be “yes” that, in the limit, IPS could be provided to everyone. But what would this look like? It would boil down to having dedicated personal to help subjects find competitive employment.
Part of the IPS treatment is that it provides a means to help those looking for work find competitive work. That this would work with other groups in general is probably not too surprising. We must remember that the initial surprise after all in this literature was that this approach could and does appear to work with those with severe mental illnesses and disabilities who had previously been thought to be ‘unfit’ for work until after their mental disabilities were resolved or at least controlled.
The utility of expanding this treatment to groups without disabilities, and whether that is at all cost-effective, is not obvious. However, expansion to other groups that have a disability or suffer from some other symptoms and who are normally encouraged to follow a two-step serial process does seem to make sense. For these groups, it would be good to understand if SE via IPS could benefit them as the current results suggest it might.
Finally, within the traditional groups that this treatment was designed to help, there is a need to better differentiate between the types of severe mental illness (SMI) that subjects generally have. SMI usually includes, but is not limited to, schizophrenia, bipolar disorders, and severe depression given that each of these may have its own etiology and neurobiological underpinnings. For example, there may be a difference in the structural (white matter) between bipolar and unipolar depression [63]. Better tracking of these groups could help to elucidate when and if IPS helps with non-vocational outcomes.
Can we make supportive employment more effective?
As seen above, IPS is quite effective in helping certain populations gain CE. It is less certain how much better SE is than treatment as usual for other outcomes such as mental health. Our estimates suggest that there is little effect and that for some of the outcomes the observed effect is quite heterogeneous. Are there better alternatives to this implementation?
An early work by McGurk and colleagues looked at the impact of cognitive training (CT), specifically, the “Thinking Skills for Work Program”, in conjunction with SE [64]. They found that CT+SE outperformed SE alone in percentage of subjects employed. A different study looked to augment SE with cognitive training in subjects with schizophrenia [56]. They found that the added training significantly increased the outcome of subjects (30% vs. 9%). Most recently, they found that for those that didn’t respond to IPS, the individual receiving cognitive enhancement treatments (the “Thinking Skills for Work Program”) had higher employment (60% vs. 36%), worked longer (23.9 vs. 9.2 weeks), and earned more ($3,421 vs. $1,728) than the control group that had an enhanced version of SE [65].
Other researchers have looked at the effectiveness of skills training in conjunction with IPS [66]. They found that while the test group increased their knowledge of fundamental concepts of the workplace (e.g., job performance), they did not differ from controls on a host of other outcomes. The subjects (both treatment and control) had, however, already been in IPS prior to the study.
There have also been efforts to expand how IPS is done. For example, one study looked at expanding SE beyond the rigid requirements of IPS [67]. Instead of using specialized IPS delivery centers, these researchers were able to show that IPS can be effectively delivered in a ‘multi-server’ program center.
Conclusion
There is little doubt that for the primary outcome, which was competitive employment rates, the supported employment framework and the IPS treatment is more effective than treatment as usual. There is even good evidence that the IPS treatment can be expanded and remain effective with groups other than those with severe mental illness (who were the original targets of the intervention).
Still, it must be noted that SE/IPS is not a panacea. Subjects still work below full time and still suffer from their underlying disorder. The obvious real solution would be to eliminate fully the underlying condition, but this is, of course, often difficult.
There is some evidence that even though the above is true, the cost-benefit of SE/IPS is not necessarily better than TAU. One RCT investigated the cost-benefit of SE compared to other vocational rehabilitation programs [68]. Depending on the outcome measured, SE/IPS may or may not be more effective. A fairly recent study looked at the cost-benefit of SE for veterans with spinal cord injuries and found that it was not significantly more cost-effective than TAU [69]. Also, vocational rehabilitation closure rates (i.e., success) of SE/IPS compared to TAU has been observed to be the same [70]. However, these studies do not take into account possible increases in meaning and quality of life, the “value” of which are of course difficult to quantify.
Finally, we must ask ourselves what is the ultimate purpose of SE/IPS and vocational rehabilitation and what ought the ultimate purpose truly be. If SE/IPS provides greater employment but does not help in other respects, is it worth it to have these services? SE/IPS does provide greater competitive employment and may also contribute to a higher level of quality of life. SE/IPS does not appear to significantly decrease subjects’ mental health symptoms, but neither does it increase their symptoms, though there was some evidence that the effects may be heterogeneous across studies. Examining further and understanding that heterogeneity may be essential to developing further improvements in SE interventions to enhance an even broader range of outcomes, though much work would still be required to make this a reality.
Supporting information
S1 File. PRISMA_checklist.
PLoS One PRISMA Checklist.
https://doi.org/10.1371/journal.pone.0212208.s001
(PDF)
S2 File. Supported_employment.
R script file with source code for performing analyses used in this paper.
https://doi.org/10.1371/journal.pone.0212208.s002
(R)
S3 File. R data files (zipped) to use with S2 for reproducing analyses.
https://doi.org/10.1371/journal.pone.0212208.s003
(ZIP)
References
- 1. VanderWeele TJ. On the promotion of human flourishing. Proceedings of the National Academy of Sciences. 2017;114(31):8148–56. pmid:28705870
- 2.
Terkel S. Working: People talk about what they do all day and how they feel about what they do.: New Press; 1974.
- 3. Bond GR, Drake RE. Making the Case for IPS Supported Employment. Administration and Policy in Mental Health and Mental Health Services Research. 2014;41(1):69–73. pmid:23161326
- 4. Modini M, Joyce S, Mykletun A, Christensen H, Bryant RA, Mitchell PB, et al. The mental health benefits of employment: Results of a systematic meta-review. Australasian psychiatry: bulletin of Royal Australian and New Zealand College of Psychiatrists. 2016. PubMed Central PMCID: PMC26773063.
- 5.
Frederick DE, VanderWeele TJ. Increased job crafting is positively associated with work engagement: a longitudinal meta-analysis.
- 6. Bakker AB. An Evidence-Based Model of Work Engagement. Current Directions in Psychological Science. 2011;20(4):265–9.
- 7. Tims M, Derks D, Bakker AB. Job crafting and its relationships with person–job fit and meaningfulness: A three-wave study. Journal of Vocational Behavior. 2016;92:44–53.
- 8. Nordt C, Warnke I, Seifritz E, Kawohl W. Modelling suicide and unemployment: a longitudinal analysis covering 63 countries, 2000–11. The Lancet Psychiatry. 2015;2(3):239–45. Epub 2015/09/12. pmid:26359902.
- 9. McKee-Ryan F, Song Z, Wanberg CR, Kinicki AJ. Psychological and physical well-being during unemployment: a meta-analytic study. J Appl Psychol. 2005;90(1):53–76. Epub 2005/01/12. pmid:15641890.
- 10. Paul K, Moser K. Unemployment impairs mental health: Meta-analyses. J Vocat Behav. 2009;74:264–82.
- 11. Lucas RE, Clark AE, Georgellis Y, Diener E. Unemployment Alters the Set Point for Life Satisfaction. Psychological Science. 2004;15(1):8–13. Epub 2004/01/14. pmid:14717825.
- 12. S R, L C, M W, C M, SB R, A B, et al. Association of Returning to Work With Better Health in Working-Aged Adults: A Systematic Review. Am J Public Health. 2012;102:541–56. pmid:22390520
- 13.
G W, AK B. Is Work Good for your Health and Well-being? Office TS, editor. London 2006.
- 14. Modini M, Tan L, Brinchmann B, Wang MJ, Killackey E, Glozier N, et al. Supported employment for people with severe mental illness: Systematic review and meta-analysis of the international evidence. Br J Psychiatry. 2016;209(1):14–22. Epub 2016/04/23. pmid:27103678.
- 15. Kinoshita Y, Furukawa TA, Kinoshita K, Honyashiki M, Omori IM, Marshall M, et al. Supported employment for adults with severe mental illness. The Cochrane database of systematic reviews. 2013;(9):CD008297-CD. Epub 2013/09/14. pmid:24030739.
- 16. Ottomanelli L, Goetz LL, Suris A, McGeough C, Sinnott PL, Toscano R, et al. Effectiveness of supported employment for veterans with spinal cord injuries: results from a randomized multisite study. Archives of physical medicine and rehabilitation. 2012;93(5):740–7. Epub 2012/05/01. pmid:22541306.
- 17. Bejerholm U, Larsson ME, Johanson S. Supported employment adapted for people with affective disorders—A randomized controlled trial. Journal of Affective Disorders. 2017;207:212–20. Epub 2016/10/11. pmid:27723546.
- 18. Bond GR. Supported employment: evidence for an evidence-based practice. Psychiatric rehabilitation journal. 2004;27(4):345–59. Epub 2004/06/30. pmid:15222147.
- 19. Johnson-Kwochka A, Bond GR, Becker DR, Drake RE, Greene MA. Prevalence and Quality of Individual Placement and Support (IPS) Supported Employment in the United States. Administration and Policy in Mental Health and Mental Health Services Research. 2017;44(3):311–9. pmid:28062932
- 20.
Mathur MB, VanderWeele TJ. Sensitivity Analysis for Unmeasured Confounding in Meta-Analyses. arXiv. 2017;arXiv:1707.09076.
- 21. Viechtbauer W. Conducting meta-analyses in R with the metafor package. Journal of Statistical Software. 2010;36(3):1–48.
- 22. Drake RE, McHugo GJ, Becker DR, Anthony WA, Clark RE. The New Hampshire study of supported employment for people with severe mental illness. Journal of consulting and clinical psychology. 1996;64(2):391–9. pmid:8871423
- 23. Heslin M, Howard L, Leese M, McCrone P, Rice C, Jarrett M, et al. Randomized controlled trial of supported employment in England: 2 year follow-up of the Supported Work and Needs (SWAN) study. World Psychiatry. 2011;10(2):132–7. pmid:21633690
- 24. Ottomanelli L, Barnett SD, Goetz LL. Effectiveness of Supported Employment for Veterans With Spinal Cord Injury: 2-Year Results. Archives of Physical Medicine and Rehabilitation. 2014;95(4):784–90. pmid:24316325
- 25. Viering S, Jäger M, Bärtsch B, Nordt C, Rössler W, Warnke I, et al. Supported Employment for the Reintegration of Disability Pensioners with Mental Illnesses: A Randomized Controlled Trial. Frontiers in Public Health. 2015;3:237–. pmid:26539425
- 26. Metcalfe JD, Drake RE, Bond GR. Economic, Labor, and Regulatory Moderators of the Effect of Individual Placement and Support Among People With Severe Mental Illness: A Systematic Review and Meta-analysis. Schizophrenia Bulletin. 2018;44(1):22–31. Epub 2017/10/17. pmid:29036727; PubMed Central PMCID: PMCPMC5768052.
- 27. Bejerholm U, Areberg C, Hofgren C, Sandlund M, Rinaldi M. Individual placement and support in Sweden—a randomized controlled trial. Nordic journal of psychiatry. 2015;69(1):57–66. pmid:24983382
- 28. Bond GR, Salyers MP, Dincin J, Drake R, Becker DR, Fraser VV, et al. A randomized controlled trial comparing two vocational models for persons with severe mental illness. Journal of consulting and clinical psychology. 2007;75(6):968–82. pmid:18085913
- 29. Bond GR, Kim SJ, Becker DR, Swanson SJ, Drake RE, Krzos IM, et al. A Controlled Trial of Supported Employment for People With Severe Mental Illness and Justice Involvement. Psychiatric services (Washington, DC). 2015;66(10):1027–34.
- 30. Burns T, Catty J, Becker T, Drake RE, Fioritti A, Knapp M, et al. The effectiveness of supported employment for people with severe mental illness: a randomised controlled trial. The Lancet. 2007;370(9593):1146–52.
- 31. Davis LL, Leon AC, Toscano R, Drebing CE, Ward LC, Parker PE, et al. A randomized controlled trial of supported employment among veterans with posttraumatic stress disorder. Psychiatric services (Washington, DC). 2012;63(5):464–70. pmid:22307881
- 32. Drake RE, McHugo GJ, Bebout RR, Becker DR, Harris M, Bond GR, et al. A randomized clinical trial of supported employment for inner-city patients with severe mental disorders. Archives of general psychiatry. 1999;56(7):627–33. pmid:10401508
- 33. Drake RE, Frey W, Bond GR, Goldman HH, Salkever D, Miller A, et al. Assisting Social Security Disability Insurance beneficiaries with schizophrenia, bipolar disorder, or major depression in returning to work. The American journal of psychiatry. 2013;170(12):1433–41. pmid:23929355
- 34. Hoffmann H, Jäckel D, Glauser S, Kupper Z. A randomised controlled trial of the efficacy of supported employment. Acta psychiatrica Scandinavica. 2012;125(2):157–67. pmid:22077907
- 35. Howard LM, Heslin M, Leese M, McCrone P, Rice C, Jarrett M, et al. Supported employment: randomised controlled trial. The British journal of psychiatry: the journal of mental science. 2010;196(5):404–11. pmid:20435968
- 36. Latimer EA, Lecomte T, Becker DR, Drake RE, Duclos I, Piat M, et al. Generalisability of the individual placement and support model of supported employment: results of a Canadian randomised controlled trial. The British Journal of Psychiatry. 2006;189(1):65–73. pmid:16816308
- 37. Lehman AF, Goldberg R, Dixon LB, McNary S, Postrado L, Hackman A, et al. Improving Employment Outcomes for Persons With Severe Mental Illnesses. Archives of General Psychiatry. 2002;59(2):165–. pmid:11825138
- 38. Li-Tsang CWP, Li EJQ, Lam CS, Hui KYL, Chan CCH. The Effect of a Job Placement and Support Program for Workers with Musculoskeletal Injuries: A Randomized Control Trial (RCT) Study. Journal of Occupational Rehabilitation. 2008;18(3):299–306. pmid:18563541
- 39. Michon H, van Busschbach JT, Stant AD, van Vugt MD, van Weeghel J, Kroon H. Effectiveness of individual placement and support for people with severe mental illness in The Netherlands: a 30-month randomized controlled trial. Psychiatric rehabilitation journal. 2014;37(2):129–36. pmid:24912062
- 40. Mueser KT, Clark RE, Haines M, Drake RE, McHugo GJ, Bond GR, et al. The Hartford study of supported employment for persons with severe mental illness. Journal of consulting and clinical psychology. 2004;72(3):479–90. pmid:15279531
- 41. Oshima I, Sono T, Bond GR, Nishio M, Ito J. A randomized controlled trial of individual placement and support in Japan. Psychiatric rehabilitation journal. 2014;37(2):137–43. pmid:24912063
- 42. Poremski D, Rabouin D, Latimer E. A Randomised Controlled Trial of Evidence Based Supported Employment for People Who have Recently been Homeless and have a Mental Illness. Administration and Policy in Mental Health and Mental Health Services Research. 2017;44(2):217–24. pmid:26721567
- 43. Tsang HWH, Chan A, Wong A, Liberman RP. Vocational outcomes of an integrated supported employment program for individuals with persistent and severe mental illness. Journal of Behavior Therapy and Experimental Psychiatry. 2009;40(2):292–305. pmid:19154992
- 44. Twamley EW, Vella L, Burton CZ, Becker DR, Bell MD, Jeste DV. The efficacy of supported employment for middle-aged and older people with schizophrenia. Schizophrenia research. 2012;135(1–3):100–4. pmid:22197080
- 45. Waghorn G, Dias S, Gladman B, Harris M, Saha S. A multi-site randomised controlled trial of evidence-based supported employment for adults with severe and persistent mental illness. Australian Occupational Therapy Journal. 2014;61(6):424–36. pmid:25284162
- 46. Wong KK, Chiu R, Tang B, Mak D, Liu J, Chiu SN. A randomized controlled trial of a supported employment program for persons with long-term mental illness in Hong Kong. Psychiatric Services. 2008;59(1):84–90. PubMed Central PMCID: PMC18182544. pmid:18182544
- 47. Zhang GF, Tsui CM, Lu AJB, Yu LB, Tsang HWH, Li D. Integrated Supported Employment for People With Schizophrenia in Mainland China: A Randomized Controlled Trial. American Journal of Occupational Therapy. 2017;71(6):7106165020p1-p8. Epub 2017/11/15. pmid:29135426.
- 48. Burns T, Catty J, White S, Becker T, Koletsi M, Fioritti A, et al. The impact of supported employment and working on clinical and social functioning: results of an international study of individual placement and support. Schizophrenia bulletin. 2009;35(5):949–58. pmid:18403375
- 49. Kukla M, Bond GR. A randomized controlled trial of evidence-based supported employment: Nonvocational outcomes. Journal of Vocational Rehabilitation. 2013;38(2):91–8.
- 50. Ottomanelli L, Barnett SD, Goetz LL. A prospective examination of the impact of a supported employment program and employment on health-related quality of life, handicap, and disability among Veterans with SCI. Quality of life research: an international journal of quality of life aspects of treatment, care and rehabilitation. 2013;22(8):2133–41. pmid:23345022
- 51. Bond GR, Becker DR, Drake RE, Vogler KM. A fidelity scale for the Individual Placement and Support model of supported employment. Rehabilitation Counseling Bulletin. 1997;40(4):265–84.
- 52. Bond GR, Peterson AE, Becker DR, Drake RE. Validation of the Revised Individual Placement and Support Fidelity Scale (IPS-25). Psychiatr Serv. 2012;63(8):758–63. Epub 2012/06/05. pmid:22660842.
- 53. van Rijn RM, Carlier BE, Schuring M, Burdorf A. Work as treatment? The effectiveness of re-employment programmes for unemployed persons with severe mental health problems on health and quality of life: a systematic review and meta-analysis. Occupational and Environmental Medicine. 2016;73(4):275–9. pmid:26740687
- 54. Jäckel D, Kupper Z, Glauser S, Mueser KT, Hoffmann H. Effects of Sustained Competitive Employment on Psychiatric Hospitalizations and Quality of Life. Psychiatric Services. 2017:appi.ps.2016000-appi.ps. pmid:28142387
- 55. Davis LL, Pilkinton P, Poddar S, Blansett C, Toscano R, Parker PE. Impact of social challenges on gaining employment for veterans with posttraumatic stress disorder: An exploratory moderator analysis. Psychiatric Rehabilitation Journal. 2014;37(2):107–9. pmid:24708195
- 56. McGurk SR, Mueser KT, Feldman K, Wolfe R, Pascaris A. Cognitive training for supported employment: 2–3 year outcomes of a randomized controlled trial. The American journal of psychiatry. 2007;164(3):437–41. pmid:17329468
- 57. Bond GR, Xie H, Drake RE. Can SSDI and SSI Beneficiaries With Mental Illness Benefit From Evidence-Based Supported Employment? Psychiatric Services. 2007;58(11):1412–20. pmid:17978250
- 58. Burke-Miller J, Razzano LA, Grey DD, Blyler CR, Cook JA. Supported employment outcomes for transition age youth and young adults. Psychiatric rehabilitation journal. 2012;35(3):171–9. pmid:22246115
- 59. Ellison ML, Klodnick VV, Bond GR, Krzos IM, Kaiser SM, Fagan MA, et al. Adapting Supported Employment for Emerging Adults with Serious Mental Health Conditions. The Journal of Behavioral Health Services & Research. 2015;42(2):206–22. pmid:25391357
- 60. Lones CE, Bond GR, McGovern MP, Carr K, Leckron-Myers T, Hartnett T, et al. Individual Placement and Support (IPS) for Methadone Maintenance Therapy Patients: A Pilot Randomized Controlled Trial. Administration and Policy in Mental Health and Mental Health Services Research. 2017. pmid:28213673
- 61. Mueser KT, Bond GR, Essock SM, Clark RE, Carpenter-Song E, Drake RE, et al. The effects of supported employment in Latino consumers with severe mental illness. Psychiatric Rehabilitation Journal. 2014;37(2):113–22. pmid:24912060
- 62. Frounfelker R, Teachout A, Bond GR, Drake RE. Criminal Justice Involvement of Individuals with Severe Mental Illness and Supported Employment Outcomes. Community Mental Health Journal. 2011;47(6):737–41. pmid:20680452
- 63. Serafini G, Pompili M, Borgwardt S, Houenou J, Geoffroy PA, Jardri R, et al. Brain changes in early-onset bipolar and unipolar depressive disorders: a systematic review in children and adolescents. Eur Child Adolesc Psychiatry. 2014;23(11):1023–41. Epub 2014/09/13. pmid:25212880; PubMed Central PMCID: PMC25212880.
- 64. McGurk SR, Mueser KT, Pascaris A. Cognitive training and supported employment for persons with severe mental illness: one-year results from a randomized controlled trial. Schizophrenia bulletin. 2005;31(4):898–909. pmid:16079391
- 65. McGurk SR, Mueser KT, Xie H, Welsh J, Kaiser S, Drake RE, et al. Cognitive Enhancement Treatment for People With Mental Illness Who Do Not Respond to Supported Employment: A Randomized Controlled Trial. The American journal of psychiatry. 2015;172(9):852–61. pmid:25998278
- 66. Mueser KT, Aalto S, Becker DR, Ogden JS, Wolfe RS, Schiavo D, et al. The effectiveness of skills training for improving outcomes in supported employment. Psychiatric services (Washington, DC). 2005;56(10):1254–60. pmid:16215191
- 67. Barreira PJ, Cohen M, bullet T, Gold PB, Holley D, Macias C, et al. Adapting Evidence-Based Interventions to Fit Usual Practice: Staff Roles and Consumer Choice in Psychiatric Rehabilitation. Psychiatr Q. 2010;81:139–55. pmid:20177969
- 68. Dixon L, Hoch JS, Clark R, Bebout R, Drake R, McHugo G, et al. Cost-Effectiveness of Two Vocational Rehabilitation Programs for Persons With Severe Mental Illness. Psychiatric Services. 2002;53(9):1118–24. pmid:12221310
- 69. Sinnott PL, Joyce V, Su P, Ottomanelli L, Goetz LL, Wagner TH. Cost-effectiveness of supported employment for veterans with spinal cord injuries. Archives of physical medicine and rehabilitation. 2014;95(7):1254–61. pmid:24486426
- 70. Fraser VV, Jones AM, Frounfelker R, Harding B, Hardin T, Bond GR. VR closure rates for two vocational models. Psychiatric rehabilitation journal. 2008;31(4):332–9. pmid:18407883