Next Article in Journal
Asian Americans’ Indifference to Black Lives Matter: The Role of Nativity, Belonging and Acknowledgment of Anti-Black Racism
Previous Article in Journal
Muslim Solidarity and the Lack of Effective Protection for Rohingya Refugees in Southeast Asia
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Child Safety Assessment: Do Instrument-Based Decisions Concur with Decisions of Expert Panels?

Research Institute of Child Development and Education, University of Amsterdam, Nieuwe Achtergracht 127, 1018 WS Amsterdam, The Netherlands
*
Author to whom correspondence should be addressed.
Submission received: 1 April 2021 / Revised: 3 May 2021 / Accepted: 4 May 2021 / Published: 11 May 2021
(This article belongs to the Section Childhood and Youth Studies)

Abstract

:
To make decisions on children’s immediate safety, child welfare agencies have been using safety assessment instruments for decades. However, very little research on the quality of these instruments has been conducted. This study is the first to inspect the concurrent validity of a child safety assessment instrument by comparing its outcomes to a different measure of immediate child safety. It was examined to what extent decisions of practitioners using a safety assessment instrument concur with decisions of child maltreatment expert panels. A total of 26 experts on immediate child safety participated in 7 expert panels, in which the safety of children as described in 24 vignettes was discussed. Additionally, 74 practitioners rated the same vignettes using the ARIJ safety assessment instrument. The instrument-based safety decisions of practitioners concurred for a small majority with the safety decisions reached by the expert panels (58% agreement). Expert panels often identified more types of immediate safety threats than practitioners using the instrument; however, the latter group more often deemed the child to be in immediate danger than the first group. These findings provide indications on how the instrument can be improved and give insight into how immediate safety decisions are made.

1. Introduction

Child welfare professionals frequently make crucial decisions on the safety of children in the families they supervise. For example, a professional needs to determine whether or not a child needs to be protected immediately, and if so, how the child can be protected. If a child is in immediate danger, it can be safeguarded in different ways, for example, by an in-home safety intervention, an out-of-home safety intervention, or placement in residential care. To make these safety decisions, child welfare agencies have been using safety assessment instruments for over three decades (DePanfilis and Scannapieco 1994). However, very little research on the quality of these instruments has been conducted, and instruments are most often only practice-based (Vial et al. 2020).
In an attempt to fill the gap in research on child safety assessment instruments and to develop an evidence-based as well as a practice-based instrument, we extensively examined the quality of a widely used Dutch safety assessment instrument (the ARIJ safety assessment instrument; Actuarial Risk Assessment Instrument Youth Protection; Van der Put et al. 2016). Studies on the reliability, content validity, and usability of the ARIJ safety assessment instrument have already been conducted (Vial et al. 2019a, 2019b). Complementary to these studies, the current study was the first to thoroughly examine the concurrent validity of a safety assessment instrument by comparing its outcomes to a different measure of immediate child safety.
Child welfare decision-making tools often comprise a safety assessment instrument and a risk assessment instrument. Safety assessment instruments help professionals to determine the child’s immediate safety. In other words, these instruments help professionals to determine whether a child has recently been harmed, if it is being harmed right now, or if it may be harmed in the immediate future (Hughes and Rycus 2006; Knoke and Trocme 2005). Immediate is often defined as within 24 to 72 h of the assessment (Vial et al. 2020). If a child is deemed to be in immediate danger, immediate measures need to be taken to safeguard the child. Risk assessment instruments help professionals to assess the risk for future child maltreatment, so that those children and families with a substantial risk for child maltreatment can be identified and this risk can be lowered by offering the caregivers treatment for the identified risk factors. These two assessment types are often mixed up and sometimes used interchangeably (Hughes and Rycus 2006). However, distinguishing safety assessment from risk assessment is important, since they serve different purposes that require different approaches.
In a recent literature review, the immediate safety aspects measured in internationally used safety assessment instruments were compared (Vial et al. 2020). This review revealed several immediate safety threats that are generally measured with these instruments, such as sexual abuse, neglect, physical abuse, domestic violence, refusing access to the child by caregivers, a caregiver’s substance abuse, and behaving toward the child in a predominantly negative way. These aspects are measured with the majority of the instruments, which supports their content validity. However, the quality of most of the included instruments has not been studied and should be examined first before we can draw inferences on the validity of these immediate safety threats.
The following studies on safety assessment instruments have been conducted. A focus group study examined the usability of a South African safety assessment instrument and reported positive first experiences of practitioners working with this particular instrument (Spies et al. 2015). The participants indicated that the instrument supported their decision making, gave direction to the substantiation of their child welfare decisions, empowered them as a professional, and enhanced their report writing. Another qualitative study examined the usability and content validity of the ARIJ safety assessment instrument (Vial et al. 2019a). Professionals generally considered the instrument to be useful, but they also provided recommendations for improvement. For instance, the wording of the (potential) outcomes of the instrument could be clarified. The professionals also indicated that several immediate safety threats were missing in the instrument and specifically mentioned emotional abuse, harm to the child inflicted by individuals from whom caregivers are unable or unwilling to protect the child, a caregiver’s psychiatric disorder that poses an immediate threat to the child, and a child’s psychiatric problems that pose an immediate threat to themselves. It was concluded that the content validity of the safety assessment instrument could be improved by adding these immediate safety threats to the instrument.
Three other studies have focused on the reliability of different safety assessment instruments. A Dutch safety assessment instrument (LIRIK) showed a low to fair interrater reliability of the individual items, and moderate interrater reliability of the overall safety outcome (Bartelink et al. 2017). Additionally, Orsi et al. (2014) studied the interrater reliability of the items of multiple American safety assessment instruments. The interrater reliability of the items varied largely from a low to substantial reliability. Further, the reliability of the ARIJ safety assessment instrument has been studied and was found to be moderate to high (Vial et al. 2019b).
Other studies have focused on the criterion validity of safety assessment instruments, in particular their predictive validity (Bartelink et al. 2017; Fuller and Wells 1998, 2003; Fuller et al. 2001; Wells and Correia 2012) and concurrent validity (Baird and Rycus 2004; Johnson 2004). However, these studies did not provide the information that is needed to draw conclusions on the quality of these instruments, as safety assessments were compared to measures of child safety in the future, such as child maltreatment recurrence, re-entry into out-of-home care, and risk assessments. Although these studies gave some indication that safety assessment outcomes predicted (future) child safety, they did not provide information on how well these instruments assessed immediate child safety.
Safety assessment instruments assess immediate child safety and should therefore be compared with other measures of a child’s immediate safety. As there are no safety assessment instruments available that have been studied thoroughly, we studied the concurrent validity of the ARIJ safety assessment instrument by comparing its outcomes with safety assessment outcomes produced by expert panels. In such panels, experts are presented with a vignette in which a child safety situation is described, and they are asked to reach consensus on the immediate safety of the child described in the vignette.
Three reasons can be put forward as to why safety assessments performed by expert panels can be an appropriate measure with which to compare ARIJ safety assessments. First, individual professionals are often advised to make decisions on a child’s safety in collaboration with a colleague, supervisor, or their (multidisciplinary) team rather than making decisions on their own. It is therefore not uncommon to discuss a child’s immediate safety with other professionals, which resembles experts reaching consensus in a panel. Second, the experts in these panels were expected to thoroughly discuss each vignette, which should result in a comprehensive argument as to why the child is considered to be safe or in immediate danger. All experts in a panel have to agree on the final decision, which encourages discussion between the experts. Third, researchers in different fields also use group decision methods to come to better decisions (e.g., Grofman et al. 1983; Schulz-Hardt et al. 2006).
The current study is an important contribution to studies on safety assessment instruments, as it is the first to study the concurrent validity of a safety assessment instrument by comparing it to another measure of immediate child safety. Additionally, it provides information on the quality of the ARIJ safety assessment instrument and the decisions made with this instrument, which is essential given the great impact these decisions have on the lives of children. In studying the concurrent validity, we not only examined the validity of the immediate safety outcomes, but also the validity of the individual immediate safety threats that are measured with the ARIJ safety assessment instrument. Thus, the aim of this study was to examine the extent to which decisions of individual practitioners using the ARIJ safety assessment instrument concur with the decisions of child maltreatment expert panels, which do not use an instrument, on immediate child safety. First, we compared the final safety decisions reached by practitioners using the ARIJ safety assessment instrument with the safety decisions reached by the expert panels. Second, the immediate safety threats identified by the practitioners using the ARIJ were compared with the immediate safety threats identified by the expert panels. As the expert panels did not identify the immediate safety threats in a structured manner, we used qualitative analyses to determine what immediate safety threats were identified by the expert panels.

2. Method

2.1. Participants

Twenty-six experts on immediate child safety (21 women, 5 men; Mage = 41 years, SD = 10) participated in seven expert panels. They were (child) psychologists or (child) social workers who worked at different agencies that provide child protection services, child and family support services, hotline (i.e., crisis) services, and community outreach services. On average, they conducted 7.6 child safety assessments each week (SD = 8.4, range: 0–35) and had 16 years of experience in youth services (SD = 9.4, range: 1.5–40).
Additionally, a total of 74 practitioners rated the vignettes using the ARIJ safety assessment instrument. These practitioners worked at a child and family support agency or a child protection agency. A description of these participants as well as more information on the ARIJ safety assessments can be found in Vial et al. (2019b).

2.2. Procedure

We used expert sampling to recruit participants for the expert panels, which is a purposive sampling method (Etikan et al. 2016). Participants were recruited by contacting both child welfare services and professionals in the social network of the authors of this study (for example, through social media). Professionals could only participate if assessing immediate child safety was an important aspect of their daily work, either because they conduct safety assessments themselves or because they supervise others conducting safety assessments. Our goal was to include four participants in six different expert panels and to recruit professionals that work at child protection services, child and family support services, hotline/crisis services, and community outreach services to obtain interdisciplinary assessments in each panel. Each panel assessed four different vignettes. During the study, a few experts cancelled the panel meetings, and therefore, one panel was split in two panels of two experts who held separate meetings.
In total, 26 experts on immediate child safety participated in the expert panels. Of the 24 vignettes, 20 vignettes were assessed by 1 expert panel and 4 vignettes were assessed by 2 expert panels (the split panel). All the experts were asked to assess 4 vignettes individually, and to return their assessments before the expert panels were formed. Of the experts, 73% indicated that they normally use an instrument or a structured method to assess a child’s immediate safety. However, they indicated that they did not actively apply this method to assess the vignettes in this study.
Each panel had one meeting. In these meetings, the vignettes were discussed one by one, and after discussion of each vignette, the panel had to decide on the immediate safety of the child. Each panel meeting lasted no longer than 1.5 h and took place in a meeting room at the university where this study was conducted. All meetings were led by the first author of this manuscript. Audio recordings were collected with the experts’ informed consent, and the experts received reimbursement as compensation for their time spent participating in this study.

2.3. Measures of the Immediate Child Safety

2.3.1. The ARIJ Safety Assessment Instrument

The ARIJ safety assessment instrument was developed to help professionals to determine whether a child is in immediate danger (Van der Put et al. 2016). The instrument consists of eight items that all describe a different immediate safety threat. A short description of the items of the ARIJ safety assessment instrument can be found in Appendix A. When an immediate safety threat is considered to be present, the child immediately needs to be safeguarded to prevent harm. Each of the items can be responded to with one of three categories: “Yes” (implying the threat described in the item is present), “No” (implying the threat described in the item is not present), and “Unknown” (implying there is insufficient information available at time of the assessment for a proper response). When at least one of the items is answered with “yes”, the instrument concludes that the child is in immediate danger. For the purpose of this study, the response categories “No” and “Unknown” were combined into a single category. In practice, professionals are often required to make a decision on a child’s immediate safety. If a professional decides it is unknown whether a safety threat is present or not, then no immediate safety measures will be taken at that time. Therefore, the safety conclusion of the instrument was “Safe” in cases where no immediate safety threats were deemed to be present in a vignette. Research has shown that the items and outcome of the ARIJ safety assessment instrument have a moderate to high interrater and intrarater reliability (Vial et al. 2019b).
The ARIJ safety assessment instrument is used for families and children that are already assigned to an agency, and both the ARIJ safety and risk assessments are performed as a part of the intake process. The ARIJ is not used as a gatekeeping assessment by agencies. The professionals who perform the assessment will also discuss the safety measures with the family, develop a safety plan, and perform further assessments to determine risk and family needs.

2.3.2. The Individual Expert Questionnaire

The questionnaire filled out by the experts in the panels started with a short explanation on the study procedures, after which eight questions followed on several characteristics of the expert, such as their work experience. Next, a definition of immediate child safety was given, which was followed by the subsequent presentation of four vignettes. For each vignette, experts were asked whether they thought the child as described in the vignette is safe or in immediate danger. The experts were asked to provide an explanation for their decision.

2.3.3. The Expert Panel

In the expert panels, the vignettes were discussed one by one. The experts had to agree on their final group decision on the child’s immediate safety.

2.4. Vignettes

A total of 24 vignettes were assessed, of which half were based on real cases of child and family support services (Vignettes 1–12), and the other half were based on real cases of child protection services (Vignettes 13–24). A fictional English vignette, which is similar to the vignettes used in this study, can be found in Appendix B. All the Dutch vignettes are available upon request. The child and family support vignettes had been created and used in a previous study by Bartelink et al. (2017). The vignettes described a variety of family compositions, social backgrounds, cultural backgrounds, child maltreatment forms (physical, sexual, emotional abuse, and neglect), and maltreatment severity levels. More information on these vignettes can be found in Bartelink et al. (2017). The child protection services vignettes were also created for a previous study (Vial et al. 2019b) and describe a variety of immediate safety problems in families. The vignettes were reviewed by practitioners of the child protection agency to assure they were representative of cases in their daily practice. Since the child protection agency usually handles more cases of children in immediate danger than the child and family support services, the vignettes designed for the former have a higher prevalence of possible immediate safety threats. An example of a vignette similar to the vignettes that were used in this study as well as more information on these vignettes can be found in Vial et al. (2019b).

2.5. Data Analyses

First, we compared the safety decisions reached by practitioners using the ARIJ safety assessment instrument with the safety decisions reached by the expert panels by calculating the percentage agreement. The three individual measures of immediate safety were all compared with each other (i.e., ARIJ assessments vs. expert panels; ARIJ assessments vs. individual expert assessments; individual expert assessments vs. expert panels). The calculated percentages show how often the measures of immediate safety were the same. For the ARIJ assessments and the individual expert assessments, we looked at the decision that was reached by the majority of the ARIJ or expert assessments. Thus, if in 80% of the ARIJ assessments, the child was deemed to be safe, the overall safety decision of the ARIJ assessments was set at “safe” for that particular vignette.
Second, the immediate safety threats identified by the practitioners using the ARIJ were compared with the immediate safety threats identified by the expert panels. For the ARIJ assessments, the prevalence of the response category “yes” showed which immediate safety threats were identified as present in a vignette. For the expert panels, the transcripts of the panel discussions were analyzed qualitatively to determine what immediate safety threats were identified as present in a vignette by the panels. For each vignette, the immediate safety threats were coded. All vignettes were coded by two research assistants who were carefully instructed to identify the immediate safety threats (i.e., the reasons the experts decided that the child was in immediate danger). Next, the first author of this manuscript merged the codes made by the assistants, identifying the immediate safety threats for each vignette. This same procedure was followed to identify the reasons of the expert panel to identify the child as safe.
Last, the immediate safety threats identified in the individual expert assessments were compared to the threats identified in the other two immediate safety measures. To do this, all questionnaires were coded by two research assistants and subsequently merged by the first author of this manuscript to identify the immediate safety threats mentioned for each vignette. The software program ATLAS.ti version 8 (ATLAS.ti Scientific Software Development GmbH, Berlin, Germany) was used for all qualitative analyses.

3. Results

3.1. Comparison of the Safety Decisions

Table 1 shows the prevalence of the safety decisions for (1) the ARIJ assessments, (2) the individual expert assessments, and (3) the expert panel assessments. The children described in the vignettes were more often determined as being in immediate danger in the ARIJ assessments (69%) than in the expert panel assessments (52%) and individual expert assessments (56%). For 58% of the vignettes (n = 14), the majority of the ARIJ safety decisions concurred with the safety decision of the expert panels. Both assessment types led to the conclusion that the child was in immediate danger for 10 vignettes and that the child was safe for 4 vignettes.
In 29% of the vignettes (n = 7), the ARIJ safety decisions differed from the safety decision by the expert panels. For six of these vignettes, the child was supposed to be in immediate danger by the majority of the ARIJ safety assessments, whereas the expert panels deemed the children to be safe, and for one vignette, the expert panel deemed the child to be in immediate danger, whereas the majority of ARIJ assessments deemed the child to be safe. In 13% of the vignettes (n = 3), either the ARIJ safety assessments (n = 2) or the expert panels (n = 1) were inconclusive on the child’s immediate safety (for practical reasons, one of the expert panels was split into two meetings; for one vignette, the safety decision differed between these two meetings).
For 83% of the vignettes (n = 20), the majority of the individual expert decisions concurred with the final expert panel’s decision. For the remaining 17% of the vignettes (n = 4), the individual expert decisions were inconclusive, because half of the individual experts judged the child to be safe and the other half judged the child to be in immediate danger.

3.2. Comparison of Immediate Safety Threats

Appendix A presents the identified immediate safety threats and the reasons a child was identified as safe, separately for the expert panel, individual experts, and ARIJ assessments. If the child was deemed to be in immediate danger by the expert panel, then the immediate safety threats identified by the experts are presented. If the child was deemed safe by the expert panels, then the explanations of the expert as to why the child was deemed to be safe are presented.
First, we looked into the immediate safety threats identified for the vignettes in which both the expert panel and the majority of the ARIJ safety assessments identified the child as being in immediate danger (Vignettes 1, 4, 6, 12, 13, 14, 15, 17, 19, and 21). In these vignettes, the safety threats identified in the ARIJ assessments and by the expert panels were similar. However, for most vignettes, the expert panels described more different types of safety threats than the ARIJ assessments. These additional threats were often related to the child’s behavior, the child’s vulnerability, mental health problems of the caregivers, the availability of the caregivers, and other family members (e.g., a brother).
Only in one vignette (Vignette 18) did the expert panel identify the child as being in immediate danger, whereas the majority of the ARIJ safety assessment decisions indicated that the child was safe. The expert panel mainly identified safety threats related to the child: “Child makes and shares her own nude pictures”, “Child runs away multiple nights at a time”, “Child does not want help”, “Child has contact with multiple men/boys”, “Child has money and expensive clothes/objects”, “Child uses substances”, and “Parents are not able to protect her”. The safety threats identified in the minority of the ARIJ safety assessments were “Physical abuse” (14%), “Sexual abuse” (14%), and “Parental availability” (14%).
In six vignettes (Vignettes 3, 9, 16, 20, 22, and 23) the majority of the ARIJ assessment decisions indicated that the child was in immediate danger, whereas the expert panels decided that the child was safe. For these vignettes, we describe briefly what the explanations of the experts were as to why they considered the children to be safe.
For Vignette 3, none of the identified safety threats in the ARIJ safety assessments were identified by the majority of the assessments. The most prevalent immediate safety threat in this vignette was “Parental availability” (46%). The expert panel mostly argued that the child was not in immediate danger due to factors related to the child’s father: “Father wants to learn and seems able to learn”, “Father asks for help”, and “Father knows that change is necessary”. Additionally, they explained that the child’s grandfather was able to help the family, and that the child goes to school and a sports club.
For Vignette 9, the majority of the ARIJ safety assessments described “Domestic violence” (73%) as an immediate safety threat, whereas the expert panel reasoned that “Parents seem to manage fairly”, and that “The child danger is chronic but not immediate”.
Notably, four immediate safety threats were identified by the majority of the ARIJ assessments for Vignette 16: “Child abduction and honor-related violence” (100%), “Domestic violence” (100%), “Physical abuse” (67%), and “Parental availability” (67%). However, the expert panel described the child as being safe. Most of the reasons of the expert panel as to why the child was considered to be safe were related to the mother (e.g., “Mother can reflect on her own behavior” and “Mother recognizes her shortcomings, which caused danger to her child”). Additionally, they described the current living situation as protective: “Mother and child currently stay in a safety house”.
The ARIJ assessments identified “Physical abuse” (89%) as an immediate safety threat for Vignette 20. In contrast, the expert panel reasoned that the child was not in immediate danger, because “The child has no injuries”, “The child is 16 years old”, and “The incident was not recent”.
For Vignette 22, “Psychiatric problems” (75%) was identified as an immediate safety threat by the majority of the ARIJ assessments. The expert panel argued that the child was not in immediate danger because, “The problems are chronic, and not immediate”, “The parents recognize the brother’s disorder (which is harmful to the child)”, “Their social network is involved”, and “The parents want help for their own problems”.
Half of the ARIJ assessments identified “Parental availability” (50%) as an immediate safety threat in Vignette 23. However, the expert panel considered the child to be safe, because “There is a social network available”, “The unofficial foster parent indicated that the child is doing fine at her place”, “Child still has a place to life”, “Father is involved with the child”, and because of “The child’s age”.
Finally, another interesting vignette is Vignette 24, as half of the ARIJ safety assessments indicated that the child was in immediate danger, whereas the other half of the ARIJ assessments identified the child as safe. The most prevalent identified immediate safety threat in the ARIJ assessments was “Parental availability” (40%), followed by “Physical abuse” (30%). The expert panel decided that the child was in immediate danger and identified the following immediate safety threats: “The child’s grandfather hits mother and child”, “Grandfather is unpredictable”, “Grandfather has Alzheimer’s disease”, “The child has behavioral problems”, “Child’s behavioral problems increase the chance that grandfather hits him”, and “Child assaulted someone”.

4. Discussion

The safety decisions reached by practitioners with the ARIJ safety assessment instrument concur with the safety decisions reached by the expert panels for a small majority of the cases (58%). Thus, 42% of the safety decisions reached by the ARIJ assessments did not concur with the expert panel decisions. In these cases, the ARIJ safety assessments often deemed the child to be in immediate danger, whereas the expert panels deemed the child to be safe. The immediate safety threats identified across the two assessment types were often comparable. However, the expert panels often identified more types of immediate safety threats than the practitioners using the assessment instrument. In general, the following threats were added to the threats mentioned in the ARIJ: threats related to the child’s behavior, the child’s vulnerability, other family members (e.g., a brother), and mental health problems of the caregivers. These added safety aspects are also measured in most internationally used safety assessment instruments (Vial et al. 2020). Moreover, a previous study on the content validity of the ARIJ safety assessment instrument showed that these threats should be included in the instrument (Vial et al. 2019a). As these immediate safety threats are not measured in the ARIJ safety assessment instrument, it is important to improve the instrument by adding these threats.
Interestingly, the expert panels also mention immediate safety aspects that can often be classified as risk factors. Safety and risk assessment instruments often assess factors that describe very similar problematic behaviors of caregivers. However, they need to be assessed differently in these different assessment types. This, for instance, applies to substance abuse of caregivers. In a risk assessment, this factor should be assessed as present if a caregiver uses substances problematically. In a safety assessment, however, this factor must only be assessed as present if the caregiver’s substance abuse causes an immediate safety threat to the child. The experts sometimes mention factors without explaining how they pose an immediate threat to the child. For example, for Vignette 1, the experts mentioned two factors as safety threats (i.e., “Mother suffered from child maltreatment as a child” and “Mother’s boyfriend went to prison”), whereas the experts seemed to use these factors as indicators of the severity of problems in the family rather than safety threats.
Also noticeable is that experts weighed child characteristics in their assessments of the child’s immediate safety, such as the child’s age or how well the child is functioning. The experts reasoned in some cases (e.g., Vignette 23) that a child is not in immediate danger, as it was relatively old (e.g., 16 years old) or because it seemed to function normally. This type of reasoning can be problematic, as studies on incident reports in the Netherlands and the United Kingdom have shown that practitioners tend to underestimate immediate safety threats if the child does not have any (behavioral) problems or does not show any signs of abuse (Trench and Griffiths 2014; Health and Youth Care Inspectorate 2016). Further, aspects related to the capacities of caregivers are also often mentioned by the experts as to why the child is not in immediate danger. Especially caregivers who are willing to change their behavior are mentioned often by the experts (e.g., Vignette 16). This may also be problematic, as risk assessment research showed that risk factors have a larger impact on child outcomes than protective factors (Luthar and Goldstein 2004; Miller et al. 1999; Vanderbilt-Adriance and Shaw 2008a, 2008b; Van der Put et al. 2016). Protective capacities of caregivers may not (always) be able to mitigate immediate safety threats. Thus, even though aspects related to the child and caregiver’s capacities are often measured with safety assessment instruments (Vial et al. 2020), it is debatable whether these aspects should be assessed in this manner in safety assessments. Future research should specifically examine the impact of child characteristics and caregiver capacities on the quality of safety assessments.
An underlying assumption of this study is that a group decision is better than an individual decision, which can be criticized. In this study, the final decision of the expert panel was very often the same as the final decision of the majority of the individual experts. However, in some panels, there were experts who held a strong opinion, which had a large impact on the final decision of the panel. In Vignette 20, for example, three experts decided that the child was in immediate danger in their individual assessment. However, the final decision of the expert panel was that the child was safe, which was in line with the decision of only one expert. Noticeable was that particularly the experts who worked at the domestic violence and child maltreatment hotline crisis services had a large impact on the final decisions of the expert panels. In the discussion of some vignettes, it was even noticed that the other experts seemed to avoid a discussion, as the crisis services experts were seen as an authority on the subject, even though all panel members had dealt with the safety of children on a daily basis. Therefore, not all vignettes were discussed as extensively as would be desirable. That the crisis services professionals were seen as an authority could also negatively influence decision making in practice, as their authority could undermine the views of other professionals working on a case. This is especially problematic because the crisis services also provide consultation to anyone worried about a child.
Important to note is that the experts in the panels worked at different agencies, which do not use exactly the same definitions of immediate child safety. This was most apparent for the experts working at the crisis services. There, the time that has passed since the last incident has a large impact on decisions, as this period is also an important aspect of the assessment instrument that is normally used by experts working at the crisis services. Additionally, in the crisis services instrument, a distinction is made between immediate safety problems and chronic safety problems, which became apparent in the explanations of the experts working at the crisis services (e.g., “Child danger is chronic but not immediate”). On the other hand, the time that has passed since the last incident and the chronicity of the safety problems was much less relevant for experts working at other agencies. Not every panel did have an expert working at the crisis services, and this may have caused differences between the final panel decisions. In future research, it would be interesting to use more homogeneous expert panels and to compare how these professionals with different backgrounds assess child safety.
It is important to mention several limitations of this study. As this is a vignette study, the professionals do not need to act on the decisions they reached. In practice, stating that a child is in immediate danger actually means that the professional should come into immediate action and safeguard the child. Given the large impact this decision has on a child, a professional could in reality be more reluctant to state that a child is in immediate danger. This effect is supported by the fact that the children described in the vignettes that were used in this study were often deemed to be in immediate danger, whereas in practice, these same children were not deemed to be in immediate danger as much, as we varied the severity of the cases. Additionally, for many vignettes, multiple immediate safety threats were identified by de practitioners, whereas in practice, it is rare that multiple immediate safety threats are deemed to be present in a single family. The practitioners who assessed the vignettes in this study may not have taken into account that—in reality—a child needs to be safeguarded immediately whenever a safety threat is assessed as present, even though this was described in the questionnaire.
Another limitation of a vignette study is the rather low level of ecological validity. It was, for example, not possible for the participants to obtain more information if they felt that they needed more information to decide on the child’s safety. Future research should try to study the concurrent validity of an instrument for cases that are actually being handled in practice. Additionally, an extensive investigation of the immediate child safety established by a multidisciplinary team of experts, such as a pediatrician, a psychologist, a social worker, etc., using multiple sources of information on the child and its living environment should be used as measure of immediate safety in future research. For this type of research, ethical limitations should be taken into account, as a comprehensive investigation is needed for children who are in immediate danger, but also for children who are not in danger.
A final limitation is that the majority of the experts use an instrument or a structured method on a daily basis to assess children’s immediate safety. Even though the experts did not use the instruments they are familiar with in the current study, their conclusions could have been influenced by these instruments. One expert working at the hotline services even explicitly disclosed all criteria described in the instrument that this expert was very familiar with. For further research, it would be interesting to compare the outcomes of these different safety assessment methods for the same cases, even though these methods have not been validated yet.
Despite these limitations, this study gives important indications on how the ARIJ safety assessment instrument needs to be improved. Some of the immediate safety threats identified by the experts should be added to the instrument: threats caused by the child’s behavior, and threats caused by other family members (e.g., a brother). Adding these threats to the instrument will most likely improve its validity and help to prevent cases where professionals overlook these immediate safety threats in their assessments.
This study also shows that there is still much room for improvement of the assessments performed in practice. For instance, professionals could be better informed on how safety assessment differs from risk assessment. The results showed that even very experienced professionals struggled to keep these two assessment types apart. Additionally, it is important to align different agency types when it comes to the definitions and operationalizations of child safety that are used in daily practice across agencies. Between expert panel members, there were large discrepancies in immediate child safety definitions. Much more work is required to achieve more unity on child safety decisions.
Finally, the decision on a child’s safety is only the first in line of many decisions that practitioners need to make in order to safeguard a child. Reliable and valid decisions on children’s safety need to be followed by effective and appropriate safety measures described in a safety plan. Further research should study whether using an instrument improves the development of an effective safety plan.

5. Conclusions

The current study was the first to examine the concurrent validity of a safety assessment instrument by comparing its outcomes to another measure of the immediate child safety. This type of research is essential to determine the quality of safety assessment instruments and the accuracy of decisions that are made with such an instrument. The decisions made with the ARIJ safety assessment instrument concurred for a small majority with the expert panel decisions. The results provide important indications on how the instrument can be improved, so that the likelihood of professionals missing relevant threats in their assessments reduces. However, deciding on the presence of immediate safety threats remains a clinical decision that is susceptible to bias. The ARIJ safety assessment instrument helps to structure this decision, but merely implementing an instrument such as the ARIJ is not sufficient. Many steps need to be taken to achieve more consensus in safety decision making. Training and educating professionals on how to thoroughly and properly perform a safety assessment and conduct an interview is highly needed. Also important are adequate supervision and the realization of safe work environments.

Author Contributions

Conceptualization, C.V.d.P., M.A., and A.V.; methodology, A.V., C.V.d.P., and M.A.; software, A.V.; validation, A.V., C.V.d.P., and M.A.; formal analysis, A.V.; investigation, A.V.; resources, A.V.; data curation, A.V.; writing—original draft preparation, A.V.; writing—review and editing, A.V., M.A., C.V.d.P., and G.J.S.; supervision, M.A. and C.V.d.P.; project administration, C.V.d.P.; funding acquisition, C.V.d.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Dutch organization for Health research and Development (ZonMW), grant number: 729300108.

Data Availability Statement

Not applicable.

Acknowledgments

We thank all the professionals that participated in this study. Additionally, we would like to thank Franziska Yasrebi-de Kom and Isidora Stolwijk for coding all the safety assessments.

Conflicts of Interest

Annemiek Vial declares that she has no conflict of interest. Mark Assink, Geert Jan Stams, and Claudia van der Put were involved in the development of the ARIJ Safety assessment instrument. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Appendix A

Table A1. Immediate safety threats and reasons the child was identified as safe in the expert panel, individual expert, and ARIJ assessments.
Table A1. Immediate safety threats and reasons the child was identified as safe in the expert panel, individual expert, and ARIJ assessments.
Expert PanelIndividual Experts 1ARIJ % Yes
Vignette 1
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (75%)
Overall safety decision:
Immediate danger (62%)
(lack of clarity on) Domestic violence (caused by mother’s boyfriend)
Mother’s boyfriend is violent when drunk
Mother’s boyfriend went to prison
Mother’s boyfriend has many (mental health) problems
Brother causes many problems in the home situation
Brother causes suffering of the child
Mother can’t protect the child from his brother
Mother plays down the problems
Mother asks too much from the child
Mother rejects services
Mother does not want to talk
Mother lacks understanding of the severity of the problems
Mother has unresolved trauma
Mother suffered from child maltreatment as a child
Mother burdens the child with adult problems
Chronic problematic child-rearing situations
Emotional unsafety
The child development in danger
Multiple life events in a short period
Mother is not capable of making a change
Limited social network
The child feels responsible for his mother
Mother has mental health problems
The child takes care of mother
Physical abuse23
Sexual abuse0
Neglect23
Child abduction and honor-related violence0
Parental refusal of immediate care8
Psychiatric problems0
Domestic violence8
Parental availability54
Vignette 2
Overall safety decision:
Safe
Overall safety decision:
Safe (100%)
Overall safety decision:
Safe (100%)
Father indicates that he does not hit anymore
Father is open for conversation
Father admitted that he used to hit the child
Parents want help
Many protective factors are present
Parents want help (with the child’s behavior)
Parents are open about how they punish the child
Father indicates that he does not hit anymore
Father knows that hitting a child is not good
Father admitted that he used to hit the child
Negative effects of the problems on the child are not visible
Parents are protective
Parents sufficiently supervise the child
Physical abuse0
Sexual abuse0
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems0
Domestic violence0
Parental availability0
Vignette 3
Overall safety decision:
Safe
Overall safety decision:
Safe (75%)
Overall safety decision:
Immediate danger (54%)
Father had a positive childhood
Father thinks positively of his child
Father knows he lacks the knowledge on how to raise his child
Father knows change is necessary
Father his intelligence is above average
Father wants to learn and seems able to learn
Father asks for help
Many protective factors are present
Grandfather helps
Child goes to school and a sports club
Grandfather is involved
Father thinks positively about himself and his daughter
Father had a good childhood
Father is open to help
Father his intelligence is above average, so he is probably able to learn
Sports club does not report any problems
No immediate incident
The problematic situation existed for a longer period
Physical abuse0
Sexual abuse0
Neglect8
Child abduction and honor-related violence0
Parental refusal of immediate care8
Psychiatric problems0
Domestic violence0
Parental availability46
Vignette 4
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Immediate danger (86%)
Physical fights between father and sons
Father has substance abuse problems
Parents are not able to care for the child
Parents can’t support children
Mother has physical problems
Mother is not mentally resilient
The child wants to leave the family
The child has a low IQ
The child has behavior problems
The child has been in out-of-home placement twice
The child has problems in all aspects of his life
The child is vulnerable
The child is on multiple waiting lists for care
Parents possibly have a mental disability
Mother does not speak Dutch
Parents suffered from child maltreatment as a child
Parents are skeptical about care
Mother is not able to disagree with/resist father
Parents do not see that their child needs care
There is no care for the child’s delinquent behavior
Emotional neglect
Physical abuse71
Sexual abuse0
Neglect21
Child abduction and honor-related violence0
Parental refusal of immediate care7
Psychiatric problems0
Domestic violence0
Parental availability36
Vignette 5
Overall safety decision:
Safe
Overall safety decision:
Safe (100%)
Overall safety decision:
Safe (54%)
Child does not want to run away
Last incident was two weeks ago
The child’s age
Child wants to change
Child recognizes the problems
Parents are available
Parents are involved
Child goes to school
Sexual abuse was stopped
Many protective factors are present (such as a social network)
No sexual abuse
No neglect
No physical abuse
Parents are involved and available
Social network available
Sexual abuse was stopped
Many protective factors
The child goes to school
The child lives at home
Family members are involved with the family
Parents supervise the child sufficiently
Physical abuse8
Sexual abuse23
Neglect8
Child abduction and honor-related violence0
Parental refusal of immediate care39
Psychiatric problems8
Domestic violence0
Parental availability8
Vignette 6
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Immediate danger (100%)
The child is being hit
Domestic violence
Father has substance abuse problems
Father is absent
Mother is incapable to protect the child
Psychological violence towards the children
The child has behavioral problems
The child runs away and without anyone knowing where she is
The child has no connection to peers
The child burdened with adult problems
The child is wary and aggressive
The child stabbed a peer with a scissor
The child does not accept authority
The child is young
Parents ask too much from the child
Chronic unsafety
The child cannot count on her parents
The child witnesses the fights between parents
Father is unavailable
Physical abuse85
Sexual abuse8
Neglect46
Child abduction and honor-related violence0
Parental refusal of immediate care8
Psychiatric problems0
Domestic violence8
Parental availability100
Vignette 7
Overall safety decision:
Safe
Overall safety decision:
Safe (100%)
Overall safety decision:
Safe (80%)
Child is protected
Parents are loving
Negative effects of parents’ disagreement on the child are not visible
Child does well in school
No physical violence during a pregnancy
No injuries to a very young child
No strangling
No sexual abuse
No injuries that need medical care
No weapons involved
No severe physical neglect
The child’s life is not in danger
Physical abuse0
Sexual abuse0
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems0
Domestic violence13
Parental availability7
Vignette 8
Overall safety decision:
Safe (50%) 2
Overall safety decision:
Inconclusive (50%)
Overall safety decision:
Immediate danger (93%)
Grandmother is available
Grandmother provides a stable living situation
There are parenting arrangements in placePhysical abuse73
Sexual abuse0
Neglect13
Child abduction and honor-related violence0
Parental refusal of immediate care13
Psychiatric problems0
Domestic violence13
Parental availability73
Vignette 8
Overall safety decision:
Immediate danger (50%) 2
Overall safety decision:
Inconclusive (50%)
Overall safety decision:
Immediate danger (93%)
Father uses physical violence (uncertain if this is also towards the children)
Father is verbally violent
Father is dangerous
Father has narcissistic personality problems
Father is very unreliable
The children stay at father every weekend
Mother is absent
Father is absent
Mother is not concerned about the needs of the children
The child has behavioral problems
Mother suffered from child maltreatment as a child
The child mimics the (violent) behavior of the fatherPhysical abuse73
Sexual abuse0
Neglect13
Child abduction and honor-related violence0
Parental refusal of immediate care13
Psychiatric problems0
Domestic violence13
Parental availability73
Vignette 9
Overall safety decision:
Safe
Overall safety decision:
Safe (100%)
Overall safety decision:
Immediate danger (80%)
Parents seem to manage fairly
Child danger is chronic but not immediate
The child danger is structural but not immediatePhysical abuse7
Sexual abuse0
Neglect7
Child abduction and honor-related violence0
Parental refusal of immediate care7
Psychiatric problems0
Domestic violence73
Parental availability33
Vignette 10
Overall safety decision:
Safe
Overall safety decision:
Safe (100%)
Overall safety decision:
Safe (94%)
Child has a strong bond with mother
Chronic problems, not immediate problems
No immediate danger to the child development
Parents are available
Mother is motivated to change
Mother recognized the child’s need
Social network is available
No physical violence
No domestic violence
No psychological violence
No sexual abuse
It is not necessary to act within two hours
Mother is emotionally and physically available
No physical violence
No domestic violence
No emotional violence
No sexual abuse
Mother acknowledges the problems
Mother is protective
Mother wants to help the child
Grandmother is involved
Mother provides basis care
Mother recognizes the child’s needsMother had therapy, which helped
Physical abuse0
Sexual abuse6
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems0
Domestic violence0
Parental availability0
Vignette 11
Overall safety decision:
Safe
Overall safety decision:
Safe (100%)
Overall safety decision:
Inconclusive (50%)
Concerns about the child’s safety, but they are not immediate
Regular services are sufficient at this time
The child’s life is not in danger
No physical danger
The child has behavior problems, but it is unknown what causes this
Physical abuse17
Sexual abuse0
Neglect17
Child abduction and honor-related violence8
Parental refusal of immediate care0
Psychiatric problems17
Domestic violence0
Parental availability33
Vignette 12
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Immediate danger (100%)
Mother has a very negative view of her son
Mother views the child as the cause of her incapability
Mother is scared to be abusive towards her children
Mother hit the child before
Mother has mental health problems
Mother is depressed
Mother is unstable
Mother suffered from child maltreatment as a child
Mother feels like the child does not want to go home
Mother is struggling
The child frequently has bruises
The child is isolated
The child is bullied
The child does not want to go home
The children are completely dependent on mother
The child shows internalizing behavioral problems
Father is not often home
Very little social control
The child is young
Mother admitted that she hit the child
There is no other adult that can protect the children
Mother is isolated
Physical abuse75
Sexual abuse0
Neglect8
Child abduction and honor-related violence0
Parental refusal of immediate care17
Psychiatric problems0
Domestic violence0
Parental availability75
Vignette 13
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Immediate danger (90%)
Domestic violence
The child had bruises before
Previous hotline report
Father is absent
Young child that is completely dependent on mother
Mother has attachment problems with her son
Stressful events
Mother has mental health problems
Mother has substance abuse problems
No social network available
Mother attempted suicide in the past
Mother is not sufficiently emotionally available
Mother has financial problems
Mother has personal problems
Police were involved after domestic violence
The child witnesses violence
Nobody sees how much mother is drinking, there is no supervision
Mother drinks alcohol when she is stressed
Mother admitted that she hit the child
Father is absent during the day and therefor mother is the one in charge
Parents have relationship problems
Physical abuse50
Sexual abuse0
Neglect20
Child abduction and honor-related violence10
Parental refusal of immediate care10
Psychiatric problems10
Domestic violence50
Parental availability50
Vignette 14
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Immediate danger (90%)
The child witnesses domestic violence
Physical violence between parents
Verbal violence between parents
Basic (physical) care is neglectful
Parents are limited available
Father has a substance abuse problem
Father stopped substance abuse treatment
Father returns to old habits
Father does not want help
Parents suffered from child maltreatment as a child
Many risk factors are present
Limited social control
Lack of hygiene
The child is in physical danger just by being in the house
The child is young
Parents mental health (is unknown)
Parents lack understanding of the severity of the problems
Financial problems
Physical abuse20
Sexual abuse0
Neglect60
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems0
Domestic violence80
Parental availability20
Vignette 15
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (75%)
Overall safety decision:
Immediate danger (100%)
Mother is aggressive
Instable child-rearing situation
Mother is dependent on father
Mother is inconsistent in her behavior
Mother has personality problems
Father indicates that the mother was physically violent towards him multiple times
Severe violent incidents in the presence of the children
Father does not listen to the mother
High conflict divorce which has been going on for years
No improvements despite recent care
Young children
Mother indicates she cannot handle the care of her children
Parents are both financially unstable
Parent both have unstable living situations
Parents are not enough concerned with the needs of the children
The child talks about suicide
Father does not want to participate in the care
Parents are occupied with their problems
All three children show behavioral problems
The youngest child shows speech development delay
The children are young and dependent of their parentsPhysical abuse50
Sexual abuse0
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems67
Domestic violence100
Parental availability33
Vignette 16
Overall safety decision:
Safe
Overall safety decision:
Safe (75%)
Overall safety decision:
Immediate danger (100%)
Mother and child currently stay in a safety house
Mother recognizes problems
Mother can reflect on her own behavior
Mother asks for help (indirectly)
Mother is open about her inability to care for the child
Mother recognizes her shortcomings, which cause danger to her child
Mother is open about her incapacity to care for the child
Only one incident occurred
Child welfare is involved
Mother and child currently stay in a safe house
Mother can reflect on her parenting skills
Mother indicates that parenting is hard for her
Mother is protective
Mother is loving
Physical abuse67
Sexual abuse0
Neglect33
Child abduction and honor-related violence100
Parental refusal of immediate care0
Psychiatric problems0
Domestic violence100
Parental availability67
Vignette 17
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Immediate danger (83%)
The child indicated that she has been sexually abused by peers (repeatedly)
Possible child prostitution
The child shows self-harming behavior
The child still sees the peers that sexually abused her
The child shows behavioral problems at school
The child shows risky behavior around boys/men
Parents are incapable to protect the child (from itself)
Parents have negative thoughts on the child
Parents are unavailable
Parents have a mental disability
The child often does not go to school
The child has unresolved trauma
Parents lack understanding of the severity of the problems
Parents do not provide basic care
The child has problems in multiple aspects of her life
There are no adults to support the child (when needed)
The child is not in care
Parents were maltreated as a child
Physical abuse0
Sexual abuse67
Neglect17
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems67
Domestic violence0
Parental availability17
Vignette 18
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Safe (71%)
The child makes and shares her nude pictures
The child runs away multiple nights at a time
Unclear were the child is when she runs away
The child does not want help
The child has contact with multiple men/boys
The child has behavioral problems
The child has money and expensive clothes/objects
The child uses substances
Mother is not capable to set rules and boundaries
Father allows the child to drink
Parents are not able to protect her (from sexual abuse)
The child is vulnerable
The child does not go to school
The child can respond aggressivelyPhysical abuse14
Sexual abuse14
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems0
Domestic violence0
Parental availability14
Vignette 19
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (100%)
Overall safety decision:
Immediate danger (67%)
(possible) Ongoing sexual abuse
Family member (preparator) who previously sexually abused the child lives close
The child is young
The child has panic attacks
Parents have different explanations on child’s panic attacks
The child has symptoms related to medicine
Parents are occupied with their problems
Parents are occupied with the problems between them
Parents disagree on (the severity of) the problems
Mother has unresolved trauma
Mental health problems of parents
Father has an autism spectrum disorder
Chronic involvement of services
The child has ongoing physical injuries
Perpetrator had a mental disability
Father does not recognize the risk of the perpetrator
Father has a mental disability
There are signs that the child still feels unsafe
Parents do not have explanations for the physical injuries
Parents do not know enough about the basic need for a child to be able to develop
Many conflicts between the parents
Physical abuse22
Sexual abuse56
Neglect11
Child abduction and honor-related violence0
Parental refusal of immediate care11
Psychiatric problems22
Domestic violence0
Parental availability22
Vignette 20
Overall safety decision:
Safe
Overall safety decision:
Inconclusive (50%)
Overall safety decision:
Immediate danger (100%)
Child is 16 years old
The incident was not recent
Child has no injuries
Danger to the child today is equal to the danger tomorrowPhysical abuse89
Sexual abuse11
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care22
Psychiatric problems0
Domestic violence0
Parental availability33
Vignette 21
Overall safety decision:
Immediate danger
Overall safety decision:
Immediate danger (80%)
Overall safety decision:
Immediate danger (100%)
The child witnessed domestic violence multiple times
Parents have serious conflicts
Father has an attention deficit disorder
Mother suffered from child maltreatment as a child
Mother has borderline personality problems
Chronic problematic situation
No (family) support available in Vignette of crisis
Mother is dependent on father
Mother indicated that she cannot handle the care for her child alone
Mother is instable
Parents are not always available
Structural care is lacking
The child has multiple addresses due to the fight between parents
Instable family
Young child
Withdrawal of medical care
Parents are emotionally unavailable due to personal problems
Parents are physically unavailable due to personal problems
Parents have mental health problems
Previous care did not lead to improvements
Parents have relationship problems
Parents do not know what their child needs
The child is dependent on its parents
Parents do not vaccinate the child
The child cannot protect itself
Physical abuse0
Sexual abuse0
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care30
Psychiatric problems0
Domestic violence100
Parental availability60
Vignette 22
Overall safety decision:
Safe
Overall safety decision:
Inconclusive (50%)
Overall safety decision:
Immediate danger (88%)
Network is involved
Parents recognize brother’s disorder (which harms the child)
Parents want help for their own problems
Parents mentioned their own concerns
Parents know what their son needs
Problems are chronic, not immediate
Child is protected
Parents are loving
Child safety problems are chronic
Parents acknowledge the child’s disorder
Parents know what the child needs
Parents ask for help
Network is involved
Physical abuse38
Sexual abuse0
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care13
Psychiatric problems75
Domestic violence0
Parental availability38
Vignette 23
Overall safety decision:
Safe
Overall safety decision:
Inconclusive (50%)
Overall safety decision:
Immediate danger (60%)
Father is involved with the child
There is a social network available
Negative effects of mother’s behavior on the child are not visible
The child’s age
The unofficial foster parent indicated that the child is doing fine at her place
Child has a residence permit
Child is not abused by the unofficial foster parent
Child still has a place to life
No urgent matters
The child currently stays at a safe space
Father is involved
The child’s age
The child has a residence permit
The unofficial foster parent protects the child
The child lives with an unofficial foster parent
The child does not stay at mother’s place anymore
Physical abuse40
Sexual abuse0
Neglect40
Child abduction and honor-related violence0
Parental refusal of immediate care10
Psychiatric problems0
Domestic violence0
Parental availability50
Vignette 24
Overall safety decision:
Immediate danger
Overall safety decision: Immediate danger (60%)Overall safety decision:
Inconclusive (50%)
Grandfather hits mother and child
Grandfather is unpredictable
Grandfather has Alzheimer’s disease
The child has behavioral problems
The child’s behavioral problems increase the chance that grandfather hits him
The child threatened others with a knife
The child assaulted someone
Mother not able to protect the child
Mother is instable
Problems in multiple life areas
Grandfather is not able to control himselfPhysical abuse30
Sexual abuse0
Neglect0
Child abduction and honor-related violence0
Parental refusal of immediate care0
Psychiatric problems10
Domestic violence0
Parental availability40
1 We only report the immediate safety threats that were mentioned in the individual expert assessments but were not mentioned in the expert panels in this column. All the aspects mentioned in the expert panels were mentioned in the individual expert assessments. 2 Due to practical reasons this expert panel was split into two different meetings with other experts.

Appendix B. A Fictional Vignette in English

The Smith family was referred to child protection by the emergency hotline after a recent report. The family consists of a father, a mother, and their son C. (11 years old) and daughter F. (10 years old). The parents are divorced. Mother lives with both children in a single-family home, in which each child has their own room. Father recently moved into a small apartment, after moving around a lot. The children don’t have their own room at their father’s place, but father soon hopes to find a bigger place to live with more space for the children. Mother works as a cashier two days a week and father works full-time as a plumber. The father’s intelligence was previously assessed and yielded an IQ of 77. The IQ of mother is unknown. Both parents grew up in a rather stable home environment.
C. was diagnosed with a bipolar disorder and autism. It is also known that C. has an IQ of 85. He has a need for structure, clear boundaries, repetition, consistency, and one-on-one directions (mother refers to handling her son as ‘staying on top of it’). C. struggles with handling changes, implying that minor changes in the structure of his daily life can cause sensory overload. Both parents have indicated that these sensory overloads can lead to temper tantrums, which happen quite regularly. A temper tantrum of C is characterized by hitting family members, screaming, banging his head against the wall, and destroying things, such as breaking windows. According to the parents, it is impossible to have contact with C during these temper tantrums. Parents have indicated that it is very important to remain calm and to not raise your voice during one of these tantrums. Taking C. outside for a walk may sometimes help calming him down. According to the parents, timing is important, because C. may run away.
F. (C.’s sister) suffers a lot from the disrupting behavior of her brother. It makes her sad, and she regularly expresses her sadness. According to herself, she fights a lot with C, and she is regularly confronted with C.’s temper tantrums. Because C’s disrupting behavior also happens at night, she regularly sleeps in the hall at her father’s place.
Throughout the years, the parents have begun several parenting programs to improve their parenting skills for handling C’s behavior. However, they have repeatedly decided to drop out of these programs. For example, both parents have terminated parental guidance and psycho-education on their own initiative and without discussing her reasons for dropping out with professionals. The parents felt that the program was too intense, and that they were too busy to follow the program. Consequently, interventions have not been successful, and the behavior of C remains problematic.
Mother says that she is worn out. Both parents indicate that they are suffering from C.’s behavior and that they have multiple parenting questions. They don’t know how to successfully cope with C. and feel helpless. Father says he is experiencing depressed feelings.
Since the divorce two years ago, the communication between the parents has been difficult. They strongly distrust each other. The parents hold onto old grudges against each other, for instance regarding the (ex-)in-laws, causing heated fights. Recently, neighbors have reported verbal domestic violence to the emergency hotline. It seems that parents lose sight of C.’s and F.’s needs because of these conflicts. In addition, the parents do not agree on the upbringing of C., and they seem negatively influenced by the behavior of their son. Because of C.’s disrupting behavior, both parents have rather little attention for F.
Mother has a sister and only one good friend. The maternal grandfather and grandmother live in close proximity to the mother of C. and F., and both try to be supportive. The father of C. and F. receives somewhat support from his parents and two sisters. C. and F. sometimes stay with their paternal aunt and C. sometimes plays with his nephews. F. has two best friends, and she sometimes stays over at their places.

References

  1. Baird, Christopher, and Judith S. Rycus. 2004. The contribution of decision theory to promoting child safety. APSAC Advisor 16: 2–10. [Google Scholar]
  2. Bartelink, Cora, Leontien de Kwaadsteniet, Ingrid J. ten Berge, and Cilla L. M. Witteman. 2017. Is it safe? Reliability and validity of structured versus unstructured child safety judgments. Child & Youth Care Forum 46: 745–68. [Google Scholar]
  3. DePanfilis, Daine, and Maria Scannapieco. 1994. Assessing the safety of children at risk of maltreatment: Decision-making models. Child Welfare 73: 229. [Google Scholar] [PubMed]
  4. Etikan, Ilker, Sulaiman Abubakar Musa, and Rukayya Sunusi Alkassim. 2016. Comparison of convenience sampling and purposive sampling. American journal of theoretical and applied statistics 5: 1–4. [Google Scholar] [CrossRef] [Green Version]
  5. Fuller, Tamara L., and Susan J. Wells. 1998. Illinois Child Endangerment Risk Assessment Protocol: A Technical Report Concerning the Implementation. Available online: https://cfrc.illinois.edu/pubs/rp_19980814_IllinoisCERAPATechnicalReportConcerningTheImplementationAndEvaluationOfTheProtocol.pdf (accessed on 10 May 2021).
  6. Fuller, Tamara L., and Susan J. Wells. 2003. Predicting maltreatment recurrence among CPS cases with alcohol and other drug involvement. Children and Youth Services Review 25: 553–69. [Google Scholar] [CrossRef]
  7. Fuller, Tamara L., Susan J. Wells, and Edward E. Cotton. 2001. Predictors of maltreatment recurrence at two milestones in the life of a case. Children and Youth Services Review 23: 49–78. [Google Scholar] [CrossRef]
  8. Grofman, Bernard, Guillermo Owen, and Scott L. Feld. 1983. Thirteen theorems in search of the truth. Theory and Decision 15: 261–78. [Google Scholar] [CrossRef]
  9. Health and Youth Care Inspectorate [Samenwerkend Toezicht Jeugd/Toezicht Sociaal Domein]. 2016. Leren van Calamiteiten 2: Veiligheid van Kinderen in Kwetsbare Gezinnen. Den Haag: Ministerie van Volksgezondheid, Welzijn en Sport. [Google Scholar]
  10. Hughes, Ronald C., and Judith S. Rycus. 2006. Issues in risk assessment in child protective services. Journal of Public Child Welfare 1: 85–116. [Google Scholar] [CrossRef]
  11. Johnson, Will. 2004. Effectiveness of California’s Child Welfare Structured Decision Making (SDM) Model: A Prospective Study of the Validity of the California Family Risk Assessment. Madison: Children’s Research Center, Available online: http://www.nccdglobal.org/sites/default/files/publication_pdf/ca_sdm_model_feb04.pdf (accessed on 10 May 2021).
  12. Knoke, Della, and Nico Trocme. 2005. Reviewing the evidence on assessing risk for child abuse and neglect. Brief Treatment and Crisis Intervention 5: 310. [Google Scholar] [CrossRef] [Green Version]
  13. Luthar, Suniya S., and Adam Goldstein. 2004. Children’s exposure to community violence: Implications for understanding risk and resilience. Journal of Clinical Child and Adolescent Psychology 33: 499–505. [Google Scholar] [CrossRef] [Green Version]
  14. Miller, Laurie S., Gail A. Wasserman, Richard Neugebauer, Deborah Gorman-Smith, and Dimitra Kamboukos. 1999. Witnessed community violence and antisocial behavior in high-risk, urban boys. Journal of Clinical Child Psychology 28: 2–11. [Google Scholar] [CrossRef] [PubMed]
  15. Orsi, Rebecca, Ida J. Drury, and Marc J. Mackert. 2014. Reliable and valid: A procedure for establishing item-level interrater reliability for child maltreatment risk and safety assessments. Children and Youth Services Review 43: 58–66. [Google Scholar] [CrossRef]
  16. Schulz-Hardt, Stefan, Felix C. Brodbeck, Andreas Mojzisch, Rudolf Kerschreiter, and Dieter Frey. 2006. Group decision making in hidden profile situations: Dissent as a facilitator for decision quality. Journal of Personality and Social Psychology 91: 1080. [Google Scholar] [CrossRef] [Green Version]
  17. Spies, Gloudina M., Catharina S. Delport, and Magdalena P. le Roux. 2015. Developing safety and risk assessment tools and training materials: A researcher-practice dialogue. Research on Social Work Practice 25: 670–80. [Google Scholar] [CrossRef]
  18. Trench, Sally, and Sian Griffiths. 2014. Serious Case Review: Family L. Available online: https://www.norfolklscb.org/wp-content/uploads/2015/03/Norfolk-SCR_Case-L_FINAL.pdf (accessed on 10 May 2021).
  19. Van der Put, Claudia, Mark Assink, and Geert Jan Stams. 2016. Predicting relapse of problematic child-rearing situations. Children and Youth Services Review 61: 288–95. [Google Scholar] [CrossRef]
  20. Vanderbilt-Adriance, Ella, and Daniel S. Shaw. 2008a. Conceptualizing and re-evaluating resilience across levels of risk, time, and domains of competence. Clinical Child and Family Psychology Review 11: 30. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  21. Vanderbilt-Adriance, Ella, and Daniel S. Shaw. 2008b. Protective factors and the development of resilience in the context of neighborhood disadvantage. Journal of Abnormal Child Psychology 36: 887–901. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  22. Vial, Annemiek, Claudia van der Put, Geert Jan Stams, and Mark Assink. 2019a. The content validity and usability of a child safety assessment instrument. Children and Youth Services Review 107: 104538. [Google Scholar] [CrossRef]
  23. Vial, Annemiek, Mark Assink, Geert Jan Stams, and Claudia van der Put. 2019b. Safety and risk assessment in child welfare: A reliability study using multiple measures. Journal of Child and Family Studies 28: 3533–44. [Google Scholar] [CrossRef] [Green Version]
  24. Vial, Annemiek, Mark Assink, Geert Jan Stams, and Claudia van der Put. 2020. Safety assessment in child welfare: A comparison of instruments. Children and Youth Services Review 108: 104555. [Google Scholar] [CrossRef]
  25. Wells, Melissa, and Melissa Correia. 2012. Reentry into out-of-home care: Implications of child welfare workers’ assessments of risk and safety. Social Work Research 36: 181–95. [Google Scholar] [CrossRef]
Table 1. Prevalence of the safety decision immediate danger for the ARIJ assessments, the individual expert assessments, and the expert panel assessments.
Table 1. Prevalence of the safety decision immediate danger for the ARIJ assessments, the individual expert assessments, and the expert panel assessments.
VignetteARIJIndividual ExpertsExpert Panel
%
Danger
Total Number of Assessments%
Danger
Total Number of Assessments%
Danger
Total Number of Assessments
162137541002 1
20120401
3541325401
4861410041001
546130501
61001310041001
720150401
89315504502 1
980150401
106160501
1150120401
121001210041001
13901010051001
14901010041001
1510067541002 1
16100625401
1783610041001
1829710041002 1
1967910041001
20100950401
21100108051001
2288850401
23601050401
2450106051001
Mean percentage
%SD%SD%SD
696.39568.265210.20
The percentage of the assessments that judged the child to be in immediate danger or safe for the ARIJ assessments, individual expert assessments, and expert panel assessments. SD = standard deviation. 1 Due to practical reasons, this expert panel was split into two different meetings with other experts.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Vial, A.; Assink, M.; Stams, G.J.; Van der Put, C. Child Safety Assessment: Do Instrument-Based Decisions Concur with Decisions of Expert Panels? Soc. Sci. 2021, 10, 167. https://0-doi-org.brum.beds.ac.uk/10.3390/socsci10050167

AMA Style

Vial A, Assink M, Stams GJ, Van der Put C. Child Safety Assessment: Do Instrument-Based Decisions Concur with Decisions of Expert Panels? Social Sciences. 2021; 10(5):167. https://0-doi-org.brum.beds.ac.uk/10.3390/socsci10050167

Chicago/Turabian Style

Vial, Annemiek, Mark Assink, Geert Jan Stams, and Claudia Van der Put. 2021. "Child Safety Assessment: Do Instrument-Based Decisions Concur with Decisions of Expert Panels?" Social Sciences 10, no. 5: 167. https://0-doi-org.brum.beds.ac.uk/10.3390/socsci10050167

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop