Stage one: Consultation with experts to determine the response categories, modes of administration, and administrative instructions of the BI-SS.

Two meetings of the expert panel were held in stage one. The panels consisted of 2 senior occupational therapists, 2 psychometricians, and 3 researchers in the field of occupational therapy. The main purpose of the first meeting was to decide response categories and modes of administration (e.g., face-to-face interview or performance observation) for the Self-perceived Difficulty Scale and the Ability Scale, respectively. The definitions of ADL constructs were explained and the 10 items of the BI were provided to the panel members to act as reference items for the Self-perceived Difficulty Scale and the Ability Scale. In the second meeting, the expert panel developed the standardized administrative instructions for each item of the BI-SS based on the modes of administration determined from the first meeting. In addition, the panel members determined whether any tools or materials would be needed for assessment. All 7 of the panel members attended and participated in these two meetings. It was considered that consensus was achieved when at least 80% of the panel members indicated agreement with a proposal.

Stage 2: A pilot test of the BI-SS in patients with stroke.

A pilot test was conducted with the patients to examine the clarity of the administrative instructions and the feasibility of administration of the BI-SS. We tried to recruit participants having characteristics similar to those of the target patients. All procedures were carried out by the first author in an assessment room. Participants were individually tested and encouraged to identify any administrative instructions or response categories that seemed difficult to understand or ambiguous to them. The comments were reviewed and changes were made after 4 participants were tested. This process (testing and revisions) was repeated until no more substantial comments were made.

Subjects.

Patients undergoing outpatient or inpatient rehabilitation were recruited from 7 rehabilitation departments in Taiwan (including northern (4 hospitals), central (2 hospitals), and southern (1 hospital) parts of Taiwan) between January 2011 and August 2012.

Participants were included in the study if they met the following criteria: (1) diagnosis (International Classification of Disease, Ninth Revision, Clinical Modification codes) of cerebral hemorrhage (431), cerebral infarction (434), or other (430, 432, 433, 436, 437); and (2) ability to follow instructions. In addition, we excluded patients with any co-morbidity (e.g., dementia, Parkinsonism, limb amputation, or spinal cord injury) that might otherwise affect the patient’s performance on ADL. All participants gave informed consent prior to their inclusion in the study. Demographic characteristics and information on co-morbidities were collected from their medical records.

Ethics statement.

This study was approved by the Research Ethics Committee Office of E-DA Hospital and the Institutional Review Board of Kaohsiung Medical University Chung-Ho Memorial Hospital.

Procedure.

Each participant was assessed with the BI-SS and BI once by one of the two trained raters in an assessment room. The BI was administered to the participants via face-to-face interview with the original scoring criteria [23].

Prior to the study, the raters (independent of the expert panel) familiarized themselves with the BI-SS and BI. Both raters studied the user manual of the BI-SS and BI and received 2 hours of training on the administration of the BI-SS and BI. At the end of the training, both raters individually administered the BI-SS and BI to two patients while the first author observed and scored the patients at the same time. The raters’ scoring results were checked by the first author. Any discrepancies in score results were discussed to ensure that the raters were thoroughly familiar with the standardized process of administration and scoring criteria.

Data analysis.

We validated the construct validity of the BI-SS by examining the unidimensionality and convergent validity of the BI-SS.

Unidimensionality. We examined the unidimensionality of each scale of the BI-SS individually using Mokken scale analysis with the MSP5.0 computer program [3]. Mokken scale analysis is a nonparametric item response theory (IRT). The model of monotone homogeneity (MH) of Mokken scale analysis examines the accuracy of ordering of between persons’ raw sum scores on a measure to determine undimensionality [3], [22]. The MH model of the Mokken scale was used because it is believed to exemplify the simplest form of unidimensionality [24], [25]. Other parametric IRT models, such as the Rasch model, further require a parametric functional form of the item response function (IRF) [25], [26]. However, with rigorous assumptions, the Rasch model tends to exclude items that do fit the unidimensionality assumption (e.g., the Mokken model’s expectations) but not the parametric IRF form assumption. Thus, the Mokken model is likely to include more items from a pool of items in a scale while still holding the essential of unidimensionality [24].

The MH model has three assumptions: (1) items form a unidimensional scale (measuring the same construct; e.g., ADL ability); (2) item scores are locally independent (e.g., the scores on a given set of items are stochastically independent of each other within a group of persons with the same level of ADL ability); and (3) the item response function for each item is a steadily increasing function of the latent trait which means that patients with a higher level of ADL function would have a higher probability of scoring higher for an item that fits MH [24], [27]. Given a set of items (e.g., 8 items of the Ability Scale) that satisfies the assumptions of the MH model, then unidimensionality will hold, and it is justified to sum the score of each items to create a total score to represent the construct of interest (e.g., ADL ability) [25], [28].

The fit of the MH model was evaluated by calculating the scalability coefficient H for each of the individual items i (Hi) and for the entire measure (H). The Hi value was evaluated to determine whether an item was coherent enough to be included in a unidimensional scale. In general, all Hi in a unidimensional scale should be ≥0.3 [27]. Thus, we removed items from the BI-SS that had a Hi below 0.3. The H value is a global indicator of the degree to which participants can be accurately ordered on the underlying construct by means of their sum scores. Higher values of H indicate fewer violations of the assumption and a better scale [3], [24], [25]. Therefore, unidimensionality was considered to be strongly supported if H≥0.5 [3].

Convergent validity. Spearman’s rho correlation coefficient was used to examine the association between the BI-SS and the BI to determine the convergent validity of the BI-SS. A rho value ≥0.75 was considered high, 0.40–0.74 moderate, and ≤0.39 low [29]. We expected that the three scales would have moderate to high associations with each other.

We further examined the agreement between each pair of scores of the three scales (i.e., BI and BI-SS) to confirm that they were distinguished scales. First, the Wilcoxon signed rank test was used to examine whether the scores of the three scales were significantly differently from each other. The Wilcoxon signed rank test is a nonparametric statistical hypothesis test used to investigate the difference between the magnitudes of paired (dependent) observations (i.e., the BI and BI-SS in this study) [30]. Second, the minimal important difference (MID; also known as the minimal clinically importance difference [31], [32]) of the BI (i.e., 1.85 points) [33] was used as a threshold to present a meaningful difference in the responses of each participant between the scales. The proportions of the patients whose response differences between each pair of scales exceeded 1.85 points were calculated. To visualize the magnitude of response differences and the degree of agreement between scales, Bland-Altman plots with 95% limits of agreement (LOA) [34] were also plotted. The LOA provided insight into the amount of variation between scales. The agreement and variation of each patient’s responses to each pair of the three scales could also be seen on the plot. The range of difference was largely defined by interval between the upper bound and the lower bound of the 95% LOA (d±1.96×SD), where d represents the mean differences of the each pair of scores and SD represents the standard deviation of differences [34]. If a pair of scale assesses the same construct, then the pair of scores will agree very closely and the ranges of differences between both scales will be small. Third, we compared the numbers of patients with the lowest and highest scores in the BI against the two BI-SS scales.

Stage one: Consultation with experts to determine the response categories, modes of administration, and administrative instructions of the BI-SS.

Based on the results of expert panel discussions, each of the 10 items of the Self-perceived Difficulty Scale used 3 response categories ranging from 0 (with much difficulty), 1 (with some difficulty), and 2 (without any difficulty), with a total score of 20 (Appendix S1 in File S1). The higher the score, the lower the patient’s self-perceived difficulty in performing ADL.

Regarding the mode of administration, the face-to-face interviews method was decided for the Self-perceived Difficulty Scale. Thus, the Self-perceived Difficulty Scale was administered by asking patients to respond to questions such as “How much difficulty do you have in performing grooming?” Because self-perceived difficulty is based on a patient’s own perception, it will be valid only if the responses are from the patient him/herself.

Two items (bowel and bladder control) were removed from the Ability Scale due to their infeasibility and non-practicality to be assessed in clinical settings, leaving only eight items. The items of the Ability Scale had 3 or 4 response categories. For example, ‘grooming’ could be rated 0 (unable to perform), 1 (able to complete partially), or 2 (able to complete), while ‘transferring’ could be rated 0 (unable to perform), 1 (barely able to complete), 2 (almost able to complete), or 3 (able to complete) (Appendix S1 in File S1). The total score ranged from 0 to 18, with higher scores implying a higher level of ability to carry out the ADL. Further detailed instructions for scoring the Ability Scale can be found in Appendix S1 in File S1.

Regarding the mode of administration, observation-based testing was used for the Ability Scale. In addition, the panel members recommended that the Ability Scale be assessed in a standardized context (e.g., an assessment room without distractors such as physical obstacles or other people) to eliminate the varying impacts of different contexts on the performance of a patient. Furthermore, panel members decided on the tools/materials to be used for assessing the items of feeding, grooming, dressing, and bathing in the Ability Scale: chopsticks, spoons, a bowl, a brush, toothpaste, clothes, and towels. Thus, the Ability Scale was assessed by observing patients as they carried out a specific ADL task, such as “put the jacket on and zip it up”. Then the rater rated the patient’s level of ability in doing this task.

Stage two: A pilot test of the BI-SS in patients with stroke.

A total of 12 patients participated in the stage two of pilot testing to confirm the administrative instructions and feasibility of the BI-SS. Three rounds of testing were carried out. In the first and second rounds of testing, patients gave comments on the ambiguous wordings of instructions for, e.g., the eating task with chopsticks and the dressing task. Thus, the revisions were made accordingly. In the third round of testing, no substantial changes were suggested. The final version of standardized administrative instructions for each item of the BI-SS was clear, and the modes of administration and response categories were understandable to the patients. Thus, no further testing was conducted. In addition, on the basis of the third round of testing, the time required to complete the BI-SS was about 15 minutes.