Speech perception in noise loads on perceptual, cognitive, and linguistic processes

Successful communication in challenging listening conditions requires a complex interdependence among perceptual, cognitive, and linguistic processes. In order to select a desired signal from competing sound sources, the auditory system must extract key acoustic features from the incoming stimulus stream. Interference with perceptual processes occurs when degradation caused by background noise renders portions of the target signal imperceptible [8]. In turn, this may lead to challenges in mapping key acoustic features to phonetic representations, and consequently to lexical representations (for a review see, [9]). A large body of work also implicates a role for cognitive processes in the extraction of speech from background noise. For example, the ease of language understanding (ELU) model indicates that for speech comprehension to occur sublexical information is Rapidly, Automatically, and Multimodally Bound into a PHOnological representation (RAMBPHO), a type of temporary storage system [10]. Poor speech signal quality due to noise results in ambiguous information in RAMBPHO which cannot be easily matched to stored speech representations. In such cases, working memory is needed to explicitly reprocess the incoming speech signal to resolve the mismatch. Working memory, a subcomponent of executive function [11], is engaged during speech recognition in adverse listening conditions to temporarily store salient acoustic cues, such as the fundamental frequency of the target speaker [9]. Cognitive demands tend to be greatest when the competing sound source has meaningful content, e.g. informational maskers [9, 12–14]. Like energetic masking (e.g. steady-state noise), informational masking involves selective attention and signal separation in the auditory periphery. However, recognizing speech in informational maskers not only requires listeners to cope with signal degradation due to energetic masking, but also to expend greater cognitive resources to ignore the meaningful distractors and attend to the target signal [13, 15]. Finally, speech recognition in challenging listening conditions is also dependent upon linguistic factors, such as vocabulary knowledge and lexical access. For example, a recent study demonstrated a positive correlation between phonemic restoration benefit and receptive vocabulary size and verbal intelligence, as measured by the Peabody Picture Vocabulary Test (PPVT-III-NL[16])[17]. The ELU additionally posits that individual differences in the quality of speaker-internal phonological representations in long-term memory also contributes to individual differences in the perception of speech in adverse listening conditions [10]. Taken together, these findings suggest that linguistic knowledge significantly contributes to the restoration of degraded speech.

Bilingualism differentially impacts perceptual, cognitive, and linguistic processes

The bilingual experience differentially shapes perceptual, cognitive, and linguistic mechanisms across the lifespan resulting in outcomes ranging from better to poorer performance, when compared to monolingual age-matched peers [4]. Constant communication in two languages can fine-tune subcortical auditory responses to incoming speech signals [18]. Bilinguals have also shown advantages in executive function (specifically, selective attention and inhibitory control) across the lifespan [19–23], with evidence suggesting that these processes emerge earlier in bilingual children, compared to age-matched monolingual peers [21]. These studies have shown that bilingual children are less distracted by irrelevant stimulus features compared to monolinguals, and in turn demonstrate faster or more accurate identification of target stimulus features [20, 24]. Finally, studies investigating verbal fluency and lexical retrieval often report poorer performance in bilinguals relative to monolingual peers [25–28]. The exact reason for the corroborated bilingual deficit in verbal fluency and lexical access is unclear. Some studies have suggested that verbal fluency differences are ameliorated between monolinguals and bilinguals when language proficiency is matched between groups [29]. Other studies that have specifically investigated lexical access abilities in bilinguals and monolinguals suggest that poorer performance in bilinguals may arise from greater linguistic processing demands due to competition between two lexicons [30]. This assumption stems from a large body of evidence that indicates that bilinguals co-activate both languages during language comprehension [30, 31].

As reviewed, bilingualism differentially impacts perceptual, cognitive, and linguistic processes. Therefore, it can be argued that a significant advantage or disadvantage in one or more of these underlying perceptual, cognitive (e.g. better executive control), or linguistic processes (e.g. poorer vocabulary knowledge), may contribute to differences in speech perception in noise performance for bilingual listeners compared to monolingual peers. Likewise, performance differences between groups may arise across varied speech-in-noise experimental designs, since speech perception in noise loads on perceptual, cognitive, and linguistic processes in distinct ways through different listening conditions (e.g. various noise levels and noise types).

Bilingualism and audio-only speech perception in noise: Mixed evidence

Prior work has revealed mixed evidence for the impact of bilingualism on audio-only speech-in-noise performance. Some studies have shown that early and simultaneous bilingual listeners exhibit poorer performance on speech-in-noise tasks relative to age-matched monolingual peers [32–34]. For example, one study investigated the impact of early bilingualism on speech perception in noise. While there was no difference between Spanish-English bilingual (age of English acquisition: ≤ six yrs.) and monolingual participants’ monosyllabic word perception in quiet, bilingual participants exhibited lower performance than monolingual participants in noise conditions. In another study, the Speech Perception in Noise (SPIN) test [35] was used to investigate the impact of speech noise and reverberation on the ability to recognize high- and low-predictable words in five groups of adult listeners who differed in their age of English acquisition [33]. The results revealed that while simultaneous and early bilinguals (age of English acquisition ≤ 5–7 yrs.) performed comparably to monolingual controls in mildly degraded listening conditions, monolinguals outperformed the two groups in more challenging listening conditions. In sum, these findings suggest that differences between monolingual and bilingual performance on speech-in-noise tasks may only be observable in conditions with high task demand (i.e. more challenging listening conditions). The results also indicate that age of English acquisition is a significant predictor of bilingual listeners’ identification of degraded target words in various combinations of noise, reverberation [33, 34] and context [33].

In contrast, recent evidence has demonstrated that simultaneous bilinguals exhibit comparable performance on speech-in-noise tasks, relative to monolingual age-matched peers [36, 37]. For example, one study found similar performance between English monolingual and simultaneous Spanish-English children (age of English acquisition ≤ 5 yrs.) on a forced-choice, picture-pointing paradigm that assessed speech reception thresholds across both English and Spanish two-talker masking conditions, as well as spectrally matched noise [37]. In another study with young adult listeners similar performance was found between English monolingual and simultaneous Greek–English bilingual listeners (age of English acquisition ≤ 2 yrs.) for recognition of target words across both English and Greek speech masker types [36]. In contrast to studies that revealed poorer speech-in-noise performance for bilinguals [33, 34], these studies [36, 37] suggest that simultaneous bilingualism does not negatively impact speech perception in noise. Several methodological variations can be noted in the type of target stimuli selected (e.g. monosyllabic words, SPIN sentences, BEL sentences), the maskers implemented (e.g. reverberation, spectral noise, various multi-talker babble tracks), as well as the SNR at which the stimuli were presented (e.g. +4 dB to -5 dB SNR).

These conflicting results and methodological variations pose a challenge to clinicians who need to determine the extent to which English speech-in-noise assessment tools can reliably test bilingual listeners with early English language acquisition [38]. Further, with the exception of a single study [37], there is a dearth of evidence for simultaneous bilingual children’s performance on English speech recognition in noise tasks.

Bilingualism and audiovisual speech perception in noise: lack of evidence

The existing studies on early and simultaneous bilingual speech-in-noise performance have only considered the contribution of auditory information to speech perception in adverse listening conditions. This is surprising since it is well-established that viewing a speaker’s articulatory movements improves speech-in-noise for both monolingual and non-native listeners [39–41]. Currently, the role of visual cues in bilingual speech perception has been examined in infants simultaneously learning two languages [42–44] and early bilingual adult listeners perceiving speech in a non-native language [41, 45–50]. In sum, these studies reveal that although bilingual infants show an audiovisual advantage relative to age-matched monolingual counterparts, bilingual adults do not. For example, bilingual infants exhibit better discriminability of silent talking faces [42, 44], and also exploit audiovisual speech cues more than monolingual infants [43]. These observations have led some to propose that bilingual infants may capitalize on audiovisual cues more than monolinguals to disambiguate their native languages as they simultaneously acquire two languages [51]. In contrast, studies that have investigated audiovisual phoneme identification in bilingual adults have shown that although monolingual and bilingual listeners both benefit from visual cues, monolingual adults outperform bilingual age-matched peers [50]. The evidence here suggests that reliance on audiovisual cues may change as a function of age in bilinguals. The use of audiovisual speech cues should be investigated in school-age children to gain a better understanding of this developmental change in reliance on visual cues in bilinguals since evidence is current lacking for this age group.

Bilingualism and audiovisual speech perception in noise: Study design and motivation

As reviewed, prior studies have yielded mixed results regarding bilingual speech perception in noise performance across different experimental designs. While this body of evidence is important, these studies have not examined audiovisual speech recognition in noise for bilingual listeners. Expanding on previous research, we investigated the effect of simultaneous bilingualism through a comprehensive speech-in-noise battery that assessed sentence recognition across: different modalities [audio-only (AO), audiovisual (AV)], since monolingual and bilingual listeners have been found to rely on visual cues differently across the lifespan; and two masker types that distinctively engage cognitive processes (energetic masker: steady-state pink noise; informational masker: two-talker babble). Perceiving speech in competing information maskers is more critically dependent upon executive processes, such as selective attention and inhibitory control [9], cognitive processes for which bilingual children have demonstrated advantages [20, 23, 52]. Finally, our comprehensive speech-in-noise battery assessed sentence recognition across a more expanded range of signal-to-noise ratios (SNRs; 0 to -16 dB) in school-age children. The implementation of more challenging SNRs has been found to tax the auditory system in such a way to elicit individual differences in the breakdown of speech sound encoding that may not otherwise occur in more favorable listening conditions [53, 54]. To control for linguistic bias between groups, we recruited a group of simultaneous bilingual children with similar linguistic backgrounds as similultaneous bilingual participants reported in [37] for: English language onset (age of English acquisition ≤3 years), as well as proficiency and usage of both languages (see Table 1). We also ensured that our two groups were matched on socioeconomic status (SES), since SES and bilingualism are found to contribute independently to cognitive and linguistic development [23, 55]. Prior studies have not matched monolingual and bilingual groups on SES [6, 7, 33, 34, 37]. Finally, we utilized the BEL sentences, which were developed to have simpler English lexicon and syntax compared to other standardized speech-in-noise sentences [56]. With this experimental design, we aimed to assess whether performance differences between monolingual and simultaneous bilingual school-age children, if present, are restricted to certain listening conditions when socioeconomic status and age of English acquisition is similar between groups.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Age of English acquisition (AoEA), daily language usage, and reported language proficiency of the twelve bilingual participants.

https://doi.org/10.1371/journal.pone.0168048.t001

Participants

Twenty-four elementary school-age children (age range: 6–10 years) from the Austin community participated in the experiment. Inclusionary criteria consisted of: normal hearing defined as hearing thresholds < 20 dB, normal hearing level (nHL) for octaves from 250 to 8000 Hz, no history or current diagnosis of a speech, language, or neurodevelopmental disorder (confirmed via parent report), and normal intelligence (M = 100, SD = 15), as measured by the Kaufman Brief Intelligence Test-Second Edition, KBIT-2, matrices subtest [57]; monolinguals: M = 104.83, SD = 17.02; bilinguals: M = 118.42, SD = 19.51; [F(1,22) = 3.30, P = 0.08]. The KBIT-2 matrices subtest assesses non-verbal intelligence through the evaluation of an individual’s ability to perceive relationships and complete visual analogies [57]. Monolinguals (n = 12; 6 female) and simultaneous bilinguals (n = 12; 8 female) were matched on age (monolinguals: M = 7.33 yrs, SD = 1.23 yrs; bilinguals: M = 7.33 yrs, SD = 1.23 yrs; [F(1,22) = 0, P = 1]) and family socioeconomic status (Family-SES; monolinguals: M = 55.75, SD = 6.18; bilinguals: M = 54.77, SD = 14.36; [F(1,22) = 0.05, P = 0.83]). The Family-SES score was calculated based on the Four Factor Index of Social Status [58] This metric takes into account parents’ occupation, education, sex and marital/cohabitation status and returns a six-tier classification of social strata based on scores ranging from 8 to 66. All participants’ family social strata fell into two categories: (a) medium business, minor professional, technical (social stratum range = 40–54), or (b) major business and professional (social stratum range = 55–66). SES is an important factor to control, as SES and bilingualism have both been found to contribute independently to cognitive and linguistic development [23].

Each parent completed a language history questionnaire [59]. Results derived from this language history questionnaire have been found to correlate well with linguistic proficiency [37]. Through the questionnaire parents rated their child’s speaking and comprehension proficiency on a scale ranging from 1 to 5 (see Table 1). The questionnaire specified that speaking proficiency referred to how easily the child could be understood in each language, while comprehension proficiency indicated how easily the child could understand each language. The 12 bilingual speakers consisted of 7 Chinese-English, 4 Arabic-English, and 1 English-Spanish participant. All parents of bilingual participants reported that their child was born in the United States, except one participant. The parent of this participant reported that the child was born in Japan, and soon after, the family immigrated to the United States. Five participants were reported to start acquiring both languages from birth, three before 20 months, and four before 36 months of age. All parents reported that their child’s use of the other language exceeded 20% throughout the day (see Table 1 for detailed demographics of bilingual participants).

Based on previously reported bilingualism classification, we labeled these children as simultaneous bilinguals. The parents of the 12 monolingual speakers stated that their child began acquiring English from birth and was not exposed to a second language throughout the child's lifespan.

Target Stimuli

Disadvantages for bilinguals, relative to monolinguals, have been found for vocabulary knowledge (for reviews, [4, 22]). Therefore, it is unclear if poor speech-in-noise performance in bilinguals arises because of poor linguistic proficiency in the target language. To control for the influence of linguistic knowledge on speech perception in noise, we selected 80 target sentence stimuli from the Basic English Lexicon [56]. The BEL corpus was developed for native and non-native English speech-recognition testing with the specific aim to select lexical items that would be familiar to non-native listeners who may have limited knowledge of English lexicon and syntax [56]. This complete corpus consists of 20 lists of 25 sentences. The 80 sentences that were selected for the current experiment comprised of simple vocabulary that would be present in an elementary school-age child’s lexicon (e.g. “mouse”, “cheese”, “rabbit”, “toy”, “tree”). Each sentence contained four keywords for intelligibility scoring. For example, “The black₁ cat₂ climbed₃ the tree₄.” A target word was counted incorrect if the child added or deleted morphemes. For example, if the child reported: "cats" for "cat” or “climb” for “climbed," the target words would be scored as incorrect. One male native speaker of American English was recorded producing the full set of 80 sentences. The audio was recorded at a sampling rate of 48000 Hz. The video and audio for each sentence were segmented and separated, and the audio tracks were equalized for RMS amplitude using Praat [60]. Similar stimuli have been utilized across a range of studies from our lab (e.g. [41, 61, 62]).

Maskers

The protocol for creating the maskers, as well as mixing the target speech with maskers, was additionally consistent with methodology reported in these past publications. There were two masker types created for this experiment: two-talker babble and steady-state pink noise. For the two-talker babble tracks, two male native speakers of American English were recorded in a sound-attenuated booth at Northwestern University as part of the Wildcat Corpus project [63]. Each speaker produced a set of 30 simple, meaningful English sentences (from [64]). All sentences were segmented from the recorded files and equalized for RMS amplitude in Praat [60]. The sentences from each talker were concatenated in random order to create 30-sentence strings without silence between the sentences. Two of these strings were mixed using the mix paste function in Audacity (Version 1.2.5; www.audacity.sourceforge.net) to generate two-talker babble. A second masker track consisting of 10 seconds of steady-state pink noise was created using the Noise Generator option in Audacity 1.2.5. Both masker tracks were equated for RMS amplitude to 50dB, 54dB, 58dB, 62dB and 66dB to create ten discrete, long masker tracks. Each masker track was segmented using Praat to create 80 unique noise clips, for a total of 5 noise clips per target sentence per masker type.

Mixing targets and maskers

Each audio clip was mixed with the five corresponding noise clips from each masker track using Adobe Audition to create ten final stimuli of the same target sentence with the following signal to noise ratios: 0 dB, -4 dB, -8 dB, -12 dB, and -16 dB. The mixed audio clips served as the stimuli for the audio-only condition. For the audiovisual condition, audio clips were reattached to the corresponding videos using Final Cut Pro. A freeze frame of the speaker was displayed during the 500 ms noise leader and 500 ms noise trailer. The final mixed target sentence with masker was approximately 2000 ms. In total, for each noise type, there were 400 final audio files (80 sentences × 5 SNRs) and 400 corresponding audiovisual files (80 sentences × 5 SNRs).

Procedures

Steady-state pink noise masker

In the mixed-effects model, the intercept was significant, b = -3.65, SE = 0.36, Z = -10.02, P < 0.001. The simple effect of language group was not significant, b = 0.60, SE = 0.42, Z = 1.42, P = 0.156, indicating that proportion of correctly identified target words in noise were not significantly different between the two language groups. The simple effect of SNR was significant b = 0.53, SE = 0.03, Z = 17.02, P < 0.0001, where improving SNR increased the probability of correct target word identification. The simple effect of modality was significant, b = 1.79, SE = 0.26, Z = 6.87, P < 0.0001, suggesting that the probability of correct target word identification was significantly higher in AV compared to the AO condition. The SNR by modality interaction was significant, b = -0.12, SE = 0.04, Z = -3.28, P = 0.001, indicating that slope of increase for intelligibility benefit from the improvement of SNR was less for the AV condition compared to the AO condition. There were no other significant two- or three-way interactions.

Two-talker masker

In the mixed-effects model, the intercept was significant, b = -3.37, SE = 0.44, Z = -7.65, P < 0.0001. The simple effect of language group was not significant, b = -0.46, SE = 0.56, Z = -0.82, P = 0.414, demonstrating that proportion of correctly identified target words in noise was not significantly different between the two language groups. The simple effect of SNR was significant, b = 0.42, SE = 0.03, Z = 16.10, P < 0.0001, where improving SNR increased the probability of correct target word identification. The simple effect of modality was significant, b = 1.34, SE = 0.24, Z = 5.57, P < 0.0001, suggesting that target words in sentences were more correctly identified in AV relative to AO conditions. There were no significant two- or three-way interactions.

Conclusion

In conclusion, our results indicate that when the age of English acquisition and socioeconomic status are similar between groups, monolingual and simultaneous bilingual children exhibit the same degree of speech-in-noise difficulties across a range of adverse listening conditions. For these simultaneous bilingual listeners, bilingualism does not negatively affect English speech recognition across a combination of masker (steady-state pink noise, two-talker babble), SNR (0 to -16 dB), and presentation modality (AO, AV). These findings suggest that despite increasing linguistic demands that may arise from two competing lexicons, simultaneous bilingual listeners have the ability to perform equally to monolinguals on the identification of degraded target words in masked sentences.