It’s about time

Shadows in a Cave

As cognitive neuroscientists, we want to understand how the dynamics of the brain lead to dynamics in cognition and behavior. But the brain is perhaps the most complex, mysterious, and enigmatic information processing system that we know of, and cognitive processes remain debated in terms of how they should be defined, categorized, and tested. Thus, the problems cognitive neuroscientists try to solve are poorly defined on both the cognitive and neuroscience sides.

In some sense, we have come a long way: Specific cognitive/perceptual/emotional/motor functions have been linked to activity in specific brain regions or circuits of brain regions; neurochemicals have been identified as necessary or relevant for certain aspects of emotion, learning, memory, and action; white matter pathways linking different brain regions have been implicated in specific diseases and behavioral characteristics. Compared to a century ago, when Overton (1897) suggested that thinking is done by “forehead cells,” our understanding of the brain has increased tremendously. But even “simple” processes like categorizing a visual stimulus as animal or automobile, maintaining a specific amount of force on a grip, or slowing response time after errors, turn out to have complex neural correlates that remain debated and poorly understood.

In his famous cave allegory, Plato describes prisoners who spend their lives chained to the side of a bridge in the middle of a cave. The bridge is a passageway, and people, animals, and vehicles traverse the bridge to get through the cave. The cave is lit from behind the bridge by a fire. But the prisoners are chained such that they cannot see what is behind them; they see only the flickering shadows cast in front of them on the cave wall. The prisoners do not know what shapes produce the shadows, and, because they spent their entire lives looking at shadows, these shadows – and not the shapes that produce them – are their view of reality.

We are like the prisoners in the cave. There are platonic “biological bases of behavior” that we want to discover (the figures walking on the pathway behind the prisoners), but all we can observe are the shadows cast on the wall (empirical data) by the flame in the back of the cave (methods and technologies); our concept of the nature and shape of the figures (theories) are shaped by past experience, intuition, and, perhaps most importantly, how the light of the flame defines the shadows (Figure 1).

However, we have one important advantage over the prisoners in Plato’s cave: We can, to some extent, control the flame. We can develop new technologies and methodologies, and we can combine methodologies in interesting, novel, and insightful ways. We can compare the shadows cast on the wall using different materials to fuel the fire.

Here I will argue that too much attention has been focused on investigating neurocognitive function based on attempts to localize processes in space (i.e., functional localization). Instead, fruitful insights might arise from considering time to be an important factor in neurocognitive function. Indeed, for some neurocognitive processes, time may be as important, or possibly more important, than space in terms of the underlying neurocomputational mechanisms.

Functional Localization: The Standard Approach

The way we as cognitive neuroscientists typically link dynamics of the brain to dynamics of behavior is by correlating increases or decreases of some measure of brain activity with the cognitive or emotional state we hope the subject is experiencing at the time. The primary dependent measure in the majority of these studies is whether the average amount of activity – measured through spiking, event-related-potential or -field component amplitude, blood flow response, light scatter, etc. – in a region of the brain goes up or down. In this approach, the aim is to reduce this complex and enigmatic neural information processing system to two dimensions: Space and activation (up/down). The implicit assumption is that cognitive processes can be localized to specific regions of the brain, can be measured by an increase in average activity levels, and in different experimental conditions, either operate or do not.

It is naïve to think that these two dimensions are sufficient for characterizing neurocognitive function. The range and flexibility of cognitive, emotional, perceptual, and other mental processes is huge, and the scale of typical functional localization claims – on the order of several cubic centimeters – is large compared to the number of cells with unique physiological, neurochemical, morphological, and connectional properties contained in each MRI voxel. Further, there are no one-to-one mappings between cognitive processes and brain regions: Different cognitive processes can activate the same brain region, and activation of several brain regions can be associated with single cognitive processes. In the analogy of Plato’s cave, our current approach to understanding the biological foundations of cognition is like looking at shadows cast on a region of the wall of the cave without observing how they change dynamically over time. This makes it difficult to disentangle shadows cast by different but overlapping shapes (Figure 1).

Are Cognitive Processes Localizable in Space?

Yes and no, depending on the spatial resolution and the cognitive process in question. At a gross level (e.g., several cubic centimeters), some cognitive functions appear to be localized to specific brain regions when using specific statistical thresholding: The hippocampus seems to be the locus of some aspects of long-term memory formation and retrieval; decoding the visual world occurs largely in occipito-temporal regions, volitional control of the body occurs in cortical motor areas and basal ganglia nuclei.

The fusiform gyrus has received considerable attention for the issue of functional localization. There is a region of the fusiform gyrus that increases in activation for a wide range of visual object categories, but that activates preferentially for faces relative to other comparable visual stimuli (Sergent et al., 1992; Haxby et al., 1994; Puce et al., 1996). For this reason, the “fusiform face area” is argued to be a modular region in the brain that specializes in processing faces (Kanwisher et al., 1997; Kanwisher, 2000). It seems clear that some computations related to face perception can be localized to this particular brain area (Kanwisher, 2010). Even in the case of face recognition, however, which is one of the strongest examples of a localized “module” of cognitive processes, it remains unclear what this localization of function entails – alternative accounts suggest that this area is specialized not for faces per se, but instead for categories of expertise (Gauthier et al., 2000a,b; Tarr and Gauthier, 2000) or individuation (Gauthier et al., 2000b; Rhodes et al., 2004). Further, distributed and overlapping representations of faces and objects may exist in larger regions of ventral temporal cortex (Haxby et al., 2001).

There are many spatial scales in the brain that differ in size by several orders of magnitude, ranging from single neurons to cortical columns to meso- and macroscopic populations. It is unclear what the appropriate spatial scale is for functional localization, or whether different neurocognitive processes can or should be localized at different spatial scales. Dynamics at some spatial scales may or may not be relevant for dynamics at other spatial scales (Kiebel et al., 2008).

It is tempting to argue that lesion studies provide the most compelling evidence for functional localization: The necessity of a region in a particular cognitive process. However, lesion studies must be interpreted with caution. First, there may be functional and anatomical reorganization/compensation, even within hours or days of damage (Sanes et al., 1988). Second, lesions may cause impairments indirectly by destroying a “way-point” for information flow or fibers-of-passage (Goulet et al., 1998). Third, it is uncertain how the functioning of a damaged (human or non-human animal) brain can be generalized to functioning of a healthy human brain.

Since the emergence of functional neuroimaging, most studies are based on the principle of functional localization – searching for a one-to-one mapping between region of the brain and psychological process. It is no exaggeration to state that standard preprocessing and analysis protocols (mass-univariate statistics and cluster-based thresholding) are specifically designed to find relationships between particular brain locations and particular cognitive functions. The underlying assumptions are that a mass of brain tissue uniformly increases in activity in response to an experiment event, and that this region must be “big enough” to be considered significant. Typically, however, in these studies there is not a single region that is activated, but rather many regions, producing long tables of activated areas. These long tables are difficult to reconcile with the assumptions of the functional localization approach. This lack of one-to-one mappings between brain region and function (Price and Friston, 2005) may in fact be an accurate reflection of the physically separated but functionally linked networks that underlie neurocognitive function (Varela et al., 2001; Guye et al., 2008; Bassett and Bullmore, 2009; Bullmore and Sporns, 2009).

At a spatial scale finer than possible with fMRI, cognitive processes appear even less localizable and more dependent on sparse spatial patterns (Fujisawa et al., 2008; Quian Quiroga and Panzeri, 2009). For example, the homuncular mapping of the body on primary motor area M1 is considerably more distributed and with considerably more overlap among regions than traditionally thought (Schieber, 2001). Even the concept of a cortical “column” – a small-scale functionally homologous unit of cortex – upon close inspection turns out to be a mysterious and fluid idea that oversimplifies the complex mesoscopic organization of the brain; columns are less functionally and anatomically homogeneous, and more dynamic over time and space, than previously thought (Rockland, 2010).

It should be noted that some physiological functions appear to be more reliably localized in the brain, for example the super-chiasmatic nucleus may be the “location” of our circadian rhythm. Clearly, these areas influence cognitive function, but the cognitive processes typically under investigation in cognitive neuroscience studies do not appear to be precisely localizable.

In summary, functional localization has been a useful approach and was critical for the development of cognitive neuroscience theories, experiments, and statistical measures. However, its simplicity may be its greatest limitation. It is likely that the brain uses more dimensions for information processing than just space and activation magnitude. This is not meant to imply that space is irrelevant for information coding/processing, or that functional localization is inappropriate or invalid. Rather, after this initial period of studying functional localization and learning about its merits and limitations, it is perhaps useful to consider time as an important factor for neural information processing. Time will help forge new brain-behavior links and may provide insights into human neurocognitive function beyond what can be learned from focusing on space.

It’s about Time

Time is a factor that is often though not always ignored in human cognitive neuroscience, and yet several considerations suggest that neural systems may use time as a factor for information coding, processing, and transmission. Indeed, time may be as important – if not more important – than space for information processing, particularly at the level of small populations of cells (spatial scale of millimeters to a few centimeters). As described below, “time” refers to rapid dynamics in electrochemical signals that are often but not necessarily oscillatory. Time as latency in functional MRI (e.g., the hemodynamic response peaks about 6–8 s after a stimulus) or an event-related component (e.g., the average voltage deflection 300–600 ms after a stimulus) is not taking into account the rich information that appears to be embedded in the temporal dynamics of neural activity.

There are several empirical and theoretical reasons why time may be in important factor in neural information processing.

(1) There appears to be information carried in the precise timing of activity within and across physically separated areas of the brain that cannot be measured by overall activity levels in any individual brain region. “Information” here can refer simply to quantifiable measures of brain activity that predict the cognitive state or behavioral response of the subject. In some cases, temporal dynamics of neural activity are significantly related to the task events while the overall amount of activity averaged over time is not. These kinds of results provide direct evidence that information in the brain is embedded in the rich temporal landscape of electrophysiological activity, and is lost when averaging activity over larger periods of time. Examples will be outlined in a subsequent section.

(2) In an information-theoretic sense, time provides a large number of possibilities for information to be represented and processed continuously, rapidly, and simultaneously (in parallel) in multiple functionally distinct networks that overlap in time and space. Time provides a rich source of complex multi-dimensional data in which information can be represented and processed. The large amount of information provided by the temporal dynamics of neural activity arises in part because electrophysiological activity of the brain is strongly oscillatory. These oscillations reflect rhythmic fluctuations in the excitability of populations of neurons (Tiesinga et al., 2008; Wang, 2010). Oscillations occur in multiple temporal and spatial scales, ranging from ultra-slow oscillations with a periodicity of tens of seconds over much of the cortex during deep sleep (Steriade, 2006) to ultra-fast oscillations with a periodicity of a few milliseconds within patches of somatosensory cortex (Curio, 2000). Oscillations that seem most relevant for cognitive processes range from delta (∼1–4 Hz) and theta (∼4–8 Hz) to gamma (∼30–100 Hz; for general reviews of neural oscillations, see Varela et al., 2001; Buzsaki and Draguhn, 2004; Traub et al., 2004). Because activity in one frequency band may occur independently of, and in parallel with, activity in other frequency bands, there is considerable bandwidth for information processing. For example, it has been suggested that multiple alpha sub-bands can be functionally dissociated in their roles in memory processes (Klimesch et al., 2007). Thus, if different neurons are “tuned” to different frequency bands (Jacobs et al., 2007), multiple functionally distinct neural networks can spatially coexist and be dissociated according to frequency band or spatiotemporal patterns (Akam and Kullmann, 2010).

Even from the activity recorded from a single electrode, there are already multiple domains of information, including frequency (the speed of the oscillation), power (the amount of the energy in a frequency band at a point in time), and phase angle (the position of the oscillation along the sine wave, driven by the state of excitation of the population of neurons; Figure 2; see also Makeig et al., 2004). And because these dimensions are largely independent of each other, a single electrode in a single location in the brain can measure multi-dimensional local neural dynamics. Interactions can occur across these dimensions (e.g., phase–amplitude cross-frequency coupling), suggesting that information may be embedded not only in the dynamics of one of these dimensions, but also in the interactions among dimensions. And because they all occur in the neural population contributing to one electrode, analyses based on spatial localization of average activity levels might be blind to some of these dynamics. Indeed, space is another dimension that increases the potential for information processing and complexity: Interactions can occur across physically separated networks over different frequency bands, and among power and phase.

What this means is that information processing schemes that take advantage of time can utilize many dimensions (information processing possibilities) simultaneously. This is in contrast with the standard functional localization approach, which, as discussed earlier, is limited to two dimensions: Functions occur in specific regions or are indexed by specific ERP components, and are either operating or are not.

(3) There is arguably selection pressure for individuals and species carrying neural systems that can decode the sensory world, make decisions, and adapt behavior faster and more efficiently. The fastest known behavioral response that is mediated by a neural connection is the snap closure of the mandible of the Odontomachus ant. It takes 8 ms for a hair to trigger receptor cells in the jaw, the message to be sent via the largest axons in insects or vertebrates to the brain, and then another signal to be sent back to the muscles to snap the jaw closed (Holldobler and Wilson, 1998). This mechanism provides the Odontomachus an unparalleled ability to attack and to escape (by snapping the jaws against a hard surface, the ant can fly backward over 40 cm). Although this neurally mediated response is rigid, its speed and efficiency give this ant a significant advantage during battle.

As neural systems and the environments in which they operate become more complex and less predictable, flexibility, and adaptability become critical. However, speed of processing has not entirely been compromised. Relatively simple perceptual decisions such as gradient orientation or color discrimination can be done with high accuracy with as little as 10 ms presentation time (Bodelon et al., 2007). Indeed, the macaque brain may require as little as 30 ms to make simple color discriminations (Stanford et al., 2010). Cortical responses to sensory deviancy occur within 100 ms of stimulus onset (e.g., the mismatch negativity; Garrido et al., 2009; Kujala and Naatanen, 2010). Seemingly complex processes in humans also occur rapidly. Electrical signals generated in the medial frontal cortex that reflect errors or response conflict (when multiple responses are activated but only one can be selected) begin shortly before the button press or stimulus onset, and peak about 80 ms after the button press (van Veen and Carter, 2002; Padilla et al., 2006). When subjects are given external performance feedback, electrical activity over medial frontal cortex begins to distinguish positive from negative feedback as early as 200 ms after feedback presentation (Gehring and Willoughby, 2002; Holroyd and Coles, 2002). In some cases, these signals predict decisions subjects make a few seconds later (Cohen and Ranganath, 2007; Cavanagh et al., 2010). Some of these processes occur in absence of conscious awareness (van Gaal et al., 2008; Cohen et al., 2009d).

Though not conclusive in establishing that there must be information processing/transfer schemes based on temporal coding, these observations indicate that processing information as rapidly as possible while maintaining flexibility and adaptability is ubiquitously needed and often observed. In other words, to the extent that speed and flexibility in responding to unpredictable changes in the environment provides an advantage for survival, animals containing neural systems that take advantage of the processing capabilities afforded by time may have been more likely to reproduce and pass on their genes (and neural processing power and coding schemes).

(4) Neural activity is inextricably linked to cognition and behavior in time, but not in space. There is a direct relationship between the timeframe of brain processes and the timeframe of the corresponding cognitive and behavioral processes. A fast neural process implies a fast cognitive process, and this in turn determines the speed of initiating or adjusting behavior. Indeed, our cognitive and neural systems appear well equipped for estimating and attending to time (Ivry, 1996; Fuster, 2001; Coull, 2004; Coull et al., 2004; Nobre et al., 2007; Ivry and Schlerf, 2008). In contrast, the spatial organization of neural processes is arbitrarily related to cognitive processes. Thus, whether a neural process is at one or another location, or distributed throughout the brain, has no implications for the corresponding (location of) behavior, except if physical location can constrain temporal dynamics. For example, would we be any different if our amygdalae were 3 cm more dorsal than they currently are? What about if they took 3,000 instead of 170 ms to respond to a threatening facial expression? The fact that brain activity is time-locked, rather than space-locked, to behavior implies that time will be highly informative about behaviorally relevant neurocognitive mechanisms. The brain does indeed exhibit some spatial relationships with the body (e.g., homuncular organization of sensorimotor regions, retinotopic organization of visual areas), although these examples still exhibit some arbitrary relations to behavior, such as the left–right and up–down crossovers.

(5) Controversies develop in cognitive neuroscience over the precise functional role of a specific region, but some of these controversies may be moot because multiple functionally distinct neural networks may coexist in the same space. Indeed, in these cases, empirical evidence may seem conflicting because different theories can be supported by different experiments. For example, it is widely accepted that the anterior cingulate cortex and surrounding medial frontal cortex is involved in monitoring actions (Ridderinkhof et al., 2004). However, different theoretical accounts have argued over whether this region monitors conflict, errors, or error likelihood (Botvinick, 2007; Carter and van Veen, 2007); signals that behavioral adjustments are needed (Kerns et al., 2004) or implements them directly (Taylor et al., 2007); integrates information about reward history or uncertainty (Rudebeck et al., 2008; Rushworth and Behrens, 2008), and so on. Considering that there is empirical evidence for all of these propositions, it seems likely that functionally different networks can emerge from the same population of anterior cingulate cortex neurons, depending on task demands (Fujisawa et al., 2008). Indeed, event-related potential studies demonstrate that within ∼300 ms, medial frontal scalp potentials can code for several different behaviorally relevant experiment parameters including reward magnitude, probability, and significance for learning (Philiastides et al., 2010).

In other words, rather than attempting to resolve a grand-unified-theory for the function of a region of the brain (in this example, the anterior cingulate cortex), attention might be better spent trying to understand how that area may utilize temporal schemes to compute and coordinate the diverse functions suggested by empirical evidence.

Event-Related Potentials

Most cognitive EEG studies report event-related potentials. The event-related potential is simply the time-domain average of EEG traces locked to the onset of some experimental event such as stimulus onset or button press. The reasoning behind this approach is that background noise in the EEG is averaged out over many trials, and the remaining fluctuations reflect activation of different cognitive systems. Peaks are named according to voltage sign relative to a pre-stimulus period and approximate peak time (N100, P200, etc.). The components are thought to reflect activation of modular cognitive or perceptual systems (Luck, 2005). The dependent variable is usually the peak amplitude of some component (e.g., the P300), the average voltage over some larger time window, or a peak-to-peak or base-to-peak amplitude difference.

To the extent that neuroelectric dynamics are oscillatory and non-phase-locked to the event, a considerable amount of cognitively relevant information in EEG may be lost in time-domain averaging (Makeig et al., 2004). This is illustrated in Figure 3. Although an extreme example, non-phase-locked (and, therefore, not observed in ERP) dynamics are often observed in real data. Indeed, many of the findings reviewed in the next section would not be observed using event-related potentials. For this reason, a lack of differences in event-related potentials between conditions may be difficult to interpret because many aspects of neural dynamics that are not apparent in event-related potentials might differ between conditions, e.g., if the neurocognitive processes under investigation recruits non-phase-locked dynamics or high frequency oscillations.

Thus, event-related potentials are useful for providing a glimpse of the global neural processing, but may be of limited use for elucidating complex electrophysiological dynamics.

Examples of Time-Embedded Information in Human Electrophysiological Activity

Considerable work has been done in animals and in computational models regarding how time may be used to encode information. Examples include: The timing of the first post-stimulus action potential in auditory cortex encodes sound amplitude (Heil, 2004); the timing of hippocampal place cells with respect to simultaneous theta phase improves statistical localization of the rats’ position based on physiology data (Jensen and Lisman, 2000; Jensen, 2001) and the timing of those action potentials provides independent information compared to average firing rate (Huxter et al., 2003); the timing of specific neuron firing in early visual cortex encodes gradient phase (Aronov et al., 2003); synchronization of local field potentials has been suggested to link disparate sensory modalities to form unified representations (i.e., binding; Engel et al., 1997; Singer, 1999; Fries et al., 2007); computational simulations suggest that the phase of neural activity may be sufficient for pattern completion (Knoblauch and Palm, 2001; Gutierrez-Galvez and Gutierrez-Osuna, 2003).

These and other findings in animals have laid important groundwork for understanding how the brain might use time to encode information, and have inspired many studies in human neuroscience. But because the spatial scale investigated in animals and model simulations is smaller than what is typically available in humans, and because it is not known to what extent human neurocognitive functions operate the same as those of other animals (though presumably some fundamental principles are conserved across species), this section will focus on relevant work in humans.

Although the literature on time-based coding schemes and sophisticated analyses of electrophysiological data is overshadowed by the literature on fMRI-based localization studies, there are too many relevant and insightful findings to discuss all them all here. Instead, this section will highlight three examples of how mathematical analyses of the temporal dynamics of human electrophysiological recordings have shed insight into neurocognitive function. These examples also illustrate cases in which standard localization- and hemodynamic-based analyses would be unlikely to reveal these brain dynamics (e.g., because no overall increase in space-averaged activity occurs).

(1) Cross-frequency coupling. Cross-frequency coupling refers to a relationship between activities in two different frequency bands. For example, the power of gamma (∼30–80 Hz) oscillations may vary as a function of the phase of theta (∼4–8 Hz). Cross-frequency coupling may be used for information coding if the lower frequency oscillations coordinate the activity of sub-populations of cells that use higher frequency oscillations to process information. Cross-frequency coupling has been suggested to be a generic brain mechanism for information processing (Lisman, 2005; Jensen and Colgin, 2007), and it likely involves dynamics of multiple neural populations that overlap in time and space. There are several ways in which cross-frequency coupling can be quantified (Mormann et al., 2005; Canolty et al., 2006; Cohen, 2008; Tort et al., 2010); different methods may be suited for different purposes, but all methods generally test for a modulation of activity in one frequency band as a function of activity in another (typically, relatively lower) frequency band.

Cross-frequency coupling has been linked to a variety of human cognitive processes (Canolty et al., 2006; Sauseng et al., 2008; Axmacher et al., 2010; Griesmayr et al., 2010). For example, increases in gamma–theta synchronization strength were reported to increase with working memory load, although there was no reported significant change in overall oscillation power at those specific frequency bands (Axmacher et al., 2010). These findings support a model of working memory that predicts that theta acts to coordinate activation of stimulus representations stored in gamma band activity (Lisman, 2010). The human nucleus accumbens exhibits robust gamma–alpha coupling that differentiates reward from punishment feedback (even though average gamma power does not differentiate these conditions; Cohen et al., 2009a), and predicts, on a trial-by-trial level, the extent to which patients adjusted their decision-making time on subsequent trials (Cohen et al., 2009b). Posterior alpha and gamma power have been linked to frontal theta phase during familiar visual perception (Demiralp et al., 2007), and errors due to lapses in attention (Mazaheri et al., 2009), consistent with alpha–gamma coupling in posterior cortex during visual perception (Osipova et al., 2008). Within the medial frontal cortex, alpha–theta coupling has been linked to reward and punishment evaluation (Cohen et al., 2009c).

Cross-frequency coupling is unlikely to elicit a hemodynamic response as measured with fMRI. The reason is that frequency band-specific activity levels may not change over a timescale measurable with fMRI; rather, it is the precise timing of activity that fluctuates, e.g., 10–20 times per fMRI measurement (Figure 4A).

Although there are potential methodological issues to be considered (discussed later), examination of cross-frequency coupling has the potential to shed insight into human neurocognitive function beyond what is possible though fMRI or time-domain averages ERP averaging. Cross-frequency coupling measures a putative mechanism by which spatially overlapping but functionally heterogeneous neural networks can be activated and coordinated in a rapid timescale.

(2) Inter-regional oscillatory synchronization. In addition to dynamics across frequency bands within the same region of space, information may be embedded in the temporal relationship of activity over space. Inter-regional phase synchronization (a frequency band-specific measure of functional connectivity) may underlie information transfer and co-processing (Knight, 2007; Womelsdorf et al., 2007). And because changes in phase synchronization may occur without any concomitant changes in power (Heinzle et al., 2007), there might be information embedded in the temporal relationship between areas that is not localized to either region alone.

For example, inter-regional oscillatory synchronization may be the mechanism by which the medial frontal cortex interacts with other brain systems, such as lateral prefrontal cortex to implement cognitive control after errors in speeded reaction-time (Hanslmayr et al., 2008; Cavanagh et al., 2009) or reinforcement learning (Cavanagh et al., 2010) tasks, with occipital cortex to bias sensory processing during go/no-go tasks in which no-go cues were difficult to perceive (Cohen et al., 2009d), or with the nucleus accumbens during reinforcement learning and reward anticipation (Cohen et al., 2009b, 2011). Long-range cortico-cortical phase synchronization has also been linked to conscious visual perception (Melloni et al., 2007), working memory (Palva et al., 2010), and other aspects of top-down control (Engel et al., 2001). Further, several brain disorders ranging from schizophrenia to ADHD to Alzheimer’s are associated with aberrant patterns of oscillatory phase synchronization (Uhlhaas and Singer, 2006; Stam, 2010).

Although some measures of phase synchronization are non-directional, meaning it is not possible to determine the direction with which oscillations are traveling, the temporal precision of EEG allows one to estimate directional flow of activity. For example, using spectral Granger causality (Granger, 1969; Cui et al., 2008), we found that directed synchronization from the medial frontal cortex to the occipital cortex increased after response errors in a visually guided go/no-go task in which no-go cues were difficult to perceive (Cohen et al., 2009d). These patterns of inter-regional directional synchrony were not mirrored in the levels of activation (oscillation power) of either frontal or visual cortices, suggesting that there was information contained in the inter-regional interactions that could not be localized to either region alone. Granger causality has also been used to show that consciously perceived words, compared to subliminally presented words, are associated with long-range directional synchronization (Gaillard et al., 2009). Partial directed coherence analyses demonstrate changes in directional cortical wave activity during sleep (Kaminski et al., 1997; De Gennaro et al., 2004), and interactions among motor, frontal, and parietal systems during response switching (Gladwin et al., 2006).

More generally, these findings illustrate an important feature of brain network phenomena: Widespread neural networks may synchronize and desynchronize within hundreds of milliseconds (Varela et al., 2001). This is important because ideas about connectivity and network functioning are becoming increasing popular in human and cognitive neuroscience (Sporns, 2010). From the findings reviewed here, it seems that many instances of brain functional network formation are transient and may be best measured using measurement technologies that take maximal advantage of time. Indeed, transient synchronizations in absence of changes in local power might not elicit a hemodynamic response or event-related potential (Figure 4B).

(3) Microstates and other transient electrophysiological events. Microstates refer to brief periods of cortical electrophysiological activity that are topographically stable over tens to hundreds of milliseconds (Lehmann et al., 2006). Microstates fluctuate 1–2 orders of magnitude faster than the hemodynamic response, and have been linked to visual perception, error processing, and resting state (Muller et al., 2005; Britz and Michel, 2010). They are sometimes accompanied by hemodynamic responses (Britz et al., 2010; Musso et al., 2010). Other brief cortical events include endogenous “bursts” of frontal alpha asymmetry (Allen and Cohen, 2010) that have been linked to depression. Transient bursts of synchronized electrophysiological activity also occur during sleep, namely spindles and ripples, which have been linked to memory formation and dream recall (Axmacher et al., 2008).

Relatedly, transient pre-stimulus oscillatory dynamics predict performance on the upcoming trial. For example, specific pre-stimulus alpha and theta phases predict performance on perceptual (Mathewson et al., 2009; Busch and VanRullen, 2010), memory (Guderian et al., 2009), cognitive control (Mazaheri et al., 2009; O’Connell et al., 2009; Eichele et al., 2010), and switching (Gladwin et al., 2006) tasks. These and other similar findings demonstrate that there are transient but cognitively meaningful brain states that modulate upcoming task-related performance. Indeed, these pre-stimulus dynamics may be causally involved in perceptual processes, as suggested by TMS manipulations (Romei et al., 2010). Because these pre-stimulus dynamics are transient and driven by phase and not amplitude, they are unlikely to be observed with time-domain averaging or the hemodynamic response (see Mathewson et al., 2009, Figure 4 for an example with empirical data).

In conclusion, these examples illustrate cases in which cognitively meaningful brain dynamics are related to the subjects’ cognitive state, or predict upcoming performance, while overall levels of average activity do not. The point here is not to argue that spatiotemporal averaging or low temporal resolution imaging is invalid or inappropriate; rather, the point is to stress that there are vast and complex neural dynamics that are relevant for understanding neurocognitive function that occur “below the radar” of fMRI and event-related potentials.

How to Study Time-Based Processing Schemes

In humans, the primary tools for studying electrophysiological activity are EEG and MEG. In some cases, it is possible to record EEG intracranially directly from patients with electrodes implanted for epilepsy or deep brain stimulation. Due to the high temporal resolution of EEG – a sample of electrical/magnetic brain activity can be recorded from each of dozens or hundreds of channels multiple times each millisecond – it is possible to examine the rich temporal landscape of cortical activity, and observe phasic changes in synchronization and desynchronization that occur over tens to hundreds of milliseconds. As discussed earlier, these complex cortical dynamics occur 1–2 orders of magnitude faster than the BOLD response, and may be lost in time-domain EEG averaging during standard event-related potential analyses.

There are several advanced mathematical tools that are appropriate for extracting the fine spatiotemporal oscillation dynamics from EEG data, including but not limited to short-time fast Fourier transform, complex wavelet convolution, multi-taper (in combination with Fourier transform or wavelet convolution), autoregressive coefficients, and bandpass filtering with the Hilbert transform. With careful parameter selection, these methods can produce nearly identical results (Quian Quiroga et al., 2002; Bruns, 2004), although in practice each approach has advantages and limitations. Functional connectivity can be estimated through spectral coherence, phase synchronization, power correlations, Granger causality, partial directed coherence, cross-correlations, etc. Several free toolboxes for analyzing data exist, including but not limited to EEGLAB, fieldtrip, spm8, BIOSIG, and BSMART. Some labs write in-house code for processing and analyses.

The analyses used in this research are not simple, nor are the methods standardized and widely used. New methods and ideas are continuously injected into the field. Although this “wild west” atmosphere provides researchers with the flexibility and freedom to custom-tailor mathematical and statistical approaches that can be optimized for the hypotheses at hand, it also makes entry into the field difficult for scientists who lack the background and experience in signal processing and programming.

However, even without performing relatively complex frequency- or synchronization-based analyses, single-trial analyses (in the time-domain or time–frequency domain) can better link cognitive/behavioral to neural dynamics compared to cross-trial averaging (Debener et al., 2007; Mars et al., 2008; Rousselet et al., 2008), and may be particularly relevant for linking EEG to fMRI (Debener et al., 2005, 2007; De Martino et al., 2010).

One practical issue is that because of volume conduction – the influence of deep and/or distant sources to many electrodes – it may be difficult to distinguish true inter-regional synchronization from artificially high synchronization due to different electrodes recording the same activity. This is a larger issue for EEG than for MEG. Ignoring zero-phase lag synchronizations may combat this issue, although there may be biologically relevant zero-phase lag synchronizations in the brain (Konig et al., 1995; Rajagovindan and Ding, 2008; Vicente et al., 2008). Another approach is to apply a spatial high-pass filter or other spatial transform such as current-source-density or Laplacian, which helps minimize contributions of deep/distant sources that project to many electrodes (Kayser and Tenke, 2006; Srinivasan et al., 2007), or independent components analysis, which estimates unique sources of variance in the brain (Makeig et al., 1997). Finally, one can estimate the cortical generators via beamforming, minimum-norm estimates, or dipole modeling, and then perform analyses in source-space. This approach is also not without drawbacks, because there is no unique inverse solution for any given cortical topography, and different methods may yield different estimates of source activity. Of course, there are advantages and limitations to every methodological approach that one must consider when interpreting results.

Limitations of Time-Based Information Coding/Processing Schemes

Recording electrophysiological or electromagnetic activity is not a perfect measurement of neurocognitive function. M/EEG recordings, like any methodology, have limitations that must be considered.

Mixing in the temporal or spatial domain is a critical issue. Mixing refers to when multiple spatially overlapping populations contribute to the signal recorded at a single electrode. An example of mixing in the temporal domain that cannot be recovered through time–frequency analyses is illustrated in Figure 5. Another (extreme) example of mixing is if two populations of pyramidal cells are equally simultaneously active, but aligned in opposing orientation (e.g., on different sides of a sulcus). In this case, their electrical fields will cancel and the researcher may be left with the misleading conclusion that no neural activity has occurred. Mixing is particularly problematic in EEG due to volume conduction and smearing/smoothing of the signal through the skull. Spatial filters such as current-source-density seem to be appropriate for obtaining relatively finer spatial resolution and linking inter-regional synchronization to cognitive processes (Srinivasan et al., 2007; Winter et al., 2007). Independent components analysis may recover some activity from mixed sources if those sources are temporally differentiable. Complex mixing from spatially overlapping and non-stationary sources may be less mathematically tractable to separate.

Another limitation of M/EEG is that they are limited to recording only certain kinds of activity from certain kinds of neurons (e.g., pyramidal and not interneuron) arranged in certain geometric orientations relative to the skull. Another theoretical limitation is that EEG measures only neural populations that are tangentially aligned to the skull, whereas MEG measures neural populations that are radially aligned to the skull. This is mostly a theoretical argument, however, because in practice many cognitive processes recruit areas of cortex that span gyri and sulci. Finally, due to the large number of dimensions available in EEG data and thus a huge number of possible statistical comparisons across time, space, frequency, and power/phase (and interactions among these dimensions such as inter-electrode cross-frequency coupling), there is a large potential for spurious Type I errors, particularly during exploratory analyses. There is a fine balance between, on the one hand, being driven by and constraining oneself to a priori hypotheses based on theory and previous research, and, on the other hand, being open to unexpected and unpredicted but robust patterns of results in the data.

Despite the limitations, examination of neural temporal dynamics has the potential to provide insight into human neurocognitive function beyond what is possible using approaches based on spatial localization (e.g., fMRI) or time-domain averages (e.g., ERP). Arguably, these limitations and considerations reinforce the idea that analyzing the rich temporal dynamics of neural activity bring us closer to the true complexity of brain function.

What about Space?

Accepting that information in the brain is coded precisely in time but distributed in space does not necessarily imply that space is irrelevant for neural representations and computations. Indeed, a logical consequent of this proposition is that space-based analyses should focus on distributed patterns rather than localization. Space and time may even have similar hierarchical computational organizational principles (Kiebel et al., 2008).

The best spatial resolution currently possible is a few cubic millimeters with high-resolution fMRI, although this resolution refers to the hemodynamic response, which may be spatially dispersed from its neural origin, following vascular features (Disbrow et al., 2000). Nonetheless, to the extent that representations extend over space at the level of several millimeters or centimeters, fMRI seems to be a valid tool for uncovering sparse or distributed representations. Spatial multivariate approaches that analyze patterns of activity over space (voxels) have sometimes proven more sensitive than standard approaches (i.e., testing whether the activity at all voxels is different from zero, or different between conditions) at linking brain states to cognitive states. On the one hand, this might be expected considering that multivariate regressions use more parameters to characterize the data – indeed, it remains to be discussed in the literature what the actual probability of Type I errors are and what preventative statistical measures are appropriate – but nonetheless the pattern of spatial activation sometimes predicts subjects’ cognitive state better than the overall amount of activity averaged over space (Haynes and Rees, 2006; Norman et al., 2006). Still, many applications of spatial multivariate approaches continue to rest on functional localization assumptions, for example by considering multivariate patterns only from small clusters of contiguous voxels, and then moving this “spotlight” around the brain (Kriegeskorte et al., 2006), or by selecting voxels for multivariate analyses that exhibit significant modulation by condition in a standard localization-based general linear model (Norman et al., 2006).

Relatively low spatial resolution techniques like EEG and MEG can also be used to examine distributed spatial patterns of electrical activity. For example, spatial multivariate patterns in EEG have been used to dissociate neural computations of magnitude from valence in a feedback-driven learning task (Philiastides et al., 2010) and word categories (Chan et al., 2010). Multivariate pattern analyses may also be informative across frequencies in one brain region. For example, visual gradient orientation can be predicted from frequency multivariate patterns (Duncan et al., 2010), even though the neurons coding for directional gradients are at a spatial scale too small to be resolved by MEG.

Distributed, multivariate spatial analyses in fMRI vs. multivariate time–frequency analyses in M/EEG may provide complementary information: Whereas examining complex spatial patterns in fMRI data may be suitable for understanding how representations are “stored” or activated, examining complex temporal patterns in M/EEG data may be more amenable for understanding the operations/computations performed on those representations, and how those representations are shared or transferred across space and over time.

Open Questions and Future Directions

Following is a non-exhaustive list of important foci for future research on temporal coding and processing schemes in human neurocognitive function.

(1) What is the neurobiological meaning of different features of oscillation dynamics? The different features are power and phase within each frequency range, and all interactions they entail (e.g., power–power correlation, phase–phase synchronization, power–phase synchronization). Within each frequency band, estimates of power and phase are independent of each other (with the exception of zero power, in which case it is not possible to estimate phase, although in practice this is not often observed in real data at neurocognitively relevant frequencies). Sometimes, results obtained from power and from phase are convergent; other times, divergent. How do we interpret results from power, phase, phase-power coherence, phase–phase synchrony, etc., and what do they mean for network dynamics, brain function, and information processing transfer?

It is tempting to speculate that the number of simultaneously active neurons drives power whereas the timing of the activity of those neurons drives phase. However, this is likely overly simplistic. For example, both power and phase information can be used to predict spike-timing (Rasch et al., 2008). For human neuroscience, perhaps the best way to dissociate the roles of power and phase for neurocomputation may come from careful and clever experimental design in which different predictions are made for how measures of power vs. phase are related to different cognitive processes.

(2) What spatial scales are relevant and how are dynamics at different spatial scales related? Dynamic and oscillatory neural activity can be measured at a large range of spatial scales, from within a single neuron to populations of millions of neurons (Varela et al., 2001; Kiebel et al., 2008; Moran and Bar-Gad, 2010). What is the appropriate spatial scale for neurocognitive function? Are different scales more appropriate for different cognitive functions? Are multi-spatial-scale interactions relevant for cognition?

There have been few investigations into how electrophysiological measurements at different spatial scales are related to each other. For example, in a study investigating the relationship between single-/multi-unit activity and EEG in a monkey, even the best combination of EEG characteristics (in this case, delta phase and gamma power) recorded from a small electrode accounted for only ∼15% of the variance of multi-unit activity (Whittingstall and Logothetis, 2009). Similarly, in an intracranial EEG study in humans, we found that time-domain correlation coefficients and theta-band phase synchrony between Cz and each intracranial electrode in the medial frontal cortex showed significant correlations/synchrony, but the magnitude was low (in the range of 0.1–0.2; Cohen et al., 2008). Thus, surface EEG may reflect a complex mixture of spatiotemporal dynamics from widespread areas. Whether and to what extent the divergence between activities recorded from multiple spatial scales is meaningful for cognitive function deserves more empirical attention.

(3) How does anatomical connectivity shape functional connectivity? Synchronous activity across widespread brain regions is believed to reflect functionally unified networks, such that physically separate neural ensembles are co-processing the same information or transferring information back and forth. Presumably, functional interactions – the nature and strength with which different nodes in a brain network communicate with each other – are shaped by the anatomical connections bridging those nodes. But what aspects of functional connectivity are shaped by anatomical connectivity: The strength of connectivity? Frequency range of synchronous interactions? Timing and phase delay? Which aspects of structure–function relationships are relevant for cognitive/behavioral functioning?

This question may be best addressed by linking EEG measures to white matter properties, measured through diffusion tensor imaging (DTI; Johansen-Berg and Rushworth, 2009), which takes advantage of the fact that the diffusion of water molecules in the brain is constrained by white matter fiber bundles. DTI data provides meaningful information about local white matter integrity and also the strength of tracts connecting different brain regions (Johansen-Berg, 2010). For example, visual stimulus-evoked gamma oscillations are correlated across subjects with corpus callosum white matter integrity (Zaehle and Herrmann, 2010). Similar findings have been observed with resting state EEG connectivity (Teipel et al., 2009) and medial frontal cortical responses to errors (Westlye et al., 2009).

(4) Are oscillations causally involved in neurobiological phenomena? Establishing causation is critical to science. To date, much of the current work on the role of oscillations in human neurocognitive function has been correlative. Although this is a necessary initial step, once spatial–temporal-frequency characteristics of neurocognitive processes are characterized, oscillation dynamics should be experimentally manipulated, ideally without explicitly manipulating the cognitive process thought to rely on those dynamics. There are several tools for addressing issues of causality, including transcranial magnetic stimulation, which has been shown to transiently perturb ongoing oscillations (Van Der Werf and Paus, 2006) that are dominant to each cortical region (Rosanova et al., 2009), and can impair cognitive processes such as attention that are thought to rely on specific oscillation patterns (Hamidi et al., 2009; Sauseng et al., 2009; Romei et al., 2010). Pharmacological manipulations may also be useful, although pharmacological agents may have complex effects on several brain systems and functions, so it may be difficult to interpret such results solely in the context of oscillations.

Oscillations can also be exogenously manipulated through stimulus flicker: When a visual stimulus is flashed at a particular frequency (like a strobe-light), regions of the brain that process that stimulus begin to oscillate at that frequency. In addition to “tagging” particular stimulus features (Herrmann, 2001; Ding et al., 2006), flicker has been shown to module – in task/frequency band-specific ways – attention and memory processes (Silberstein et al., 2001; Williams, 2001; Ellis et al., 2006; Williams et al., 2006; Wu and Yao, 2007; Mayes et al., 2009). It may also be possible to train subjects to modulate intrinsic oscillatory activity through “neurofeedback,” which can modulate cognitive processes (Gruzelier et al., 2006; Keizer et al., 2010; Zoefel et al., 2010) as well as neuroplasticity and corticomuscular excitability (Ros et al., 2010).

(5) Are all time-based coding/processing schemes oscillatory? Time–frequency decomposition is often used because of visually observable oscillations in EEG data, the link to animal research examining local field potential oscillations (Buzsaki and Draguhn, 2004), the fact that time–frequency methods are becoming increasingly common in the field, and the continuous advances in computing power, which facilitate analyses. But is all (or even most) time-based information in the brain contained in oscillation dynamics?

In fact, in time–frequency decomposition analyses such as wavelet convolution or Fourier transform, oscillations themselves in different frequency bands are not directly measured; instead, what is measured is the extent to which the time-domain signal correlates with wavelets or sine waves at specific frequency bands with specific windows in time. Because any time-domain signal can be represented as a sum of sine waves of different phases, frequencies, and amplitudes, non-oscillatory responses will be captured by time–frequency decomposition. Yeung et al. (2004) attempted to use this feature of Fourier’s theorem to argue that the error-related negativity may not be an oscillatory theta response, although their simulation of a “non-oscillatory response” was a half-sine wave at theta frequency (for more discussion, see Trujillo and Allen, 2007). Yeung et al.’s theoretical point is well taken, however, and there may be non-oscillatory dynamics that appear oscillatory due to time–frequency decomposition. Indeed, even in absence of sharp peaks in EEG power spectra over extended recording periods, frequency band-specific temporal dynamics are apparent (He et al., 2010), and can be characterized using 1/f functions (Miller et al., 2009). Broadband activity also seems to be relevant for some aspects of sensory–motor functioning (Onton and Makeig, 2009; Miller et al., 2010).

One could argue that whether the neural dynamics are truly oscillatory is not important; rather, what is important is that a time–frequency approach to analyzing electrophysiology data may provide new insights into neurocognitive function beyond what could be learned from simple time-domain averaging. However, whether the dynamics are truly oscillatory in nature might be relevant to linking human work to in vivo animal recordings, computational models, etc.

There are other ways in which information can be encoded in time that are not necessarily oscillatory. For example, there might be temporal “states” or patterns (Stam and van Dijk, 2002; Osterhage et al., 2007). Dynamics unfolding over time could be decoded using pattern-based analyses like support-vector machines or multivariate regressions, in which activity at different points in time are weighted to produce a linear or non-linear integration of activity over time (and/or frequency; Duncan et al., 2010) that best predicts the subjects’ internal mental state or cognitive process.

(6) What exactly is the information embedded in time? If the brain uses time-based information coding and processing schemes, what is the nature of this information? Is there a “temporographic” code, such that the precise timing of activity contains information? Or is the “information” simply reflecting passive maintenance of space-based information? Or does it reflect the passive transmission of information from one population of cells to another? Perhaps the oscillations themselves do not code information per se, but rather act as an organizing filter or selector for different populations of cells that actually encode/process information through action potentials or other means.

Conclusion

The stomatogastric ganglion is a nucleus in the crustacean stomach that comprises approximately 30 neurons and controls the stomach and parts of the digestive system. The complete circuitry and connectivity of this Lilliputian network is known. And yet, with two central pattern generators, modulation by at least 20 neurochemicals, multiple modes of firing ranging from synchronous bursting to arrhythmic, and pacemaker functions, there is an incomplete understanding as to how stomatogastric ganglion operates, changes neural states, and supports digestion (Hooper and DiCaprio, 2004; Stein, 2009). The human brain, in super-Brobdingnagian contrast, contains perhaps eight orders of magnitude more neurons, even more possible connectivity patterns among these neurons, and controls an impressive and complex array of perceptual, emotional, motor, linguistic, social, and creative processes. In trying to understand how dynamics of the brain lead to dynamics of behavior, it may seem we are doomed to remain prisoners in a platonic cave, forever straining to make out faint and fuzzy shadows cast by the most fantastically complex and enigmatic information processing system we know of.

Here it is argued that considerable neural information is embedded in the rich temporal landscape of electrophysiological dynamics, that much of this information may be lost when confining analyses to spatial dimensions, and that at least some of this information can be extracted non-invasively in humans using EEG and MEG. Approaches and analyses focused on temporal dynamical coding schemes will not render useless other approaches that are based on different (e.g., spatial) assumptions of neurocognitive function. However, ideas about time-based information coding schemes, and the approach of examining the temporal dynamics of brain electrical activity, are an important next step in theoretical and empirical human neuroscience developments. This nascent but growing literature on human neural temporal dynamics will provide a new impetus in uncovering fundamental neurocognitive mechanisms, linking research in humans to that in animals, and improving clinical diagnosis and treatment assessment. It’s about time.

Acknowledgments

Several of the ideas presented here were outlined at a workshop entitled “Opinions and Discussions in Cognitive Neuroscience: Amsterdam” in October 2009. The author is supported by a VIDI grant from the Dutch Organization for Scientific Research (NWO). Thanks to Dr. Sanne de Wit for Figure 1, and to Romke Rouw for comments.

References