Beyond Trialbased Paradigms Continuous Behavior Ongoing Neural Activity and Natural Stimuli

TechSights

Beyond Trial-Based Paradigms: Continuous Behavior, Ongoing Neural Activity, and Natural Stimuli

Journal of Neuroscience 29 August 2018, 38 (35) 7551-7558; DOI: https://doi.org/10.1523/JNEUROSCI.1920-17.2018

Abstract

The vast majority of experiments examining perception and behavior are conducted using experimental paradigms that adhere to a rigid trial structure: each trial consists of a brief and discrete series of events and is regarded as independent from all other trials. The assumptions underlying this structure ignore the reality that natural behavior is rarely discrete, brain activity follows multiple time courses that do not necessarily conform to the trial structure, and the natural environment has statistical structure and dynamics that exhibit long-range temporal correlation. Modern advances in statistical modeling and analysis offer tools that make it feasible for experiments to move beyond rigid independent and identically distributed trial structures. Here we review literature that serves as evidence for the feasibility and advantages of moving beyond trial-based paradigms to understand the neural basis of perception and cognition. Furthermore, we propose a synthesis of these efforts, integrating the characterization of natural stimulus properties with measurements of continuous neural activity and behavioral outputs within the framework of sensory-cognitive-motor loops. Such a framework provides a basis for the study of natural statistics, naturalistic tasks, and/or slow fluctuations in brain activity, which should provide starting points for important generalizations of analytical tools in neuroscience and subsequent progress in understanding the neural basis of perception and cognition.

Introduction

In neuroscience, our classical conception of sensorimotor experiments is invariably built from the notion of the trial, a brief and discrete series of events that allow the experimenter to select certain input parameters and then measure the resulting output of the nervous system. Each subsequent trial can be executed in conceptual isolation from the prior ones, typically with statistically independent input parameters. The outputs of the nervous system occurring during or right after each of these trials are then subjected to classical analyses derived from well-established tools, such as signal detection theory (SDT). The near-exclusive reliance on conventional experimental paradigms and analyses reflects a preference for apparent simplicity and control for the experiment and analysis, over the ecological validity of the tasks and stimuli used to probe the brain. Here, we examine approaches for quantitative neuroscience experiments that acknowledge that natural behavior is rarely discrete and that brain activity follows multiple time courses which do not necessarily obey experimenter-imposed trial structure. We conclude that the synthesis of such approaches has the potential to enrich our understanding of neural computation and how neural activity supports perception and behavior without a loss of quantitative rigor.

In this article, we focus on two primary reasons to consider approaches beyond conventional experimental paradigms. First, consider the fact that your own reading of this paper does not involve a series of events that could be well described as brief, independent trials, but still arises from coordinated patterns of sensory input, neural processing, mental functions, and motor behavior. Likewise, riding your bike to work, searching for a lost key, or deciding whether to continue reading this—from simple sensorimotor behaviors to the highest forms of metacognition—involve continuous chains of sensory-cognitive-motor loops of processing that continue over time frames longer and less well-defined than that of a conventional experimental trial. Of course, some behaviors, such as saccades, are at least individually quite ballistic, although how they play out across a longer time course is not yet well understood (Henderson, 2003; Hayhoe and Ballard, 2005; Najemnik and Geisler, 2005). Here, we explain that analytical tools exist for characterizing these sequences of behaviors. We then argue that continuous sensory-cognitive-motor loops are not merely tractable but should be thought of as the most appropriate framework for studying many forms of behavior, perception, and cognition that are currently shoehorned into trials (or not studied due to the difficulty in doing so).

The second main reason for moving beyond near-exclusive reliance on trial-based analyses is that they do not reflect the realistic structures and dynamics that exist and occur in the environment and the brain. The statistics of the inputs that define the environment in which the brain evolved and normally functions do not necessarily follow the standard assumptions (e.g., statistically independent and identically distributed, Gaussian) often made to facilitate simple quantitative analyses. Instead, the sensory environment is typically broadband, with visual patterns exhibiting multiple spatial frequencies, and dynamical visual stimuli and natural soundscapes exhibiting multiple temporal frequencies. The distribution of such spatial and temporal frequencies (i.e., its power) is often well approximated as 1/f ^β (where β typically ranges from 0 to 2), implying that sensory systems typically exhibit a wide set of time scales and temporal dynamics (Bernacchia et al., 2011; Murray et al., 2014; Hasson et al., 2015), quite distinct from the usual single-frequency (or otherwise tightly restricted) nature of experimental inputs. Although the presentation of a single sinusoidal input allows for powerful and intuitive analyses derived from systems identification approaches, frequency-based analytic tools are capable of handling more complex inputs, and broad spectrums with both fast/brief and slow/long timescales are present both in natural stimuli and in recorded patterns of brain activity. In a typical trial-based paradigm, such relationships can produce trial-to-trial serial dependence in behavioral performance which may be more suitably studied by a continuous paradigm.

More broadly, we take this opportunity to explain an experimental and analytic framework that stands to make continuous behaviors and mental processes, ongoing brain activity, and natural statistical structure more tractable and more integrated. The value of this approach is not just ecological validity. By taking on the continuous, broadband, and generally more complex nature of both sensory inputs and neural activity, analytic tools can become more powerful. They become capable of capturing elements typically left over as unexplained variance ("noise"), such as slow temporal fluctuations in neural activity that do not obey the faster timing of individual trials. Experiments also gain efficiency, placing the subject/nervous system in contexts with a higher duty cycle, with less time spent in secondary phases, such as the dreaded intertrial interval. And perhaps most reassuringly, the tools for thinking this way not only already exist (and have been applied to neuroscience in certain situations) but are also quantitatively relatable to many more familiar analyses. We now consider this approach in more detail, synthesizing a generic framework for analysis and a corresponding prescription for experimental design.

Moving from discrete to continuous paradigms in the study of sensorimotor transformations

Signal detection theory (SDT) is central to the study of the relationship between neural activity and cognitive function (Green and Swets, 1966). In the context of sensory systems, it posits that each sensory stimulus is represented in the form of a scalar "internal response," which reflects the intensity of the sensory stimulus, but which is perturbed by noise. This noise, often working in tandem with low stimulus intensities, places the internal response for a particular trial in an ambiguous regimen: it is unclear how much of the internal response is driven by the stimulus, and how much is driven by noise on that particular trial.

As suggested by its name, SDT is most straightforwardly applied to the challenge of detecting a weak signal (Fig. 1, left). In such tasks, an observer (i.e., a human or a trained animal) is presented a stimulus, and their task is to indicate whether the stimulus was present or not. Most readers will be familiar with the comparison of each trial's internal response to an (also unobserved) internal criterion, as well as the four possible resulting outcomes (hits, misses, correct rejections, and false alarms). It is also common to apply the SDT framework to tasks other than simple detection; for example, to the identification of a single stimulus (e.g., was a motion display of varying strength moving more to the left or to the right), and to two alternative forced choice tasks (e.g., which of the two moving patches was faster). Such trial-based approaches are easy to integrate with standard statistical hypothesis testing, and other standard tools and analyses established in the literature. A variety of classic and more modern primers provide detailed and thorough mathematical treatments in many extended domains, although the majority of applications of SDT are of the simpler cases (Green and Swets, 1966; Nevin, 1969; Banks, 1970; Stanislaw and Todorov, 1999; Swets, 2014). What we will focus on here are the core assumptions when SDT is related to neural activity.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Left, Schematic of SDT. The noisy sensory process develops over time given some input. The trial ends, and the decision is made based on endpoint of the sensory process relative to criterion. The resulting decision provides 1 bit of information about the entire sensory process. Middle, Schematic of drift diffusion model. The noisy sensory process develops over time given some input. The sensory evidence accumulates to some bound, and the decision is made based upon that bound. The resulting decision gives 2 pieces of information about the entire process (i.e., which bound was hit, and when). Right, Schematic of the Kalmer filter approach. The noisy sensory process develops over time given some input. The subject provides continuous estimates of the signal based on that noisy sensory process. The resulting behavior provides a full time series of information about the noisy sensory process over time, and trials tend to be longer because each trial is associated with a continuous series of estimates rather than a discrete output.

In detection or simple identification/discrimination tasks, connecting SDT to measurements of neural activity is superficially straightforward. The internal response is presumably the appropriate neural activity, which is described as the noisy spike count during stimulus presentation. Thus, neurophysiological recordings can be thought of as providing direct access to the internal representations (Fig. 1, middle), which are otherwise treated as unobserved variables in purely behavioral experiments, and which are usually estimated from analysis in such contexts.

One limitation of SDT is that it is couched in discrete, singular terms. A single value of noisy internal response is compared against a criterion value. In this instance, the internal response might be the spike count for the entire stimulus presentation, which therefore collapses any temporal dynamics within the trial. Although convenient and appropriate prima facie for tasks in which the sensory event is correspondingly brief and/or discrete, sensory decisions in the real world often take variable amounts of time to complete, and the sensory stimuli themselves can have noise that changes over time, requiring a process of evidence accumulation. Thus, it seems worthwhile to consider extensions of SDT that explicitly capture the temporal dimension.

The best-known temporal extension of SDT is the drift diffusion model (Wald, 1947; Ratcliff, 1978; Smith and Ratcliff, 2004; Palmer et al., 2005; Ratcliff and McKoon, 2008). Loosely, diffusion can be thought of as signal detection over time, in which each instantaneous time point has a corresponding noisy internal representation (Fig. 1, middle). These repeated "pulls" from a signal-detection theory type mechanism are then accumulated over time. When this accumulated evidence reaches a requisite level (the "decision bound"), the decision is made. The overt behavioral response is then generated and, in the simplest case, assumed to be a relatively rapid process that is brief and stereotyped relative to the decision phase. The rate of accumulation can depend on the strength of the sensory stimulus, and thus diffusion to bound can be formulated to make a prediction for both the accuracy and speed of decisions as a function of different stimulus conditions.

The value of accumulating evidence should be intuitive, but drift diffusion is only one way that the brain could benefit from evaluating evidence over time. Any mechanism that interrogates more than the initial impulse of a stimulus is capable of producing increases in accuracy with additional time. Classical drift diffusion is specified in continuous time and has no leak of the integration mechanism, but both discrete and leaky variants of accumulator models are often capable of fitting speed and accuracy data from psychophysical tasks (Usher and McClelland, 2001; Ditterich, 2006). Indeed, large bodies of literature have focused on distinguishing between these models, and substantial effort has been put into elaborating these models to include additional mechanisms, such as competing accumulators (Smith and Vickers, 1988; Mazurek et al., 2003; Reddi et al., 2003), and trial-to-trial parameter variability (Ratcliff, 1978, 2002; Ratcliff and Rouder, 1998, 2000; Smith and Ratcliff, 2004). It has recently been argued that the majority of such elaborated models are not falsifiable given standard tasks and data (Jones and Dzhafarov, 2014).

The ambiguities associated with testing between various formulations of decision making mechanisms comes in part from the relatively limited amount of data collected on each trial. While the drift diffusion model acknowledges the noisy time-varying internal process involved in sensory processing, the matching experimental paradigm (i.e., some sort of forced choice task) still only results in a single discrete behavior at the end of that internal process. This has the advantage of producing data that are simple to analyze (i.e., whether the choice was accurate, and when the response was made), but standard paradigms that wait for the end of the trial to record discrete behavioral outputs are by construction only indirectly shed light upon the noisy time-varying internal process meant to be studied within these paradigms.

Logically, an alternative approach would be to somehow measure a more continuous series of behavioral outputs in response to a presented stimulus. This time series could then be used to better model and understand the noisy internal processes that underlie sensory information processing. This is unwieldy if one envisions extending standard tasks to include multiple intermediate reports, but is indeed straightforward outside the context of forced-choice tasks. One such class of tasks that provides a time series of behavioral observations are tracking tasks (Baddeley et al., 2003; Burge et al., 2008; Mulligan et al., 2013; Bonnen et al., 2015, 2017). In these tasks, subjects track targets with their eyes or by pointing with their finger. These tasks are more natural for subjects and generate a large amount of behavioral data in a relatively short period of time.

Behavior in such tasks can be modeled by simple dynamic linear systems and their solutions. For example: where x_t is the stimulus parameter tracked by the subject at time t, F_t is the process transition matrix, w_t is the process noise, y_t is the noisy internal response, H_t is the observation model that maps the true state space to the observation space, and v_t is the internal noise. Here we assume Gaussian noise models for both process and internal noise (the former of which can be enforced in stimulus design). Under this assumption, the Kalman filter provides the following estimator: where x̂_t is estimate of x_t ; x̂ _t|t−1 is the estimate of x_t given all the information up to but not including the current time step, t; and K_t is the Kalman gain, which is calculated from estimates of the covariance (i.e., an estimate of the level of uncertainty in the system). The Kalman filter solution combines knowledge of the structure of the linear system in question (i.e., Eqs. 1, 2), as well as incoming sensory information (y_t ), to make a prediction for the current time step.

Tracking tasks in conjunction with Kalman filter models (or state-space models more generally) form the basis for a more detailed study of the temporal dynamics of sensory processing (Fig. 1, right). Typically, the Kalman filter is used to produce estimates of x_t , given the noisy measurements y_t . However, by flipping the estimation framework over to become a fitting problem, Bonnen et al. (2015) showed how the state-space model and Kalman filter solution can also be used to estimate the parameters associated with the noisy internal response, given the stimulus parameter (i.e., the true state x_t ) and the behavioral response (i.e., the estimates, x̂_t ). The intent of the Kalman filter in this context is to provide an analysis framework for comparing/modeling results via the estimation of sensory noise distributions, just as one might estimate d′ values for forced choice tasks using SDT. There are significant similarities between the Kalman filter model of tracking and the drift diffusion model of forced choice tasks. A noisy internal response is also a component of the Kalman filter model; the internal noise is part of the noisy measurement of some underlying state of the world. In the drift diffusion framework, the internal noise affects the accumulated evidence and is related to the behavioral outcome when the process hits the decision bound. The Kalman filter solution, meanwhile, gives an equation for relating the behavioral estimates to the noisy internal response over time. The advantage of this approach (tracking psychophysics in conjunction with the Kalman filter for analysis) is that the time series of noisy internal responses is not related to a single behavioral outcome but rather to series of behavioral estimates.

Here we have laid out the math for a linear state-space model with Gaussian noise and its Kalman filter solution: these are the standard assumptions for the Kalman filter. We recognize that such assumptions may be incorrect for some applications, but we also note that, fortunately, a variety of extensions already exist which provide solutions for other noise models as well as nonlinear and non-Gaussian state-space models (Sorenson, 1966; Uhlmann, 1992; e.g., Julier and Uhlmann, 1997).

Furthermore, work across a range of subfields already uses the Kalman filter to model neural and behavioral data. Work in reinforcement learning uses the Kalman filter (and similar filtering approaches) to model uncertainty and learning across successive trials and relate that to neural activity (Sutton and Barto, 1998; Seo et al., 2009; Gershman, 2015). The brain-machine interface community uses Kalman filters to perform decoding on neural activity to control cursors, robotic arms, etc. (Carmena et al., 2003; Wu et al., 2004). There is behavioral and theoretical evidence for the nervous system approximating the function of Kalman filters in implementing state-space estimators (Denève et al., 2007; Makin et al., 2015). The application we propose here is distinct in being more pragmatic and more general, putting forth the Kalman filter (and other filtering solutions to state-space models) as a framework for relating sensory perception and neural activity via behavior in the same manner that SDT has been used in psychophysics. Throughout this section, we have laid out the advantages to such a framework, in particular that it models sensory information processing as a noisy time-varying process, and furthermore that the behavioral observations collected in such a framework would be collected at a temporal resolution more aligned with the underlying neural dynamics. It is also important to note that simpler descriptions of the tracking paradigm stimulus-response relation (e.g., height of a simple cross-correlogram) covaries tightly with the principled estimates from the Kalman framework analysis (Bonnen et al., 2015). Future work developing this framework might focus in particular on examining paradigms which include observer prediction and thus require estimates of the transition matrix or paradigms that rely on the extensions of the framework (including non-Gaussian and nonlinear state-space models).

The development of continuous paradigms for relating behavior and neural activity is further motivated by a number of current revisitations of the time course of neural activity and its correlation with behavior in the context of trial-based paradigms (Churchland et al., 2010; Goris et al., 2014; Lundqvist et al., 2016; Bolkan et al., 2017; Schmitt et al., 2017; Yates et al., 2017). Many classic lines of work have been reopened to reveal that even the simplest forms of temporal computations linking sensory and motor stages remain unclear. For example, current debate surrounds whether persistent neural activity actually exists at the single-trial level during oculomotor working memory tasks or whether more transient bursts with variable times are the essential neural substrate (Lundqvist et al., 2016; Bolkan et al., 2017; Schmitt et al., 2017). It is also contentious as to whether ramping activity during simple perceptual decisions is a straightforward neural correlate of evidence accumulation or whether it reflects alternate dynamics, a mixture of simpler factors, and/or secondary signals not functionally necessary for performing the task (Gold and Shadlen, 2007; Freedman and Assad, 2016; Huk et al., 2017).

While the primary example we provided for the use of a tracking paradigm is focused on the measurement of sensory noise, we contend that tracking paradigms are much more broadly useful across sensory, motor, and cognitive domains. For example, ongoing work in the area of visual attention using multiobject tracking provides a concrete example of how tracking paradigms prove useful in more cognitive domains (Kuo et al., 2017). However, just as the case for smooth pursuit, the dynamics of the motor response sometimes need to be disambiguated from the sensory dynamics. For example, Bonnen et al. (2017) teased apart the relative contributions of the motor plant (arm/hand) and perception to human tracking performance in three dimensions by flexibly changing the mappings between visual stimuli and motor responses (something difficult to dissect in the case of more tightly aligned continuous sensorimotor behaviors, such as smooth pursuit).

Finally, the development of continuous paradigms is motivated by the need to move beyond the stimulus and analysis constraints present in conventional trial-based paradigms. While there are certainly instances in which a discrete trial-based approach is appropriate particularly for the study of a discrete behavior (e.g., saccades, brief periods of smooth pursuit), a variety of other phenomena have accumulated that are poorly understood and not well integrated, but which are all related to long time-scale fluctuations in natural stimuli, cortical dynamics, and perceptual behavior. Most of these do not fit nicely into assumptions of short time scales and temporal statistical independence. We believe that increased development of continuous paradigms relating behavior and neural activity will advance the understanding such phenomena and further extensions of the Kalman filter analysis framework may provide a powerful tool for dissecting long time-scale cortical dynamics and how the brain processes natural stimuli. The following section examines these phenomena in greater detail.

Removing the independence assumption: natural stimuli, ongoing brain activity, and serial dependencies in perception and behavior

A repeated finding using trial-based paradigms is that human perception and behavior exhibit long-range temporal correlation, manifesting as trial-to-trial correlation in the perceptual judgments or behavioral outputs. For example, in simple tasks such as threshold-level detection or reaction time tasks, as well as reproduction of a particular level of force or a particular time interval, the trial-to-trial fluctuations of hit rate, reaction time, force output, and time-interval output exhibit long-range temporal correlation such that their power spectra follow a P ∝ 1/f ^β form, where P is power, f is temporal frequency, and β is a scaling parameter typically between 0 and 1 (Gilden et al., 1995; Gilden, 2001; Monto et al., 2008). Such a 1/f-type power spectrum indicates that performance many trials ago is still correlated with that in the current trial, with the magnitude of this relation falling off with increasing time interval. Interestingly, long-range temporal correlations in reaction time fluctuations are modulated by task difficulty (Clayton and Frey, 1997) in a manner similar to task modulations of long-range temporal correlations in neural activity (He et al., 2010), suggesting that slow fluctuations in neural activity may underlie trial-to-trial behavioral dependence, a point we elaborate on below.

Although long-range temporal correlation in human behavioral output has been long described, (positive) trial-to-trial serial dependence in human perception was only recently discovered (Chopin and Mamassian, 2012; Fischer and Whitney, 2014; Liberman et al., 2014), and remains controversial (Maus et al., 2013; Fritsche et al., 2017). Presumably, this is due to the fact that perception is also strongly influenced by adaptation, which could produce negative trial-to-trial correlation that would cancel out serial dependence. This may result in the net effect varying across experiments depending on the exact paradigm and subject population. Nonetheless, there is now strong evidence suggesting the existence of positive trial-to-trial correlations in both perceptual and behavioral outcomes.

What is the neural basis of trial-to-trial serial dependence in perception and behavior? A recent study found that, in an orientation judgment task, orientation signals in V1 measured by fMRI were positively correlated from trial to trial, similar to the perceptual decisions made by subjects; in addition, both the behavioral and neural serial dependence was spatially specific (St John-Saaltink et al., 2016). More broadly, ongoing brain activity at the level of population signals recorded by local field potentials (LFPs) (Manning et al., 2009; Milstein et al., 2009), electrocorticography (Miller et al., 2009; He et al., 2010), MEG/EEG (Dehghani et al., 2010; Lin et al., 2016), and fMRI (Bullmore et al., 2001; He, 2011) exhibit long-range temporal correlations, manifesting as power spectra following a P ∝ 1/f ^β form, with β typically between 0 and 2 (He, 2014). This long-range temporal correlation in neural activity extends to the time scale of tens of seconds, minutes (with the corresponding 1/f-type power spectrum extending down to well below 0.1 Hz (He, 2011; Lin et al., 2016; Mitra et al., 2018), and is thus well positioned to produce trial-to-trial correlations in ongoing neural activity with standard trial-based behavioral paradigms. Thus, long-range temporal correlation in neural activity is a natural cause for serial dependence in perception and behavior.

A now-extensive literature describes the rich network structures embedded in spontaneous fMRI signals (for review, see Buckner et al., 2013; Petersen and Sporns, 2015; Raichle, 2015). Spontaneous fMRI signals correlate with the low-frequency (<5 Hz) component of neural field potentials, named "slow cortical potentials" (SCPs) (He et al., 2008; Pan et al., 2013). Like the spontaneous fMRI signals, ongoing fluctuations in the SCPs are also coherent within intrinsic large-scale brain networks (He et al., 2008). Both types of signals contain very slow fluctuations in the order of seconds to minutes and exhibit long-range temporal correlations (He et al., 2010; He, 2011) that are well poised to drive trial-to-trial serial dependence in perception and behavior. Consistent with this idea, studies have demonstrated that prestimulus spontaneous fMRI and SCP activity influences perceptual outcome and motor output (Boly et al., 2007; Fox et al., 2007; Hesselmann et al., 2008; Monto et al., 2008; Li et al., 2014; Baria et al., 2017). However, much work remains to be done to directly probe the connection between slow fluctuations in fMRI signals and SCPs and trial-to-trial correlations in perception and behavior.

Along an apparently distinct, but likely related vein, it is well known that many natural stimuli exhibit temporal or spatial power spectra following a P ∝ 1/f ^β form, with β commonly ranging between 0 and 2. In the visual domain, natural movies typically follow a P ∝ 1/f ^β type temporal power spectrum (Dong and Atick, 1995). In the auditory domain, loudness and pitch fluctuations of natural soundscapes, such as urban and rural environmental noise (De Coensel et al., 2003), speech, and music (Voss and Clarke, 1975), also exhibit 1/f-type temporal power spectra. Thus, the temporal dynamics of natural stimuli exhibit long-range temporal correlation, in a manner similar to trial-to-trial fluctuations of human behavioral output as well as slow, ongoing neural activity recorded by fMRI or SCP.

Might there be a relationship between these three phenomena: temporal statistical regularities in natural stimuli, trial-to-trial correlations in perception and behavior, and long-range temporal correlations in ongoing neural activity? As mentioned earlier, slow fluctuations in ongoing neural activity are well positioned to contribute to serial dependence in perception and behavior. However, the other link, between neural activity and perception/behavior on the one hand and natural stimuli on the other hand, has been more elusive. This is partly because natural stimuli are less analytically tractable than simpler, artificial stimuli with narrower temporal/spatial frequency bandwidth or the sorts of Gaussian and trial-to-trial independence assumptions often made in trial-based frameworks. However, tools for analyzing neural activity in response to natural stimuli are developing quickly, such as assessing similarity in evolving neural dynamics between repeated presentations of the same temporally extended natural stimulus (Hasson et al., 2010), and encoding models relating multiple stimulus parameters to neural activity at each time point (Naselaris et al., 2011). In addition, mathematically constructed artificial stimuli that capture the second-order statistical structures (i.e., power spectrum, autocorrelation) of natural stimuli but are nonetheless precisely controlled have proven to be a powerful tool for probing how the nervous system processes statistical regularities present in natural stimuli (Palmer et al., 2015; Lin et al., 2016). For instance, long-range temporal correlations exhibited by MEG activity recorded from the human brain not only reflect long-range temporal correlations in stimulus input but also predict individual subject's ability to discriminate different levels of temporal correlations in the stimulus input (Lin et al., 2016).

Last but not least, being able to make valid predictions about environmental stimuli confers an obvious evolutionary advantage. So far, studies on predictive processing based on statistical regularities in stimulus input have typically adopted simple, artificial stimuli that involve repeated presentations of items or sequences (Bekinschtein et al., 2009; Yaron et al., 2012; Gavornik and Bear, 2014). And many trial-based frameworks (including the vast majority within sensory and motor neuroscience) enforce that there is nothing predictable about the next trial based on the preceding ones. Yet, the long-range temporal correlations prevalent in natural stimuli suggest that natural stimuli have a substantial degree of predictability, and it seems plausible that the nervous system has evolved to capitalize on such dependencies to make and exploit predictions about its environment. Thus, a key question for future studies is how predictive processing based on natural statistical regularities is implemented in the brain. Using continuous stimuli and paradigms provides a natural arena for the consideration of temporal correlations in both inputs and neural activity (Honey et al., 2012; Lin et al., 2016).

Importantly, effective tools for addressing these questions in both stimulus design and data analysis already exist. As mentioned earlier, temporally varying natural stimuli often exhibit temporal power spectra following a P ∝ 1/f ^β form, with β typically ranging between 0 and 2. This second-order statistical structure is what confers temporal redundancy or predictability for the continuous natural stimuli. When β is in the range of 0–1) (indeed, anywhere between −1 and 1), the corresponding time-domain stimulus input forms a stationary sequence (technically referred to as "fractional Gaussian noise" or fGn). When such a sequence has zero-mean (the mean can be added back after estimation), the mathematically optimal linear prediction of order K for the upcoming item in the sequence x̂_n based on past samples {x _n−1, x _n−2, …, x _n−K } is written as follows: where the vector ̱a = {a ₁, a ₂, …, a _k} is to be estimated so as to minimize the average squared prediction error. Linear algebra leads to an explicit theoretical solution for â (Scharf and Demeure, 1991): where ̱r_x is the covariance sequence of process x, and denotes the K × K square matrix, with entry ( ) _p,p′ = ̱r_x (|p − p′|) for p, p′ ϵ {1,…,K}².

When β is in the range 1–2 (indeed, anywhere between 1 and 3), the corresponding time-domain stimulus input forms a nonstationary sequence (technically referred to as "fractional Brownian motion" or fBm). fBm sequences are cumulative sums of their corresponding fGn sequences, whose β exponents differ by 2. The mathematically optimal prediction for an upcoming item in an fBm sequence can be estimated by the sum of the current item in the fBm sequence and the optimal prediction for the upcoming item in the corresponding fGn sequence.

Together, these tools allow the mathematical calculation of the optimal prediction for the upcoming stimulus input given the past history of any stimulus sequence exhibiting a P ∝ 1/f ^β -type power spectrum, where β ϵ (−1, 3). Using this mathematical framework, a recent study created a set of stimulus sequences exhibiting 1/f ^β temporal power spectra, where β ϵ (0, 2). To dissociate sensory processing of the current stimulus input from predictive processing of the upcoming stimulus input, these sequences converged onto the same value for the penultimate item, while their different history prescribed different values for the expected upcoming item. The actually presented last item was randomly drawn from a fixed distribution and subjects gave surprise ratings for this last item based on the previous stimulus history. Analysis of these psychophysical data suggested that human subjects indeed capitalized on the natural temporal statistical regularities to make valid predictions about the upcoming item (Maniscalco et al., 2018). In addition, concurrent MEG recordings revealed that slow, arrhythmic activity in the SCP range reflected integration of stimulus sequence history over time, and that such history integration contained in slow neural activity predicted the expected value of the upcoming stimulus input, providing a concrete computational mechanism for forming predictions based on natural statistical regularities.

Conclusion: moving to naturalistic and continuous stimuli, behavior, and neural measurements without a loss of quantitative tractability

The time seems ripe to consider ways of doing experiments that allow for continuous stimuli, behaviors, and assessments of neural activity. Hopefully, we have shown that the analytic machinery for such designs is rooted in the same tools used within trial-based paradigms for understanding the neural basis of perception and cognition. The idea is certainly not to call for the replacement of conventional trial-based paradigms, but to highlight that continuous approaches can directly complement what can be learned from punctuate trials, ballistic behaviors, and/or brief measures of activity. In this review, we have discussed the following: (1) analytic frameworks that are amenable to continuous input–output relations, while still being relatable to the signal-detection framework; (2) the existence of behavioral and neural responses that do not conform to the time scales of individual trials, and thus violate trial-based independence assumptions; and (3) the statistics of natural stimuli also span timescales distinct from trials, and thus the nervous system typically functions with inputs and goals that are not comprehensively probed within standard experimental paradigms and trials.

We propose a synthesis of these sorts of efforts into a generic framework that characterizes the broadband properties of stimuli (as opposed to attempting to simplify these properties), and which measures continuous neural activity and behavioral outputs (instead of summarizing neural activity with simple statistics and/or considering binary behavioral outputs). With these philosophies of stimulus-task design and measurement in place, the analytic framework appears within reach. We conclude by identifying three key areas for continued development. First, the Kalman filter framework initially proposed by Bonnen et al. (2015) implements the simplest proof-of-concept assuming the stimulus is a random walk composed of Gaussian noise. A variety of extensions of the original Kalman filter exist, extending the solution to nonlinear and non-Gaussian state-space models (Sorenson, 1966; Uhlmann, 1992; Julier and Uhlmann, 1997). Future work will need to both identify appropriate stimuli and the corresponding Kalman filter solutions. Second, tools for analyzing temporally continuous neural data will need to be adapted. There are several promising instances of such tools, from the GLM framework used to characterize spike trains, to the frequency-based tools used for field potential recordings. However, linking these tools together and to behavior will better integrate this endeavor. Third, the larger experimental design and analysis loop needs to be knitted tighter, with the previously discussed analytic developments pointing to a class of broadband and/or continuous stimuli and behavioral measures that are not just appropriate but maximally efficient and/or insightful.

It is also important to note that the use of continuous behavioral responses has long been seen as valuable for insight into how the brain works. In one line of work, limb movements have been recognized as providing richer insight than discrete button presses or fast saccadic eye movements because of their more continuous and slower nature (Wolpert and Ghahramani, 2000; e.g., Janssen and Scherberger, 2015). These are elegant experiments, working within the confines of trial-based designs and analyses to show the isolated power of even brief bursts of continuous output.

And of course, extensive work has focused on smooth pursuit eye movements (Lisberger et al., 1987; Barnes and Ruddock, 1989; Boudet et al., 2006; Lisberger, 2015; Liu et al., 2016; Mukherjee et al., 2017). This line of work is even more conceptually related to our framework. Although it is trial-based, it involves the continuous presentation of a moving stimulus and the corresponding analysis of eye movement timeseries. As opposed to "lower" oculomotor behaviors, the task is volitional, in that subjects can choose whether or not to pursue a target. Furthermore, the moving target can be manipulated and perturbed to further probe the sensorimotor loop. Our approach expands upon the impacts of pursuit paradigms, as it allows focus on longer periods of behavior and far more flexible stimulus-response mappings. For example, subjects need not directly track motion with the movement of their eyes or finger, and tasks can include more strategic components, such as following the center of an expanding optic flow field (Knoell et al., 2017). Thus, our proposal certainly does not stand in isolation but may generalize the power of trial-based and specialized continuous behaviors to richer representations of the environment, and a wider range of links between sensation and action.

In conclusion, the convenience of chopping inputs and outputs up into trials makes a lot of sense for allowing straightforward analyses of both brain function and behavior. However, at this time, enough pressure has accumulated to suggest that strict adherence to the trial is fated to providing only a partial and somewhat artificial understanding of how intelligent actions are generated by the brain. Somehow, the brain grapples with slow internal fluctuations in its own activity, slow external fluctuations in sensory stimuli, and the need to not just respond continuously, but to do so in a predictive manner. The analysis of natural statistics, naturalistic tasks, and/or slow fluctuations in brain activity should no longer be seen as niche enterprises, but rather as the starting points for important generalizations of our tools and subsequent understanding.

Footnotes

• A.H. was supported by National Eye Institute R01-EY020592. K.B. was supported by the National Science Foundation DGE-1110007 and the Harrington Fellowship Program (University of Texas at Austin). B.J.H. was supported by National Institutes of Health/National Institute of Neurological Disorders and Stroke Intramural Research Program, Klingenstein-Simons Neuroscience Fellowship, and Leon Levy Neuroscience Fellowship.

• The authors declare no competing financial interests.

• Correspondence should be addressed to either of the following: Dr. Biyu J. He, 435 E 30th St, Rm 1303, Neuroscience Institute, New York University Langone Medical Center, New York, New York 10016, biyu.jade.he{at}gmail.com ; or Dr. Alexander Huk, 1 University Station, A8000, The University of Texas at Austin, Austin, TX 78712, huk{at}utexas.edu

walshardes1989.blogspot.com

Source: https://www.jneurosci.org/content/38/35/7551

Share this post