SUBJECT. One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning. TASK. The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing.
Craig M. Bennett1, Abigail A. Baird2, Michael B. Miller1, and George L. Wolford3. Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction.” Neuroimage 47.Suppl 1 (2009): S125. (pdf)
1 Psychology Department, University of California Santa Barbara, Santa Barbara, CA;
2 Department of Psychology, Vassar College, Poughkeepsie, NY;
3 Department of Psychological & Brain Sciences, Dartmouth College, Hanover, NH
With the extreme dimensionality of functional neuroimaging data comes extreme risk for false positives. Across the 130,000 voxels in a typical fMRI volume the probability of a false positive is almost certain. Correction for multiple comparisons should be completed with these datasets, but is often ignored by investigators. To illustrate the magnitude of the problem we carried out a real experiment that demonstrates the danger of not correcting for chance properly.
Subject. One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning.
Task. The task administered to the salmon involved completing an open-ended mentalizing task. The salmon was shown a series of photographs depicting human individuals in social situations with a specified emotional valence. The salmon was asked to determine what emotion the individual in the photo must have been experiencing.
Design. Stimuli were presented in a block design with each photo presented for 10 seconds followed by 12 seconds of rest. A total of 15 photos were displayed. Total scan time was 5.5 minutes.
Preprocessing. Image processing was completed using SPM2. Preprocessing steps for the functional imaging data included a 6-parameter rigid-body affine realignment of the fMRI timeseries, coregistration of the data to a T1-weighted anatomical image, and 8 mm full-width at half-maximum (FWHM) Gaussian smoothing.
Analysis. Voxelwise statistics on the salmon data were calculated through an ordinary least-squares estimation of the general linear model (GLM). Predictors of the hemodynamic response were modeled by a boxcar function convolved with a canonical hemodynamic response. A temporal high pass filter of 128 seconds was include to account for low frequency drift. No autocorrelation correction was applied.
Voxel Selection. Two methods were used for the correction of multiple comparisons in the fMRI results. The first method controlled the overall false discovery rate (FDR) and was based on a method defined by Benjamini and Hochberg (1995). The second method controlled the overall familywise error rate (FWER) through the use of Gaussian random field theory. This was done using algorithms originally devised by Friston et al. (1994).
A t-contrast was used to test for regions with significant BOLD signal change during the photo condition compared to rest. The parameters for this comparison were t(131) > 3.15, p(uncorrected) < 0.001, 3 voxel extent threshold.
Several active voxels were discovered in a cluster located within the salmon’s brain cavity (Figure 1, see above). The size of this cluster was 81 mm3 with a cluster-level significance of p = 0.001. Due to the coarse resolution of the echo-planar image acquisition and the relatively small size of the salmon brain further discrimination between brain regions could not be completed. Out of a search volume of 8064 voxels a total of 16 voxels were significant.
Identical t-contrasts controlling the false discovery rate (FDR) and familywise error rate (FWER) were completed. These contrasts indicated no active voxels, even at relaxed statistical thresholds (p = 0.25).
To examine the spatial configuration of false positives we completed a variability analysis of the fMRI timeseries. On a voxel-by-voxel basis we calculated the standard deviation of signal values across all 140 volumes.
We observed clustering of highly variable voxels into groups near areas of high voxel signal intensity. Figure 2a shows the mean EPI image for all 140 image volumes. Figure 2b shows the standard deviation values of each voxel. Figure 2c shows thresholded standard deviation values overlaid onto a high-resolution T1-weighted image.
To investigate this effect in greater detail we conducted a Pearson correlation to examine the relationship between the signal in a voxel and its variability. There was a significant positive correlation between the mean voxel value and its variability over time (r = 0.54, p < 0.001). A scatterplot of mean voxel signal intensity against voxel standard deviation is presented to the right.
Can we conclude from this data that the salmon is engaging in the perspective-taking task? Certainly not. What we can determine is that random noise in the EPI timeseries may yield spurious results if multiple comparisons are not controlled for. Adaptive methods for controlling the FDR and FWER are excellent options and are widely available in all major fMRI analysis packages. We argue that relying on standard statistical thresholds (p < 0.001) and low minimum cluster sizes (k > 8) is an ineffective control for multiple comparisons. We further argue that the vast majority of fMRI studies should be utilizing multiple comparisons correction as standard practice in the computation of their statistics.
Benjamini Y and Hochberg Y (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57:289-300.
Friston KJ, Worsley KJ, Frackowiak RSJ, Mazziotta JC, and Evans AC. (1994). Assessing the significance of focal activations using their spatial extent. Human Brain Mapping, 1:214-220.
IgNobel Prize in Neuroscience: The dead salmon study
I have to say that I am incredibly pleased that this study won the Ignobel. Not just because it’s a really fun study, but also because it really is one of those studies that makes you laugh, and then makes you THINK. And in the case of this study in particular, it has changed a lot about how we think about making corrections in fMRI, and may have actually really affected the way the data are published. And so, I present to you: the dead salmon study.
Bennett et al. “Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument For Proper Multiple Comparisons Correction” Journal of Serendipitous and Unexpected Results, 2010.
This study began as a fun trial, and almost never saw the light of day. But since it has, it has become a really important study in the field of functional magnetic resonance imaging, which measures changes in blood oxygenation levels in the brain during tasks (usually in humans). These studies are widely used and very widely cited in the media, and they have told us a great deal about the brain, our mental abilities, and certain disease states. But until this study, not all of these studies were really adequately controlled. And now, it looks like things are getting better. And all because of a dead salmon.
It all started when Bennett et al were setting up their…um…real experiment. They were going to look at humans and their responses to social stimuli. But in order to do this, they have to first test the machine. Apparently, when you usually test an fMRI machine, you put a big balloon in there filled with mineral oil. Just to test it and look for contrast, etc. The authors of this study wanted something different, they wanted something with more contrast and different types of texture. So first they bought a pumpkin.
(Courtesy of the authors. You can see the seeds on the inside if you look carefully!)
They got some good signals, but not very good contrast. They next tried a Cornish Game Hen (dead, defeathered, from the store). That also produced good visuals but wasn’t quite what they needed. The authors needed something with good contrast, but also with several clearly defined and distinguishable types of tissue: fat, bone, muscle, etc.
Enter the salmon.
The lead author, Dr. Craig Bennett, wanted to get something fresh, so he headed in to the grocery story first thing in the morning. At the fish counter, he spoke the words that will echo down the centuries as a testimony to the dedication and drive of neuroscientists throughout the ages:
“I need a full length Atlantic Salmon. For science.”
I am still shocked that the guys at the fish counter didn’t give him a discount. Can’t you get a discount for science?!
Having procured the specimen, the authors placed the salmon in the fMRI and ran all the usual anatomical scans, and then ran the experimental set of the study as well. In this study, the salmon was shown images of people in social situations, either socially inclusive situations or socially exclusive situations. The salmon was asked to respond, saying how the person in the situation must be feeling. The salmon, as far as I can tell from the paper, did not comply with instructions. Naughty salmon.
The results were set aside and not looked at for a good while, until one of the other authors of the study was running a seminar on how to properly analyze fMRI data. They wanted to do some improper analysis on something improbably, and remembered that they had the salmon data on the computer. And a study was born.
Now, to clarify: what exactly were they doing? Well, when you do fMRI studies in the brain, there’s a ton of information there. The information is generally broken down into sections called voxels. Up 130,000 of them in a single study and contrast selection, looking at each one to see if it is ‘activated’ compared to the others. And doing the statistics on these studies gets to be a problem. You have to do thousands of comparisons, and you being to run into something called the “multiple comparisons problem“. If you do a lot of tests, at least some of them will come out positive, even if they are not real. These are called false positives, and they are something you really want to watch out for.
To solve this problem, there are various methods for correcting the multiple comparisons, but this also means that you lose a lot of statistical power. In other words, you get rid of your false positives, but it might mean you don’t see things that are really there, you might find false negatives instead. There is a running debate in the fMRI field over whether false positives or false negatives are more dangerous. The authors of this study contend (and I am inclined to agree) that the false positives are more likely to get overblown and lead to problems down the line. For a really good wrapup on the stats questions, I recommend neuroskeptic’s piece on the topic.
So in the final results, the authors compared the normal multiple comparisons, with the multiple CORRECTED comparisons. When they used the multiple corrected comparisons, the dead salmon showed nothing. When they did the multiple comparisons without the correction, the salmon showed significant increases in “activation”, coincidentally, in the brain and spinal cord. This shows the importance of correcting for multiple comparisons and avoiding false positives.
The original poster almost didn’t make it to a conference, but when it did, it made a major splash, and reactions were very positive. Some people like to use the salmon study as proof that fMRI is woo, but this isn’t the case, it’s actually a study to show the importance of correcting your stats.
And the poster, and the paper that was eventually published, may have had an effect on the field. The authors note that at the time the poster was presented, between 25-40% of studies on fMRI being published were NOT using the corrected comparisons. But by the time this group won the Ignobel last week, that number had dropped to 10%. And who knows, it might, in part, be due to a dead fish.
*I should note that at least one of the authors of this study, Dr. Craig Bennett,reads this blog!! I was so thrilled (and slightly scared, I mean, I know people read it, but…I mean what if I screw up?! Clearly I need to attend the session on “when people start taking your online ramblings seriously” at #scio13) when he told me! He’s on twitter and you can follow his sciency ramblings at @prefrontal. And he also provided the excellently seasonal fMRI photos of the pumpkin. Dr. Bennett also notes that he was not reimbursed for the salmon, but that was because they ate it later.