Viewpoint Expert opinions on trends and controversies in autism research.
Opinion / Viewpoint

Registered reports

by  /  14 June 2013

CONNECTIONS SFARI.org’s columnist Jon Brock explores the connections between the brain, social skills and autism.Read more columns »

These are fast-moving times for autism research. Every week brings a new swath of research findings that promise fresh insights into the causes of autism, its diagnosis and treatment. Yet, beneath the flurry of publications, the reality is that progress has been painstakingly slow.

One reason, unfortunately, is that many published studies contain results that turn out not to be true. This isn’t because scientists are lying or fabricating their data. It is a consequence of the way science is done and the pressures on researchers to produce results.

The scientific approach is meant to guard against erroneous findings. Statistical analyses are used to indicate how likely it would be to get similar results just by chance. By convention, a result is only considered to be statistically significant if there’s less than a five percent chance of it being a fluke. So, naively, we might expect 95 percent of published research findings to be true.

The problem is that, as researchers, we are under pressure to find and report significant results. It’s much harder to get results published — particularly in high-profile journals — if they’re not statistically significant. And often, we have a vested interest in the hypothesis we’re testing. We’re testing our own ideas and, naturally, we want our ideas to be right.

We also have a lot of freedom in the analyses we choose to conduct and the outcomes we choose to report. If the data don’t quite come out as significant, we can always try a slightly different analysis, collect a few more data points, or find a reason to throw out data that don’t fit the general pattern. And we can keep going until we get a statistically significant result.

The more we poke around, the more likely we are to find a significant effect — and the more likely that the effect we end up reporting is just a fluke.

Research plans:

Awareness of this problem is growing. Last week, in a letter to the British newspaper The Guardian, U.K. neuroscientists Chris Chambers and Marcus Munafo called for the widespread introduction of a new form of journal article, the ‘registered report,’ which they hope will address the issue.

The journal Cortex, where Chambers is an associate editor, is already trying out registered reports. Before they even begin to collect their data, researchers have to submit a proposal to the journal, detailing exactly what they are going to do. This includes the analyses they plan to conduct, as well as the methods they will use to collect the data. 

Bastian – statistically-funny.blogspot.comIt was getting harder and harder to find a truly meaningful relationship at the medical journal happy hour.

Once the data are in, the paper is accepted not based on the results, but on whether the researchers did exactly what they said they would. 

There’s no room for fudging. Whatever the results turn out to be (significant or otherwise), we will know exactly how much we can trust them.

A similar approach is already standard practice in medical research. Before testing a new drug in a clinical trial, researchers have to state exactly what outcomes they are looking for.

However, trials of other forms of treatment or intervention often go unregistered. Researchers don’t want to commit to a single outcome measure, presumably because they’re not sure from the outset what the best measure might be. But if they’re allowed to pick and choose their outcome measures after the event, there’s a good chance of there being at least one measure that appears to support the treatment — even if the treatment is completely ineffective.

The proposal for registered reports is a recognition that we need to apply the same standards to all research, not just clinical trials.

Of course, it’s often the case that the most interesting findings are the ones we least expect. Many important scientific findings were discovered by accident. And there’s nothing intrinsically wrong with reporting findings that weren’t predicted.

Still, we should always be skeptical about findings that weren’t predicted. And of course, if the effects are real, then further studies should replicate the results.

Registered reports should be seen as the gold standard for research. They’re not the solution to all of the challenges facing autism research — or science in general. But they may be a way of identifying those findings in which we can have a high degree of confidence.

Jon Brock is a research fellow at Macquarie University in Sydney, Australia. He also blogs regularly on his website, Cracking the Enigma. Read more Connections columns at SFARI.org/connections »


7 responses to “Registered reports”

  1. D.A. says:

    -“A similar approach is already standard practice in medical research. Before testing a new drug in a clinical trial, researchers have to state exactly what outcomes they are looking for.
    (…)
    However, trials of other forms of treatment or intervention often go unregistered.”

    I always found this strange: think about all psychological research related to psychological disorders/ findings, which to me seem directly related to people’s well-being. It amazes me that pre-registration hasn’t been required decades ago for these very important research findings.

    -“Registered reports should be seen as the gold standard for research.”

    Hear, hear !!

  2. ASDDad says:

    “Yet, beneath the flurry of publications, the reality is that progress has been painstakingly slow.”

    Only if your a geneticist … forty years without much fruit from the tree probably equates to barking up the wrong tree.

    Looking at other fields of medical research like immunology and neurology shows that there has been substantial progress and the first treatment regimes are just around the corner.

    • Steve White says:

      I am not a scientist but seems to me this “not much” comment is not accurate.
      My own kid’s autism was probably caused by maternal antibodies. The molecules the more common antibodies react to have been identified, a commercial test is being developed, so I don’t think one can say “not much” happened in that research area.
      There are reports, accurate I don’t know, of some autism traits being reversed in mouse models after drug treatment. That will be “much” not “not much” if it’s borne out in humans.
      A lot of candidate genes have been identified.
      If autism had one cause, or two, or a dozen, then complaints about failure to find them would be I think reasonable. After all a dozen causes is still only one cause for every 200 researchers at the IMFAR conference. But what if there are hundreds of causes?
      Just the musings of a laymen, but personally I think they will never get rid of autism. Too many things to go wrong. But it will probably be cut down by half in the wealthy nations within ten years or so.

  3. Dow Dae Dung says:

    I think many areas of autism research (and not just genetics) have a lot of false positive results, ASDDad, so if you perceive rapid progress, unfortunately you may be thinking about things that were published, but sad to say, are just not so…

    Autism has gobbled up enormous resources (because so many well off families have lobbied hard to put resources there). Well, the resources were spent, but what is there to show for it? Not much.

  4. Jon Brock says:

    Thanks for all the comments.

    One thing that perhaps didn’t come across very well is that these issues aren’t specific to autism research. Researchers across various fields are beginning to realise just how easy it is to get false positives – and how biased and messy the literature is as a result. Some fields are I think further ahead than others, both in terms of awareness of the problem, and in finding ways to deal with it.

    The wider problem is that, until enough people are aware of the issues, the publishing system actually penalises researchers who are careful in how they analyse their data. It’s sometimes easier to get a false positive published than it is to get a true null result published.

    And the main message, really, is to never put too much faith in a single study. There’s a very good chance that it’s wrong. It’s really only when several different (and hopefully independent) studies arrive at the same conclusion that we can have confidence that an effect is real.

    So, in answer to some of the comments, I do think we’re making progress in autism research, but I also think greater awareness of these issues will lead to faster progress. Less heat, more light.

  5. RA Jensen says:

    “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’, but ‘That’s funny…'” -Isaac Asimov

    To reject a study that fails to meet an expected result but finds an unexpected result would have delayed many of the important discoveries in the history of science. Too many researchers in the field of autism live in narrow silos that ignore the wider literature. Behavioral geneticists consider genes as the only important arena worthy of study and environmental researchers consider the environment as the only important arena worthy of study. Autism is a multifactorial disorder. To claim that autism is only genetic or only environmental is to deny that autism is a multifactorial disorder. Several examples below show how narrow concepts of autism have led to the glacial pace of new discoveries.

    The MET (7q31) gene region harbors a polymorphism, the MET promoter variant rs1858830 allele “C” that is present in 47% of the general population and is associated with immune function, gastrointestinal repair, neuronal growth, migration and brain development.

    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3507704/

    Multiple studies have claimed that the Met promoter variant rs1858830 allele “C” is an autism candidate gene. This common inherited polymorphism is almost as common as the presence of a Y chromosome and these claims are astonishingly exaggerated. The claims aren’t meaningless but do suggest that environmental risk factors, epigenetics and transposable elements (jumping genes) have in important role to play in disrupting normal functioning of the rs1858830 allele “C”.

    Copy number variations (CNV’s) are associated with autism risk. All of us possess CNV’s inherited or de novo. How does CNV’s contribute to health and disease? An important clue has been provided by AIDS researchers. They have discovered that CNV’s in the gene CCL3L1 (17q11.2) has both risk and protective effects. CNV’s lower than population norms increase risk for HIV infection while CNV’s higher than population norms confers protection against HIV infection.

    http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0015778

    Genome wide association studies have identified 17q11.2, the region that harbors the CCL3L1 gene, as an autism candidate gene. CNV‘s in the 17q11.2 region may also be associated with autism risk via maternal infections. CNV’s in CCL3L1 is not restricted to risk for HIV infection. Genes within this region harbor CNV’s associated with a variety of diseases, with viral infections and autoimmune diseases being the most represented categories.

    http://www.ncbi.nlm.nih.gov/pubmed/15647115

    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2990928/

Leave a Reply

Your email address will not be published. Required fields are marked *