A few months ago, I spent almost a week trying to replicate a published “causal” association which had received >500 citations in the last 5 years. My aim was to provide a better effect estimate and to do this, I used two different datasets, one with similar and another with a larger sample size. However, both of my analyses returned null results (i.e. no effect of exposure on outcome). Positive controls were carried out to make sure the analysis pipeline was working correctly. Ultimately, I moved on to other ‘more interesting’ projects as there was no point spending time writing a paper that was probably going to end up in a ‘not-so-prestigious’ journal and never going to get >500 citations or be weighted heavily when I apply for grants/fellowships.
Consequently, inadvertently I contributed to publication bias on this issue – and no other analyses on the subject matter were published since the original publication, so I am sure others have found similar results and chose not to publish.
However, I have changed my mind about publishing null/negative results after encountering Russell, Wittgenstein and others’ long debates on proving ‘negative’ truths/facts (and in a nutshell, how hard it is to prove negatives – which should make it especially important to publish conclusive null findings). These giants of philosophy thought it was an important issue and spent years structuring their ideas but here I am, not seeing my conclusive null results worthy of publication. I (and the others who found similar results) should have at least published a preprint to right a wrong – and this sentiment doesn’t just apply to the scientific literature. I also think academics should spend some time on social media to issue corrections to common misconceptions in the general public.
This also got me thinking about my university education: I was not taught any philosophy other than bioethics during my undergraduate course in biological sciences (specialising in Genetics in the final year). I am now more convinced than ever that ‘relevant’ philosophy (e.g. importance of publishing all results, taking a step back and revisiting what ‘knowledge’ is and how to attain ‘truth’, how to construct an argument1, critical thinking/logical fallacies, what is an academic’s intellectual responsibility?) should be embedded and mandatory in all ‘natural science’ courses. This way, I believe future scientists and journal editors would appreciate the importance of publishing negative/null results more and allow well-done experiments to be published in ‘prestigious’ journals more. This way, hopefully, less published research findings are going to be false2.
References/Further reading:
- Think Again I: How to Understand Arguments (Coursera MOOC)
- Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Med. 2(8): e124 (2015)
- How Life Sciences Actually Work: Findings of a Year-Long Investigation (Blog post)
- An interesting Quora discussion: Why do some intelligent people lose all interest in academia?
- Calculating the ‘worth’ of an academic (Blog post)
*the title comes from the fact that today’s natural scientists would have been called ‘natural philosophers’ back in the day