Recent content on Median Watch
My friend and medical colleague John Fraser once said to me that I could sniff out a bad dataset like he could sniff out a patient in trouble. Regardless of their blood results or history, John can look at a patient, and his instincts tell him that’s something wrong. John has seen a lot of patients, I have seen a lot of numbers. Your skills build up. Homeless numbers Earlier this year, my local…
RORI’s interesting report on the future of peer review noted a lack of cross-over of ideas between peer review in funding and peer review in journals. Having worked on both, I have ideas. Lotteries for journals Conditional lotteries are becoming popular with funders. After we were publicly made fun of for suggesting them with the Australian NHMRC in 2015, they are now all over the world. They are…
Why aren’t ethics applications publicly available? If not the full application, then at least the investigators' names, the title, a plain language summary, the application number, and the date it was approved. A public record would help confirm that the research actually took place. We live in a world where entire papers can be written in under 30 minutes. Many thousands of researchers are…
There’s been another call to “Bring back panels!” for the NHMRC, this time published in Science. These calls normally appear on social media on the day of major funding announcements, when the majority of people find out that they’ve been unsuccessful and vent their frustrations. There are some scientists who, when their applications are rejected, don’t consider that…
Breadcrumbs feature in the fairy tale Hansel and Gretel where two lone children drop them along their journey into the forest so that they can find their way home. I’ve been thinking a lot about breadcrumbs and paper trails because the research world is rapidly moving to widely using artificial intelligence, including for generating questions, collecting data, and writing papers. Whilst…
Last week, I blocked out two hours of protected time in my diary for “grant writing”. I’ve done this before, but the difference this time was that at the end of two hours I had a nearly finished NHMRC Ideas Grant. Of course I used AI. I used PRISM, a new free tool from OpenAI designed for academic writing. I gave PRISM the Ideas Grant criteria, a document on what makes a good application, and a…
Science prides itself on being self-correcting. While scientific fakery has always been a problem, cases of fraud have been isolated, and a combination of scepticism and scrutiny has up to now generally worked to highlight published papers that are unreliable. The 30 minute paper However, the world of research and publishing is changing. The introduction of agentic artificial intelligence (AI)…
Last month I wrote a piece in Nature to announce that I’m going to halve my research output. For me, the reasons are clear. Publication numbers are skyrocketing and are now being supercharged by researchers using AI to write papers. Many papers are now created simply to pad CVs and have no scientific value. The scientific community needs to be careful with the limited resource that is peer review…
When fraudsters make up research data, they can make mistakes. Real data is rich and complex whilst fraudsters are on a get-rich-quick scheme and make slapdash errors. One mistake they make is in randomised trials, where it’s standard to have a baseline table that compares the randomised groups. As the groups are randomised, the summary statistics should be similar. Fraudsters have no sense…
Originally published in the Deeble and AusHSI newsletter. Vincent van Gogh only sold a few paintings during his lifetime and died a poor man. After he died, his sister-in-law, Johanna van Gogh-Bonger, cleverly built his reputation and created a thriving market for his paintings. Recent sales have been over USD $80 million. Vincent was both truly gifted and desperately unlucky. Soon after his…
My eleven year-old uses lots of words that go over my head, such as “skibidi”, “bruh” and “floptropica”. I’m not meant to understand their conversations and that’s rad. I also struggle to understand many scientific papers. Sometimes it’s my fault, as I’m too tired or too dumb. But sometimes it’s the authors’ fault, as they’ve drowned their ideas in verbiage and acronyms. We wrote a fun paper…
“What is the purpose of research funding?” Phew, that is a big question. If I could hear your answers, I am sure there would be a wide range of opinions. Let’s put that in the too hard basket. What about this question, “What should research funding not do?" Again, I can imagine lots of answers, but there are likely some answers that we could all agree on. How about an axiom that “A research…
Reading Adrian Edmondson’s excellent autobiography, he mentioned Sturgeon’s law which is: “Ninety percent of everything is crap”. Adrian is a comedian and was applying the law to his creative work. Sturgeon was using it talk about science fiction, but I think it also applies to scientific research, and Sturgeon’s number is strikingly similar to the estimate from…
Reproduced from the DORA blog. The winter months can get cold in Belfast, the largest city in Northern Ireland where the Titanic was designed, built and launched in the early 1900s. Not seriously cold of course, never sub-zero depths of cold that were the undoing of the Titanic on its maiden voyage, but a ‘nippy’ cold… a cold that comes with biting winds and rain that means only the bravest of…
Reproduced from the LSE impact blog. Here’s their introduction: Based on an analysis of over 32 million abstracts published over the last fifty years, Adrian Barnett and Nicole White find a marked rise in common spelling errors. Evidence they suggest of a culture of quantity over quality in academic writing. Pressure Many academics feel pressure to publish lots of papers every year because…
Reproduced from the LSE impact blog. Here’s their introduction: Peer review decisions are definitive, and depending on the style of peer review practiced at a journal, reviewers can usually make one of three recommendations: accept, reject, revise and resubmit. Discussing a new study into the levels of certainty reviewers have making these choices, Adrian Barnett suggests how embracing this…
I say real job, because I’d been working whilst studying as a dishwasher, packer, waiter, bingo operator, book-binder, and in a bookies. The interviews for these jobs were straightforward, turning up on time and in the right place got you half way there. My first real interview – way back in 1994 – was for a graduate statistician position at a big drug company in the UK. They paid my…
We’ve just published the world’s first randomised trial of funding (paper available here). We ran a truly novel study, using a gold standard study design, and with a published protocol. So why did it take two years to get through peer review? Too much care Our team also recently published a large randomised trial about reducing unnecessary care at the end of life (paper available…
Last week I saw Alison Avenell give a great talk titled “Improving the integrity of published research: How, when, and if?’’ This was on her experience of finding fraudulent papers and what actions the journals took to correct the record – which was too often nothing. One of Alison’s recommendations was to avoid inadvertently citing retracted papers by checking against the…
Reproduced from The Conversation Our lives are becoming increasingly data driven. Our phones monitor our time and internet usage and online surveys discern our opinions and likes. These data harvests are used for telling us how well we’ve slept or what we might like to buy. Numbers are becoming more important for everyday life, yet people’s numerical skills are falling behind. For example, the…
A cover of Science from 2018 (available here) shows two giant scientists peering down on other scientists at work. It is a great image to describe metascience, also known as meta-research or the “science of science”. This is the growing field of scientifically studying science, with the aim of understanding why it sometimes fails and how we can make it better. At AIMOS (Association for…
As a teenager I watched a lot of films starring Paul Newman and Steve McQueen and loved them all. As an adult I was disappointed to learn they had petty fights about whose name appeared first in the posters. Two big egos fighting about prestige will be familiar to many in academia. Academia presents itself as an intellectual pursuit, but off-stage there are always squabbles about who deserves the…
A colleague sent me a draft manuscript with the typo “statically significant”. A typo that passes a spell check but would surely not pass reviewers and editors? Oh dear, a PubMed search reveals that it has snuck past reviewers and editors, many many times. There are 975 abstracts that have used this nonsense phrase. There should be a celebration for the 1000th paper!…
Reposted from The Conversation. Fraud in science is alarmingly common. Sometimes researchers lie about results and invent data to win funding and prestige. Other times, researchers might pay to stage and publish entirely bogus studies to win an undeserved pay rise – fuelling a “paper mill” industry worth an estimated €1 billion a year. Some of this rubbish can be easily spotted by peer reviewers,…
Reposted with permission from Campus Morning Mail. Researchers have a dark sense of humour when it comes to research funding. One of my favourite sneers was from the ecologist Terry McGlynn who said, “instead of writing this grant, I should walk the whole country and get a penny from each person. Same amount of money, but less hassle.” Applicants to last year’s early- and mid-career Medical…
Reposted with permission from 360info. New Zealander Nigel Richards has won the French Scrabble championship twice. What’s more remarkable than double wins is that Nigel doesn’t speak French. He applied his prodigious brain to the task of memorising words from the French dictionary, bypassing the need for understanding. In 2022, The Lancet medical journal achieved a feat that has parallels with…
I am a statistician. I rely on data for my career. Luckily data is everywhere, so I’m often spoilt for choice. One place where I no longer look for existing data is the health and medical system in Australia. This comes from multiple painful experiences of spending months on approval processes that end in failure. Ad nauseum examples Many other researchers have spoken about the difficulties…
In my talk on bad statistics in medical research, I showed the infamous plot of Z-values created by Erik van Zwet. A version of the plot made with David Borg is shown below. The sample size is over 1.1 million Z-values. {width=450px} The two large spikes in Z-values are just below and above the statistically significant threshold of ± 1.96, corresponding to a p-value of less than 0.05. The plot…
Re-posted from Campus Morning Mail with permission. The National Statement on Ethical Conduct in Human Research is a vitally important document for researchers and the public, because it determines what research is acceptable and guides how researchers should design studies to adhere to ethical practice. There is a current public consultation on changes to the statement. One proposed change is…
Two weeks ago I gave a fun online talk on statistics for the Young Scientist Forum of the German Society for Biomaterials. I had some great chats with the organisers and there were good questions from the audience. One good question was about how to interpret the analysis results when things are not clear cut. During my presentation I had talked about not deleting difficult outliers and not…
Sometimes I fantasise about being in an earthquake. Not a big one, just enough to feel the ground shake, to see the earth move for a few seconds. I should be careful what I wish for. Last Sunday I was in the plughole of a proper natural disaster as massive amounts of rain tried to barge its way into our house. It wasn’t any fun. Our house is fine, and other friends have had far worse…
Re-posted from Campus Morning Mail with permission. Last week I pledged to work for any Australian university for free if they abandoned league tables. So far no university has been bold enough to take up my offer. I am one of many academics who have criticised league tables using science, demonstrating their many statistical flaws and negative consequences. But all this scientific criticism has…
Johnson & Johnson have a funding scheme for women in STEM. Of course this is a great idea and much needed given the chronic under-representation of women in health and medical research, which was confirmed again this week with the latest NHMRC figures. But there’s a huge potential problem with this scheme: the success rate is under 1%. The last round had 650 applications and 6 awarded. Given…
One year ago after another stupid fight with a journal about p-values, I made a pledge to go without them for a year. Here’s how it went. But first, why? I am aware of the arguments for and against p-values. I have used p-values for a long while and they can be a useful statistic. The reason I ditched them is because almost nobody in health and medical research interprets them correctly, wrongly…
In 1998, the French men’s team won the football World Cup for the first time in their history. For most of the country, even non-football fans, it was a joyous moment and the trigger for a massive national party, especially as the tournament was in France with the final in Paris. At least one Frenchman did not join the party: the rancorous politician Jean Marie Le Pen, who complained that the team…
Sorry state. A few years ago I thought about writing an article that apologised to the public about the poor state of health and medical research. Their tax pays for this research and they give their time and data, and yet far too often the final results are totally unreliable. In the end I bottled it; too worried about the potential harm to my career. But today I’ve read this important…
Blog pause It’s been a long while between blogs. I won’t bore you with how busy I’ve been, but one thing that’s kept me away from the fun of writing blogs is the “fun” of writing grants. Beans To get more research funding we rightly have to explain the impact of our past funding. It’s possible to get examples of impact by seeing how other researchers have…
Show me the shortcut Scientists are busy people. Busy people love a short-cut because it gives them more time to be busy. The p-value is a well-used scientific short-cut. It can decide for us whether something is important or not, and it’s based on an equation so it must be right. Another heaven-sent shortcut is the h-index which allows us to decide the careers of researchers based on just…
Re-posted from this 2016 AusHSI blog because this is still an issue. Let’s start with the obvious. Winning funding for health and medical research is soul-crushingly hard. Success rates for major schemes are under 20%, so failure is the norm. Your application will be judged by a panel of 6 to 12 senior researchers. A key marker of success is your track record, which may simply mean the…
Yet another p-value run-in. For a recent observational study I tried to limit the use of p-values in the paper. My colleagues wanted more p-values and I had to politely push back. During one team meeting I even offered to put the p-values in if someone could accurately tell me what they meant … silence. Predicting that the reviewers would also want to see more p-values, I added this sentence to…
Our paper examining trends in acronyms in abstracts was recently published in eLife. We examined over 26 million abstracts from the PubMed database, which is easily the largest data set I’ve ever used. In this post I talk about some of the challenges and benefits of dealing with such a massive data set. Data greed One of the most common mistakes I see researchers — new and experienced — make is to…
Today I end my two year stint as president of the Statistical Society of Australia. As I press “submit” on my presidency, here’s a hodge-podge of reflections. I was delighted to be president, and I will miss being able to say, “I’m the president of the Stats Society”. Statistics has given me an incredible career and I feel I owe something back. It might sound…
Re-posted from the AusHSI blog (8 May 2015). After yet another failed fellowship application I considered if I should leave research. I now have seven fellowship failures and no successes, and that seems like a lot. Success rates for grants are nose-diving and even the former head of the NHMRC says that researchers should be considering other careers (pay-walled). I’ve spent a lot of time running…
My daughter loves a song by Parry Gripp called “I did the dishes”. The chorus is “I did the dishes, I’m a hero.” I can’t work out if the song is trying to encourage kids to do the dishes, or if it’s making fun of people who think their housework is heroic, or a bit of both. “Plates, done! Bowls, done! Gravy boat, done!” I helped some people,…
Re-posted from the March 2020 issue of Advocate. The smell of pool chlorine reminds us of work. Both of us admit to doing emails while sitting at the local pool waiting for the kids to finish their lessons, as well as at cafés, on airplanes, waiting in queues. At least one of us does the odd email or two from the bathroom. We are academics and therefore work anywhere, anytime (maybe most of the…
Re-posted from Campus Morning Mail. Poor quality medical research is nothing new. A major cause is the race to be published first, which means researchers do not adequately check their work. The COVID-19 crisis has put an already pressured system under even more strain, and the cracks are clearly visible. Small studies have been used to justify massive changes in clinical practice, such as the…
I’ve done a lot of research on seasonal models. I’ve always enjoyed playing with the elegantly simple sinusoid function, which can fit a remarkable array of shapes with just a few parameters. Put enough sinusoids together and you can create any sound. When Fourier discovered this, it was considered too simple to be true. A great number of diseases are seasonal, so these models are useful as well…
Yet another COVID model. I did this modelling because I was asked to provide some COVID estimates for a hospital. There have been lots of models in the last few weeks and I don’t want to reduce the signal-to-noise ratio in this vitally important area, but I’m sharing this in case someone finds my approach useful. All the code is here. I have used similar models before to simulate disease numbers…
Statistics can be hard Here’s a great quote about working with statistics from the fantastic statistician David Spiegelhalter: “I am often asked why people tend to find probability a difficult and unintuitive idea, and I reply that, after forty years researching and teaching in this area, I have finally concluded that it is because probability really is a difficult and unintuitive idea”…
The gender disparity in research funding. There is a large gender disparity in the number of research grants awarded in Australia. For years men have won more funding than women. This disparity in success is driven by a disparity in applications, as women apply far less often. Success rates for men and women are relatively close, so policies to reduce the gender disparity in funding should focus…