Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
Public understanding
In the course of thinking about metrics, I keep coming across cases of over-promoted research. An early case was “Why honey isn’t a wonder cough cure: more academic spin“. More recently, I noticed these examples.
“Effect of Vitamin E and Memantine on Functional Decline in Alzheimer Disease".(Spoiler -very little), published in the Journal of the American Medical Association. ”
and ” Primary Prevention of Cardiovascular Disease with a Mediterranean Diet” , in the New England Journal of Medicine (which had second highest altmetric score in 2013)
and "Sleep Drives Metabolite Clearance from the Adult Brain", published in Science
In all these cases, misleading press releases were issued by the journals themselves and by the universities. These were copied out by hard-pressed journalists and made headlines that were certainly not merited by the work. In the last three cases, hyped up tweets came from the journals. The responsibility for this hype must eventually rest with the authors. The last two papers came second and fourth in the list of highest altmetric scores for 2013
Here are to two more very recent examples. It seems that every time I check a highly tweeted paper, it turns out that it is very second rate. Both papers involve fMRI imaging, and since the infamous dead salmon paper, I’ve been a bit sceptical about them. But that is irrelevant to what follows.
Boost your memory with electricity
That was a popular headline at the end of August. It referred to a paper in Science magazine:
“Targeted enhancement of cortical-hippocampal brain networks and associative memory” (Wang, JX et al, Science, 29 August, 2014)
This study was promoted by the Northwestern University "Electric current to brain boosts memory". And Science tweeted along the same lines.
|
Science‘s link did not lead to the paper, but rather to a puff piece, "Rebooting memory with magnets". Again all the emphasis was on memory, with the usual entirely speculative stuff about helping Alzheimer’s disease. But the paper itself was behind Science‘s paywall. You couldn’t read it unless your employer subscribed to Science.
|
 |
|
All the publicity led to much retweeting and a big altmetrics score. Given that the paper was not open access, it’s likely that most of the retweeters had not actually read the paper. |
|
When you read the paper, you found that is mostly not about memory at all. It was mostly about fMRI. In fact the only reference to memory was in a subsection of Figure 4. This is the evidence.
That looks desperately unconvincing to me. The test of significance gives P = 0.043. In an underpowered study like this, the chance of this being a false discovery is probably at least 50%. A result like this means, at most, "worth another look". It does not begin to justify all the hype that surrounded the paper. The journal, the university’s PR department, and ultimately the authors, must bear the responsibility for the unjustified claims.
Science does not allow online comments following the paper, but there are now plenty of sites that do. NHS Choices did a fairly good job of putting the paper into perspective, though they failed to notice the statistical weakness. A commenter on PubPeer noted that Science had recently announced that it would tighten statistical standards. In this case, they failed. The age of post-publication peer review is already reaching maturity
Boost your memory with cocoa
Another glamour journal, Nature Neuroscience, hit the headlines on October 26, 2014, in a paper that was publicised in a Nature podcast and a rather uninformative press release.
"Enhancing dentate gyrus function with dietary flavanols improves cognition in older adults. Brickman et al., Nat Neurosci. 2014. doi: 10.1038/nn.3850.".
The journal helpfully lists no fewer that 89 news items related to this study. Mostly they were something like “Drinking cocoa could improve your memory” (Kat Lay, in The Times). Only a handful of the 89 reports spotted the many problems.
A puff piece from Columbia University’s PR department quoted the senior author, Dr Small, making the dramatic claim that
“If a participant had the memory of a typical 60-year-old at the beginning of the study, after three months that person on average had the memory of a typical 30- or 40-year-old.”
|
Like anything to do with diet, the paper immediately got circulated on Twitter. No doubt most of the people who retweeted the message had not read the (paywalled) paper. The links almost all led to inaccurate press accounts, not to the paper itself. |
|
But some people actually read the paywalled paper and post-publication review soon kicked in. Pubmed Commons is a good site for that, because Pubmed is where a lot of people go for references. Hilda Bastian kicked off the comments there (her comment was picked out by Retraction Watch). Her conclusion was this.
"It’s good to see claims about dietary supplements tested. However, the results here rely on a chain of yet-to-be-validated assumptions that are still weakly supported at each point. In my opinion, the immodest title of this paper is not supported by its contents."
(Hilda Bastian runs the Statistically Funny blog -“The comedic possibilities of clinical epidemiology are known to be limitless”, and also a Scientific American blog about risk, Absolutely Maybe.)
NHS Choices spotted most of the problems too, in "A mug of cocoa is not a cure for memory problems". And so did Ian Musgrave of the University of Adelaide who wrote "Most Disappointing Headline Ever (No, Chocolate Will Not Improve Your Memory)",
Here are some of the many problems.
- The paper was not about cocoa. Drinks containing 900 mg cocoa flavanols (as much as in about 25 chocolate bars) and 138 mg of (−)-epicatechin were compared with much lower amounts of these compounds
- The abstract, all that most people could read, said that subjects were given "high or low cocoa–containing diet for 3 months". Bit it wasn’t a test of cocoa: it was a test of a dietary "supplement".
- The sample was small (37ppeople altogether, split between four groups), and therefore under-powered for detection of the small effect that was expected (and observed)
- The authors declared the result to be "significant" but you had to hunt through the paper to discover that this meant P = 0.04 (hint -it’s 6 lines above Table 1). That means that there is around a 50% chance that it’s a false discovery.
- The test was short -only three months
- The test didn’t measure memory anyway. It measured reaction speed, They did test memory retention too, and there was no detectable improvement. This was not mentioned in the abstract, Neither was the fact that exercise had no detectable effect.
- The study was funded by the Mars bar company. They, like many others, are clearly looking for a niche in the huge "supplement" market,
The claims by the senior author, in a Columbia promotional video that the drink produced "an improvement in memory" and "an improvement in memory performance by two or three decades" seem to have a very thin basis indeed. As has the statement that "we don’t need a pharmaceutical agent" to ameliorate a natural process (aging). High doses of supplements are pharmaceutical agents.
To be fair, the senior author did say, in the Columbia press release, that "the findings need to be replicated in a larger study—which he and his team plan to do". But there is no hint of this in the paper itself, or in the title of the press release "Dietary Flavanols Reverse Age-Related Memory Decline". The time for all the publicity is surely after a well-powered study, not before it.
The high altmetrics score for this paper is yet another blow to the reputation of altmetrics.
One may well ask why Nature Neuroscience and the Columbia press office allowed such extravagant claims to be made on such a flimsy basis.
What’s going wrong?
These two papers have much in common. Elaborate imaging studies are accompanied by poor functional tests. All the hype focusses on the latter. These led me to the speculation ( In Pubmed Commons) that what actually happens is as follows.
- Authors do big imaging (fMRI) study.
- Glamour journal says coloured blobs are no longer enough and refuses to publish without functional information.
- Authors tag on a small human study.
- Paper gets published.
- Hyped up press releases issued that refer mostly to the add on.
- Journal and authors are happy.
- But science is not advanced.
It’s no wonder that Dorothy Bishop wrote "High-impact journals: where newsworthiness trumps methodology".
It’s time we forgot glamour journals. Publish open access on the web with open comments. Post-publication peer review is working
But boycott commercial publishers who charge large amounts for open access. It shouldn’t cost more than about £200, and more and more are essentially free (my latest will appear shortly in Royal Society Open Science).
Follow-up
Hilda Bastian has an excellent post about the dangers of reading only the abstract "Science in the Abstract: Don’t Judge a Study by its Cover"
4 November 2014
I was upbraided on Twitter by Euan Adie, founder of Almetric.com, because I didn’t click through the altmetric symbol to look at the citations "shouldn’t have to tell you to look at the underlying data David" and "you could have saved a lot of Google time". But when I did do that, all I found was a list of media reports and blogs -pretty much the same as Nature Neuroscience provides itself.
More interesting, I found that my blog wasn’t listed and neither was PubMed Commons. When I asked why, I was told "needs to regularly cite primary research. PubMed, PMC or repository links”. But this paper is behind a paywall. So I provide (possibly illegally) a copy of it, so anyone can verify my comments. The result is that altmetric’s dumb algorithms ignore it. In order to get counted you have to provide links that lead nowhere.
So here’s a link to the abstract (only) in Pubmed for the Science paper http://www.ncbi.nlm.nih.gov/pubmed/25170153 and here’s the link for the Nature Neuroscience paper http://www.ncbi.nlm.nih.gov/pubmed/25344629
It seems that altmetrics doesn’t even do the job that it claims to do very efficiently.
It worked. By later in the day, this blog was listed in both Nature‘s metrics section and by altmetrics. com. But comments on Pubmed Commons were still missing, That’s bad because it’s an excellent place for post-publications peer review.
Here is a record of a couple of recent newspaper pieces. Who says the mainstream media don’t matter any longer? Blogs may be in the lead now when it comes to critical analysis. The best blogs have more expertise and more time to read the sources than journalists. But the mainstream media get the message to a different, and much larger, audience.
The Observer ran a whole page interview with me as part of their “Rational Heroes” series. I rather liked their subtitle [pdf of article]
“Professor of pharmacology David Colquhoun is the take-no-prisoners debunker of pseudoscience on his unmissable blog”
It was pretty accurate apart from the fact that the picture was labelled as “DC in his office”. Actually it was taken (at the insistence of the photographer) in Lucia Sivilotti’s lab.
Photo by Karen Robinson.
The astonishing result of this was that on Sunday the blog got a record 24,305 hits. Normally it gets 1,000-1,400 hits a day . between posts, fewer on Sunday, and the previous record was around 7000/day
A week later it was still twice normal. It remains to be seen whether the eventual plateau stays up.
I also gained around 1000 extra followers on twitter, though some dropped away quite soon, and 100 or so people signed for email updates. The dead tree media aren’t yet dead. I’m happy to say.
3 June 2013
Perhaps as a result of the foregoing piece, I got asked to write a column for The Observer, at barely 48 hours notice. This is the best I could manage in the time. The web version has links.
This attracted the usual "it worked for me" anecdotes in the comments, but I spent an afternoon answering them. It seems important to have a dialogue, not just to lecture the public. In fact when I read a regular scientific paper, I now find myself looking for the comment section. That may say something about the future of scientific publishing.
It is for others to judge how succesfully I engage with the public, but I was quite surprised to discover that UCL’s public engagement unit, @UCL_in_public, has blocked me on twitter. Hey ho. They have 1574 follower and I have 7597. I wish them the best of luck.
Follow-up
Being interested in science communication, I was pleased when the BMJ asked me to review Unscientific America , by Chris Monney and Sheril Kirshenbaum.
The BMJ provides a link that allows you access to the whole review. They have made very few changes from the submitted version, which is reproduced below (with live links in the text. [Download pdf of print version]
I very soon discovered that the book had already caused ructions in the USA, as a result of its advocacy of appeasement of religious groups. In particular there was all out war with P.Z.Myers, whose very popular blog, Pharyngula. documented the battle in detail).
It is an American book through and through, and in the USA the biggest threat to reason comes from the far-right religious fundamentalists who preach young-earth creationism. It is said that 46% of US citizens believe that the earth is less than 10,000 years old. The same far-right religious groups also preach that carrying guns is good, that Iraq was responsible for 9/11, that climate change is a socialist conspiracy and that health care for everyone is a communist plot. And they never hesitate to lie in the promotion of their ‘religious’ views. The US situation is totally different from that in Europe, where religion is all but dead, and young earth creationism is the preserve of a few cranky used-car dealers (and possibly Tony Blair?)
|
Review of the Week Trust me, I’m a scientist David Colquhoun, professor of pharmacology, University College London Unscientific America sounds like a fascinating topic, not least because the book is a follow-up from Mooney’s The Republican War on Science. It is written entirely from a US perspective (the USA sequenced the genome and invented the internet, apparently unaided). It’s reported that 46 percent of Americans believe that the earth is less than 10,000 years old. That’s certainly cause for alarm and Mooney & Kirshenbaum are certainly alarmed. They think that the public needs to be educated in science. They identify the obvious problems, evolution, climate change and quackery and ask what can be done. The problem is that they propose no good solutions, and some bad ones. Their aims are worthy but sometimes the book reads like an over-long and somewhat condescending whine about why science and scientists are not sufficiently appreciated. I simply don’t think that it’s true that the public are not interested in science, nor that they can’t understand it at a level that is sufficient to be useful. It’s true that they have been let down badly by some sections of the media. Think particularly of the “great MMR hoax”1. The disastrous fall in vaccination is more attributable to talk show presenters and air-headed celebrities than to lack of interest from the public. People are systematically deceived by anti-vaxers, climate change denialists, vitamin pill salesmen and a horde of crackpot alternative therapists. There is one problem that Mooney & Kirshenbaum don’t talk about at all, yet it seems to me to be one of the biggest problems in science communication. It isn’t lack of interest by the public, nor even lack of understanding, but lack of trust. The tendency of real science to indulge in hyperbolic self-promotion is one reason for the lack of trust. Sometimes this descends into outright dishonesty2,3. That is a tendency that is promoted by government and funding agencies by their insistence on imposing silly performance measures. The public is quite sensible enough to take with a pinch of salt the almost daily announcements of “cancer cures” that emanate from university press offices. On the face of it, one should be encouraged that ‘public engagement in science’ is the mode du jour. It isn’t quite that simple though. Only too often, universities regard public engagement as a branch of their own PR machine4. They even instruct you about what tone of voice to use when talking publicly. One reason why scientists need to talk to people outside the lab is precisely to counteract this tide of nonsense from PR people, who are paid to deceive. The problem for academics is usually time. We already do three jobs, teaching, research and coping with HR bollocks. How can we find time for a fourth job? That’s not easy, especially for the best researchers (those that do research themselves, not just lead a team). Mooney & Kirshenbaum suggest that the solution is to create a “cadre of communication and outreach experts”. I don’t think this would work. They would, by and large, be outsiders, writing uncritical paeans, dictated by big name scientists. A new cadre of PR hangers-on does not sound like a great idea. A better, and very much cheaper, solution would be to provide a course in free blogging software and we’ll do it ourselves. The two chapters that I looked forward to reading, on religion and on “The bloggers cannot save us”, proved deeply disappointing. The authors are firmly in the camp of what Richard Dawkins called the “Neville Chamberlain school of evolutionists&rdquo.; They maintain “if the goal is to create an America more friendly to science and reason, the combativeness of the New Atheists is strongly counterproductive”. They are particularly critical of P.Z. Myers5, the University of Minnesota developmental biologist who is splendidly clear in his views. Of the communion wafer, he famously said “It’s a frackin’ cracker”. But he, and Dawkins, are right. When it comes to young earth creationists we have a war on our hands, and nowhere more than in the USA. What’s more it’s a winnable war. Mooney & Kirshenbaum are all for appeasement, but appeasement won’t work. It might please the more moderate wings of the church, but they already believe in evolution and are regarded by fundamentalists as being just as big an enemy as Myers and Dawkins. And, one must ask, who has done best at getting a wide public readership? P.Z Myers’ blog, Pharyngula, has up to two million page views a month. Dawkins’ book The God Delusion has sold three million copies. In comparison the bland and often rather condescending corporate science web sites get tiny numbers of hits. In Europe in general, and the UK in particular, young earth creationists are not the major problem that they are in the USA, despite being supported by Tony Blair6. Perhaps the nearest analogy in Europe is the threat to reason from various sorts of crackpot medicine. The appeasers are widespread. The Royal Colleges and the Department of Health are at the forefront of the Neville Chamberlain approach. But appeasement hasn’t worked there either. What has worked is the revelation that university courses are teaching that “amethysts emit high yin energy”7. Or, in a lecture on herbal approaches for patients with cancer, “Legally, you cannot claim to cure cancer. This is not a problem because: ‘we treat people, not diseases’ “8. This is shocking stuff but it has not been unearthed by the corporate media, but by bloggers. I think Mooney and Kirshenbaum have it all wrong. They favour corporate communications, which are written by people outside science and which easily become mere PR machines for individuals and institutions. Such blogs are rarely popular and at their worst they threaten the honesty of science. More and more individual scientists have found that they can write their own blog. It costs next to nothing and you can say what you think. A few clicks and the world can read what you have to say. Forget corporate communications. Just do it yourself. It’s fun. And think of the money you’d save for doing science if the PR people were just fired. (1) Goldacre, B. The media’s MMR hoax. 2008 http://www.badscience.net/2008/08/the-medias-mmr-hoax/ (2) PLoS One. Ghostwriting documents now fully available on PLoS Medicine website. 21-8-2009 http://speakingofmedicine.plos.org/2009/08/21/ghostwriting-documents-now-fully-available-on-plos-medicine-website/ (3) Colquhoun, D. Universities Inc. in the UK. The Corporate Corruption of Higher Education: part 2. 6-12-2007 https://www.dcscience.net/?p=193 (4) Corbyn, Z. Nottingham raises eyebrows over definition of ‘public engagement’. 21-8-2008 http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=403234 (5) Myers, P. Z. Pharyngula. 2009 http://scienceblogs.com/pharyngula/ (6) Pyke, N. Revealed: Blair’s link to schools that take the Creation literally (Independent 13 June 2004). 13-7-2004 http://www.independent.co.uk/news/uk/politics/revealed-blairs-link-to-schools-that-take-the-creation-literally-732032.html (7) Colquhoun, D. Westminster University BSc: "amethysts emit high yin energy". 23-4-2008 https://www.dcscience.net/?p=227 (8) Colquhoun, D. Herbal approaches for patients with cancer. 10-8-2009 https://www.dcscience.net/?p=2043 |
Follow-up
P.Z. Myers has posted about thie book review, on Pharyngula, as Is this book dead yet? There are a lot more comments there than here, though few of them address the question of science communications..
Butterflies and Wheels is generating a lot of hits