Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
good science
[This an update of a 2006 post on my old blog]
The New York Times (17 January 2006) published a beautiful spoof that illustrates only too clearly some of the bad practices that have developed in real science (as well as in quackery). It shows that competition, when taken to excess, leads to dishonesty.
More to the point, it shows that the public is well aware of the dishonesty that has resulted from the publish or perish culture, which has been inflicted on science by numbskull senior administrators (many of them scientists, or at least ex-scientists). Part of the blame must attach to "bibliometricians" who have armed administrators with simple-minded tools the usefulness is entirely unverified. Bibliometricians are truly the quacks of academia. They care little about evidence as long as they can sell the product.
The spoof also illustrates the folly of allowing the hegemony of a handful of glamour journals to hold scientists in thrall. This self-inflicted wound adds to the pressure to produce trendy novelties rather than solid long term work.
It also shows the only-too-frequent failure of peer review to detect problems.
The future lies on publication on the web, with post-publication peer review. It has been shown by sites like PubPeer that anonymous post-publication review can work very well indeed. This would be far cheaper, and a good deal better than the present extortion practised on universities by publishers. All it needs is for a few more eminent people like mathematician Tim Gowers to speak out (see Elsevier – my part in its downfall).
Recent Nobel-prizewinner Randy Schekman has helped with his recent declaration that "his lab will no longer send papers to Nature, Cell and Science as they distort scientific process"
The spoof is based on the fraudulent papers by Korean cloner, Woo Suk Hwang, which were published in Science, in 2005. As well as the original fraud, this sad episode exposed the practice of ‘guest authorship’, putting your name on a paper when you have done little or no work, and cannot vouch for the results. The last (‘senior’) author on the 2005 paper, was Gerald Schatten, Director of the Pittsburgh Development Center. It turns out that Schatten had not seen any of the original data and had contributed very little to the paper, beyond lobbying Scienceto accept it. A University of Pittsburgh panel declared Schatten guilty of “research misbehavior”, though he was, amazingly, exonerated of “research misconduct”. He still has his job. Click here for an interesting commentary.
The New York Times carried a mock editorial to introduce the spoof..
One Last Question: Who Did the Work? By NICHOLAS WADE In the wake of the two fraudulent articles on embryonic stem cells published in Science by the South Korean researcher Hwang Woo Suk, Donald Kennedy, the journal’s editor, said last week that he would consider adding new requirements that authors “detail their specific contributions to the research submitted,” and sign statements that they agree with the conclusions of their article. A statement of authors’ contributions has long been championed by Drummond Rennie, deputy editor of The Journal of the American Medical Association, Explicit statements about the conclusions could bring to light many reservations that individual authors would not otherwise think worth mentioning. The article shown [below] from a future issue of the Journal of imaginary Genomics, annotated in the manner required by Science‘s proposed reforms, has been released ahead of its embargo date. |
The old-fashioned typography makes it obvious that the spoof is intended to mock a paper in Science.
The problem with this spoof is its only too accurate description of what can happen at the worst end of science.
Something must be done if we are to justify the money we get and and we are to retain the confidence of the public
My suggestions are as follows
- Nature Science and Cell should become news magazines only. Their glamour value distorts science and encourages dishonesty
- All print journals are outdated. We need cheap publishing on the web, with open access and post-publication peer review. The old publishers would go the same way as the handloom weavers. Their time has past.
- Publish or perish has proved counterproductive. You’d get better science if you didn’t have any performance management at all. All that’s needed is peer review of grant applications.
- It’s better to have many small grants than fewer big ones. The ‘celebrity scientist’, running a huge group funded by many grants has not worked well. It’s led to poor mentoring and exploitation of junior scientists.
- There is a good case for limiting the number of original papers that an individual can publish per year, and/or total grant funding. Fewer but more complete papers would benefit everyone.
- Everyone should read, learn and inwardly digest Peter Lawrence’s The Mismeasurement of Science.
Follow-up
3 January 2014.
Yet another good example of hype was in the news. “Effect of Vitamin E and Memantine on Functional Decline in Alzheimer Disease“. It was published in the Journal of the American Medical Association. The study hit the newspapers on January 1st with headlines like Vitamin E may slow Alzheimer’s Disease (see the excellent analyis by Gary Schwitzer). The supplement industry was ecstatic. But the paper was behind a paywall. It’s unlikely that many of the tweeters (or journalists) had actually read it.
The trial was a well-designed randomised controlled trial that compared four treatments: placebo, vitamin E, memantine and Vitamin E + memantine.
Reading the paper gives a rather different impression from the press release. Look at the pre-specified primary outcome of the trial.
The primary outcome measure was
" . . the Alzheimer’s Disease Cooperative Study/Activities of Daily Living (ADCSADL) Inventory.12 The ADCS-ADL Inventory is designed to assess functional abilities to perform activities of daily living in Alzheimer patients with a broad range of dementia severity. The total score ranges from 0 to 78 with lower scores indicating worse function."
It looks as though any difference that might exist between the four treaments is trivial in size. In fact the mean difference between Vitamin E and placebos was only 3.15 (on a 78 point scale) with 95% confidence limits from 0.9 to 5.4. This gave a modest P = 0.03 (when properly corrected for multiple comparisons), a result that will impress only those people who regard P = 0.05 as a sort of magic number. Since the mean effect is so trivial in size that it doesn’t really matter if the effect is real anyway.
It is not mentioned in the coverage that none of the four secondary outcomes achieved even a modest P = 0.05 There was no detectable effect of Vitamin E on
- Mean annual rate of cognitive decline (Alzheimer Disease Assessment Scale–Cognitive Subscale)
- Mean annual rate of cognitive decline (Mini-Mental State Examination)
- Mean annual rate of increased symptoms
- Mean annual rate of increased caregiver time,
The only graph that appeared to show much effect was The Dependence Scale. This scale
“assesses 6 levels of functional dependence. Time to event is the time to loss of 1 dependence level (increase in dependence). We used an interval-censored model assuming a Weibull distribution because the time of the event was known only at the end of a discrete interval of time (every 6 months).”
It’s presented as a survival (Kaplan-Meier) plot. And it is this somewhat obscure secondary outcome that was used by the Journal of the American Medical Assocciation for its publicity.
Note also that memantine + Vitamin E was indistinguishable from placebo. There are two ways to explain this: either Vitamin E has no effect, or memantine is an antagonist of Vitamin E. There are no data on the latter, but it’s certainly implausible.
The trial used a high dose of Vitamin E (2000 IU/day). No toxic effects of Vitamin E were reported, though a 2005 meta-analysis concluded that doses greater than 400 IU/d "may increase all-cause mortality and should be avoided".
In my opinion, the outcome of this trial should have been something like “Vitamin E has, at most, trivial effects on the progress of Alzheimer’s disease”.
Both the journal and the authors are guilty of disgraceful hype. This continual raising of false hopes does nothing to help patients. But it does damage the reputation of the journal and of the authors.
|
This paper constitutes yet another failure of altmetrics. (see more examples on this blog). Not surprisingly, given the title, It was retweeted widely, but utterly uncritically. Bad science was promoted. And JAMA must take much of the blame for publishing it and promoting it. |
|
|
I had never intended to write about climate. It is too far from the things I know about. But recent events have unleashed a Palin-esque torrent of comments from people who clearly know even less about it than I do. In any case, it provides a good context to think about trust in science, |
 Earthrise from moon. (click to enlarge) |
My interest in it, apart from little matters like the future of the planet, lies in the reputation of science and scientists.
I have been going on for years now about the lack of trust in science, and the extent to which it is a self-inflicted problem. The latest reactions to the developments at the University of East Anglia and the IPCC may show the nature of the problem with dreadful clarity,
Many of us came into science because, apart from the sheer beauty of nature, it seemed like one of the few honest ways of earning a living. Most scientists that I know still think like that, but recent
events invite some reexamination of honesty in science.
How dishonest is science?
The first thing to say is that I have never come across anything in my own field that would qualify as fraud, or even dishonest. I did once have a visit from a rather distressed postdoc (not in my area of work) who felt pressurised by her boss into putting an interpretation on her work that she did not agree with. In the end, the bit of work in question was left out of the paper. That could be held to be dishonest, in that the omission wasn’t mentioned, but it could also be held that the omitted result was too ambiguous to contribute much to the paper. It was just short of the point where I’d have felt compelled to do something about it. But only just. That is about the worst thing I’ve encountered in a lifetime.
There is, of course, an enormous difference between being wrong and being dishonest. Any research that is worth doing has an outcome that can’t be predicted before the work is done. At best, one can hope for an approximation to the truth. Mistakes in observations, analysis or interpretation will sometimmes mean the announced result is completely wrong, with no trace of dishonesty being involved. But when that happens, others soon fiind the mistake. It is that self-correcting characteristic of science that keeps it honest in the long run.
Of course there have been occasional cases of outright fraud, simple
falsification or fabrication of data. How often it occurs is not really known. There is a recent analysis in PLoS One, about verified cases of misconduct in the USA suggested that 1 in 100,000 scientists per year are to blame, but other ways of counting give larger numbers. For example, if asked around 2 in 100 scientists claim to be aware of misconduct by someone else., The numbers aren’t huge but they are much bigger than they should be.
It isn’t perhaps surprising that the Fanelli study found misconduct was most frequent in “medical (including clinical and pharmacological) research studies”, which are often funded by the pharmaceutical industry, Basic biomedical research and other subjects were better, though sadly that could be only because they are less often offered money.
What gives rise to dishonesty?
It seems obvious that one motive is money, as suggested by the worst rates of misconduct being found in the clinical pharmacological studies, It is well known that studies funded by industry are more likely to produce results that favour the product than those funded in other ways.
The other reason is presumably the human desire to win fame, promotion and to get grants.
It is no excuse, but it is perhaps a reason for misconduct that the pressure to publish and produce results is now enormous in academia. Even in good universities people are judged by the numbers (rather than the quality) of papers they produce and by what journal they happen to be published in. Contrary to public perception, even quite senior people have no guarantee that they can’t be fired, and life for postdoctoral fellows, who do a large fraction of experimental research, is harsh to the point of cruelty. They exist on a series of short term contracts, they work exceedigly hard and have poor prospects of getting a secure job. In conditions like that, the only surprising thing is that there is so little dishonesty.
The pressure to publish in particular journals is particularly invidious because it is known that the number of citations that a paper gets (itself a fallible measure of quality) is independent of the journal in which it appears. Bibliometrists are the curse of our age. (See, for example Challenging the tyranny of impact factors, 2003; and How to get good science, 2007 or its web version; and Peter Lawrence’s article, The mismeasurement of science)
The enormous competitive pressure under which academics work is imposed by vice-chancellors, research councils and other senior people who should know better, It is a self-inflicted wound.
In other words, the authorities provide a strong incentive to do poor, over-hurried and occasionally dishonest science. Perhaps the surprising thing in the circumstances is that there is so little outright fabrication. The very measures that have the aim of improving science actually have just the opposite effect. That is what happens when science is run by people who don’t do it.
For an idea of what life is like in science now, try Peter Lawrence’s Real Lives and White Lies in the Funding of Scientific Research. Or, for someone at the other end of their career, Jennifer Rohn’s account on Nature blogs.
Given the high degree of insecurity for young researchers, compounded by well-intentioned but vacuous “training” from daft Robert’s’ "training courses", or the dismaly ineffective Concordat, the only surprise is that so many people remain honest and devoted to good science. Nothing raises the ire of hard-pressed scientists more than the constant emails form HR trying to force people to go to gobbledygook courses on "wellbeing". Times Higher Education recently did a piece on "Get happy", The comments are worth reading.
So what about climate change?
Out of thousands of pages in the IPCC reports, a single mistake was found, On page 493 of the IPCC’s second 1000-page Working Group report on “Impacts, Adaptation and Vulnerability” (WGII) it was said that Himalayan glaciers were “very likely” to disappear by 2035. Glaciers are melting but that date can’t be justified. This single mistake has been blown out of all proportion. Furthermore it is important to notice that the mistake was found by scientists, not by ‘sceptics’. It is a good example of the self-correcting nature of science. Nevertheless this single mistake has provoked something close to hysteria among those who want to deny that something needs to be done.
On the other hand, the hacked emails from the University of East Anglia (UEA) look bad. It simply isn’t possible at the moment to say whether they are as bad as they seem at first sight, We just don’t know whether anything of importance was concealed, but we should know.
One thing can be said with certainty, and that is that the reaction to their revelation by Dr Phil Jones, and by the vice-chancellor of the University of East Anglia, was nothing short of disastrous. Fred Pearce put it very well in Climate emails cannot destroy proof that humans are warming the planet
Most unforgivably of all, UEA refused to comply with requests under the Freedom of Information Act, and there is some reason to think that relevant material was deleted. The deputy information commissioner, Graham Smith, said: in a statement that
“The emails which are now public reveal that Mr Holland’s requests under the Freedom of Information Act were not dealt with as they should have been under the legislation. Section 77 of the Freedom of Information Act makes it an offence for public authorities to act so as to prevent intentionally the disclosure of requested information.”.
That seems to me to be a matter that requires the resignation of the vice-chancellor. On this matter, I think George Monbiot is spot on in his article “Climate change email scandal shames the university and requires resignations“.
There was a big feature about academic freedom in Times Higher Education recently. One of the problems was what happens to someone who brings their own university into disrepute. But when that term is used, it is always used about junior partners in the organisation (you know, professors and the like). It should apply equally to heads of communications and vice-chancellors who bring their own university into disrepute, whether the disrepute is brought about by failing to comply with the Freedom of Information Act, or by promoting courses in junk medicine.
In general, conspiracy theories are wrong. I’m not sure how much of the distortion of climate data results from surreptitious funding of opposition to doing anything by the fossil fuel industry. The Royal Society is an organisation that is not usually prone to conspiritorialist views. That means one must take it seriously the fact that in 2006, the Royal Society wrote to ExxonnMobil to ask them to stop funding climate denialist organisations. This is a bit like the way Big Pharma has been caught funding “user groups” that endorse their products. Some newspapers like to stir up controversies about things that aren’t very controversial. For example there is a good analysis of a recent Sunday Times piece here.
Of course it is often alleged that "quackbusters" are funded by Big Pharma, though in fact the amounts of money involved are far too small for Big Pharma to bother. Climate deniers too like to suggest that there is some sort of conspiracy, arranged between hundreds of labs in the world to conceal the fact that there is no such thing as warming. I guess that shows only that deniers know little about how science works. it is an exceedingly competitive business, and getting hundreds of labs to say the same thing would be like trying to herd cats.
If there is a problem, it is the other way round. Labs are in such intense competition with each other, that it lcan lead to undesirable levels of secrecy.
Blogs in which researches have a direct dialogue with the public are a big help. As always in the blogosphere, the problem is to find the reliable sources. Two excellent sites, in which scientists (not journalists or lobbyists) talk directly with the public are realclimate.org and Andrew Russell’s blog. The post on RealClimate, IPCC errors: facts and spin, is especially worth reading.
Total openness is the only cure
All the raw data and all emails have to be disclosed openly. Everything should be put on the web as soon as possible. By appearing to go to ground, UEA has made enormous problems for itself and for the rest of the world. Some people object to total openness on the grounds that the other side tells lies. In the case of climate change (and in the case of junk medicine too) that is undoubtedly true. The opponents are ruthlessly dishonest about facts. The only way to counter that is by being ruthlessly and visibly honest about what you know, and why.
The UK’s Meteorological Office has, to its great credit, put raw data on line. That policy has already paid off, because a science blogger found a mistake in the way that some Australian data had been incorporated into forecasts. The Met Office thanked him and corrected the mistake. In fact the error makes no substantial difference to the warming trend, but the principle is just great. The more people who can check analyses and eliminate slip-ups the better.
Putting raw data on the web is an idea that has been gathering force for a while, in all areas of work, not just climate change. In my own are (stochastic properties of single ion channel molecules) our analysis programs have always been available on the web, free to anyone who wants them, despite the large amount of work that has gone into them. And we run a course. almost free, on the theory that underlies our analyses. Within the last couple of months we have been discussion ways of making public all our raw data (in any case, we would always have sent it to anyone who asked). Digitised single channel records are big files (around 100 Mb) and it is only recently that the web has been able to deal with such large amounts of raw data. There are also problems of how to format data so other people can read it, The way we are all heading is clear, and the fact that some people in climate science appeared to be hiding raw data is a disgrace.
Public relations is not the cure
,
It is not uncommon to read that science needs better PR. That is precisely what is not needed. PR exists to put only one side of the story. That makes it an essentially dishonest occupation. Its aims are the very opposite of those of science. The public aren’t stupid: often they recognise when they are getting half the story.
It is particularly unfortunate that many universities have developed departments with names like "corporate communications". Externally they are seen as giving information about science, and indeed some of the things they do are successful public engagement in science. Only too often, though, it is made clear internally that an important aim of these departments is to improve the image of the university.
But you have to choose. You can engage the public in science or you can be a PR image-builder. You can’t be both.
The matter came to a head in 2008 when, according to a report in Times Higher Education, the University of Nottingham issued a memo that defined public engagement as: “The range of activities of which the primary functions are to raise awareness of the university’s capabilities, expertise and profile to those not already engaged with the institution”.
The mainstream media and political blogs
The biggest problem of all with climate change is that it has become more about politics than about facts. It has become an essential credential for any conservative to deny that climate is changing. It is part of their public image, and most conservatives neither know nor care about evidence. Like Sarah Palin, they just know. In the USA especially, the argument is not really about climate at all. It is really about discrediting Barack Obama -a sort of swift-boat treatment that uses whatever lies are needed.
Just as with the great MMR fiasco and the promotion of its false link to autism, reports in newspapers and blogs must bear much of the blame for failing to inform readers of the actual underlying facts and, just as important, the uncertainties. Of course some papers have done a pretty good job, particularly the Guardian and the Independent in the UK, and the New York Times.. The political blogs, by and large, haven’t. The Huffington Post has made little effort (and publishes some appalling nonsense about medicine too).
The problem with political blogs and tabloid newspapers is that they are much more interested in sensation and circulation than they are in giving accurate news and information. Take, for example, the Guido Fawkes blog. To be fair, the blog itself says "The primary motivation for the creation of the blog was purely to make mischief at the expense of politicians and for the author’s own self-gratification. Its writer", so you know not to expect much, Paul Staines, was at the Westminster Skeptics event, Does Political Blogging Make a Difference? He makes no pretence of taking the news seriously, which, I guess, is why I don’t read his blog. After the talks I asked why his blog did little about climate change. His answer was "where are your sandals?". On the way home I tweeted, from a very overcrowded train (most trains from Euston being cancelled that night),
"On way home from #sitp political blogging. Learned that Guido serious about nothing but Guido. Narcissist not journalist."
At least one other person there agreed (thanks, Dave Cole).
It was good to hear Sunny Hundall of Liberal Conspiracy (the only one I read), but I found myself agreeing mostly with the chair, Nick Cohen. It would be a tragedy if the great national and local papers were to vanish. Guido Fawkes and Huffington Post are not remotely like proper newspapers.
Specialist blogs like this one are fine if you are interested in the topics we write about, but we don’t begin to supplant proper newspapers. Bloggers can and do occasionally get good stories. Those that are written by scientists can analyse more critically than most journalists have either the knowledge or the time to do. Bur they don’t come close to supplanting the detailed reporting in good newspapers of local events, what happens in law courts or in parliament. That’s why it is vital to buy newspapers, not just read them free on the web.
Follow-up
James Hayton, who is in nanoscience has posted his thoughts obout trust in science on his blog. I discovered this via Twitter (@James_Hayton). He also posted a beautiful clip from the Ascent of Man, in which Jacob Bronowski speaks, from Auschwitz, of the consequences of irrational dogma. I’m old enough to remember Bronowski on a 1950s radio programme, the Brains Trust, though James Hayton clearly isn’t. Now I enjoy equally his daughter, Lisa Jardine‘s talks about science and history.
1 March 2010. Phil Jones, and the vice-chancellor of the University of East Anglia, appeared before a parliamentary committee. I found their responses to questions very disappointing. The evidence submitted by the Institute of Physics was strongly worded, but spot on.
“The CRU e-mails as published on the internet provide prima facie evidence of determined and co-ordinated refusals to comply with honourable scientific traditions and freedom of information law. The principle that scientists should be willing to expose their ideas and results to independent testing and replication by others, which requires the open exchange of data, procedures and materials, is vital.”
7 March 2010. Thanks to some kind remarks from Michael Kenward (see first comment). I sought wider coverage of this item in the mainstream media. Consequently, on Thursday 4 March, a much shortened version of this article appeared on the Guardian environment site. That piece has accumulated so far, 230 comments. The discussion of it has spread to the two blogs that I recommended, Andy Russell’s blog and RealClimate.org, though it has been diverted onto the side-issue of the letter from the Institute of Physics. The seemingly innocent idea that total openness would increase trust has, to my real astonishment, resulted in hysterical accusations that I’m a crypto-denialist. The constant politically-motivated attacks on climate science seem to have induced a paranoid siege mentality in some of them. There is a real danger that such people will harm their own cause, and that would be tragic.