Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
David Colquhoun
It makes a nice change to be able to compliment an official government report.
|
Ever since the House of Lords report in 2000, the government has been vacillating about what should be done about herbalists. At the moment both western herbalists and traditional Chinese medicine (TCM) are essentially unregulated. Many (but not all) herbalists have been pushing for statutory regulation, which they see as government endorsement. It would give them a status like the General Medical Council. |
|
A new report has ruled out this possibility, for very good reasons [download local copy].
Back story (abridged!)
My involvement began with the publication in 2008 of a report on the Regulation of Practitioners of Acupuncture, Herbal Medicine, Traditional Chinese Medicine . That led to my post, A very bad report: gamma minus for the vice-chancellor. The report was chaired by the late Professor Michael Pittilo BSc PhD CBiol FIBiol FIBMS FRSH FLS FRSA, Principal and Vice-Chancellor of The Robert Gordon University, Aberdeen. The membership of the group consisted entirely of quacks and the vice -chancellor’s university ran a course in homeopathy (now closed).
The Pittilo report recommended statutory regulation and "The threshold entry route to the register will normally be through a Bachelor degree with Honours". It ignored entirely the little problem that you can’t run a BSc degree in a subject that’s almost entirely devoid of evidence. It said, for example that acupuncturists must understand " yin/yang, 5 elements/phases, eight principles, cyclical rhythms, qi ,blood and body fluids". But of course there is nothing to "understand"! They are all pre-scientific myths. This “training dilemma” was pointed out in one of my earliest posts, You’d think it was obvious, but nonetheless the then Labour government seemed to take this absurd report seriously.
In 2009 a consultation was held on the Pittilo report. I and many of my friends spent a lot of time pointing out the obvious. Eventually the problem was again kicked into the long grass.
The THR scheme
Meanwhile European regulations caused the creation of the Traditional Herbal Registration (THR) scheme. It’s run by the Medicines and Healthcare products Regulatory Authority (MHRA). This makes it legal to put totally misleading claims on labels of herbal concoctions, as long as they are registered with THR, They also get an impressive-looking certification mark. All that’s needed to get THR registration is that the ‘medicines’ are not obviously toxic and they have been in use for 30 years. There is no need to supply any information whatsoever about whether they work or not. This appears to contradict directly the MHRA’s brief:
"”We enhance and safeguard the health of the public by ensuring that medicines and medical devices work and are acceptably safe."
After much effort, I elicited an admission from the MHRA that there was no reason to think that any herbal concoctions were effective, and that there was nothing to prevent them from adding a statement to say so on the label. They just chose not to do so. That’s totally irresponsible in my opinion. See Why does the MHRA refuse to label herbal products honestly? Kent Woods and Richard Woodfield tell me. Over 300 herbal products have been registered under the THR scheme (a small percentage of the number of products being used). So far only one product of Tibetan medicine and one traditional Chinese medicine have been registered under THR. These are the only ones that can be sold legally now, because no herbs whatsoever have achieved full marketing authorisation -that requires good evidence of efficacy and that doesn’t exist for any herb.
The current report
Eventually, in early 2014, the Tory-led government set up yet another body, "Herbal Medicines and Practitioners Working Group " (HMPWG). My heart sank when I saw its membership (Annex A.2). The vice-chair was none other that the notorious David Tredinnick MP (Con, Bosworth). It was stuffed with people who had vested interests. I wrote to the chair and to the few members with scientific credentials to put my views to them.
But my fears were unfounded, because the report of the HMPWG was not written by the group, but by its chair only. David Walker is deputy chief medical officer and he had clearly listened. Here are some quotations.
The good thing about the European laws is that
"This legislation effectively banned the importation and sale of large-scale manufactured herbal medicine products. This step severely limited the scope of some herbal practitioners to continue practising, particularly those from the Traditional Chinese Medicine (TCM) and Ayurvedic traditions."
The biggest loophole is that
"At present under UK law it is permitted for a herbal practitioner to see individual patients, offer diagnoses and prepare herbal treatments on their own premises, as long as these preparations do not contain banned or restricted substances. This is unchanged by the Traditional Herbal Medicinal Products Directive. "
Walker recognised frankly that there is essentially no good evidence that any herb, western or Chinese, works well enough to make an acceptable treatment. And importantly he, unlike Pittilo, realised that this precludes statutory regulation.
"There are a small number of studies indicating benefit from herbal medicine in a limited range of conditions but the majority of herbal medicine practice is not supported by good quality evidence. A great deal of international, primary research is of poor quality. "
"ts. Herbal medicine practice is therefore currently based upon traditional practice rather than science. It is difficult to differentiate good practice from poor practice on the basis of this evidence in a way that could establish standards for statutory regulation"
The second problem was the harms done by herbs. Herbalists, western and Chinese, have no satisfactory way of reporting side effects
" . . . there is very limited understanding of the risks to patient safety from herbal medicines and herbal practice. A review of safety data was commissioned from HMAC as part of this review. This review identified many anecdotal reports and case studies but little systematically collected data. Most herbal medicine products have not been through the rigorous licensing process that is required of conventional pharmaceutical products to establish their safety and efficacy. Indeed, only a small proportion have even been subject to the less rigorous Traditional Herbal Registration (THR) process. "
"The anecdotal evidence of risk to patients from herbal products in the safety review highlighted the prominence of manufactured herbal medicines in the high profile serious incidents which have been reported in recent years. Many of these reports relate to harm thought to be caused by industrially manufactured herbal products which contained either dangerous herbs, the wrong constituents, toxic contaminants or adulterants. All such industrially manufactured products are now only available under European regulations if their safety is assured through MHRA licensing or THR
accreditation; and specific dangerous herbs have been banned under UK law. This has weakened the case for introduction of statutory regulation as a further safety measure. "
Then Walker identified correctly the training dilemma. Although it seems obvious, this is a big advance for a government document. Degrees that teach nonsense are not good training: they are miseducation.
"The third issue is the identification of educational standards for training practitioners and the benchmarking of standards for accrediting practitioners. With no good data on efficacy or safety, it is difficult for practitioners and patients to understand or quantify the potential benefits and risks of a proposed therapeutic intervention. Training programmes could accredit knowledge and skills in some areas including pharmacology and physiology, professional ethics and infection control but without a credible evidence base relating to the safety and effectiveness of herbal medicine it is hard to see how they could form the basis of accreditation in this field of practice.
There are a number of educational university programmes offering courses in herbal medicine although the number has declined in recent years. Some of these courses are accredited by practitioner organisations which is a potential governance risk as the accreditation may be based on benchmarks established by tradition and custom rather than science.
"
"The herbal medicine sector is in a dilemma" is Walker’s conclusion.
"Some practitioners would like to continue to practise as
they do now, with no further regulation, and accept that their practice is based on tradition and personal experience rather than empirical science. The logical consequence of adopting this form of practice is that we should take a precautionary approach in order to ensure public safety. The public should be protected through consumer legislation to prevent false claims, restricting the use of herbal products which are known to be hazardous to health"
The problem with this is, if course, is that although there is plenty of law, it’s rarely enforced : see Most alternative medicine is illegal Trading Standards very rarely enforce the Consumer Protection Regulations (2008) but Walker is too diplomatic to mention that fact.
"The herbals sector must recognise that its overall approach (including the rationale for use of products and methods of treatment, education and training, and interaction with the NHS) needs to be more science and evidence based if in order to be established as a profession on the same basis as other groups that are statutorily regulated."
So what happens next?
In the short term nothing will happen.
The main mistake has been avoided: there wil be no statutory regulation.
The other options are (a) do nothing, or (b) go for accreditation of a voluntary register (AR) by the Professional Standards Authority for Health and Social Care (PSA). Walker ends up recommending the latter, but only after a lot more work (see pages 28-29 of report). Of particular interest is recommendation 5.
"As a first step it would be helpful for the sector organisations to develop an umbrella voluntary register that could support the development of standards and begin to collaborate on the collection of safety data and the establishment of an academic infrastructure to develop training and research. This voluntary register could in due course seek accreditation from the Professional Standards Authority for Health and Social Care (PSA)."
So it looks as though nothing will happen for a long time, and herbalists and TCM may end up with the utterly ineffectual PSA. After all, the PSA have accredited voluntary registers of homeopaths, so clearly nothing is too delusional for them. It’s very obvious that, unlike Walker, the PSA are quite happy to ignore the training dilemma.
Omissions from the report
Good though this report is, by Department of Health standards, it omits some important points.
Endangered species and animal cruelty aren’t mentioned in the report. Traditional Chinese medicine, and its variants, are responsible for the near-extinction of rhinoceros, tiger and other species because of the superstitious belief that they have medicinal value. It’s not uncommon to find animal parts in Chinese medicines sold in the UK despite it being illegal
And the unspeakably cruel practice of farming bears to collect bile is a direct consequence of TCM.
A bile bear in a “crush cage” on Huizhou Farm, China (Wikipedia)
Statutory regulation of Chiropractors
The same arguments used in Walker’s report to deny statutory regulation of herbalism, would undoubtedly lead to denial of statutory regulation of chiropractors. The General Chiropractic Council was established in 1994, and has a status that’s the same as the General Medical Council. That was a bad mistake. The GCC has not protected the public, in fact it has acted as an advertising agency for chiropractic quackery.
Perhaps Prof. Walker should be asked to review the matter.
Follow-up
You can also read minutes of the HMPWG meetings (and here). But, as usual, all the interesting controversies have been sanitised.
Edzard Ernst has also commented on this topic: Once again: the regulation of nonsense will generate nonsense – the case of UK herbalists.
DOI: 10.15200/winn.142809.94999
The Research Excellence Framework (REF) is the latest in a series of 6-yearly attempts to assess the quality of research in UK universities. It’s used to decide how to allocate about £1.6 billion per year of taxpayers’ money, the so-called "quality-related" (QR) allocation.
It could have been done a lot worse. One of the best ideas was that only four papers could be submitted, whatever the size of a research group. After much argument, the judgment panels were told not to use journal impact factors as a proxy for quality (or, for lack of quality), though it’s clear that many people did not believe that this would be obeyed, But it cost at least £60 million. At UCL alone, it took 50 – 75 person-years of work. and the papers that were submitted were assessed by people who often would have no deep knowledge about the field, It was a shocking waste of time and money, and its judgements in the end were much the same as last time.
Did the REF benefit science?
It’s frequently said that the REF improved the UK’s science output. The people who claim this need a course in the critical assessment of evidence. Firstly, there is no reason to think that science has improved in quality in the last 6 years, and secondly any changes that might have occurred are hopelessly confounded with the passage of time, the richest source of false correlations.
I’d argue that the REF has harmed science by encouraging the perverse incentives that have done so much to corrupt academia. The REF, and all the other university rankings produced by journalists, are taken far too seriously by vice-chancellors and that does active harm. As one academic put it
"This isn’t about science – it’s about bragging rights, or institutional willy-waving. "
There are now serious worries about lack of reproducibility of published work, waste of money spent on unreliable studies, publication of too many small under-powered studies, bad statistical practice (like ignoring the false discovery rate), and about exaggerated claims by journals, university PR people and authors themselves. These result in no small part from the culture of metrics and the mismeasurement of science. The REF has added to the pressures.
It is highly unsatisfactory, so the only real question becomes what should be done instead?
What’s to be done?
Transferring all the QR money to Research Councils won’t work. It would merely encourage the grossly bad behaviour that we’ve seen at Imperial College London, Warwick University, Kings College London and Queen Mary College London, all of whom have fired successful senior staff simple because their grant income wasn’t deemed big enough. (This is odd because the same managers whine continually that they make a loss on research grants, but that’s another question.) It’s been suggested that this could be avoided by reducing considerably the overheads that come with grants, but this would leave a shortfall that, without QR, would be impossible to make up.
At present a HEFCE working group is considering the possibility that metrics might be used in the next REF. It’s a sensible group of people, and they are well aware of the corrupting influence of metrics, and the lack of evidence that they measure the quality of research. So if reading papers takes too much time and money, and metrics are likely to lead to widespread "gaming" (a euphemism for cheating), what should be done?
I made a suggestion in 2010, but it seems to have been totally ignored, despite appearing in the Times (in their premier
Thunderer opinion column. So I’ll try to make the case again, in the context of the REF.
A complete re-thinking of tertiary education is needed,
Proposal for a two stage higher education system
It seems to be a good thing that such a large proportion of the population now get higher education. But the university system has failed to change to cope with the huge increase in the number of students.
The system of highly specialist honours degrees might have been adequate when 5% of the population did degrees, but that system seems quite inappropriate when 50% are doing them.
There are barely enough university teachers who are qualified to teach specialist 3rd year or postgraduate courses. And many teachers must have suffered from (in my field) trying to teach the subtleties of the exponential probability density function to a huge third year class, most of whom have already decided that they want to be bankers or estate agents.
These considerations have driven me to conclude, somewhat reluctantly, that the whole system needs to be altered.
Honours degrees were intended as a prelude to research and 50% of the population are not going to do research (fortunately for the economy). Vice-chancellors have insisted on imposing on large numbers of undergraduates, highly specialist degrees which are not what they want or need.
I believe that all first degrees should be ordinary degrees, and these should be less specialist than now. Some institutions would specialise in teaching such degrees, others would become predominantly postgraduate institutions, which would have the time. money and expertise to do proper advanced teaching, rather than the advanced Powerpoint courses that dominate what passes for Graduate Schools in the UK.
There would, of course, be almighty rows about which universities would be re-allocated to teach ordinary degrees. That’s not a reason to educate students in 2015 using a pre-war system.
The two-stage system would be more egalitarian than the present one
I anticipate that some people might think that this system is a reversion to the pre-1992 divide between polytechnics and universities. It isn’t. The pre-1992 system labelled you as either polytechnic or university: it was a two-tier system. I’m proposing a two stage system. The two sorts of institution work in series, not in parallel.
Such a system would be more egalitarian than now, not less.
Everyone would start out with the same broad undergraduate education, and the decision about whether to specialise, and the area in which to specialise, would not have to be made before leaving (high) school, as now, but would be postponed until two or three years later. That’s a lot better, especially for people from poorer backgrounds.
If this were done, most research would be done in the postgraduate institutions. Of course there are some good researchers in institutions that would become essentially teaching-only, so there would have to be chances for such people to move to postgraduate universities, and for some people to move in the other direction.
This procedure would, no doubt, result in a reduction in the huge number of papers that are published (but read by nobody). That is another advantage of my proposal. It’s commonly believed that there is a large amount of research that is either trivial or wrong. In biomedical research, it’s been estimated that 85% of resources are wasted (Macleod et al., 2014).
It’s well-known that any paper, however bad, can be published in a peer-reviewed journal. Pubmed, amazingly, indexes something like 30 jouranls devoted to quack medicine, in which papers by quacks are peer-reviewed by other quacks, and which are then solemnly counted by bean-counters as though they were real research. The pressure to publish when you have nothing to say is one of the perverse incentives of the metrics culture.
It seems likely that standards of research in second-stage universities would be at least as high as at present. It that’s the case then QR could simply be allocated on the basis of the number of people in a department. Dorothy Bishop has shown that even under the present system, the amount of QR money received is strongly correlated with the size of the department (correlation coefficient = 0.995 for psychology/neuroscience).
From Dorothy Bishop’s blog. r = 0.995
Using metrics produces only a tiny increase in the correlation coefficient for RAE data. It could hardly be any higher than 0.995
In other words, after all the huge amount of time, effort and money that’s been put into assessment of research, every submitted researcher ends up getting much the same amount of money.
That system wouldn’t work at the moment, because, sadly, universities would, no doubt, submit the departmental cat for a share of the cash. But it could work under a system such as I’ve described. The allocation of QR would take microseconds and cost nothing.
How much would the two-stage system cost?
To have any hope of being accepted by politicians, the two-stage system would probably have to cost no more than the existing system. As far as I know. nobody seems to have made any serious attempt to work out the costs. Perhaps they should. It won’t be easy because an important element of the two-stage system is to improve postgraduate education, and postgraduate education was forgotten in the government’s "reforms"
Much would depend on whether the first stage, ordinary degrees could be taught in two years. In an institution that does little research, there would be no justification for the long summer vacation. Something comparable with (high) school holidays would be more appropriate, and if a decent job could be done in two years, that could save enough money to pay for the rest. It would also mimimise the debt that hangs round the neck of graduates.
The cost of running the second stage would depend on how many students opted (and qualified) to carry on to do an honours degree, and on how many of those wanted go on to graduate school and higher degrees. The numbers of people that went on to specialist honours degrees would inevitably be smaller than now, so their education would be cheaper. But, crucially, they could be educated better. And because of the specialist researchers in a postgraduate institution, it would be possible to have real postgraduate education in advanced research methods,
At present, Graduate Schools in the UK (unlike those in the USA) rarely teach topics beyond advanced Powerpoint, and that’s a recipe for later mediocrity.
In order to estimate the actual cost, we’d need to know how many people wanted to go beyond the first degree (and qualified to do so). If this were not to large, the proposed system could well be cheaper than the presnet one, as well as being more egalitarian, and providing better postgraduate education. The Treasury should like that.
The California System
It will not have escaped the readers’ attention that the two stage system proposed here has much in common with higher education in the USA. In particular, it resembles the University of California system, which was started in 1960. It became a model for the rest of the world.
Meanwhile, the UK persists with a pre-war system of specialist honours degrees that is essentially unchanged since only a handful of people went to universities.
It’s time for the UK to have a serious debate about whether we need to change.
Follow-up
I just noticed this, from the inimitable Laurie Taylor. It is dated 4 July 2013. Who says the REF does not encourage cheating?
AppointmentsAre you a distinguished academic researcher looking to supplement your income? Then look no further. Poppleton is offering 24 extraordinarily well-paid and extraordinarily part-time posts to leading scholars in almost any discipline who will help to raise its profile in the research excellence framework. These posts will follow what is known as the Cardiff-Swansea paradigm in that successful candidates need not have conducted any of their distinguished research at Poppleton, have no need to ever visit the actual campus, and can be assured that their part-time contracts will expire immediately after the date of the REF census. Please apply marking your application “REF FARCE”. |
3 February 2015
The day after this post appeared the Guardian published a version of it which discussed only the two-stage degree proposals but omits the bit about the Research Excellence Framework (REF 2014). The title was "Honours degrees aren’t for all – some unis should only teach two-year courses". There are a lot more comments there than than here.. I assume that the headline was written by one of those pesky subeditors who failed to understand what’s important (the two year degrees were just a suggestion, nothing to do with the main proposals).
3 April 2015
As an experiment, this blog has been re-posted on the Winnower. The advantage of this is that it now has a digital object identifier, DOI: 10.15200/winn.142809.94999
Sarah Ferguson, ex-wife of Prince Andrew, Duke of York, seems to need a lot of money. Some of her wheezes are listed in today’s Times. That’s behind a paywall, as is the version reproduced in The Australian (Murdoch connection presumably). You can read it (free) here, with more details below the article.
|
Thomas Ough and David Brown Published at 12:01AM, January 15 2015 In her seemingly endless quest to make money, Sarah, Duchess of York, has had little hesitation using her title to generate sales. This week, though, she landed herself in trouble after appearing to use the name of Britain’s foremost scientific university to lend credibility to a promotion for her new diet system. The duchess told NBC’s Today show during an interview to promote her “emulsifier” programme that she was aware of the dangers of obesity through her work as an ambassador for the Institute of Global Health Improvement at Imperial College London. Last night she apologised for “any misunderstanding” after Imperial College, ranked the joint second-best university in the world, sought to distance itself from the duchess’s promotion. A spokesman said: “The commercial activities promoted by Sarah Ferguson in the interview with Today are not connected in any way to Imperial’s staff or research activities, and the college does not endorse the suggestion of any possible link.” The institute, which has more than 160 specialists, including clinicians, engineers, scientists and psychologists, is headed by Lord Darzi of Denham, a former Labour health minister. The duchess told the Today presenter Matt Lauer that she had been a comfort eater since the age of 12 but the “turning point” was when she realised that she was the same weight as when pregnant with Princess Beatrice, now 25. “I couldn’t bear looking at myself any minute longer,” she confided. “In fact, the size of my ass probably saved my life.” She said she discovered that the “emulsifier” was “a solution for behavioural change” and helped her to lose 55lbs. The $99 kit, which includes a blender, a couple of recipe books and some workout DVDs, is produced by Tristar Products, a direct marketing company for home and health items. The duchess told the breakfast show: “I have just found out on my discoveries with Imperial College London . . . I’m an ambassador for the Institute for Global Health Innovation, and I found out that children, little children, are going to die before their parents because of obesity.” The benefits of the kit were questioned yesterday by Ayela Spiro, a senior scientist at the British Nutrition Foundation. She said: “In terms of the particular product, no juicer or blender on their own can enhance how much nutrition your body will absorb. Any claims made about such products such that it accelerates weight loss, boosts energy and strengthens the immune system need to be treated with caution.” Professor David Colquhoun of University College London, said: “I find it pretty amazing that Imperial chose someone like her to be an ‘ambassador’. Imperial does have an interest in appetite suppression but hasn’t come up with any usable product yet and this research has nothing to do with blenders. “[Her television appearance] was sheer name-dropping, something she’s quite good at. The only ‘discovery’ she seems to have made is that if you eat less you’ll lose weight. The $100 blender has nothing to do with it.” A spokesman for the duchess said: “She is not trying to use her association with the institute to promote her personal interests. She was talking about ‘behavioural change’, which is endorsed by the institute, and her own behavioural change.” |
With the article there’s an inset that gives details of other ways in which Sarah Ferguson has exploited her title to make money.
Fergie’s latest wheeze, Duchess Discoveries is being promoted heavily on US television. It bears a close resemblance to those ghastly daytime TV advertising channels. Watch her interview on a US TV programme, "Today".
It’s partly promoting her latest diet scam, and partly a vigorous defence of her ex-husbands innocence in the face of allegations of sexual shenanigans. Of course she doesn’t know whether the allegations are true. The Queen doesn’t know (so why bother with the denial from Buckingham Palace?). And I don’t know. We know plenty about Prince Andrew’s bad behaviour, but we don’t know whether he’s had sex with minors.
Worse still is the promotional video on the “Duchess Discoveries” site itself.
I quote:
“I’m SO excited about my fusion accelerator system, accelerates weight loss, accelerates your energy, accelerates and strengthens your immune system.”
"accelerates weight loss" is certainly unproven. Mere hype
"accelerates your energy" is totally meaningless. It’s the sort of sciencey-sounding words that are loved by all quacks.
"accelerates and strengthens your immune system". Sigh. "strengthening the immune system is the perpetual mantra of just about every quack. It’s totally meaningless. Just made-up nutribollocks.
The promotional video is fraudulent nonsense. If it were based in the UK I have no doubt that it would be quickly slapped down by the Advertising Standards Authority. But in the USA the first amendment allows people to lie freely about nutrition, which is why it’s such big business.
It bothers me that the most that the best that the British Nutrition Foundation could manage was to say that such claims "need to be treated with caution". They are mendacious nonsense. Why not just say so?
Follow-up
The tragedy of the apparent suicide of Stefan Grimm is now known worldwide. His last email has been read by more than 160,000 people from over 200 countries. This post gathers together some of the reactions to his death. It’s a Christmas card for the people who are responsible.
 Alice Gast (president) |
 James Stirling (provost) |
 Dermot Kelleher (VP (health) |
“This isn’t about science – it’s about bragging rights, or institutional willy-waving.” from Grimm’s Tale
The back story
On Monday 1st December I published Stefan Grimm’s last email. It has been read by more than 160,000 people from over 200 different countries.
On Tuesday 2nd December, Stefan Grimm’s immediate boss, Martin Wilkins, wrote to me. He claimed “We met from time to time to discuss science and general matters. These meetings were always cordial. ”
On Wednesday 3rd December, the Dean of Medicine, Dermot Kelleher, mailed all Faculty of Medicine staff (not the rest of the College). Read the letter. It said very little. But it did include the words
“I regret I did not know Stefan personally, and I looked to colleagues to describe to me his life and the impact of his work at Imperial “
It seems a bit odd that the Dean of Medicine did not know a senior professor, but that seems to be life at Imperial.
On Thursday 4th December, Times Higher Education printed the same last email, and also the text of a threatening letter sent to Grimm in March.by his boss, Martin Wilkins. The letter was very far from being cordial, contrary to what Wilkins claimed. It included these words.
“I am of the opinion that you are struggling to fulfil the metrics of a Professorial post at Imperial College which include maintaining established funding in a programme of research with an attributable share of research spend of £200k p.a and must now start to give serious consideration as to whether you are performing at the expected level of a Professor at Imperial College.”
For a successful 51 year old with a good publication record to get a letter like that must have been devastating.
On Friday 5th December, Imperial made its first public announcement of his death. more than three months after it happened. By this time a damning account of his death had appeared even in the Daily Mail. The announcement read as though the world was unaware of his last words. It was a PR disaster: weasel words and crocodile tears. It made Imperial College appear to be totally heartless. The official announcement was accompanied by the phone numbers for the Samaritans. the chaplaincy and mental health first-aiders. Giving a person a phone number to call when you’ve destroyed their life is not an adequate substitute for treating staff properly.
Imperial are still trying to pretend that Grimm’s death is nothing to do with them, despite the fact that the whole world now knows quite enough of the facts to see otherwise.
The Coroner’s Inquest
The inquest into Grimm’s death was adjourned on October 8th, pending investigations into its cause. If you know anything relevant you should email the Coroner’s officer who is responsible for the investigation. That’s Molly Stewart (Molly.Stewart@lbhf.gov.uk). It is rather important that all the information doesn’t come from the College authorities, which cannot be relied on to tell the truth.
Some reports about the regime at Imperial College
Since my post went up on December 1st, I’ve had a stream of emails which testify to the reign of terror operated by the senior management at Imperial. The problem is by no means restricted to the Faculty of Medicine, though the problems seem to be worst there.
Many of these correspondents don’t want to speak in public. That’s certainly true of people who still work at Imperial, who have been warned to deflect all enquiries to HR. Here are some of the stories that I can reveal.
The Research Excellence Framework (REF) results were announced on 18th December. All university PR people hunted through the results, and all found something to boast tediously about. The letter from Imperial’s provost, James Stirling (read it), is pretty standard stuff. as is the letter from the Dean of Medicine, Dermot Kelleher (read it). Needless to say, neither letter mentioned the price in human misery, and even death, that Imperial had paid for its high ranking. I felt compelled to tweet
Imperial is second in #REF2014. And cost only one life #Grimmdeath. Well worth it uh? #HRbollocks http://t.co/25QNe8agjs
— David Colquhoun (@david_colquhoun) December 18, 2014
Kelleher promoted. Astonishingly, the very next day, the Dean of Medicine, on whose watch Grimm died, was promoted. You can read the letter from Imperial’s president, Alice Gast, in which this is announced. He is to be Vice President (Health), as a reward no doubt, for the cruel regime he ran as Dean. The letter has all the usual vacuous managerial buzzwords, e.g. “to support and grow the multidisciplinary paradigm in health”. Remember DC’s rule number one: never trust anyone who uses the word ‘paradigm’. Needless to say, still no mention of treating staff better.
Dr William J Astle.
Dr Astle is one of many people who wrote to me about his experiences at Imperial College. Although he still appears on Imperial’s web site, he now works as a statistician in a bioinformatics team at the University of Cambridge (see their web site).
He wrote again on 23 October 2014, to pass on an email (read the mail) that was sent to Department staff after Grimm’s last email had been circulated.(on 21 October). It is from a Faculty Operating Officer and ends with a warning to refer media enquries to a PR person (the Press and Internal Communications Manager, John-Paul Jones).
When he saw the internal email from Provost James Stirling with the usual self-congratulatory stuff about the REF, Astle wrote again to Stirling, His letter ends thus.
“Putting university staff in fear of losing their jobs leads to an atmosphere of obsequiousness and obedience to authority that prevents academics from fulfilling their institutional role. In a free society it is essential academics have the autonomy to determine their line of work, to question institutional and state authority and to do risky research. Once again I emphasise – in my experience the atmosphere in the faculty of medicine at IC is not conducive to this.”
Stirling did not reply to this letter. Neither Gast nor Stirling have replied to mine either. Discourtesy seems to be part of the job description of senior managers.
Christine Yates
Christine Yates says
“I was employed at Imperial College London from s” August 2002 to October 2012. For these 10 years I was the College’s Equality and Diversity Consultant in the Human Resources Department, reporting to the HR deputy director, Kim Everitt. In turn, Kim Everitt reports to the HR director, Louise Lindsay. Throughout this time I was the College’s sole equalities consultant, and over time built up the Equalities Unit and managed a team of five.”
“I was dismissed on 8th October 2012 following a Disciplinary Hearing in response to an allegation of gross misconduct “for continued wilful refusal to follow your Head of Department’s (HOD) instructions not to be involved in individual cases”.
As part of her job, she was responsible for establishing and maintaining the Harassment Support Contact Scheme, which was designed to help staff who felt they were being harassed, bullied, and victimised. She was also responsible for the College’s first Athena SWAN (scientific women’s academic network) .successful application, along with the establishment of disabilities, race equalities, and sexual orientation networks, all of which attained quality professional kite marks over time. The Athena Swan award is particularly ironic, given that Imperial’s present brutal assessment system must be even more unfair to women than it is to men. In 2003 (when Richard Sykes was still in charge), a third of female employees at Imperial reported that they were bullied. The improvement since then seems to have been small.
One of many cases she dealt with involved the harassment and bullying of a senior female academic by her male boss. Yates maintains, with good evidence, that complaints about this behaviour were never investigated properly by HR. This displeased HR. Incidents like this undoubtedly contributed to her dismissal.
“In Dr ***’s [female] case, it is clear to me that no independent investigations have been held and that College procedures are being flouted or rnanipulated with the alleged harasser (Professor **** [male]) being protected and permitted to continue his misconduct.”
“In my position as the College’s Equalities Consultant, I was aware of many cases and outcomes. Or ***’s is one of the most distressing and badly handled cases I was witness to, and the manner in which HR protect senior academics who have gravely offended, and who under any reasonable circumstances would be found to be guilty of gross misconduct, is a sad indictment of Imperial College”
You can read the statement that Christine Yates has already sent to the Coroner’s officer. Unfortunately the attachments have had to be removed here because they deal with specific cases.
“The Coroner’s Office needs to be aware of the pattern of behaviour that ensues whenever bad practice is brought to the College’s attention. In response to whistle blows and other complaints the College tries to discredit the complainant. When this fails they will invariably state that they will hold a ‘review’ usually undertaken by those responsible for the bad behaviour and thus with a vested interest in covering up any misconduct and impropriety. It is noted this pattern remains unchanged, “
A problem with a paper
An anonymous correspondent has sent me a lot of emails that concern a paper that was in revision at the time of Grimm’s death. The title of the paper is “Role of non-coding RNAs in apoptosis revealed in a functional genome-wide screen”.
On October 6th, one author wrote to his co-authors “I worked closely with Stefan on the screen data this year. We re-interpreted the mathematical analysis performed in the original manuscript, providing a more rigorous statistical foundation of the gene rankings. As a result, the gene list Stefan and I have generated is now different.”.
Clearly Grimm was aware of the need for revision before he died. Given that everyone was under such intense pressure to publish, it’s likely that the prospect of a prolonged delay in publication might well have contributed to his depression and his death.
The author who wrote on October 6th outlined some options. One was to leave the paper as it was, but to include all the raw data and submit to a journal such as Plos One or the preprint server BioRxiv. This option “requires minimal work, and would result in no change in the author list. However we would aim for a lower-impact journal.”. His preferred option, though, was to rewrite the paper altogether (and for himself to become first co-author) “as it is in all our shared interest to get the work published in as good a journal as possible. “.
Two days later, on October 8th, the same author thanked his co-authors for their responses. As a result of the responses he got, he asked to have his name removed from the paper because he did not agree with what was contained in the manuscript. “However, given that I believe the gene list is wrong, I request my name to be removed from the author list. If any other authors do not wish for the raw data to be disclosed then I hope you think it’s reasonable for me to close off my involvement with the paper.”.
The paper has 11 authors, including Stefan Grimm. . I have written to all but one of the authors to try to ascertain the facts. Of the four co-authors who have replied, all but one said that they hadn’t seen the final paper. One said that they were unaware that they were on the author list, and said they probably shouldn’t be.
I have tried to protect the authors (some of whom are still at Imperial) by not mentioning their names. But one co-author is sufficiently senior to be mentioned by name. Alan Boobis answered by my mail cordially enough when I first wrote to him, but declined to give much useful information, apart from confirming that Grimm was the senior author on the paper. On October 9th he wrote to all co-authors, thus.
|
From: Boobis, Alan R [a.boobis@imperial.ac.uk] Dear all The situation regarding this manuscript needs to be dealt with rationally. There is a real danger that the reputations of individuals and of the College will be harmed. I suggest that we all need to agree the most appropriate way forward. I am out of the country this week but will have my secretary liaise with you next week to arrange a suitable time (face to face or by phone) to discuss this. Best wishes, Alan |
I have no idea what the outcome of this meeting was. Personally. I always worry a bit when people want meetings “face to face or by phone”. Written records are much more informative.
I should like to make it clear that I’m not suggesting any misconduct whatsoever. The author who wished to withdraw acted with principle and courage, and mistakes happen. They are perhaps especially likely in multi-author papers where some authors don’t understand the input from others. But it is sad to see the emphasis on the long-discredited journal impact factor that was forced on them by Imperial’s policies. And it’s sad to see that several co-authors had not actually seen the final paper. This smacks of “citation-mongering”, yet another bad effect of the metrics culture that has pervaded all of academia, and which is enforced in an especially simple-minded way at Imperial.
This sad episode is yet another illustration of the way that Imperial’s policies are damaging people, and, in the end, damaging science.
Some discussions of the Imperial problem
Since Grimm’s last email was revealed, it’s been discussed in many blogs and articles. Here are a few of them.
Grimm’s tale (2 December). This perceptive blog reproduces part of the nasty threatening letter sent by Martin Wilkins to Grimm.
“Your current level of funding does not constitute the appropriate level for a professor at Imperial College. Unless you submit and are awarded a Platform grant as PI in the next 12 months we will seek to initiate disciplinary action against you.”
The author comments (my emphasis).
“This isn’t about science – it’s about bragging rights, or institutional willy-waving. Grimm was informed – in public – that he was to be fired, and left waiting for the axe to fall while the axe-wielder marauded around the campus boasting about it like an even more pathetic Alan Sugar.”
That sums it up for me. It’s very sad.
Another blog comments
“Martin Wilkins to Professor Stefan Grimm, a few months before the latter committed suicide. Imperial College had been pressuring Grimm to get 200, 000 pounds in grants in order for him to remain employed. They threatened to sack him as he only had 135,000 pounds.
Sounds a lot like loan sharks.”
Clearly universities like Imperial are no longer places for scholarship. They are more like anxiety machines.
The Nuffield Council on Bioethics produced an important report in the midst of the scandal about Grimm: The culture of scientific research 2014. Paragraph 1.7 produced a chilling statistic
1.7 Compromising on research integrity and standards
- Almost six in ten (58%) respondents are aware of scientists feeling under pressure to
compromise on research integrity and standards, with poor methodology and data fraud
frequently mentioned in the free text responses.- Just over a quarter (26%) of those taking part in the survey have felt tempted to compromise
on research integrity.
Stefan Grimm and the British University system. This blog, written by a geneticist. Federico Calboli, based in Helsinki, gives an indication of the harm that Imperial is inflicting not only on itself, but on the whole of UK academia, and hence on the UK economy
“As always in the real world the best laid plans often conflict with how the world actually works, and this conflict gives rise to a number of unintended consequences. The first unintended consequence is that the pursuit of what managements defines as ‘novel’ and ‘glamorous’ will diminish the intellectual value of British academia as a whole.”
“Unfortunately, since academia, funding bodies and the editorial boards of papers have been taken over by top down management culture, solid rigorous science is penalised in favour of anything that can be branded as ’novel’, ‘cutting edge’, ‘state of the art’ and similar platitudes.”
“This policy will leave British academia directionless and intellectually empty, and will transform any research in technology and data driven drivel that can at most pick up low hanging fruits and will deliver less and less as time goes on.”
Still more shaming, Calboli continues thus.
“The second problem with how British academia is managed is the culture of intellectual dishonesty that is forced upon people. People are not allowed to just express their goals in simple honest terms. They are required to spin and embellish everything in order to have half a chance of getting some funding or publishing in a high impact journal – both crucial to contribute to the ‘excellence metric”.
“Only the shameless cynics thrive in such environment”.
The blog finishes with a rallying cry.
“the email that Prof Grimm sent in October did not magically make its way to the press by itself. While many people are feeling disenchanted with academia and leave, more and more insiders are taking a combative stance against the mindless hogwash that threatens the foundations of British academia and the people that push it. We should all stand up and be counted, or we will not be able to complain in the future. It would be great if management could live up to its role and abandon the idea that scientific research is simple, predictable and quickly profitable, and actually help build the future of British academia.”
All this reflects similar sentiments to those that I expressed in 2007 [the RAE was the predecessor of the REF]
“The policies described here will result in a generation of ‘spiv’ scientists, churning out 20 or even more papers a year, with very little originality. They will also, inevitably, lead to an increase in the sort of scientific malpractice that was recently pilloried viciously, but accurately, in the New York Times, and a further fall in the public’s trust in science. That trust is already disastrously low, and one reason for that is, I suggest, pressures like those described here which lead scientists to publish when they have nothing to say.”
““All of us who do research (rather than talk about it) know the disastrous effects that the Research Assessment Exercise has had on research in the United Kingdom: short-termism, intellectual shallowness, guest authorships and even dishonesty”. Now we can add to that list bullying, harassment and an incompetent box-ticking style of assessment that tends to be loved by HR departments.
This process might indeed increase your RAE score in the short term (though there is no evidence that it it does even that). But, over a couple of decades, it will rid universities of potential Nobel prize winners.”
Conclusions
The policies adopted by Imperial College have harmed Imperial’s reputation throughout the world. Worse still, they have tainted the reputation of all UK universities. They have contributed to the corruption of science. and they have, in all probability, killed a successful man,
I hope that Alice Gast (president), James Stirling (provost). Dermot Kelleher (Dean, now vice president), and Martin Wilkins (who was left to weild the knife) have a good Christmas. If I were in their shoes, I’d feel so guilty that I wouldn’t be able to sleep at night.
They should spend Christmas reading Peter Lawrence’s wonderful essay on The Mismeasurement of Science. Please download a copy
Their proposal that HR policies should be investigated by, inter alia, the head of HR has provoked worldwide derision.
Their refusal to set up an independent external inquiry is reprehensible.
Not for the first time, a fine institution is being brought into disrepute by its leadership. Council please note.
|
Alice Gast |
James Stirling |
Dermot Kelleher |
Perhaps the best description of what’s going on is from Grimm’s Tale “This isn’t about science – it’s about bragging rights, or institutional willy-waving.”. Gast, Stirling and Kelleher should stop the willy-waving. They should either set about rectifying the damage they’ve done. Or they should resign. Now.
The chair of universities HR association, Kim Frost, said
“Bullying is a very emotive term, and what one person experiences as bullying will often be simple performance management from their manager’s point of view.”.
That’s scary because it shows that she hasn’t the slightest idea about “performance management”. I have news for HR people. They are called experiments because we don’t know whether they will work. If they don’t work that’s not a reason to fire anyone. No manager can make an experiment come out as they wish. The fact of the matter is that it’s impossible to manage research. If you want real innovation you have to tolerate lots and lots of failure. “Performance management” is an oxymoron. Get used to it.
This sorry episode has far more general lessons for the way the REF is conducted and for the metrics sales industry. Both share some of the guilt.
That will have to wait for another post.
Follow-up
25 December 2014. Universities "eliminate tenure because Starbucks does not have tenure"
I was struck by this excerpt from a Christmas newsletter from a colleague. Buried among the family news was buried this lament. He’s writing about Rush University, Chicago, but much the same could be said about many universities, not only in the USA.
|
Rush Medical Center built an $800 million hospital building that is clinically state-of-the-art and architecturally unique. Now it is poised to become a world class center of basic and clinical research. Sadly, rather than listen to researchers who have devoted their careers to Rush, senior administration hears advice from fly-by-night financial consultants who apply the same “Business Model” to medical care, education, and research as to a shoe factory. Perhaps because fiscal consulting requires little skill or training*, they do not distinguish between a researcher and a Starbucks employee [literally true!]. They eliminate tenure because Starbucks does not have tenure. {To be fair, they have only eliminated “tenure of salary” – one may continue working with a title, but without pay!} They cannot imagine that world-class research is an art that requires years of training, cultivating an international network of colleagues, and most importantly, continuity of funding. Because their work is so trivial, they cannot fathom that researchers could be utterly unique and irreplacable. And they do not care – they will destroy research at Rush, collect their multi-million dollar fee, and move on to the next shoe factory. *Lesson 1. Fire people who do real work, cut wages, steal from pension funds, eliminate unions and job security. Congratulations you are now a qualified fiscal consultant! |
26 December 2014
|
Grimm is not the only one. In the same month, September 2014, Tony Veitch was found dead. He was a senior scientist in the lab at Kew Botanical Gardens. He was 49, much the same age as Stefan Grimm. It’s presumed that he committed suicide after being told to reapply for his own job. |
 |
!7 January 2015
I hear that Imperial College’s UCU passed this motion.
|
Motion 3: Branch condemns bullying and harassment of staff at Imperial This branch strongly condemns the bullying and harassment of staff at Imperial, particularly by some managers. We call upon the senior management of the College to ensure that all managers are properly trained to deal with staff in a fair and considerate manner and on how to refrain from bullying and harassment. In light of a recent tragic case at Imperial, the College management must ensure that they fulfil their duty of care to all staff at all times. |
Of course every employer claims that they do this.
I wonder how the officials can mouth these platitudes when the facts, now well known, show them to be untrue,
The first post and this one have been viewed over 173.000 times, from at least 170 countries (UK, USA,and then almost 10,000 views from China). I realise that this must have harmed Imperial, but they have brought it on themselves. Neither the president nor the rector have had the courtesy to answer perfectly polite letters.
I wrote also on 29 December to the chair of Council. Eliza Manningham-Buller. She has still not acknowledged receipt, never mind replied.
I am amazed by the discourtesy of people who regard themselves as too important to reply to letters.
|
To chair of Council, Imperial College London 29 December 2014 Dear Lady Manningham-Buller A problem with management at Imperial It cannot have escaped your notice that a senior member of Imperial’s staff was found dead, after being told that he’d lose his job if he didn’t raise £200,000 in grants within a year. When I posted Stefan Grimm’s last email on my blog on December 1st it went viral (Publish and perish at Imperial College London), It has been read by over 160,000 people from over 200 countries. That being the case, Imperial’s first official mention of the matter on December 4th looked pretty silly. It was written as though his email was not already common knowledge –totally hamfisted public relations. After posting Grimm’s last mail, I was deluged with mails about people who had been badly treated at Imperial. I posted a few of them on December 23rd (Some experiences of life at Imperial College London. An external inquiry is needed after the death of Stefan Grimm). The policy of telling staff that their research must be expensive is not likely to be appreciated by the taxpayer. Neither will it improve the quality of science. On the contrary, the actions of the College are very likely to deter good scientists from working there (I already heard of two examples of people who turned down jobs at Imperial). I think it is now clear that the senior management team is pursuing policies that are damaging the reputation of Imperial. I hope that Council will take appropriate action. Best regards David Colquhoun |
20 January 2015
Today I got a reply to the letter (above) that I sent to Eliza Manningham-Buller on 29 December. You can download it.
I guess it’s not surprising that the reply says nothing helpful.
It endorses the idea that HR should investigate their own practices, an idea that the outside world greets with ridicule.
It reprimands me for making "unprofessional" comments about individuals. That’s what happens when people behave badly. It would be unprofessional to fail to point out what’s going on. It’s the job of journalists to name people. All else is PR.
It suggests that I may have not followed the Samaritans’ guidelines for reporting of suicide. I’ve read their document and I don’t believe that either I, or Times Higher Education, have breached the guidelines.
The letter says. essentially, please shut up, you are embarrassing Imperial. It’s fascinating to see the rich and powerful close ranks when criticised. But it is very disappointing. It seems to me to be very much in the public interest to have published the last email of Stefan Grimm.
But I guess the last person you’d expect to champion transparency is an ex-head of MI5.
Felix, Imperial’s student newspaper, carried an interesting article Death of Professor Grimm: the world reacts. The events at Imperial have been noted all over the world (at least 170 countries according to my own Google analytics) but the response has been especially big in China. Alienating a country like China seems to me to rank as bringing the College into disrepute.
9 February 2015
Death in Academia and the mis-measurement of science. Good article in Euroscientist by Arren Frood
25 February 2015
I see that Dermot Kelleher is leaving Imperial for the University of British Columbia. Perhaps he hopes that he’ll be able to escape his share of the blame for the death of Stefan Grimm? Let’s hope, for the sake of UBC, that he’s learned a lesson from the episode.
10 March 2015
The Vancouver Sun has been asking questions. An article by Pamela Fayerman includes the following.
"Recently, Imperial College was engulfed in a controversy involving a tragedy. . . . a medical school professor, Stefan Grimm, took his own life last fall. He left an email that accused unnamed superiors of bullying through demands that he garner more research grants. The “publish or peril” adage that scientists so often cite seems like it may apply in this case.
The college said it would set up an internal inquiry into the circumstances around the toxicology professor’s death, but the results have not been released. UBC provost Dave Farrar said in an interview that the death of the professor at Imperial College was never even discussed during the recruitment process.
Kelleher said in a long distance phone interview that the tragedy had nothing to do with his reasons for leaving Imperial. And he can’t speak about the case since it is currently under review by a coroner."
Well, I guess he would say that, wouldn’t he? Kelleher has been at Imperial for less than three years, and the generous intepretation of his departure is that he didn’t like the bullying regime. It had been going on long before Kelleher arrived, as documented on this blog in in 2007.
It’s interesting to speculate about why he wasn’t asked about Grimm’s death (if that’s true).
Did the University of British Columbia think it was irrelevant?
Or did they want him to establish a similar regime of “performance management” at UBC?
Or were the senior people at UBC not even aware of the incident?
Perhaps the third option is the most likely: it’s only too characteristic of senior managers to be unaware of what’s happening on the shop floor. Just as in banks.
11 March 2015
It’s beginning to look like an exodus. The chair of Imperial’s council, Eliza Manningham-Buller, is also leaving. Despite her condescending response to my inquiries, perhaps she too is scared of what will be revealed about bullying. I just hope that she doesn’t bring Imperial’s ideas about "performance management" to the Wellcome Trust.
|
This week’s Times Higher Education carried a report of the death, at age 51, of Professor Stefan Grimm: Imperial College London to ‘review procedures’ after death of academic. He was professor of toxicology in the Faculty of Medicine at Imperial. |
 |
Now Stefan Grimm is dead. Despite having a good publication record, he failed to do sufficiently expensive research, so he was fired (or at least threatened with being fired).
“Speaking to Times Higher Education on condition of anonymity, two academics who knew Professor Grimm, who was 51, said that he had complained of being placed under undue pressure by the university in the months leading up to his death, and that he had been placed on performance review.”
Having had cause to report before on bullying at Imperial’s Department of Medicine, I was curious to know more.
Martin Wilkins wrote to Grimm on 10 March 2014. The full text is on THE.
"I am of the opinion that you are struggling to fulfil the metrics of a Professorial post at Imperial College which include maintaining established funding in a programme of research with an attributable share of research spend of £200k p.a and must now start to give serious consideration as to whether you are performing at the expected level of a Professor at Imperial College."
"Please be aware that this constitutes the start of informal action in relation to your performance, however should you fail to meet the objective outlined, I will need to consider your performance in accordance with the formal College procedure for managing issues of poor performance (Ordinance D8) which can be found at the following link.
http://www3.imperial.ac.uk/secretariat/collegegovernance/provisions/ordinances/d8"
[The link to ordinances in this letter doesn’t work now. But you can still read them here (click on the + sign).]
It didn’t take long to get hold of an email from Grimm that has been widely circulated within Imperial. The mail is dated a month after his death. It isn’t known whether it was pre-set by Grimm himself or whether it was sent by someone else. It’s even possible that it wasn’t written by Grimm himself, though if it is an accurate description of what happened, that’s not crucial.
No doubt any Imperial staff member would be in great danger if they were to publish the mail. So, as a public service, I shall do so.
The email from Stefan Grimm, below, was prefaced by an explanation written by the person who forwarded it (I don’t know who that was).
|
Dear Colleagues,
You may have already heard about the tragic death of Professor Stefan Grimm a former member of the Faculty of Medicine at Imperial College. He died suddenly and unexpectedly in early October. As yet there is no report about the cause of his death. Some two weeks later a delayed email from him was received by many of the senior staff of the medical school, and other researchers worldwide. It has been forwarded to me by one of my research collaborators. From my reading of it I believe that Stefan wanted it circulated as widely as possible and for that reason I am sending it to you. It is appended below. This email represents just one side of an acrimonious dispute, but it may be indicative of more deep seated problems. best wishes Begin forwarded message: From: Stefan Grimm <professorstefangrimm@gmail.com> Date: 21 October 2014 23:41:03 BST To: <big-email-list> Subject: How Professors are treated at Imperial College Dear all, If anyone is interested how Professors are treated at Imperial College: Here is my story. On May 30th ’13 my boss, Prof Martin Wilkins, came into my office together with his PA and ask me what grants I had. After I enumerated them I was told that this was not enough and that I had to leave the College within one year – “max” as he said. He made it clear that he was acting on behalf of Prof Gavin Screaton, the then head of the Department of Medicine, and told me that I would have a meeting with him soon to be sacked. Without any further comment he left my office. It was only then that I realized that he did not even have the courtesy to close the door of my office when he delivered this message. When I turned around the corner I saw a student who seems to have overheard the conversation looking at me in utter horror. Prof Wilkins had nothing better to do than immediately inform my colleagues in the Section that he had just sacked me. Why does a Professor have to be treated like that? All my grant writing stopped afterwards, as I was waiting for the meeting to get sacked by Prof Screaton. This meeting, however, never took place. In March ’14 I then received the ultimatum email below. 200,000 pounds research income every year is required. Very interesting. I was never informed about this before and cannot remember that this is part of my contract with the College. Especially interesting is the fact that the required 200,000.- pounds could potentially also be covered by smaller grants but in my case a programme grant was expected. Our 135,000.- pounds from the University of Dammam? Doesn’t count. I have to say that it was a lovely situation to submit grant applications for your own survival with such a deadline. We all know what a lottery grant applications are. There was talk that the Department had accepted to be in dept for some time and would compensate this through more teaching. So I thought that I would survive. But the email below indicates otherwise. I got this after the student for whom I “have plans” received the official admission to the College as a PhD student. He waited so long to work in our group and I will never be able to tell him that this should now not happen. What these guys don’t know is that they destroy lives. Well, they certainly destroyed mine. The reality is that these career scientists up in the hierarchy of this organization only look at figures to judge their colleagues, be it impact factors or grant income. After all, how can you convince your Department head that you are working on something exciting if he not even attends the regular Departmental seminars? The aim is only to keep up the finances of their Departments for their own career advancement. These formidable leaders are playing an interesting game: They hire scientists from other countries to submit the work that they did abroad under completely different conditions for the Research Assessment that is supposed to gauge the performance of British universities. Afterwards they leave them alone to either perform with grants or being kicked out. Even if your work is submitted to this Research Assessment and brings in money for the university, you are targeted if your grant income is deemed insufficient. Those submitted to the research assessment hence support those colleagues who are unproductive but have grants. Grant income is all that counts here, not scientific output. We had four papers with original data this year so far, in Cell Death and Differentiation, Oncogene, Journal of Cell Science and, as I informed Prof Wilkins this week, one accepted with the EMBO Journal. I was also the editor of a book and wrote two reviews. Doesn’t count. This leads to a interesting spin to the old saying “publish or perish”. Here it is “publish and perish”. Did I regret coming to this place? I enormously enjoyed interacting with my science colleagues here, but like many of them, I fell into the trap of confusing the reputation of science here with the present reality. This is not a university anymore but a business with very few up in the hierarchy, like our formidable duo, profiteering and the rest of us are milked for money, be it professors for their grant income or students who pay 100.- pounds just to extend their write-up status. If anyone believes that I feel what my excellent coworkers and I have accomplished here over the years is inferior to other work, is wrong. With our apoptosis genes and the concept of Anticancer Genes we have developed something that is probably much more exciting than most other projects, including those that are heavily supported by grants. Was I perhaps too lazy? My boss smugly told me that I was actually the one professor on the whole campus who had submitted the highest number of grant applications. Well, they were probably simply not good enough. I am by far not the only one who is targeted by those formidable guys. These colleagues only keep quiet out of shame about their situation. Which is wrong. As we all know hitting the sweet spot in bioscience is simply a matter of luck, both for grant applications and publications. Why does a Professor have to be treated like that? One of my colleagues here at the College whom I told my story looked at me, there was a silence, and then said: “Yes, they treat us like sh*t”. Best regards, Stefan Grimm |
There is now a way for staff to register their opinions of their employers.The entries for Imperial College on Glassdoor.com suggest that bullying there is widespread (on contrast, the grumbles about UCL are mostly about lack of space).
Googling ‘imperial college employment tribunal’ shows a history of bullying that is not publicised. In fact victims are often forced to sign gagging clauses. In fairness, AcademicFOI.com shows that the problems are not unique to Imperial. Over 3 years (it isn’t clear which years) , 810 university staff went to employment tribunals. And 5528 staff were gagged. Not a proud record
Imperial’s Department of Medicine web site says that one of its aims is to “build a strong and supportive academic community”. Imperial’s spokesman said “Stefan Grimm was a valued member of the Faculty of Medicine”.
The ability of large organisations to tell barefaced lies never ceases to amaze me.
I asked Martin Wilkins to comment on the email from Grimm. His response is the standard stuff that HR issues on such occasions. Not a word of apology, no admission of fault. It says “Imperial College London seeks to give every member of its community the opportunity to excel and to create a supportive environment in which their careers may flourish.”. Unless, that is, your research is insufficiently expensive, in which case we’ll throw you out on the street at 51. For completeness, you can download Wilkins’ mail.
After reading this post, Martin Wilkins wrote again to me (12.21 on 2nd December), He said
“You will appreciate that I am unable to engage in any further discussion – not because of any institutional policy but because there is an ongoing inquest into the circumstances of his death. What I can say is that there was no ongoing correspondence. We met from time to time to discuss science and general matters. These meetings were always cordial. My last meeting with him was to congratulate him on his recent paper, accepted by EMBOL "
The emails now revealed show that the relationship could hardly have been less “cordial”. Martin Wilkins appears to be less than frank about what happened.
If anyone has more correspondence which ought to be known, please send it to me. I don’t reveal sources (if you prefer, use my non-College email david.colquhoun72 (at) gmail.com).
The problem is by no means limited to Imperial. Neither is it universal at Imperial: some departments are quite happy about how they are run. Kings College London, Warwick University and Queen Mary College London have been just as brutal as Imperial. But in these places nobody has died. Not yet.
Follow-up
Here are a few of the tweets that appeared soon after this post appeared.
OMG. If you are an academic at a UK uni read this. Saddest thing I’ve read. Both for the individual and us all. http://t.co/g8xlQbLvDD
— GaryFoster (@Prof_GD_Foster) December 1, 2014
Utterly tragic consequences of running a University Department as a business, via @david_colquhoun. #UofABudgetModel http://t.co/Ql2AJcQqWR
— Dr. Andy Holt (@DrAndyHolt) December 1, 2014
Accusation of a culture of bullying at Imperial from Stefan Grimm, written just before he died. http://t.co/NeXysWFjZM
— Suzi Gage (@soozaphone) December 1, 2014
Bastards “@david_colquhoun: Publish *and* perish at Imperial College: the last email of Stefan Grimm http://t.co/DcQQlfwf78 #bullying”
— John Canning (@johngcanning) December 1, 2014
@david_colquhoun thank you for publishing this. Tragic.
— Melanie Byng (@ThetisMercurio) December 1, 2014
Good grief. This is what happens when science is run like a business http://t.co/3eArX3pRPz (via @david_colquhoun)
— Stephan Neuhaus (@stephanneuhaus1) December 1, 2014
Having worked at Imperial as a PostDoc, I recognise the vile atmosphere portrayed here (via @david_colquhoun). http://t.co/7ogAekAcLR
— Michel Valstar (@MichelValstar) December 1, 2014
Haunting stuff. @david_colquhoun writes of a tragic death and potential case of bullying at #Imperial College London http://t.co/uicA2wPE7S
— Michael Head (@michaelghead) December 1, 2014
@Cesar_F1000 @david_colquhoun Not the kind of "life of the mind" that anyone would want. Can Academia ever be reformed?
— Eric Kansa (@ekansa) December 1, 2014
@david_colquhoun @AMCELL Thank you. This is like something out of Terminator: the rise of the (administrator) machines.
— Dawn Bazely (@dawnbazely) December 1, 2014
@david_colquhoun @timeshighered @ChrisParrTHE "review" by Director of HR? Wonder if they'll find any fault with HR processes?
— calloutloud (@blowthatwhistl1) December 2, 2014
@david_colquhoun @timeshighered @ChrisParrTHE Stefan Grimm case order up a lorry load of whitewash for delivery to @imperialcollege DofHR
— calloutloud (@blowthatwhistl1) December 2, 2014
Shame on my old Uni for this! @david_colquhoun Publish and perish at @imperialcollege: the death of Stefan Grimm http://t.co/l0LZf8u8BX
— Scott Edmunds (@SCEdmunds) December 2, 2014
I saw the best minds of my generation destroyed by madness. @imperialcollege chews up profs. http://t.co/8ibqkk9fjJ @david_colquhoun
— Robert Davidson (@bobbledavidson) December 2, 2014
Saddest blog post I've read for ages: about the pressure on high performing academics by @david_colquhoun http://t.co/gH95I1bMCa
— Mark Brandon (@icey_mark) December 2, 2014
This breaks my heart @david_colquhoun Publish *and* perish at Imperial College London: last email of Stefan Grimm http://t.co/mUsEiM6OJU
— Sylvia McLain (@girlinterruptin) December 2, 2014
"Not a university but a business": Publish AND perish | http://t.co/6aOU6j9CRS | Via @david_colquhoun | Tragic, shameful, urgent
— Tom Farsides (@TomFarsides) December 2, 2014
3 December 2014
The day after this post went public, I wrote to the vice-chancellor of Imperial College, thus.
|
To: alice.gast@imperial.ac.uk cc: w.j.stirling@imperial.ac.uk, s.johal@imperial.ac.uk. d.humphris@imperial.ac.uk Dear Professor Gast You may be aware that last night, at 18.30, I published Stefan Grimm’s last email, see https://www.dcscience.net/?p=6834 In the 12 hours that it’s been public it’s had at least 10,000 views. At the moment, 230 people. from all round the world, are reading it. It seems to be going viral. I appreciate that you are new to the job of rector, so you may not realise that this sort of behaviour has been going on for years at Imperial (especially in Medicine) -I last wrote about the dimwitted methods being used to assess people in Medicine on 2007 -see https://www.dcscience.net/?p=182 Now it seems likely that the policy has actually killed someone (itwas quite predictable that this would happen, sooner or later). I hope that your your humanity will ensure a change of policy in your approach to “performance management”. Failing that, the bad publicity that you’re getting may be enough to persuade you to do so. Best regards David Colquhoun __________________________________ |
Today I updated the numbers: 44,000 hits after 36 hours.
I tried to put it politely, but I have not yet had a reply.
4 December 2014
More than one source at Imperial has sent me a copy of an email sent to staff by the dean of the Faculty of Medicine. It’s dated 03 December 2014 16:44. It was sent almost 24 hours after my post. It is, I suppose, just possible that Kelleher was unaware of my post. But he must surely have seen the internally-circulated version of Grimm’s letter. It isn’t mentioned: that makes the weasel words and crocodile tears in the email even more revolting than they otherwise would be. Both his account and Wilkins’ account contradict directly the account in Grimm’s mail.
Somebody is not telling the truth.
This post has broken all records (for this blog). It has been viewed over 50,000 times in 48 hours. It is still getting 35-40 visitors per minute, as it has for the last 2 days. How much longer will managers at Imperial be able to pretend that the cat hasn’t escaped from the bag?
5 December 2014
|
Late last night. Imperial made, at last. a public comment on the death of Stefan Grimm: Statement on Professor Stefan Grimm by Caroline Davis (Communications and Public Affairs). This bit of shameless public relations appears under a tasteful picture of lilies. |
|
It says “Members of Imperial’s community may be aware of media reports of the tragic loss of Stefan Grimm, professor of toxicology in the Faculty of Medicine”. They could hardly have missed the reports. As of 07.25 this morning, this post alone has been viewed 97,626 times, from all over the world. The statement is a masterpiece of weasel words, crocodile tears and straw man arguments. “Contrary to claims appearing on the internet, Professor Grimm’s work was not under formal review nor had he been given any notice of dismissal”. I saw no allegations that he had actually been fired. He was undoubtedly threatened with being fired. That’s entirely obvious from the email sent by Martin Wilkins to Stefan Grimm. on 10 March. The full text of that mail was published yesterday in Times Higher Education.
It’s worth reproducing the full text of that mail. To write like that to a successful professor, aged 51, is simply cruel. It is obviously incompatible with the PR guff that was issued yesterday. It seems to me to be very silly of Imperial College to try to deny the obvious.
I don’t know how people like Martin Wilkins and Caroline Davis manage to sleep at night.
|
Date: 10 March 2014 Dear Stefan I am writing following our recent meetings in which we discussed your current grant support and the prospects for the immediate future. The last was our discussion around your PRDP, which I have attached. As we discussed, any significant external funding you had has now ended. I know that you have been seeking further funding support with Charities such as CRUK and the EU commission but my concern is that despite submitting many grants, you have been unsuccessful in persuading peer-review panels that you have a competitive application. Your dedication to seek funding is not in doubt but as time goes by, this can risk becoming a difficult situation from which to extricate oneself. In other words, grant committees can become fatigued from seeing a series of unsuccessful applications from the same applicant. I am of the opinion that you are struggling to fulfil the metrics of a Professorial post at Imperial College which include maintaining established funding in a programme of research with an attributable share of research spend of £200k p.a and must now start to give serious consideration as to whether you are performing at the expected level of a Professor at Imperial College. Over the course of the next 12 months I expect you to apply and be awarded a programme grant as lead PI. This is the objective that you will need to achieve in order for your performance to be considered at an acceptable standard. I am committed to doing what I can to help you succeed and will meet with you monthly to discuss your progression and success in achieving the objective outlined. You have previously initiated discussions in our meetings regarding opportunities outside of Imperial College and I know you have been exploring opportunities elsewhere. Should this be the direction you wish to pursue, then I will do what I can to help you succeed. Please be aware that this constitutes the start of informal action in relation to your performance, however should you fail to meet the objective outlined, I will need to consider your performance in accordance with the formal College procedure for managing issues of poor performance (Ordinance D8) which can be found at the following link. Should you have any questions on the above, please do get in touch. Best wishes Martin |
These fixed performance targets are simply absurd. It’s called "research" because you don’t know how ir will come out. I’m told that if you apply for an Academic Clinical Fellowship at Imperial you are told
“Objectives and targets: The goal would be to impart sufficient training in the chosen subspecialty, as to enable the candidate to enter a MD/PhD programme at the end of the fellowship. During the entire academic training programme, the candidate is expected to publish at least five research articles in peer-reviewed journals of impact factor greater than 4.”
That’s a recipe for short term, unoriginal research. It’s an incentive to cut corners. Knowing that a paper has been written under that sort of pressure makes me less inclined to believe that the work has been done thoroughly. It is a prostitution of science.
Later on 5 December. This post has now had 100,000 views in a bit less that four days. At 13.30, I was at Kings College London, to talk to medical students about quackery etc. They were a smart lot, but all the questions were about Stefan Grimm.
The national press have begun to notice the tragedy. The Daily Mail, of all "newspapers" has a fair account of the death. It quotes Professor James Stirling, Provost of Imperial College London, as intoning the standard mantra:
“Imperial seeks to give every member of its community the opportunity to excel and to create a supportive environment in which their careers may flourish. Where we become aware that the College is falling short of this standard of support to its members, we will act”.
In my opinion the email above shows this is simply untrue. This sort of absurd and counterproductive pressure has been the rule in the Department of Medicine for years. I can’t believe that James Stirling didn’t now about it. If he did know, he should be fired for not anticipating the inevitable tragic consequences of his policies. If he didn’t know what was going on, he should be fired for not knowing. .
It is simply absurd for Imperial to allow (In)human resources to investigate itself. Nobody will believe the result.
An independent external inquiry is needed. Soon.
Stefan Grimm’s death is, ultimately, the fault of the use of silly metrics to mismeasure people. If there were no impact factors, no REF, no absurd university rankings, and no ill-educated senior academics and HR people who take them seriously, he’d probably still be alive.
8 December 2014
After one week, I wrote again to the senior management at Imperial (despite the fact that my earlier letters had been ignored). This time I had one simple suggestion. If Imperial want genuinely to set things right they should get an independent external inquiry. Their present proposal that the people who let things go so far should investigate themselves has been greeted with the scepticism that it so richly deserves. I still live in hope that someone will be sufficiently courteous to answer this time.
|
To: alice.gast@imperial.ac.uk cc: w.j.stirling@imperial.ac.uk, s.johal@imperial.ac.uk. d.humphris@imperial.ac.uk, d.kelleher@imperial.ac.uk Dear Professor Gast My post of Stefan Grimm’s email last Monday evening, has been viewed 130,000 times from at least 175 different countries. Your failure to respond to my letters is public knowledge. When you finally posted a statement about Grimm on Thursday it so obviously contradicted the emails which I, and Times Higher Education had already published, that it must have done your reputation more harm than good. May I suggest that the best chance to salvage your reputation would be to arrange for an independent external inquiry into the policies that contributed to Grimm’s death. You must surely realise that your announcement that HR will investigate its own policies has been greeted with universal scepticism. Rightly or wrongly, its conclusions will simply not be believed. I believe that an external inquiry would show Imperial is genuine in wishing to find out how to improve the way it treats the academics who are responsible for its reputation. Best regards David Colquhoun __________________________________ |
Here is a map of the location of 200 hits on 4 December (one of 20 such maps in a 4 hour period).
10 December 2014
Eventually I got a reply, of sorts, from Dermot Kelleher. It’s in the style of the true apparatchik "shut up and go away".
|
Dear Dr Colquhoun
Many thanks for your enquiry. Can I just say that College will liaise with the Coroner as required on this issue. In light of this, I do not believe that further correspondence will be helpful at present.
Best wishes
Dermot
|
After an interchange on Twitter about how blogs get noticed, I commented that the best thing for me was being thrown off the UCL web site by Malcolm Grant, and the subsequent support that I got from Ben Goldacre. I am a big fan of just about everything that Goldacre has done. So are a lot of other people and his support was crucial.
When I looked up his 2007 post, I found a lot of links were now broken, and some characters didn’t render properly. So, as a matter of historical record, I’m reproducing the whole post with updated links where possible.
Goldacre’s comments, of course, greatly exaggerated my virtues. But they were very useful at the time, they quadrupled my readership overnight, and I’m eternally grateful to him.
Some of the history of this saga has already been transferred to this blog. The aftermath was interesting.
The Mighty David Colquhoun
[Update: Letter from Provost below]
Ben Goldacre
Saturday June 9, 2007
The Guardian
I’ve always said you’d get a lot more kids interested in science if you told them it involves fighting – which of course it does. This week, for example, Professor David Colquhoun FRS – one of the most eminent scientists in the UK – has been forced to remove his quackbusting blog from the UCL servers where it has lived for many years, after complaints from disgruntled alternative therapists.
They objected, for example, to his use of the word “gobbledygook” to describe Red Clover as a “blood cleanser” or a “cleanser of the lymphatic system”. Somebody from the “European Herbal and Traditional Medicine Practitioners Association” complained that he’d slightly misrepresented one aspect of herbalists’ practice. One even complained about Colquhoun infringing copyright, simply for quoting the part of their website that he was examining. They felt, above all, that this was an inappropriate use of UCL facilities.
Now I don’t want to get into the to and fro here, but it is striking that none of them engaged the Prof himself on the issue of the ideas. In fact, they all ran behind his back to the Provost, or rather, to teacher; and the Provost, after serving up a sterling defense of academic freedom in responses to them, quietly asked Colquhoun to take his blog elsewhere, on the grounds that it was bringing the university too much flak. Rousing defenses of Colquhoun have already been written by Professors from Stanford, and senior academics from the UK. [Some are linked here, I’ve got the rest archived. The provost’s initial letter was actually rather stirring]
This episode reveals some unfortunate contrasts. Firstly, in a world where most orthodox "public engagement with science" activity consists of smug, faux radical "science meets art" projects where ballet dancers watch each other prance about in brain scanners (and I am hardly caricaturing here) Colquhoun was showing the world what science really does.
He took dodgy scientific claims, or “hypotheses” as we call them in the trade, and examined the experimental evidence for them, in everyday language, with humour and verve. For all that being a world expert on single ion channels might make Colquhoun glamorous to me, I would say his blog is a bit more of a treat for the wider public, and arguably a rather good use of the time and resources of a public servant who has devoted his entire life to academia, on its relatively low wages, never once working for industry. Sharing ideas is an employment perk in academia.
Secondly, giving special attention to a blog shows that we may not have got to grips with new forms of social media yet. His blog is the problem in hand, but I’ve heard Prof Colquhoun speak about quackery in UCL lecture theatres. Was the electricity, the publicity material, the room rent, a misuse of public funds and resources? I’ve done talks myself, in universities and schools: are they all guilty of wasting public money on robust, challenging, childish and sarcastic discussion of ideas?
But lastly, if you’re worrying about the appropriate use of a science department’s resources, Prof Colquhoun is the bloke who made the fuss in Nature -the biggest academic journal in the world – about British universities giving away science degrees in quackery. The people who run the BSc "science" degrees in these pseudoscientific alternative therapies have still refused to answer questions from David, and from me, about what "science" they teach in their science degrees.
I notice that nobody is making the jokers behind these Quackery BSc’s take their gobbledygook -a word that sounds best being snorted through Colquhoun’s impressive nasal hair – off university webservers. Although courses in gobbledygook make money. And they are flattered by the Prince. And nobody can criticise them, because they actually refuse to tell us what they’e teaching. Now you tell me who should be booted out of a seat of learning.
Please send your bad science to ben@badscience.net
So:
Prof Colquhoun doesn’t really have impressive nasal hair, I just didn’t want the column to come across as too gushing. His quack page is definitely worth rooting about on:
And as you can see, he needs WordPress advice even more than I do. Also his politics feed is quite jolly and if I could work out, for example, how to link directly to the Greenhalgh story, I would. Rummage away.
[DC edit: one of the best side effects of the move was getting a proper blog, rather than a bloated web page. The old politics page is archived and the Greenhalgh story link now works]
Letter from Provost:
This is an email from the Provost to someone who emailed him this morning, which he has allowed me to post, I understand he will be sending something similar to those who email him. It’s very much worth reading. I believe – as you can imagine – that an emeritus professor of pharmacology in his seventies making the link between science and real world claims for free in everyday language is a treat, but of course I have absolutely no doubt that Colquhoun’s public engagement with science activity did pose difficulties for UCL.
These difficulties were thrown into sharp relief by the fact that those who disagreed with Colquhoun enacted their grievances through the Freedom of Information Act, UK libel law, copyright law, complaints about the use of academic resources, and efforts to lean on senior figures from the university, rather than engaging on the science, or contacting Colquhoun.
There is a balance to be struck on whether Colquhoun’s public engagement with science activities were valued enough to be worth defending (through the miracle ofinstant context you can decide for yourself) and that is of course a decision for UCL to make.
If you are going to write to the Provost I hope I can rely on you to be polite and understanding about this balance, and understand that he’s a busy man who has already been leant on over what ideally should never have been a Provost’s concerns at such an early stage.
Andrew
If UCL had behaved in the way you seem to believe then your comments would be wholly justified, but of course it hasn’t.
Allow me to supply the missing facts. I;m copying this message also to Ben Goldacre and David Colquhoun.
Academic freedom is a fundamental precept of any institution fit to style itself a university. Like all freedoms, it comes with conditions, largely those that are necessary to underpin the freedoms of other people under the law, including criminal law, human rights, copyright, the laws of tort and contract, and statutory regulation.
When a university hosts a website it is taken to be the publisher of the material on it. That means that it is liable in law for any breaches of copyright, data protection and defamation. It is possible of course to engage in robust academic debate without infringing any of these rules.
But breaches of all of them have now been claimed in legal claims against UCL regarding David Colquhoun’s website, and with good reason.
A university can of course safeguard its position by moderating the content of the website. That is what I assume the Guardian does with its various blogs, and certainly is what it does with all its editorial content. Nobody sees that as a major assault on the freedom of expression of the press. To do this in a university would of course raise concerns that it constituted an incursion into academic freedom, and I also think it would be completely impractical.
Yet not to take appropriate action to protect UCL would be to expose us to potentially expensive legal action in respect of activity over which we have absolutely no control.
For the most part, academic websites don’t infringe the law. Indeed, in over 35 years as an academic this is the first such instance that I have any detailed knowledge of. If it has unlawful material that the author believes is essential for conveying his/her message, then there is no reason why they shouldn’t host it themselves and assume the consequences.
UCL has taken legal advice, which is to the effect that the website does contain material which breaks the law in several respects. Some of them have now been fixed: alleged breaches of copyright and data protection. But libel proceedings are now also in play, and Professor Colquhoun and I have a meeting on Monday with a senior defamation QC to explore the potential extent of UCL’s vicarious liability for certain statements on the website, and our possible options. There is also the question of Professor Colquhoun’s own personal liability, but of course a plaintiff will always prefer to go against a major institution because of our deep pockets.
On the basis of the advice that I receive then I shall have to determine UCL’s future course of action, and Professor Colquhoun likewise.
Malcolm Grant
Just to be absolutely clear:
The item that has caused the fuss and complaint is this one. It has not been changed since the complaint, so you can decide for yourself how awful it is.
++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Scienceand Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++
56 Responses
In the course of thinking about metrics, I keep coming across cases of over-promoted research. An early case was “Why honey isn’t a wonder cough cure: more academic spin“. More recently, I noticed these examples.
“Effect of Vitamin E and Memantine on Functional Decline in Alzheimer Disease".(Spoiler -very little), published in the Journal of the American Medical Association. ”
and ” Primary Prevention of Cardiovascular Disease with a Mediterranean Diet” , in the New England Journal of Medicine (which had second highest altmetric score in 2013)
and "Sleep Drives Metabolite Clearance from the Adult Brain", published in Science
In all these cases, misleading press releases were issued by the journals themselves and by the universities. These were copied out by hard-pressed journalists and made headlines that were certainly not merited by the work. In the last three cases, hyped up tweets came from the journals. The responsibility for this hype must eventually rest with the authors. The last two papers came second and fourth in the list of highest altmetric scores for 2013
Here are to two more very recent examples. It seems that every time I check a highly tweeted paper, it turns out that it is very second rate. Both papers involve fMRI imaging, and since the infamous dead salmon paper, I’ve been a bit sceptical about them. But that is irrelevant to what follows.
Boost your memory with electricity
That was a popular headline at the end of August. It referred to a paper in Science magazine:
“Targeted enhancement of cortical-hippocampal brain networks and associative memory” (Wang, JX et al, Science, 29 August, 2014)
This study was promoted by the Northwestern University "Electric current to brain boosts memory". And Science tweeted along the same lines.
|
Science‘s link did not lead to the paper, but rather to a puff piece, "Rebooting memory with magnets". Again all the emphasis was on memory, with the usual entirely speculative stuff about helping Alzheimer’s disease. But the paper itself was behind Science‘s paywall. You couldn’t read it unless your employer subscribed to Science.
|
 |
|
All the publicity led to much retweeting and a big altmetrics score. Given that the paper was not open access, it’s likely that most of the retweeters had not actually read the paper. |
|
When you read the paper, you found that is mostly not about memory at all. It was mostly about fMRI. In fact the only reference to memory was in a subsection of Figure 4. This is the evidence.
That looks desperately unconvincing to me. The test of significance gives P = 0.043. In an underpowered study like this, the chance of this being a false discovery is probably at least 50%. A result like this means, at most, "worth another look". It does not begin to justify all the hype that surrounded the paper. The journal, the university’s PR department, and ultimately the authors, must bear the responsibility for the unjustified claims.
Science does not allow online comments following the paper, but there are now plenty of sites that do. NHS Choices did a fairly good job of putting the paper into perspective, though they failed to notice the statistical weakness. A commenter on PubPeer noted that Science had recently announced that it would tighten statistical standards. In this case, they failed. The age of post-publication peer review is already reaching maturity
Boost your memory with cocoa
Another glamour journal, Nature Neuroscience, hit the headlines on October 26, 2014, in a paper that was publicised in a Nature podcast and a rather uninformative press release.
"Enhancing dentate gyrus function with dietary flavanols improves cognition in older adults. Brickman et al., Nat Neurosci. 2014. doi: 10.1038/nn.3850.".
The journal helpfully lists no fewer that 89 news items related to this study. Mostly they were something like “Drinking cocoa could improve your memory” (Kat Lay, in The Times). Only a handful of the 89 reports spotted the many problems.
A puff piece from Columbia University’s PR department quoted the senior author, Dr Small, making the dramatic claim that
“If a participant had the memory of a typical 60-year-old at the beginning of the study, after three months that person on average had the memory of a typical 30- or 40-year-old.”
|
Like anything to do with diet, the paper immediately got circulated on Twitter. No doubt most of the people who retweeted the message had not read the (paywalled) paper. The links almost all led to inaccurate press accounts, not to the paper itself. |
|
But some people actually read the paywalled paper and post-publication review soon kicked in. Pubmed Commons is a good site for that, because Pubmed is where a lot of people go for references. Hilda Bastian kicked off the comments there (her comment was picked out by Retraction Watch). Her conclusion was this.
"It’s good to see claims about dietary supplements tested. However, the results here rely on a chain of yet-to-be-validated assumptions that are still weakly supported at each point. In my opinion, the immodest title of this paper is not supported by its contents."
(Hilda Bastian runs the Statistically Funny blog -“The comedic possibilities of clinical epidemiology are known to be limitless”, and also a Scientific American blog about risk, Absolutely Maybe.)
NHS Choices spotted most of the problems too, in "A mug of cocoa is not a cure for memory problems". And so did Ian Musgrave of the University of Adelaide who wrote "Most Disappointing Headline Ever (No, Chocolate Will Not Improve Your Memory)",
Here are some of the many problems.
- The paper was not about cocoa. Drinks containing 900 mg cocoa flavanols (as much as in about 25 chocolate bars) and 138 mg of (−)-epicatechin were compared with much lower amounts of these compounds
- The abstract, all that most people could read, said that subjects were given "high or low cocoa–containing diet for 3 months". Bit it wasn’t a test of cocoa: it was a test of a dietary "supplement".
- The sample was small (37ppeople altogether, split between four groups), and therefore under-powered for detection of the small effect that was expected (and observed)
- The authors declared the result to be "significant" but you had to hunt through the paper to discover that this meant P = 0.04 (hint -it’s 6 lines above Table 1). That means that there is around a 50% chance that it’s a false discovery.
- The test was short -only three months
- The test didn’t measure memory anyway. It measured reaction speed, They did test memory retention too, and there was no detectable improvement. This was not mentioned in the abstract, Neither was the fact that exercise had no detectable effect.
- The study was funded by the Mars bar company. They, like many others, are clearly looking for a niche in the huge "supplement" market,
The claims by the senior author, in a Columbia promotional video that the drink produced "an improvement in memory" and "an improvement in memory performance by two or three decades" seem to have a very thin basis indeed. As has the statement that "we don’t need a pharmaceutical agent" to ameliorate a natural process (aging). High doses of supplements are pharmaceutical agents.
To be fair, the senior author did say, in the Columbia press release, that "the findings need to be replicated in a larger study—which he and his team plan to do". But there is no hint of this in the paper itself, or in the title of the press release "Dietary Flavanols Reverse Age-Related Memory Decline". The time for all the publicity is surely after a well-powered study, not before it.
The high altmetrics score for this paper is yet another blow to the reputation of altmetrics.
One may well ask why Nature Neuroscience and the Columbia press office allowed such extravagant claims to be made on such a flimsy basis.
What’s going wrong?
These two papers have much in common. Elaborate imaging studies are accompanied by poor functional tests. All the hype focusses on the latter. These led me to the speculation ( In Pubmed Commons) that what actually happens is as follows.
- Authors do big imaging (fMRI) study.
- Glamour journal says coloured blobs are no longer enough and refuses to publish without functional information.
- Authors tag on a small human study.
- Paper gets published.
- Hyped up press releases issued that refer mostly to the add on.
- Journal and authors are happy.
- But science is not advanced.
It’s no wonder that Dorothy Bishop wrote "High-impact journals: where newsworthiness trumps methodology".
It’s time we forgot glamour journals. Publish open access on the web with open comments. Post-publication peer review is working
But boycott commercial publishers who charge large amounts for open access. It shouldn’t cost more than about £200, and more and more are essentially free (my latest will appear shortly in Royal Society Open Science).
Follow-up
Hilda Bastian has an excellent post about the dangers of reading only the abstract "Science in the Abstract: Don’t Judge a Study by its Cover"
4 November 2014
I was upbraided on Twitter by Euan Adie, founder of Almetric.com, because I didn’t click through the altmetric symbol to look at the citations "shouldn’t have to tell you to look at the underlying data David" and "you could have saved a lot of Google time". But when I did do that, all I found was a list of media reports and blogs -pretty much the same as Nature Neuroscience provides itself.
More interesting, I found that my blog wasn’t listed and neither was PubMed Commons. When I asked why, I was told "needs to regularly cite primary research. PubMed, PMC or repository links”. But this paper is behind a paywall. So I provide (possibly illegally) a copy of it, so anyone can verify my comments. The result is that altmetric’s dumb algorithms ignore it. In order to get counted you have to provide links that lead nowhere.
So here’s a link to the abstract (only) in Pubmed for the Science paper http://www.ncbi.nlm.nih.gov/pubmed/25170153 and here’s the link for the Nature Neuroscience paper http://www.ncbi.nlm.nih.gov/pubmed/25344629
It seems that altmetrics doesn’t even do the job that it claims to do very efficiently.
It worked. By later in the day, this blog was listed in both Nature‘s metrics section and by altmetrics. com. But comments on Pubmed Commons were still missing, That’s bad because it’s an excellent place for post-publications peer review.
|
Maurice Nathan Saatchi, Baron Saatchi is an advertising man who, with his brother, Charles Saatchi ("‘why tell the truth when a good lie will do?), became very rich by advertising cigarettes and the Conservative party. After his second wife died of cancer he introduced a private members bill in the House of Lords in 2012. The Medical Innovation Bill came back to the Lords for its second reading on 24 October 2014. |
 |
The debate was deeply depressing: very pompous and mostly totally uninformed. You would never have guessed that the vast majority of those who understand the problem are against the bill. The Bill has failed to win the support of The British Medical Association, The Motor Neurone Disease Association, the Royal College of Physicians, the Royal College of Surgeons, the Medical Research Society, the Royal College of Radiologists, The Medical Defence Union, the Academy of Medical Sciences, the Royal College of Pathologists, the Royal College of General Practitioners, the Academy for Healthcare Science, the Wellcome Trust, Action Against Medical Accidents, and patient advocacy charities Health Watch and the Nightingale Collaboration, and others. Conservative MP Sarah Wollaston, a former GP, has “very serious concerns” about it.
Grief is not a good basis for legislation. In the Daily Telegraph, Lord Saatchi attributes the lack of a "cure for cancer" to fear of litigation.
“Any deviation by a doctor is likely to result in a verdict of guilt for medical negligence. The law defines medical negligence as deviation from that standard procedure. But as innovation is deviation, non-deviation is non-innovation. This is why there is no cure for cancer.”
This statement is utterly bizarre. Evidently Lord Saatchi knows much more about how to sell cigarettes than he does about how to discover new drugs. The reason there is no cure is that it’s a very difficult problem. It has nothing to do with litigation
Almost every medical organisation, and many lawyers, have pointed out the flaws in his ideas,. But slick, and often mendacious, advertising of his cure has deceived many of our scientifically-illiterate parliamentarians, and the bill is making progress
Lord Saatchi’s Bill would allow uncontrolled testing of treatments on any patient. It is not limited to cancer, nor to terminally ill patients (though some amendments, yet to be accepted, might change that). This sort of uncontrolled experimentation is likely to impede advances in treatment rather than to help them. And the vagueness of the wording of the bill could lead to an increase in litigation, rather than the intended decrease.
It is no coincidence that the legion of cancer quacks is in favour of the bill. It opens the door to their nonsense. Big pharma is likely to benefit too, because they will be able to sell improperly tested drugs with little or no effectiveness and immunity from prosecution if they do harm.
For more information about the Bill, see the excellent site, Stop the Saatchi Bill. There are sections on Dispelling Myths, on Professional Concerns, and on the many good articles and blogs that have pointed out the many problems with the bill.
Two recent articles are well worth reading
Saatchi’s ‘Medical Innovation Bill’ will benefit lawyers and charlatans, not patients, by neurologist David Nicholl. And it’s particularly interesting that the Telegraph’s arch-conservative, Peter Oborne has come out strongly against the bill, in The ‘Saatchi Bill’ is dangerous and will bring nothing but harm. I can’t agree with his opening words ("Lord Saatchi is rightly regarded as a national treasure. In his early days he was the advertising genius who played a fundamental role in selling the Thatcherite message"), but I agree entirely with
“The PR campaign distorts the facts, exploiting ignorance and desperation to plant false hope in the rich soil of multiple fallacies.”
“Quacks will be given free rein. No “treatment” is so loopy (or potentially dangerous) that the Saatchi Bill won’t protect doctors who prescribe or administer it from prosecution.”
“The propagation of falsehoods and fallacies surrounding and arising from this Bill will have a corrosive effect on medical ethics and society as a whole.”
Lord Saatchi, who proposed the bill said “To prevent more needless cancer deaths, doctors must be free to innovate”. This statement shows he has no idea of the efforts that go into cancer research. We are doing our best, but it is a very hard problem. In some areas (like breast cancer) there have been big advances. In others areas (e.g. pancreatic or ovarian cancer) there has not yet been any perceptible progress. It will take many more years of hard scientific work. Individual doctors tinkering with speculative untested treatments will not solve the problem.
- The only people who will benefit from the bill will be lawyers, quacks and big pharma.
- “Innovation” is undefined –the bill is based on the myth that there exists a miraculous cure waiting to be found. Sadly, the probability of this happening is vanishingly small.
- To some extent, medicine is victim of its own hype. The public feels it has a right to demand the latest miracle cure. Too often, they don’t exist.
- There is no need for the bill because doctors can already prescribe whatever they want. The Medical Defence Union says it has no reason to think that doctors are deterred from innovating by the fear of litigation.
- The bill won’t help to discover new cures –in fact it is more likely to hinder it, especially if it deters people from joining properly organised clinical trials. The bill will generate many separate anecdotes which may or may not be published. That’s the worst possible way to do research.
- The bill removes protections from patients and is more likely to increase litigation than reduce it.
- Anyone unlucky enough to get cancer is immediately a target for a legion of quacks trying to sell you crazy “treatments”. There is nothing to stop even the lunatic fringe of alternative medicine, homeopaths, promoting their sugar pills. There are, disgracefully, several hundred homeopaths with medical degrees –all a quack has to do is to ask another quack doctor to endorse their daft ideas.
- For untested treatments there is, by definition, no reason to think they’ll do more good than harm. The cancer surgeon, Michael Baum said, rightly, that in cases of terminal cancer "there will be many patients we can’t help. but none we can’t harm".
As a result of tweeting about the bill, I got a call on 21 October from RT TV (UK) asking me speak on the Saatchi Bill. That was a pleasure. Here it is (done from home, on Skype).
And on the day of the Lords debate (24 October 2014) I was asked to talk again about the Saatchi Bill, on the flagship Radio 4 morning news programme, Today. It was only a 3 minute interview with Mishal Husain, but it was unopposed so I managed to get in the main points.
Click to play interview
This is the Today Programme studio.
The BBC doesn’t pay you, but it’s good at driving you to/from the studio, though the cars are unnecessarily posh. On the way home, I had my first ever ride in an all-electric Tesla. It does 0 – 60 in 4.2 seconds but costs £50k -more than twice the price of a Prius.
|
|
Follow-up
25 October 2014 A summary of the misunderstandings in yesterday’s House of Lords debate: What the Lords Missed.
I have always been insanely proud to work at UCL. My first job was as an assistant lecturer. The famous pharmacologist, Heinz Otto Schild gave me that job in 1964, and apart from nine years, I have been there ever since. That’s 50 years. I love its godless tradition. I love its multi-faculty nature. And I love its relatively democratic ways (with rare exceptions).
|
From the start, the intellectual heart of UCL has been the staff Common Room. As I so often say, failing to waste time drinking coffee with people who are cleverer than yourself can seriously damage your career (and your happiness). And there’s no better place for that than the Housman room. |
 |
It is there that I met the great statistician Alan Hawkes, without whom much of my research would never have happened. It was there that Hyman Kestelman (among others) gave me informal tutorials on matrix algebra over lunch. It was there where I have met John Sutherland (English), Mary Fulbrook (German), many historians and people from the Slade school of Art. And it was there where, yesterday, I had an illuminating conversation with Steve Jones about the problems of twin studies for measuring heritability.
I was astonished when I arrived at UCL to discover that the Housman room was male only. I’d just come from Edinburgh which still had separate men’s and women’s student unions and some men-only bars. But Edinburgh also had a wonderful staff club, open to all. It’s true that UCL had also a women-only common room and a mixed common room, the Haldane room (which is where I went usually). But the biggest and most impressive room, the Housman room, was for men only. I found this very odd in the 1960s, the age of sexual liberation. Reform was in the air in the 1960s.
A lot of other people, not all female, thought it odd too. Direct action was called for (I was in CND at the time). So we’d go into the Housman room with a woman and join the queue for coffee. It never took long before some pompous prat would tap the woman on the shoulder and eject her. I can’t remember now the names of any of the feisty women who braved the lions’ den (perhaps this blog will remind someone).
|
I had any ally in Brian Woledge. He was Fielden Professor of French at UCL from 1939 (when I was 3) to 1971 so he was on the brink of retirement. I was a young lecturer, but our thinking on segregation was much the same. His obituary in the Guardian says “Of robustly secular beliefs and Fabian views, in important respects he was an heir to the ideals of the Enlightenment”. It’s no wonder we got on well. The picture, from around 1970, was supplied by his son, Roger Woledge, who was in the Physiology department at UCL for most of his life, and who did his PhD with my great hero, A.V. Hill. |
 |
In 1967 we proposed a motion at the Housman AGM to desegregate all common rooms. It was defeated. The next year we did it again, and were defeated again.. But at the third attempt, in 1969, we succeeded. I was very happy to have had a small role in upholding UCL’s liberal traditions.
|
It is now quite impossible to imagine that UCL was segregated. After all, UCL was the first English university to admit women on equal terms to men, in 1878 (the Scots were a bit ahead) And UCL was home to Kathleen Lonsdale (1903 -1971), one of the first two female fellows of the Royal Society, and the first female professor at UCL. |
 |
Nevertheless, in the mid-1960s, women were very far from being regarded as equal, even at UCL. At the time, segregation was more common than people now remember.
I was spurred to write this post when Melissa Terras, UCL’s professor of digital humanities, retweeted a reminder that it was in 1967 that a woman first ran in a an official marathon, and suffered physical attack from a male organiser for her temerity.
I responded
I was urged to record this history by both Terras and by Lisa Jardine, Director of UCL’s Centre for Interdisciplinary Research in the Humanities. So I have done it.
I was very aware of Kathy Switzer at the time, and I’ve no doubt she is part of the reason why I felt strongly about segregation. You can read about the 1967 Boston marathon in her own words. I thought it was a wonderful story, though I wasn’t yet into distance running myself (I was still sailing and boxing).
One of the great thing about marathons is that women and men run in the same race. That means that almost all men have had to get used to being overtaken by very many women. That has been wonderfully good for deflating male egos. When I was training for marathons in the 1980s, my training partner, Annie Briggs was on the elite start -a good hour faster than I could manage.
|
Now we are accustomed to watching Paula Radcliffe run marathons faster than any but the very best men. She’s the world record holder with the spectacular time of 2 hours 15 min in the 2003 London Marathon (my best is 3 hr 57 min). That’s only a bit over 26 consecutive 5 minute miles. And that’s faster than I could run a single mile at my peak.[Picture from Wikipedia: NYC marathon 2008 2:23:56] |
 |
It’s now utterly beyond belief that in the 1960s men were saying that women were too feeble to run 26 miles. It was sheer blind arrogance. After Switzer, progress was fast. In 1972 women were allowed to run in Boston, and within 10 years, the women’s record time had fallen by a full hour. Physiology hadn’t changed, but confidence had.
Of course it wasn’t until the 2012 Olympics that women gained total equality in sport. Everyone who said that women were incapable of competing in combat sports should see Rosi Sexton in action.
She’s the ultimate high-achiever. She’s an accomplished musician (grade 7 cello, ALCM piano) and she played at the Albert Hall with the Reading Youth Orchestra. She went on to get a first in maths (Cambridge, Trinity College), where her tutor was Tim Gowers. Then she did a PhD in theoretical computer science from Manchester (read her thesis). And she’s had a distinguished career as professional athlete, competing at the highest level in MMA. Why? “The other things I did, the music, the maths, just weren’t quite hard enough“.
 Taking bow at school concert |
 PhD, Manchester |
 Athlete in MMA |
Not many athletes have a paper in the Journal of Pure and Applied Algebra. I’d be very happy if I could do any one of these things as well as she does.
|
It could not be more appropriate than to be writng this in the week when the Fields medal was won by a woman, Maryam Mirzakhani, for the first time since it started, in 1936. Genetics hasn’t changed since 1936. Confidence has. UCL mathematician, Helen Wilson, points out the encouragement this will give to female mathematicians. |
 |
On 15 July 2017, Maryam Mirzakhani died, at a mere 40 years old. It’s tragic that having achieved so much, against all the odds, the dice rolled the wrong way for her, and cancer destroyed her. Her life will inspire generations to come.
As in marathons, confidence, role models and zeitgeist matter as much as genetics.
It’s examples like these that have made me profoundly suspicious of generalisations about what particular groups of people can and cannot do. Whether it is working class boys. black boys, or women, such generalisations can be shattered over a decade or two, once the zeitgeist changes.
That’s one reason that I am so unsympathetic to the IQ enthusiasts. Great harm has stemmed from the belief that it’s possible to sum up human achievements in a single number. What’s more, it’s a number that measures your resemblance to white male psychologists. It is because politicians believed the over-hyped claims of psychologists in the 1930s, that three-quarters of the population was written off. Much the same thing has happened with women, and with skin colour.
Don’t believe it.
And the job of desegregation may not be entirely finished. In fact now it is harder to combat, since it’s unspoken. Once again, I’m reminded of Peter Lawrence’s essay, The Mismeasurement of Science. Speaking of the perverse incentives and over-competitiveness that has invaded academia, he says
“Gentle people of both sexes vote with their feet and leave a profession that they, correctly, perceive to discriminate against them [17]. Not only do we lose many original researchers, I think science would flourish more in an understanding and empathetic workplace.”
The perverse incentives that make academic life hard for women (and for many men too) are administered by HR departments (with the collusion of mostly elderly male academics). They are the very same people who write fine-sounding diversity documents and lecture you about work-life balance.
It’s time they woke up.
Note. The minutes of Housman AGMs from the 1960s are missing at the moment. If they come to light, this post will be modified accordingly.
Follow-up
29 August 2014
|
As I’d hoped, this post elicited the name of one of the women who braved the rules and went into the Housman room when it was still men-only. I had an email from Lynn Bindman, and she told me that one of them was Gertrude Falk (1925 – 2008), who had worked in Bernard Katz’s Biophysics Department since 1961. |
In 1967 she must have been about 42. The episode is mentioned in Gertrude’s obituary in the Guardian. She also sent me a copy of the Physiologocal Society’s obituary, which recounts the story thus.
“Her indifference to conventions is well illustrated by the occasion when, drinking coffee in the men’s staff common room, at that time still segregated, she responded calmly to the Beadle summoned to escort her out, “well, I am certainly going to finish my coffee first”, and did so at her leisure.”
I have another story about Gertrude’s feistiness. Every year the Royal Society has a soirée for fellows and guests. It’s a sort of private view for the Summer Science exhibition. Men are required to dress like penguins despite the heat, and the invitation says “decorations will be worn”. The food is good though it’s all a bit pompous for my taste. Some years ago I met Gertrude at a soirée and I saw she was wearing a medal round her neck. I said “have they made you a Dame of the British Empire?”. She held up the medal and I saw it said “Erasmus High School Economics Prize”. She is why I usually go to the soirée wearing my London Marathon medal.
12 May 2015
Surprising as it seems now that the Housman room excluded women until 1969, there are other UCL institutions that were almost as slow as Oxford and Camridge to join the modern age.
One of these is the Professors’ Dining Club (it isn’t actually restricted to professors). I recall going to one of their dinners in the 1960s, as a guest of Heinz Otto Schild, the then head of Pharmacology, who gave me my first job. He was a lovely man, but I was horrified that it didn’t allow women to join. I recently discovered that its records reveal that it didn’t see the light until 1981. It wasn’t until after that happened that I joined the club. It seems now to be a shameful record.
This post follows directly from “Some pharmacological history: an exam from 1959“. In that post, I related how two of my teachers in Leeds, James Dare and George Mogey, had encouraged my interest in statistcs. George Mogey had worked previously at the famous Wellcome Research Labs in Beckenham, Kent. He had been there at the same time as J.W. Trevan, who pioneered accurate methods of biological assay.
Another person who overlapped with Mogey and Trevan at Beckenham was C.L. Oakley. I’m told by Audrey Mogey, George’s widow, that they were good friends of the Oakleys and that probably explains why George Mogey introduced me to Cyril Oakley, who had the chair of bacteriology at Leeds while I was an undergraduate there. Oakley’s Biographical Memoir makes no mention of statistics. The only person I’ve located who knew him is Keith Holland (professor of microbiology at Leeds). He told me
“I was trained by CLO between !961-65 and he inspired me to remain in research into aspects of anaerobic bacteriology and I attended his lectures on statistics, which were highly stimulating and humorous. He frequently used examples of magicians turning lead into gold and I can not recall examples of goats and men.”
The statistical connection stems from an article that was written by Oakley in 1943, Oakley, C. L. (1943). “He-goats into young men: first steps in statistics”, University College Hospital Magazine Vol 28, 16-21. Now you can download a copy of this rather obscure publication.
The action occurs on the Brocken. The paper starts by citing the Illustrated London News (the internet of its age). In 1932 an experiment was done which allegedly dispelled the legend of the Brocken. Here it is.
Oakley uses the Brocken experiment to explain the statistical method known as probit analysis. This was obviously something he’d learned from J.W. Trevan during his time at the Beckenham lab (e.g his classic 1927 paper, The Error of Determination of Toxicity) . And it was my meeting with Oakley, as an undergraduate, that caused me to use his paper as the basis of a section in Lectures on Biostatistics.
It also explains why, ever since the late 50s, I’ve wanted to visit the Brocken. It’s only about 100 km from Göttingen, where I worked often between 1980 and 1985, but at that time the Brocken was in East Germany. I remember looking across the wall at the Harz mountains, when Erwin Neher took us into the country to pick wild bilberries (blueberries, Heidelbeeren). Reunification of Germany occurred while I was working in Heidelberg in 1991 but it was not until a month ago that I got there. We took a rail tour of Germany, and spent four days in the Harz town of Wernigerode, from where we took the Harzer Schmalspurbahn, the steam powered narrow gauge railway, to the Brocken. Here are some pictures of the trip (click first picture for an album)..
All I got was a teapot stand, with a witch on a broomstick (and I don’t even drink the stuff myself).
Interestingly, although there is plenty of tourist tat about the connection with Dr Faust and Goethe, I didn’t find any German who’d heard of the he-goat conversion legend. One of the people involved in the experiment, Harry Price (1881 – 1948) of the National Laboratory of Psychical Research seems to have been behind it, and the history is described by him in the “Bloksberg Tryst” (Blocksberg is another name for the Brocken). Another person who conducted the experiment was Professor Joad (1891 – 1953). I can just remember hearing him on the BBC Home Service (radio) programme, the Brains Trust, which also featured Julian Huxley and Jacob Bronowski (1908 -1974). They were the public intellectuals of the early 1950s. (Much later, I discovered that Bronowski was the father of Lisa Jardine, who now works at UCL).
Oakley (1943) starts by citing the account in the Illustrated London News.
“The legend of the Brocken (the famous peak in the Harz Mountains, noted for its spectre and as the haunt of witches on Walpurgis Night), according to which a virgin he-goat can be converted into “a youth of surpassing beauty” by spells performed in a magic circle at midnight, was tested on June 17 by British and German scientists and investigators, including Professor Joad and Mr. Harry Price, of the National Laboratory of Psychical Research. The object was to expose the fallacy of Black Magic and also to pay a tribute to Goethe, who used the legend in Faust. Some wore evening dress. The goat was anointed with the prescribed compound of scrapings from church bells, bats’ blood, soot and honey. The necessary maiden pure in heart, who removed the white sheet from the goat at the critical moment, was Fräulein Urta Bohn, daughter of one of the German professors taking part in the test. Her mother was a Scotswoman (formerly Miss Gordon). The scene was flood-lit and filmed. As our photographs show, the goat remained a goat and the legend of the Brocken was dispelled”.
Oakley then proposes a biological assay to measure purity in heart.
“It will he observed that the only incompletely controllable variables in the experiment (excluding Iocal variations in the church bells, bat’s blood, soot and honey) are the virgin he-goat and the maiden (virgin?) pure in heart. Virginity may for the present be regarded as an absolute character —purity in heart no doubt varies from person to person.. If, therefore, a reasonably uniform supply of virgin he-goats be obtained, and the percentage of he-goats converted bears
any relation to the purity in heart of the maiden used, we ought appear “>to
be able to measure the degree of purity in heart of the virgins available.”
The argument he uses is based directly on J.W. Trevan. The story reappeared in Chapter 7 (section 7.8, page 111) of Lectures on Biostatistics, where I used it to illustrate confidence intervals for a binomial proportion.
“We shall assume, as Oakley did, that the conversion of he-goats into young men is an all-or-nothing process; either complete conversion or nothing occurs. Oakley supposed, on this basis, that a comparison could be made between, on one hand, the percentage of he-goats converted by maidens of various degrees of purity in heart, and, on the other hand, the sort of pharmacological experiment that involves the measurement of the percentage of individuals showing a specified
effect in response to various doses of a drug. In conformity with the common pharmacological practice he supposed that a plot of percentage he-goat conversion against log purity in heart index (log PHI) would have the sigmoid form shown in Fig. 14.2.4. As explained in Chapter 14, this implies that log PHI required to convert individual he-goats is a normally distributed variable. Furthermore it means that infinite purity in heart is required to produce a population he-goat
conversion rate (HGCR) of 100 per cent..
Although there is a lack of experimental evidence on this point, the present author feels that the assumption of a normal distribution is, as so often happens, without foundation (see § 4.2). The implication of the normality assumption, that there exist he-goats so resistant to conversion that infinite purity in heart is needed to affect them, has
not been (and cannot be) experimentally verified. Furthermore the very idea of infinite purity in heart seems likely to cause despondency in most people, and should therefore be avoided until such time as its necessity may be demonstrated experimentally.”
In the light of these remarks it appears to the present author desirable that the purity in heart index should be redefined simply as the population percentage of he-goats converted. This simple operational definition means that the PHI of all maidens will fall between 0 and 100, and confidence limits for the true PHI can be found easily from the observed conversion rate (which should be binomially distributed, see §§ 3.2-3.5) using Table A2, as explained in §7.7.
For example, if it were observed that a particular maiden caused conversion of r = 2 out of n = 4 he-goats, the estimated PHI would be 100 × 2/4 = 50 per cent, and, from Table A2, confidence limits (P = 0·95) for true PHI are 6.8 – 93.2 per cent. Clearly the information be gained from a sample of only four he-goats is so imprecise that it difficult to conceive what use it could be put to. Oakley recommended that for preliminary experiments at least n = 10 he-goats should be used. If r = 5 (50 per cent) of these were observed to be converted Table A2 would give the confidence limits (P = 0·95) for the true PHI as 18·7 — 81·3 per cent. While the most extreme forms of vice and of virtue appear to be ruled out by this result, there is still considerable uncertainty about the PHI. If a greater degree of confidence were required, as for example, if a potential husband demanded a certain minimum (or, alternatively, a certain maximum) PHI before committing himself, the P = 0.99 confidence limits could found from Table A2. They are 12.8 — 87.2 per cent. The most tolerant suitor might be forgiven for requiring a larger sample.”
The statistics are pretty standard stuff. You can find out more by downloading Lectures on Biostatistics. The binomial distribution in Chapters 3, 7 and 8. Probit analysis is described in Chapter 14.
For some real statistics, please look at “An investigation of the false discovery rate and the misinterpretation of P values“, now available as a preprint on arXiv.
Follow-up
The two posts on this blog about the hazards of s=ignificance testing have proved quite popular. See Part 1: the screening problem, and Part 2: Part 2: the false discovery rate. They’ve had over 20,000 hits already (though I still have to find a journal that will print the paper based on them).
Yet another Alzheiner’s screening story hit the headlines recently and the facts got sorted out in the follow up section of the screening post. If you haven’t read that already, it might be helpful to do so before going on to this post.
This post has already appeared on the Sense about Science web site. They asked me to explain exactly what was meant by the claim that the screening test had an "accuracy of 87%". That was mentioned in all the media reports, no doubt because it was the only specification of the quality of the test in the press release. Here is my attempt to explain what it means.
The "accuracy" of screening tests
Anything about Alzheimer’s disease is front line news in the media. No doubt that had not escaped the notice of Kings College London when they issued a press release about a recent study of a test for development of dementia based on blood tests. It was widely hailed in the media as a breakthrough in dementia research. For example, the BBC report is far from accurate). The main reason for the inaccurate reports is, as so often, the press release. It said
"They identified a combination of 10 proteins capable of predicting whether individuals with MCI would develop Alzheimer’s disease within a year, with an accuracy of 87 percent"
The original paper says
"Sixteen proteins correlated with disease severity and cognitive decline. Strongest associations were in the MCI group with a panel of 10 proteins predicting progression to AD (accuracy 87%, sensitivity 85% and specificity 88%)."
What matters to the patient is the probability that, if they come out positive when tested, they will actually get dementia. The Guardian quoted Dr James Pickett, head of research at the Alzheimer’s Society, as saying
"These 10 proteins can predict conversion to dementia with less than 90% accuracy, meaning one in 10 people would get an incorrect result."
That statement simply isn’t right (or, at least, it’s very misleading). The proper way to work out the relevant number has been explained in many places -I did it recently on my blog.
The easiest way to work it out is to make a tree diagram. The diagram is like that previously discussed here, but with a sensitivity of 85% and a specificity of 88%, as specified in the paper.
In order to work out the number we need, we have to specify the true prevalence of people who will develop dementia, in the population being tested. In the tree diagram, this has been taken as 10%. The diagram shows that, out of 1000 people tested, there are 85 + 108 = 193 with a positive test result. Out ot this 193, rather more than half (108) are false positives, so if you test positive there is a 56% chance that it’s a false alarm (108/193 = 0.56). A false discovery rate of 56% is far too high for a good test.
This figure of 56% seems to be the basis for a rather good post by NHS Choices with the title “Blood test for Alzheimer’s ‘no better than coin toss’
If the prevalence were taken as 5% (a value that’s been given for the over-60 age group) that fraction of false alarms would rise to a disastrous 73%.
How are these numbers related to the claim that the test is "87% accurate"? That claim was parroted in most of the media reports, and it is why Dr Pickett said "one in 10 people would get an incorrect result".
The paper itself didn’t define "accuracy" anywhere, and I wasn’t familiar with the term in this context (though Stephen Senn pointed out that it is mentioned briefly in the Wiikipedia entry for Sensitivity and Specificity). The senior author confirmed that "accuracy" means the total fraction of tests, positive or negative, that give the right result. We see from the tree diagram that, out of 1000 tests, there are 85 correct positive tests and 792 correct negative tests, so the accuracy (with a prevalence of 0.1) is (85 + 792)/1000 = 88%, close to the value that’s cited in the paper.
Accuracy, defined in this way, seems to me not to be a useful measure at all. It conflates positive and negative results and they need to be kept separate to understand the problem. Inspection of the tree diagram shows that it can be expressed algebraically as
accuracy = (sensitivity × prevalence) + (specificity × (1 − prevalence))
It is therefore merely a weighted mean of sensitivity and specificity (weighted by the prevalence). With the numbers in this case, it varies from 0.88 (when prevalence = 0) to 0.85 (when prevalence = 1). Thus it will inevitably give a much more flattering view of the test than the false discovery rate.
No doubt, it is too much to expect that a hard-pressed journalist would have time to figure this out, though it isn’t clear that they wouldn’t have time to contact someone who understands it. But it is clear that it should have been explained in the press release. It wasn’t.
In fact, reading the paper shows that the test was not being proposed as a screening test for dementia at all. It was proposed as a way to select patients for entry into clinical trials. The population that was being tested was very different from the general population of old people, being patients who come to memory clinics in trials centres (the potential trials population)
How best to select patients for entry into clinical trials is a matter of great interest to people who are running trials. It is of very little interest to the public. So all this confusion could have been avoided if Kings had refrained from issuing a press release at all, for a paper like this.
I guess universities think that PR is more important than accuracy.
That’s a bad mistake in an age when pretentions get quickly punctured on the web.
This post first appeared on the Sense about Science web site.
The Higher Education Funding Council England (HEFCE) gives money to universities. The allocation that a university gets depends strongly on the periodical assessments of the quality of their research. Enormous amounts if time, energy and money go into preparing submissions for these assessments, and the assessment procedure distorts the behaviour of universities in ways that are undesirable. In the last assessment, four papers were submitted by each principal investigator, and the papers were read.
In an effort to reduce the cost of the operation, HEFCE has been asked to reconsider the use of metrics to measure the performance of academics. The committee that is doing this job has asked for submissions from any interested person, by June 20th.
This post is a draft for my submission. I’m publishing it here for comments before producing a final version for submission.
Draft submission to HEFCE concerning the use of metrics.
I’ll consider a number of different metrics that have been proposed for the assessment of the quality of an academic’s work.
Impact factors
The first thing to note is that HEFCE is one of the original signatories of DORA (http://am.ascb.org/dora/ ). The first recommendation of that document is
:"Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions"
.Impact factors have been found, time after time, to be utterly inadequate as a way of assessing individuals, e.g. [1], [2]. Even their inventor, Eugene Garfield, says that. There should be no need to rehearse yet again the details. If HEFCE were to allow their use, they would have to withdraw from the DORA agreement, and I presume they would not wish to do this.
Article citations
Citation counting has several problems. Most of them apply equally to the H-index.
- Citations may be high because a paper is good and useful. They equally may be high because the paper is bad. No commercial supplier makes any distinction between these possibilities. It would not be in their commercial interests to spend time on that, but it’s critical for the person who is being judged. For example, Andrew Wakefield’s notorious 1998 paper, which gave a huge boost to the anti-vaccine movement had had 758 citations by 2012 (it was subsequently shown to be fraudulent).
- Citations take far too long to appear to be a useful way to judge recent work, as is needed for judging grant applications or promotions. This is especially damaging to young researchers, and to people (particularly women) who have taken a career break. The counts also don’t take into account citation half-life. A paper that’s still being cited 20 years after it was written clearly had influence, but that takes 20 years to discover,
- The citation rate is very field-dependent. Very mathematical papers are much less likely to be cited, especially by biologists, than more qualitative papers. For example, the solution of the missed event problem in single ion channel analysis [3,4] was the sine qua non for all our subsequent experimental work, but the two papers have only about a tenth of the number of citations of subsequent work that depended on them.
- Most suppliers of citation statistics don’t count citations of books or book chapters. This is bad for me because my only work with over 1000 citations is my 105 page chapter on methods for the analysis of single ion channels [5], which contained quite a lot of original work. It has had 1273 citations according to Google scholar but doesn’t appear at all in Scopus or Web of Science. Neither do the 954 citations of my statistics text book [6]
- There are often big differences between the numbers of citations reported by different commercial suppliers. Even for papers (as opposed to book articles) there can be a two-fold difference between the number of citations reported by Scopus, Web of Science and Google Scholar. The raw data are unreliable and commercial suppliers of metrics are apparently not willing to put in the work to ensure that their products are consistent or complete.
- Citation counts can be (and already are being) manipulated. The easiest way to get a large number of citations is to do no original research at all, but to write reviews in popular areas. Another good way to have ‘impact’ is to write indecisive papers about nutritional epidemiology. That is not behaviour that should command respect.
- Some branches of science are already facing something of a crisis in reproducibility [7]. One reason for this is the perverse incentives which are imposed on scientists. These perverse incentives include the assessment of their work by crude numerical indices.
- “Gaming” of citations is easy. (If students do it it’s called cheating: if academics do it is called gaming.) If HEFCE makes money dependent on citations, then this sort of cheating is likely to take place on an industrial scale. Of course that should not happen, but it would (disguised, no doubt, by some ingenious bureaucratic euphemisms).
- For example, Scigen is a program that generates spoof papers in computer science, by stringing together plausible phases. Over 100 such papers have been accepted for publication. By submitting many such papers, the authors managed to fool Google Scholar in to awarding the fictitious author an H-index greater than that of Albert Einstein http://en.wikipedia.org/wiki/SCIgen
- The use of citation counts has already encouraged guest authorships and such like marginally honest behaviour. There is no way to tell with an author on a paper has actually made any substantial contribution to the work, despite the fact that some journals ask for a statement about contribution.
- It has been known for 17 years that citation counts for individual papers are not detectably correlated with the impact factor of the journal in which the paper appears [1]. That doesn’t seem to have deterred metrics enthusiasts from using both. It should have done.
Given all these problems, it’s hard to see how citation counts could be useful to the REF, except perhaps in really extreme cases such as papers that get next to no citations over 5 or 10 years.
The H-index
This has all the disadvantages of citation counting, but in addition it is strongly biased against young scientists, and against women. This makes it not worth consideration by HEFCE.
Altmetrics
Given the role given to “impact” in the REF, the fact that altmetrics claim to measure impact might make them seem worthy of consideration at first sight. One problem is that the REF failed to make a clear distinction between impact on other scientists is the field and impact on the public.
Altmetrics measures an undefined mixture of both sorts if impact, with totally arbitrary weighting for tweets, Facebook mentions and so on. But the score seems to be related primarily to the trendiness of the title of the paper. Any paper about diet and health, however poor, is guaranteed to feature well on Twitter, as will any paper that has ‘penis’ in the title.
It’s very clear from the examples that I’ve looked at that few people who tweet about a paper have read more than the title. See Why you should ignore altmetrics and other bibliometric nightmares [8].
In most cases, papers were promoted by retweeting the press release or tweet from the journal itself. Only too often the press release is hyped-up. Metrics not only corrupt the behaviour of academics, but also the behaviour of journals. In the cases I’ve examined, reading the papers revealed that they were particularly poor (despite being in glamour journals): they just had trendy titles [8].
There could even be a negative correlation between the number of tweets and the quality of the work. Those who sell altmetrics have never examined this critical question because they ignore the contents of the papers. It would not be in their commercial interests to test their claims if the result was to show a negative correlation. Perhaps the reason why they have never tested their claims is the fear that to do so would reduce their income.
Furthermore you can buy 1000 retweets for $8.00 http://followers-and-likes.com/twitter/buy-twitter-retweets/ That’s outright cheating of course, and not many people would go that far. But authors, and journals, can do a lot of self-promotion on twitter that is totally unrelated to the quality of the work.
It’s worth noting that much good engagement with the public now appears on blogs that are written by scientists themselves, but the 3.6 million views of my blog do not feature in altmetrics scores, never mind Scopus or Web of Science. Altmetrics don’t even measure public engagement very well, never mind academic merit.
Evidence that metrics measure quality
Any metric would be acceptable only if it measured the quality of a person’s work. How could that proposition be tested? In order to judge this, one would have to take a random sample of papers, and look at their metrics 10 or 20 years after publication. The scores would have to be compared with the consensus view of experts in the field. Even then one would have to be careful about the choice of experts (in fields like alternative medicine for example, it would be important to exclude people whose living depended on believing in it). I don’t believe that proper tests have ever been done (and it isn’t in the interests of those who sell metrics to do it).
The great mistake made by almost all bibliometricians is that they ignore what matters most, the contents of papers. They try to make inferences from correlations of metric scores with other, equally dubious, measures of merit. They can’t afford the time to do the right experiment if only because it would harm their own “productivity”.
The evidence that metrics do what’s claimed for them is almost non-existent. For example, in six of the ten years leading up to the 1991 Nobel prize, Bert Sakmann failed to meet the metrics-based publication target set by Imperial College London, and these failures included the years in which the original single channel paper was published [9] and also the year, 1985, when he published a paper [10] that was subsequently named as a classic in the field [11]. In two of these ten years he had no publications whatsoever. See also [12].
Application of metrics in the way that it’s been done at Imperial and also at Queen Mary College London, would result in firing of the most original minds.
Gaming and the public perception of science
Every form of metric alters behaviour, in such a way that it becomes useless for its stated purpose. This is already well-known in economics, where it’s know as Goodharts’s law http://en.wikipedia.org/wiki/Goodhart’s_law “"When a measure becomes a target, it ceases to be a good measure”. That alone is a sufficient reason not to extend metrics to science. Metrics have already become one of several perverse incentives that control scientists’ behaviour. They have encouraged gaming, hype, guest authorships and, increasingly, outright fraud [13].
The general public has become aware of this behaviour and it is starting to do serious harm to perceptions of all science. As long ago as 1999, Haerlin & Parr [14] wrote in Nature, under the title How to restore Public Trust in Science,
“Scientists are no longer perceived exclusively as guardians of objective truth, but also as smart promoters of their own interests in a media-driven marketplace.”
And in January 17, 2006, a vicious spoof on a Science paper appeared, not in a scientific journal, but in the New York Times. See https://www.dcscience.net/?p=156
The use of metrics would provide a direct incentive to this sort of behaviour. It would be a tragedy not only for people who are misjudged by crude numerical indices, but also a tragedy for the reputation of science as a whole.
Conclusion
There is no good evidence that any metric measures quality, at least over the short time span that’s needed for them to be useful for giving grants or deciding on promotions). On the other hand there is good evidence that use of metrics provides a strong incentive to bad behaviour, both by scientists and by journals. They have already started to damage the public perception of science of the honesty of science.
The conclusion is obvious. Metrics should not be used to judge academic performance.
What should be done?
If metrics aren’t used, how should assessment be done? Roderick Floud was president of Universities UK from 2001 to 2003. He’s is nothing if not an establishment person. He said recently:
“Each assessment costs somewhere between £20 million and £100 million, yet 75 per cent of the funding goes every time to the top 25 universities. Moreover, the share that each receives has hardly changed during the past 20 years.
It is an expensive charade. Far better to distribute all of the money through the research councils in a properly competitive system.”
The obvious danger of giving all the money to the Research Councils is that people might be fired solely because they didn’t have big enough grants. That’s serious -it’s already happened at Kings College London, Queen Mary London and at Imperial College. This problem might be ameliorated if there were a maximum on the size of grants and/or on the number of papers a person could publish, as I suggested at the open data debate. And it would help if univerities appointed vice-chancellors with a better long term view than most seem to have at the moment.
Aggregate metrics? It’s been suggested that the problems are smaller if one looks at aggregated metrics for a whole department. rather than the metrics for individual people. Clearly looking at departments would average out anomalies. The snag is that it wouldn’t circumvent Goodhart’s law. If the money depended on the aggregate score, it would still put great pressure on universities to recruit people with high citations, regardless of the quality of their work, just as it would if individuals were being assessed. That would weigh against thoughtful people (and not least women).
The best solution would be to abolish the REF and give the money to research councils, with precautions to prevent people being fired because their research wasn’t expensive enough. If politicians insist that the "expensive charade" is to be repeated, then I see no option but to continue with a system that’s similar to the present one: that would waste money and distract us from our job.
1. Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. British Medical Journal 314: 498-502. [Download pdf]
2. Colquhoun D (2003) Challenging the tyranny of impact factors. Nature 423: 479. [Download pdf]
3. Hawkes AG, Jalali A, Colquhoun D (1990) The distributions of the apparent open times and shut times in a single channel record when brief events can not be detected. Philosophical Transactions of the Royal Society London A 332: 511-538. [Get pdf]
4. Hawkes AG, Jalali A, Colquhoun D (1992) Asymptotic distributions of apparent open times and shut times in a single channel record allowing for the omission of brief events. Philosophical Transactions of the Royal Society London B 337: 383-404. [Get pdf]
5. Colquhoun D, Sigworth FJ (1995) Fitting and statistical analysis of single-channel records. In: Sakmann B, Neher E, editors. Single Channel Recording. New York: Plenum Press. pp. 483-587.
6. David Colquhoun on Google Scholar. Available: http://scholar.google.co.uk/citations?user=JXQ2kXoAAAAJ&hl=en17-6-2014
7. Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2: e124.[full text]
8. Colquhoun D, Plested AJ Why you should ignore altmetrics and other bibliometric nightmares. Available: https://www.dcscience.net/?p=6369
9. Neher E, Sakmann B (1976) Single channel currents recorded from membrane of denervated frog muscle fibres. Nature 260: 799-802.
10. Colquhoun D, Sakmann B (1985) Fast events in single-channel currents activated by acetylcholine and its analogues at the frog muscle end-plate. J Physiol (Lond) 369: 501-557. [Download pdf]
11. Colquhoun D (2007) What have we learned from single ion channels? J Physiol 581: 425-427.[Download pdf]
12. Colquhoun D (2007) How to get good science. Physiology News 69: 12-14. [Download pdf] See also https://www.dcscience.net/?p=182
13. Oransky, I. Retraction Watch. Available: http://retractionwatch.com/18-6-2014
14. Haerlin B, Parr D (1999) How to restore public trust in science. Nature 400: 499. 10.1038/22867 [doi].[Get pdf]
Follow-up
Some other posts on this topic
Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation
Gaming Google Scholar Citations, Made Simple and Easy
Manipulating Google Scholar Citations and Google Scholar Metrics: simple, easy and tempting
Driving Altmetrics Performance Through Marketing
Death by Metrics (October 30, 2013)
Not everything that counts can be counted
Using metrics to assess research quality By David Spiegelhalter “I am strongly against the suggestion that peer–review can in any way be replaced by bibliometrics”
1 July 2014
My brilliant statistical colleague, Alan Hawkes, not only laid the foundations for single molecule analysis (and made a career for me) . Before he got into that, he wrote a paper, Spectra of some self-exciting and mutually exciting point processes, (Biometrika 1971). In that paper he described a sort of stochastic process now known as a Hawkes process. In the simplest sort of stochastic process, the Poisson process, events are independent of each other. In a Hawkes process, the occurrence of an event affects the probability of another event occurring, so, for example, events may occur in clusters. Such processes were used for many years to describe the occurrence of earthquakes. More recently, it’s been noticed that such models are useful in finance, marketing, terrorism, burglary, social media, DNA analysis, and to describe invasive banana trees. The 1971 paper languished in relative obscurity for 30 years. Now the citation rate has shot threw the roof.
The papers about Hawkes processes are mostly highly mathematical. They are not the sort of thing that features on twitter. They are serious science, not just another ghastly epidemiological survey of diet and health. Anybody who cites papers of this sort is likely to be a real scientist. The surge in citations suggests to me that the 1971 paper was indeed an important bit of work (because the citations will be made by serious people). How does this affect my views about the use of citations? It shows that even highly mathematical work can achieve respectable citation rates, but it may take a long time before their importance is realised. If Hawkes had been judged by citation counting while he was applying for jobs and promotions, he’d probably have been fired. If his department had been judged by citations of this paper, it would not have scored well. It takes a long time to judge the importance of a paper and that makes citation counting almost useless for decisions about funding and promotion.
Stop press. Financial report casts doubt on Trainor’s claims
Science has a big problem. Most jobs are desperately insecure. It’s hard to do long term thorough work when you don’t know whether you’ll be able to pay your mortgage in a year’s time. The appalling career structure for young scientists has been the subject of much writing by the young (e.g. Jenny Rohn) and the old, e.g Bruce Alberts. Peter Lawrence (see also Real Lives and White Lies in the Funding of Scientific Research, and by me.
Until recently, this problem was largely restricted to post-doctoral fellows (postdocs). They already have PhDs and they are the people who do most of the experiments. Often large numbers of them work for a single principle investigator (PI). The PI spends most of his her time writing grant applications and traveling the world to hawk the wares of his lab. They also (to variable extents) teach students and deal with endless hassle from HR.
The salaries of most postdocs are paid from grants that last for three or sometimes five years. If that grant doesn’t get renewed. they are on the streets.
Universities have come to exploit their employees almost as badly as Amazon does.
The periodical research assessments not only waste large amounts of time and money, but they have distorted behaviour. In the hope of scoring highly, they recruit a lot of people before the submission, but as soon as that’s done with, they find that they can’t afford all of them, so some get cast aside like worn out old boots. Universities have allowed themselves to become dependent on "soft money" from grant-giving bodies. That strikes me as bad management.
The situation is even worse in the USA where most teaching staff rely on research grants to pay their salaries.
I have written three times about the insane methods that are being used to fire staff at Queen Mary College London (QMUL).
Is Queen Mary University of London trying to commit scientific suicide? (June 2012)
Queen Mary, University of London in The Times. Does Simon Gaskell care? (July 2012) and a version of it appeared th The Times (Thunderer column)
In which Simon Gaskell, of Queen Mary, University of London, makes a cock-up (August 2012)
The ostensible reason given there was to boost its ratings in university rankings. Their vice-chancellor, Simon Gaskell, seems to think that by firing people he can produce a university that’s full of Nobel prize-winners. The effect, of course, is just the opposite. Treating people like pawns in a game makes the good people leave and only those who can’t get a job with a better employer remain. That’s what I call bad management.
At QMUL people were chosen to be fired on the basis of a plain silly measure of their publication record, and by their grant income. That was combined with terrorisation of any staff who spoke out about the process (more on that coming soon).
Kings College London is now doing the same sort of thing. They have announced that they’ll fire 120 of the 777 staff in the schools of medicine and biomedical sciences, and the Institute of Psychiatry. These are humans, with children and mortgages to pay. One might ask why they were taken on the first place, if the university can’t afford them. That’s simply bad financial planning (or was it done in order to boost their Research Excellence submission?).
Surely it’s been obvious, at least since 2007, that hard financial times were coming, but that didn’t dent the hubris of the people who took an so many staff. HEFCE has failed to find a sensible way to fund universities. The attempt to separate the funding of teaching and research has just led to corruption.
The way in which people are to be chosen for the firing squad at Kings is crude in the extreme. If you are a professor at the Institute of Psychiatry then, unless you do a lot of teaching, you must have a grant income of at least £200,000 per year. You can read all the details in the Kings’ “Consultation document” that was sent to all employees. It’s headed "CONFIDENTIAL – Not for further circulation". Vice-chancellors still don’t seem to have realised that it’s no longer possible to keep things like this secret. In releasing it, I take ny cue from George Orwell.
"Journalism is printing what someone else does not want printed: everything else is public relations.”
There is no mention of the quality of your research, just income. Since in most sorts of research, the major cost is salaries, this rewards people who take on too many employees. Only too frequently, large groups are the ones in which students and research staff get the least supervision, and which bangs per buck are lowest. The university should be rewarding people who are deeply involved in research themselves -those with small groups. Instead, they are doing exactly the opposite.
Women are, I’d guess, less susceptible to the grandiosity of the enormous research group, so no doubt they will suffer disproportionately. PhD students will also suffer if their supervisor is fired while they are halfway through their projects.
An article in Times Higher Education pointed out
"According to the Royal Society’s 2010 report The Scientific Century: Securing our Future Prosperity, in the UK, 30 per cent of science PhD graduates go on to postdoctoral positions, but only around 4 per cent find permanent academic research posts. Less than half of 1 per cent of those with science doctorates end up as professors."
The panel that decides whether you’ll be fired consists of Professor Sir Robert Lechler, Professor Anne Greenough, Professor Simon Howell, Professor Shitij Kapur, Professor Karen O’Brien, Chris Mottershead, Rachel Parr & Carol Ford. If they had the slightest integrity, they’d refuse to implement such obviously silly criteria.
Universities in general. not only Kings and QMUL have become over-reliant on research funders to enhance their own reputations. PhD students and research staff are employed for the benefit of the university (and of the principle investigator), not for the benefit of the students or research staff, who are treated as expendable cost units, not as humans.
One thing that we expect of vice-chancellors is sensible financial planning. That seems to have failed at Kings. One would also hope that they would understand how to get good science. My only previous encounter with Kings’ vice chancellor, Rick Trainor, suggests that this is not where his talents lie. While he was president of the Universities UK (UUK), I suggested to him that degrees in homeopathy were not a good idea. His response was that of the true apparatchik.
“. . . degree courses change over time, are independently assessed for academic rigour and quality and provide a wider education than the simple description of the course might suggest”
That is hardly a response that suggests high academic integrity.
The students’ petition is on Change.org.
Follow-up
The problems that are faced in the UK are very similar to those in the USA. They have been described with superb clarity in “Rescuing US biomedical research from its systemic flaws“, This article, by Bruce Alberts, Marc W. Kirschner, Shirley Tilghman, and Harold Varmus, should be read by everyone. They observe that ” . . . little has been done to reform the system, primarily because it continues to benefit more established and hence more influential scientists”. I’d be more impressed by the senior people at Kings if they spent time trying to improve the system rather than firing people because their research is not sufficiently expensive.
10 June 2014
Progress on the cull, according to an anonymous correspondent
“The omnishambles that is KCL management
1) We were told we would receive our orange (at risk) or green letters (not at risk, this time) on Thursday PM 5th June as HR said that it’s not good to get bad news on a Friday!
2) We all got a letter on Friday that we would not be receiving our letters until Monday, so we all had a tense weekend
3) I finally got my letter on Monday, in my case it was “green” however a number of staff who work very hard at KCL doing teaching and research are “orange”, un bloody believable
As you can imagine the moral at King’s has dropped through the floor”
18 June 2014
Dorothy Bishop has written about the Trainor problem. Her post ends “One feels that if KCL were falling behind in a boat race, they’d respond by throwing out some of the rowers”.
The students’ petition can be found on the #KCLHealthSOS site. There is a reply to the petition, from Professor Sir Robert Lechler, and a rather better written response to it from students. Lechler’s response merely repeats the weasel words, and it attacks a few straw men without providing the slightest justification for the criteria that are being used to fire people. One can’t help noticing how often knighthoods go too the best apparatchiks rather than the best scientists.
14 July 2014
A 2013 report on Kings from Standard & Poor’s casts doubt on Trainor’s claims
Download the report from Standard and Poor’s Rating Service
A few things stand out.
- KCL is in a strong financial position with lower debt than other similar Universities and cash reserves of £194 million.
- The report says that KCL does carry some risk into the future especially that related to its large capital expansion program.
- The report specifically warns KCL over the consequences of any staff cuts. Particularly relevant are the following quotations
- Page p3 “Further staff-cost curtailment will be quite difficult …pressure to maintain its academic and non-academic service standards will weigh on its ability to cut costs further.”
- page 4 The report goes on to say (see the section headed outlook, especially the final paragraph) that any decrease in KCL’s academic reputation (e.g. consequent on staff cuts) would be likely to impair its ability to attract overseas students and therefore adversely affect its financial position.
- page 10 makes clear that KCL managers are privately aiming at 10% surplus, above the 6% operating surplus they talk about with us. However, S&P considers that ‘ambitious’. In other words KCL are shooting for double what a credit rating agency considers realistic.
One can infer from this that
- what staff have been told about the cuts being an immediate necessity is absolute nonsense
- KCL was warned against staff cuts by a credit agency
- the main problem KCL has is its overambitious building policy
- KCL is implementing a policy (staff cuts) which S & P warned against as they predict it may result in diminishing income.
What on earth is going on?
16 July 2014
I’ve been sent yet another damning document. The BMA’s response to Kings contains some numbers that seem to have escaped the attention of managers at Kings.
10 April 2015
King’s draft performance management plan for 2015
This document has just come to light (the highlighting is mine).
It’s labelled as "released for internal consultation". It seems that managers are slow to realise that it’s futile to try to keep secrets.
The document applies only to Institute of Psychiatry, Psychology and Neuroscience at King’s College London: "one of the global leaders in the fields" -the usual tedious blah that prefaces every document from every university.
It’s fascinating to me that the most cruel treatment of staff so often seems to arise in medical-related areas. I thought psychiatrists, of all people, were meant to understand people, not to kill them.
This document is not quite as crude as Imperial’s assessment, but it’s quite bad enough. Like other such documents, it pretends that it’s for the benefit of its victims. In fact it’s for the benefit of willy-waving managers who are obsessed by silly rankings.
Here are some of the sillier bits.
"The Head of Department is also responsible for ensuring that aspects of reward/recognition and additional support that are identified are appropriately followed through"
And, presumably, for firing people, but let’s not mention that.
"Academics are expected to produce original scientific publications of the highest quality that will significantly advance their field."
That’s what everyone has always tried to do. It can’t be compelled by performance managers. A large element of success is pure luck. That’s why they’re called experiments.
" However, it may take publications 12-18 months to reach a stable trajectory of citations, therefore, the quality of a journal (impact factor) and the judgment of knowledgeable peers can be alternative indicators of excellence."
It can also take 40 years for work to be cited. And there is little reason to believe that citations, especially those within 12-18 months, measure quality. And it is known for sure that "the quality of a journal (impact factor)" does not correlate with quality (or indeed with citations).
Later we read
"H Index and Citation Impact: These are good objective measures of the scientific impact of
publications"
NO, they are simply not a measure of quality (though this time they say “impact” rather than “excellence”).
The people who wrote that seem to be unaware of the most basic facts about science.
Then
"Carrying out high quality scientific work requires research teams"
Sometimes it does, sometimes it doesn’t. In the past the best work has been done by one or two people. In my field, think of Hodgkin & Huxley, Katz & Miledi or Neher & Sakmann. All got Nobel prizes. All did the work themselves. Performance managers might well have fired them before they got started.
By specifying minimum acceptable group sizes, King’s are really specifying minimum acceptable grant income, just like Imperial and Warwick. Nobody will be taken in by the thin attempt to disguise it.
The specification that a professor should have "Primary supervision of three or more PhD students, with additional secondary supervision." is particularly iniquitous. Everyone knows that far too many PhDs are being produced for the number of jobs that are available. This stipulation is not for the benefit of the young. It’s to ensure a supply of cheap labour to churn out more papers and help to lift the university’s ranking.
The document is not signed, but the document properties name its author. But she’s not a scientist and is presumably acting under orders, so please don’t blame her for this dire document. Blame the vice-chancellor.
Performance management is a direct incentive to do shoddy short-cut science.
No wonder that The Economist says "scientists are doing too much trusting and not enough verifying—to the detriment of the whole of science, and of humanity".
Feel ashamed.
This is a web version of a review of Peter Gotzsche’s book. It appeared in the April 2014 Healthwatch Newsletter. Read the whole newsletter. It has lots of good stuff. Their newsletters are here. Healthwatch has been exposing quackery since 1989. Their very first newsletter is still relevant.
|
Most new drugs and vaccines are developed by the pharmaceutical industry. The industry has produced huge benefits for mankind. But since the Thatcherite era it has come to be dominated by marketing people who appear to lack any conscience. That’s what gave rise to the Alltrials movement. It was founded in January 2013 with the aim of ensuring that all past and present clinical trials are registered before they start and that and their results are published The industry has been dragged, kicking and screaming, towards a new era of transparency, with two of the worst offenders, GSK and Roche, now promising to release all data. Let’s hope this is the beginning of real open science. |
|
This version is not quite identical with the published version in which several changes were enforced by Healthwatch’s legal adviser. They weren’t very big changes, but here is the original.
Deadly Medicines and Organised Crime
By Peter Gøtzsche, reviewed by David Colquhoun
Published by Radcliffe Publishing Ltd on 1 August 2013. RRP £24.99 (320 pages, paperback)
ISBN-10: 1846198844 ISBN-13: 978-1846198847
As someone who has spent a lifetime teaching pharmacology, this book is a bitter pill to swallow. It makes Goldacre’s Bad Pharma seem quite mild.
In fairness, the bits of pharmacology that I’ve taught concern mostly drugs that do work quite well. Things like neuromuscular blocking agents, local anaesthetics, general anaesthetics, anticoagulants, cardiac glycosides and thyroid drugs all do pretty much what is says on the label.
Peter Gøtzsche is nothing if not evidence man. He directs the Nordic Cochrane group, and he talks straight. His book is about drugs that don’t work as advertised. There is no doubt whatsoever that the pharmaceutical industry has behaved very badly indeed in the last couple of decades. You don’t have to take my word for it, nor Peter Gotzche’s, nor Ben Goldacre’s. They have told us about it themselves. Not voluntarily of course, but in internal emails that have been revealed during court proceedings, and from whistleblowers.
Peter Rost was vice president marketing for the huge pharmaceutical company, Pfizer, until he was fired after the company failed to listen to his complaints about illegal marketing of human growth hormone as an anti-ageing drug. After this he said:
“It is scary how many similarities there are between this industry and the mob. The mob makes obscene amounts of money, as does this industry. The side effects of organized crime are killings and deaths, and the side effects are the same in this industry. The mob bribes politicians and others, and so does the drug industry … “
The pharmaceutical industry is the biggest defrauder of the US federal government under the False Claims Act. Roche led a cartel that, according to the US Justice Department’s antitrust division, was the most pervasive and harmful criminal antitrust conspiracy ever uncovered. Multibillion dollar fines have been levied on all of the big companies (almost all in the USA, other countries have been supine), though the company’s profits are so huge they are regarded as marketing expenses.
It’s estimated that adverse effects of drugs kill more people than anything but cancer and heart disease, roughly half as many as cigarettes. This horrifying statistic is announced at the beginning of the book, though you have to wait until Chapter 21 to find the data. I’d have liked to see a more critical discussion of the problems of causality in deciding why someone died, which are just as big as those in deciding why somebody recovered. Nevertheless, nobody seems to deny that the numbers who are killed by their treatments are alarmingly high.
Gøtzsche’s book deals with a wide range of drugs that don’t do what it says on the label, but which have made fortunes because of corruption of the scientific process. These include non-steroidal anti-inflammatory drugs (NSAIDs), an area described as “a horror story filled with extravagant claims, bending of the rules, regulatory inaction, . . .”. Other areas where there has been major misbehaviour include diabetes (Avandia), and the great Tamiflu scandal. and the great Tamiflu scandal. It took five years of pressure before Roche released the hidden data about Tamiflu trials. It barely works. Goldacre commented “government’s Tamiflu stockpile wouldn’t have done us much good in the event of a flu epidemic”
But the worst single area is psychiatry.
Two of the chapters in the book deal with psychiatry. Nobody has the slightest idea how the brain works (don’t believe the neuroscience hype) or what causes depression or psychosis. Treatments are no more than guesses and none of them seems to work very well.
The problems with the SSRI antidepressant, paroxetine (Seroxat in UK, Paxil in USA) were brought to public attention, not by a regulator, but by a BBC Panorama television programme. The programme revealed that a PR company, which worked for GSK, had written
"Originally we had planned to do extensive media relations surrounding this study until we actually viewed the results. Essentially the study did not really show it was effective in treating adolescent depression, which is not something we want to publicise."
This referred to the now-notorious study 329. It was intended to show that paroxetine should be recommended for adolescent depression. The paper that eventually appeared in 2001 grossly misrepresented the results. The conclusions stated “Paroxetine is generally well tolerated and effective for major depression in adolescents”, despite the fact that GSK already knew this wasn’t true. The first author of this paper was Martin Keller, chair of psychiatry at Brown University, RI, with 21 others.
But the paper wasn’t written by them, but by ghost authors working for GSK. Keller admitted that he hadn’t checked the results properly.
That’s not all. Gøtzsche comments thus.
“Keller is some character. He double- billed his travel expenses, which were reimbursed both by his university and the drug sponsor. Further, the Massachusetts Department of Mental Health had paid Brown’s psychiatry department, which Keller chaired, hundreds of thousands of dollars to fund research that wasn’t being conducted. Keller himself received hundreds of thousands of dollars from drug companies every year that he didn’t disclose.”
His department received $50 million in research funding. Brown University has never admitted that there was a problem. It still boasts about this infamous paper
The extent of corruption at Brown University rivals the mob.
The infamous case of Richard Eastell at Sheffield university is no better. He admitted in print to lying about who’d seen the data. The university did nothing but fire the whistleblower.
Another trial, study 377, also showed that paroxetine didn’t work. GSK suppressed it.
“There are no plans to publish data from Study 377” (Seroxat/Paxil Adolescent Depression. Position piece on the phase III clinical studies. GlaxoSmithKline document. 1998 Oct.)
Where were the regulatory agencies during all this? The MHRA did ban use of paroxetine in adolescents in 2003, but their full investigation didn’t report until 2008. It came to much the same conclusions as the TV programme six years earlier about the deceit. But despite that, no prosecution was brought. GSK got away with a deferential rap on the knuckles.
Fiona Godlee (editor of the BMJ, which had turned down the paper) commented
“We shouldn’t have to rely on investigative journalists to ask the difficult questions”
Now we can add bloggers to that list of people who ask difficult questions. The scam operated by the University of Wales, in ‘validating’ external degrees was revealed by my blog and by BBC TV Wales. The Quality Assurance Agency came in only at the last moment. Regulators regularly fail to regulate.
|
Despite all this, the current MHRA learning module on SSRIs contains little hint that SSRIs simply don’t work for mild or moderate depression. Neither does the current NICE guidance. Some psychiatrists still think they do work, despite there being so many negative trials. |
 |
The psychiatrists’ narrative goes like this. You don’t expect to see improvements for many weeks (despite the fact that serotonin uptake is stopped immediately). You may get worse before you get better. And if the first sort of pill doesn’t work, try another one. That’s pretty much identical with what a homeopath will tell you. The odds are that its meaning is, wait a while and you’ll get better eventually, regardless of treatment.
It’s common to be told that they must work because when you stop taking them, you get worse. But, perhaps more likely, when you stop taking them you get withdrawal symptoms, because the treatment itself caused a chemical imbalance. Gøtzsche makes a strong case that most psychiatric drugs do more harm than good, if taken for any length of time. Marcia Angell makes a similar case in The Illusions of Psychiatry.
Gøtzsche will inevitably be accused of exaggerating. Chapter 14 ends thus.
“Merck stated only 6 months before it withdrew Vioxx that ‘MSD is fully committed to the highest standards of scientific integrity, ethics, and protection of patient’s wellbeing in our research. We have a tradition of partnership with leaders in the academic research community. Great. Let’s have some more of such ethical partnerships. They often kill our patients while everyone else prospers.
Perhaps Hells Angels should consider something similar in their PR: We are fully committed to the highest standards of integrity, ethics and protection of citizens’ well- being when we push narcotic drugs. We have a tradition of partnership with leaders in the police force”.
But the evidence is there. The book has over 900 references. Much of the wrongdoing has been laid bare by legal actions. I grieve for the state of my subject.
The wrongdoing by pharma is a disgrace.
The corruption of universities and academics is even worse, because they are meant to be our defence against commercial corruption.
All one can do is to take consolation from the fact that academics, like Gøtzsche and Goldacre, and a host of bloggers, are the people who are revealing what’s wrong. As a writer for the business magazine, Fortune, said
“For better or worse, the drug industry is going to have to get used to Dr. Peter Rost – and others like him.”
At a recent meeting I said that it was tragic that medicine, the caring profession, was also the most corrupt (though I’m happy to admit that other jobs might be as bad if offered as much money).
At present there is little transparency. There is no way that I can tell whether my doctor is taking money from pharma, data are still hidden from public scrutiny by regulatory agencies (which are stuffed with people who take pharma money) as well as by companies. Governments regard business as more important than patients. In the UK, the Government continued promotion of the fake bomb detector for many years after they’d been told it was fake. Their attitude to fake medicines is not much different. Business is business, right?
One side effect of the horrific corruption is that it’s used as a stick by the alternative medicine industry. That’s silly of them, because their business is more or less 100% mendacious marketing of ineffective treatments. At least half of pharma products really do work.
Fines are useless. Nothing will change until a few CEOs, a few professors and a few vice-chancellors spend time in jail for corruption.
Read this book. Get angry. Do something.
Follow-up
This post is now a bit out of date: there is a summary of my more recent efforts (papers, videos and pop stuff) can be found on Prof Sivilotti’s OneMol pages.
What follows is a simplified version of part of a paper that appeared as a preprint on arXiv in July. It appeared as a peer-reviewed paper on 19th November 2014, in the new Royal Society Open Science journal. If you find anything wrong, or obscure, please email me. Be vicious.
There is also a simplified version, given as a talk on Youtube..
It’s a follow-up to my very first paper, which was written in 1959 – 60, while I was a fourth year undergraduate(the history is at a recent blog). I hope this one is better.
‘”. . . before anything was known of Lydgate’s skill, the judgements on it had naturally been divided, depending on a sense of likelihood, situated perhaps in the pit of the stomach, or in the pineal gland, and differing in its verdicts, but not less valuable as a guide in the total deficit of evidence” ‘George Eliot (Middlemarch, Chap. 45)
“The standard approach in teaching, of stressing the formal definition of a p-value while warning against its misinterpretation, has simply been an abysmal failure” Sellke et al. (2001) `The American Statistician’ (55), 62–71
The last post was about screening. It showed that most screening tests are useless, in the sense that a large proportion of people who test positive do not have the condition. This proportion can be called the false discovery rate. You think you’ve discovered the condition, but you were wrong.
Very similar ideas can be applied to tests of significance. If you read almost any scientific paper you’ll find statements like “this result was statistically significant (P = 0.047)”. Tests of significance were designed to prevent you from making a fool of yourself by claiming to have discovered something, when in fact all you are seeing is the effect of random chance. In this case we define the false discovery rate as the probability that, when a test comes out as ‘statistically significant’, there is actually no real effect.
You can also make a fool of yourself by failing to detect a real effect, but this is less harmful to your reputation.
It’s very common for people to claim that an effect is real, not just chance, whenever the test produces a P value of less than 0.05, and when asked, it’s common for people to think that this procedure gives them a chance of 1 in 20 of making a fool of themselves. Leaving aside that this seems rather too often to make a fool of yourself, this interpretation is simply wrong.
The purpose of this post is to justify the following proposition.
|
If you observe a P value close to 0.05, your false discovery rate will not be 5%. It will be at least 30% and it could easily be 80% for small studies.
|
This makes slightly less startling the assertion in John Ioannidis’ (2005) article, Why Most Published Research Findings Are False. That paper caused quite a stir. It’s a serious allegation. In fairness, the title was a bit misleading. Ioannidis wasn’t talking about all science. But it has become apparent that an alarming number of published works in some fields can’t be reproduced by others. The worst offenders seem to be clinical trials, experimental psychology and neuroscience, some parts of cancer research and some attempts to associate genes with disease (genome-wide association studies). Of course the self-correcting nature of science means that the false discoveries get revealed as such in the end, but it would obviously be a lot better if false results weren’t published in the first place.
How can tests of significance be so misleading?
Tests of statistical significance have been around for well over 100 years now. One of the most widely used is Student’s t test. It was published in 1908. ‘Student’ was the pseudonym for William Sealy Gosset, who worked at the Guinness brewery in Dublin. He visited Karl Pearson’s statistics department at UCL because he wanted statistical methods that were valid for testing small samples. The example that he used in his paper was based on data from Arthur Cushny, the first holder of the chair of pharmacology at UCL (subsequently named the A.J. Clark chair, after its second holder)
The outcome of a significance test is a probability, referred to as a P value. First, let’s be clear what the P value means. It will be simpler to do that in the context of a particular example. Suppose we wish to know whether treatment A is better (or worse) than treatment B (A might be a new drug, and B a placebo). We’d take a group of people and allocate each person to take either A or B and the choice would be random. Each person would have an equal chance of getting A or B. We’d observe the responses and then take the average (mean) response for those who had received A and the average for those who had received B. If the treatment (A) was no better than placebo (B), the difference between means should be zero on average. But the variability of the responses means that the observed difference will never be exactly zero. So how big does it have to be before you discount the possibility that random chance is all you were seeing. You do the test and get a P value. Given the ubiquity of P values in scientific papers, it’s surprisingly rare for people to be able to give an accurate definition. Here it is.
|
The P value is the probability that you would find a difference as big as that observed, or a still bigger value, if in fact A and B were identical.
|
If this probability is low enough, the conclusion would be that it’s unlikely that the observed difference (or a still bigger one) would have occurred if A and B were identical, so we conclude that they are not identical, i.e. that there is a genuine difference between treatment and placebo.
This is the classical way to avoid making a fool of yourself by claiming to have made a discovery when you haven’t. It was developed and popularised by the greatest statistician of the 20th century, Ronald Fisher, during the 1920s and 1930s. It does exactly what it says on the tin. It sounds entirely plausible.
What could possibly go wrong?
Another way to look at significance tests
One way to look at the problem is to notice that the classical approach considers only what would happen if there were no real effect or, as a statistician would put it, what would happen if the null hypothesis were true. But there isn’t much point in knowing that an event is unlikely when the null hypothesis is true unless you know how likely it is when there is a real effect.
We can look at the problem a bit more realistically by means of a tree diagram, very like that used to analyse screening tests, in the previous post.
In order to do this, we need to specify a couple more things.
First we need to specify the power of the significance test. This is the probability that we’ll detect a difference when there really is one. By ‘detect a difference’ we mean that the test comes out with P < 0.05 (or whatever level we set). So it’s analogous with the sensitivity of a screening test. In order to calculate sample sizes, it’s common to set the power to 0.8 (obviously 0.99 would be better, but that would often require impracticably large samples).
The second thing that we need to specify is a bit trickier, the proportion of tests that we do in which there is a real difference. This is analogous to the prevalence of the disease in the population being tested in the screening example. There is nothing mysterious about it. It’s an ordinary probability that can be thought of as a long-term frequency. But it is a probability that’s much harder to get a value for than the prevalence of a disease.
If we were testing a series of 30C homeopathic pills, all of the pills, regardless of what it says on the label, would be identical with the placebo controls so the prevalence of genuine effects, call it P(real), would be zero. So every positive test would be a false positive: the false discovery rate would be 100%. But in real science we want to predict the false discovery rate in less extreme cases.
Suppose, for example, that we test a large number of candidate drugs. Life being what it is, most of them will be inactive, but some will have a genuine effect. In this example we’d be lucky if 10% had a real effect, i.e. were really more effective than the inactive controls. So in this case we’d set the prevalence to P(real) = 0.1.
We can now construct a tree diagram exactly as we did for screening tests.
Suppose that we do 1000 tests. In 90% of them (900 tests) there is no real effect: the null hypothesis is true. If we use P = 0.05 as a criterion for significance then, according to the classical theory, 5% of them (45 tests) will give false positives, as shown in the lower limb of the tree diagram. If the power of the test was 0.8 then we’ll detect 80% of the real differences so there will be 80 correct positive tests.
The total number of positive tests is 45 + 80 = 125, and the proportion of these that are false positives is 45/125 = 36 percent. Our false discovery rate is far bigger than the 5% that many people still believe they are attaining.
In contrast, 98% of negative tests are right (though this is less surprising because 90% of experiments really have no effect).
The equation
You can skip this section without losing much.
As in the case of screening tests, this result can be calculated from an equation. The same equation works if we substitute power for sensitivity, P(real) for prevalence, and siglev for (1 – specificity) where siglev is the cut off value for “significance”, 0.05 in our examples.
The false discovery rate (the probability that, if a “signifcant” result is found, there is actually no real effect) is given by
\[FDR = \frac{siglev\left(1-P(real)\right)}{power.P(real) + siglev\left(1-P(real)\right) }\; \]
In the example above, power = 0.8, siglev = 0.05 and P(real) = 0.1, so the false discovery rate is
\[\frac{0.05 (1-0.1)}{0.8 \times 0.1 + 0.05 (1-0.1) }\; = 0.36 \]
So 36% of “significant” results are wrong, as found in the tree diagram.
Some subtleties
The argument just presented should be quite enough to convince you that significance testing, as commonly practised, will lead to disastrous numbers of false positives. But the basis of how to make inferences is still a matter that’s the subject of intense controversy among statisticians, so what is an experimenter to do?
It is difficult to give a consensus of informed opinion because, although there is much informed opinion, there is rather little consensus. A personal view follows. Colquhoun (1970), Lectures on Biostatistics, pp 94-95.
This is almost as true now as it was when I wrote it in the late 1960s, but there are some areas of broad agreement.
There are two subtleties that cause the approach outlined above to be a bit contentious. The first lies in the problem of deciding the prevalence, P(real). You may have noticed that if the frequency of real effects were 50% rather than 10%, the approach shown in the diagram would give a false discovery rate of only 6%, little different from the 5% that’s embedded in the consciousness of most experimentalists.
But this doesn’t get us off the hook, for two reasons. For a start, there is no reason at all to think that there will be a real effect there in half of the tests that we do. Of course if P(real) were even bigger than 0.5, the false discovery rate would fall to zero, because when P(real) = 1, all effects are real and therefore all positive tests are correct.
There is also a more subtle point. If we are trying to interpret the result of a single test that comes out with a P value of, say, P = 0.047, then we should not be looking at all significant results (those with P < 0.05), but only at those tests that come out with P = 0.047. This can be done quite easily by simulating a long series of t tests, and then restricting attention to those that come out with P values between, say, 0.045 and 0.05. When this is done we find that the false discovery rate is at least 26%. That’s for the best possible case where the sample size is good (power of the test is 0.8) and the prevalence of real effects is 0.5. When, as in the tree diagram, the prevalence of real effects is 0.1, the false discovery rate is 76%. That’s enough to justify Ioannidis’ statement that most published results are wrong.
One problem with all of the approaches mentioned above was the need to guess at the prevalence of real effects (that’s what a Bayesian would call the prior probability). James Berger and colleagues (Sellke et al., 2001) have proposed a way round this problem by looking at all possible prior distributions and so coming up with a minimum false discovery rate that holds universally. The conclusions are much the same as before. If you claim to have found an effects whenever you observe a P value just less than 0.05, you will come to the wrong conclusion in at least 29% of the tests that you do. If, on the other hand, you use P = 0.001, you’ll be wrong in only 1.8% of cases. Valen Johnson (2013) has reached similar conclusions by a related argument.
A three-sigma rule
As an alternative to insisting on P < 0.001 before claiming you’ve discovered something, you could use a 3-sigma rule. In other words, insist that an effect is at least three standard deviations away from the control value (as opposed to the two standard deviations that correspond to P = 0.05).
The three sigma rule means using P= 0.0027 as your cut off. This, according to Berger’s rule, implies a false discovery rate of (at least) 4.5%, not far from the value that many people mistakenly think is achieved by using P = 0.05 as a criterion.
Particle physicists go a lot further than this. They use a 5-sigma rule before announcing a new discovery. That corresponds to a P value of less than one in a million (0.57 x 10−6). According to Berger’s rule this corresponds to a false discovery rate of (at least) around 20 per million. Of course their experiments can’t be randomised usually, so it’s as well to be on the safe side.
Underpowered experiments
All of the problems discussed so far concern the near-ideal case. They assume that your sample size is big enough (power about 0.8 say) and that all of the assumptions made in the test are true, that there is no bias or cheating and that no negative results are suppressed. The real-life problems can only be worse. One way in which it is often worse is that sample sizes are too small, so the statistical power of the tests is low.
The problem of underpowered experiments has been known since 1962, but it has been ignored. Recently it has come back into prominence, thanks in large part to John Ioannidis and the crisis of reproducibility in some areas of science. Button et al. (2013) said
“We optimistically estimate the median statistical power of studies in the neuroscience field to be between about 8% and about 31%”
This is disastrously low. Running simulated t tests shows that with a power of 0.2, not only do you have only a 20% chance of detecting a real effect, but that when you do manage to get a “significant” result there is a 76% chance that it’s a false discovery.
And furthermore, when you do find a “significant” result, the size of the effect will be over-estimated by a factor of nearly 2. This “inflation effect” happens because only those experiments that happen, by chance, to have a larger-than-average effect size will be deemed to be “significant”.
What should you do to prevent making a fool of yourself?
The simulated t test results, and some other subtleties, will be described in a paper, and/or in a future post. But I hope that enough has been said here to convince you that there are real problems in the sort of statistical tests that are universal in the literature.
The blame for the crisis in reproducibility has several sources.
One of them is the self-imposed publish-or-perish culture, which values quantity over quality, and which has done enormous harm to science.
The mis-assessment of individuals by silly bibliometric methods has contributed to this harm. Of all the proposed methods, altmetrics is demonstrably the most idiotic. Yet some vice-chancellors have failed to understand that.
Another is scientists’ own vanity, which leads to the PR department issuing disgracefully hyped up press releases.
In some cases, the abstract of a paper states that a discovery has been made when the data say the opposite. This sort of spin is common in the quack world. Yet referees and editors get taken in by the ruse (e.g see this study of acupuncture).
The reluctance of many journals (and many authors) to publish negative results biases the whole literature in favour of positive results. This is so disastrous in clinical work that a pressure group has been started; altrials.net “All Trials Registered | All Results Reported”.
Yet another problem is that it has become very hard to get grants without putting your name on publications to which you have made little contribution. This leads to exploitation of young scientists by older ones (who fail to set a good example). Peter Lawrence has set out the problems.
And, most pertinent to this post, a widespread failure to understand properly what a significance test means must contribute to the problem. Young scientists are under such intense pressure to publish, they have no time to learn about statistics.
Here are some things that can be done.
- Notice that all statistical tests of significance assume that the treatments have been allocated at random. This means that application of significance tests to observational data, e.g. epidemiological surveys of diet and health, is not valid. You can’t expect to get the right answer. The easiest way to understand this assumption is to think about randomisation tests (which should have replaced t tests decades ago, but which are still rare). There is a simple introduction in Lectures on Biostatistics (chapters 8 and 9). There are other assumptions too, about the distribution of observations, independence of measurements), but randomisation is the most important.
- Never, ever, use the word “significant” in a paper. It is arbitrary, and, as we have seen, deeply misleading. Still less should you use “almost significant”, “tendency to significant” or any of the hundreds of similar circumlocutions listed by Matthew Hankins on his Still not Significant blog.
- If you do a significance test, just state the P value and give the effect size and confidence intervals (but be aware that this is just another way of expressing the P value approach: it tells you nothing whatsoever about the false discovery rate).
- Observation of a P value close to 0.05 means nothing more than ‘worth another look’. In practice, one’s attitude will depend on weighing the losses that ensue if you miss a real effect against the loss to your reputation if you claim falsely to have made a discovery.
- If you want to avoid making a fool of yourself most of the time, don’t regard anything bigger than P < 0.001 as a demonstration that you’ve discovered something. Or, slightly less stringently, use a three-sigma rule.
Despite the gigantic contributions that Ronald Fisher made to statistics, his work has been widely misinterpreted. We must, however reluctantly, concede that there is some truth in the comment made by an astute journalist:
“The plain fact is that 70 years ago Ronald Fisher gave scientists a mathematical machine for turning baloney into breakthroughs, and °flukes into funding. It is time to pull the plug“. Robert Matthews Sunday Telegraph, 13 September 1998.
There is now a video on YouTube that attempts to explain explain simply the essential ideas. The video has now been updated. The new version has better volume and it used term ‘false positive risk’, rather than the earlier term ‘false discovery rate’, to avoid confusion with the use of the latter term in the context of multiple comparisons.
The false positive risk: a proposal concerning what to do about p-values (version 2)
Follow-up
31 March 2014 I liked Stephen Senn’s first comment on twitter (the twitter stream is storified here). He said ” I may have to write a paper ‘You may believe you are NOT a Bayesian but you’re wrong'”. I maintain that the analysis here is merely an exercise in conditional probabilities. It bears a formal similarity to a Bayesian argument, but is free of more contentious parts of the Bayesian approach. This is amplified in a comment, below.
4 April 2014
I just noticed that my first boss, Heinz Otto Schild.in his 1942 paper about the statistical analysis of 2+2 dose biological assays (written while he was interned at the beginning of the war) chose to use 99% confidence limits, rather than the now universal 95% limits. The later are more flattering to your results, but Schild was more concerned with precision than self-promotion.
woodchopper said,
June 9, 2007 at 7:45 am
I’m quite shocked. If people complain about the lack of understanding of science then they know where to look for an answer.
SomeBeans said,
June 9, 2007 at 8:03 am
I’m guessing that if we all write the Provost a letter, it won’t really be troublesome enough to make him change his mind.
This is really exactly the sort of thing a university should be supporting and encouraging, rather than censoring.
evidencebasedeating said,
June 9, 2007 at 8:44 am
Rather depressing proof-positive (in a holistic, meaningful ,empathic way)of how our previously august and independent universities increasingly pander to the lowest-common-denominator ‘science-lite’ approach amidst concerns from woo practitioners and regal missives from Charlie Boy (Ernst at Exeter springs to mind).
I wonder if the Provost took the decision unilaterally? Perhaps the university Senate should review both the case and the decision. They could take into account the dichotomy of Provost Malcolm Grant’s actions, versus his opening paragraph states his ‘vision’ of UCL – taken from the website, that states:
“UCL is an exceptional institution, with a radical tradition and a distinctive character. The university’s commitment to excellence and innovation in research and teaching is central to its vision of enriching society’s intellectual, cultural, scientific, economic, environmental and medical spheres.”
Er, so his role as Provost is to eradicate that ‘radical tradition’, ‘distinctive character’ and ‘vision of enriching society’s cultural and scientific spheres’.
But I note his Professorship is in Law, not science.
Explains a lot.
Never mind, Colquhuon’s status in his professional and public spheres is independent of UCL. Just makes me consider the organisation in a much more ambivalent manner.
Mojo said,
June 9, 2007 at 9:04 am
(Off-topic)
This is odd: when I looked at this page first thing this morning, before there were any comments on it, it displayed fine. Now the text has slipped down the left-hand side again.
jackpt said,
June 9, 2007 at 9:36 am
I missed out on all of this because I hadn’t checked his site for some time. You’d think UCL would be better than this, especially from the standpoint of precedents of which this is an appalling one. On a positive note I’m sure he could get free hosting or mirrors from places and people way out of the reach of scum trading on red clover etc. I for one would happily mirror any material under legal or informal threat from bread headed scum flogging false hopes and pseudoscience. The problem here is one of precedents, other universities may take note…
le canard noir said,
June 9, 2007 at 9:44 am
It is most important that all fellow bloggers and site owners, change their links to DC’s pages asap!
Need to get Google onto the move and make sure the pageranks for his stuff is up there again!
terryhamblin said,
June 9, 2007 at 10:53 am
This is not just any university. This is UCL. Jeremy Bentham must be turning in his box seat.
doctormonkey said,
June 9, 2007 at 11:32 am
This is a sad state of affairs
Another large institution bullied into dropping something as good and funny as DC’s blog
Personally I think they should drop the quackology BSc’s but failing that should allow parity and keep DC’s blog
Then again, I have always disliked UCL but I am sad to have my un-thinking, I’m-from-another-London-college prejudice actually supported by fact
andrew said,
June 9, 2007 at 12:21 pm
The more you look at it, the worse it gets.
Tobacco companies, anti-MMGW groups and other lobbyists frequently fire off
legal challenges against individual scientists to maintain a general climate of harrassment.
UCL’s message to the world is that their staff are easy meat, the college won’t stand by them.
From Steven Shafer’s letter on Colquhoun’s web-site:
“As a counter example, the University of California at San Francisco stood solidly behind Stanton Glantz when the cigarette industry tried to destroy him for his efforts to expose their activities. Had he agreed to ‘shoulder directly the burden’, we would never have known of the extensive research conducted by the cigarette industry over two decades that identified the health risks, and guided their extensive disinformation campaign. I would hope that Stanford University would following the UCSF example, and devote the necessary resources to defend my academic freedom, rather than the UCL example, and ask me to ‘shoulder the burden.’ “
JohnD said,
June 9, 2007 at 12:59 pm
I can’t belive that the Provost’s decision will stand. Less than a year ago, UCL signed the Magna Charta Universitatum, and bragged of it. That charter includes that, “all members of that institution’s academic community should have the freedom to work, teach and learn.”
See:www.ucl.ac.uk/news/news-articles/0609/06092601
I hope Professor Michael Worton, who signed on behalf of UCL is as uncomfortable as he should be with this.
John
j said,
June 9, 2007 at 1:29 pm
Great idea for the column: when ‘alternative’ practitioners get a website shut down by moaning about it, I think it’s important to give them as much publicity as possible as a result.
Just to add a couple of extra details: the complaint that got DC’s site moved from UCL came from Alan Lakin (the husband of Ann Walker). Walker is (or at least was) the director of New Vitality – www.newvitality.org.uk/index.htm. She also has quite a few interesting online articles on herbal medicine which come up when you google her (e.g. www.healthspan.co.uk) Given the way in which DC was forced to move his site, it might be appropriate if a few people with health/science-related blogs collaborated to post articles fisking different pieces of Walker’s work: I like the idea of a load of critical articles springing up when one is forced to move
Anyway, just going to update my blogroll link to DC’s excellent site.
Ben Goldacre said,
June 9, 2007 at 3:50 pm
dear all
please see the email from the provost that has been added above.
PK said,
June 9, 2007 at 4:20 pm
I do not find that letter remotely convincing. Sure, Colquhoun must not engage in libel, but it is hard never to (accidentally) stray into libelous territory when you are dealing with these people. If UCL is serious about academic freedom and scientific integrity, then they should fight this one.
SomeBeans said,
June 9, 2007 at 5:11 pm
Thanks for posting the letter from the provost – most illuminating.
Doesn’t give the impression of UCL helping David Colquhoun very much. I wonder if they still use his papers for their RAE’s.
What’s the Guardian policy on this type of thing? I seem to remember that they fought Jonathan Aitken and won…
jackpt said,
June 9, 2007 at 5:17 pm
The problem with the letter is is that it’s all couched in such vague terms. It seems to me that they’ve acted on the basis of something that could be libellous/in breach of copyright/etc rather than anything clear cut. If it were clear-cut there would be specific examples that he could point to. It’s the approach of a chicken because the letter is saying “we may be right but it’s not worth our trouble to fight” setting himself up as an arbiter of just causes. So if it’s not clear cut don’t expect any help from UCL. Grey areas not wanted.
Andrew Clegg said,
June 9, 2007 at 8:25 pm
I also sent a letter complaining (being another less than impressed alumnus like Dr Nicholas above). Here’s some helpful thoughts…
1. When you get a long personal reply back from the provost, it’s worth checking to see whether other people got the same reply word for word…
2. … rather than being so surprised that when you forward it to Ben and David with comments…
3. … you forget to take Prof. Grant’s email off the header and end up looking like a muppet.
But a since and well-intentioned muppet at least.
Andrew.
Andrew Clegg said,
June 9, 2007 at 8:29 pm
Err, unless that response Ben posted was his reply to my letter (just noticed the Dear Andrew at the top), not a standard form response, in which case I take back what I said about word-for-word copies, and look like slightly more of a muppet instead.
I think I need to eat something, brain not working at 100% today.
Andrew.
pv said,
June 9, 2007 at 9:41 pm
They might well have acted on the threat of a libel action and just decided to cave in. A University is primarily a business these days while aspiring to be a centre of educational excellence is either secondary or coincidental. On that basis no-one should be surprised that it is compelled to act in a way that protects the interests of its financial supporters and sponsors – namely their money – before any wider academic interests or unnecessary luxuries like freedom of speech. I know it all appears to be lacking in integrity but freedom of business comes first these days, even (or especially) the right of quacks and charlatans to do business without hindrance.
Art5 said,
June 9, 2007 at 9:52 pm
Perhaps it shouldn’t be surprising that they did just decide to cave in, but why does that entail asking DC to remove the whole blog and not just the contentious article? That looks incredibly unsupportive to me.
igb said,
June 10, 2007 at 12:43 am
One of the defining characteristics of the `management’ of public sector bodies is their utter, craven cowardice in face of things that even smell of a court case. I don’t know when it happens in their career, but your typical school headmaster, hospital manager or (it would appear) University provost regards a dog-eared piece of paper saying “Oi will zue youse for libil” as being as frightening as the jury coming back in and saying `guilty’.
Hence the rise in schools and universities being cowed by not even solicitor’s letters (which are, it should be noted, simply a letter from someone who happens to be a solicitor) but the threat of the same. If public bodies fought such cases through the courts, and then bankrupted the claimants when they lost (as they almost always would), after a year or so they and the ambulance chasers would get the hint. As things stand, public sector managers are encouraged to pay tribute, rather than spend on defence, and worse they are paying tribute to people with cardboard swords.
Those who get their fortnightly dose of poor typography (and it’s not as funny as it was, is it?) will know of `Arkell vs Pressdram’. The rest of you can google for it. UCL’s response to a threatened libel case should be `bring it on’, with a plea of justification.
The reason we know that David Irvine is a fraud and is because Deborah Lipstadt’s book, a copy of which is sat a few feet from me, was defended to the hilt by its publishers. Penguin Books have principles, and made a stand. It’s a shame that UCL appears to have a yellow stripe painted down its back where its spine used to be.
rob said,
June 10, 2007 at 6:14 am
Pitiful cowardice from an institution that claims to be a world-class university. Until it is proven that the material is actually illegal, it should be their part to stand up for academic freedom.
Moganero said,
June 10, 2007 at 7:46 am
le canard noir “Need to get Google onto the move and make sure the pageranks for his stuff is up there again!”
UCL’s webmaster could set up a permanent redirect to Prof Colquhoun’s new URL – this would send the search engines to it and they’d index the new location. Anyone trying to see the blog at the old URL would automatically see it at the new location.
TINSTAFL said,
June 10, 2007 at 9:16 am
Step on their toes until they apologize. They can wave their jargon at us and threaten libel, but they WILL NEVER ACTUALLY WANT TO BE IN COURT AND LOSE. And they want all this to happen quietly. Now that UCL has backed off, they will want to put pressure on UCL to censure the Prof. even more. And this is exactly what UCL is doing in response to a minor complaint. They are censuring him: cutting off his voice and officially rebuking his work on the site.
It may even be possible that rather than protect themselves, they have opened themselves to litigation from both sides. 1) Dumping suggest merit to the complain and 2) that UCL provided the site in the place and then took it away means then have placed the good Prof. in an unsupported/dangerous situation.
igb has the right idea. Fight them now and hard.
I sent an email to the provost and I suggest that others do so as well. Even letters from well-intentioned muppets will help (I have certainly sent my own in my time, misspellings and all!). Certainly the provost responded the original bad-intentioned muppets who made the complaint. Even if he does not read them, having Prof. Colquoun’s name in the subject line of a large number of message will lend him the support that he needs and will make the provost think a bit. I will also make a head link link from my own anti quack site to his.
I am willing to post my email if other are interested, but this may be up to Dr. Goldacre to decide if this is appropriate.
Mojo said,
June 10, 2007 at 10:06 am
le canard noir said,
(June 9, 2007 at 9:44 am) “Need to get Google onto the move and make sure the pageranks for his stuff is up there again!”
Well, as long as Google aren’t as spineless as they were in the case of Howard’s page about TAPL:
http://www.hakwright.co.uk/rants/Gillian_McKeith.html
A search of google.co.uk still brings up the message at the bottom of the page saying “In response to a legal request submitted to Google, we have removed 1 result(s) from this page. If you wish, you may read more about the request at ChillingEffects.org.”
Interestingly, while Howard’s page still appears on the first page for google.com, google.ca and google.com.au, the results for google.co.uk seem to have different rankings so that the message about the legal threat, sorry, request, now appears on the second page of results.
Mojo said,
June 10, 2007 at 10:35 am
Incidentally, in some parts of intellectual property law (trade marks, patents and registered designs, but not copyright, unfortunately) it is a tort to make an unjustified threat to sue. Perhaps a case could be made for extending this to defamation.
Dr Aust said,
June 10, 2007 at 11:53 am
I also wrote to the UCL Provost (as an academic scientist and UCL alumnus) and got the stock response several other people have mentioned about the time that had gone into handling complaints etc.
I can see where he’s coming from, although on balance I think he is wrong (see the Stanford letter for why).
I think the wider point about UK Univs turning pale at the merest whiff of a threat of legal action that igb mentioned is a genuine problem. It appears that in this case they have at least taken real legal advice… but I have seen many examples where merely the threat of (e.g.) a student sueing is enough to cause a fit of the vapours, and would trigger tens or even hundreds of person hrs of administrative hot air.
I used to argue, without much success, that Univs should fight all these cases when they were sure they were right, especially when they dealt with “academic integrity” in the wider sense. And they should seek to recoup their adminstrative and legal costs against frivolous complainants like Walker and Lakin.
…the point being that if people think complaining and shouting “lawyer” will get them an undeserved second or third chance at an exam resit, or a website taken down, or whatever, people will keep doing it. As igb says, you have to give them a real potential DOWNSIDE to doing it, as well as a potential upside.
Incidentally, it is worth noting that Dr Walker is employed (although apparently now only in a part-time capacity, according to DC’s blog) by Reading University. Presumably they are happy about an academic from their School of Food Biosciences making public claims about unproven supplements and herbs that are scientific nonsense, and then waving M’Learned Friends when these claims are exposed. I wonder if she still teaches on their BSc in “Nutrition and Food Science”.
Mojo said,
June 10, 2007 at 1:05 pm
Dr Aust said,
“I used to argue, without much success, that Univs should fight all these cases when they were sure they were right, especially when they dealt with “academic integrity†in the wider sense. And they should seek to recoup their adminstrative and legal costs against frivolous complainants like Walker and Lakin.”
While legal costs are recoverable (assumong the Uni won the case), I’m not sure that this would apply to the Uni’s administrative costs.
Hence my suggestion above that the tort of falsely threatening to sue, at present only available in patent, trade mark and registered design disputes, might usefully be extended to libel. If it were, the Uni could then sue the frivolous complainants for their administrative costs as well.
JohnK said,
June 10, 2007 at 2:15 pm
I don’t understand why UCL didn’t just ask DC to remove the offending material, which he has done anyway. Booting him off the server seems to be an attempt to hang him out to dry (“there is also the question of Professor Colquhoun’s own personal liability.”), but if UCL are deemed to be publishers, removing the content does not alter the past; if it was illegal, stopping doing it doesn’t redeem them. To paraphrase an old joke, “Have you stopped hosting allegedly defamatory material on your website?” – both answers get you in trouble.
I wonder how much a lawsuit would actually cost if it came to it, and I wonder how much monetary value could be ascribed to DC’s RAE contribution.
Dr Aust said,
June 10, 2007 at 3:34 pm
Mojo wrote: “While legal costs are recoverable (assuming the Uni won the case), I’m not sure that this would apply to the Uni’s administrative costs.”
Shame. The main context for this was typically students contesting results, or complaining they had been treated unfairly, or denying they had been caught cheating, BTW. My argument was that a basic investigation of any alleged mistakes / irregularities was warranted and fair. For stuff beyond that we should be prepared to make people pay for the time and inconvenience caused by unfounded and often frivolous complaints.
What would happen was that the Univ would investigate (at Faculty level) and write back and say: “We have investigated your allegation and found it to be groundless… (gives details). However, if you are not satisfied with this, you may…. (appeal to next rung up).
The problem was that this gave people who were alleging a grievance no downside whatsoever to continuing to pursue groundless and often ludicrous claims, apart from their own time. In many cases it would go up the next one, or two, rungs in turn to the University’s senior administrator(s), with the same info being picked over multiple times by increasingly high-powered and expensive people.
I thought we should say “…if you are not satified you may (appeal to next rung up). HOWEVER, as your complaint has been investigated by our standard procedures and judged groundless, any further administrative time, and costs of expert advice we find it necessary to take, incurred by us through your pursuit of a complaint will be recorded. In the event that your complaint is ultimately judged groundless, it will be our practise in all cases to pursue you in civil court for the recovery of all these costs.”
Please somebody tell me that there is a case in law for doing this? Mojo’s posts above suggest not, which is sad.
If there isn’t, there ought to be…!
The point is that at some stage there needs to be a mechanism for making complainants judge whether they really have a case, or are just blustering for some other reason (like that they can’t admit, either for public consumption or even to themselves, that they were rumbled). They have to be made to do a “cost-benefit analysis” of wasting everyone’s time. Sadly at the moment cheats, charlatans, and obsessed nutters too often get a free ride.
Coming back to Univs, I suspect the cost and “negative publicity” is the factor the administrators prioritize when pressing for settlement or (as in DC’s case) “minimizing the University’s liability”. But if Universities are mainly selling themselves on their academic reputation (which in the final analysis they are), they have to be prepared to defend that reputation in the open, every time, and without compromise.
PS In terms of DC’s scientific standing and it’s worth to UCL, it has doubtless been worth a lot over the years.
RAE rankings contain a lot of nonsense, as DC himself has eloquently argued elsewhere:
voltaire.members.beeb.net/goodscience.htm
– but it is fair to say the UCL Pharmacology Dept has generally been regarded as one of the two or three, or arguably the best, pharmacology dept in the UK for all of the 25 yrs I have been in the business. As for DC himself, the FRS (judged by your peers to be a top scientist, and the only such thing British scientists rate) says it all.
Pepper said,
June 11, 2007 at 1:31 am
I see here 43 comments and a lot of people, which try to defend Prof. Colquhoun. But I’d like to know – is here just one man from UCL? And if the answer is “no”, then – what does this silence suggest? If DC is right, then why do his Alma Mater remain silent?
It is merely question. And I’d like merely to learn answer.
Filias Cupio said,
June 11, 2007 at 2:32 am
I know of one case where there was a significant downside to students for pushing too hard.
Two students had been caught cheating in a terms test. A friend of mine (from whom I have the story) summoned them to his office, and told them that they would get zero for the test, and for all assignments they’d done up to this point, but they could appeal to the university’s disciplinary committee. They did so, and instead were expelled for a year.
igb said,
June 11, 2007 at 10:32 am
“Isn’t the problem not so much that UCL are cowards as that the legal advice they have taken says they may lose with a heavy financial penalty. ”
So suddenly the `precautionary principle’, which most people with the vaguest scientific background regard as silly, has become respectable? No lawyer can tell you that you will not lose, just as no scientist can tell you that mobile phones are absolutely safe. So `may’ is the coward’s shield.
The reality is that a libel case fought by an individual against a large institution is almost imposssible to win, as legal aid is not available and most decisions can be appealed. In fact, “ the real issue the fact that people can use libel laws to restrict free speech” conceals the fact that current libel laws allow newspapers to accuse you of being a kiddie-fiddler whilst providing you with no redress, because libel cases are the strict preserve of the affluent.
Bearing in mind the requirements of a libel case, the risk to UCL is approximately zero. But it’s not actually zero.
Gimpy said,
June 11, 2007 at 10:54 am
igb I don’t see what the merits or otherwise of the precautionary principle have to do with this. I’m not defending UCL here, I’m just pointing out that libel law is abused as you correctly point out by the affluent. In this case the accusers are relatively wealthy.
I’m assuming that libel is the main legal argument being used against UCL because breaches of copyright rarely stand up in court if swiftly corrected and apology issued (which has been done in this case).
In this country the burden of proof in libel cases is on the defendant and there is no limit on the financial awards for damages. UCL obviously think there is a reasonable possibility that they may be liable for such damages and have taken what they consider appropriate action while they review the facts.
Dr Aust said,
June 11, 2007 at 11:56 am
Gimpy
That may be part of the reason, but what heinous libel would DC have committed against Walker and Lakin? He pointed out that terms like “blood cleanser” or “lymphatic cleanser” have no meaning as applied to drugs; he pointed out that their claims had no foundation in published research; he pointed out that certain organisations were not neutral information services but actually exist to promote supplements; and he used the word “gobbledegook”, which in the context used could be taken to mean “scientically meaningless or nonsensical”.
Would Walker and La kin they really ever want all this aired in open court? That is, that they use the pretence of “science”, and stuff that is arguably in breach of the trades descriptions, to relieve the gullible of their money? I find this inconceivable.
vinnyr said,
June 11, 2007 at 12:12 pm
I’m sure UCL are covered by the same legislation as websites such as YouTube when it comes to copyright infringements.
As they are only hosting the blog, all they need to do is inform Dr Colquhoun of problem with his blog and take down the page if he does not correct the infringement within a reasonable amount of time (usually ~24 hours).
Gimpy said,
June 11, 2007 at 12:12 pm
Dr Aust – “but what heinous libel would DC have committed against Walker and Lakin?”
I have no idea. All I was trying to do was see things from UCL’s side. It does seem a hasty decision on the part of UCL though. Anyway, the courts are not the place to establish the veracity of science nor indeed the truth in libel trials as the cases of Jeffry Archer and Jonathan Aitken prove.
andrew said,
June 11, 2007 at 12:20 pm
Well, I’m no lawyer, and I see that Prof. Grant is.
Nor am I going to start second-guessing that senior defamation QC they’ll be meeting today.
On the other hand, for background info, outlaw.com is a solid source of information on internet law.
Here’s their stuff on “User-generated content”
www.out-law.com/page-7807
and on “Liability of ISPs for third party material”
www.out-law.com/page-488
Dr Aust said,
June 11, 2007 at 12:50 pm
Point taken, Andrew.
I think what worries us here is the possibility that UCL, and other comparable institutions, will seek to position themselves to have NO conceivable liability.
I would imagine it is virtually impossible to utterly exclude liability unless (i) every page on a University’s website is scrutinized by a libel QC, or (ii) anything thought to be even vaguely “controversial” (read : “critical”) is blanket forbidden.
In which case critics of misinformation stand a good chance of being silenced.
andrew said,
June 11, 2007 at 1:24 pm
To clarify, the previous post is mainly to attention to outlaw.com‘s explanation of the E-commerce Directive and related material, e.g.
“Article 12 [of the E-commerce Directive] provides that each member state shall ensure that service providers (which will include ISP s, VISPs and Web Hosts) will not be held liable for information transmitted on their sites provided that the relevant service provider:
– Does not initiate the transmission;
– Does not select the receiver of the transmission; and
– Does not select or modify the information contained in the transmission.
In other words, if the above criteria are met a service provider will be treated as a mere conduit as opposed to an author, editor or publisher. However, a service provider will still be required to remove unlawful and/or defamatory material from its site once it has received a complaint.”
All I’m saying is that I’m not qualified to comment on how it applies in this case, you’ll have to make of it what you will.
raygirvan said,
June 11, 2007 at 1:55 pm
> minor breaches of copyright, which DC could have (and has) corrected. And there was no “malicious intent†behind the infringement, since he did not do it specifically to steal their trademarked words. He did it to highlight that what they were saying was untrue.
… which I would have said put it well into the territory of fair use for the purposes of comment or criticism.
Symball said,
June 11, 2007 at 2:56 pm
I think the real shame here has been the obvious victory of harassment over principle. I don’t believe that UCL has done anything other than protect itself financially and try to draw a line between personal comment and university statements. To be honest there are not many organisations that would allow its IT resources to be used for anything other than some ‘fair use’ surfing. so it is not surprising it has asked for the blog to be removed.
However it is sad that the woo’s have used similar tactics to the animal rights mob in simply harassing organisations into doing their bidding. Perhaps UCL could redress the balance by looking into the subject and publishing something in its own name instead
Dr Aust said,
June 11, 2007 at 3:45 pm
I suspect UCL probably couldn’t use the “ISP defence” indicated by Andrew above. This is because a complainant could argue, with some plausibility, that DC’s “pseudoscience debunking” clearly stems from his work for UCL as a scientist. So hard to separate the two.
But this just brings back to the “Is what DC said true?” issue.
Quoting from a site talking about the law of defamation:
webjcli.ncl.ac.uk/2005/issue3/lewis3.html
“Where defamation is alleged, the first step is to consider the ordinary and natural meaning of the words used and what an ordinary person will infer.”
“If a defendant can prove the substantial truth of the words complained about the defence of justification is established.”
“Another defence in the law of defamation is that everyone is allowed to comment so long as the subject is a matter of public interest and the views were honestly held. The public interest has never been satisfactorily defined for these purposes but it is clear that it is to be broadly construed.”
All these seem to offer fairly obvious defences.
Of course, the UCL Provost has stated for the record that it was the “admin bother and nuisance” that was the issue, rather than the risk of liability at law. I still think, though, that they had some sort of wider moral obligation, as an institute of learning and “enlightenment”, to be SEEN to defend the right of scholars to oppose obfuscation and inaccuracy, especially when the latter were being used to sell things.
igb said,
June 11, 2007 at 8:10 pm
“igb I don’t see what the merits or otherwise of the precautionary principle have to do with this. ”
Because the basic argument seems to be “a lawyer says this bad thing _may_ happen” or even “a lawyer says this bad thing cannot be said never to happen”. That’s exactly the argument that idiots use about wifi: “can you tell me it’s absolutely safe with no caveats? No? Then we should assume the worst”.
“libel law is abused as you correctly point out by the affluent. In this case the accusers are relatively wealthy.”
I may be mis-judging the finances of alternatives, but I seriously doubt that the people making the theats have pockets as deep as would be required. UCL could quite justifiably demand that measures be taken to ensure their costs are paid should they win: that’s where the rubber meets the road.
“I’m assuming that libel is the main legal argument being used against UCL because breaches of copyright rarely stand up in court if swiftly corrected and apology issued (which has been done in this case).”
The same’s true of libel, because…
“In this country the burden of proof in libel cases is on the defendant”
No, it isn’t. If the defendant opts to run a defence of justification, the burden is on them (albeit only to a civil, “balance of probabilities” standard). But the burden resides with the plaintiff to show that the words are capable of having a defamtory meaning (which might be _very_ difficult in this case) and that the plaintiff suffered harm to their repution. And there’s a whole stack of defences which might apply in this case (notably a Reynolds defence, see Reynolds vs Times Newspaper) for which the reverse burden doesn’t apply in the same way.
“UCL obviously think there is a reasonable possibility that they may be liable for such damages and have taken what they consider appropriate action while they review the facts.”
I don’t see where `reasonable’ comes from. I might just as (in)acurrately say `remote’ in the same place. A case in which a University was held to be vicariously liable for the public statements of a professor, writing in a field which is his exact speciality, requires a sequence of events all of which have a probability distinctly less than one (the writ being served, the case making it to court, the case being held to be answerable, the judge being prepared to join UCL to the case, the case making it past a jury, the case making it past an appeal, the case having damages greater than the hundred quid that UCL will have paid into the court).
John Craddock said,
June 11, 2007 at 8:47 pm
Re: mch’s comment;
“Why has UCL a moral obligation to defend our rights? It’s a university – it has a business to run, students to teach, research to, well, search. Making a stand and getting sued will cost (and maybe not just money), and who is going to refund it?”
UCL has an obligation to defend the freedom of its academics. If it doesn’t, then it reduces its role to that of a degree factory.
I don’t know what the situation is in the UK but the universities act in Ireland (quoted below) is clear on the issue, I presume you have similar principles and laws over there.
14.—(1) A university, in performing its functions shall—
( a ) have the right and responsibility to preserve and promote the traditional principles of academic freedom in the conduct of its internal and external affairs
…
(2) A member of the academic staff of a university shall have the freedom, within the law, in his or her teaching, research and any other activities either in or outside the university, to question and test received wisdom, to put forward new ideas and to state controversial or unpopular opinions
Dr Aust said,
June 11, 2007 at 11:21 pm
I suppose if a “justification defence” is deemed too risky there is always “fair comment in voicing a sincerely held view on a matter of public interest” (see my post above). The sincerity is not in doubt and the whole tenor of DC’s blog is malice-free – it always just asks “do these statements have scientific meaning” or sometimes “do these people have hidden interests they have not made clear?”
I have read the words about Walker and Lakin and their product very carefully, first with my amateur barrack-room lawyer’s hat on, then as a scientist with an interest in the use of words, and finally as a “member of the public” – and I still can’t see anything that could not be construed as “DC’s sincerely held opinion”.
I would still hope that in an analogous situation in the future a (any?) University would have the stones to put up the justification defence when the statements could be easily argued to be true. The point of pubically taking a stand specifically on justification would be, as mentioned by many here,
“We stand by our guy and his right to try and inform the public about a matter of public interest, no matter what”.
If Universities don’t stand for stuff like this, then mch is right and they are just businesses. But when they admit that, they are on the slide, because their business is based at bottom on their academic REPUTATION, which is based on their not being “biddable” by financial considerations alone. That is why, in science, research from Univs is by and large more trusted than research from drug companies.
Stanford, though a private institution (and thus more of a “business” than UCL), seems to have understood this, judging by the tobacco company example the Stanford prof gave on DC’s blog:
www.dcscience.net/quack.html#move1
UCL has misjudged the same, IMHO.
What I sincerely HOPE is happening behind the scenes is UCL offering DC legal advice about how to avoid problems going forward with his now “privatised” blog. That would go some way to restoring my faith in my old alma mater.
Kells said,
June 12, 2007 at 12:02 pm
Whilst shutting down DC UCL would like you to give generously to this
www.uclh.nhs.uk/New+developments/RLHH+redevelopment/
they need 1/4 million to house thier CAM library full of non evidence based periodicals of absolute bullshit.
Please give generously
Dr Aust said,
June 12, 2007 at 1:40 pm
It’s more interesting than that – from the webpage:
“…
New specialist electronic library on complementary and alternative medicine (NeLCAM)
The RLHH recently won the contract to provide the NHS "new specialist electronic library on complementary and alternative medicine (NeLCAM) in collaboration with the Research Council for Complementary Medicine (RCCM) and the University of Westminster’s School of Integrated Health. ..”
This is, of course, the same Univ of Westminster School of Integrated Health that DC has been chiding on his blog and in the pages of Nature for awarding BScs in antiscience, and which awards a “B.Sc. in Homeopathy” for which the External Examiner is (surprise surprise) a non-scientifically qualified homeopath.
The RLHH appeal is for money to fund their “open access CAM Information Centre”. Oh goody. They say this Centre will “work with other bodies within the world of complementary medicine, including the Research Council for Complementary Medicine, the British Homoeopathic Association, and The Prince of Wales’s Foundation for Integrated Health”.
Boosters all, of course. Now why doesn’t that leave me feeling reassured?
Pepper said,
June 12, 2007 at 5:34 pm
Well…
And what next?
DC’s webpage is expelled from UCL server. Quacks intend to frame up a case against DC. Homoeopaths are trying to edge in UCL.
Scientific people have written to provost. Provost has answered.
That’s all.
And strange silence has settled…
What is it? Is it defeat? Or the hush before the storm?
Hey! Defenders of freedom and real Science! Or will this problem leave in the air? Will it exist further in present state?
That will never do, IMHO. It’s unscientifically, after all.
It is necessary right solution of this question.
andrew said,
June 13, 2007 at 10:19 am
As an aside, Malcolm Grant is also catching flak over UCL’s armaments investments (£900k in Cobham PLC).
New Statesman 11 June 2007:
“Despite the overwhelming support of the Disarm UCL campaign, Grant refused to genuinely engage with the issue of divestment from Cobham. Instead he concentrated on criticizing students and suggested we were campaigning against UCL.”
It’s been a rough week for poor Grant, and it’s still only Wednesday…
Pepper said,
June 13, 2007 at 1:38 pm
Aha, Malcolm Grant gains money for UCL and UCL’s students by armaments investments.
But UCL students can’t even tackle his provost to gain money by other way! The students and staff in other universities have done it. And UCL student can merely yelp against provost like silly pups and unroll antiwar banners. One question, please! Do they like to get stipends and salaries ill-gotten by their provost for them? Eh?
No?? Then – let UCL students and staff propose their provost OTHER way to gain money for UCL. There are a lot of methods to get money from development of modern, knowledge-intensive, advanced technologies, from applied scientific research, etc., etc., etc.
Who is richest man in the world? Bill Gates! Does Bill Gates sells the arms? He makes and cells computers.
UCL students and staff must propose your provost best way to gain money. But if he refuse, then there will be only remaining resource – to put question about discharge him for inaptitude, so in this case his words about business and progress for UCL would be empty words and he would be merely wild aggressive politician of last centuries with backward opinions and policy.
Pepper said,
June 13, 2007 at 3:00 pm
DAVID COLQUHOUN WON!!!
Here is ad from his website:
Announcement 13 June 2007. UCL restores DC’s IMPROBABLE SCIENCE page.
After taking legal advice, the provost and I have agreed a joint statememt. Read it on the UCL web site.
" . . . the Provost and Professor Colquhoun have taken advice from a senior defamation Queen’s Counsel, and we are pleased to announce that Professor Colquhoun’s website – with some modifications effected by him on counsel’s advice – will shortly be restored to UCL’s servers."
I am grateful to UCL for its legal support, and I’m very grateful too for the enormous support I’ve had from many people, especially since Ben Goldacre mentioned the site move. Now all I need is a bit of help to get it into a more convenient format. The page will stay at its present address until there is time to sort things out.
MY CONGRATULATIONS, DEAR DAVID!!!
BE HAPPY AND HEALTHY!!!
Pepper said,
June 13, 2007 at 3:04 pm
Here is link of UCL website about DC:
www.ucl.ac.uk/news/news-articles/0706/07061303
Tabazan said,
June 13, 2007 at 4:20 pm
Good statement . . nice to see common sense won through in the end
Grathuln said,
June 14, 2007 at 12:55 pm
Perhaps the UK would benefit from “safe habour” laws, making site hosts immune from prosecution for content; I thought we must have something like this already but the Provos statement suggests otherwise. Perhaps we would also benefit from fair usage copyright laws, allowing the kind of use Dr. Colquhoun.
I hope that if this does go to court on defamation it gets summarily kicked out and used as example of how such cases will be treated in the future.
ihid said,
December 18, 2009 at 10:37 am
Yeah, this is really shocking!