The reproducibility of Science. A meeting report.

Published April 14, 2015

There is a widespread belief that science is going through a crisis of reproducibility. A meeting was held to discuss the problem. It was organised by Academy of Medical Sciences, the Wellcome Trust, MRC and BBSRC, and It was chaired by Dorothy Bishop (of whose blog I’m a huge fan). It’s good to see that scientific establishment is beginning to take notice. Up to now it’s been bloggers who’ve been making the running. I hadn’t intended to write a whole post about it, but some sufficiently interesting points arose that I’ll have a go.

The first point to make is that, as far as I know, the “crisis” is limited to, or at least concentrated in, quite restricted areas of science. In particular, it doesn’t apply to the harder end of sciences. Nobody in physics, maths or chemistry talks about a crisis of reproducibility. I’ve heard very little about irreproducibility in electrophysiology (unless you include EEG work). I’ve spent most of my life working on single-molecule biophysics and I’ve never encountered serious problems with irreproducibility. It’s a small and specialist field so I think if I would have noticed if it were there. I’ve always posted on the web our analysis programs, and if anyone wants to spend a year re-analysing it they are very welcome to do so (though I have been asked only once).

The areas that seem to have suffered most from irreproducibility are experimental psychology, some areas of cell biology, imaging studies (fMRI) and genome studies. Clinical medicine and epidemiology have been bad too. Imaging and genome studies seem to be in a slightly different category from the others. They are largely statistical problems that arise from the huge number of comparisons that need to be done. Epidemiology problems stem largely from a casual approach to causality. The rest have no such excuses.

The meeting was biased towards psychology, perhaps because that’s an area that has had many problems. The solutions that were suggested were also biased towards that area. It’s hard to see some of them could be applied to electrophysiology for example.

There was, it has to be said, a lot more good intentions than hard suggestions. Pre-registration of experiments might help a bit in a few areas. I’m all for open access and open data, but doubt they will solve the problem either, though I hope they’ll become the norm (they always have been for me).

All the tweets from the meeting hve been collected as a Storify. The most retweeted comment was from Liz Wager

@SideviewLiz: Researchers are incentivised to publish, get grants, get promoted but NOT incentivised to be right! #reprosymp

This, I think, cuts to the heart if the problem. Perverse incentives, if sufficiently harsh, will inevitably lead to bad behaviour. Occasionally it will lead to fraud. It’s even led to (at least) two suicides. If you threaten people in their forties and fifties with being fired, and losing their house, because they don’t meet some silly metric, then of course people will cut corners. Curing that is very much more important than pre-registration, data-sharing and concordats, though the latter occupied far more of the time at the meeting.

The primary source of the problem is that there is not enough money for the number of people who want to do research (a matter that was barely mentioned). That leads to the unpalatable conclusion that the only way to cure the problem is to have fewer people competing for the money. That’s part of the reason that I suggested recently a two-stage university system. That’s unlikely to happen soon. So what else can be done in the meantime?

The responsibility for perverse incentives has to rest squarely on the shoulders of the senior academics and administrators who impose them. It is at this level that the solutions must be found. That was said, but not firmly enough. The problems are mostly created by the older generation It’s our fault.

IncidentalIy, I was not impressed by the fact that the Academy of Medical Sciences listed attendees with initials after peoples’ names. There were eight FRSs but I find it a bit embarrassing to be identified as one, as though it made any difference to the value of what I said.

It was suggested that courses in research ethics for young scientists would help. I disagree. In my experience, young scientists are honest and idealistic. The problems arise when their idealism is shattered by the bad example set by their elders. I’ve had a stream of young people in my office who want advice and support because they feel they are being pressured by their elders into behaviour which worries them. More than one of them have burst into tears because they feel that they have been bullied by PIs.

One talk that I found impressive was Ottloline Leyser who chaired the recent report on The Culture of Scientific Research in the UK, from the Nuffield Council on Bioethics. But I found that report to be bland and its recommendations, though well-meaning, unlikely to result in much change. The report was based on a relatively small, self-selected sample of 970 responses to a web survey, and on 15 discussion events. Relatively few people seem to have spent time filling in the text boxes, For example

“Of the survey respondents who provided a negative comment on the effects of competition in science, 24 out of 179 respondents (13 per cent) believe that high levels of competition between individuals discourage research collaboration and the sharing of data and methodologies.&rdquo:

Such numbers are too small to reach many conclusions, especially since the respondents were self-selected rather than selected at random (poor experimental design!). Nevertheless, the main concerns were all voiced. I was struck by

“Almost twice as many female survey respondents as male respondents raise issues related to career progression and the short term culture within UK research when asked which features of the research environment are having the most negative effect on scientists”

But no conclusions or remedies were put forward to remedy this problem. It was all put rather better, and much more frankly, some time ago by Peter Lawrence. I do have the impression that bloggers (including Dorothy Bishop) get to the heart of the problems much more directly than any official reports.

The Nuffield report seemed to me to put excessive trust in paper exercises, such as the “Concordat to Support the Career Development of Researchers”. The word “bullying” does not occur anywhere in the Nuffield document, despite the fact that it’s problem that’s been very widely discussed and a problem that’s critical for the problems of reproducibility. The Concordat (unlike the Nuffield report) does mention bullying.

"All managers of research should ensure that measures exist at every institution through which discrimination, bullying or harassment can be reported and addressed without adversely affecting the careers of innocent parties. "

That sounds good, but it’s very obvious that there are many places simply ignore it. All universities subscribe to the Concordat. But signing is as far as it goes in too many places. It was signed by Imperial College London, the institution with perhaps the worst record for pressurising its employees, but official reports would not dream of naming names or looking at publicly available documentation concerning bullying tactics. For that, you need bloggers.

On the first day, the (soon-to-depart) Dean of Medicine at Imperial, Dermot Kelleher, was there. He seemed a genial man, but he would say nothing about the death of Stefan Grimm. I find that attitude incomprehensible. He didn’t reappear on the second day of the meeting.

The San Francisco Declaration on Research Assessment (DORA) is a stronger statement than the Concordat, but its aims are more limited. DORA states that the impact factor is not to be used as a substitute “measure of the quality of individual research articles, or in hiring, promotion, or funding decisions”. That’s something that I wrote about in 2003, in Nature. In 2007 it was still rampant, including at Imperial College. It still is in many places. The Nuffield Council report says that DORA has been signed by “over 12,000 individuals and 500 organisations”, but fails to mention the fact that only three UK universities have signed up to DORA (oneof them, I’m happy to say, is UCL). That’s a pretty miserable record. And, of course, it remains to be seen whether the signatories really abide by the agreement. Most such worthy agreements are ignored on the shop floor.

The recommendations of the Nuffield Council report are all worthy, but they are bland and we’ll be lucky if they have much effect. For example

“Ensure that the track record of researchers is assessed broadly, without undue reliance on journal impact factors”

What on earth is “undue reliance”? That’s a far weaker statement than DORA. Why?

And

“Ensure researchers, particularly early career researchers, have a thorough grounding in research ethics”

In my opinion, what we should say to early career researchers is “avoid the bad example that’s set by your elders (but not always betters)”. It’s the older generation which has produced the problems and it’s unbecoming to put the blame on the young. It’s the late career researchers who are far more in need of a thorough grounding in research ethics than early-career researchers.

Although every talk was more or less interesting, the one I enjoyed most was the first one, by Marcus Munafo. It assessed the scale of the problem (though with a strong emphasis on psychology, plus some genetics and epidemiology), and he had good data on under-powered studies. It also made a fleeting mention of the problem of the false discovery rate. Since the meeting was essentially about the publication of results that aren’t true, I would have expected the statistical problem of the false discovery rate to have been given much more prominence than it was. Although Ioannidis’ now-famous paper “Why most published research is wrong” got the occasional mention, very little attention (apart from Munafo and Button) was given to the problems which he pointed out.

I’ve recently convinced myself that, if you declare that you’ve made a discovery when you observe P = 0.047 (as is almost universal in the biomedical literature) you’ll be wrong 30 – 70% of the time (see full paper, "An investigation of the false discovery rate and the misinterpretation of p-values".and simplified versions on Youtube and on this blog). If that’s right, then surely an important way to reduce the publication of false results is for journal editors to give better advice about statistics. This is a topic that was almost absent from the meeting. It’s also absent from the Nuffield Council report (the word “statistics” does not occur anywhere).

In summary, the meeting was very timely, and it was fun. But I ended up thinking it had a bit too much of preaching good intentions to the converted. It failed to grasp some of the nettles firmly enough. There was no mention of what’s happening at Imperial, or Warwick, or Queen Mary, or at Kings College London. Let’s hope that when it’s written up, the conclusion will be a bit less bland than those of most official reports.

It’s overdue that we set our house in order, because the public has noticed what’s going on. The New York Times was scathing in 2006. This week’s Economist said

"Modern scientists are doing too much trusting and not enough verifying -to the detriment of the whole of science, and of humanity.
Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis"

"Careerism also encourages exaggeration and the cherrypicking of results."

This is what the public think of us. It’s time that vice-chancellors did something about it, rather than willy-waving about rankings.

Conclusions

After criticism of the conclusions of official reports, I guess that I have to make an attempt at recommendations myself. Here’s a first attempt.

The heart of the problem is money. Since the total amount of money is not likely to increase in the short term, the only solution is to decrease the number of applicants. This is a real political hot-potato, but unless it’s tackled the problem will persist. The most gentle way that I can think of doing this is to restrict research to a subset of universities. My proposal for a two stage university system might go some way to achieving this. It would result in better postgraduate education, and it would be more egalitarian for students. But of course universities that became “teaching only” would see (wrongly) as demotion, and it seems that UUK is unlikely to support any change to the status quo (except, of course, for increasing fees).
Smaller grants, smaller groups and fewer papers would benefit science.
Ban completely the use of impact factors and discourage use of all metrics. None has been shown to measure future quality. All increase the temptation to “game the system” (that’s the usual academic euphemism for what’s called cheating if an undergraduate does it).
“Performance management” is the method of choice for bullying academics. Don’t allow people to be fired because they don’t achieve arbitrary targets for publications or grant income. The criteria used at Queen Mary London, and Imperial, and Warwick and at Kings, are public knowledge. They are a recipe for employing spivs and firing Nobel Prize winners: the 1991 Nobel Laureate in Physiology or Medicine would have failed Imperial’s criteria in 6 years out of 10 years when he was doing the work which led to the prize.
Universities must learn that if you want innovation and creativity you have also to tolerate a lot of failure.
The ranking of universities by ranking businesses or by the REF encourages bad behaviour by encouraging vice-chancellors to improve their ranking, by whatever means they can. This is one reason for bullying behaviour. The rankings are totally arbitrary and a huge waste of money. I’m not saying that universities should be unaccountable to taxpayers. But all you have to do is to produce a list of publications to show that very few academics are not trying. It’s absurd to try to summarise a whole university in a single number. It’s simply statistical illiteracy
Don’t waste money on training courses in research ethics. Everyone already knows what’s honest and what’s dodgy (though a bit more statistics training might help with that). Most people want to do the honest thing, but few have the nerve to stick to their principles if the alternative is to lose your job and your home. Senior university people must stop behaving in that way.
University procedures for protecting the young are totally inadequate. A young student who reports bad behaviour of his seniors is still more likely to end up being fired than being congratulated (see, for example, a particularly bad case at the University of Sheffield). All big organisations close ranks to defend themselves when criticised. Even extreme cases, as when an employee commits suicide after being bullied, universities issue internal reports which blame nobody.
Universities must stop papering over the cracks when misbehaviour is discovered. It seems to be beyond the wit of PR people to realise that often it’s best (and always the cheapest) to put your hands up and say “sorry, we got that wrong”
There an urgent need to get rid of the sort of statistical illiteracy that allows P = 0.06 to be treated as failure and P = 0.04 as success. This is almost universal in biomedical papers, and given the hazards posed by the false discovery rate, could well be a major contribution to false claims. Journal editors need to offer much better statistical advice than is the case at the moment.

Follow-up

Tagged Alice Gast, bibliometrics, Dermot Kelleher, impact factor, Imperial College, irreproducibility, James Stirling, King's College London, metrics, Queen Mary, Queen Mary University of London, reproducibility, University of Warwick | 1 Comment

Some experiences of life at Imperial College London. An external inquiry is needed after the death of Stefan Grimm

Published December 23, 2014

Jump to follow-up

The tragedy of the apparent suicide of Stefan Grimm is now known worldwide. His last email has been read by more than 160,000 people from over 200 countries. This post gathers together some of the reactions to his death. It’s a Christmas card for the people who are responsible.

Alice Gast (president)

James Stirling (provost)

Dermot Kelleher (VP (health)

“This isn’t about science – it’s about bragging rights, or institutional willy-waving.” from Grimm’s Tale

The back story

On Monday 1st December I published Stefan Grimm’s last email. It has been read by more than 160,000 people from over 200 different countries.

On Tuesday 2nd December, Stefan Grimm’s immediate boss, Martin Wilkins, wrote to me. He claimed “We met from time to time to discuss science and general matters. These meetings were always cordial. ”

On Wednesday 3rd December, the Dean of Medicine, Dermot Kelleher, mailed all Faculty of Medicine staff (not the rest of the College). Read the letter. It said very little. But it did include the words

“I regret I did not know Stefan personally, and I looked to colleagues to describe to me his life and the impact of his work at Imperial “

It seems a bit odd that the Dean of Medicine did not know a senior professor, but that seems to be life at Imperial.

On Thursday 4th December, Times Higher Education printed the same last email, and also the text of a threatening letter sent to Grimm in March.by his boss, Martin Wilkins. The letter was very far from being cordial, contrary to what Wilkins claimed. It included these words.

“I am of the opinion that you are struggling to fulfil the metrics of a Professorial post at Imperial College which include maintaining established funding in a programme of research with an attributable share of research spend of £200k p.a and must now start to give serious consideration as to whether you are performing at the expected level of a Professor at Imperial College.”

For a successful 51 year old with a good publication record to get a letter like that must have been devastating.

On Friday 5th December, Imperial made its first public announcement of his death. more than three months after it happened. By this time a damning account of his death had appeared even in the Daily Mail. The announcement read as though the world was unaware of his last words. It was a PR disaster: weasel words and crocodile tears. It made Imperial College appear to be totally heartless. The official announcement was accompanied by the phone numbers for the Samaritans. the chaplaincy and mental health first-aiders. Giving a person a phone number to call when you’ve destroyed their life is not an adequate substitute for treating staff properly.

Imperial are still trying to pretend that Grimm’s death is nothing to do with them, despite the fact that the whole world now knows quite enough of the facts to see otherwise.

The Coroner’s Inquest

The inquest into Grimm’s death was adjourned on October 8th, pending investigations into its cause. If you know anything relevant you should email the Coroner’s officer who is responsible for the investigation. That’s Molly Stewart (Molly.Stewart@lbhf.gov.uk). It is rather important that all the information doesn’t come from the College authorities, which cannot be relied on to tell the truth.

Some reports about the regime at Imperial College

Since my post went up on December 1st, I’ve had a stream of emails which testify to the reign of terror operated by the senior management at Imperial. The problem is by no means restricted to the Faculty of Medicine, though the problems seem to be worst there.

Many of these correspondents don’t want to speak in public. That’s certainly true of people who still work at Imperial, who have been warned to deflect all enquiries to HR. Here are some of the stories that I can reveal.

The Research Excellence Framework (REF) results were announced on 18th December. All university PR people hunted through the results, and all found something to boast tediously about. The letter from Imperial’s provost, James Stirling (read it), is pretty standard stuff. as is the letter from the Dean of Medicine, Dermot Kelleher (read it). Needless to say, neither letter mentioned the price in human misery, and even death, that Imperial had paid for its high ranking. I felt compelled to tweet

Imperial is second in #REF2014. And cost only one life #Grimmdeath. Well worth it uh? #HRbollocks http://t.co/25QNe8agjs

— David Colquhoun (@david_colquhoun) December 18, 2014

Kelleher promoted. Astonishingly, the very next day, the Dean of Medicine, on whose watch Grimm died, was promoted. You can read the letter from Imperial’s president, Alice Gast, in which this is announced. He is to be Vice President (Health), as a reward no doubt, for the cruel regime he ran as Dean. The letter has all the usual vacuous managerial buzzwords, e.g. “to support and grow the multidisciplinary paradigm in health”. Remember DC’s rule number one: never trust anyone who uses the word ‘paradigm’. Needless to say, still no mention of treating staff better.

Dr William J Astle.

Dr Astle is one of many people who wrote to me about his experiences at Imperial College. Although he still appears on Imperial’s web site, he now works as a statistician in a bioinformatics team at the University of Cambridge (see their web site).

He wrote again on 23 October 2014, to pass on an email (read the mail) that was sent to Department staff after Grimm’s last email had been circulated.(on 21 October). It is from a Faculty Operating Officer and ends with a warning to refer media enquries to a PR person (the Press and Internal Communications Manager, John-Paul Jones).

When he saw the internal email from Provost James Stirling with the usual self-congratulatory stuff about the REF, Astle wrote again to Stirling, His letter ends thus.

“Putting university staff in fear of losing their jobs leads to an atmosphere of obsequiousness and obedience to authority that prevents academics from fulfilling their institutional role. In a free society it is essential academics have the autonomy to determine their line of work, to question institutional and state authority and to do risky research. Once again I emphasise – in my experience the atmosphere in the faculty of medicine at IC is not conducive to this.”

Stirling did not reply to this letter. Neither Gast nor Stirling have replied to mine either. Discourtesy seems to be part of the job description of senior managers.

Christine Yates

Christine Yates says

“I was employed at Imperial College London from s” August 2002 to October 2012. For these 10 years I was the College’s Equality and Diversity Consultant in the Human Resources Department, reporting to the HR deputy director, Kim Everitt. In turn, Kim Everitt reports to the HR director, Louise Lindsay. Throughout this time I was the College’s sole equalities consultant, and over time built up the Equalities Unit and managed a team of five.”

“I was dismissed on 8th October 2012 following a Disciplinary Hearing in response to an allegation of gross misconduct “for continued wilful refusal to follow your Head of Department’s (HOD) instructions not to be involved in individual cases”.

As part of her job, she was responsible for establishing and maintaining the Harassment Support Contact Scheme, which was designed to help staff who felt they were being harassed, bullied, and victimised. She was also responsible for the College’s first Athena SWAN (scientific women’s academic network) .successful application, along with the establishment of disabilities, race equalities, and sexual orientation networks, all of which attained quality professional kite marks over time. The Athena Swan award is particularly ironic, given that Imperial’s present brutal assessment system must be even more unfair to women than it is to men. In 2003 (when Richard Sykes was still in charge), a third of female employees at Imperial reported that they were bullied. The improvement since then seems to have been small.

One of many cases she dealt with involved the harassment and bullying of a senior female academic by her male boss. Yates maintains, with good evidence, that complaints about this behaviour were never investigated properly by HR. This displeased HR. Incidents like this undoubtedly contributed to her dismissal.

“In Dr ***’s [female] case, it is clear to me that no independent investigations have been held and that College procedures are being flouted or rnanipulated with the alleged harasser (Professor **** [male]) being protected and permitted to continue his misconduct.”

“In my position as the College’s Equalities Consultant, I was aware of many cases and outcomes. Or ***’s is one of the most distressing and badly handled cases I was witness to, and the manner in which HR protect senior academics who have gravely offended, and who under any reasonable circumstances would be found to be guilty of gross misconduct, is a sad indictment of Imperial College”

You can read the statement that Christine Yates has already sent to the Coroner’s officer. Unfortunately the attachments have had to be removed here because they deal with specific cases.

“The Coroner’s Office needs to be aware of the pattern of behaviour that ensues whenever bad practice is brought to the College’s attention. In response to whistle blows and other complaints the College tries to discredit the complainant. When this fails they will invariably state that they will hold a ‘review’ usually undertaken by those responsible for the bad behaviour and thus with a vested interest in covering up any misconduct and impropriety. It is noted this pattern remains unchanged, “

A problem with a paper

An anonymous correspondent has sent me a lot of emails that concern a paper that was in revision at the time of Grimm’s death. The title of the paper is “Role of non-coding RNAs in apoptosis revealed in a functional genome-wide screen”.

On October 6th, one author wrote to his co-authors “I worked closely with Stefan on the screen data this year. We re-interpreted the mathematical analysis performed in the original manuscript, providing a more rigorous statistical foundation of the gene rankings. As a result, the gene list Stefan and I have generated is now different.”.

Clearly Grimm was aware of the need for revision before he died. Given that everyone was under such intense pressure to publish, it’s likely that the prospect of a prolonged delay in publication might well have contributed to his depression and his death.

The author who wrote on October 6th outlined some options. One was to leave the paper as it was, but to include all the raw data and submit to a journal such as Plos One or the preprint server BioRxiv. This option “requires minimal work, and would result in no change in the author list. However we would aim for a lower-impact journal.”. His preferred option, though, was to rewrite the paper altogether (and for himself to become first co-author) “as it is in all our shared interest to get the work published in as good a journal as possible. “.

Two days later, on October 8th, the same author thanked his co-authors for their responses. As a result of the responses he got, he asked to have his name removed from the paper because he did not agree with what was contained in the manuscript. “However, given that I believe the gene list is wrong, I request my name to be removed from the author list. If any other authors do not wish for the raw data to be disclosed then I hope you think it’s reasonable for me to close off my involvement with the paper.”.

The paper has 11 authors, including Stefan Grimm. . I have written to all but one of the authors to try to ascertain the facts. Of the four co-authors who have replied, all but one said that they hadn’t seen the final paper. One said that they were unaware that they were on the author list, and said they probably shouldn’t be.

I have tried to protect the authors (some of whom are still at Imperial) by not mentioning their names. But one co-author is sufficiently senior to be mentioned by name. Alan Boobis answered by my mail cordially enough when I first wrote to him, but declined to give much useful information, apart from confirming that Grimm was the senior author on the paper. On October 9th he wrote to all co-authors, thus.

From: Boobis, Alan R [a.boobis@imperial.ac.uk]
Sent: 09 October 2014 18:15
To: xxxxxx [co-authors]
Subject: Re: News About Stefan & Screen Paper

Dear all

The situation regarding this manuscript needs to be dealt with rationally. There is a real danger that the reputations of individuals and of the College will be harmed. I suggest that we all need to agree the most appropriate way forward. I am out of the country this week but will have my secretary liaise with you next week to arrange a suitable time (face to face or by phone) to discuss this.

Best wishes,

Alan

I have no idea what the outcome of this meeting was. Personally. I always worry a bit when people want meetings “face to face or by phone”. Written records are much more informative.

I should like to make it clear that I’m not suggesting any misconduct whatsoever. The author who wished to withdraw acted with principle and courage, and mistakes happen. They are perhaps especially likely in multi-author papers where some authors don’t understand the input from others. But it is sad to see the emphasis on the long-discredited journal impact factor that was forced on them by Imperial’s policies. And it’s sad to see that several co-authors had not actually seen the final paper. This smacks of “citation-mongering”, yet another bad effect of the metrics culture that has pervaded all of academia, and which is enforced in an especially simple-minded way at Imperial.

This sad episode is yet another illustration of the way that Imperial’s policies are damaging people, and, in the end, damaging science.

Some discussions of the Imperial problem

Since Grimm’s last email was revealed, it’s been discussed in many blogs and articles. Here are a few of them.

Grimm’s tale (2 December). This perceptive blog reproduces part of the nasty threatening letter sent by Martin Wilkins to Grimm.

“Your current level of funding does not constitute the appropriate level for a professor at Imperial College. Unless you submit and are awarded a Platform grant as PI in the next 12 months we will seek to initiate disciplinary action against you.”

The author comments (my emphasis).

“This isn’t about science – it’s about bragging rights, or institutional willy-waving. Grimm was informed – in public – that he was to be fired, and left waiting for the axe to fall while the axe-wielder marauded around the campus boasting about it like an even more pathetic Alan Sugar.”

That sums it up for me. It’s very sad.

Another blog comments

“Martin Wilkins to Professor Stefan Grimm, a few months before the latter committed suicide. Imperial College had been pressuring Grimm to get 200, 000 pounds in grants in order for him to remain employed. They threatened to sack him as he only had 135,000 pounds.

Sounds a lot like loan sharks.”

Clearly universities like Imperial are no longer places for scholarship. They are more like anxiety machines.

The Nuffield Council on Bioethics produced an important report in the midst of the scandal about Grimm: The culture of scientific research 2014. Paragraph 1.7 produced a chilling statistic

1.7 Compromising on research integrity and standards

Almost six in ten (58%) respondents are aware of scientists feeling under pressure to
compromise on research integrity and standards, with poor methodology and data fraud
frequently mentioned in the free text responses.

Just over a quarter (26%) of those taking part in the survey have felt tempted to compromise
on research integrity.

Stefan Grimm and the British University system. This blog, written by a geneticist. Federico Calboli, based in Helsinki, gives an indication of the harm that Imperial is inflicting not only on itself, but on the whole of UK academia, and hence on the UK economy

“As always in the real world the best laid plans often conflict with how the world actually works, and this conflict gives rise to a number of unintended consequences. The first unintended consequence is that the pursuit of what managements defines as ‘novel’ and ‘glamorous’ will diminish the intellectual value of British academia as a whole.”

“Unfortunately, since academia, funding bodies and the editorial boards of papers have been taken over by top down management culture, solid rigorous science is penalised in favour of anything that can be branded as ’novel’, ‘cutting edge’, ‘state of the art’ and similar platitudes.”

“This policy will leave British academia directionless and intellectually empty, and will transform any research in technology and data driven drivel that can at most pick up low hanging fruits and will deliver less and less as time goes on.”

Still more shaming, Calboli continues thus.

“The second problem with how British academia is managed is the culture of intellectual dishonesty that is forced upon people. People are not allowed to just express their goals in simple honest terms. They are required to spin and embellish everything in order to have half a chance of getting some funding or publishing in a high impact journal – both crucial to contribute to the ‘excellence metric”.

“Only the shameless cynics thrive in such environment”.

The blog finishes with a rallying cry.

“the email that Prof Grimm sent in October did not magically make its way to the press by itself. While many people are feeling disenchanted with academia and leave, more and more insiders are taking a combative stance against the mindless hogwash that threatens the foundations of British academia and the people that push it. We should all stand up and be counted, or we will not be able to complain in the future. It would be great if management could live up to its role and abandon the idea that scientific research is simple, predictable and quickly profitable, and actually help build the future of British academia.”

All this reflects similar sentiments to those that I expressed in 2007 [the RAE was the predecessor of the REF]

“The policies described here will result in a generation of ‘spiv’ scientists, churning out 20 or even more papers a year, with very little originality. They will also, inevitably, lead to an increase in the sort of scientific malpractice that was recently pilloried viciously, but accurately, in the New York Times, and a further fall in the public’s trust in science. That trust is already disastrously low, and one reason for that is, I suggest, pressures like those described here which lead scientists to publish when they have nothing to say.”

““All of us who do research (rather than talk about it) know the disastrous effects that the Research Assessment Exercise has had on research in the United Kingdom: short-termism, intellectual shallowness, guest authorships and even dishonesty”. Now we can add to that list bullying, harassment and an incompetent box-ticking style of assessment that tends to be loved by HR departments.

This process might indeed increase your RAE score in the short term (though there is no evidence that it it does even that). But, over a couple of decades, it will rid universities of potential Nobel prize winners.”

Conclusions

The policies adopted by Imperial College have harmed Imperial’s reputation throughout the world. Worse still, they have tainted the reputation of all UK universities. They have contributed to the corruption of science. and they have, in all probability, killed a successful man,

I hope that Alice Gast (president), James Stirling (provost). Dermot Kelleher (Dean, now vice president), and Martin Wilkins (who was left to weild the knife) have a good Christmas. If I were in their shoes, I’d feel so guilty that I wouldn’t be able to sleep at night.

They should spend Christmas reading Peter Lawrence’s wonderful essay on The Mismeasurement of Science. Please download a copy

Their proposal that HR policies should be investigated by, inter alia, the head of HR has provoked worldwide derision.

Their refusal to set up an independent external inquiry is reprehensible.

Not for the first time, a fine institution is being brought into disrepute by its leadership. Council please note.

Alice Gast

James Stirling

Dermot Kelleher

Perhaps the best description of what’s going on is from Grimm’s Tale “This isn’t about science – it’s about bragging rights, or institutional willy-waving.”. Gast, Stirling and Kelleher should stop the willy-waving. They should either set about rectifying the damage they’ve done. Or they should resign. Now.

The chair of universities HR association, Kim Frost, said

“Bullying is a very emotive term, and what one person experiences as bullying will often be simple performance management from their manager’s point of view.”.

That’s scary because it shows that she hasn’t the slightest idea about “performance management”. I have news for HR people. They are called experiments because we don’t know whether they will work. If they don’t work that’s not a reason to fire anyone. No manager can make an experiment come out as they wish. The fact of the matter is that it’s impossible to manage research. If you want real innovation you have to tolerate lots and lots of failure. “Performance management” is an oxymoron. Get used to it.

This sorry episode has far more general lessons for the way the REF is conducted and for the metrics sales industry. Both share some of the guilt.

That will have to wait for another post.