Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
assessment
The tragedy of the apparent suicide of Stefan Grimm is now known worldwide. His last email has been read by more than 160,000 people from over 200 countries. This post gathers together some of the reactions to his death. It’s a Christmas card for the people who are responsible.
 Alice Gast (president) |
 James Stirling (provost) |
 Dermot Kelleher (VP (health) |
“This isn’t about science – it’s about bragging rights, or institutional willy-waving.” from Grimm’s Tale
The back story
On Monday 1st December I published Stefan Grimm’s last email. It has been read by more than 160,000 people from over 200 different countries.
On Tuesday 2nd December, Stefan Grimm’s immediate boss, Martin Wilkins, wrote to me. He claimed “We met from time to time to discuss science and general matters. These meetings were always cordial. ”
On Wednesday 3rd December, the Dean of Medicine, Dermot Kelleher, mailed all Faculty of Medicine staff (not the rest of the College). Read the letter. It said very little. But it did include the words
“I regret I did not know Stefan personally, and I looked to colleagues to describe to me his life and the impact of his work at Imperial “
It seems a bit odd that the Dean of Medicine did not know a senior professor, but that seems to be life at Imperial.
On Thursday 4th December, Times Higher Education printed the same last email, and also the text of a threatening letter sent to Grimm in March.by his boss, Martin Wilkins. The letter was very far from being cordial, contrary to what Wilkins claimed. It included these words.
“I am of the opinion that you are struggling to fulfil the metrics of a Professorial post at Imperial College which include maintaining established funding in a programme of research with an attributable share of research spend of £200k p.a and must now start to give serious consideration as to whether you are performing at the expected level of a Professor at Imperial College.”
For a successful 51 year old with a good publication record to get a letter like that must have been devastating.
On Friday 5th December, Imperial made its first public announcement of his death. more than three months after it happened. By this time a damning account of his death had appeared even in the Daily Mail. The announcement read as though the world was unaware of his last words. It was a PR disaster: weasel words and crocodile tears. It made Imperial College appear to be totally heartless. The official announcement was accompanied by the phone numbers for the Samaritans. the chaplaincy and mental health first-aiders. Giving a person a phone number to call when you’ve destroyed their life is not an adequate substitute for treating staff properly.
Imperial are still trying to pretend that Grimm’s death is nothing to do with them, despite the fact that the whole world now knows quite enough of the facts to see otherwise.
The Coroner’s Inquest
The inquest into Grimm’s death was adjourned on October 8th, pending investigations into its cause. If you know anything relevant you should email the Coroner’s officer who is responsible for the investigation. That’s Molly Stewart (Molly.Stewart@lbhf.gov.uk). It is rather important that all the information doesn’t come from the College authorities, which cannot be relied on to tell the truth.
Some reports about the regime at Imperial College
Since my post went up on December 1st, I’ve had a stream of emails which testify to the reign of terror operated by the senior management at Imperial. The problem is by no means restricted to the Faculty of Medicine, though the problems seem to be worst there.
Many of these correspondents don’t want to speak in public. That’s certainly true of people who still work at Imperial, who have been warned to deflect all enquiries to HR. Here are some of the stories that I can reveal.
The Research Excellence Framework (REF) results were announced on 18th December. All university PR people hunted through the results, and all found something to boast tediously about. The letter from Imperial’s provost, James Stirling (read it), is pretty standard stuff. as is the letter from the Dean of Medicine, Dermot Kelleher (read it). Needless to say, neither letter mentioned the price in human misery, and even death, that Imperial had paid for its high ranking. I felt compelled to tweet
Imperial is second in #REF2014. And cost only one life #Grimmdeath. Well worth it uh? #HRbollocks http://t.co/25QNe8agjs
— David Colquhoun (@david_colquhoun) December 18, 2014
Kelleher promoted. Astonishingly, the very next day, the Dean of Medicine, on whose watch Grimm died, was promoted. You can read the letter from Imperial’s president, Alice Gast, in which this is announced. He is to be Vice President (Health), as a reward no doubt, for the cruel regime he ran as Dean. The letter has all the usual vacuous managerial buzzwords, e.g. “to support and grow the multidisciplinary paradigm in health”. Remember DC’s rule number one: never trust anyone who uses the word ‘paradigm’. Needless to say, still no mention of treating staff better.
Dr William J Astle.
Dr Astle is one of many people who wrote to me about his experiences at Imperial College. Although he still appears on Imperial’s web site, he now works as a statistician in a bioinformatics team at the University of Cambridge (see their web site).
He wrote again on 23 October 2014, to pass on an email (read the mail) that was sent to Department staff after Grimm’s last email had been circulated.(on 21 October). It is from a Faculty Operating Officer and ends with a warning to refer media enquries to a PR person (the Press and Internal Communications Manager, John-Paul Jones).
When he saw the internal email from Provost James Stirling with the usual self-congratulatory stuff about the REF, Astle wrote again to Stirling, His letter ends thus.
“Putting university staff in fear of losing their jobs leads to an atmosphere of obsequiousness and obedience to authority that prevents academics from fulfilling their institutional role. In a free society it is essential academics have the autonomy to determine their line of work, to question institutional and state authority and to do risky research. Once again I emphasise – in my experience the atmosphere in the faculty of medicine at IC is not conducive to this.”
Stirling did not reply to this letter. Neither Gast nor Stirling have replied to mine either. Discourtesy seems to be part of the job description of senior managers.
Christine Yates
Christine Yates says
“I was employed at Imperial College London from s” August 2002 to October 2012. For these 10 years I was the College’s Equality and Diversity Consultant in the Human Resources Department, reporting to the HR deputy director, Kim Everitt. In turn, Kim Everitt reports to the HR director, Louise Lindsay. Throughout this time I was the College’s sole equalities consultant, and over time built up the Equalities Unit and managed a team of five.”
“I was dismissed on 8th October 2012 following a Disciplinary Hearing in response to an allegation of gross misconduct “for continued wilful refusal to follow your Head of Department’s (HOD) instructions not to be involved in individual cases”.
As part of her job, she was responsible for establishing and maintaining the Harassment Support Contact Scheme, which was designed to help staff who felt they were being harassed, bullied, and victimised. She was also responsible for the College’s first Athena SWAN (scientific women’s academic network) .successful application, along with the establishment of disabilities, race equalities, and sexual orientation networks, all of which attained quality professional kite marks over time. The Athena Swan award is particularly ironic, given that Imperial’s present brutal assessment system must be even more unfair to women than it is to men. In 2003 (when Richard Sykes was still in charge), a third of female employees at Imperial reported that they were bullied. The improvement since then seems to have been small.
One of many cases she dealt with involved the harassment and bullying of a senior female academic by her male boss. Yates maintains, with good evidence, that complaints about this behaviour were never investigated properly by HR. This displeased HR. Incidents like this undoubtedly contributed to her dismissal.
“In Dr ***’s [female] case, it is clear to me that no independent investigations have been held and that College procedures are being flouted or rnanipulated with the alleged harasser (Professor **** [male]) being protected and permitted to continue his misconduct.”
“In my position as the College’s Equalities Consultant, I was aware of many cases and outcomes. Or ***’s is one of the most distressing and badly handled cases I was witness to, and the manner in which HR protect senior academics who have gravely offended, and who under any reasonable circumstances would be found to be guilty of gross misconduct, is a sad indictment of Imperial College”
You can read the statement that Christine Yates has already sent to the Coroner’s officer. Unfortunately the attachments have had to be removed here because they deal with specific cases.
“The Coroner’s Office needs to be aware of the pattern of behaviour that ensues whenever bad practice is brought to the College’s attention. In response to whistle blows and other complaints the College tries to discredit the complainant. When this fails they will invariably state that they will hold a ‘review’ usually undertaken by those responsible for the bad behaviour and thus with a vested interest in covering up any misconduct and impropriety. It is noted this pattern remains unchanged, “
A problem with a paper
An anonymous correspondent has sent me a lot of emails that concern a paper that was in revision at the time of Grimm’s death. The title of the paper is “Role of non-coding RNAs in apoptosis revealed in a functional genome-wide screen”.
On October 6th, one author wrote to his co-authors “I worked closely with Stefan on the screen data this year. We re-interpreted the mathematical analysis performed in the original manuscript, providing a more rigorous statistical foundation of the gene rankings. As a result, the gene list Stefan and I have generated is now different.”.
Clearly Grimm was aware of the need for revision before he died. Given that everyone was under such intense pressure to publish, it’s likely that the prospect of a prolonged delay in publication might well have contributed to his depression and his death.
The author who wrote on October 6th outlined some options. One was to leave the paper as it was, but to include all the raw data and submit to a journal such as Plos One or the preprint server BioRxiv. This option “requires minimal work, and would result in no change in the author list. However we would aim for a lower-impact journal.”. His preferred option, though, was to rewrite the paper altogether (and for himself to become first co-author) “as it is in all our shared interest to get the work published in as good a journal as possible. “.
Two days later, on October 8th, the same author thanked his co-authors for their responses. As a result of the responses he got, he asked to have his name removed from the paper because he did not agree with what was contained in the manuscript. “However, given that I believe the gene list is wrong, I request my name to be removed from the author list. If any other authors do not wish for the raw data to be disclosed then I hope you think it’s reasonable for me to close off my involvement with the paper.”.
The paper has 11 authors, including Stefan Grimm. . I have written to all but one of the authors to try to ascertain the facts. Of the four co-authors who have replied, all but one said that they hadn’t seen the final paper. One said that they were unaware that they were on the author list, and said they probably shouldn’t be.
I have tried to protect the authors (some of whom are still at Imperial) by not mentioning their names. But one co-author is sufficiently senior to be mentioned by name. Alan Boobis answered by my mail cordially enough when I first wrote to him, but declined to give much useful information, apart from confirming that Grimm was the senior author on the paper. On October 9th he wrote to all co-authors, thus.
|
From: Boobis, Alan R [a.boobis@imperial.ac.uk] Dear all The situation regarding this manuscript needs to be dealt with rationally. There is a real danger that the reputations of individuals and of the College will be harmed. I suggest that we all need to agree the most appropriate way forward. I am out of the country this week but will have my secretary liaise with you next week to arrange a suitable time (face to face or by phone) to discuss this. Best wishes, Alan |
I have no idea what the outcome of this meeting was. Personally. I always worry a bit when people want meetings “face to face or by phone”. Written records are much more informative.
I should like to make it clear that I’m not suggesting any misconduct whatsoever. The author who wished to withdraw acted with principle and courage, and mistakes happen. They are perhaps especially likely in multi-author papers where some authors don’t understand the input from others. But it is sad to see the emphasis on the long-discredited journal impact factor that was forced on them by Imperial’s policies. And it’s sad to see that several co-authors had not actually seen the final paper. This smacks of “citation-mongering”, yet another bad effect of the metrics culture that has pervaded all of academia, and which is enforced in an especially simple-minded way at Imperial.
This sad episode is yet another illustration of the way that Imperial’s policies are damaging people, and, in the end, damaging science.
Some discussions of the Imperial problem
Since Grimm’s last email was revealed, it’s been discussed in many blogs and articles. Here are a few of them.
Grimm’s tale (2 December). This perceptive blog reproduces part of the nasty threatening letter sent by Martin Wilkins to Grimm.
“Your current level of funding does not constitute the appropriate level for a professor at Imperial College. Unless you submit and are awarded a Platform grant as PI in the next 12 months we will seek to initiate disciplinary action against you.”
The author comments (my emphasis).
“This isn’t about science – it’s about bragging rights, or institutional willy-waving. Grimm was informed – in public – that he was to be fired, and left waiting for the axe to fall while the axe-wielder marauded around the campus boasting about it like an even more pathetic Alan Sugar.”
That sums it up for me. It’s very sad.
Another blog comments
“Martin Wilkins to Professor Stefan Grimm, a few months before the latter committed suicide. Imperial College had been pressuring Grimm to get 200, 000 pounds in grants in order for him to remain employed. They threatened to sack him as he only had 135,000 pounds.
Sounds a lot like loan sharks.”
Clearly universities like Imperial are no longer places for scholarship. They are more like anxiety machines.
The Nuffield Council on Bioethics produced an important report in the midst of the scandal about Grimm: The culture of scientific research 2014. Paragraph 1.7 produced a chilling statistic
1.7 Compromising on research integrity and standards
- Almost six in ten (58%) respondents are aware of scientists feeling under pressure to
compromise on research integrity and standards, with poor methodology and data fraud
frequently mentioned in the free text responses.- Just over a quarter (26%) of those taking part in the survey have felt tempted to compromise
on research integrity.
Stefan Grimm and the British University system. This blog, written by a geneticist. Federico Calboli, based in Helsinki, gives an indication of the harm that Imperial is inflicting not only on itself, but on the whole of UK academia, and hence on the UK economy
“As always in the real world the best laid plans often conflict with how the world actually works, and this conflict gives rise to a number of unintended consequences. The first unintended consequence is that the pursuit of what managements defines as ‘novel’ and ‘glamorous’ will diminish the intellectual value of British academia as a whole.”
“Unfortunately, since academia, funding bodies and the editorial boards of papers have been taken over by top down management culture, solid rigorous science is penalised in favour of anything that can be branded as ’novel’, ‘cutting edge’, ‘state of the art’ and similar platitudes.”
“This policy will leave British academia directionless and intellectually empty, and will transform any research in technology and data driven drivel that can at most pick up low hanging fruits and will deliver less and less as time goes on.”
Still more shaming, Calboli continues thus.
“The second problem with how British academia is managed is the culture of intellectual dishonesty that is forced upon people. People are not allowed to just express their goals in simple honest terms. They are required to spin and embellish everything in order to have half a chance of getting some funding or publishing in a high impact journal – both crucial to contribute to the ‘excellence metric”.
“Only the shameless cynics thrive in such environment”.
The blog finishes with a rallying cry.
“the email that Prof Grimm sent in October did not magically make its way to the press by itself. While many people are feeling disenchanted with academia and leave, more and more insiders are taking a combative stance against the mindless hogwash that threatens the foundations of British academia and the people that push it. We should all stand up and be counted, or we will not be able to complain in the future. It would be great if management could live up to its role and abandon the idea that scientific research is simple, predictable and quickly profitable, and actually help build the future of British academia.”
All this reflects similar sentiments to those that I expressed in 2007 [the RAE was the predecessor of the REF]
“The policies described here will result in a generation of ‘spiv’ scientists, churning out 20 or even more papers a year, with very little originality. They will also, inevitably, lead to an increase in the sort of scientific malpractice that was recently pilloried viciously, but accurately, in the New York Times, and a further fall in the public’s trust in science. That trust is already disastrously low, and one reason for that is, I suggest, pressures like those described here which lead scientists to publish when they have nothing to say.”
““All of us who do research (rather than talk about it) know the disastrous effects that the Research Assessment Exercise has had on research in the United Kingdom: short-termism, intellectual shallowness, guest authorships and even dishonesty”. Now we can add to that list bullying, harassment and an incompetent box-ticking style of assessment that tends to be loved by HR departments.
This process might indeed increase your RAE score in the short term (though there is no evidence that it it does even that). But, over a couple of decades, it will rid universities of potential Nobel prize winners.”
Conclusions
The policies adopted by Imperial College have harmed Imperial’s reputation throughout the world. Worse still, they have tainted the reputation of all UK universities. They have contributed to the corruption of science. and they have, in all probability, killed a successful man,
I hope that Alice Gast (president), James Stirling (provost). Dermot Kelleher (Dean, now vice president), and Martin Wilkins (who was left to weild the knife) have a good Christmas. If I were in their shoes, I’d feel so guilty that I wouldn’t be able to sleep at night.
They should spend Christmas reading Peter Lawrence’s wonderful essay on The Mismeasurement of Science. Please download a copy
Their proposal that HR policies should be investigated by, inter alia, the head of HR has provoked worldwide derision.
Their refusal to set up an independent external inquiry is reprehensible.
Not for the first time, a fine institution is being brought into disrepute by its leadership. Council please note.
|
Alice Gast |
James Stirling |
Dermot Kelleher |
Perhaps the best description of what’s going on is from Grimm’s Tale “This isn’t about science – it’s about bragging rights, or institutional willy-waving.”. Gast, Stirling and Kelleher should stop the willy-waving. They should either set about rectifying the damage they’ve done. Or they should resign. Now.
The chair of universities HR association, Kim Frost, said
“Bullying is a very emotive term, and what one person experiences as bullying will often be simple performance management from their manager’s point of view.”.
That’s scary because it shows that she hasn’t the slightest idea about “performance management”. I have news for HR people. They are called experiments because we don’t know whether they will work. If they don’t work that’s not a reason to fire anyone. No manager can make an experiment come out as they wish. The fact of the matter is that it’s impossible to manage research. If you want real innovation you have to tolerate lots and lots of failure. “Performance management” is an oxymoron. Get used to it.
This sorry episode has far more general lessons for the way the REF is conducted and for the metrics sales industry. Both share some of the guilt.
That will have to wait for another post.
Follow-up
25 December 2014. Universities "eliminate tenure because Starbucks does not have tenure"
I was struck by this excerpt from a Christmas newsletter from a colleague. Buried among the family news was buried this lament. He’s writing about Rush University, Chicago, but much the same could be said about many universities, not only in the USA.
|
Rush Medical Center built an $800 million hospital building that is clinically state-of-the-art and architecturally unique. Now it is poised to become a world class center of basic and clinical research. Sadly, rather than listen to researchers who have devoted their careers to Rush, senior administration hears advice from fly-by-night financial consultants who apply the same “Business Model” to medical care, education, and research as to a shoe factory. Perhaps because fiscal consulting requires little skill or training*, they do not distinguish between a researcher and a Starbucks employee [literally true!]. They eliminate tenure because Starbucks does not have tenure. {To be fair, they have only eliminated “tenure of salary” – one may continue working with a title, but without pay!} They cannot imagine that world-class research is an art that requires years of training, cultivating an international network of colleagues, and most importantly, continuity of funding. Because their work is so trivial, they cannot fathom that researchers could be utterly unique and irreplacable. And they do not care – they will destroy research at Rush, collect their multi-million dollar fee, and move on to the next shoe factory. *Lesson 1. Fire people who do real work, cut wages, steal from pension funds, eliminate unions and job security. Congratulations you are now a qualified fiscal consultant! |
26 December 2014
|
Grimm is not the only one. In the same month, September 2014, Tony Veitch was found dead. He was a senior scientist in the lab at Kew Botanical Gardens. He was 49, much the same age as Stefan Grimm. It’s presumed that he committed suicide after being told to reapply for his own job. |
 |
!7 January 2015
I hear that Imperial College’s UCU passed this motion.
|
Motion 3: Branch condemns bullying and harassment of staff at Imperial This branch strongly condemns the bullying and harassment of staff at Imperial, particularly by some managers. We call upon the senior management of the College to ensure that all managers are properly trained to deal with staff in a fair and considerate manner and on how to refrain from bullying and harassment. In light of a recent tragic case at Imperial, the College management must ensure that they fulfil their duty of care to all staff at all times. |
Of course every employer claims that they do this.
I wonder how the officials can mouth these platitudes when the facts, now well known, show them to be untrue,
The first post and this one have been viewed over 173.000 times, from at least 170 countries (UK, USA,and then almost 10,000 views from China). I realise that this must have harmed Imperial, but they have brought it on themselves. Neither the president nor the rector have had the courtesy to answer perfectly polite letters.
I wrote also on 29 December to the chair of Council. Eliza Manningham-Buller. She has still not acknowledged receipt, never mind replied.
I am amazed by the discourtesy of people who regard themselves as too important to reply to letters.
|
To chair of Council, Imperial College London 29 December 2014 Dear Lady Manningham-Buller A problem with management at Imperial It cannot have escaped your notice that a senior member of Imperial’s staff was found dead, after being told that he’d lose his job if he didn’t raise £200,000 in grants within a year. When I posted Stefan Grimm’s last email on my blog on December 1st it went viral (Publish and perish at Imperial College London), It has been read by over 160,000 people from over 200 countries. That being the case, Imperial’s first official mention of the matter on December 4th looked pretty silly. It was written as though his email was not already common knowledge –totally hamfisted public relations. After posting Grimm’s last mail, I was deluged with mails about people who had been badly treated at Imperial. I posted a few of them on December 23rd (Some experiences of life at Imperial College London. An external inquiry is needed after the death of Stefan Grimm). The policy of telling staff that their research must be expensive is not likely to be appreciated by the taxpayer. Neither will it improve the quality of science. On the contrary, the actions of the College are very likely to deter good scientists from working there (I already heard of two examples of people who turned down jobs at Imperial). I think it is now clear that the senior management team is pursuing policies that are damaging the reputation of Imperial. I hope that Council will take appropriate action. Best regards David Colquhoun |
20 January 2015
Today I got a reply to the letter (above) that I sent to Eliza Manningham-Buller on 29 December. You can download it.
I guess it’s not surprising that the reply says nothing helpful.
It endorses the idea that HR should investigate their own practices, an idea that the outside world greets with ridicule.
It reprimands me for making "unprofessional" comments about individuals. That’s what happens when people behave badly. It would be unprofessional to fail to point out what’s going on. It’s the job of journalists to name people. All else is PR.
It suggests that I may have not followed the Samaritans’ guidelines for reporting of suicide. I’ve read their document and I don’t believe that either I, or Times Higher Education, have breached the guidelines.
The letter says. essentially, please shut up, you are embarrassing Imperial. It’s fascinating to see the rich and powerful close ranks when criticised. But it is very disappointing. It seems to me to be very much in the public interest to have published the last email of Stefan Grimm.
But I guess the last person you’d expect to champion transparency is an ex-head of MI5.
Felix, Imperial’s student newspaper, carried an interesting article Death of Professor Grimm: the world reacts. The events at Imperial have been noted all over the world (at least 170 countries according to my own Google analytics) but the response has been especially big in China. Alienating a country like China seems to me to rank as bringing the College into disrepute.
9 February 2015
Death in Academia and the mis-measurement of science. Good article in Euroscientist by Arren Frood
25 February 2015
I see that Dermot Kelleher is leaving Imperial for the University of British Columbia. Perhaps he hopes that he’ll be able to escape his share of the blame for the death of Stefan Grimm? Let’s hope, for the sake of UBC, that he’s learned a lesson from the episode.
10 March 2015
The Vancouver Sun has been asking questions. An article by Pamela Fayerman includes the following.
"Recently, Imperial College was engulfed in a controversy involving a tragedy. . . . a medical school professor, Stefan Grimm, took his own life last fall. He left an email that accused unnamed superiors of bullying through demands that he garner more research grants. The “publish or peril” adage that scientists so often cite seems like it may apply in this case.
The college said it would set up an internal inquiry into the circumstances around the toxicology professor’s death, but the results have not been released. UBC provost Dave Farrar said in an interview that the death of the professor at Imperial College was never even discussed during the recruitment process.
Kelleher said in a long distance phone interview that the tragedy had nothing to do with his reasons for leaving Imperial. And he can’t speak about the case since it is currently under review by a coroner."
Well, I guess he would say that, wouldn’t he? Kelleher has been at Imperial for less than three years, and the generous intepretation of his departure is that he didn’t like the bullying regime. It had been going on long before Kelleher arrived, as documented on this blog in in 2007.
It’s interesting to speculate about why he wasn’t asked about Grimm’s death (if that’s true).
Did the University of British Columbia think it was irrelevant?
Or did they want him to establish a similar regime of “performance management” at UBC?
Or were the senior people at UBC not even aware of the incident?
Perhaps the third option is the most likely: it’s only too characteristic of senior managers to be unaware of what’s happening on the shop floor. Just as in banks.
11 March 2015
It’s beginning to look like an exodus. The chair of Imperial’s council, Eliza Manningham-Buller, is also leaving. Despite her condescending response to my inquiries, perhaps she too is scared of what will be revealed about bullying. I just hope that she doesn’t bring Imperial’s ideas about "performance management" to the Wellcome Trust.
The Higher Education Funding Council England (HEFCE) gives money to universities. The allocation that a university gets depends strongly on the periodical assessments of the quality of their research. Enormous amounts if time, energy and money go into preparing submissions for these assessments, and the assessment procedure distorts the behaviour of universities in ways that are undesirable. In the last assessment, four papers were submitted by each principal investigator, and the papers were read.
In an effort to reduce the cost of the operation, HEFCE has been asked to reconsider the use of metrics to measure the performance of academics. The committee that is doing this job has asked for submissions from any interested person, by June 20th.
This post is a draft for my submission. I’m publishing it here for comments before producing a final version for submission.
Draft submission to HEFCE concerning the use of metrics.
I’ll consider a number of different metrics that have been proposed for the assessment of the quality of an academic’s work.
Impact factors
The first thing to note is that HEFCE is one of the original signatories of DORA (http://am.ascb.org/dora/ ). The first recommendation of that document is
:"Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions"
.Impact factors have been found, time after time, to be utterly inadequate as a way of assessing individuals, e.g. [1], [2]. Even their inventor, Eugene Garfield, says that. There should be no need to rehearse yet again the details. If HEFCE were to allow their use, they would have to withdraw from the DORA agreement, and I presume they would not wish to do this.
Article citations
Citation counting has several problems. Most of them apply equally to the H-index.
- Citations may be high because a paper is good and useful. They equally may be high because the paper is bad. No commercial supplier makes any distinction between these possibilities. It would not be in their commercial interests to spend time on that, but it’s critical for the person who is being judged. For example, Andrew Wakefield’s notorious 1998 paper, which gave a huge boost to the anti-vaccine movement had had 758 citations by 2012 (it was subsequently shown to be fraudulent).
- Citations take far too long to appear to be a useful way to judge recent work, as is needed for judging grant applications or promotions. This is especially damaging to young researchers, and to people (particularly women) who have taken a career break. The counts also don’t take into account citation half-life. A paper that’s still being cited 20 years after it was written clearly had influence, but that takes 20 years to discover,
- The citation rate is very field-dependent. Very mathematical papers are much less likely to be cited, especially by biologists, than more qualitative papers. For example, the solution of the missed event problem in single ion channel analysis [3,4] was the sine qua non for all our subsequent experimental work, but the two papers have only about a tenth of the number of citations of subsequent work that depended on them.
- Most suppliers of citation statistics don’t count citations of books or book chapters. This is bad for me because my only work with over 1000 citations is my 105 page chapter on methods for the analysis of single ion channels [5], which contained quite a lot of original work. It has had 1273 citations according to Google scholar but doesn’t appear at all in Scopus or Web of Science. Neither do the 954 citations of my statistics text book [6]
- There are often big differences between the numbers of citations reported by different commercial suppliers. Even for papers (as opposed to book articles) there can be a two-fold difference between the number of citations reported by Scopus, Web of Science and Google Scholar. The raw data are unreliable and commercial suppliers of metrics are apparently not willing to put in the work to ensure that their products are consistent or complete.
- Citation counts can be (and already are being) manipulated. The easiest way to get a large number of citations is to do no original research at all, but to write reviews in popular areas. Another good way to have ‘impact’ is to write indecisive papers about nutritional epidemiology. That is not behaviour that should command respect.
- Some branches of science are already facing something of a crisis in reproducibility [7]. One reason for this is the perverse incentives which are imposed on scientists. These perverse incentives include the assessment of their work by crude numerical indices.
- “Gaming” of citations is easy. (If students do it it’s called cheating: if academics do it is called gaming.) If HEFCE makes money dependent on citations, then this sort of cheating is likely to take place on an industrial scale. Of course that should not happen, but it would (disguised, no doubt, by some ingenious bureaucratic euphemisms).
- For example, Scigen is a program that generates spoof papers in computer science, by stringing together plausible phases. Over 100 such papers have been accepted for publication. By submitting many such papers, the authors managed to fool Google Scholar in to awarding the fictitious author an H-index greater than that of Albert Einstein http://en.wikipedia.org/wiki/SCIgen
- The use of citation counts has already encouraged guest authorships and such like marginally honest behaviour. There is no way to tell with an author on a paper has actually made any substantial contribution to the work, despite the fact that some journals ask for a statement about contribution.
- It has been known for 17 years that citation counts for individual papers are not detectably correlated with the impact factor of the journal in which the paper appears [1]. That doesn’t seem to have deterred metrics enthusiasts from using both. It should have done.
Given all these problems, it’s hard to see how citation counts could be useful to the REF, except perhaps in really extreme cases such as papers that get next to no citations over 5 or 10 years.
The H-index
This has all the disadvantages of citation counting, but in addition it is strongly biased against young scientists, and against women. This makes it not worth consideration by HEFCE.
Altmetrics
Given the role given to “impact” in the REF, the fact that altmetrics claim to measure impact might make them seem worthy of consideration at first sight. One problem is that the REF failed to make a clear distinction between impact on other scientists is the field and impact on the public.
Altmetrics measures an undefined mixture of both sorts if impact, with totally arbitrary weighting for tweets, Facebook mentions and so on. But the score seems to be related primarily to the trendiness of the title of the paper. Any paper about diet and health, however poor, is guaranteed to feature well on Twitter, as will any paper that has ‘penis’ in the title.
It’s very clear from the examples that I’ve looked at that few people who tweet about a paper have read more than the title. See Why you should ignore altmetrics and other bibliometric nightmares [8].
In most cases, papers were promoted by retweeting the press release or tweet from the journal itself. Only too often the press release is hyped-up. Metrics not only corrupt the behaviour of academics, but also the behaviour of journals. In the cases I’ve examined, reading the papers revealed that they were particularly poor (despite being in glamour journals): they just had trendy titles [8].
There could even be a negative correlation between the number of tweets and the quality of the work. Those who sell altmetrics have never examined this critical question because they ignore the contents of the papers. It would not be in their commercial interests to test their claims if the result was to show a negative correlation. Perhaps the reason why they have never tested their claims is the fear that to do so would reduce their income.
Furthermore you can buy 1000 retweets for $8.00 http://followers-and-likes.com/twitter/buy-twitter-retweets/ That’s outright cheating of course, and not many people would go that far. But authors, and journals, can do a lot of self-promotion on twitter that is totally unrelated to the quality of the work.
It’s worth noting that much good engagement with the public now appears on blogs that are written by scientists themselves, but the 3.6 million views of my blog do not feature in altmetrics scores, never mind Scopus or Web of Science. Altmetrics don’t even measure public engagement very well, never mind academic merit.
Evidence that metrics measure quality
Any metric would be acceptable only if it measured the quality of a person’s work. How could that proposition be tested? In order to judge this, one would have to take a random sample of papers, and look at their metrics 10 or 20 years after publication. The scores would have to be compared with the consensus view of experts in the field. Even then one would have to be careful about the choice of experts (in fields like alternative medicine for example, it would be important to exclude people whose living depended on believing in it). I don’t believe that proper tests have ever been done (and it isn’t in the interests of those who sell metrics to do it).
The great mistake made by almost all bibliometricians is that they ignore what matters most, the contents of papers. They try to make inferences from correlations of metric scores with other, equally dubious, measures of merit. They can’t afford the time to do the right experiment if only because it would harm their own “productivity”.
The evidence that metrics do what’s claimed for them is almost non-existent. For example, in six of the ten years leading up to the 1991 Nobel prize, Bert Sakmann failed to meet the metrics-based publication target set by Imperial College London, and these failures included the years in which the original single channel paper was published [9] and also the year, 1985, when he published a paper [10] that was subsequently named as a classic in the field [11]. In two of these ten years he had no publications whatsoever. See also [12].
Application of metrics in the way that it’s been done at Imperial and also at Queen Mary College London, would result in firing of the most original minds.
Gaming and the public perception of science
Every form of metric alters behaviour, in such a way that it becomes useless for its stated purpose. This is already well-known in economics, where it’s know as Goodharts’s law http://en.wikipedia.org/wiki/Goodhart’s_law “"When a measure becomes a target, it ceases to be a good measure”. That alone is a sufficient reason not to extend metrics to science. Metrics have already become one of several perverse incentives that control scientists’ behaviour. They have encouraged gaming, hype, guest authorships and, increasingly, outright fraud [13].
The general public has become aware of this behaviour and it is starting to do serious harm to perceptions of all science. As long ago as 1999, Haerlin & Parr [14] wrote in Nature, under the title How to restore Public Trust in Science,
“Scientists are no longer perceived exclusively as guardians of objective truth, but also as smart promoters of their own interests in a media-driven marketplace.”
And in January 17, 2006, a vicious spoof on a Science paper appeared, not in a scientific journal, but in the New York Times. See https://www.dcscience.net/?p=156
The use of metrics would provide a direct incentive to this sort of behaviour. It would be a tragedy not only for people who are misjudged by crude numerical indices, but also a tragedy for the reputation of science as a whole.
Conclusion
There is no good evidence that any metric measures quality, at least over the short time span that’s needed for them to be useful for giving grants or deciding on promotions). On the other hand there is good evidence that use of metrics provides a strong incentive to bad behaviour, both by scientists and by journals. They have already started to damage the public perception of science of the honesty of science.
The conclusion is obvious. Metrics should not be used to judge academic performance.
What should be done?
If metrics aren’t used, how should assessment be done? Roderick Floud was president of Universities UK from 2001 to 2003. He’s is nothing if not an establishment person. He said recently:
“Each assessment costs somewhere between £20 million and £100 million, yet 75 per cent of the funding goes every time to the top 25 universities. Moreover, the share that each receives has hardly changed during the past 20 years.
It is an expensive charade. Far better to distribute all of the money through the research councils in a properly competitive system.”
The obvious danger of giving all the money to the Research Councils is that people might be fired solely because they didn’t have big enough grants. That’s serious -it’s already happened at Kings College London, Queen Mary London and at Imperial College. This problem might be ameliorated if there were a maximum on the size of grants and/or on the number of papers a person could publish, as I suggested at the open data debate. And it would help if univerities appointed vice-chancellors with a better long term view than most seem to have at the moment.
Aggregate metrics? It’s been suggested that the problems are smaller if one looks at aggregated metrics for a whole department. rather than the metrics for individual people. Clearly looking at departments would average out anomalies. The snag is that it wouldn’t circumvent Goodhart’s law. If the money depended on the aggregate score, it would still put great pressure on universities to recruit people with high citations, regardless of the quality of their work, just as it would if individuals were being assessed. That would weigh against thoughtful people (and not least women).
The best solution would be to abolish the REF and give the money to research councils, with precautions to prevent people being fired because their research wasn’t expensive enough. If politicians insist that the "expensive charade" is to be repeated, then I see no option but to continue with a system that’s similar to the present one: that would waste money and distract us from our job.
1. Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. British Medical Journal 314: 498-502. [Download pdf]
2. Colquhoun D (2003) Challenging the tyranny of impact factors. Nature 423: 479. [Download pdf]
3. Hawkes AG, Jalali A, Colquhoun D (1990) The distributions of the apparent open times and shut times in a single channel record when brief events can not be detected. Philosophical Transactions of the Royal Society London A 332: 511-538. [Get pdf]
4. Hawkes AG, Jalali A, Colquhoun D (1992) Asymptotic distributions of apparent open times and shut times in a single channel record allowing for the omission of brief events. Philosophical Transactions of the Royal Society London B 337: 383-404. [Get pdf]
5. Colquhoun D, Sigworth FJ (1995) Fitting and statistical analysis of single-channel records. In: Sakmann B, Neher E, editors. Single Channel Recording. New York: Plenum Press. pp. 483-587.
6. David Colquhoun on Google Scholar. Available: http://scholar.google.co.uk/citations?user=JXQ2kXoAAAAJ&hl=en17-6-2014
7. Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2: e124.[full text]
8. Colquhoun D, Plested AJ Why you should ignore altmetrics and other bibliometric nightmares. Available: https://www.dcscience.net/?p=6369
9. Neher E, Sakmann B (1976) Single channel currents recorded from membrane of denervated frog muscle fibres. Nature 260: 799-802.
10. Colquhoun D, Sakmann B (1985) Fast events in single-channel currents activated by acetylcholine and its analogues at the frog muscle end-plate. J Physiol (Lond) 369: 501-557. [Download pdf]
11. Colquhoun D (2007) What have we learned from single ion channels? J Physiol 581: 425-427.[Download pdf]
12. Colquhoun D (2007) How to get good science. Physiology News 69: 12-14. [Download pdf] See also https://www.dcscience.net/?p=182
13. Oransky, I. Retraction Watch. Available: http://retractionwatch.com/18-6-2014
14. Haerlin B, Parr D (1999) How to restore public trust in science. Nature 400: 499. 10.1038/22867 [doi].[Get pdf]
Follow-up
Some other posts on this topic
Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation
Gaming Google Scholar Citations, Made Simple and Easy
Manipulating Google Scholar Citations and Google Scholar Metrics: simple, easy and tempting
Driving Altmetrics Performance Through Marketing
Death by Metrics (October 30, 2013)
Not everything that counts can be counted
Using metrics to assess research quality By David Spiegelhalter “I am strongly against the suggestion that peer–review can in any way be replaced by bibliometrics”
1 July 2014
My brilliant statistical colleague, Alan Hawkes, not only laid the foundations for single molecule analysis (and made a career for me) . Before he got into that, he wrote a paper, Spectra of some self-exciting and mutually exciting point processes, (Biometrika 1971). In that paper he described a sort of stochastic process now known as a Hawkes process. In the simplest sort of stochastic process, the Poisson process, events are independent of each other. In a Hawkes process, the occurrence of an event affects the probability of another event occurring, so, for example, events may occur in clusters. Such processes were used for many years to describe the occurrence of earthquakes. More recently, it’s been noticed that such models are useful in finance, marketing, terrorism, burglary, social media, DNA analysis, and to describe invasive banana trees. The 1971 paper languished in relative obscurity for 30 years. Now the citation rate has shot threw the roof.
The papers about Hawkes processes are mostly highly mathematical. They are not the sort of thing that features on twitter. They are serious science, not just another ghastly epidemiological survey of diet and health. Anybody who cites papers of this sort is likely to be a real scientist. The surge in citations suggests to me that the 1971 paper was indeed an important bit of work (because the citations will be made by serious people). How does this affect my views about the use of citations? It shows that even highly mathematical work can achieve respectable citation rates, but it may take a long time before their importance is realised. If Hawkes had been judged by citation counting while he was applying for jobs and promotions, he’d probably have been fired. If his department had been judged by citations of this paper, it would not have scored well. It takes a long time to judge the importance of a paper and that makes citation counting almost useless for decisions about funding and promotion.
Stop press. Financial report casts doubt on Trainor’s claims
Science has a big problem. Most jobs are desperately insecure. It’s hard to do long term thorough work when you don’t know whether you’ll be able to pay your mortgage in a year’s time. The appalling career structure for young scientists has been the subject of much writing by the young (e.g. Jenny Rohn) and the old, e.g Bruce Alberts. Peter Lawrence (see also Real Lives and White Lies in the Funding of Scientific Research, and by me.
Until recently, this problem was largely restricted to post-doctoral fellows (postdocs). They already have PhDs and they are the people who do most of the experiments. Often large numbers of them work for a single principle investigator (PI). The PI spends most of his her time writing grant applications and traveling the world to hawk the wares of his lab. They also (to variable extents) teach students and deal with endless hassle from HR.
The salaries of most postdocs are paid from grants that last for three or sometimes five years. If that grant doesn’t get renewed. they are on the streets.
Universities have come to exploit their employees almost as badly as Amazon does.
The periodical research assessments not only waste large amounts of time and money, but they have distorted behaviour. In the hope of scoring highly, they recruit a lot of people before the submission, but as soon as that’s done with, they find that they can’t afford all of them, so some get cast aside like worn out old boots. Universities have allowed themselves to become dependent on "soft money" from grant-giving bodies. That strikes me as bad management.
The situation is even worse in the USA where most teaching staff rely on research grants to pay their salaries.
I have written three times about the insane methods that are being used to fire staff at Queen Mary College London (QMUL).
Is Queen Mary University of London trying to commit scientific suicide? (June 2012)
Queen Mary, University of London in The Times. Does Simon Gaskell care? (July 2012) and a version of it appeared th The Times (Thunderer column)
In which Simon Gaskell, of Queen Mary, University of London, makes a cock-up (August 2012)
The ostensible reason given there was to boost its ratings in university rankings. Their vice-chancellor, Simon Gaskell, seems to think that by firing people he can produce a university that’s full of Nobel prize-winners. The effect, of course, is just the opposite. Treating people like pawns in a game makes the good people leave and only those who can’t get a job with a better employer remain. That’s what I call bad management.
At QMUL people were chosen to be fired on the basis of a plain silly measure of their publication record, and by their grant income. That was combined with terrorisation of any staff who spoke out about the process (more on that coming soon).
Kings College London is now doing the same sort of thing. They have announced that they’ll fire 120 of the 777 staff in the schools of medicine and biomedical sciences, and the Institute of Psychiatry. These are humans, with children and mortgages to pay. One might ask why they were taken on the first place, if the university can’t afford them. That’s simply bad financial planning (or was it done in order to boost their Research Excellence submission?).
Surely it’s been obvious, at least since 2007, that hard financial times were coming, but that didn’t dent the hubris of the people who took an so many staff. HEFCE has failed to find a sensible way to fund universities. The attempt to separate the funding of teaching and research has just led to corruption.
The way in which people are to be chosen for the firing squad at Kings is crude in the extreme. If you are a professor at the Institute of Psychiatry then, unless you do a lot of teaching, you must have a grant income of at least £200,000 per year. You can read all the details in the Kings’ “Consultation document” that was sent to all employees. It’s headed "CONFIDENTIAL – Not for further circulation". Vice-chancellors still don’t seem to have realised that it’s no longer possible to keep things like this secret. In releasing it, I take ny cue from George Orwell.
"Journalism is printing what someone else does not want printed: everything else is public relations.”
There is no mention of the quality of your research, just income. Since in most sorts of research, the major cost is salaries, this rewards people who take on too many employees. Only too frequently, large groups are the ones in which students and research staff get the least supervision, and which bangs per buck are lowest. The university should be rewarding people who are deeply involved in research themselves -those with small groups. Instead, they are doing exactly the opposite.
Women are, I’d guess, less susceptible to the grandiosity of the enormous research group, so no doubt they will suffer disproportionately. PhD students will also suffer if their supervisor is fired while they are halfway through their projects.
An article in Times Higher Education pointed out
"According to the Royal Society’s 2010 report The Scientific Century: Securing our Future Prosperity, in the UK, 30 per cent of science PhD graduates go on to postdoctoral positions, but only around 4 per cent find permanent academic research posts. Less than half of 1 per cent of those with science doctorates end up as professors."
The panel that decides whether you’ll be fired consists of Professor Sir Robert Lechler, Professor Anne Greenough, Professor Simon Howell, Professor Shitij Kapur, Professor Karen O’Brien, Chris Mottershead, Rachel Parr & Carol Ford. If they had the slightest integrity, they’d refuse to implement such obviously silly criteria.
Universities in general. not only Kings and QMUL have become over-reliant on research funders to enhance their own reputations. PhD students and research staff are employed for the benefit of the university (and of the principle investigator), not for the benefit of the students or research staff, who are treated as expendable cost units, not as humans.
One thing that we expect of vice-chancellors is sensible financial planning. That seems to have failed at Kings. One would also hope that they would understand how to get good science. My only previous encounter with Kings’ vice chancellor, Rick Trainor, suggests that this is not where his talents lie. While he was president of the Universities UK (UUK), I suggested to him that degrees in homeopathy were not a good idea. His response was that of the true apparatchik.
“. . . degree courses change over time, are independently assessed for academic rigour and quality and provide a wider education than the simple description of the course might suggest”
That is hardly a response that suggests high academic integrity.
The students’ petition is on Change.org.
Follow-up
The problems that are faced in the UK are very similar to those in the USA. They have been described with superb clarity in “Rescuing US biomedical research from its systemic flaws“, This article, by Bruce Alberts, Marc W. Kirschner, Shirley Tilghman, and Harold Varmus, should be read by everyone. They observe that ” . . . little has been done to reform the system, primarily because it continues to benefit more established and hence more influential scientists”. I’d be more impressed by the senior people at Kings if they spent time trying to improve the system rather than firing people because their research is not sufficiently expensive.
10 June 2014
Progress on the cull, according to an anonymous correspondent
“The omnishambles that is KCL management
1) We were told we would receive our orange (at risk) or green letters (not at risk, this time) on Thursday PM 5th June as HR said that it’s not good to get bad news on a Friday!
2) We all got a letter on Friday that we would not be receiving our letters until Monday, so we all had a tense weekend
3) I finally got my letter on Monday, in my case it was “green” however a number of staff who work very hard at KCL doing teaching and research are “orange”, un bloody believable
As you can imagine the moral at King’s has dropped through the floor”
18 June 2014
Dorothy Bishop has written about the Trainor problem. Her post ends “One feels that if KCL were falling behind in a boat race, they’d respond by throwing out some of the rowers”.
The students’ petition can be found on the #KCLHealthSOS site. There is a reply to the petition, from Professor Sir Robert Lechler, and a rather better written response to it from students. Lechler’s response merely repeats the weasel words, and it attacks a few straw men without providing the slightest justification for the criteria that are being used to fire people. One can’t help noticing how often knighthoods go too the best apparatchiks rather than the best scientists.
14 July 2014
A 2013 report on Kings from Standard & Poor’s casts doubt on Trainor’s claims
Download the report from Standard and Poor’s Rating Service
A few things stand out.
- KCL is in a strong financial position with lower debt than other similar Universities and cash reserves of £194 million.
- The report says that KCL does carry some risk into the future especially that related to its large capital expansion program.
- The report specifically warns KCL over the consequences of any staff cuts. Particularly relevant are the following quotations
- Page p3 “Further staff-cost curtailment will be quite difficult …pressure to maintain its academic and non-academic service standards will weigh on its ability to cut costs further.”
- page 4 The report goes on to say (see the section headed outlook, especially the final paragraph) that any decrease in KCL’s academic reputation (e.g. consequent on staff cuts) would be likely to impair its ability to attract overseas students and therefore adversely affect its financial position.
- page 10 makes clear that KCL managers are privately aiming at 10% surplus, above the 6% operating surplus they talk about with us. However, S&P considers that ‘ambitious’. In other words KCL are shooting for double what a credit rating agency considers realistic.
One can infer from this that
- what staff have been told about the cuts being an immediate necessity is absolute nonsense
- KCL was warned against staff cuts by a credit agency
- the main problem KCL has is its overambitious building policy
- KCL is implementing a policy (staff cuts) which S & P warned against as they predict it may result in diminishing income.
What on earth is going on?
16 July 2014
I’ve been sent yet another damning document. The BMA’s response to Kings contains some numbers that seem to have escaped the attention of managers at Kings.
10 April 2015
King’s draft performance management plan for 2015
This document has just come to light (the highlighting is mine).
It’s labelled as "released for internal consultation". It seems that managers are slow to realise that it’s futile to try to keep secrets.
The document applies only to Institute of Psychiatry, Psychology and Neuroscience at King’s College London: "one of the global leaders in the fields" -the usual tedious blah that prefaces every document from every university.
It’s fascinating to me that the most cruel treatment of staff so often seems to arise in medical-related areas. I thought psychiatrists, of all people, were meant to understand people, not to kill them.
This document is not quite as crude as Imperial’s assessment, but it’s quite bad enough. Like other such documents, it pretends that it’s for the benefit of its victims. In fact it’s for the benefit of willy-waving managers who are obsessed by silly rankings.
Here are some of the sillier bits.
"The Head of Department is also responsible for ensuring that aspects of reward/recognition and additional support that are identified are appropriately followed through"
And, presumably, for firing people, but let’s not mention that.
"Academics are expected to produce original scientific publications of the highest quality that will significantly advance their field."
That’s what everyone has always tried to do. It can’t be compelled by performance managers. A large element of success is pure luck. That’s why they’re called experiments.
" However, it may take publications 12-18 months to reach a stable trajectory of citations, therefore, the quality of a journal (impact factor) and the judgment of knowledgeable peers can be alternative indicators of excellence."
It can also take 40 years for work to be cited. And there is little reason to believe that citations, especially those within 12-18 months, measure quality. And it is known for sure that "the quality of a journal (impact factor)" does not correlate with quality (or indeed with citations).
Later we read
"H Index and Citation Impact: These are good objective measures of the scientific impact of
publications"
NO, they are simply not a measure of quality (though this time they say “impact” rather than “excellence”).
The people who wrote that seem to be unaware of the most basic facts about science.
Then
"Carrying out high quality scientific work requires research teams"
Sometimes it does, sometimes it doesn’t. In the past the best work has been done by one or two people. In my field, think of Hodgkin & Huxley, Katz & Miledi or Neher & Sakmann. All got Nobel prizes. All did the work themselves. Performance managers might well have fired them before they got started.
By specifying minimum acceptable group sizes, King’s are really specifying minimum acceptable grant income, just like Imperial and Warwick. Nobody will be taken in by the thin attempt to disguise it.
The specification that a professor should have "Primary supervision of three or more PhD students, with additional secondary supervision." is particularly iniquitous. Everyone knows that far too many PhDs are being produced for the number of jobs that are available. This stipulation is not for the benefit of the young. It’s to ensure a supply of cheap labour to churn out more papers and help to lift the university’s ranking.
The document is not signed, but the document properties name its author. But she’s not a scientist and is presumably acting under orders, so please don’t blame her for this dire document. Blame the vice-chancellor.
Performance management is a direct incentive to do shoddy short-cut science.
No wonder that The Economist says "scientists are doing too much trusting and not enough verifying—to the detriment of the whole of science, and of humanity".
Feel ashamed.
Academic staff are going to be fired at Queen Mary University of London (QMUL). It’s possible that universities may have to contract a bit in hard times, so what’s wrong?
What’s wrong is that the victims are being selected in a way that I can describe only as insane. The criteria they use are guaranteed to produce a generation of second-rate spiv scientists, with a consequent progressive decline in QMUL’s reputation.
The firings, it seems, are nothing to do with hard financial times, but are a result of QMUL’s aim to raise its ranking in university league tables.
In the UK university league table, a university’s position is directly related to its government research funding. So they need to do well in the 2014 ‘Research Excellence Framework’ (REF). To achieve that they plan to recruit new staff with high research profiles, take on more PhD students and post-docs, obtain more research funding from grants, and get rid of staff who are not doing ‘good’ enough research.
So far, that’s exactly what every other university is trying to do. This sort of distortion is one of the harmful side-effects of the REF. But what’s particularly stupid about QMUL’s behaviour is the way they are going about it. You can assess your own chances of survival at QMUL’s School of Biological and Chemical Sciences from the following table, which is taken from an article by Jeremy Garwood (Lab Times Online. July 4, 2012). The numbers refer to the four year period from 2008 to 2011.
|
Category of staff |
Research Output Quantity |
Research Output |
Research Income (£) |
Research Income (£) |
|
Professor |
11 |
2 |
400,000 |
at least 200,000 |
|
Reader |
9 |
2 |
320,000 |
at least 150,000 |
|
Senior Lecturer |
7 |
1 |
260,000 |
at least 120,000 |
|
Lecturer |
5 |
1 |
200,000 |
at least 100,000 |
|
In addition to the three criteria, ‘Research Output ‐ quality’, ‘Research Output – quantity’, and ‘Research Income’, there is a minimum threshold of 1 PhD completion for staff at each academic level. All this data is “evidenced by objective metrics; publications cited in Web of Science, plus official QMUL metrics on grant income and PhD completion.” To survive, staff must meet the minimum threshold in three out of the four categories, except as follows: Demonstration of activity at an exceptional level in either ‘research outputs’ or ‘research income’, termed an ‘enhanced threshold’, is “sufficient” to justify selection regardless of levels of activity in the other two categories. And what are these enhanced thresholds? |
|
The university notes that the above criteria “are useful as entry standards into the new school, but they fall short of the levels of activity that will be expected from staff in the future. These metrics should not, therefore, be regarded as targets for future performance.” This means that those who survived the redundancy criteria will simply have to do better. But what is to reassure them that it won’t be their turn next time should they fail to match the numbers? To help them, Queen Mary is proposing to introduce ‘D3’ performance management (www.unions.qmul.ac.uk/ucu/docs/d3-part-one.doc). Based on more ‘administrative physics’, D3 is shorthand for ‘Direction × Delivery × Development.’ Apparently “all three are essential to a successful team or organisation. The multiplication indicates that where one is absent/zero, then the sum is zero!” D3 is based on principles of accountability: “A sign of a mature organisation is where its members acknowledge that they face choices, they make commitments and are ready to be held to account for discharging these commitments, accepting the consequences rather than seeking to pass responsibility.” Inspired? |
I presume the D3 document must have been written by an HR person. It has all the incoherent use of buzzwords so typical of HR. And it says "sum" when it means "product" (oh dear, innumeracy is rife).
The criteria are utterly brainless. The use of impact factors for assessing people has been discredited at least since Seglen (1997) showed that the number of citations that a paper gets is not perceptibly correlated with the impact factor of the journal in which it’s published. The reason for this is the distribution of the number of citations for papers in a particular journal is enormously skewed. This means that high-impact journals get most of their citations from a few articles.
|
The distribution for Nature is shown in Fig. 1. Far from being gaussian, it is even more skewed than a geometric distribution; the mean number of citations is 114, but 69% of papers have fewer than the mean, and 24% have fewer than 30 citations. One paper has 2,364 citations but 35 have 10 or fewer. ISI data for citations in 2001 of the 858 papers published in Nature in 1999 show that the 80 most-cited papers (16% of all papers) account for half of all the citations (from Colquhoun, 2003)
|
|
The Institute of Scientific Information, ISI, is guilty of the unsound statistical practice of characterizing a distribution by its mean only, with no indication of its shape or even its spread. School of Biological and Chemical Sciences-QMUL is expecting everyone has to be above average in the new regime. Anomalously, the thresholds for psychologists are lower because it is said that it’s more difficult for them to get grants. This undermines even the twisted logic applied at the outset.
All this stuff about skewed distributions is, no doubt, a bit too technical for HR people to understand. Which, of course, is precisely why they should have nothing to do with assessing people.
At a time when so may PhDs fail to get academic jobs we should be limiting the numbers. But QMUL requires everyone to have a PhD student, not for the benefit of the student, but to increase its standing in league tables. That is deeply unethical.
The demand to have two papers in journals with impact factor greater than seven is nonsense. In physiology, for example, there are only four journals with an impact factor greater that seven and three of them are review journals that don’t publish original research. The two best journals for electrophysiology are Journal of Physiology (impact factor 4.98, in 2010) and Journal of General Physiology (IF 4.71). These are the journals that publish papers that get you into the Royal Society or even Nobel prizes. But for QMUL, they don’t count.
I have been lucky to know well three Nobel prize winners. Andrew Huxley. Bernard Katz, and Bert Sakmann. I doubt that any of them would pass the criteria laid down for a professor by QMUL. They would have been fired.
The case of Sakmann is analysed in How to Get Good Science, [pdf version]. In the 10 years from 1976 to 1985, when Sakmann rose to fame, he published an average of 2.6 papers per year (range 0 to 6). In two of these 10 years he had no publications at all. In the 4 year period (1976 – 1979 ) that started with the paper that brought him to fame (Neher & Sakmann, 1976) he published 9 papers, just enough for the Reader grade, but in the four years from 1979 – 1982 he had 6 papers, in 2 of which he was neither first nor last author. His job would have been in danger if he’d worked at QMUL. In 1991 Sakmann, with Erwin Neher, got the Nobel Prize for Physiology or Medicine.
The most offensive thing of the lot is the way you can buy yourself out if you publish 26 papers in the 4 year period. Sakmann came nowhere near this. And my own total, for the entire time from my first paper (1963) until I was elected to the Royal Society (May 1985) was 27 papers (and 7 book chapters). I would have been fired.
Peter Higgs had no papers at all from the time he moved to Edinburgh in 1960, until 1964 when his two paper’s on what’s now called the Higgs’ Boson were published in Physics Letters. That journal now has an impact factor less than 7 so Queen Mary would not have counted them as “high quality” papers, and he would not have been returnable for the REF. He too would have been fired.
The encouragement to publish large numbers of papers is daft. I have seen people rejected from the Royal Society for publishing too much. If you are publishing a paper every six weeks, you certainly aren’t writing them, and possibly not even reading them. Most likely you are appending your name to somebody else’s work with little or no checking of the data. Such numbers can be reached only by unethical behaviour, as described by Peter Lawrence in The Mismeasurement of Science. Like so much managerialism, the rules provide an active encouragement to dishonesty.
In the face of such a boneheaded approach to assessment of your worth, it’s the duty of any responsible academic to point out the harm that’s being done to the College. Richard Horton, in the Lancet, did so in Bullying at Barts. There followed quickly letters from Stuart McDonald and Nick Wright, who used the Nuremburg defence, pointing out that the Dean (Tom Macdonald) was just obeying orders from above. That has never been as acceptable defence. If Macdonald agreed with the procedure, he should be fired for incompetence. If he did not agree with it he should have resigned.
It’s a pity, because Tom Macdonald was one of the people with whom I corresponded in support of Barts’ students who, very reasonably, objected to having course work marked by homeopaths (see St Bartholomew’s teaches antiscience, but students revolt, and, later, Bad medicine. Barts sinks further into the endarkenment). In that case he was not unreasonable, and, a mere two years later I heard that he’d taken action.
To cap it all, two academics did their job by applying a critical eye to what’s going on at Queen Mary. They wrote to the Lancet under the title Queen Mary: nobody expects the Spanish Inquisition
"For example, one of the “metrics” for research output at professorial level is to have published at least two papers in journals with impact factors of 7 or more. This is ludicrous, of course—a triumph of vanity as sensible as selecting athletes on the basis of their brand of track suit. But let us follow this “metric” for a moment. How does the Head of School fair? Zero, actually. He fails. Just consult Web of Science. Take care though, the result is classified information. HR’s “data” are marked Private and Confidential. Some things must be believed. To question them is heresy."
Astoundingly, the people who wrote this piece are now under investigation for “gross misconduct”. This is behaviour worthy of the University of Poppleton, as pointed out by the inimitable Laurie Taylor, in Times Higher Education (June 7)
|
The rustle of censorship It appears that last week’s edition of our sister paper, The Poppleton Evening News, carried a letter from Dr Gene Ohm of our Biology Department criticising this university’s metrics-based redundancy programme. We now learn that, following the precedent set by Queen Mary, University of London, Dr Ohm could be found guilty of “gross misconduct” and face “disciplinary proceedings leading to dismissal” for having the effrontery to raise such issues in a public place. Louise Bimpson, the corporate director of our ever-expanding human resources team, admitted that this response might appear “severe” but pointed out that Poppleton was eager to follow the disciplinary practices set by such soon-to-be members of the prestigious Russell Group as Queen Mary. Thus it was only to be expected that we would seek to emulate its espousal of draconian censorship. She hoped this clarified the situation. |
David Bignell, emeritus professor of zoology at Queen Mary hit the nail on the head.
"These managers worry me. Too many are modest achievers, retired from their own studies, intoxicated with jargon, delusional about corporate status and forever banging the metrics gong. Crucially, they don’t lead by example."
What the managers at Queen Mary have failed to notice is that the best academics can choose where to go.
People are being told to pack their bags and move out with one day’s notice. Access to journals stopped, email address removed, and you may need to be accompanied to your (ex)-office. Good scientists are being treated like criminals.
What scientist in their right mind would want to work at QMUL, now that their dimwitted assessment methods, and their bullying tactics, are public knowledge?
The responsibility must lie with the principal, Simon Gaskell. And we know what the punishment is for bringing your university into disrepute.
Follow-up
Send an email. You may want to join the many people who have already written to QMUL’s principal, Simon Gaskell (principal@qmul.ac.uk), and/or to Sir Nicholas Montagu, Chairman of Council, n.montagu@qmul.ac.uk.
Sunday 1 July 2012. Since this blog was posted after lunch on Friday 29th June, it has had around 9000 visits from 72 countries. Here is one of 17 maps showing the origins of 200 of the hits in the last two days
The tweets about QMUL are collected in a Storify timeline.
I’m reminded of a 2008 comment, on a post about the problems imposed by HR, In-human resources, science and pizza.
Thanks for that – I LOVED IT. It’s fantastic that the truth of HR (I truly hate that phrase) has been so ruthlessly exposed. Should be part of the School Handbook. Any VC who stripped out all the BS would immediately retain and attract good people and see their productivity soar.
That’s advice that Queen Mary should heed.
Part of the reason for that popularity was Ben Goldacre’s tweet, to his 201,000 followers
“destructive, unethical and crude metric incentives in academia (spotlight QMUL) bit.ly/MFHk2H by @david_colquhoun”
3 July 2012. I have come by a copy of this email, which was sent to Queen Mary by a senior professor from the USA (word travels fast on the web). It shows just how easy it is to destroy the reputation of an institution.
|
Sir Nicholas Montagu, Chairman of Council, and Principal Gaskell, I was appalled to read the criteria devised by your University to evaluate its faculty. There are so flawed it is hard to know where to begin. Your criteria are antithetical to good scientific research. The journals are littered with weak publications, which are generated mainly by scientists who feel the pressure to publish, no matter whether the results are interesting, valid, or meaningful. The literature is flooded by sheer volume of these publications. Your attempt to require “quality” research is provided by the requirement for publications in “high Impact Factor” journals. IF has been discredited among scientists for many reasons: it is inaccurate in not actually reflecting the merit of the specific paper, it is biased toward fields with lots of scientists, etc. The demand for publications in absurdly high IF journals encourages, and practically enforces scientific fraud. I have personally experienced those reviews from Nature demanding one or two more “final” experiments that will clinch the publication. The authors KNOW how these experiments MUST turn out. If they want their Nature paper (and their very academic survival if they are at a brutal, anti-scientific university like QMUL), they must get the “right” answer. The temptation to fudge the data to get this answer is extreme. Some scientists may even be able to convince themselves that each contrary piece of data that they discard to ensure the “correct” answer is being discarded for a valid reason. But the result is that scientific misconduct occurs. I did not see in your criteria for “success” at QMUL whether you discount retracted papers from the tally of high IF publications, or perhaps the retraction itself counts as yet another high IF publication! Your requirement for each faculty to have one or more postdocs or students promotes the abusive exploitation of these individuals for their cheap labor, and ignores the fact that they are being “trained” for jobs that do not exist. The “standards” you set are fantastically unrealistic. For example, funding is not graded, but a sharp step function – we have 1 or 2 or 0 grants and even if the average is above your limits, no one could sustain this continuously. Once you have fired every one of your faculty, which will almost certainly happen within 1-2 rounds of pogroms, where will you find legitimate scientists who are willing to join such a ludicrous University? |
4 July 2012.
Professor John F. Allen is Professor of Biochemistry at Queen Mary, University of London, and distinguished in the fields of Photosynthesis, Chloroplasts, Mitochondria, Genome function and evolution and Redox signalling. He, with a younger colleague, wrote a letter to the Lancet, Queen Mary: nobody expects the Spanish Inquisition. It is an admirable letter, the sort of thing any self-respecting academic should write. But not according to HR. On 14 May, Allen got a letter from HR, which starts thus.
|
14th May 2012 Dear Professor Allen I am writing to inform you that the College had decided to commence a factfinding investigation into the below allegation: That in writing and/or signing your name to a letter entitled "Queen Mary: nobody expects the Spanish Inquisition," (enclosed) which was published in the Lancet online on 4th May 2012, you sought to bring the Head of School of Biological and Chemical Sciences and the Dean for Research in the School of Medicine and Dentistry into disrepute. . . . . Sam Holborn |
Download the entire letter. It is utterly disgraceful bullying. If anyone is bringing Queen Mary into disrepute, it is Sam Holborn and the principal, Simon Gaskell.
Here’s another letter, from the many that have been sent. This is from a researcher in the Netherlands.
|
Dear Sir Nicholas,
I am addressing this to you in the hope that you were not directly involved in creating this extremely stupid set of measures that have been thought up, not to improve the conduct of science at QMUL, but to cheat QMUL’s way up the league tables over the heads of the existing academic staff. Others have written more succinctly about the crass stupidity of your Human Resources department than I could, and their apparent ignorance of how science actually works. As your principal must bear full responsibility for the introduction of these measures, I am not sending him a copy of this mail. I am pretty sure that his “principal” mail address will no longer be operative. We have had a recent scandal in the Netherlands where a social psychology professor, who even won a national “Man of the Year” award, as well as as a very large amount of research money, was recently exposed as having faked all the data that went into a total number of articles running into three figures. This is not the sort of thing one wants to happen to one’s own university. He would have done well according to your REF .. before he was found out. Human Resources departments have gained too much power, and are completely incompetent when it comes to judging academic standards. Let them get on with the old dull, and gobbledigook-free, tasks that personnel departments should be carrying out. |
5 July 2012.
Here’s another letter. It’s from a member of academic staff at QMUL, someone who is not himself threatened with being fired. It certainly shows that I’m not making a fuss about nothing. Rather, I’m the only person old enough to say what needs to be said without fear of losing my job and my house.
|
Dear Prof. Colquhoun,
I am an academic staff member in SBCS, QMUL. I am writing from my personal email account because the risks of using my work account to send this email are too great. I would like to thank you for highlighting our problems and how we have been treated by our employer (Queen Mary University of London), in your blog. I would please urge you to continue to tweet and blog about our plight, and staff in other universities experiencing similarly horrific working conditions. I am not threatened with redundancy by QMUL, and in fact my research is quite successful. Nevertheless, the last nine months have been the most stressful of all my years of academic life. The best of my colleagues in SBCS, QMUL are leaving already and I hope to leave, if I can find another job in London. Staff do indeed feel very unfairly treated, intimidated and bullied. I never thought a job at a university could come to this.
Thank you again for your support. It really does matter to the many of us who cannot really speak out openly at present.
Best regards,
|
In a later letter, the same person pointed out
"There are many of us who would like to speak more openly, but we simply cannot."
"I have mortgage . . . . Losing my job would probably mean losing my home too at this point."
"The plight of our female staff has not even been mentioned. We already had very few female staff. And with restructuring, female staff are more likely to be forced into teaching-only contracts or indeed fired"."
"total madness in the current climate – who would want to join us unless desperate for a job!"
“fuss about nothing” – absolutely not. It is potentially a perfect storm leading to teaching and research disaster for a university! Already the reputation of our university has been greatly damaged. And senior staff keep blaming and targeting the “messengers"."
6 July 2012.
Througn the miracle of WiFi, this is coming from Newton, MA. The Lancet today has another editorial on the Queen Mary scandal.
"As hopeful scientists prepare their applications to QMUL, they should be aware that, behind the glossy advertising, a sometimes harsh, at times repressive, and disturbingly unforgiving culture awaits them."
That sums it up nicely.
24 July 2012. I’m reminded by Nature writer, Richard van Noorden (@Richvn) that Nature itself has wriiten at least twice about the iniquity of judging people by impact factors. In 2005 Not-so-deep impact said
"Only 50 out of the roughly 1,800 citable items published in those two years received more than 100 citations in 2004. The great majority of our papers received fewer than 20 citations."
"None of this would really matter very much, were it not for the unhealthy reliance on impact factors by administrators and researchers’ employers worldwide to assess the scientific quality of nations and institutions, and often even to judge individuals."
And, more recently, in Assessing assessment” (2010).
29 July 2012. Jonathan L Rees. of the University of Edinburgh, ends his blog:
"I wonder what career advice I should offer to a young doctor circa 2012. Apart from not taking a job at Queen Mary of course. "
How to select candidates
I have, at various times, been asked how I would select candidates for a job, if not by counting papers and impact factors. This is a slightly modified version of a comment that I left on a blog, which describes roughly what I’d advocate
After a pilot study the entire Research Excellence Framework (which attempts to assess the quality of research in every UK university) made the following statement.
“No sub-panel will make any use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs”
It seems that the REF is paying attention to the science not to bibliometricians.
It has been the practice at UCL to ask people to nominate their best papers (2 -4 papers depending on age). We then read the papers and asked candidates hard questions about them (not least about the methods section). It’s a method that I learned a long time ago from Stephen Heinemann, a senior scientist at the Salk Institute. It’s often been surprising to learn how little some candidates know about the contents of papers which they themselves select as their best. One aim of this is to find out how much the candidate understands the principles of what they are doing, as opposed to following a recipe.
Of course we also seek the opinions of people who know the work, and preferably know the person. Written references have suffered so much from ‘grade inflation’ that they are often worthless, but a talk on the telephone to someone that knows both the work, and the candidate, can be useful, That, however, is now banned by HR who seem to feel that any knowledge of the candidate’s ability would lead to bias.
It is not true that use of metrics is universal and thank heavens for that. There are alternatives and we use them.
Incidentally, the reason that I have described the Queen Mary procedures as insane, brainless and dimwitted is because their aim to increase their ratings is likely to be frustrated. No person in their right mind would want to work for a place that treats its employees like that, if they had any other option. And it is very odd that their attempt to improve their REF rating uses criteria that have been explicitly ruled out by the REF. You can’t get more brainless than that.
This discussion has been interesting to me, if only because it shows how little bibliometricians understand how to get good science.
It’s hard enough to communicate basic ideas about how to assess evidence to adults without having the effort hindered by schools.
The teaching of quackery to 16 year-olds has been approved by a maze of quangos, none of which will take responsibility, or justify their actions. So far I’ve located no fewer than eight of them.
[For non-UK readers, quango = Quasi-Autonomous Non-Governmental Organisation].
A lot of odd qualifications are accredited by OfQual (see here). Consider, for example, Edexcel Level 3 BTEC Nationals in Health and Social Care (these exams are described here), Download the specifications here and check page 309.
Unit 23: Complementary Therapies for Health and Social Care
NQF Level 3: BTEC National
Guided learning hours: 60Unit abstract
“In order to be able to take a holistic view towards medicine and health care, health and social care professionals need to understand the potential range of complementary therapies available and how they may be used in the support of conventional medicine.”
Well, Goldacre has always said that homeopathy makes the perfect vehicle for teaching how easy it is to be deceived by bad science, so what’s wrong? But wait
“Learners will consider the benefits of complementary therapies to health and wellbeing, as well as identifying any contraindications and health and safety issues in relation to their use.”
Then later
“The holistic approach to illnesses such as cancer could be used as a focus here. For example, there could be some tutor input to introduce ideas about the role of complementary therapies in the treatment and management of cancer, this being followed up by individual or small group research by learners using both the internet and the services available locally/regionally. If available, a local homeopathic hospital, for example, would be an interesting place to visit.”
It’s true that to get a distinction, you have to “evaluate the evidence relating to the use of complementary therapies in contemporary society”, but it isn’t at all clear that this refers to evidence about whether the treatment works.
The really revealing bit comes when you get to the
“Indicative reading for learners
There are many resources available to support this unit.Websites
www.acupuncture.org.uk British Acupuncture Council
www.bant.org.uk British Association for Nutritional Therapy
www.exeter.ac.uk/sshs/compmed Exeter University’s academic department of Complementary medicine
www.gcc-uk.org General Chiropractic Council
www.nimh.org.uk National Institute of Medical Herbalists
www.nursingtimes.net The Nursing Times
www.osteopathy.org.uk General Osteopathic Council
www.the-cma.org.uk The Complementary Medical Association”
This list is truly astonishing. Almost every one of them can be relied on to produce self-serving inaccurate information about the form of “therapy” it exists to promote. The one obvious exception is the reference to Exeter University’s academic department of complementary medicine (and the link to that one is wrong). The Nursing Times should be an exception too, but their articles about CAM are just about always written by people who are committed to it.
It is no consolation that the 2005 version was even worse. In its classification of ‘therapies’ it said “Pharmaceutically mediated: eg herbalism, homeopathy “. Grotesque! And this is the examinng body!
The Teacher
This particular educational disaster came to my attention when I had a letter from a teacher. She had been asked to teach this unit, and wanted to know if I could provide any resources for it. She said that Edexcel hadn’t done so. She asked ” Do you know of any universities that teach CT’s [sic] so I could contact them about useful teaching resources?.” She seemed to think that reliable information about homeopathy could be found from a ‘university’ homeopathy teacher. Not a good sign. It soon emerged why.
She said.
“My students are studying BTEC National Health Studies and the link is Edexcel BTEC National Complimentary [sic] studies.”
“I am a psychotherapist with an MA in Education and Psychology. I am also trained in massage and shiatsu and have plenty of personal experience of alternative therapy”
Shiatsu uh? It seems the teacher is already committed to placebo medicine. Nevertheless I spent some time looking for some better teaching material for 16 year-old children. There is good stuff at Planet
Science, and in some of the pamphlets from Sense about Science, not least their latest, I’ve got nothing to lose by trying it – A guide to weighing up claims about cures and treatments. I sent all this stuff to her, and prefaced the material by saying
“First of all, I should put my cards on the table and say that I am quite appalled by the specification of Unit 23. In particular, it has almost no emphasis at all on the one thing that you want to know about any therapy, namely does it work? The reference list for reading consists almost entirely of organisations that are trying to sell you various sorts of quackery, There is no hint of balance; furthermore it is all quite incompatible with unit 22, which IS concerned with evidence.”
At this point the teacher the teacher came clean too, As always, anyone who disagrees with the assessment (if any) of the evidence by a true believer is unmeasured and inflammatory.
“I have found your responses very unmeasured and inflammatory and I am sorry to say that this prejudicial attitude has meant that I have not found your comments useful.”
shortly followed by
“I am not coming from a scientific background, neither is the course claiming to be scientific.”
That will teach me to spend a couple of hours trying to help a teacher.
What does Edexcel say?
I wrote to Edexcel’s science subject advisors with some questions about what was being taught. The response that I got was not from the science subject advisors but from the Head of Customer support, presumably a PR person.
| From: (Bola Arabome) 12/11/2008 04.31 PM
Dear Professor Colquhoun Thank you for email communication concerning the complementary therapies unit which is available in our BTEC National in Health and BTEC National in Health and Social Care qualifications. I have replied on behalf of Stephen Nugus, our science subject advisor, because your questions do not refer to a science qualification. I would like to answer your questions as directly as possible and then provide some background information relating to the qualifications. The units and whole qualifications for all awarding bodies are accredited by the regulator, the Qualifications and Curriculum Authority. The resource reading list is also produced by us to help teachers and learners. The qualification as a whole is related to the National Occupational Standards for the vocational sectors of Health and Health and social care with consultation taken from the relevant sector skills councils . As you will be aware many of these complementary therapies are available in care centres and health centres under the NHS and in the private sector. The aim of BTEC qualifications is to prepare people for work in these particular sectors. Clearly a critical awareness is encouraged with reference to health and safety and regulation. There are other units, in some cases compulsory, within the qualification with a scientific approach. ‘ ‘ ‘ ‘ ‘ Stephen Harris Head of Customer Support |
Aha, so it seems that teaching people to treat sick patients is “not a science qualification”. Just a business qualification perhaps?. I haven’t yet managed to reach the people who make these decisions, so I persisted with the PR man. Here is part of the next letter (Edexcel’s reply in italic).
| 19 November
I find it quite fascinating that Edexcel regards the treatment of sick patients as not being part of science (“do not refer to a science qualification”). Does that mean Edexcel regard the “Health” part of “Health and Social Care” as being nothing to do with science, and that it therefore doesn’t matter if Health Care is unscientific, or even actively anti-scientific? I am sorry if my answer lacked clarity. My comment, that I had taken your enquiry on behalf of our Science Advisor because this was not a science qualification, was intended to explain why I was replying. It was not intended as a comment on the relationship between Health and Social Care and science. At Edexcel we use bureaucratic categories where we align our management of qualifications with officially recognised occupational sectors. Often we rely on sector bodies such as Sector Skills Councils to endorse or even approve the qualifications we offer. Those involved in production of our Science qualifications and our (4) You say “The qualification as a whole is related to the National Occupational Standards for the vocational sectors of Health and Health and social care with consultation taken from the relevant sector skills councils”. Are you aware that the Skills for Health specifications for Alternative medicine were written essentially by the Prince of Wales Foundation? The qualification was approved by both ‘Skills for Health’ and ‘Skills for Care and Development’ prior to being accredited by QCA. It uses the NOS in Health and Social Care as the basis for many of the mandatory units. The ‘Complementary Therapies’ NOS were not used. This was not a requirement of a ‘Health and Social Care’ qualification. “Are the NOS in Health and Social Care that you mention the ones listed here? http://www.ukstandards.org/Find_Occupational_Standards.aspx?NosFindID=1&ClassificationItemId=174 If so, I can see nothing there about ‘complementary therapies’. if I have missed it, I’d be very grateful if you could let me know where it is. If it is not there, I remain very puzzled about the provenance of Unit 23, since you say it is not based on Skills for Health.” |
Now we are immediately at sea, struggling under a tidal wave of acronyms for endless overlapping quangos. In this one short paragraph we have no fewer than four of them. ‘Skills for Health’, ‘Skills for Care and Development’ , ‘Quality and Curriculum Authority (QCA) and NOS.
It seems that the specification of unit 23 was written by Edexcel, but Harris (25 Nov) declines to name those responsible
“When I refer to our “Health and Social care team” I mean the mix of Edexcel Staff and the associates we employ on a contract basis as writers, examiners and external verifiers. The writers are generally recruited from those who are involved in teaching and assessment the subjects in schools and colleges. The editorial responsibility lies with the Edexcel Staff. I do not have access to the names of the writers and in any case would not be able to pass on this information. Specifications indicate the managers responsible for authorising publication”
“Edexcel takes full responsibility for its ethical position on this and other issues. However we can not accept responsibility for the opinions expressed in third party materials. There is a disclaimer to this effect at the beginning of the specification. ”
” You have the correct link to the Health NOS . These are the standards, which where appropriate, influence our qualifications. However in the case of Unit 23 I understand that there is no link with the Health NOS. I don’t know if the NOS cover the unit 23 content.”
So, contrary to what I was told at first, neither Skills for Health, nor NOS were involved Or were they (see below)?
So who does take responsibility? Aha that is secret. And the approval by the QCA is also secret.
“I cannot provide you with copies of any correspondence between Skills for Health and Edexcel. We regard this as confidential. “
What does the QCA say?
The strapline of the QCA is
“We are committed to building a world-class education and training framework. We develop and modernise the curriculum, assessments, examinations and qualifications.”
Referring school children to the Society of Homeopaths for advice seems to be world-class bollocks rather than world-class education.
When this matter was brought to light by Graeme Paton in the Daily Telegraph, he quoted Kathleen Tattersall, CEO of the QCA. She said
“The design of these diplomas has met Ofqual’s high standards. We will monitor them closely as they are delivered to make sure that learners get a fair deal and that standards are set appropriately.”
Just the usual vacuous bureaucratic defensive sound-bite there. So I wrote to Kathleen Tattersall myself with some specific questions. The letter went on 2nd September 2008. Up to today, 26 November, I had only letters saying
“Thank you for your email of 12 November addressed to Kathleen Tattersall, a response is being prepared which will be forwarded to you shortly.”
“Thank you for your email of 25th November addressed to Kathleen Tattersall. A more detailed response is being prepared which will be sent to you shortly.”
Here are some of the questions that I asked.
| I wrote to Edexcel’s subject advisors about unit 23 and I was told “your questions do not refer to a science qualification”. This seems to mean that if it comes under the name “Health Care” then the care of sick patients is treated as though it were nothing to do with science, That seems to me to be both wrong and dangerous, and I should like to hear your view about that question.
Clearly the fundamental problem here is that the BTEC is intended as a vocational training for careers in alternative medicine, As a body concerned with education, surely you cannot ignore the view of 99% of scientists and doctors that almost all alternative medicine is fraud. That doesn’t mean that you can’t make a living from it, but it surely does create a dilemma for an educational organisation. What is your view of that dilemma? |
Eventually, on 27th November, I get a reply (of sorts) It came not from the Kathleen Tattersall of the QCA but from yet another regulatory body, OfQual, the office of the Qualifications and Examinations Regulator. You’d think that they’d know the answers, but if they do they aren’t telling, [download whole letter. It is very short. The “more detailed response” says nothing.
|
Ofqual does not take a view on the detailed content of vocational qualifications as that responsibility sits with the relevant Sector Skills Council which represents employers and others involved in the sector. Ofqual accredits the specifications, submitted by sector-skilled professionals, after ensuring they meet National Occupational Standards. Ofqual relies on the professional judgement of these sector-skilled professionals to include relevant subjects and develop and enhance the occupational standards in their profession. The accreditation of this BTEC qualification was supported by both Skills for Health, and Skills for Care and Development, organisations which represent the emerging Sector Qualifications Strategies and comply with the relevant National Occupational Standards Isabel Nisbet Acting Chief Executive |
So no further forward. Every time I ask a question, the buck gets passed to another quango (or two, or three). This letter, in any case, seems to contradict what Edexcel said about the involvement of Skills for Health (that’s the talking to trees outfit),
A nightmare maze of quangos
You may well be wondering what the relationship is between Ofqual and the QCA. There is an ‘explanation’ here.
Ofqual will take over the regulatory responsibilities of the Qualifications and Curriculum Authority (QCA), with stronger powers in relation to safeguarding the standards of qualifications and assessment and an explicit remit as a market regulator. The QCA will evolve into the Qualifications and Curriculum Development Agency (QCDA): supporting Ministers with advice and undertaking certain design and delivery support functions in relation to the curriculum, qualifications, learning and development in the Early Years Foundation Stage, and National Curriculum and Early Years Foundation Stage assessments.
Notice tha QCA won’t be abolished. There will be yet another quango.
The result of all this regulatory bureaucracy seems to be worse regulation, Exactly the same thing happens with accreditiation of dodgy degrees in universities.
At one time, a proposal for something like Unit 23 would have been shown to any competent science teacher, who would have said”you must be joking” and binned it. Now a few hundred bureaucrats tick their boxes and rubbish gets approved.
There seems to be nobody in any of these quangos with the education to realise that if you want to know the truth about homeopathy, the last person you ask is the Society of Homeopaths or the Prince of Wales.
What next?
So the mystery remains. I can’t find out who is responsible for the provenance of the appallingly anti-science Unit 23, and I can’t find out how it got approved. Neither can I get a straight answer to the obvious question about whether it is OK to encourage vocational qualifications for jobs that are bordering on being fraudulent.
.All I can get is platitudes and bland assurances. Everything that might be informative is clouded in secrecy.
The Freedom of Information requests are in. Watch this space. But don’t hold your breath.
Follow-up
Here are some attempts to break through the wall of silence.
Edexcel. I sent them this request.
Freedom of Information Act
Hello
I should like to see please all documents from Edexcel and OfQual or QCA (and communications between then) that concern the formulation and approval of Unit 23 (Complementary Therapies) in the level3 BTEC (page 309 in attached document). In vew of the contentious nature of the subject matter, I believe that is is in the public interest that this information be provided
David Colquhoun
The answer was quite fast, and quite unequivocal, Buzz off.
| Dear Mr Colquhoun,
Thank you of your e-mail of today’s date. I note your request for information pursuant to The Freedom of Information Act. As you may know this Act only applies to public bodies and not to the private sector. Edexcel Limited is privately owned and therefore not subject to this Act. Edexcel is therefore not obliged to provide information to you and is not prepared to give you the information you seek. Please do not hesitate to contact me again if you have any further queries. Kate Gregory |
This lack of public accountability just compounds their appalling inability to distinguish education from miseducation.
International Therapy Examination Council (ITEC)
Mojo’s comment, below, draws attention to the Foundation degree in Complementary Therapies offered by Cornwall College, Camborne, Cornwall (as well as to the fact that the Royal National Lifeboat Institution has been wasting money on ‘research’ on homeopathy –write to them).
At least the courses are held on the Camborne campus of Cornwall College, not on the Duchy campus (do we detect the hand of the Quacktitioner Royal in all this nonsense?).
Cornwall College descends to a new level of barminess in its course Crystal Healing VTCT Level 3
“Who is this course for?
This course is designed to enhance the skills of the Holistic Therapist. Crystals may be used on their own in conjunction with other therapies such as Indian Head Massage, Aromatherapy and Reflexology. Due to the nature of the demands of the holistic programme this course is only suitable for students over the age of 18.”
“What will I be doing on the course?
Students will study the art of Crystal healing which is an energy based treatment where crystals and gemstones are used to channel and focus various energy frequencies.”
| .VTCT stands for the Vocational Training Charitable Trust.
It is yet another organisation that runs vocational exams, and it is responsible for this particular horror |
|
The crystals are here. I quote.
Objectives
- the use of interpersonal skills with client
- how to complement other therapies with crystals
- the types and effects of different crystals
- uses of crystals including cleansing, energising, configurations
- concepts of auras and chakras
This is, of course, pure meaningless nonsense. Utter bollocks being offered as further education
Cornwall College has many courses run by ITEC.
The College says
“You will become a professional practitioner with the International Therapy Examination Council (ITEC), study a number of essential modules to give a vocational direction to your study that include: Homeopathy and its application,”
| Who on earth, I hear you cry, are ITEC? That brings us to the seventh organisation in the maze of quangos and private companies involved in the miseducation of young people about science and medicine. It appears, like Edexcel, to be a private company though its web site is very coy about that. |  |
After the foundation degree you can go on to “a brand new innovative BSc in Complementary Health Studies (from Sept 2009)”
The ITEC web site says
- ITEC qualifications are accredited by the Office of the Qualifications and Examination Regulator (OFQUAL)
- ITEC qualifications are funded in the uk by the on behalf of Department for Innovation, Universities and Skills (DIUS)
- ITEC qualifications have been mapped to the National Occupational Standards, where they exist
Oddly enough, there is no mention of accreditation by a University (not that that is worth much). So a few more Freedom of Information requests are going off, in an attempt to find out why are kids are being miseducated about science and medicine.
Meanwhile you can judge the effect of all that education in physiology by one of the sample questions for ITEC Unit 4, reflexology.
The pancreas reflex:
A Extends across both feet
B Is on the right foot only
C Is on the left foot only
D Is between the toes on both feet
Uhuh, they seem to have forgotten the option ‘none of the above’.
Or how about a sample question from ITEC Unit 47 – Stone Therapy Massage
Which organ of the body is associated with the element fire?
A Heart
B Liver
C Spleen
D Pancreas
Or perhaps this?
Which incantation makes hot stones work best?
A Incarcerous
B Avada Kedavra,
C Dissendium
D Expelliarmus.
(OK I made the last one up, with help from Harry Potter, but it makes just about as much sense as the real ones).
And guess what? You can’t use the Freedom of Information Act to find out how this preposterous rubbish got into the educational system because ” ITEC is a private organisation therefore does not come under this legislation”. The ability to conduct business in secret is a side effect of the privatisation of public education is another reason why it’s a bad idea.
Ofsted
|
Ofsted has inspected Cornwall College. They say “We inspect and regulate to achieve excellence in the care of children and young people, and in education and skills for learners of all ages.”. I can find no mention of this nonsense in their report, so I’ve asked them. |
|
Ofsted has admitted a spectacular failure in its inspection of child care in the London Borough of Haringey. Polly Curtis wrote in the Guardian (6 Dec 2008) “We failed over Haringey – Ofsted head”. It was the front page story. But of course Ofsted don’t take the blame, they say they were supplied with false information,
That is precisely what happens whenever a committee or quango endorses rubbish. They look only at the documents sent to them and they don’t investigate, don’t engage their brains.
In the case of these courses in utter preposterous rubbish, it seems rather likely that the ultimate source of the misinformation is the Princes’ Foundation for Integrated Health. Tha views of the Prince of Wales get passed on to the ludicrous Skills for Health and used as a criterion by all the other organisations, without a moment of critical appraisal intervening at any point.
2 December 2008 A link from James Randi has sent the hit rate for this post soaring. Someone there left are rather nice comment.
“A quango seems to be a kind of job creation for the otherwise unemployable ‘educated ‘( degree in alternative navel contemplation) middle classes who can’t be expected to do anything useful like cleaning latines ( the only other thing they seem qualified for ). I really hate to think of my taxes paying for this codswollop.”