Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
HEFCE
This is very quick synopsis of the 500 pages of a report on the use of metrics in the assessment of research. It’s by far the most thorough bit of work I’ve seen on the topic. It was written by a group, chaired by James Wilsdon, to investigate the possible role of metrics in the assessment of research.
The report starts with a bang. The foreword says
|
"Too often, poorly designed evaluation criteria are “dominating minds, distorting behaviour and determining careers.”1 At their worst, metrics can contribute to what Rowan Williams, the former Archbishop of Canterbury, calls a “new barbarity” in our universities." "The tragic case of Stefan Grimm, whose suicide in September 2014 led Imperial College to launch a review of its use of performance metrics, is a jolting reminder that what’s at stake in these debates is more than just the design of effective management systems." "Metrics hold real power: they are constitutive of values, identities and livelihoods " |
And the conclusions (page 12 and Chapter 9.5) are clear that metrics alone can measure neither the quality of research, nor its impact.
"no set of numbers,however broad, is likely to be able to capture the multifaceted and nuanced judgements on the quality of research outputs that the REF process currently provides"
"Similarly, for the impact component of the REF, it is not currently feasible to use quantitative indicators in place of narrative impact case studies, or the impact template"
These conclusions are justified in great detail in 179 pages of the main report, 200 pages of the literature review, and 87 pages of Correlation analysis of REF2014 scores and metrics
The correlation analysis shows clearly that, contrary to some earlier reports, all of the many metrics that are considered predict the outcome of the 2014 REF far too poorly to be used as a substitute for reading the papers.
There is the inevitable bit of talk about the "judicious" use of metrics tp support peer review (with no guidance about what judicious use means in real life) but this doesn’t detract much from an excellent and thorough job.
Needless to say, I like these conclusions since they are quite similar to those recommended in my submission to the report committee, over a year ago.
Of course peer review is itself fallible. Every year about 8 million researchers publish 2.5 million articles in 28,000 peer-reviewed English language journals (STM report 2015 and graphic, here). It’s pretty obvious that there are not nearly enough people to review carefully such vast outputs. That’s why I’ve said that any paper, however bad, can now be printed in a journal that claims to be peer-reviewed. Nonetheless, nobody has come up with a better system, so we are stuck with it.
It’s certainly possible to judge that some papers are bad. It’s possible, if you have enough expertise, to guess whether or not the conclusions are justified. But no method exists that can judge what the importance of a paper will be in 10 or 20 year’s time. I’d like to have seen a frank admission of that.
If the purpose of research assessment is to single out papers that will be considered important in the future, that job is essentially impossible. From that point of view, the cost of research assessment could be reduced to zero by trusting people to appoint the best people they can find, and just give the same amount of money to each of them. I’m willing to bet that the outcome would be little different. Departments have every incentive to pick good people, and scientists’ vanity is quite sufficient motive for them to do their best.
Such a radical proposal wasn’t even considered in the report, which is a pity. Perhaps they were just being realistic about what’s possible in the present climate of managerialism.
Other recommendations include
"HEIs should consider signing up to the San Francisco Declaration on Research Assessment (DORA)"
4. "Journal-level metrics, such as the Journal Impact Factor (JIF), should not be used."
It’s astonishing that it should be still necessary to deplore the JIF almost 20 years after it was totally discredited. Yet it still mesmerizes many scientists. I guess that shows just how stupid scientists can be outside their own specialist fields.
DORA has over 570 organisational and 12,300 individual signatories, BUT only three universities in the UK have signed (Sussex, UCL and Manchester). That’s a shocking indictment of the way (all the other) universities are run.
One of the signatories of DORA is the Royal Society.
"The RS makes limited use of research metrics in its work. In its publishing activities, ever since it signed DORA, the RS has removed the JIF from its journal home pages and marketing materials, and no longer uses them as part of its publishing strategy. As authors still frequently ask about JIFs, however, the RS does provide them, but only as one of a number of metrics".
That’s a start. I’ve advocated making it a condition to get any grant or fellowship, that the university should have signed up to DORA and Athena Swan (with checks to make sure they are actually obeyed).
And that leads on naturally to one of the most novel and appealing recommendations in the report.
|
"A blog will be set up at http://www.ResponsibleMetrics.org "every year we will award a “Bad Metric” prize to the most |
This should be really interesting. Perhaps I should open a book for which university is the first to win "Bad Metric" prize.
The report covers just about every aspect of research assessment: perverse incentives, whether to include author self-citations, normalisation of citation impact indicators across fields and what to do about the order of authors on multi-author papers.
It’s concluded that there are no satisfactory ways of doing any of these things. Those conclusions are sometimes couched in diplomatic language which may, uh, reduce their impact, but they are clear enough.
The perverse incentives that are imposed by university rankings are considered too. They are commercial products and if universities simply ignored them, they’d vanish. One important problem with rankings is that they never come with any assessment of their errors. It’s been known how to do this at least since Goldstein & Spiegelhalter (1996, League Tables and Their Limitations: Statistical Issues in Comparisons Institutional Performance). Commercial producers of rankings don’t do it, because to do so would reduce the totally spurious impression of precision in the numbers they sell. Vice-chancellors might bully staff less if they knew that the changes they produce are mere random errors.
Metrics, and still more altmetrics, are far too crude to measure the quality of science. To hope to do that without reading the paper is pie in the sky (even reading it, it’s often impossible to tell).
The only bit of the report that I’m not entirely happy about is the recommendation to spend more money investigating the metrics that the report has just debunked. It seems to me that there will never be a way of measuring the quality of work without reading it. To spend money on a futile search for new metrics would take money away from science itself. I’m not convinced that it would be money well-spent.
Follow-up
Stop press. Financial report casts doubt on Trainor’s claims
Science has a big problem. Most jobs are desperately insecure. It’s hard to do long term thorough work when you don’t know whether you’ll be able to pay your mortgage in a year’s time. The appalling career structure for young scientists has been the subject of much writing by the young (e.g. Jenny Rohn) and the old, e.g Bruce Alberts. Peter Lawrence (see also Real Lives and White Lies in the Funding of Scientific Research, and by me.
Until recently, this problem was largely restricted to post-doctoral fellows (postdocs). They already have PhDs and they are the people who do most of the experiments. Often large numbers of them work for a single principle investigator (PI). The PI spends most of his her time writing grant applications and traveling the world to hawk the wares of his lab. They also (to variable extents) teach students and deal with endless hassle from HR.
The salaries of most postdocs are paid from grants that last for three or sometimes five years. If that grant doesn’t get renewed. they are on the streets.
Universities have come to exploit their employees almost as badly as Amazon does.
The periodical research assessments not only waste large amounts of time and money, but they have distorted behaviour. In the hope of scoring highly, they recruit a lot of people before the submission, but as soon as that’s done with, they find that they can’t afford all of them, so some get cast aside like worn out old boots. Universities have allowed themselves to become dependent on "soft money" from grant-giving bodies. That strikes me as bad management.
The situation is even worse in the USA where most teaching staff rely on research grants to pay their salaries.
I have written three times about the insane methods that are being used to fire staff at Queen Mary College London (QMUL).
Is Queen Mary University of London trying to commit scientific suicide? (June 2012)
Queen Mary, University of London in The Times. Does Simon Gaskell care? (July 2012) and a version of it appeared th The Times (Thunderer column)
In which Simon Gaskell, of Queen Mary, University of London, makes a cock-up (August 2012)
The ostensible reason given there was to boost its ratings in university rankings. Their vice-chancellor, Simon Gaskell, seems to think that by firing people he can produce a university that’s full of Nobel prize-winners. The effect, of course, is just the opposite. Treating people like pawns in a game makes the good people leave and only those who can’t get a job with a better employer remain. That’s what I call bad management.
At QMUL people were chosen to be fired on the basis of a plain silly measure of their publication record, and by their grant income. That was combined with terrorisation of any staff who spoke out about the process (more on that coming soon).
Kings College London is now doing the same sort of thing. They have announced that they’ll fire 120 of the 777 staff in the schools of medicine and biomedical sciences, and the Institute of Psychiatry. These are humans, with children and mortgages to pay. One might ask why they were taken on the first place, if the university can’t afford them. That’s simply bad financial planning (or was it done in order to boost their Research Excellence submission?).
Surely it’s been obvious, at least since 2007, that hard financial times were coming, but that didn’t dent the hubris of the people who took an so many staff. HEFCE has failed to find a sensible way to fund universities. The attempt to separate the funding of teaching and research has just led to corruption.
The way in which people are to be chosen for the firing squad at Kings is crude in the extreme. If you are a professor at the Institute of Psychiatry then, unless you do a lot of teaching, you must have a grant income of at least £200,000 per year. You can read all the details in the Kings’ “Consultation document” that was sent to all employees. It’s headed "CONFIDENTIAL – Not for further circulation". Vice-chancellors still don’t seem to have realised that it’s no longer possible to keep things like this secret. In releasing it, I take ny cue from George Orwell.
"Journalism is printing what someone else does not want printed: everything else is public relations.”
There is no mention of the quality of your research, just income. Since in most sorts of research, the major cost is salaries, this rewards people who take on too many employees. Only too frequently, large groups are the ones in which students and research staff get the least supervision, and which bangs per buck are lowest. The university should be rewarding people who are deeply involved in research themselves -those with small groups. Instead, they are doing exactly the opposite.
Women are, I’d guess, less susceptible to the grandiosity of the enormous research group, so no doubt they will suffer disproportionately. PhD students will also suffer if their supervisor is fired while they are halfway through their projects.
An article in Times Higher Education pointed out
"According to the Royal Society’s 2010 report The Scientific Century: Securing our Future Prosperity, in the UK, 30 per cent of science PhD graduates go on to postdoctoral positions, but only around 4 per cent find permanent academic research posts. Less than half of 1 per cent of those with science doctorates end up as professors."
The panel that decides whether you’ll be fired consists of Professor Sir Robert Lechler, Professor Anne Greenough, Professor Simon Howell, Professor Shitij Kapur, Professor Karen O’Brien, Chris Mottershead, Rachel Parr & Carol Ford. If they had the slightest integrity, they’d refuse to implement such obviously silly criteria.
Universities in general. not only Kings and QMUL have become over-reliant on research funders to enhance their own reputations. PhD students and research staff are employed for the benefit of the university (and of the principle investigator), not for the benefit of the students or research staff, who are treated as expendable cost units, not as humans.
One thing that we expect of vice-chancellors is sensible financial planning. That seems to have failed at Kings. One would also hope that they would understand how to get good science. My only previous encounter with Kings’ vice chancellor, Rick Trainor, suggests that this is not where his talents lie. While he was president of the Universities UK (UUK), I suggested to him that degrees in homeopathy were not a good idea. His response was that of the true apparatchik.
“. . . degree courses change over time, are independently assessed for academic rigour and quality and provide a wider education than the simple description of the course might suggest”
That is hardly a response that suggests high academic integrity.
The students’ petition is on Change.org.
Follow-up
The problems that are faced in the UK are very similar to those in the USA. They have been described with superb clarity in “Rescuing US biomedical research from its systemic flaws“, This article, by Bruce Alberts, Marc W. Kirschner, Shirley Tilghman, and Harold Varmus, should be read by everyone. They observe that ” . . . little has been done to reform the system, primarily because it continues to benefit more established and hence more influential scientists”. I’d be more impressed by the senior people at Kings if they spent time trying to improve the system rather than firing people because their research is not sufficiently expensive.
10 June 2014
Progress on the cull, according to an anonymous correspondent
“The omnishambles that is KCL management
1) We were told we would receive our orange (at risk) or green letters (not at risk, this time) on Thursday PM 5th June as HR said that it’s not good to get bad news on a Friday!
2) We all got a letter on Friday that we would not be receiving our letters until Monday, so we all had a tense weekend
3) I finally got my letter on Monday, in my case it was “green” however a number of staff who work very hard at KCL doing teaching and research are “orange”, un bloody believable
As you can imagine the moral at King’s has dropped through the floor”
18 June 2014
Dorothy Bishop has written about the Trainor problem. Her post ends “One feels that if KCL were falling behind in a boat race, they’d respond by throwing out some of the rowers”.
The students’ petition can be found on the #KCLHealthSOS site. There is a reply to the petition, from Professor Sir Robert Lechler, and a rather better written response to it from students. Lechler’s response merely repeats the weasel words, and it attacks a few straw men without providing the slightest justification for the criteria that are being used to fire people. One can’t help noticing how often knighthoods go too the best apparatchiks rather than the best scientists.
14 July 2014
A 2013 report on Kings from Standard & Poor’s casts doubt on Trainor’s claims
Download the report from Standard and Poor’s Rating Service
A few things stand out.
- KCL is in a strong financial position with lower debt than other similar Universities and cash reserves of £194 million.
- The report says that KCL does carry some risk into the future especially that related to its large capital expansion program.
- The report specifically warns KCL over the consequences of any staff cuts. Particularly relevant are the following quotations
- Page p3 “Further staff-cost curtailment will be quite difficult …pressure to maintain its academic and non-academic service standards will weigh on its ability to cut costs further.”
- page 4 The report goes on to say (see the section headed outlook, especially the final paragraph) that any decrease in KCL’s academic reputation (e.g. consequent on staff cuts) would be likely to impair its ability to attract overseas students and therefore adversely affect its financial position.
- page 10 makes clear that KCL managers are privately aiming at 10% surplus, above the 6% operating surplus they talk about with us. However, S&P considers that ‘ambitious’. In other words KCL are shooting for double what a credit rating agency considers realistic.
One can infer from this that
- what staff have been told about the cuts being an immediate necessity is absolute nonsense
- KCL was warned against staff cuts by a credit agency
- the main problem KCL has is its overambitious building policy
- KCL is implementing a policy (staff cuts) which S & P warned against as they predict it may result in diminishing income.
What on earth is going on?
16 July 2014
I’ve been sent yet another damning document. The BMA’s response to Kings contains some numbers that seem to have escaped the attention of managers at Kings.
10 April 2015
King’s draft performance management plan for 2015
This document has just come to light (the highlighting is mine).
It’s labelled as "released for internal consultation". It seems that managers are slow to realise that it’s futile to try to keep secrets.
The document applies only to Institute of Psychiatry, Psychology and Neuroscience at King’s College London: "one of the global leaders in the fields" -the usual tedious blah that prefaces every document from every university.
It’s fascinating to me that the most cruel treatment of staff so often seems to arise in medical-related areas. I thought psychiatrists, of all people, were meant to understand people, not to kill them.
This document is not quite as crude as Imperial’s assessment, but it’s quite bad enough. Like other such documents, it pretends that it’s for the benefit of its victims. In fact it’s for the benefit of willy-waving managers who are obsessed by silly rankings.
Here are some of the sillier bits.
"The Head of Department is also responsible for ensuring that aspects of reward/recognition and additional support that are identified are appropriately followed through"
And, presumably, for firing people, but let’s not mention that.
"Academics are expected to produce original scientific publications of the highest quality that will significantly advance their field."
That’s what everyone has always tried to do. It can’t be compelled by performance managers. A large element of success is pure luck. That’s why they’re called experiments.
" However, it may take publications 12-18 months to reach a stable trajectory of citations, therefore, the quality of a journal (impact factor) and the judgment of knowledgeable peers can be alternative indicators of excellence."
It can also take 40 years for work to be cited. And there is little reason to believe that citations, especially those within 12-18 months, measure quality. And it is known for sure that "the quality of a journal (impact factor)" does not correlate with quality (or indeed with citations).
Later we read
"H Index and Citation Impact: These are good objective measures of the scientific impact of
publications"
NO, they are simply not a measure of quality (though this time they say “impact” rather than “excellence”).
The people who wrote that seem to be unaware of the most basic facts about science.
Then
"Carrying out high quality scientific work requires research teams"
Sometimes it does, sometimes it doesn’t. In the past the best work has been done by one or two people. In my field, think of Hodgkin & Huxley, Katz & Miledi or Neher & Sakmann. All got Nobel prizes. All did the work themselves. Performance managers might well have fired them before they got started.
By specifying minimum acceptable group sizes, King’s are really specifying minimum acceptable grant income, just like Imperial and Warwick. Nobody will be taken in by the thin attempt to disguise it.
The specification that a professor should have "Primary supervision of three or more PhD students, with additional secondary supervision." is particularly iniquitous. Everyone knows that far too many PhDs are being produced for the number of jobs that are available. This stipulation is not for the benefit of the young. It’s to ensure a supply of cheap labour to churn out more papers and help to lift the university’s ranking.
The document is not signed, but the document properties name its author. But she’s not a scientist and is presumably acting under orders, so please don’t blame her for this dire document. Blame the vice-chancellor.
Performance management is a direct incentive to do shoddy short-cut science.
No wonder that The Economist says "scientists are doing too much trusting and not enough verifying—to the detriment of the whole of science, and of humanity".
Feel ashamed.
The Scottish Universities Medical Journal asked me to write about the regulation of alternative medicine. It’s an interesting topic and not easy to follow because of the veritable maze of more than twenty overlapping regulators and quangos which fail utterly to protect the public against health fraud. In fact they mostly promote health fraud. The paper is now published, and here is a version with embedded links (and some small updates).
We are witnessing an increasing commercialisation of medicine. It’s really taken off since the passage of the Health and Social Security Bill into law. Not only does that mean having NHS hospitals run by private companies, but it means that “any qualified provider” can bid for just about any service. The problem lies, of course, in what you consider “qualified” to mean. Any qualified homeopath or herbalist will, no doubt, be eligible. University College London Hospital advertised for a spiritual healer. The "person specification" specified a "quallfication", but only HR people think that a paper qualification means that spiritual healing is anything but a delusion.
The vocabulary of bait and switch
First, a bit of vocabulary. Alternative medicine is a term that is used for medical treatments that don’t work (or at least haven’t been shown to work). If they worked, they’d be called “medicine”. The anti-malarial, artemesinin, came originally from a Chinese herb, but once it had been purified and properly tested, it was no longer alternative. But the word alternative is not favoured by quacks. They prefer their nostrums to be described as “complementary” –it sounds more respectable. So CAM (complementary and alternative medicine became the politically-correct euphemism. Now it has gone a stage further, and the euphemism in vogue with quacks at the moment is “integrated” or “integrative” medicine. That means, very often, integrating things that don’t work with things that do. But it sounds fashionable. In reality it is designed to confuse politicians who ask for, say, integrated services for old people.
Put another way, the salespeople of quackery have become rather good at bait and switch. The wikepedia definition is as good as any.
Bait-and-switch is a form of fraud, most commonly used in retail sales but also applicable to other contexts. First, customers are “baited” by advertising for a product or service at a low price; second, the customers discover that the advertised good is not available and are “switched” to a costlier product.
As applied to the alternative medicine industry, the bait is usually in the form of some nice touchy-feely stuff which barely mentions the mystical nonsense. But when you’ve bought into it you get the whole panoply of nonsense. Steven Novella has written eloquently about the use of bait and switch in the USA to sell chiropractic, acupuncture, homeopathy and herbal medicine: "The bait is that CAM offers legitimate alternatives, the switch is that it primarily promotes treatments that don’t work or are at best untested and highly implausible.".
The "College of Medicine" provides a near-perfect example of bait and switch. It is the direct successor of the Prince of Wales’ Foundation for Integrated Health. The Prince’s Foundation was a consistent purveyor of dangerous medical myths. When it collapsed in 2010 because of a financial scandal, a company was formed called "The College for Integrated Health". A slide show, not meant for public consumption, said "The College represents a new strategy to take forward the vision of HRH Prince Charles". But it seems that too many people have now tumbled to the idea that "integrated", in this context, means barmpottery. Within less than a month, the new institution was renamed "The College of Medicine". That might be a deceptive name, but it’s a much better bait. That’s why I described the College as a fraud and delusion.
Not only did the directors, all of them quacks, devise a respectable sounding name, but they also succeeded in recruiting some respectable-sounding people to act as figureheads for the new organisation. The president of the College is Professor Sir Graham Catto, emeritus professor of medicine at the University of Aberdeen. Names like his make the bait sound even more plausible. He claims not to believe that homeopathy works, but seems quite happy to have a homeopathic pharmacist, Christine Glover, on the governing council of his college. At least half of the governing Council can safely be classified as quacks.
So the bait is clear. What about the switch? The first thing to notice is that the whole outfit is skewed towards private medicine: see The College of Medicine is in the pocket of Crapita Capita. The founder, and presumably the main provider of funds (they won’t say how much) is the huge outsourcing company, Capita. This is company known in Private Eye as Crapita. Their inefficiency is legendary. They are the folks who messed up the NHS computer system and the courts computer system. After swallowing large amounts of taxpayers’ money, they failed to deliver anything that worked. Their latest failure is the court translation service.. The president (Catto), the vice president (Harry Brunjes) and the CEO (Mark Ratnarajah) are all employees of Capita.
The second thing to notice is that their conferences and courses are a bizarre mixture of real medicine and pure quackery. Their 2012 conference had some very good speakers, but then it had a "herbal workshop" with Simon Mills (see a video) and David Peters (the man who tolerates dowsing as a way to diagnose which herb to give you). The other speaker was Dick Middleton, who represents the huge herbal company, Schwabe (I debated with him on BBC Breakfast), In fact the College’s Faculty of Self-care appears to resemble a marketing device for Schwabe.
Why regulation isn’t working, and can’t work
There are various levels of regulation. The "highest" level is the statutory regulation of osteopathy and chiropractic. The General Chiropractic Council (GCC) has exactly the same legal status as the General Medical Council (GMC). This ludicrous state of affairs arose because nobody in John Major’s government had enough scientific knowledge to realise that chiropractic, and some parts of osteopathy, are pure quackery,
The problem is that organisations like the GCC function more to promote chiropractic than to regulate them. This became very obvious when the British Chiropractic Association (BCA) decided to sue Simon Singh for defamation, after he described some of their treatments as “bogus”, “without a jot of evidence”.
In order to support Singh, several bloggers assessed the "plethora of evidence" which the BCA said could be used to justify their claims. When, 15 months later, the BCA produced its "plethora" it was shown within 24 hours that the evidence was pathetic. The demolition was summarised by lawyer, David Allen Green, in The BCA’s Worst Day.
In the wake of this, over 600 complaints were made to the GCC about unjustified claims made by chiropractors, thanks in large part to heroic work by two people, Simon Perry and Allan Henness. Simon Perry’s Fishbarrel (browser plugin) allows complaints to be made quickly and easily -try it). The majority of these complaints were rejected by the GCC, apparently on the grounds that chiropractors could not be blamed because the false claims had been endorsed by the GCC itself.
My own complaint was based on phone calls to two chiropractors, I was told such nonsense as "colic is down to, er um, faulty movement patterns in the spine". But my complaint never reached the Conduct and Competence committee because it had been judged by a preliminary investigating committee that there was no case to answer. The impression one got from this (very costly) exercise was that the GCC was there to protect chiropractors, not to protect the public.
The outcome was a disaster for chiropractors, wno emerged totally discredited. It was also a disaster for the GCC which was forced to admit that it hadn’t properly advised chiropractors about what they could and couldn’t claim. The recantation culminated in the GCC declaring, in August 2010, that the mythical "subluxation" is a "historical concept " "It is not supported by any clinical research evidence that would allow claims to be made that it is the cause of disease.". Subluxation was a product of the fevered imagination of the founder of the chiropractic cult, D.D. Palmer. It referred to an imaginary spinal lesion that he claimed to be the cause of most diseases. .Since ‘subluxation’ is the only thing that’s distinguished chiropractic from any other sort of manipulation, the admission by the GCC that it does not exist, after a century of pretending that it does, is quite an admission.
The President of the BCA himself admitted in November 2011
“The BCA sued Simon Singh personally for libel. In doing so, the BCA began one of the darkest periods in its history; one that was ultimately to cost it financially,”
As a result of all this, the deficiencies of chiropractic, and the deficiencies of its regulator were revealed, and advertisements for chiropractic are somewhat less misleading. But this change for the better was brought about entirely by the unpaid efforts of bloggers and a few journalists, and not at all by the official regulator, the GCC. which was part of the problem. not the solution. And it was certainly not helped by the organisation that is meant to regulate the GCC, the Council for Health Regulatory Excellence (CHRE) which did nothing whatsoever to stop the farce.
At the other end of the regulatory spectrum, voluntary self-regulation, is an even worse farce than the GCC. They all have grand sounding "Codes of Practice" which, in practice, the ignore totally.
The Society of Homeopaths is just a joke. When homeopaths were caught out recommending sugar pills for prevention of malaria, they did nothing (arguably such homicidal advice deserves a jail sentence).
The Complementary and Natural Healthcare Council (CNHC) is widely know in the blogosphere as Ofquack. I know about them from the inside, having been a member of their Conduct and Competence Committee, It was set up with the help of a £900,000 grant from the Department of Health to the Prince of Wales, to oversee voluntary self-regulation. It fails utterly to do anything useful.. The CNHC code of practice, paragraph 15 , states
“Any advertising you undertake in relation to your professional activities must be accurate. Advertisements must not be misleading, false, unfair or exaggerated”.
When Simon Perry made a complaint to the CNHC about claims being made by a CNHC-registered reflexologist, the Investigating Committee upheld all 15 complaints. But it then went on to say that there was no case to answer because the unjustified claims were what the person had been taught, and were made in good faith.
This is precisely the ludicrous situation which will occur again and again if reflexologists (and many other alternative therapies) are “accredited”. The CNHC said, correctly, that the reflexologist had been taught things that were not true, but then did nothing whatsoever about it apart from toning down the advertisements a bit. They still register reflexologists who make outrageously false claims.
Once again we see that no sensible regulation is possible for subjects that are pure make-believe.
The first two examples deal (or rather, fail to deal) with regulation of outright quackery. But there are dozens of other quangos that sound a lot more respectable.
European Food Standards Agency (EFSA). One of the common scams is to have have your favourite quack treatment classified as a food not as a medicine. The laws about what you can claim have been a lot laxer for foods. But the EFSA has done a pretty good job in stopping unjustified claims for health benefits from foods. Dozens of claims made by makers of probiotics have been banned. The food industry, needless to say, objects very strongly to be being forced to tell the truth. In my view, the ESFA has not gone far enough. They recently issued a directive about claims that could legally be made. Some of these betray the previously high standards of the EFSA. For example you are allowed to say that "Vitamin C contributes to the reduction of tiredness and fatigue" (as long as the product contains above a specified amount of Vitamin C. I’m not aware of any trials that show vitamin C has the slightest effect on tiredness or fatigue, Although these laws do not come into effect until December 2012, they have already been invoked by the ASA has a reason not to uphold a complaint about a multivitamin pill which claimed that it “Includes 8 nutrients that can contribute to the reduction in tiredness and fatigue”
The Advertising Standards Authority (ASA). This is almost the only organisation that has done a good job on false health claims. Their Guidance on Health Therapies & Evidence says
"Whether you use the words ‘treatment’, ‘treat’ or ‘cure’, all are likely to be seen by members of the public as claims to alleviate effectively a condition or symptom. We would advise that they are not used"
"Before and after’ studies with little or no control, studies without human subjects, self-assessment studies and anecdotal evidence are unlikely to be considered acceptable"
"Before and after’ studies with little or no control, studies without human subjects, self-assessment studies and anecdotal evidence are unlikely to be considered acceptable"
They are spot on.
The ASA’s Guidance for Advertisers of Homeopathic Services is wonderful.
"In the simplest terms, you should avoid using efficacy claims, whether implied or direct,"
"To date, the ASA has have not seen persuasive evidence to support claims that homeopathy can treat, cure or relieve specific conditions or symptoms."
That seems to condemn the (mis)labelling allowed by the MHRA as breaking the rules.. Sadly, though, the ASA has no powers to enforce its decisions and only too often they are ignored. The Nightingale collaboration has produced an excellent letter that you can hand to any pharmacist who breaks the rules
The ASA has also judged against claims made by "Craniosacral therapists" (that’s the lunatic fringe of osteopathy). They will presumably uphold complaints about similar claims made (I’m ashamed to say) by UCLH Hospitals.
The private examination company Edexcel sets exams in antiscientific subjects, so miseducating children. The teaching of quackery to 16 year-olds has been approved by a maze of quangos, none of which will take responsibility, or justify their actions. So far I’ve located no fewer than eight of them. The Office of the Qualifications and Examinations Regulator (OfQual), Edexcel, the Qualifications and Curriculum Authority (QCA), Skills for Health, Skills for Care, National Occupational Standards (NOS), private exam company VTCT and the schools inspectorate, Ofsted.. Asking any of these people why they approve of examinations in imaginary subjects meets with blank incomprehension. They fail totally to protect tha public from utter nonsense.
The Department of Education has failed to do anything about the miseducation of children in quackery. In fact it has encouraged it by, for the first time, giving taxpayers’ money to a Steiner (Waldorf) school (at Frome, in Somerset). Steiner schools are run by a secretive and cult-like body of people (read about it). They teach about reincarnation, karma, gnomes, and all manner of nonsense, sometimes with unpleasant racial overtones. The teachers are trained in Steiner’s Anthroposophy, so if your child gets ill at school they’ll probably get homeopathic sugar pills. They might well get measles or mumps too, since Steiner people don’t believe in vaccination.
Incredibly, the University of Aberdeen came perilously close to appointing a chair in anthroposophical medicine. This disaster was aborted by bloggers, and a last minute intervention from journalists. Neither the university’s regulatory mechanisms. nor any others, seemed to realise that a chair in mystical barmpottery was a bad idea.
Trading Standards offices and the Office of Fair Trading.
It is the statutory duty of Trading Standards to enforce the Consumer Protection Regulations (2008) This European legislation is pretty good. it caused a lawyer to write " Has The UK Quietly Outlawed “Alternative” Medicine?". Unfortunately Trading Standards people have consistently refused to enforce these laws. The whole organisation is a mess. Its local office arrangement fails totally to deal with the age of the internet. The situation is so bad that a group of us decided to put them to the test. The results were published in the Medico-Legal Journal, Rose et al., 2012. "Spurious Claims for Health-care Products: An Experimental Approach to Evaluating Current UK Legislation and its Implementation". They concluded "EU directive 2005/29/EC is
largely ineffective in preventing misleading health claims for consumer products in
the UK"
Skills for Health is an enormous quango which produces HR style "competences" for everything under the son. They are mostly quite useless. But those concerned with alternative medicine are not just useless. They are positively harmful. Totally barmy. There are competences and National Occupational Standards for every lunatic made-up therapy under the sun. When I phoned them to discover who’d written them, I learned that the had been drafted by the Prince of Wales’ Foundation for Magic Medicine. And when I joked by asking if they had a competence for talking to trees, I was told, perfectly seriously, “You’d have to talk to LANTRA, the land-based organisation for that.”
That was in January 2008. A lot of correspondence with the head of Skills for Health got nowhere at all. She understood nothing and it hasn’t improved a jot.
This organisation costs a lot of taxpayers’ money and it should have been consigned to the "bonfire of the quangos" (but of course there was no such bonfire in reality). It is a disgrace.
The Quality Assurance Agency (QAA) is supposed to ensure the quality of university courses. In fact it endorses courses in nonsense alternative medicine and so does more harm than good. The worst recent failure of the QAA was in the case of the University of Wales: see Scandal of the University of Wales and the Quality Assurance Agency. The university was making money by validating thousands of external degrees in everything from fundamentalist theology to Chinese Medicine. These validations were revealed as utterly incompetent by bloggers, and later by BBC Wales journalist Ciaran Jenkins (now working for Channel 4).
The mainstream media eventually caught up with bloggers. In 2010, BBC1 TV (Wales) produced an excellent TV programme that exposed the enormous degree validation scam run by the University of Wales. The programme can be seen on YouTube (Part 1, and Part 2). The programme also exposed, incidentally, the uselessness of the Quality Assurance Agency (QAA) which did nothing until the scam was exposed by TV and blogs. Eventually the QAA sent nine people to Malaysia to investigate a dodgy college that had been revealed by the BBC. The trip cost £91,000. It could have been done for nothing if anyone at the QAA knew how to use Google.
The outcome was that the University of Wales stopped endorsing external courses, and it was soon shut down altogether (though bafflingly, its vice-chancellor, Marc Clement was promoted). The credit for this lies entirely with bloggers and the BBC. The QAA did nothing to help until the very last moment.
Throughout this saga Universities UK (UUK), has maintained its usual total passivity. They have done nothing whatsoever about their members who give BSc degrees in anti-scientific subjects. (UUK used to known as the Committee of Vice-Chancellors and Principals).
Council for Health Regulatory Excellence (CHRE), soon to become the PSAHSC,
Back now to the CHRE, the people who failed so signally to sort out the GCC. They are being reorganised. Their consultation document says
"The Health and Social Care Act 20122 confers a new function on the Professional Standards Authority for Health and Social Care (the renamed Council for Healthcare Regulatory Excellence). From November 2012 we will set standards for organisations that hold voluntary registers for people working in health and social care occupations and we will accredit the register if they meet those standards. It will then be known as an ‘Accredited Register’. "
They are trying to decide what the criteria should be for "accreditation" of a regulatory body. The list of those interested has some perfectly respectable organisations, like the British Psychological Society. It also contains a large number of crackpot organisations, like Crystal and Healing International, as well as joke regulators like the CNHC.
They already oversee the Health Professions Council (HPC) which is due to take over Herbal medicine and Traditional Chinese Medicine, with predictably disastrous consequences.
Two of the proposed criteria for "accreditation" appear to be directly contradictory.
Para 2.5 makes the whole accreditation pointless from the point of view of patients
2.5 It will not be an endorsement of the therapeutic validity or effectiveness of any particular discipline or treatment.
Since the only thing that matters to the patient is whether the therapy works (and is safe), accrediting of organisations that ignore this will merely give the appearance of official approval of crystal healing etc etc. This appears to contradict directly
A.7 The organisation can demonstrate that there either is a sound knowledge base underpinning the profession or it is developing one and makes that explicit to the public.
A "sound knowledge base", if it is to mean anything useful at all, means knowledge that the treatment is effective. If it doesn’t mean that, what does it mean?
It seems that the official mind has still not grasped the obvious fact that there can be no sensible regulation of subjects that are untrue nonsense. If it is nonsense, the only form of regulation that makes any sense is the law.
Please fill in the consultation. My completed return can be downloaded as an example, if you wish.
Medicines and Healthcare products Regulatory Agency (MHRA) should be a top level defender of truth. Its strapline is
"We enhance and safeguard the health of the public by ensuring that medicines and medical devices work and are acceptably safe."
The MHRA did something (they won’t tell me exactly what) about one of the most cruel scams that I’ve ever encountered, Esperanza Homeopathic Neuropeptide, peddled for multiple sclerosis, at an outrageous price ( £6,759 for 12 month’s supply). Needless to say there was not a jot of evidence that it worked (and it wasn’t actually homeopathic).
Astoundingly, Trading Standards officers refused to do anything about it.
The MHRA admit (when pushed really hard) that there is precious little evidence that any of the herbs work, and that homeopathy is nothing more than sugar pills. Their answer to that is to forget that bit about "ensuring that medicines … work"
Here’s the MHRA’s Traditional Herbal Registration Certificate for devils claw tablets.
The wording "based on traditional use only" has to be included because of European legislation. Shockingly, the MHRA have allowed them to relegate that to small print, with all the emphasis on the alleged indications. The pro-CAM agency NCCAM rates devil’s claw as "possibly effective" or "insufficient evidence" for all these indications, but that doesn’t matter because the MHRA requires no evidence whatsoever that the tablets do anything. They should, of course, added a statement to this effect to the label. They have failed in their duty to protect and inform the public by allowing this labelling.
But it gets worse. Here is the MHRA’s homeopathic marketing authorisation for the homeopathic medicinal product Arnicare Arnica 30c pillules
It is nothing short of surreal.
|
|
Since the pills contain nothing at all, they don’t have the slightest effect on sprains, muscular aches or bruising. The wording on the label is exceedingly misleading.
If you "pregnant or breastfeeding" there is no need to waste you doctor’s time before swallowing a few sugar pills.
"Do not take a double dose to make up for a missed one". Since the pills contain nothing, it doesn’t matter a damn.
"If you overdose . . " it won’t have the slightest effect because there is nothing in them
And it gets worse. The MHRA-approved label specifies ACTIVE INGREDIENT. Each pillule contains 30c Arnica Montana
No, they contain no arnica whatsoever.
|
|
It truly boggles the mind that men with dark suits and lots of letters after their names have sat for hours only to produce dishonest and misleading labels like these.
When this mislabeling was first allowed, it was condemned by just about every scientific society, but the MHRA did nothing.
The Nightingale Collaboration.
This is an excellent organisation, set up by two very smart skeptics, Alan Henness and Maria MacLachlan. Visit their site regularly, sign up for their newsletter Help with their campaigns. Make a difference.
Conclusions
The regulation of alternative medicine in the UK is a farce. It is utterly ineffective in preventing deception of patients.
Such improvements as have occurred have resulted from the activity of bloggers, and sometime the mainstream media. All the official regulators have, to varying extents, made things worse.
The CHRE proposals promise to make matters still worse by offering "accreditation" to organisations that promote nonsensical quackery. None of the official regulators seem to be able to grasp the obvious fact that is impossible to have any sensible regulation of people who promote nonsensical untruths. One gets the impression that politicians are more concerned to protect the homeopathic (etc, etc) industry than they are to protect patients.
Deception by advocates of alternative medicine harms patients. There are adequate laws that make such deception illegal, but they are not being enforced. The CHRE and its successor should restrict themselves to real medicine. The money that they spend on pseudo-regulation of quacks should be transferred to the MHRA or a reformed Trading Standards organisation so they can afford to investigate and prosecute breaches of the law. That is the only form of regulation that makes sense.
Follow-up
The shocking case of the continuing sale of “homeopathic vaccines” for meningitis, rubella, pertussis etc was highlighted in an excellent TV programme by BBC South West. The failure of the MHRA and the GPC do take any effective action is a yet another illustration of the failure of regulators to do their job. I have to agree with Andy Lewis when he concludes
“Children will die. And the fault must lie with Professor Sir Kent Woods, chairman of the regulator.”
This is a slightly-modified version of the article that appeared in BMJ blogs yesterday, but with more links to original sources, and a picture. There are already some comments in the BMJ.
The original article, diplomatically, did not link directly to UCL’s Grand Challenge of Human Wellbeing, a well-meaning initiative which, I suspect, will not prove to be value for money when it comes to practical action.
Neither, when referring to the bad effects of disempowerment on human wellbeing (as elucidated by, among others, UCL’s Michael Marmot), did I mention the several ways in which staff have been disempowered and rendered voiceless at UCL during the last five years. Although these actions have undoubtedly had a bad effect on the wellbeing of UCL’s staff, it seemed a litlle unfair to single out UCL since similar things are happening in most universities. Indeed the fact that it has been far worse at Imperial College (at least in medicine) has probably saved UCL from being denuded. One must be thankful for small mercies.
There is, i think, a lesson to be learned from the fact that formal initiatives in wellbeing are springing up at a time when university managers are set on taking actions that have exactly the opposite effect. A ‘change manager’ is not an adequate substitute for a vote. Who do they imagine is being fooled?
Click logo to go to BMJ
The A to Z of the wellbeing industry
From angelic reiki to patient-centred care
Nobody could possibly be against wellbeing. It would be like opposing motherhood and apple pie. There is a whole spectrum of activities under the wellbeing banner, from the undoubtedly well-meaning patient-centred care at one end, to downright barmy new-age claptrap at the other end. The only question that really matters is, how much of it works?
Let’s start at the fruitcake end of the spectrum.
One thing is obvious. Wellbeing is big business. And if it is no more than a branch of the multi-billion-dollar positive-thinking industry, save your money and get on with your life.
In June 2010, Northamptonshire NHS Foundation Trust sponsored a “Festival of Wellbeing” that included a complementary therapy taster day. In a BBC interview one practitioner used the advertising opportunity, paid for by the NHS, to say “I’m an angelic reiki master teacher and also an angel therapist.” “Angels are just flying spirits, 100 percent just pure light from heaven. They are all around us. Everybody has a guardian angel.” Another said “I am a member of the British Society of Dowsers and use a crystal pendulum to dowse in treatment sessions. Sessions may include a combination of meditation, colour breathing, crystals, colour scarves, and use of a light box.” You couldn’t make it up.
The enormous positive-thinking industry is no better. Barbara Ehrenreich’s book, Smile Or Die: How Positive Thinking Fooled America and the World, explains how dangerous the industry is, because, as much as guardian angels, it is based on myth and delusion. It simply doesn’t work (except for those who make fortunes by promoting it). She argues that it fosters the sort of delusion that gave us the financial crisis (and pessimistic bankers were fired for being right). Her interest in the industry started when she was diagnosed with cancer. She says
”When I was diagnosed, what I found was constant exhortations to be positive, to be cheerful, to even embrace the disease as if it were a gift. If that’s a gift, take me off your Christmas list,”
It is quite clear that positive thinking does nothing whatsoever to prolong your life (Schofield et al 2004; Coyne et al 2007; 2,3), any more than it will cure tuberculosis or cholera. “Encouraging patients to “be positive” only may add to the burden of having cancer while providing little benefit” (Schofield et al 2004). Far from being helpful, it can be rather cruel.
Just about every government department, the NHS, BIS, HEFCE, and NICE, has produced long reports on wellbeing and stress at work. It’s well known that income is correlated strongly with health (Marmot, M., 2004). For every tube stop you go east of Westminster you lose a year of life expectancy (London Health Observatory). It’s been proposed that what matters is inequality of income (Wilkinson & Pickett, 2009). The nature of the evidence doesn’t allow such a firm conclusion (Lynch et al. 2004), but that isn’t really the point. The real problem is that nobody has come up with good solutions. Sadly the recommendations at the ends of all these reports don’t amount to a hill of beans. Nobody knows what to do, partly because pilot studies are rarely randomised so causality is always dubious, and partly because the obvious steps are either managerially inconvenient, ideologically unacceptable, or too expensive.
Take two examples:
Sir Michael Marmot’s famous Whitehall study (Marmot, M., 2004) has shown that a major correlate of illness is lack of control over one’s own fate: disempowerment. What has been done about it?
In universities it has proved useful to managers to increase centralisation and to disempower academics, precisely the opposite of what Marmot recommends.
|
As long as it’s convenient to managers they are not going to change policy. Rather, they hand the job to the HR department which appoints highly paid “change managers,” who add to the stress by sending you stupid graphs that show you emerging from the slough of despond into eternal light once you realise that you really wanted to be disempowered after all. Or they send you on some silly “resilience” course. |
|
A second example comes from debt. According to a BIS report (Mental Capital and Wellbeing), debt is an even stronger risk factor for mental disorder than low income. So what is the government’s response to that? To treble tuition fees to ensure that almost all graduates will stay in debt for most of their lifetime. And this was done despite the fact that the £9k fees will save nothing for the taxpayer: in fact they’ll cost more than the £3k fees. The rise has happened, presumably, because the ideological reasons overrode the government’s own ideas on how to make people happy.
Nothing illustrates better the futility of the wellbeing industry than the response that is reported to have been given to a reporter who posed as an applicant for a “health, safety, and wellbeing adviser” with a local council. When he asked what “wellbeing” advice would involve, a member of the council’s human resources team said: “We are not really sure yet as we have only just added that to the role. We’ll want someone to make sure that staff take breaks, go for walks — that kind of stuff.”
The latest wellbeing notion to re-emerge is the happiness survey. Jeremy Bentham advocated “the greatest happiness for the greatest number,” but neglected to say how you measure it. A YouGov poll asks, “what about your general well-being right now, on a scale from 1 to 10.” I have not the slightest idea about how to answer such a question. As always some things are good, some are bad, and anyway wellbeing relative to whom? Writing this is fun. Trying to solve an algebraic problem is fun. Constant battling with university management in order to be able to do these things is not fun. The whole exercise smacks of the sort of intellectual arrogance that led psychologists in the 1930s to claim that they could sum up a person’s intelligence in a single number. That claim was wrong and it did great social harm.
HEFCE has spent a large amount of money setting up “pilot studies” of wellbeing in nine universities. Only one is randomised, so there will be no evidence for causality. The design of the pilots is contracted to a private company, Robertson Cooper, which declines to give full details but it seems likely that the results will be about as useless as the notorious Durham fish oil “trials”(Goldacre, 2008).
Lastly we get to the sensible end of the spectrum: patient-centred care. Again this has turned into an industry with endless meetings and reports and very few conclusions. Epstein & Street (2011) say
“Helping patients to be more active in consultations changes centuries of physician-dominated dialogues to those that engage patients as active participants. Training physicians to be more mindful, informative, and empathic transforms their role from one characterized by authority to one that has the goals of partnership, solidarity, empathy, and collaboration.”
That’s fine, but the question that is constantly avoided is what happens when a patient with metastatic breast cancer expresses a strong preference for Vitamin C or Gerson therapy, as advocated by the YesToLife charity. The fact of the matter is that the relationship can’t be equal when one party, usually (but not invariably) the doctor, knows a lot more about the problem than the other.
What really matters above all to patients is getting better. Anyone in their right mind would prefer a grumpy condescending doctor who correctly diagnoses their tumour, to an empathetic doctor who misses it. It’s fine for medical students to learn social skills but there is a real danger of so much time being spent on it that they can no longer make a correct diagnosis. Put another way, there is confusion between caring and curing. It is curing that matters most to patients. It is this confusion that forms the basis of the bait and switch tactics (see also here) used by magic medicine advocates to gain the respectability that they crave but rarely deserve.
If, as is only too often the case, the patient can’t be cured, then certainly they should be cared for. That’s a moral obligation when medicine fails in its primary aim. There is a lot of talk about individualised care. It is a buzzword of quacks and also of the libertarian wing which says NICE is too prescriptive. It sounds great, but it helps only if the individualised treatment actually works.
Nobody knows how often medicine fails to be “patient-centred.”. Even less does anyone know whether patient-centred care can improve the actual health of patients. There is a strong tendency to do small pilot trials that are as likely to mislead as inform. One properly randomised trial (Kinmonth et al., 1998) concluded
“those committed to achieving the benefits of patient centred consulting should not lose the focus on disease management.”
Non-randomised studies may produce more optimistic conclusions (e.g. Hojat et al, 2011), but there is no way to tell if this is simply because doctors find it easy to be empathetic with patients who have better outcomes.
Obviously I’m in favour of doctors being nice to patients and to listening to their wishes. But there is a real danger that it will be seen as more important than curing. There is also a real danger that it will open the doors to all sorts of quacks who claim to provide individualised empathic treatment, but end up recommending Gerson therapy for metastatic breast cancer. The new College of Medicine, which in reality is simply a reincarnation of the late unlamented Prince’s Foundation for Integrated Health, lists as its founder Capita, the private healthcare provider that will, no doubt, be happy to back the herbalists and homeopaths in the College of Medicine, and, no doubt, to make a profit from selling their wares to the NHS.
In my own experience as a patient, there is not nearly as much of a problem with patient centred care as the industry makes out. Others have been less lucky, as shown by the mid-Staffordshire disaster (Delamothe, 2010), That seems to have resulted from PR being given priority over patients. Perhaps all that’s needed is to save money on all the endless reports and meetings (“the best substitute for work”), ban use of PR agencies (paid lying) and to spend the money on more doctors and nurses so they can give time to people who need it. This is a job that will be hindered considerably by the government’s proposals to sell off NHS work to private providers who will be happy to make money from junk medicine.
Reference
Wilkinson. R & Pickett, K. 2009 , The Spirit Level, ISBN 978 1 84614 039 6
A footnote on Robertson Cooper and "resilience"
I took up the offer of Robertson Cooper to do their free "resilience" assessment, the company to which HEFCE has paid an undisclosed amount of money.
The first problem arose when it asked about your job. There was no option for scientist, mathematician, university or research, so I was forced to choose "education and training". (a funny juxtaposition since training is arguably the antithesis of education). It had 195 questions. mostly as unanswerable as in the YouGov happiness survey. I particularly liked question 124 "I see little point in many of the theoretical models I come across". The theoretical models that I come across most are Markov models for the intramolecular changes in a receptor molecule when it binds a ligand (try, for example, Joint distributions of apparent open and shut times of single-ion channels and maximum likelihood fitting of mechanisms). I doubt the person who wrote the question has ever heard of a model of that sort. The answer to that question (and most of the others) would not be worth the paper they are written on.
The whole exercise struck me as the worst sort of vacuous HR psychobabble. It is worrying that HEFCE thinks it is worth spending money on it.