What to do about research assessment (the REF)?
A proposal for two-stage university education

Jump to follow-up

DOI: 10.15200/winn.142809.94999

The Research Excellence Framework (REF) is the latest in a series of 6-yearly attempts to assess the quality of research in UK universities. It’s used to decide how to allocate about £1.6 billion per year of taxpayers’ money, the so-called "quality-related" (QR) allocation.

It could have been done a lot worse. One of the best ideas was that only four papers could be submitted, whatever the size of a research group. After much argument, the judgment panels were told not to use journal impact factors as a proxy for quality (or, for lack of quality), though it’s clear that many people did not believe that this would be obeyed, But it cost at least £60 million. At UCL alone, it took 50 – 75 person-years of work. and the papers that were submitted were assessed by people who often would have no deep knowledge about the field, It was a shocking waste of time and money, and its judgements in the end were much the same as last time.

Did the REF benefit science?

It’s frequently said that the REF improved the UK’s science output. The people who claim this need a course in the critical assessment of evidence. Firstly, there is no reason to think that science has improved in quality in the last 6 years, and secondly any changes that might have occurred are hopelessly confounded with the passage of time, the richest source of false correlations.

I’d argue that the REF has harmed science by encouraging the perverse incentives that have done so much to corrupt academia. The REF, and all the other university rankings produced by journalists, are taken far too seriously by vice-chancellors and that does active harm. As one academic put it

"This isn’t about science – it’s about bragging rights, or institutional willy-waving. "

There are now serious worries about lack of reproducibility of published work, waste of money spent on unreliable studies, publication of too many small under-powered studies, bad statistical practice (like ignoring the false discovery rate), and about exaggerated claims by journals, university PR people and authors themselves. These result in no small part from the culture of metrics and the mismeasurement of science. The REF has added to the pressures.

It is highly unsatisfactory, so the only real question becomes what should be done instead? 

What’s to be done?

Transferring all the QR money to Research Councils won’t work. It would merely encourage the grossly bad behaviour that we’ve seen at Imperial College London, Warwick University, Kings College London and Queen Mary College London, all of whom have fired successful senior staff simple because their grant income wasn’t deemed big enough. (This is odd because the same managers whine continually that they make a loss on research grants, but that’s another question.) It’s been suggested that this could be avoided by reducing considerably the overheads that come with grants, but this would leave a shortfall that, without QR, would be impossible to make up.

At present a HEFCE working group is considering the possibility that metrics might be used in the next REF. It’s a sensible group of people, and they are well aware of the corrupting influence of metrics, and the lack of evidence that they measure the quality of research. So if reading papers takes too much time and money, and metrics are likely to lead to widespread "gaming" (a euphemism for cheating), what should be done?

I made a suggestion in 2010, but it seems to have been totally ignored, despite appearing in the Times (in their premier
Thunderer opinion column. So I’ll try to make the case again, in the context of the REF.

A complete re-thinking of tertiary education is needed,

Proposal for a two stage higher education system

It seems to be a good thing that such a large proportion of the population now get higher education.  But the university system has failed to change to cope with the huge increase in the number of students. 

The system of highly specialist honours degrees might have been adequate when 5% of the population did degrees, but that system seems quite inappropriate when 50% are doing them. 

There are barely enough university teachers who are qualified to teach specialist 3rd year or postgraduate courses.  And many  teachers must have suffered from (in my field) trying to teach the subtleties of the exponential probability density function to a huge third year class, most of whom have already decided that they want to be bankers or estate agents.

These considerations have driven me to conclude, somewhat reluctantly, that the whole system needs to be altered. 

Honours degrees were intended as a prelude to research and 50% of the population are not going to do research (fortunately for the economy). Vice-chancellors have insisted on imposing on large numbers of undergraduates, highly specialist degrees which are not what they want or need.

I believe that all first degrees should be ordinary degrees, and these should be less specialist than now.  Some institutions would specialise in teaching such degrees, others would become predominantly postgraduate institutions, which would have the time. money and expertise to do proper advanced teaching, rather than the advanced Powerpoint courses that dominate what passes for Graduate Schools in the UK.

There would, of course, be almighty rows about which universities would be re-allocated to teach ordinary degrees. That’s not a reason to educate students in 2015 using a pre-war system.

The two-stage system would be more egalitarian than the present one

I anticipate that some people might think that this system is a reversion to the pre-1992 divide between polytechnics and universities. It isn’t. The pre-1992 system labelled you as either polytechnic or university: it was a two-tier system. I’m proposing a two stage system. The two sorts of institution work in series, not in parallel.

Such a system would be more egalitarian than now, not less. 

Everyone would start out with the same broad undergraduate education, and the decision about whether to specialise, and the area in which to specialise, would not have to be made before leaving (high) school, as now, but would be postponed until two or three years later. That’s a lot better, especially for people from poorer backgrounds.

If this were done, most research would be done in the postgraduate institutions.  Of course there are some good researchers in institutions that would become essentially teaching-only, so there would have to be chances for such people to move to postgraduate universities, and for some people to move in the other direction.

This procedure would, no doubt, result in a reduction in the huge number of papers that are published (but read by nobody).  That is another advantage of my proposal.   It’s commonly believed that there is a large amount of research that is either trivial or wrong.  In biomedical research, it’s been estimated that 85% of resources are wasted (Macleod et al., 2014).

It’s well-known that any paper, however bad, can be published in a peer-reviewed journal.  Pubmed, amazingly, indexes something like 30 jouranls devoted to quack medicine, in which papers by quacks are peer-reviewed by other quacks, and which are then solemnly counted by bean-counters as though they were real research.  The pressure to publish when you have nothing to say is one of the perverse incentives of the metrics culture.

It seems likely that standards of research in second-stage universities would be at least as high as at present. It that’s the case then QR could simply be allocated on the basis of the number of people in a department.  Dorothy Bishop has shown that even under the present system, the amount of QR money received is strongly correlated with the size of the department (correlation coefficient = 0.995 for psychology/neuroscience).

From Dorothy Bishop’s blog. r = 0.995

Using metrics produces only a tiny increase in the correlation coefficient for RAE data. It could hardly be any higher than 0.995 

In other words, after all the huge amount of time, effort and money that’s been put into assessment of research, every submitted researcher ends up getting much the same amount of money. 

That system wouldn’t work at the moment, because, sadly, universities would, no doubt, submit the departmental cat for a share of the cash.  But it could work under a system such as I’ve described.  The allocation of QR would take microseconds and cost nothing.

How much would the two-stage system cost?

To have any hope of being accepted by politicians, the two-stage system would probably have to cost no more than the existing system. As far as I know. nobody seems to have made any serious attempt to work out the costs. Perhaps they should. It won’t be easy because an important element of the two-stage system is to improve postgraduate education, and postgraduate education was forgotten in the government’s "reforms"

Much would depend on whether the first stage, ordinary degrees could be taught in two years. In an institution that does little research, there would be no justification for the long summer vacation. Something comparable with (high) school holidays would be more appropriate, and if a decent job could be done in two years, that could save enough money to pay for the rest. It would also mimimise the debt that hangs round the neck of graduates.

The cost of running the second stage would depend on how many students opted (and qualified) to carry on to do an honours degree, and on how many of those wanted go on to graduate school and higher degrees. The numbers of people that went on to specialist honours degrees would inevitably be smaller than now, so their education would be cheaper. But, crucially, they could be educated better. And because of the specialist researchers in a postgraduate institution, it would be possible to have real postgraduate education in advanced research methods,

At present, Graduate Schools in the UK (unlike those in the USA) rarely teach topics beyond advanced Powerpoint, and that’s a recipe for later mediocrity.

In order to estimate the actual cost, we’d need to know how many people wanted to go beyond the first degree (and qualified to do so). If this were not to large, the proposed system could well be cheaper than the presnet one, as well as being more egalitarian, and providing better postgraduate education. The Treasury should like that.

The California System

It will not have escaped the readers’ attention that the two stage system proposed here has much in common with higher education in the USA. In particular, it resembles the University of California system, which was started in 1960. It became a model for the rest of the world.

Meanwhile, the UK persists with a pre-war system of specialist honours degrees that is essentially unchanged since only a handful of people went to universities.

It’s time for the UK to have a serious debate about whether we need to change.

Follow-up

I just noticed this, from the inimitable Laurie Taylor. It is dated 4 July 2013. Who says the REF does not encourage cheating?

3 February 2015

The day after this post appeared the Guardian published a version of it which discussed only the two-stage degree proposals but omits the bit about the Research Excellence Framework (REF 2014). The title was "Honours degrees aren’t for all – some unis should only teach two-year courses". There are a lot more comments there than than here.. I assume that the headline was written by one of those pesky subeditors who failed to understand what’s important (the two year degrees were just a suggestion, nothing to do with the main proposals).

3 April 2015

As an experiment, this blog has been re-posted on the Winnower. The advantage of this is that it now has a digital object identifier, DOI: 10.15200/winn.142809.94999