Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
The press releases (STOP PRESS)
Uhuh, here we go again.
All over the media we see headlines like “Honey ‘beats cough medicine’ “.
Take for example, the Daily Telegraph, where Ben Farmer writes “Honey is better at treating children’s coughs than an ingredient used in many over-the-counter medicines, according to new research”.
That is NOT what the research found This is what the research paper itself says (DM refers to the standard ‘cough suppressant’ dextromethorphan, which is already known to be ineffective).
| “honey was significantly superior to no treatment for cough frequency’ DM was not better than no treatment for any outcome. Comparison of honey with DM revealed no significant differences.” |
See it? No detectable difference between honey and standard cough medicine.
Everyone in the media misinterpreted what the paper said, but at least one blogger is already on to it, with Today’s “duh” study is a honey”.
At first sight, the results seem contradictory, No difference between honey and DM, No difference between DM and ‘no treatment’. So how can honey be better than ‘no treatment’?
The study was by Ian M. Paul, MD, MSc; Jessica Beiler, MPH; Amyee McMonagle, RN; Michele L. Shaffer, PhD; Laura Duda, MD; Cheston M. Berlin Jr, MD, published in Archives of Pediatrics & Adolescent Medicine 2007, 161, 1140 – 1146.
What was done
The design of this trial was pretty good apart from one thing Three things were compared (a) buckwheat honey, (b) a standard ‘cough suppressant’, dextromethorphan in a honey-flavoured syrup that was designed to be similar to the honey (DM for short), and (c) no treatment whatsoever.
The median age of the children who completed the study was 5.2 years (range, 2.2 – 16.9 years). They all had coughs attributed to upper respiratory tract infection. Thirty-five patients received honey, 33 received DM, and 37 received no treatment.The good thing is that the treatments were allocated randomly to the children, and that the person doing the assessment didn’t know which treatment each child had received. The children didn’t know whether they were getting honey or DM either, but they DID know when they got ‘no treatment’. The trial was carried out over two days. On day one nobody got a treatment, but they filled in a survey that asked, for example, “How frequent was your child’s coughing last night”. The parent had to tick one of seven boxes, from ‘not at all’ (score zero) to ‘extremely’ (score 6). They were then given the treatment allocated to them in a brown paper bag, so the person who gave it didn’t know which it was. The patients then went home and on the next day the same survey was completed by the same parent, over the telephone.
What happened?
First look at the raw data. Here is Figure 2 from the original paper.
The charts show the results for 5 different measures of the severity of cough, and the last chart (F) shows the aggregate score for all the criteria.
The first thing to notice is that there are no error bars on these charts. In my area, at least, no journal would accept a chart like this with no indication of scatter. There is a snag, though. Each patient acts as his/her own control, and that would not be reflected properly if errors bars were calculated for the numbers plotted in Fig. 2. It would therefore have been better to have a chart in which the difference in score between day 1 and day 2 was calculated from each patient, and the size of these differences plotted, with a standard deviation of the mean to indicate the amount of scatter in the observations. I have asked Dr Paul to send me a version that indicates the scatter of the numbers in this way (but I don’t think it will come).
The second thing to notice is that there is there is quite a big difference between the score on the first day (pale columns) and on the second day (dark columns), even in the no treatment group .
Thirdly, the pale columns are all much the same. On the first day the average score was about 4 (“a lot”) though on the second day, even with no treatment, the score fell quite a lot, to something between 2 (“a little”) and 3 (“somewhat”). This is a bit baffling because no treatment was given on either day. Presumably it results from the different settings in which the survey was given, or because the kids were getting better anyway.
Fourthly, insofar as the pale columns (baseline values) are all much the same, the thing you need to concentrate on is the difference, on each chart, between the height of the dark bars, for honey, DM and no treatment. These differences are pretty small, but on all the charts, the honey score is slightly smaller than the DM score, and the DM score is slightly smaller than the ‘no treatment’ score. What are we to make of that?
Here beginneth the statistical lesson.
Because the differences are small, and the scatter is quite big, we have to ask whether the differences are just random fluctuations rather than a result of any real difference between the treatments. That means we need statistics. Here is how the statistical argument works. Put roughly, we ask “how probable is it that the observations could arise by chance”. More precisely, the question is this. If there were no difference between the treatments, what is the probability that we would observe by chance a difference as big as, or bigger than, that seen in the experiment? (You need the subjunctive mood to explain statistics -pity it’s vanishing.)
Above each chart in the Figure we see P < 0.001. This means that there is less than a one in 1000 chance of the results arising by chance. More precisely, if all three treatments (honey, DM and no treatment) were actually identical, it is very unlikely that we’d see these results. The reasonable conclusion is, therefore, that all three treatments are not identical. The problem with this argument is that it tells you nothing about where the differences lie, so it is of no help whatsoever to a patient who is trying to decide what to do about a cough. The other problem is that it includes the ‘no treatment’ group, which was not blind. Both the children and parents were well aware that no treatment was given.
The most helpful comparison is really the properly-blinded comparison between honey and DM. And when this was looked at the result was no significant differences. In other words the small differences between the heights of the dark columns for honey and DM could perfectly well have arisen by chance if honey and DM were identical in their properties.
There isn’t any reason at all to think that honey is better than the standard (but ineffective) cough medicine.
The direct comparison between DM and ‘no treatment’ also shows no significant difference. Yet there are signs of a real difference between ‘no treatment’ and honey, though only for the cough frequency, not the other four measures. The aggregate measure (F in the figure) gave P = 0.04 for the comparison, so the authors are running a risk of 1 in 25 of being wrong in claiming a real effect. Although some people seem to regard a value of P = 0.05 as indicating a real effect, the fact that you’ll make a fool of yourself 1 time in 20 by claiming a real effect when none exists has never seemed to me to be good enough odds to stake one’s reputation on.
The ‘no treatment’ group certainly has some interest, but the fact that it was not blind means that the fact that honey was marginally better than ‘no treatment’ could perfectly well mean that taking honey has a better placebo effect that doing nothing at all. It provides no evidence at all that honey has any genuine therapeutic effect. If it had, one would then have to find out if the therapeutic effect was specific to buckwheat honey, or whether any old honey would do. It could be argued that even if the effect were real rather than placebo, the size of the effect is too small to make all that effort worthwhile.
A couple more things
It is already well known, from several good studies, that DM is useless, no better than placebo. This inconvenient fact has not yet reached many places that it should have (not even mentioned on wikipedia for example), but the American Academy of Pediatrics says
“Numerous prescription and nonprescription medications are currently available for suppression of cough, a common symptom in children. Because adverse effects and overdosage associated with the administration of cough and cold preparations in children have been reported, education of patients and parents about the lack of proven antitussive effects and the potential risks of these products is needed.”
The discussion in the paper by Paul et al, seems surprisingly upbeat about honey, in the light of their own findings. I’m surprised that they use the term ‘demulcent’ which I had thought to have died out, like the word ‘tonic’, on the grounds that it had no defined meaning
It is because meaningless terms and useless medicines die out eventually that medicine makes progress. The problem with alternative medicine is that nothing dies out: on the contrary they keep adding myths.
And finally
Always look at the end of the paper. On this one we see that the study was paid for by the National Honey Board. Dr Paul assures me that the funding source had no say in the design or analysis, which is as it should be.
Financial Disclosure: Dr Paul has been a consultant to the Consumer Healthcare Products Association and McNeil Consumer Healthcare.
Funding/Support: This work was supported by an unrestricted research grant from the National Honey Board, an industry-funded agency of the US Department of Agriculture.
So what is the practical outcome?
My conclusion from all this is simple. If you have got a cough, tough luck. There isn’t really anything available, conventional or alternative, that does much good. You’ll just have to wait for it to get better. But if you want to take something that tastes nice, why not honey? It almost certainly won’t do any good but it tastes good and it’s safer than the standard cough medicine.
The sponsor’s interpretation
It seems that the sponsor of the work is happy with the misinterpretation.
Charlotte Jordan a project manager of research at the National Honey Board, believes the finding confirms what your grandmother told you.
“This is a really exciting finding,” she said. “For a long time it’s been folklore medicine to use honey when you have a cough or a cold, but it’s exciting to have a scientific study to back that up.”
Just one problem, That is NOT what the paper says.
How did all this mis-reporting happen?
One reason is misleading press releases. Universities and Academic journals now engage in shameless PR, spin and hype. They prostitute good science.
Download press releases from Penn State, JAMA and Press Association [pdf file]
Here is the highly misleading bit of hype that came from the Press Office of the Pennsylvania State University. The headline is “Honey a better option for childhood cough than OTCs” (OTC means over-the-counter medicines that contain DM). That contradicts directly the paper which says “Comparison of honey with DM revealed no significant differences”.
Likewise the statement in the Penn State release “Honey did a better job reducing the severity, frequency and bothersome nature of nighttime cough from upper respiratory infection than DM or no treatment” is equally incompatible with “Comparison of honey with DM revealed no significant differences”. Its only possible justification is from the 3 way comparison by analysis of variance and that does not tell us what we need to know.
To make matters worse, the media office is not to blame this time. Ms Manlove told me tonight that the press release had been approved by Dr Paul himself.
| Contact: Megan W. Manlove Penn State Honey a better option for childhood cough than OTCs A new study by a Penn State College of Medicine research team found that honey may offer parents an effective and safe alternative than over the counter children’s cough medicines. The study found that a small dose of buckwheat honey given before bedtime provided better relief of nighttime cough and sleep difficulty in children than no treatment or dextromethorphan (DM), a cough suppressant found in many over-the-counter cold medications. Honey did a better job reducing the severity, frequency and bothersome nature of nighttime cough from upper respiratory infection than DM or no treatment. Honey also showed a positive effect on the sleep quality of both the coughing child and the child’s parents. DM was not significantly better at alleviating symptoms than no treatment. . . . |
All that Candice Yakel, of the Office for Research Protections at Penn State had ro say in the matter was
“Our investigators stand by the conclusions of the study as reported in the Archives of Pediatric and Adolescent Medicine and as characterized in our press release of December 3, 2007.”
And here is the equally misleading bit of hype issued by the Journal of the American Medical Association (Ms Manlove tells me that this was also approved bt Dr Paul).
| JAMA and Archives Journals Study suggests honey may help relieve children’s cough, improve sleep during colds
|
The Press Association release was equally bad, and probably the one used by many of the reporters as a basis for stories in the media. The opening statement is totally wrong.
| 1 HEALTH Honey Embargoed to 2100 Monday December 3 HONEY BEST FOR KIDDIES’ COUGHS SAY RESEARCHERS By John von Radowitz, PA Science Correspondent Natural honey is a better remedy for children’s coughs than expensive over-the-counter medicines, researchers said today. A dose of buckwheat honey before bedtime easily outperformed a cough suppressant widely used in commercial treatments, a US study found. . . . |
Follow-up
There is a review of over-the-counter cough medicines in the BMJ (2002) [free full text]. It concludes “Recommendation of over the counter cough medicines to patients is not justified by current evidence”.
32 Responses to Why honey isn’t a wonder cough cure: more academic spin
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
The folk at MedPageToday obviously need to read this as they appear to have fallen into the honey trap: http://www.medpagetoday.com/InfectiousDisease/URItheFlu/tb/7560
considering how some of the ‘active’ ingredients in cough syrups can be misused surely there is now ample evidence for them to be banned?
Surprisingly, DrJB doesn’t agree: Honey found to soothe children’s coughs.
Just as a matter of interest, I thought codeine (and other opiates) was pretty effective for coughs? Not that you’d want to give them to kids necessarily.
The HTH team seem to have got it right:
http://www.library.nhs.uk/rss/newsAndRssArticle.aspx?uri=http%3a%2f%2fwww.library.nhs.uk%2fresources%2f%3fid%3d275788
– as they usually do.
Yes I’d noticed that. Quite impressive given the the poor quality of some of the NHS advice on alternative medicine.
Perhaps the HTH folk had had a look at your post?
I find HTH normally reliable but wish they had the resources to cover more health stories. Re NHS CAM information, time to trot out my gripe about there being no single NLH specialist library for evidence-based allergy information, meaning several need to be monitored. CAM, however, which promotes itself for allergy-related disorders without, IMO, much good evidence, has its own NLH specialist library.
Given that the dextromethorphan was apparently given in a “honey-flavoured syrup”, I am puzzled why the control group was given nothing rather than, say, a honey-flavoured syrup placebo. Seems rather an obvious modification which would have ruled out “treatment effects” much more effectively.
Hello David,
Sorry to be a grammar pedant, but the subjunctive is not a tense, but a mood.
Everything else seemed spot on though…
Absolutely right Robin. I do like pedants.
Fixed it now.
David, you say of honey: “It won’t do much good but its safer than the standard cough medicine.”
That’s not my understanding of the paper. Honey had more side effects associated with it:”The combination of mild reactions that include hyperactivity, nervousness, and insomnia occurred in 5 patients treated with honey, 2 patients in the DM group, and no patients in the no-treatment arm (P = .04).”
Perhaps we should conclude dextromethorphan is not inferior to honey, and is safer to boot?
Good point, When I said safer than DM I had in mind the danger of abuse in big doses, rather than danger in routine use. Perhaps I should make that clearer.
I find it pretty hard to believe that honey causes all those side effects, so in the absence of better evidence I’d leave it as a dead heat.
Depressed to hear that the hype and spin was coming from the people who did the work. Sadly there is too much of this sort of thing about, though most academics still regard it as rather unseemly. The trouble is that “exposure” is seen as a way to get ahead, by Univs, Depts and some people.
This reminds me of something, actually. The last time I saw DC’s friend and eminent colleague Prof Bert Sakmann give a conference lecture, the person introducing him stressed how Sakmann was very much the kind of scientist other scientists looked up to in the following terms: sakmann, he said, was a brilliant experimenter, a personally modest man, and:
“…a scientist of the kind whose reputation you judge from the number of their papers that appear in journals like Nature, rather than the number of times their name appears in the papers”
Hmm… I wonder if those supposed side effects are in any way related to the popular perception that sugar (and its relatives) cause ADD or at least aggravate it. At any rate, IIRC, honey is mostly fructose which will cause a sugar rush.
Personal observation, I find almost any kind of sweetener, including sucralose goes a long way toward soothing a sore throat even if it doesn’t do much for any other symptoms.
Dr Aust, Thanks for the link to that excellent editorial. It says
I’ve been wondering about this. Presumably when a university or research institution issues a press release, it would be standard practice to get the lead authors of the study in question to approve it?
Having just had a quick chat to the press office at work, it seems that the lead author of a study will generally sign off on any press release related to that study. Not sure how it goes at other institutions, though.
For statistical curiosities sake, could you tell us what stats tests were used? I might have missed it in the blog entry but I don’t think I saw any mention. Also, did they adjust for multiple comparisons, and would that be appropriate in such a study?
I must confess that my grasp of the fine details of stats isn’t great so it’s rather nice to be looking at real world examples.
Thanks,
C.
The 3 way comparison was done by analysis of variance. and a Tukey method for pairwise tests. I thought there was very little to criticise about the work itself, It was the spin put on the results inthe discussion and especially in the press releases that was so alarming
Of course Honey is a proven cure for the cough, you just have to use it the right way.
Here is how: Open your beehive and lift out a honeycomb complete with bees to admire. I guarantee that you will NOT cough whilst you’re doing this, even if it feels like you have to. 😉
[…] in the UK in the notorious case of the University of Sheffield). We have seen ghost-writing and spin tolerated, and even encouraged, not to mention sham consultations and attempts to impose the […]
Claire has pointed out this interesting finding.
“Adding honey to dressings covering venous leg ulcers did not speed up wound healing and was associated with more adverse events than standard care, researchers here found.”
Great rebuttle to the “scientific” paper. It appears that we are one of the few buckwheat honey suppliers in the UK and we were inundated by on-line and telephone orders when the national press and some websites covered the story. There’s no doubt that buckwheat honey is a great honey – dark and strong flavoured – but as a cough remedy! Honey is one of nature’s wonderful, miracle foods and can, for some people, no doubt help with coughs and colds. But the paper failed to compare the efficay of buckwheat honey with other honeys and this was a major shortcoming. The same problem of hype and clever marketing has seen public awareness and demand grow for Manuka Honey – which attracts a premium price. Manuka honey is a great honey but there is no evidence to suggest that it is better than other pure, natural honeys. We are poorly served by our press and, it seems, by some in the scientifc community.
[adverts removed DC]
Simon Burgess
Hmmm well actually there isn’t the slightest reason to describe any honey as a “miracle food”, or to think that it helps anyone with coughs and colds. It just tastes nice. That’s enough.
I did have a friend a while ago who was adamant that honey contained no sugar. She must have learned that from a ‘health food’ ship, I guess.
“But the paper failed to compare the efficay of buckwheat honey with other honeys and this was a major shortcoming.”
Ha! Canard #46: ‘they used the wrong kind of…[insert brand of woo here]’
I’ve seen this sort of claim many times*.
I’ve also noticed that whenever I write about health foods I always get salesmen leaving comments along the lines of ‘yes, the products you’ve looked at are a bit dodgy, but our particular brand…’
*The herbs weren’t individualised, it was the wrong homeopathic remedy, the vitamins were synthetic etc…
[…] channels, nobody would read it (big mistake -it’s fascinating), but if one can use the case of honey versus cough medicine to explain the analysis of variance, there is a chance that someone might find it […]
“The aggregate measure (F in the figure) gave P = 0.04 for the comparison, so the authors are running a risk of 1 in 25 of being wrong in claiming a real effect.”
Not really. It’s much worse than that.
The P-value is the probability of their data (or, more pedantically, the probability of observing data at least as extreme as they really did observe), conditional on there being no real effect. But we don’t know if this condition holds. The condition we do know to be the case, is that they observed a certain set of data (if we’re prepared to believe they observed what they say they observed and I don’t have a problem with that) . So the probability they are wrong in claiming a real effect is conditional on their data — that’s the way round for the conditioning, not the probability of the data conditional on there being no effect. And this probability that we do want is likely to be very different from the P-value. See (e.g.) Berger and Sellke (1987), Journal of the American Statistical Association, 82, 112-139 — they give formulas that I haven’t applied in detail to this case, and anyway there are other ways of doing it, but with this sample size and this P-value they would say that the probability there’s really no effect is somewhere a bit over 50%. That is, there’s a risk of about 1 in 2 that they are wrong in claiming a real effect, not 1 in 25.
Honey is not a natural food. It is classed as a “processed” food by governments (because it is processed by bees ! Evil little communist advocates of agri-business)
All things considered the response from alvanatural is not the usual self-seeking rubbish we expect from quacks (not that I am suggesting they are quacks – merely vendors of honey).
Not sure what evidence exists that honey (or indeed anything else) is a “miracle” food though. I thought it was just processed sugar (with maybe some trace of various flowers or herbs).
I would be happy to assist in a clinical evaluation of malt whisky as a palliative for colds and sore throats should anyone manage to secure appropriate funding. I am more than happy to suffer for the sake of science (as long as I don’t get the placebo!).
[…] The first job of a scientist is to say openly when the answer to a question is not known. But scientists are under constant pressure to exaggerate the importance of their results. Last year we published an article which I feel may, if verified, turn out to be the second most important that I have ever been an author on. Because it happened to be published in Nature (not because of its quality), a press release was written (by an arts graduate!). It took some argument to prevent the distorted and exaggerated account being imposed on the public. This is typical of the sort of thing reported in Goldacre’s column. I reported a similar case a while ago, Why honey isn’t a wonder cough cure: more academic spin. […]
I have found personally that using raw, local, unprocessed honey has worked better than Delsym (considered the best OTC cough medicine). Delsym, by the way, contains dextromethorphan. So I believe there is legitimacy to honey helping to cure a cough.
Honey is also a powerful antioxidant and contains a multitude of vitamins and minerals, including vitamin C, many of the B vitamins, potassium, calcium, iron and many more.
Seeing as honey has been used medicinally for over 2500 years, wouldn’t it be worth a shot to at least try it before you fault the research? Remember, the proof of the pudding is in the eating.:-)
Honey has definitely worked better for me than dextromethorphan, so it has my vote of confidence.
@spage
Oh dear you seem to have fallen for every scam and myth of the supplements industry. Antioxidants are a myth and there isn’t the slightest reason why any of the things you list should be helpful
In fact you didn’t even read the post. The paper found that honey was not better than dextromethorphan, which is itself known to be useless. My criticism was not of that finding, but at the distortion of it that appeared in newspapers, apparently with the connivance of the authors
[…] sort of misreporting occurs, it often seems to come from the press release (another case documented here). Many journalists never get any further than the press release, Here is Nature’s press […]
[…] explaining science. Worse still, sometimes the misleading hyps stems from the authors themselves (one example here, but there are hundreds to choose from). This makes it very important that science reporters […]