Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
One of my scientific heroes is Bernard Katz. The closing words of his inaugural lecture, as professor of biophysics at UCL, hang on the wall of my office as a salutory reminder to refrain from talking about ‘how the brain works’. After speaking about his discoveries about synaptic transmission, he ended thus.
|
"My time is up and very glad I am, because I have been leading myself right up to a domain on which I should not dare to trespass, not even in an Inaugural Lecture. This domain contains the awkward problems of mind and matter about which so much has been talked and so little can be said, and having told you of my pedestrian disposition, I hope you will give me leave to stop at this point and not to hazard any further guesses." |
 Drawing ©Jenny Hersson-Ringskog |
The question of what to eat for good health is truly a topic about "which so much has been talked and so little can be said"
That was emphasized yet again by an editorial in the British Medical Journal written by my favourite epidemiologist. John Ioannidis. He has been at the forefront of debunking hype. Its title is “Implausible results in human nutrition research” (BMJ, 2013;347:f6698.
Get pdf).
The gist is given by the memorable statement
"Almost every single nutrient imaginable has peer reviewed publications associating it with almost any outcome."
and the subtitle
“Definitive solutions won’t come from another million observational papers or small randomized trials“.
Being a bit obsessive about causality, this paper is music to my ears. The problem of causality was understood perfectly by Samuel Johnson, in 1756, and he was a lexicographer, not a scientist. Yet it’s widely ignored by epidemiologists.
The problem of causality is often mentioned in the introduction to papers that describe survey data, yet by the end of the paper, it’s usually forgotten, and public health advice is issued.
Ioannidis’ editorial vindicates my own views, as an amateur epidemiologist, on the results of the endless surveys of diet and health.
- Diet and health. What can you believe: or does bacon kill you (2009) in which I look at the World Cancer Research Fund’s evidence for causality (next to none in my opinion). Through this I got to know Gary Taubes, whose explanation of causality in the New York Times is the best popular account I’ve ever seen.
- How big is the risk from eating red meat now: an update (2012) This was based on the WCRF update – the risk was roughly halved though it didn’t say that in the press release.
- Another update. Red meat doesn’t kill you, but the spin is fascinating (2013). Update after the EPIC results in which the risk essentially vanished: good news which you could find only by digging into Table 3.
There is nothing new about the problem. It’s been written about many times. Young & Karr (Significance, 8, 116 – 120, 2011: get pdf) said "Any claim coming from an observational study is most likely to be wrong". Out of 52 claims made in 12 observational studies, not a single one was confirmed when tested by randomised controlled trials.
Another article cited by Ioannidis, "Myths, Presumptions, and Facts about Obesity" (Casazza et al , NEJM, 2013), debunks many myths, but the list of conflicts of interests declared by the authors is truly horrendous (and at least one of their conclusions has been challenged, albeit by people with funding from Kellogg’s). The frequent conflicts of interest in nutrition research make a bad situation even worse.
The quotation in bold type continues thus.
"On 25 October 2013, PubMed listed 291 papers with the keywords “coffee OR caffeine” and 741 with “soy,” many of which referred to associations. In this literature of epidemic proportions, how many results are correct? Many findings are entirely implausible. Relative risks that suggest we can halve the burden of cancer with just a couple of servings a day of a single nutrient still circulate widely in peer reviewed journals.
However, on the basis of dozens of randomized trials, single nutrients are unlikely to have relative risks less than 0.90 for major clinical outcomes when extreme tertiles of population intake are compared—most are greater than 0.95. For overall mortality, relative risks are typically greater than 0.995, if not entirely null. The respective absolute risk differences would be trivial. Observational studies and even randomized trials of single nutrients seem hopeless, with rare exceptions. Even minimal confounding or other biases create noise that exceeds any genuine effect. Big datasets just confer spurious precision status to noise."
And, later,
"According to the latest burden of disease study, 26% of deaths and 14% of disability adjusted life years in the United States are attributed to dietary risk factors, even without counting the impact of obesity. No other risk factor comes anywhere close to diet in these calculations (not even tobacco and physical inactivity). I suspect this is yet another implausible result. It builds on risk estimates from the same data of largely implausible nutritional studies discussed above. Moreover, socioeconomic factors are not considered at all, although they may be at the root of health problems. Poor diet may partly be a correlate or one of several paths through which social factors operate on health."
Another field that is notorious for producing false positives, wirh false attribution of causality, is the detection of biomarkers. A critical discussion can be found in the paper by Broadhurst & Kell (2006), "False discoveries in metabolomics and related experiments".
"Since the early days of transcriptome analysis (Golub et al., 1999), many workers have looked to detect different gene expression in cancerous versus normal tissues. Partly because of the expense of transcriptomics (and the inherent noise in such data (Schena, 2000; Tu et al., 2002; Cui and Churchill, 2003; Liang and Kelemen, 2006)), the numbers of samples and their replicates is often small while the number of candidate genes is typically in the thousands. Given the above, there is clearly a great danger that most of these will not in practice withstand scrutiny on deeper analysis (despite the ease with which one can create beautiful heat maps and any number of ‘just-so’ stories to explain the biological relevance of anything that is found in preliminary studies!). This turns out to be the case, and we review a recent analysis (Ein-Dor et al., 2006) of a variety of such studies."
The fields of metabolomics, proteomics and transcriptomics are plagued by statistical problems (as well as being saddled with ghastly pretentious names).
What’s to be done?
Barker Bausell, in his demolition of research on acupuncture, said:
[Page39] “But why should nonscientists care one iota about something as esoteric as causal inference? I believe that the answer to this question is because the making of causal inferences is part of our job description as Homo Sapiens.”
The problem, of course, is that humans are very good at attributing causality when it does not exist. That has led to confusion between correlation and cause on an industrial scale, not least in attempts to work out the effects of diet on health.
More than in any other field it is hard to do the RCTs that could, in principle, sort out the problem. It’s hard to allocate people at random to different diets, and even harder to make people stick to those diets for the many years that are needed.
We can probably say by now that no individual food carries a large risk, or affords very much protection. The fact that we are looking for quite small effects means that even when RCTs are possible huge samples will be needed to get clear answers. Most RCTs are too short, and too small (under-powered) and that leads to overestimation of the size of effects.
That’s a problem that plagues experimental pyschology too, and has led to a much-discussed crisis in reproducibility.
"Supplements" of one sort and another are ubiquitous in sports. Nobody knows whether they work, and the margin between winning and losing is so tiny that it’s very doubtful whether we ever will know. We can expect irresponsible claims to continue unabated.
The best thing that can be done in the short term is to stop doing large observational studies altogether. It’s now clear that inferences made from them are likely to be wrong. And, sad to say, we need to view with great skepticism anything that is funded by the food industry. And make a start on large RCTs whenever that is possible. Perhaps the hardest goal of all is to end the "publish or perish" culture which does so much to prevent the sort of long term experiments which would give the information we want.
Ioannidis’ article ends with the statement
"I am co-investigator in a randomized trial of a low carbohydrate versus low fat diet that is funded by the US National Institutes of Health and the non-profit Nutrition Science Initiative."
It seems he is putting his money where his mouth is.
Until we have the results, we shall continue to be bombarded with conflicting claims made by people who are doing their best with flawed methods, as well as by those trying to sell fad diets. Don’t believe them. The famous "5-a-day" advice that we are constantly bombarded with does no harm, but it has no sound basis.
As far as I can guess, the only sound advice about healthy eating for most people is
- don’t eat too much
- don’t eat all the same thing
You can’t make much money out of that advice.
No doubt that is why you don’t hear it very often.
Follow-up
Two relevant papers that show the unreliability of observational studies,
"Nearly 80,000 observational studies were published in the decade 1990–2000 (Naik 2012). In the following decade, the number of studies grew to more than 260,000". Madigan et al. (2014)
“. . . the majority of observational studies would declare statistical significance when no effect is present” Schuemie et al., (2012)
20 March 2014
On 20 March 2014, I gave a talk on this topic at the Cambridge Science Festival (more here). After the event my host, Yvonne Noblis, sent me some (doubtless cherry-picked) feedback she’d had about the talk.
21 Responses to We know little about the effect of diet on health. That’s why so much is written about it
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Many nutritional claims are simply not amenable to RCTs. However, smaller RCTs of 2-6 months clearly debunk many dietary claims of the last 40 years. For example, low carbohydrate, high fat diets are clearly superior to low-fat and vegan diets in weight loss, visceral fat loss, and diabetes risk markers (over a 6 month timeframe for what it’s worth). We need to start designing smarter RCTs in which we have greater control over the food intake, rather than giving patients a pamphlet and sending them home. Certainly I admit we can find an observational study to prove anything, but there is much room for more well-design RCTs in nutrition science.
The truly appalling thing is that, despite vast amounts of work, neither diets nor exercise seem to work well for losing weight. Even Robert Lustig, the most vociferous proponent of “sugar is poison” point of view, has said in a recent paper that
In other words, it seems that total calories that matter, not sugar per se.
You are clearly right that we need better, and longer, RCTs. Taken literally that would mean hardly any more publications about diet and health for the next ten years or so. That would deprive the Daily Mail of a lot of headlines, but it would be the honest thing to do. Any bets on it happening?
If Lustig et al. were controlling for total consumption, that would mask the effects of fructose on eating behavior. From what I understand, fructose directly enters the liver for metabolism and does not stimulate leptin release, thus imparing the satiety mechanism. If such an experiment were ethical, I would imagine that with patients given either a diet high in fructose vs a diet high in starches with the same caloric content, we would see greater weight gain in the former group as well as a host of other deleterious health effects. Penelope Greene at Harvard did an interesting set of experiments which have yet to be published in a peer-reviewed journal showing as best she could that calories in do not equal calories out:http://www.news.harvard.edu/gazette/2003/10.23/03-lowcarb.html
I’ve been pressing Lustig for the evidence for his view that sugar, and fructose in particular is a cause of diabetes (as opposed to obesity). I haven’t seen anything that’s totally convincing yet. I’m puzzled about why high fructose corn syrup (HFCS), in particular, has been blamed because I understand that it’s about 55% fructose, little higher than sucrose (50%) and less than most honey.
Perhaps its just that HFCS makes it cheap and easy to add sugar to almost everything you buy?
The Greene study to which you refer sounds interesting, but it is both brief (12 weeks) and wildly under-powered (7 people in each of three groups). That means that it’s likely to overestimate the effect size. Judgement of that work should, I think, be deferred until such time as it’s published in full.
Although Robert is an obesity doctor, I think his primary concern is T2 diabetes. He seems to argue that the 2 effects of sugar intake are independent (ie, obesity and T2 diabetes). The biochemical effects of fructose argue circumstantially that they should also be a driver of obesity given the known insulin resistance and lack of effect upon leptin. We do need a proper RCT, but it will be difficult to get past the ethical review boards. Of course, this didn’t stop us from accepting the view that cigarettes cause lung cancer despite there being no true RCTs for obvious reasons.
HFCS is essentially no different than sugar as you say, but it is so ubiquitous, appearing in most processed foods and even loafs of bread, that the most vulnerable of the public are frequently exposed to it. Nixon’s corn subsidies of the early 70s saw to its widespread use and relatively low cost (as if I needed another reason to dislike Nixon).
@chesketh
I wonder if it would be able to arrange a good RCT. You’d have to agree to be randomised to you present diet (high in sugar for almost everyone) or to a low sugar (or low fructose) diet. I’d take that risk. I find very sweet pizza crust quite unpleasant.
On twitter, Christopher Snowdon alerted me to a piece on The Conversation (Australian version), https://theconversation.com/informed-consent-why-some-foods-should-carry-a-cancer-risk-warning-18618
It uses the WCRF numbers to advocate health warnings on meat etc. As explained in the posts linked at the top, I think the WCRF numbers are too uncertain a basis for policy. The risks of eating red meat were essentially zero in the European EPIC trial (not mentioned in The Conversation). The analogy with cigarettes is flimsy. Cigarettes gave a relative risk of 20. Even the first WCRF report gave read meat a relative risk of only 1.2 (and it’s fallen since).
I’m not against warning labels and taxes in principle, but they have to be based on good evidence, and that we have not got.
“The truly appalling thing is that, despite vast amounts of work, neither diets nor exercise seem to work well for losing weight.”
I’ve always wondered if that argument doesn’t slightly miss the point. I do exercise (when I do exercise) to improve my general health. What matters to me is whether it reduces mortality/morbidity, not my gut size. Is there any reason to suppose that BMI or any other measure of weight is a good proxy for the things that actually matter to people, like living longer and with better life-quality?
*Great post David. I’ve been increasingly bothered about all the implausible and unreliable claims about foods in peer-reviewed journals, never mind those from the quacks.
I can’t really see how it would ever be possible to do really rigorous trials on a particular diet or nutrient (specially micro-nutrients) because, however large your sample size, the variables will be enormous and almost impossible to allow for.
As you quote above, there does seem to be an almost inborn wish for assigning cause and effect in all walks of life, specially in relation to food, perhaps because people feel they have some sort of control over that. So maybe peer-reviewed articles should be heralding the fact that these relationships simply don’t exist in the way that is frequently suggested. Then the cynical money-making of diet-mongers would be stopped in its tracks.
I don’t doubt the sincerity of the researchers but somehow I feel that if much of this research was focused on the costs of eating a sensible varied diet plus looking at the links between poverty and obesity/poor health then maybe some real value would come out of research into food and nutrition. At present I do wonder whether the only people who react to the claims are those who Iona Heath called the ‘worried well’.
The family and I were enjoying bacon sandwiches at the weekend, so good to be reminded again that the finger-wagging warnings about bacon (faithfully conveyed to the kids by their junior school – *sigh*) are basically nonsense.
And always good to see BK’s always-understated wisdom quoted.
BTW, since you mentioned ‘Omes’, if en passant, here is something else taking the mickey out of them, though now a bit out of date.
Speaking of small effect sizes, this abstract just came out from ASHG:
http://www.ashg.org/2013meeting/abstracts/fulltext/f130123041.htm
I am privileged enough to have 2M of my own SNPs and I don’t have this “meat-death” gene :). Not that it would have stopped me from deriving 70% of my calories from animal products, but it was nice to know.
Fascinating post David. Reminds me of a big (12,000 patients) cardiovascular risk factor study I ran 30 years ago. The sponsor company saw it as case-finding for its cholesterol-lowering drug. But to make it ethical patients with high cholesterol had to be put on closely monitored diets first. Big problem for the company – 10 times as many patients responded to diet as they expected, so not much drug was sold to them. Of course, the endpoint was cholesterol level not mortality or morbidity.
@chesketh
The premise of that gene study is that read meat causes colorectal cancer, the very thing that seems not to be true, at least in Europeans. And of course, as I mentioned, gene association studies are themselves notoriously susceptible to false positive results.
It’s certainly an interesting possibility that the near-zero correlation in the population results from most of the population being able to eat red meat perfectly safely, while a small number of people with unlucky genetics have a high risk. That certainly hasn’t been established, but it remains a possibility.
@DavidC True that study suffers from the same failings of any study trying to link two observations. It tries to link genes, the environment and disease which is fraught with even more potential confounders. It is a sexy topic right now though; I’m sure even the Daily Mail had a piece about it.
@csrster
I recall recent results that suggested that you live a bit longer if you were overweight (but not if obese). I’m not aware of any data that suggest that runners live longer (or otherwise) but I’ll check. The running boom started only in the 1980s so people who were young at the time are mostly still alive and it may be too early to get a clear answer.
I trained for marathons in the 1980s with a woman, Annie Briggs who was super fit (she was on the elite start for the London marathon) but I discovered recently that she succumbed to breast cancer in her 50s. Tragic.
As always, causality is the problem. Most runners are thin, but are they thin because they run, or do they run because they are thin? In this area, nothing is obvious.
I’ll try to dig out some information.
@DavidC. Here is the famous study linking BMI to mortality: http://jama.jamanetwork.com/article.aspx?articleid=200731
Seems that being “frail” at an old age can increase mortality, but at a younger age, obesity, but not “overweight” appears to be the main concern. You don’t hear about it much in the mainstream press because it goes against the mantra we always hear, ie, being “overweight” but not obese actually reduces the number of excess deaths compared to normal-weight individuals.
@chesketh
There is a good discussion of the “overweight live longer” suggestion on NHS Choices
Do overweight people live longer than normal weight individuals?
See here:
Are Metabolically Healthy Overweight and Obesity Benign Conditions?: A Systematic Review and Meta-analysis
Caroline K. Kramer, MD, PhD; Bernard Zinman, CM, MD; and Ravi Retnakaran, MD
and the accompanying editorial:
The Myth of Healthy Obesity
James O. Hill, PhD; and Holly R. Wyatt, MD
*I forgot to give the full reference, here it is:
Are Metabolically Healthy Overweight and Obesity Benign Conditions?: A Systematic Review and Meta-analysis
Caroline K. Kramer, MD, PhD; Bernard Zinman, CM, MD; and Ravi Retnakaran, MD
Ann Intern Med. 2013;159(11):758-769. doi:10.7326/0003-4819-159-11-201312030-00008
Thanks. UCL does not subscribe to that journal. Can you send a pdf?
Everyone agrees that obese people have increased mortality. The disagreement is about overweight. Oddly their fate was not mentioned in the abstract, so, given the usual alarmist spin, I’d guess that they weren’t at increased risk. But I’ll postpone judgement until I get hold of the full paper.
[…] We know little about the effect on health. That’s why there is so much written about it. […]