Download Lectures on Biostatistics (1971). Corrected and searchable version of Google books edition
Download review of Lectures on Biostatistics (THES, 1973).
I have in the past, taken an occasional interest in the philosophy of science. But in a lifetime doing science, I have hardly ever heard a scientist mention the subject. It is, on the whole, a subject that is of interest only to philosophers.
|
It’s true that some philosophers have had interesting things to say about the nature of inductive inference, but during the 20th century the real advances in that area came from statisticians, not from philosophers. So I long since decided that it would be more profitable to spend my time trying to understand R.A Fisher, rather than read even Karl Popper. It is harder work to do that, but it seemed the way to go.
|
 |
This post is based on the last part of chapter titled “In Praise of Randomisation. The importance of causality in medicine and its subversion by philosophers of science“. A talk was given at the meeting at the British Academy in December 2007, and the book will be launched on November 28th 2011 (good job it wasn’t essential for my CV with delays like that). The book is published by OUP for the British Academy, under the title Evidence, Inference and Enquiry (edited by Philip Dawid, William Twining, and Mimi Vasilaki, 504 pages, £85.00). The bulk of my contribution has already appeared here, in May 2009, under the heading Diet and health. What can you believe: or does bacon kill you?. It is one of the posts that has given me the most satisfaction, if only because Ben Goldacre seemed to like it, and he has done more than anyone to explain the critical importance of randomisation for assessing treatments and for assessing social interventions.
Having long since decided that it was Fisher, rather than philosophers, who had the answers to my questions, why bother to write about philosophers at all? It was precipitated by joining the London Evidence Group. Through that group I became aware that there is a group of philosophers of science who could, if anyone took any notice of them, do real harm to research. It seems surprising that the value of randomisation should still be disputed at this stage, and of course it is not disputed by anybody in the business. It was thoroughly established after the start of small sample statistics at the beginning of the 20th century. Fisher’s work on randomisation and the likelihood principle put inference on a firm footing by the mid-1930s. His popular book, The Design of Experiments made the importance of randomisation clear to a wide audience, partly via his famous example of the lady tasting tea. The development of randomisation tests made it transparently clear (perhaps I should do a blog post on their beauty). By the 1950s. the message got through to medicine, in large part through Austin Bradford Hill.
Despite this, there is a body of philosophers who dispute it. And of course it is disputed by almost all practitioners of alternative medicine (because their treatments usually fail the tests). Here are some examples.
“Why there’s no cause to randomise” is the rather surprising title of a report by Worrall (2004; see also Worral, 2010), from the London School of Economics. The conclusion of this paper is
“don’t believe the bad press that ‘observational studies’ or ‘historically controlled trials’ get – so long as they are properly done (that is, serious thought has gone in to the possibility of alternative explanations of the outcome), then there is no reason to think of them as any less compelling than an RCT.”
In my view this conclusion is seriously, and dangerously, wrong –it ignores the enormous difficulty of getting evidence for causality in real life, and it ignores the fact that historically controlled trials have very often given misleading results in the past, as illustrated by the diet problem.. Worrall’s fellow philosopher, Nancy Cartwright (Are RCTs the Gold Standard?, 2007), has made arguments that in some ways resemble those of Worrall.
Many words are spent on defining causality but, at least in the clinical setting the meaning is perfectly simple. If the association between eating bacon and colorectal cancer is causal then if you stop eating bacon you’ll reduce the risk of cancer. If the relationship is not causal then if you stop eating bacon it won’t help at all. No amount of Worrall’s “serious thought” will substitute for the real evidence for causality that can come only from an RCT: Worrall seems to claim that sufficient brain power can fill in missing bits of information. It can’t. I’m reminded inexorably of the definition of “Clinical experience. Making the same mistakes with increasing confidence over an impressive number of years.” In Michael O’Donnell’s A Sceptic’s Medical Dictionary.
At the other philosophical extreme, there are still a few remnants of post-modernist rhetoric to be found in obscure corners of the literature. Two extreme examples are the papers by Holmes et al. and by Christine Barry. Apart from the fact that they weren’t spoofs, both of these papers bear a close resemblance to Alan Sokal’s famous spoof paper, Transgressing the boundaries: towards a transformative hermeneutics of quantum gravity (Sokal, 1996). The acceptance of this spoof by a journal, Social Text, and the subsequent book, Intellectual Impostures, by Sokal & Bricmont (Sokal & Bricmont, 1998), exposed the astonishing intellectual fraud if postmodernism (for those for whom it was not already obvious). A couple of quotations will serve to give a taste of the amazing material that can appear in peer-reviewed journals. Barry (2006) wrote
“I wish to problematise the call from within biomedicine for more evidence of alternative medicine’s effectiveness via the medium of the randomised clinical trial (RCT).”
“Ethnographic research in alternative medicine is coming to be used politically as a challenge to the hegemony of a scientific biomedical construction of evidence.”
“The science of biomedicine was perceived as old fashioned and rejected in favour of the quantum and chaos theories of modern physics.”
“In this paper, I have deconstructed the powerful notion of evidence within biomedicine, . . .”
The aim of this paper, in my view, is not obtain some subtle insight into the process of inference but to try to give some credibility to snake-oil salesmen who peddle quack cures. The latter at least make their unjustified claims in plain English.
The similar paper by Holmes, Murray, Perron & Rail (Holmes et al., 2006) is even more bizarre.
“Objective The philosophical work of Deleuze and Guattari proves to be useful in showing how health sciences are colonised (territorialised) by an all-encompassing scientific research paradigm “that of post-positivism ” but also and foremost in showing the process by which a dominant ideology comes to exclude alternative forms of knowledge, therefore acting as a fascist structure. “,
It uses the word fascism, or some derivative thereof, 26 times. And Holmes, Perron & Rail (Murray et al., 2007)) end a similar tirade with
“We shall continue to transgress the diktats of State Science.”
It may be asked why it is even worth spending time on these remnants of the utterly discredited postmodernist movement. One reason is that rather less extreme examples of similar thinking still exist in some philosophical circles.
Take, for example, the views expressed papers such as Miles, Polychronis and Grey (2006), Miles & Loughlin (2006), Miles, Loughlin & Polychronis (Miles et al., 2007) and Loughlin (2007).. These papers form part of the authors’ campaign against evidence-based medicine, which they seem to regard as some sort of ideological crusade, or government conspiracy. Bizarrely they seem to think that evidence-based medicine has something in common with the managerial culture that has been the bane of not only medicine but of almost every occupation (and which is noted particularly for its disregard for evidence). Although couched in the sort of pretentious language favoured by postmodernists, in fact it ends up defending the most simple-minded forms of quackery. Unlike Barry (2006), they don’t mention alternative medicine explicitly, but the agenda is clear from their attacks on Ben Goldacre. For example, Miles, Loughlin & Polychronis (Miles et al., 2007) say this.
“Loughlin identifies Goldacre [2006] as a particularly luminous example of a commentator who is able not only to combine audacity with outrage, but who in a very real way succeeds in manufacturing a sense of having been personally offended by the article in question. Such moralistic posturing acts as a defence mechanism to protect cherished assumptions from rational scrutiny and indeed to enable adherents to appropriate the ‘moral high ground’, as well as the language of ‘reason’ and ‘science’ as the exclusive property of their own favoured approaches. Loughlin brings out the Orwellian nature of this manoeuvre and identifies a significant implication.”
If Goldacre and others really are engaged in posturing then their primary offence, at least according to the Sartrean perspective adopted by Murray et al. is not primarily intellectual, but rather it is moral. Far from there being a moral requirement to ‘bend a knee’ at the EBM altar, to do so is to violate one’s primary duty as an autonomous being.”
This ferocious attack seems to have been triggered because Goldacre has explained in simple words what constitutes evidence and what doesn’t. He has explained in a simple way how to do a proper randomised controlled trial of homeopathy. And he he dismantled a fraudulent Qlink pendant, purported to shield you from electromagnetic radiation but which turned out to have no functional components (Goldacre, 2007). This is described as being “Orwellian”, a description that seems to me to be downright bizarre.
In fact, when faced with real-life examples of what happens when you ignore evidence, those who write theoretical papers that are critical about evidence-based medicine may behave perfectly sensibly. Although Andrew Miles edits a journal, (Journal of Evaluation in Clinical Practice), that has been critical of EBM for years. Yet when faced with a course in alternative medicine run by people who can only be described as quacks, he rapidly shut down the course (A full account has appeared on this blog).
It is hard to decide whether the language used in these papers is Marxist or neoconservative libertarian. Whatever it is, it clearly isn’t science. It may seem odd that postmodernists (who believe nothing) end up as allies of quacks (who’ll believe anything). The relationship has been explained with customary clarity by Alan Sokal, in his essay Pseudoscience and Postmodernism: Antagonists or Fellow-Travelers? (Sokal, 2006).
Conclusions
Of course RCTs are not the only way to get knowledge. Often they have not been done, and sometimes it is hard to imagine how they could be done (though not nearly as often as some people would like to say).
It is true that RCTs tell you only about an average effect in a large population. But the same is true of observational epidemiology. That limitation is nothing to do with randomisation, it is a result of the crude and inadequate way in which diseases are classified (as discussed above). It is also true that randomisation doesn’t guarantee lack of bias in an individual case, but only in the long run. But it is the best that can be done. The fact remains that randomization is the only way to be sure of causality, and making mistakes about causality can harm patients, as it did in the case of HRT.
Raymond Tallis (1999), in his review of Sokal & Bricmont, summed it up nicely
“Academics intending to continue as postmodern theorists in the interdisciplinary humanities after S & B should first read Intellectual Impostures and ask themselves whether adding to the quantity of confusion and untruth in the world is a good use of the gift of life or an ethical way to earn a living. After S & B, they may feel less comfortable with the glamorous life that can be forged in the wake of the founding charlatans of postmodern Theory. Alternatively, they might follow my friend Roger into estate agency — though they should check out in advance that they are up to the moral rigours of such a profession.”
The conclusions that I have drawn were obvious to people in the business a half a century ago. (Doll & Peto, 1980) said
“If we are to recognize those important yet moderate real advances in therapy which can save thousands of lives, then we need more large randomised trials than at present, not fewer. Until we have them treatment of future patients will continue to be determined by unreliable evidence.”
The towering figures are R.A. Fisher, and his followers who developed the ideas of randomisation and maximum likelihood estimation. In the medical area, Bradford Hill, Archie Cochrane, Iain Chalmers had the important ideas worked out a long time ago.
In contrast, philosophers like Worral, Cartwright, Holmes, Barry, Loughlin and Polychronis seem to me to make no contribution to the accumulation of useful knowledge, and in some cases to hinder it. It’s true that the harm they do is limited, but that is because they talk largely to each other. Very few working scientists are even aware of their existence. Perhaps that is just as well.
References
Cartwright N (2007). Are RCTs the Gold Standard? Biosocieties (2007), 2: 11-20
Colquhoun, D (2010) University of Buckingham does the right thing. The Faculty of Integrated Medicine has been fired. https://www.dcscience.net/?p=2881
Miles A & Loughlin M (2006). Continuing the evidence-based health care debate in 2006. The progress and price of EBM. J Eval Clin Pract 12, 385-398.
Miles A, Loughlin M, & Polychronis A (2007). Medicine and evidence: knowledge and action in clinical practice. J Eval Clin Pract 13, 481-503.
Miles A, Polychronis A, & Grey JE (2006). The evidence-based health care debate – 2006. Where are we now? J Eval Clin Pract 12, 239-247.
Murray SJ, Holmes D, Perron A, & Rail G (2007).
Deconstructing the evidence-based discourse in health sciences: truth, power and fascis. Int J Evid Based Healthc 2006; : 4, 180–186.
Sokal AD (1996). Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity. Social Text 46/47, Science Wars, 217-252.
Sokal AD (2006). Pseudoscience and Postmodernism: Antagonists or Fellow-Travelers? In Archaeological Fantasies, ed. Fagan GG, Routledge,an imprint of Taylor & Francis Books Ltd.
Sokal AD & Bricmont J (1998). Intellectual Impostures, New edition, 2003, Economist Books ed. Profile Books.
Tallis R. (1999) Sokal and Bricmont: Is this the beginning of the end of the dark ages in the humanities?
Worrall J. (2004) Why There’s No Cause to Randomize. Causality: Metaphysics and Methods.Technical Report 24/04 . 2004.
Worrall J (2010). Evidence: philosophy of science meets medicine. J Eval Clin Pract 16, 356-362.
Follow-up
Iain Chalmers has drawn my attention to a some really interesting papers in the James Lind Library
An account of early trials is given by Chalmers I, Dukan E, Podolsky S, Davey Smith G (2011). The adoption of unbiased treatment allocation schedules in clinical trials during the 19th and early 20th centuries. Fisher was not the first person to propose randomised trials, but he is the person who put it on a sound mathematical basis.
Another fascinating paper is Chalmers I (2010). Why the 1948 MRC trial of streptomycin used treatment allocation based on random numbers.
The distinguished statistician, David Cox contributed, Cox DR (2009). Randomization for concealment.
Incidentally, if anyone still thinks there are ethical objections to random allocation, they should read the account of retrolental fibroplasia outbreak in the 1950s, Silverman WA (2003). Personal reflections on lessons learned from randomized trials involving newborn infants, 1951 to 1967.
Chalmers also pointed out that Antony Eagle of Exeter College Oxford had written about Goldacre’s epistemology. He describes himself as a “formal epistemologist”. I fear that his criticisms seem to me to be carping and trivial. Once again, a philosopher has failed to make a contribution to the progress of knowledge.
32 Responses to Why philosophy is largely ignored by science
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
“It’s true that some philosophers have had interesting things to say about the nature of inductive inference, but during the 20th century the real advances in that area came from statisticians, not from philosophers. So I long since decided that it would be more profitable to spend my time trying to understand R.A Fisher, rather than read even Karl Popper.”
Oh dear! Those awful, squabbling frequentist statisticians, who in the 20th century ‘commandeered’ the field from the real scientists who previously had charge of it, were little better than the philosophers.
“Why do physicists see this more readily than others? Because, having created this knowledge of physical law, we have a vested interest in it and want to see it preserved and used. Frequency or propensity interpretations start by throwing away practically all the professional knowledge that we have labored for Centuries to get. Those who have not comprehended this are in no position to discourse to us on the philosophy of science or the proper methods of inference.” –Jaynes.
😉
@phayes
I wonder how typical your views among physicists. I hope they aren’t typical, though I have seen some of the rather crude approach of some physicists to statistics, under the name “propagation of errors”. After all the recent furore over the speed of light involved, if I have understood it correctly, estimation of the shift between two cumulative probability distributions. If that isn’t a statistical question, I don’t know what is.
The whole area of stochastic properties of single molecules, close to my own heart, was derived by statisticians, not by physicists.
Hi DC, this bit of text is mistakenly in the quote format I think:
“The aim of this paper, in my view, is not obtain some subtle insight into the process of inference but to try to give some credibility to snake-oil salesmen who peddle quack cures. The latter at least make their unjustified claims in plain English. ”
P.S. I would love to see you write a short blog post on the beauty of randomisation tests!
@David Colquhoun
Well I doubt my tongue-in-cheek precis of Jaynes conveyed a faithful representation even of my views. Anyway… Typically, I think, physicists are no different than other scientists and pay as little attention to deep issues in probability and inference per se as they do to intimately related ones in QM foundations. They are quite happy to just “shut up and calculate”. That’s usually not a problem and it’s no more a surprise that ‘orthodox’ statisticians have done lots of good science than it is that Brian Josephson has done some good quantum physics.
But (IMO) sometimes it does matter if a scientist hasn’t read Jaynes or a quantum physicist hasn’t read the modern foundational literature. As you may already know from other comments I’ve made here it seems to me that CTs of homeopathy are a serious and pernicious consequence of the former omission. This: http://badscience.net/forum/viewtopic.php?p=680581#p680581 is a less serious and pernicious but common consequence of the latter omission.
The fact remains that randomization is the ’only’ way to be sure of causality,
Randomisation with appropriate controls (and blinding) is the ideal to aim for; so it was disappointing to hear the President of the Royal Society, Professor Paul Nurse, have this exchange with Jim al Khalili
http://www.bbc.co.uk/programmes/b015n3b7
PN: By control I mean that you test whether what you are actually doing is a consequence of what you think it is or whether there could be other effects and quite often we scientists tend to do that a bit late………………..
JalK: It’s the boring part
PN: It’s the boring part. So exactly what I did..it’s the boring part.
One hopes that doing the boring controls at the end isn’t as widespread as suggested in the interview. Expecting the Woomeisters to conform to a higher standard than the President of the Royal Society seems jolly inequitable.
@Crews Control
Yes, I was a bit disappointed by those comments, but perhaps the important bit is than Nurse did appropriate control experiments, whereas woo-mongers (and a lot of social scientists( don’t.
“@phayes
I wonder how typical your views among physicists.”
Speaking as a physicist, I can confirm that a great many physicists are badly ignorant of statistics, which seems a massive shame to me, as I feel that ‘statistical inference’ and ‘scientific method’ should be considered as synonymous.
I strongly agree with Fisher’s arguments for randomization, but I also agree with E. T. Jaynes, that he and other ‘frequentists’ have done extraordinary damage to the science of inference.
There is nothing that frequentist stats can do that couldn’t be done with technology already known to Laplace, and in places where frequentist methods break down, Bayes’ theorem keeps going strong.
Bayes’ theorem operates without violating logic, without violating the likelihood principle, without discarding prior information, with typically (often vastly) reduced computational complexity, and with attention paid to the questions scientists are actually interested in (rather than surrogate questions regarding null hypotheses). It seems a great shame to me that so many influential statisticians have refused to use it.
Quite right, aggressivePerfector, however I think it’s important – in view of all the myths and misinformation flying around and because there is at least one “Bayesian school” – to recognise that Jaynes is not a Bayesian and his book is not a Bayesian statistics textbook. It is much more than that:
“But the final result was just the standard rules of probability theory, given already by Bernoulli and Laplace; so why all the fuss? The important new feature was that these rules were now seen as uniquely valid principles of logic in general, making no reference to “chance” or “random variables”; so their range of application is vastly greater than had been supposed in the conventional probability theory that was developed in the early twentieth Century. As a result, the imaginary distinction between “probability theory” and “statistical inference” disappears, and the field achieves not only logical unity and simplicity, but far greater technical power and flexibility in applications.”
“However, neither the Bayesian nor the frequentist approach is universally applicable, so in the present more general work we take a broader view of things. Our theme is simply: Probability Theory as Extended Logic. The “new” perception amounts to the recognition that the mathematical rules of probability theory are not merely rules for calculating frequencies of “random variables”; they are also the unique consistent rules for conducting inference (i.e. plausible reasoning) of any kind, and we shall apply them in full generality to that end. It is true that all “Bayesian” calculations are included automatically as particular cases of our rules; but so are all “frequentist” calculations. Nevertheless, our basic rules are broader than either of these, and in many applications our calculations do not fit into either category.”
I don’t want to turn this into a Bayes vs frequentist argument. Bayes is fine if you have valid priors, but we don’t so we use maximum likelihood as the basis for inference, as here http://www.ucl.ac.uk/Pharmacology/dc-bits/dcwinprogs.html
We could use Bayes as a way of ruling out values for rate constants that are physically-impossible, but there are much simpler ways of applying such constraints.
It seems to me to be quite absurd to say that Fisher has done “extraordinary damage to the science of inference”.
phayes & aggressivePerfector – Fisher wasn’t a frequentist. If you’re going to discuss the history of the philosophy of statistics, it would help if you knew the basics.
I’m also curious about aggressivePerfector’s claim that frequentists only use the technology from Laplace’s time. Do you mean computational technology (i.e. computers, for a couple of meanings of the term), the mathematical machinery for fitting (e.g. IWLS, the EM algorithm), or the statistical technology (ML, REML, etc.).
bobohara – “Fisher wasn’t a frequentist.”
From chapter 1 of “The Design of Experiments”:
“I will only state three considerations which will explain why… I shall not assume the truth of Bayes’ axiom….. advocates of inverse probability seem forced to regard probability not as an objective quantity measured by observable frequencies….”
bobohara – “aggressivePerfector’s claim that frequentists only use the technology from Laplace’s time”
In fact, I did not make this claim. I simply meant that the foundations of statistical inference were known to and very well understood by Laplace, and that (nearly, at least) all valid results obtainable by frequentist techniques are reproduced or improved upon by application of those foundations.
I don’t mean that Laplace knew, for example, all the sampling distributions that Fisher was (brilliantly) able to calculate, but that in assessing the plausibility of a hypothesis, Bayes’ theorem is not inferior to anything in the ‘orthodox’ arsenal.
David Colquhoun – ‘quite absurd to say that Fisher has done “extraordinary damage to the science of inference”.’
Let me qualify a little. I didn’t wish to assert that Fisher’s net effect was damaging. I don’t know enough of the field or its history to make that claim, and I certainly can’t speculate with confidence how the field would have developed had he chosen a career as a chimney sweep instead.
It just seems to me that there must have been considerable damage done by the dogmatic denial of the logical foundations of probability theory by so many authoritative figures.
Hopefully you can concede that my comment was at least not quite as absurd as some of the nonsense produced by some of the ‘philosophers’ you discussed in the main post.
Getting back to the point of your article, I feel that the distinction between science and philosophy is a false one, and that the real philosophers are people like Jaynes and (grinds his teeth) Fisher. It seems like there was a schism at some point, when some philosophers (who eventually started calling themselves scientists) began to think scientifically, and some others (who went on calling themselves philosophers) didn’t like the difficult mathematics or the mental discipline required, and so continued to conduct philosophy in the old, obsolete way. (Possibly over simplified – difficult without word count spiraling out of control.)
Its tempting for scientists to ignore philosophy (because of how unscientifically it tends to be conducted), but I feel science has a lot of lost ground to win back. In my view, scientists should consciously and publicly engage in philosophical thought, not just to advance their own program, but to try to bring philosophy back on track. I guess thats related to why I like your blog.
aggressivePerfector – thank you for quote-mining. It’s obvious that you don’t even know what a frequentists probability is, otherwise you wouldn’t quote something that’s utterly irrelevant to your case. Yes, it’s well known Fisher wasn’t a Bayesian, but that doesn’t make him a frequentist.
Fair enough about me mis-representing what you wrote about the ‘technology’ that frequentists used. Although I’m not sure I’d describe the foundations of statistical inference as “technology”.
bobohara – maybe I wasn’t clear enough. What is demonstrated by the quotation I used is that Fisher himself did regard probability as an ‘objective quantity, measured by observable frequencies.’ If you feel thats not enough to classify a person as a frequentist, then I’m afraid thats probably not a debate I’m interested to pursue.
aggressivePerfector – it’s not a debate, it’s a matter of fact and definition (i.e knowing what a frequentist is, and whether Fisher’s views agreed with frequentism). It’s a matter of common knowledge that Fisher’s interpretation of probability wasn’t frequentist, even though it wasn’t subjectivist either. He called them fiducial probabilities.
From the paper of 1930 in which Fisher introduces the concept of “fiducial probability”:
“It is therefore important to realise exactly what such a probability statement, bearing a strong superficial resemblance to an [hated ;-)] inverse probability statement, really means. The fiducial frequency distribution…”
Click to access 84.pdf
The frequentist, fiducial and Bayesian models all seem to be a source of dissatisfaction. Perhaps the ‘replication’ model might be more widely accepted. To take a simple example, if there is a study result of 70/100 and all studies of 70/100 were repeated a large number of times with 100 subjects, what proportion of these repeat study results would be replicated between two ‘replication limits’ of say 60/100 and 80/100? This is also a sampling model but it estimates a ‘posterior’ probability directly. If there was also a prior subjective estimate of a result equivalent to say 7/20, then the calculation could be based on (7+70)/(20+100). However, the result of 70/100 (or 77/120) would only be one feature of the paper and other features would also have to be taken into account.
In order for the probability of replication to be high, the probability of non-replication due to all other ‘causes’, has to be low. For example, the subjective estimate of the probability of non-replication due to poor reporting of results or methods (due to error, ignorance or dishonesty), poor or idiosyncratic methodology, different circumstances or subjects in the reader’s setting, etc should all be low. Finally the probability of non-replication (e.g. not being between 60/100 and 80/100) by chance due to the number of readings made must be low. If the probabilities are low for all possible ‘causes’ of non-replication, then the probability of replication should be high.
The same reasoning is used to show that the probability of a diagnosis is high. This is done by showing that the probability of each differential diagnosis is low. There is a theorem in probability theory that models ‘reasoning by elimination’. It uses Bayes theorem as its starting point. It is described in the Appendix of the 2nd edition of the Oxford Handbook of Clinical Diagnosis (http://ukcatalogue.oup.com/product/9780199232963.do).
@Huw This is Peter Killeen’s ‘cure for the p-value disease’, right?:
http://sciencewatch.com/dr/erf/2010/10octerf/10octerfKill/
Click to access 18065_Chapter_7.pdf
It’s an interesting hybrid of Frequentist ideas and Bayesian methods which would lead to scientists making very weak claims indeed (“Inferences are to future research outcomes, not to parameters.”). If I were a pessimist I’d consider that a very good idea: a kind of ‘bicycle stabiliser wheel’ for scientists who cannot or will not read *and understand* Jaynes. 😉
This more recent paper by Worrall is pretty good:
http://www.sciencedirect.com/science/article/pii/S0091743511002982
[…] sc_project=233721; sc_invisible=0; sc_partition=0; sc_security=""; ← Why philosophy is largely ignored by science […]
@phayes; that is not right. What I proposed is not Peter Kileen’s approach, which was to calculate an index called a ‘P-rep’ such that ‘P-rep’ ≈ [1 + (P/(1-P))2/3]-1. Kileen then argued that ‘P-rep’ should be regarded as the “probability of replication”. I understand that there were many objections to this, some to do with the assumptions underlying his calculations, but more importantly, that he did not take into account other things that would affect the probability of replication such as the prior probability of the outcome, the results of other similar studies, the accuracy of reporting, differences between the authors’ and readers’ settings etc.
What I have suggested also involves parameters but about a known population e.g. the set of sets of 70/100. These parameters are about the ‘known’ population from which a FUTURE range of study results could be selected and the same mathematical skills would have to be used as those used to calculate P values, confidence intervals, Bayesian credibility intervals etc. This population with ‘known’ parameters could be the set of sets with a particular proportion (e.g. the set of sets of 70/100) or the set of sets of continuous variables (e.g. the set of sets with a mean of 7.7 and a standard deviation of 1.7 say). Their parameters would be the same as for the hypothetical population that would give the maximum likelihood of selecting the result observed in the study. In each case, the statistician may wish to incorporate a prior belief about the parameters.
The agreed low probability of non replication from chance that makes it worthwhile considering other causes of non-replication could be less than 0.05 for a pair of limits (e.g. p=0.025 that it is greater than 80/100 and p=0.025 that it will be less than 60/100) or p= 0.05 that it will be less than a single limit (e.g. less than 80/100.). In practice, this is very similar to how many appear to interpret confidence limits and P values now.
Far from allowing the scientist to make weak claims, a low probability of non-replication due to chance (e.g. p≤0.05) is only the first hurdle. In order to achieve a high probability of replication, many other causes of possible non-replication must be shown to have a low probability. If any of these possibilities has a high probability of causing non-replication then the probability of replication will be low.
It is only after the issue of replication has been settled that the discussion will turn to possible scientific explanations in terms of hypotheses about physiology, pharmacology, biochemistry, genetics, etc, etc. However, the statistician will have already been considering hypotheses about the parameters of the population from which future replicating results could be drawn.
In terms of clinical probability estimates, the author’s observation might be that 70/100 patients in an emergency room presenting with a wheeze turn out to have asthma, but the author and reader would find it useful to estimate the probability of repeat studies lying between 60/100 and 80/100 (subject to the probability of other causes of non-replication being low).
In defense of Philosophy:
I find articles like this deeply upsetting. I struggle to understand the motivations for writing a piece of this sort, and can only imagine that the author, having been exposed to some bad philosophy, has decided that all philosophy is useless.
The author has managed to tar with the same brush all philosophers, as somehow akin to those who deny that randomized control trials ‘carve the world at its joints,’ or serve as adequate evidence for a causal process. Either they are like Worrell, (whose papers are not accessible on the hyperlink) unashamed navel gazers, who cannot leave the confines of their privileged thought processes long enough to pay attention to an outside reality, or they are like Holmes and Barry, so post-modern that the only thing they have failed to deconstruct is their post-modernism.
What the author fails to note of course, the large body of criticism, within the field of philosophy, for trends just like these. Of course he fails to note this, since the opening paragraphs seem to display some secret pride in this lack of engagement. Is he aware, for instance, there are a large body of philosophers who are quite disgusted at the notion of people like Derrida even being referred to by the title?
However these are not the things which dismay me most, though they are symptoms. It seems to me that the author fails to understand quite what philosophy is. Philosophy, ladies and gentlemen, is merely the study of argument and method. It is the science of true thinking. It is not prescribed by philosophy that one must regard all things as foundationless, or that one must insist that RCT trials are unnecessary. In fact, just like science, whatever position you take has to be justified, and your justification can always be brought into question by anyone. Evidence must be provided to support argument. It is not, by any means, a discipline, like alternative medicine, in which all hypotheses are given equal credence. I offer you a paper by, the dismissed, Karl Popper as example: http://www.stephenjaygould.org/ctrl/popper_falsification.html
The author asserts that in all his time in science, he has seldom heard the word philosophy mentioned. Is that scientific evidence for its uselessness? I suggest that the cause is rather that a science degree never includes explicit philosophy in its curriculum. Were it to, I believe you would find far more philosopher scientists. Of course, the evidence is not in, but without making that change, it never will be.
As for the impact philosophy has had on science? How is it, dear reader that scientific hypotheses are formed at all if not philosophically? There is so tight a bind between aspects of philosophy and science that science itself might be called experimental philosophy without any significant loss of meaning. Indeed the early scientists named themselves natural philosophers. While the discipline has grown well beyond the recognition of its parents, I find it hard to countenance a view that denies it any genetic remnant. As evidence for this I present to you one of the biologists Stephen Jay Gould’s famous essays, ‘The Spandrels of St Marco.’ http://www.aaas.org/spp/dser/03_Areas/evolution/perspectives/Gould_Lewontin_1979.shtml
Gould presents a critique of an entire school of evolutionary thinking, that being of the teleological adaptationists who desired to see all adaptation in terms of evolutionary benefit, as opposed to neutral change subsequently built upon. Gould and Kimura made room for much stochasticity within evolutionary theory. Like Gould’s punctuated equilibrium, we see experimental results subsequently confirming the theory, however the initial critique, born out of a philosophical dissatisfaction with what he was seeing, allowed us as researchers to interpret later findings, and to design experiment to confirm or deny those findings. At this level, theory and hypothesis construction is a deeply philosophical enterprise. You have to know what your findings are going to mean (not what they are going to be) before you can do a test.
What frustrates me so much about the position take by the author, is that given the linkages between science and philosophy, there is much about philosophy that can only enrich the scientific worldview. To cast philosophy in such disdainful light, is to lose out on all the insight of a 3000 year old research program, the only goal of which is to find the truth. You may not like where some philosophers have gone, but it Is regularly true that they have made contribution to the large conversation which is human progress.
As a final piece I offer Hume’s Dialogues Concerning Natural Religion, which provides excellent arguments against the existence of God, and also, interestingly to this particular conversation, excellent arguments against the validity of alternative medicine. http://www.anselm.edu/homepage/dbanach/dnr.htm
@Fardarter
Perhaps you protest too much. I was not speaking about all of philosophy, just the bits that purport to help science. Even among those I picked examples that were positively harmful, rather than the larger amount that is more or less right, though, I would contend, not very helpful in practice.
I’m perfectly happy to listen to philosophers’ views about, for example, morals. Though I must say I haven’t found their views more valuable than anybody else’s.
And yes, I’m quite aware than many. probably most, philosophers are not taken in my the pretensions of post-modernism. Nevertheless, papers continue to be published.
You say
“How is it, dear reader that scientific hypotheses are formed at all if not philosophically?”
This suggests to me that perhaps you have never done any science. Perhaps the most common way that hypotheses form are when you look at the results of an experiment and say “That’s odd: I wonder what’s going on. Could it be ….?”. I suppose you could call this process ‘philosophy’ if you want, but that
term doesn’t seem necessary to me. One problem for philosophers is perhaps that they tend to discuss science in terms of Einstein versus Newton. But that is a once-in-a-century happening. The vast bulk of science isn’t anything like that.
(Links are fixed now.)
Well my example was Gould and his discussions on evolutionary theory. Not Newton, not Einstein.
What do you make of that case?
“That’s odd: I wonder what’s going on. Could it be ….?”. I suppose you could call this process ‘philosophy’ if you want, but that
term doesn’t seem necessary to me.
It is only because you have taken position against philosophy that you desire to define philosophy out of that process. There is nothing more philosophical than curiosity. What cost is it to you to allow the term? To philosophers, the denial is to exclude them from aspects of the world that they feel that they attend to.
@Fardarter
I sympathise with both positions. I have been trying to explain diagnostic opinions and medical decisions to patients and students throughout my career. The basic principles of medicine and science are vague. When I started on the wards and in the lab, I remember being amazed that I was expected to learn to reason by a long process of imitation. I have some sympathy with philosophers who try to make sense of it all without going through a long apprenticeship in science and medicine. Because of the ‘intuitive’ nature of much of scientific reasoning, it is not surprising that there often is confusion and that the public and patients are susceptible to ‘authoritative’ nonsense, which irritates those who work hard to do things properly and to the satisfaction of their peers or strict regulatory bodies.
In medicine and science, there is therefore a non-transparent ‘intuitive’ thought process which those who are experienced recognise in those who are equally experienced (they also recognised when others are not similarly experienced). However, I think that there is also a transparent thought process that can be taught. I use one thought process as a check for the other and teach others to do this (e.g. in the Oxford Handbook of Clinical Diagnosis). ‘Reasoning by elimination’ seems to play an important role in transparent reasoning. It works in a similar way in statistics, diagnosis and scientific hypothesis testing. Another important application of this reasoning by elimination is when estimating the probability of replicating the result of a scientific observation (see @Huw// Nov 2, and Nov 5).
In order to be able to explain medical reasoning to patients and students, I have tried to show the relationship between classical logic (traditionally the field of philosophers) and probability theory. In my MD thesis in 1987, I showed how reasoning by elimination, the syllogism, converse, obverse, contra-positive, etc could be re-stated in terms of theorems based on Kolmogorov’s axioms and how all this could be applied to day to day clinical practice, teaching and research to the benefit of patients. I am now teaching this in the Oxford Handbook of Clinical Diagnosis. However, my contemporary doctors, scientists and philosophers do not find it easy to understand these ideas, perhaps because they are highly specialised. Most students, who are not so burdened with long established ways of seeing things, seem to find it a matter of common sense.
I wonder whether scientists, doctors and philosophers would have more common ground if they tried to understand the ideas on reasoning with hypotheses and probability theory in the Oxford Handbook of Clinical Diagnosis (http://ukcatalogue.oup.com/product/9780199232963.do).
[…] also refers to his October 28 piece that will certainly rile up people, “Why philosophy is largely ignored by science.“ I have in the past, taken an occasional interest in the philosophy of science. But in a […]
[…] hard-hitting essay on “Why Philosophy is largely ignored by science” is also worth reading. Although myself an exponent of philosophy and the philosophical […]
[…] David Colquhoun at Improbable Science weighed in on hy he doesn’t think philosophy has much to offer sciencew, and he uses a fairly peculiar example of causation linked to a diet study to talk about an error […]
In defence of Cartwright (I attended a talk of hers a few days ago on RCTs), the point she’s making is quite simple. RCTs aren’t the gold standard in all cases. Physics has amassed a decent body of well-established causal knowledge and no one does RCTs in physics. They use the hypothetico-deductive method and it seems to be working quite well for them. RCTs are best when we really have very little other knowledge to rely on. The control group allows us to factor out our ignorance (of which there will be plenty in the case of medicine).
This isn’t to deny that some of her points are perhaps put too strongly. The other claim she’s pushing for is that many advocates of RCTs in the social policy arena want to base interventions on one RCT conducted on one sub-population. That this is somewhat less than sensible is something that any advocate of EBM will agree with. She’s reacting to those in development economics who seem to have got a little too excited about RCTs and want to move too fast. But I appreciate that it can seem as if she’s objecting to RCTs themselves.
All these issues will be resolved when we leap to the next Prigoginic level of organization!
I am intrigued by this post. I think you made a mistake ignoring the philosophy of Karl Popper. He dedicated his entire life to advocating a decision-making method that looks at the cost of making the wrong decision. But Popper was not a statistician. So I can understand that he might not have come across as some appealing. Nevertheless he has the honor of being credited and mentioned by at least three different Nobel Prize winners (John Eccles, Peter Medawar, Paul Nurse) while being dismissed and ignored by philosophers of science.
Thanks for your comment. I guess that Popper didn’t get mentioned much is that the idea that all you can do with a hypothesis is to falsify it (or fail to falsify it) is built in to just about every approach to statistical inference for the last 100 years. But the advantage that statisticians have is that their arguments are quantitative, not just qualitative.
Statisticians have also approached quantitatively the problem decision-making that takes account of the cost of making the wrong decision. My own feeling about these is that often (not always) the assessment of the cost of making a wrong decision is so difficult, and so prone to bias, that cost-benefit calculations can rarely be trusted, but at least they ask the right questions.
I have changed by views a bit since writing this piece. I have become more sympathetic to the Bayesian approach and less sympathetic to Fisher, as described, for example, in a recent post, and in a 2019 paper.