Penny Lane, thank you very much for coming.
And I'd like to welcome you to the evening's guest lecture, which is part of the evidence based health care master's program.
I'm delighted to introduce Professor Alan Selman,
who is an epidemiologist and rheumatologist and is currently a professor of musculoskeletal health here in Oxford.
He's also the author of the core textbook that we recommend, and I started my research career out with Helen.
So a lot of what I have now is thanks to Alan,
but I'm personally very excited that he's here tonight and he's going to talk to us about how bad so many of our old examples.
Thank you. Well, thanks, Claire. So I am aware that you've been at it all day, so firstly, feel free to nod off.
I'm used to it. And secondly, I'll try and make it reasonably light hearted.
As much as epidemiology can be light hearted and really to share with you some
thoughts about kind of things that I may or may not have done differently and when.
So I'm going to about four challenging areas. One, is primary data collection worth the hassle?
Secondly, epidemiology is not the art of the perfect, the art of the possible.
Third, the that the days of the randomised controlled trial are probably at an end.
And fourthly, we spend a lot of our life trying to identify causes and do we actually know what we mean?
So those are the four areas that I'm going to attack and hopefully if do nothing else, but stimulate a little bit of discussion.
So what about primary versus secondary data collection?
What do I mean by that? Clearly, I mean,
the studies that Claire is so wonderfully involved in when we work together in Manchester is what was traditionally the view of the epidemiologist.
You went out and you interviewed people or you sent them a questionnaire by post or you did some measurement on them.
And the alternative is that you use the data that somebody else has collected.
And if you were to draw up a balance sheet, this would be on your balance sheet.
Well, of course, it's expensive collecting primary data and it takes all the time.
It's quite difficult to do it in a broad area.
But if you know what you want to collect, it can be accurate and you can collect it quite a lot in quite some detail.
And the secondary data costs are low, speed is quick.
You can cover a broad area of accuracy.
Well, very interesting, actually.
Very interesting. I was I often said as an epidemiologist, if you're interested in international health.
The one thing you can rely on is vital statistics, births and deaths.
And I was at a meeting a few months ago chatting with a colleague from India who told me
that he reckoned probably only about 30% of Indian death certificates were accurate.
So we assume that secondary data maybe has even low accuracy.
Maybe it's not even as good as that.
But I think there are an enormous number of issues in collecting primary data.
And when I think of the amount of effort that I've engaged in, and I guess more importantly for other people to engage in in collecting data.
And the first thing is this issue of will we never study a whole population and we only have a study, a sample.
And actually, when you start thinking about that,
it immediately causes a lot of anxiety because the whole basis of statistics is based on the
fact that we are taking a random sample and inferring something about a whole population.
Now, let's take the simplest idea where you're doing in a population in terms of a census, a whole national population.
How do we go about taking a true random sample?
Well, it's actually very, very difficult. And we might go to primary care.
But what is a primary care population?
How representative and how do we know that that primary care population is representative of all other populations?
Much more challenging than that is when we as clinicians study patients with a disease
in a case control study or recruitment for a longitudinal observational study.
Because what we do, we make the inference that the people we are studying are a representative sample of everybody with the disease.
As what it is that exists now has existed in the past.
Will exist in the future. Well, of course, we can't take people who will exist in the future because we don't know who they are.
And studying people who existed in the past are going to tell us about things that are of historical interest.
So once we start thinking that we're doing taking a random sample of people with a disease in terms of something,
we realise that the underpinning that we're taking a true random sample is flawed.
If I take a, you know, do a clinical trial, the people I treat with the disease in my study, the future generations,
future physicians, future health care professionals who want to know that my sample is saying, well, clearly it can't be.
So I think some of the inferences we're making are flawed.
It's why, I guess I'm saying epidemiology is the art of the possible.
However, even if we can get that, we've got to get permission now.
You can't necessarily read all those stages, but it is incredibly hard getting all the necessary permissions to get a study started.
The National Institute of Health Research reckons on from the date of letter of your award,
not the date of the letter of when you submit your grant, but when that letter lands in your mailbox.
We are pleased to tell you you have got the grant way.
Hey, you are lucky if you recruit your first patient within 18 months.
That's what they work on. 18 months from award to recruitment of a first patient.
And who are I? There isn't to predict.
And you tell you the enormous steps that you need to go through in terms of the ethical approval,
the governance and all the relationships connected with that.
And government attempts to try and simplify that process have really struggled.
I remember some years ago, a very long time ago,
I wanted to do a study of a very it's quite a common condition that some vineyards live at or cal poly myalgia rheumatic
heart condition ah where elderly people get significant ache and stiffness in their joints not associated with arthritis.
But isn't associated with increased inflammation raised sore in the blood and nobody knew how common it was.
So I decided I wanted to do a study and I thought that I would choose a convenience sample instead of a random sample.
I went to an old people's home or three old people's home in Southend on Sea.
Didn't bother getting ethical approval because you didn't need it in those days.
And I just contacted somebody about in the home, said, Yes, that's fine.
So I turned up at home and say, the doctors here to take your blood.
I shouldn't be telling you this. Well, anyway, I did.
And it got published and it was the this country's first data on the epidemiology of poly myalgia.
To do the same thing today would be very difficult.
And even if you've got all the admissions and what about getting a worthwhile response?
And I spent about 15 years of my life on the steering committee of UK Biobank.
Hands up, who's heard of UK Biobank? So lots of so UK Biobank.
Those who haven't had it was this country's Tony Blair.
Great initiative study. Half million people.
And the main aim of the study was to look for gene and environment interactions of the main diseases in the elderly.
So the aim was to recruit half a million adults aged between 45 and 64 in the UK.
And the studies that we'd been doing, we reckon when I was involved in epidemiology we wanted to recruit random sample the population.
We reckon with a bit of a push.
We could probably get up to 65 70% response rate, but UK Biobank letters would come through asking people to contribute blood, urine and answer some.
Students for the greater good of humanity and take part in something very much worthwhile.
Do you know what the response rate was to by bank anybody? No. But what about 10%?
It was about 10%. And that's really interesting because the couple of things that came out may be not surprising.
If you look maybe at the distribution of towns in school, it, as you know,
is an index of deprivation or clearly the people who took part in UK Biobank were not representative of of the great British population.
Now in some ways I remember when we when we had a big discussion about UK Biobank,
we had this great discussion about representativeness and sampling and we talked about minorities and how how did we ensure
that there was a sufficient representation of the multiple minority groups that exist in the current UK population?
And of course this is a constant change in did we you know, if they if 0.03% of the population came from Vietnam,
did we have have 0.03% of the of the respondents Vietnamese?
And if we didn't, did that mean the conclusions weren't extrapolated?
So it was a challenge. And in the end, we decided to drop it and hoped that we would have enough in in different minority groups to make a judgement.
But we've seen about Townsend score.
Well so see here in terms of cigarette smoking.
These are data comparing UK Biobank with the Health Survey of England.
And of course Health Survey for England doesn't necessarily mean that it's representative.
But clearly the people who were recruited to in UK Biobank are overrepresented.
The more affluent, the healthy, the worried well, the more educated in the population.
And yet UK Biobank has been one been a very rich resource on the epidemiology of hundreds of diseases within the UK.
Does it matter? Well, I don't know. It possibly makes it less relevant when you're comparing within the cohort to compare, say,
between never smokers and current smokers within biobank might be just as relevant as comparing between current smokers,
never smokers in the population as a whole.
That is an assumption which may or may not be true, but certainly in terms of just understanding the prevalence, there's clearly a limit.
The other thing is that the the the survey method works without problems that we believe when we are collecting that information,
the information we're collecting is giving us the answer we want.
And sometimes that's not the case.
I remember a number of years ago I was involved in a study of back pain in 35 European populations,
and we translated the question into the different languages, and then we back translated to make sure that the meaning hadn't been lost.
Okay.
And we found and these were general population samples and the prevalence of back pain varied between seven and 82% in the different populations.
I cannot believe I cannot believe that the differences we found were due to differences in the true occurrence of back pain.
What I do believe is that actually most people don't know where their back is.
If I was to ask you why your back was, you would all tell me some way.
Something different. Somebody here, some here, some might.
You can't see what I'm doing and what is pain, what tonight, etc. etc.
You take a member in UK Biobank.
We accepted long and hard about dietary survey methodology, how we ascertain what what people eat.
I've been doing some work recently about physical activity in the elderly.
Now you can come up with all the instruments you like and there are hundreds of instruments and we can look at that qualities now,
integrate and integrate reliability, etc., etc.
But most of the things we are measuring are surrogates for something that is really quite difficult.
Now that's not to say they're meaningless. Clearly they're not meaningless, but there are challenges and often we put blind faith in the data we have.
And interestingly, it's only when we have the comparative data that we realise there may be an issue.
And one example of this is I've been interested or I was interested some years ago in whether
some new drugs for treatment of rheumatoid arthritis were associated with an increased rate,
of particular of people having various kinds of infections.
And we did the same study in England and in Sweden, and we found that in the people who took this drug,
they had exactly the same rate of infection, which was fantastic.
Maybe that was a suggestion of some validation of the method we chose.
The problem was that the comparison group in England and the comparison group in Sweden were so different,
but we didn't find a significant difference and they did.
So we both had the same outcome in the treated group, but we both had very different outcomes in the untreated group,
possibly related to the way we collected data and possibly the way we collected the sample.
The other problem, particularly in in prospective longitudinal studies, is retention.
I am when I left Manchester, I was the medical director of Arthritis Research UK.
Some time I had a very ambitious programme of doing some clinical trials, major clinical trials, clinical trials that needed answering.
And we clinical trial, we had a committee of the great and the good and maybe the not so great and not
so good sometimes to determine whether this trial was achieving its goals.
And there was a real emphasis on recruitment.
And it was a bit like, you know, the, you know, the new funds for the church steeple, you know, as the it goes up.
Have we raised enough money? So has the recruitment reached the required level?
Actually, recruitment in clinical trials is a bit of an issue.
The biggest problem is retention. The bigger problem is retention.
And the people you lose means that they are not providing that information.
Now, we could have a discussion about imputation. We could have a discussion about missing at random, completely random, etc., etc.
But the one of the points I want to make is, which I think is, is a really important point, that in terms of primary data collection in studies,
the people you lose are possibly the biggest determinant of the loss of validity of a study.
And it is very hard, very hard often to retain people.
So why not use routine available data?
I mean, often routine data is collected often without any aims.
Primary care data, hospital data, census data, occupational data of many kinds is not collected to answer research specific work research questions.
Often it's done for the purpose of finance or audit.
The good thing is it's not fixed in time.
And as Claire notes, we have in the UK, we've got really good primary care data.
Now we can discuss whether it is of research quality and how different general practices, different general practitioners use different terms.
We recently were interested in CPD to assess people with hip pain and the way
that that was coded by different general practitioners was quite remarkable.
But nevertheless it exists and it exists longitudinally.
We've got very good data for some diseases, particularly cancer.
We've got some data on people who are admitted to hospital.
It's not brilliant. The coding is dependent on the coding done by the person who's responsible.
Suppose somebody comes into hospital to have a knee joint replaced and then develops
venous thrombosis and then has a pulmonary embolism and then developed for pneumonia.
What get coded? What doesn't get coded? What's the important thing?
What's not the important thing when people then have readmission, etcetera, etcetera.
There's any number of complexities with hospital data and increasingly with people treated as day care,
etc. And we're also very bad at understanding people who might come in, for example, injection treatment or some physiotherapy or something like that.
I've been heavily involved in the National Joint Register,
which is a register of everybody in the UK, actually in England who's had joint replacement.
They are joint replacement registers in many countries in the world.
They have spawned an enormous amount of data about the value and the consequences of joint replacement surgery.
But they are not without problems. They are not complete.
We are no, they're not complete, although the completeness has gone up.
They're not always accurate. We don't always have complete follow up.
And often in terms of the things that you want, that information is missing, maybe in terms of some of the outcomes.
So it's quite good on maybe short term effects and it might be very good for manufacturers, but long term health outcomes.
And I've also been involved in drug treatment registers.
And again, there's a that can be very useful, a means of understanding the long term consequences.
So yes, secondary data has its limitations, but it's there.
It's available to be used for a whole range of uses for descriptive epidemiology.
It can tell you very easily in a way that would be very difficult to do for primary data collection in terms of,
for example, the incidence in this case of different forms of skin heart disease.
It can understand the age and gender influence.
You can look at at time trends.
And over the years I've used publicly available data to look at seasonal trends in relation to certain conditions because with all its limitations,
how else can you understand that? And also you can look for geographical variation,
routine data though and to some extent is is limited because as I say, routine data is collected for routine purposes.
It's not collected to answer any specific questions and is a very big and in fact, actually in the today's today's Guardian,
there was something about dangers of implants that all of these nasty orthopaedic surgeons putting nasty bits of.
Settle into otherwise none nasty human beings.
And why didn't something like the national joint in Manchester understand that people where there was a a metal and pelvic bit and a metal hip bit,
so called metal on metal. Why didn't the national joint in Manchester come out with these things are harmful.
The metal ions get into local tissues, cause the reaction and and the joints can become loose, etc. etc. etc.
Now it wasn't the fault of the national joint in Manchester that it didn't answer the question.
The National Joint May said, Look, we've got the data. It was up to you, whoever that you is, to think of the question and answer it.
So often we kind of assume that all these large routine data sets will somehow answer questions that they have been taught,
maybe some machine learning or something to tell you.
Essentially, they are a tool, but they need at the moment a human interface to to ask questions.
I think what is useful is the potential that you can explore things within routine data.
And it may be that it's there's some potential for doing research within the register, but often a register is limited.
I think some most interesting stuff is the potential linkage between registers.
And I'm sure a number of you will have heard of the FA Institute and I went to very interesting talk
where in Wales they had linked childhood absence at school with mental illness later on in life.
So this has the potential of linking registers that can add to the richness of one register.
I sometimes feel it would be wonderful to live in countries other than Britain and in fact, I think what's happened recently,
I think it'd be necessary to live somewhere other than in Britain if you want to survive as a researcher.
But anyway, who knows, you make it through pilot.
But there's a colleague of mine in Sweden a number of years ago, and in Sweden up to four years ago, believe it or not, they had conscription.
I'm not sure what the people did in the Swedish army. I'm not sure who the enemy was, maybe the Danes.
But one of the questions that they asked the Swedish conscripts was how regularly they took cannabis,
which of course being Swedish they were very happy to tell you about.
And they were then this guy was then able to link these anonymized questionnaires that were collected on all Swedish conscripts to the
Swedish mental health register to show that there's a link between cannabis use in recruits and a schizophrenia later on in life.
Now I can not say that it was the cannabis made be, you know, other confounders, but the potential of linking or registers is really important.
As Clare invited me, how could I not talk about an example where we were able to link to registers,
where we were interested in diet and rheumatoid arthritis and it just this could do a study of a disease as rare as rheumatoid arthritis.
Collecting decent diet data would have been impractical.
And what we were able to do with Link a disease register that Claire worked on the Norfolk Arthritis
Register with a very detailed register of diet carried out fortuitously in the same area.
And we were able to show linking those two registers that diet rich in certain fruits were protective against development of rheumatoid arthritis.
So. And Biobank.
The richness of Biobank is not in terms of what data by bank collected cost actually, but what it can tell is, say linked to two primary care.
But the challenge in routine data is is is quality.
And in I talked to you about the National Joint Register,
which has been used as the data on the outcome of hip joint replacement and the quality of the data.
It's it's it's good, but it's not what we would call research quality because research quality data in requires is, you know, the kind of things that,
you know, we're constantly testing for drift for variation between observe and look at that variation within observes in the way things are collected,
completeness, etc.,
etc. and the kind of things we teach epidemiologists in terms of how you maintain data quality in a study lasting six months or a year or two years.
And here we were relying on data collected nationally over maybe decades.
And there isn't a funding there, isn't there? So we cannot accept we have to.
So we have to accept that there are limitations in the quality of the data.
And there's a trade-off between.
On the one hand, I told you how difficult is primary data, but we have to accept that there are challenges in the quality of secondary data.
So epidemiology to me, I don't think it's possible to do the the the perfect study.
I think anybody who says they've done the perfect study I don't think is telling the truth.
But I think what it's our responsibility as investigators to do two things to be clear about the possible imperfections.
Is it something bias biases it problems of the survey methodology?
Is it a problem with numbness, point loss to follow up and be open about it?
And then and then the key things. And okay.
Given that, what are the consequences?
What are the consequences? And sometimes, guys, sometimes it might be that actually the consequences can be helpful.
A bias could be helpful. What do you mean?
What I mean is the kinds of this model bias could be helpful.
And let me give you a suggestion and a real example.
So there was an at one time there was a worry that women who had influenza during
pregnancy were at risk of having an infant who would have delayed milestones.
So they took women. They took children with delayed milestones and had a passing group of children without delay milestones.
And they interviewed the mothers about whether they'd had influence event in pregnancy.
And actually,
that was probably the only way they could get that information because it wouldn't be neatly recorded because most may not have gone to their GP.
And you immediately would say, well, of course the women who had a child delay friend would be more likely to recall the things that had happened
in pregnancy and maybe would be more likely to recall having influenza than a woman who's got a normal,
healthy baby. And this would be a criticism of that study.
That particular study found no difference.
That particular study found no difference. So if there had been a bias, it would have been in the direction of finding a difference.
And because they found no difference in some ways,
that bias in the study in that third design did not really affect the conclusion which was in this particular instance we.
So biases in that sense can be helpful.
So it's important to be clear about biases and it's an.
Important to go beyond that and understand how the imperfections will have influenced the results.
So I tell my students, for example, I don't expect you to do the perfect study,
but what I do expect you I do expect you to understand in great detail the imperfections of what you've done and how they may have altered,
if at all, and by how much the results you've obtained.
Okay. Moving on swiftly. Okay.
Randomised controlled trials overblown. I could talk for a long time about randomised controlled trials.
I've only done I think three in in my life and in fact it's the only paper I ever got in the New England Journal of Medicine.
But there were challenges that expensive at the time.
It's government recruitment and retention, etc., etc.
And there is a big debate between doing a randomised controlled trial and a longitudinal observational study.
And you know, we worship the god of the randomised controlled trial and despite all their difficulties as people in clinical practice would say,
well I know we have to look at the best evidence and the best evidence is a meta analysis of clinical trials,
because we've all read the Cochrane reviews. Actually, I must tell you, when I did I was taught by Archie Cochrane.
Interesting guy, very interesting guy. Areas Colour wrote in his lapel.
Didn't make him interesting. I just found that.
But one wonders, particularly in treatment trials, which is a more useful question, which is a more useful question is drug A better than drug B?
Is treating patients with surgery better than treating people with physiotherapy?
Is treating people with CBT on the telephone better than having individual CBT?
Yeah, we know that all these things are probably useful, but we keen to have a single answer.
But maybe the more important question practically is which patients, which populations will do better with A and which with B?
Because actually, in today's world, when we do these comparative studies of treatment, actually they're probably all effective in somebody.
One of my spare time occupations is I chair the appeals panel for Nice.
And Nice is often based on whether we approve or don't approve a drug or technology for use in the NHS.
And the appeal always comes down on not whether a drug works or not or even it can be shown to be cost
effective or not because in some people it will work and in then people it will be cost effective,
but in others it won't work and in others it won't be cost effective.
So on the whole, it may be that it might not be more cost effective than some people or work more.
But the drug company or the patient advocacy group or the health care professional advocacy groups say,
but if we knew the subgroup in whom it worked, then we should use the drug.
And now we can show you clinical trials,
we can show you observational studies where we've done the subgroup analysis and shown that in this subgroup it's great and therefore nice, use it.
And we couldn't do a clinical trial of every possible subgroup because we didn't know
what subgroups there were to start off with and it would have been too expensive.
But of course, the statisticians say, well, this is post-hoc reasoning.
You can't go if you didn't start off with your analysis, you didn't start off setting out to look at this subgroup.
Don't come to us as pure statisticians and say, now it works in a subgroup.
But I just wonder, folks, I just want the facts in reality where they're actually accepting that most treatments will work.
In some people, the challenge is not a better than BP, less than.
0.05 or even p less than 0.01.
Because it could be 48% and 52% like the referendum.
And so I think and I didn't I didn't I didn't make an amendment.
And but it's actually in in in who it were.
And it's really interesting to me because for years I was a practising clinician and I was impressed.
There were there were drugs are used but worked in some patients but didn't work in others,
but different drugs worked in others and didn't work in in the other group.
And if I'd done a clinical trial, it might have told me something about the order.
I should use the drug, but it doesn't necessarily tell me how it works.
And maybe we need to rebalance some of the questions we ask in terms of doing the good longitudinal study.
And that doesn't mean to say that there's any compromise on quality to do a high quality,
longitudinal observational study that enables you to do a proper subgroup analysis.
You need to apply the same rigour in sampling, in recruitment, in the retention, in the methodological quality and reliability, validity, etc.
It's not an opportunity to do a study on the cheap, but it's maybe an opportunity to do something that's perhaps more useful.
I'm just going to in the last four or 5 minutes, I just want to just out of interest, talk a little bit about why we need epidemiology anyway.
So what actually is a cause? Two weeks ago I was in Hiroshima and I heard some harrowing stories, harrowing testimony about radiation damage.
You didn't need an epidemiologist to know that radiation causes severe radiation burn.
It was both necessary and sufficient.
When I was at school, this came out thalidomide and foetal limb damaged, which had never been this had never been seen before.
And thalidomide was advertised for drug you can safely take in pregnancy to have a good night's sleep.
That's how it was advertised on the medical journals.
You never got this condition of folk milia unless you've taken thalidomide.
It haven't been heard of before.
So the drug was necessary, but it wasn't sufficient because most women who took it, fortunately, didn't end up with a deformed child.
We know we know, in fact, from Hiroshima that a lot of people who were irradiated there developed leukaemia.
But we do know that actually people get leukaemia without being irradiated and not everybody.
And we can't get to me without being irradiated.
But if you'd had sufficient radiation, then the rates of leukaemia were very, very high.
So it was sufficient to cause the disease, but not necessary.
But that's not the world we live in. We live in a world of a mass flu vaccine and narcolepsy.
And I'm sure after a day on your course, have a good.
It is. Maybe many of you suffer from. No, actually, amazingly, no narcolepsy.
And actually,
this was something I was asked to I was sat on a panel about when they had this mass flu vaccine for H1N1 and whether it caused narcolepsy.
But of course, you know, most most people who develop narcolepsy have recently had a flu vaccine.
We knew narcolepsy existed. And most people who had the vaccine, who'd had the vaccine didn't get narcolepsy.
So, you know, most cases narcolepsy not immunised and most people did not get narcolepsy.
So in this case, that immunisation may be a cause but not be the cause.
And so my finally just very through, as we were in Oxford,
now we often look to understand causes with Bradford Hill and Bradford Hill, who is a medical statistician, who is the first person.
And to link smoking with lung cancer said, okay,
there's many cases like the the flu vaccine in narcolepsy where it's not obvious and causes are not sufficient unnecessary.
But let me give you some criteria. The effect is strong.
Not true. We know that what modern genetics is told is that genetic effects can be very real because their biological but often very weak.
The effect is consistent, makes it more likely actually not true because often we get inconsistent results because the research quality was not good.
The there is a specificity about that relationship between the perceived outcome and between a received and predicted outcome.
But actually that is often not the case. Many things like obesity or cigarette smoking, for example, associate with many outcomes.
What about dose response? Well, again, not true.
It may be that some things are threshold. Other evidence is coherent.
Well, it may be coherent. It may not be.
Is it biologically plausible? Is it biologically plausible that this vaccine was going to cause narcolepsy?
Actually, you know, probably not.
Probably not. And there are so many areas where we we we come up.
Is it biologically plausible that exercise may protect you against having a stroke?
I just pose that question to you. I'm sure they can find some evidence that makes it biologically plausible, but you might find evidence against it.
But what Bradford goes on to say is that ultimately we need experimental proof which relies on a randomised clinical trial.
And after attending this talk you probably think that is not a route that you will go down.
And on that happy note, I will take criticisms or even missiles thrown at me.
Thank you very much.