Today, I'm extraordinarily delighted to welcome and introduce So David Cox as our speaker,
and when we first started planning this summer Institute for Computational Social Science,
we were really keen to bring a statisticians perspective amongst our list of speakers.
And in particular, we were we were very and we found that there was essentially it was really important that we would have someone who would
talk about not just sort of the potential of big data and and the proliferation of data sources for social science research,
but also to talk about some of the the limitations and and and provide a sort of salient critique of it.
So then I thought, well, who better to do this than to have one of the world's most prominent statisticians?
So David Cox, come and talk to us about how we should think of statistical analysis in a world of big data.
I hope so. David is familiar to most of you already, but at least I guess his biggest claim to fame is perhaps the fortunate hazards model,
which he published in his 1972 paper, which is one of the most 100 most cited papers of all time.
And there are just many other fields of statistics that so David is contributed to and his work not just in academia,
but he's also worked in government funded research organisations. So.
So he has a vast experience that spans both, in a sense, industry and research organisations as well as working in academia.
So David has, of course, had a number of very important fellowships,
and he served as the president of a number of prestigious academic and academic organisations,
professional organisations such as the Vinoly Society, the Royal Statistical Society and the International Statistics Institute.
And in 2010, he received the the Copley Medal, the Royal Society's highest award.
And in 2016, the international prise in statistics. So he's had a number of accolades, too many for me to list in this very short introduction.
So all I have to do now is to introduce is to welcome him and thank him again for coming to come to coming, to talk, to talk with us today.
Well, thank you very much indeed. And I'm going to talk about variability and variability could mean various things.
It can mean systematic variability if we change this or make a policy alteration to that.
What are the consequences? What is what is the problem?
Very often the chain of consequences that will happen these days, the aid, investigation and careful description.
Then there's variability that is not predictable. The most obvious is survival time.
We have a group of homogeneous. In some sense, socially homogeneous individuals with a particular age and background and so forth.
How long before they die? This is not a predictable variable.
How can we describe the properties of such times?
How can we study them? How can we say what makes people survive a bit longer or a bit shorter and so forth?
How can these things they thought about in a systematic way?
And how can they be studied quantitatively in the way that?
Advances, understanding and aids communication that results.
Now I'm going to describe the techniques, the ideas that are involved in doing that.
A number of headings which are not to be taken too seriously and not to be taken in the order.
I mean, everything's to be taken seriously. Obviously, there are no no jokes in this lecture tour as yet.
The order is it is a planned one, but not to be taken too seriously, as I said,
and the first idea would be that of formulating the research question that you
want to investigate and doing that precisely enough if it is feasible to study.
Now this may sound of this, but then it's a fact of life indeed, that people,
even serious experienced investigators formulate their research questions.
They can do so in a way that's not essentially answerable.
You need to formulate your research questions not too rigidly, but increased in the form.
Does that are acceptable in principle of being answered by the data that you have or the data that you are likely to be able to obtain?
So the formulation of questions is a critical issue.
And I'm not suggesting this is a rigid thing done once in the beginning, the questions may evolve during the course of an investigation.
And this, incidentally, applies as much in pure mathematics as in more empirical studies asking the right question.
Pure mathematician, my beautiful piece of work resolve the issue of no conceivable interest to anybody.
Or they may do something that's focussed on something that has very strong consequences.
The choice was a research question is a critical one and may need revision from time to time.
And then closely associated with that. Obviously, the issue of design from one extreme, we have the design of experiments,
interventions in which the investigator has virtually total control over what's going on.
The control is never total, of course, but but it's substantial,
and the randomised clinical trial is an example of that going, which has a long history from other fields,
in particular, from agriculture and to some extent from physics work were carefully,
very, very carefully and meticulously timed experiments a crucial.
But some of the basic principles of the design, the investigations come out of agriculture and in particular,
one key question is, I think arise in many contexts.
Is it a good idea for your current investigation to think a one sharply focussed question and try and answer that?
Or is it better to ask a battery of questions that are interrelated on another?
And the great statistician or official who began his work in the nineteen twenties in an agricultural context, although his name is often prominent,
sort of is a geneticist,
and it began in agricultural contexts and set out in the mid 1920s the principles of the experimental design under four headings.
And one of them was the notion of asking nature. An interrelated series of questions.
Now that's in an experimental or interventionist context.
It's the same principle does apply more generally.
If you're using observational data, are you going to concentrate on your efforts on one sharply focussed question?
Or are you good? Excuse me. Are you going to try and study an interrelated series of things?
And then the third item in my list is what's called metrology, and that's a term used mainly by physicists to mean measurement.
How do we measure things or are measurements measuring what we think is not what are you going to do about it?
I like it all. And the most obvious example of that in the context,
I guess many of you are likely to be interested in what I certainly to personally is the issue of measuring quality of life,
perhaps in a medical context, but not totally. The key issue there is a key issue, at least, is how multidimensional they are.
You're going to say we must have a single number that measures the quality of an individual's lives.
Perceived as it is at the moment or how it might be in the future so that we can make suggestions.
This is particularly in perhaps the clinical context, but more broadly.
So it's suggestions and policies can be introduced that will in some sense maximise the quality of life and.
But it's a difficult issue.
I would dimensional I'm multi-dimensional, it's quality of life, and I'm being slightly unkind to economists, not all economists, but some economists.
But if. How can I push it if if your view of life is that it's a series of optimisations optimisation to optimise something,
the something has to be one dimensional, you can say optimise two things at the same time.
Because in general, at least there have to be collapsed into one.
So that forces the thinking about the quality of life to be the quality of a person's life.
That's to say now, or maybe estimations in the future is a single number, often expressed as a so-called quality,
quality, quality, quality, adjusted life year on the scale from the auction floor for means perfect life and so forth.
Three Main Street courses and then having gotten no light that you can.
Think about trying to make rational choices about it.
The counter to that is to say the quality of life is a multidimensional thing, and I measure it by some elaborate questionnaires.
A standard 50 50 item questionnaire is attempts to separate the quality of life into four dimensions.
And the size of that, of course, is can particularly work in patients.
Maybe, as I have patiently felt sorry for the clients, maybe that fell out 52 in ones, but you can't keep on doing that.
What I was asking them a simple question may be repeatable at an extreme, and that is a study which I think took place in, but I might be wrong.
The quality of life of patients suffering from.
Difficulties of walking on the single question was, could you put your socks on today unaided?
Yes or no? That gave. I think that is suffering from rheumatoid arthritis, lack of.
And one zero binary observation, an indication of the relevant aspect of the quality of this person's life.
And of course, that's a natural question to ask. And there's no likely to get a broadly truthful answer, which many of these self-assessed question.
That's a bit doubtful, but likely to get it to answer an impressive or repeatable.
So that's an example of metrology.
Principle of measuring things in a careful and appropriate way.
And of course, many of the advantages many, many of the developments, particularly in the medical sciences, come from physics.
The ability to measure through scanning devices various kinds of things of great subtlety,
starting really with X-rays 100 hundred years ago and developing from that amazing things that were previously not measurable at all.
So metrology, the ability to measure things in appropriate and suitable way is a key item now.
Oh, my office. So those jokes about question formulation crucial.
Designed to be a study, whether it's experimental interventionist,
whether it's observational or whether it's collecting appropriate data that already exists and deciding which,
if any of the data that might be available is appropriate for your study.
Let me move on to that, to various stages of analysis of what could divide the analysis of the data into all sorts of phases,
depending upon whether it's data that's already been fairly thoroughly analysed from different points of view well-understood
or whether it's the outcome of some totally new investigation in a field in which you have very little experience,
in which case you have to proceed much more. The multi-stage aspect becomes much more relevant.
The idea is that you in collecting the data, if you collected the data over a response or chose the data,
you must have some idea of how you're going to analyse it.
When you made the choice of the data that you're going to have to hold to that too rigidly, then it can be dangerous.
And then when the analysis is completed, what on earth is it? Or maybe what, what?
What does it, what does it suggest as a basis for action?
What new study should be set up and so forth? And so if we go through those phases?
All of them, you might say anything to do with statistics, their subject matter.
Of course, the subject matter questions. The point is that the ideas and principles involved that span.
A large range of subjects from the purest of physics to the most extreme of of application in the social sciences and everyday life.
But there are common features, and in a sense,
the theoretical statisticians job I count myself as a theoretical statistician, although my training is in mathematics.
I don't I'm not talking about mathematics. I'm talking about theory.
As a theoretical statistician, the object is to try and see some sort of unity of approach to all these problems,
whatever that feels, that will be helpful and constructive as well as getting involved in particular applications.
So that's the broad. See what I want to talk about.
Now going on to analysis the basic principles of analysis, there's one overriding principle that always applies and that's the first one,
and that's why you keep things as simple as you possibly can.
But no simpler. It's a real difficulty there.
It's got to be a face somehow. One aspect that one tends to think of this very much,
the statisticians province is the whole apparatus of significance test companies into those towns do interviews, et cetera, et cetera.
And the. There's 200 years of discussion of the quasi philosophical basis of all this,
and it's very interesting and it's an important part of what we do and what scientists do that these techniques,
which are basically principles of analysis and interpretation.
How secure is your interpretation of the analysis that you proposed?
How is that security to be expressed? The endless discussion and misunderstanding about that and its importance thing what we do?
But the more important aspect, I think in many respects is the fourth one, I've got the third one,
though, that I'm concerned with the lucid description of complex relationships.
By regression equations and other other techniques that will describe the systematic component of that
variability in is an enlightening the way it comfortable and enlightening means various things understandable,
obviously, but also linking the data that you have with other data with underlying security and so forth.
The depth and subtlety with which you can do that depends strongly on the field in which you are working and the nature of the data.
And there's an endless literature on this.
I'm referring in particular to a rather strange, you know, a diagnostic test of cosmology where Christianity Tanaka,
who some of you may know who works in Oxford, has edited an international series of papers on.
Modern statistical developments and in about 50 pages, so I think it's less than 50 pages.
That's a very excellent summary of some of the main developments in statistical methodology that are common in biomedical field.
Primary biomedical context primarily, but certainly by doubt, by no means only in that field.
So I recommend just this work.
No, I've spoken for how long? Fifteen minutes.
I've managed to avoid saying big data. Are you going to outlaw me?
Regard me as? Yes, seems to be the answer.
Well, big day, the thing about big data. Is, of course, in a sense, big data have been around a long time of.
But you couldn't analyse them. You might be examples I could give a personal application from one of the first that to work I've ever involved with.
If you're interested in the thickness of a textile yarn that's being woven, textile yarn is produced in batches of Oh.
Each single production would be perhaps 50 kilometres of yarn.
This can be measured every five mm. So let's say thickness, no problem.
You can produce paper trace of data of this so endlessly.
That's one batch of yarn. And then you have lots of others.
Well. So what you see, you can't in those days you could look at you could look at paper traces that showed these patterns,
or it might have been stress on an aircraft in flight.
It's actually a somewhat similar sets of data.
You could look at it, but you got the facilities to do any sort of numerical analysis were totally absent.
You could only take a sample. But it's not at all clear that the sample doesn't tell you that you really want to know.
It's by no means clear that just because the data are big, that's necessarily an insightful thing.
Does big data mean big amounts of information? Well, sometimes no doubt, but certainly not always.
And if, if, if the answer is, well, no, not vetted, this big doesn't make it reliable or good as a warning sign.
So the key issue is many of these big data context is the quality of the data, plus the question I put.
When is enough enough? Because there's no particular virtues, no, perhaps no great harm.
But if you take 10 or 100 times as much data as you really need in practise, you're likely to confuse yourself a bit.
But then perhaps it doesn't, doesn't matter. Then there's the more technical issue.
If you have an enormous amount of data, most measures of precision that you calculate from data that tell you how well you've done it,
estimating some feature that you're interested in. Most of these decrease, like one over the square root of this amount of data.
Not all, but most do. And if you have an enormous amount of data,
one over the square root of the enormous amount and the enormous number is something very, very, very small.
So the apparent precision of the conclusions can be very high.
But there are. I think quite powerful reasons.
And this is a largely unexplored scene, quite powerful reasons for thinking estimates of precision calculated from large amounts of data are often,
while often well, perhaps almost always a bit over overoptimistic and perhaps sometimes wildly optimistic.
They ignore features of the big data that by assuming most likely more technical systems, independent random variables that all that sort of stuff,
by assuming that they get at one of those square is a sample size effect, and that's misleading.
So the correct calculation of standard errors is.
I think a largely open question and.
To be tackled, of course, in the context of a particular case, at least to begin with, although there are some general principles involved.
Now, I haven't mentioned. I've just used the term big data.
I need to mention machine learning, data science.
So on and so forth. Each day goes by the seems to be a new branch coming largely from the computer science lexicon,
but not entirely to some extent from mathematical physics. It comes in.
You see him a new book, Information Informatics, for example.
I just write a book about in some metrics I'm getting it right.
And it's a new science on variability and data.
And some of that is, of course, much of it is driven by engineers coming to it from computer science background.
And some of it is undoubtedly very interesting and important. I'm not being dismissive about it.
I'm saying it has to somehow other things into a broader picture that looks at uncertainty
in the correct way and doesn't produce unstable and over in static conclusions.
Now, I'm so, so excited, so very brief word. About one example.
A situation where we have individuals really interested in how long they live.
Now this is a situation where they're going to be to sort of their ability presence, and we can't really escape this.
That is going to be some systematic variability as between social class medical backgrounds and their background, genetics and so forth.
It's going to be some systematic variation which will need capturing and explaining as concisely inadequately as a new cycle as we can.
But inevitably, it's going to be a large, unexplained component,
and we've got to be able to describe that unexplained component by graphically, numerically, perhaps by formulae.
And that leads into. Various highly developed parts of statistical analysis can keep going back.
Who into into the 19th century, but certainly to the early work of actuaries,
is obviously, obviously actuaries have much interest invest in people's duration of life.
Yeah. But also some physics and biological sciences and so forth.
And so we have to be able to describe the distribution of lifetime.
They have to be able to describe what might influence it systematically and but realise it'll be a large, unexplained component left over.
We've got to be able to describe that, summarise it in nice graphs or plot.
So summary statistics or whatever.
And so that's that's a relatively rich started in the actuarial science sciences, but in physics and in engineering,
my own background was just came from this case, interested in problems in engineering?
No. Being a bit more detail, many of these investigations I've talked about are essentially comes down to saying, how does this depend upon that?
But the pattern of dependence, of course, might be quite subtle and complex. That's what it's going to come down to.
And I've listed the three levels at which this problem can be studied.
The first is that of the textbook study of regression in some form, including logistic regression and survival regression and so on and so forth.
But how does one variable depend upon a modest number of individual observations with a modest number of individuals to study?
I'm not going to say what modesty is that? But that's you know what I mean, not so enormously large that we can't absorb the data as a whole.
And that's really essentially studied in statistical textbooks under the heading of regression and existing regression,
survival regression and so on and so forth. How does this depend upon that?
I put with caution because, of course, it's always easy to overinterpret any dependencies or to find.
Then the next step, which. Starting to get prominence, perhaps 40 or 50 years ago was when.
You have essentially the same situation. We've got the large numbers, not the large numbers we might talk about today,
but large numbers by the computational facilities that were available at the time.
So of. So it's financially viable as 500 individuals and things of this sort.
Multiple regression techniques is still available, but this is an issue we don't want to end up with,
typically don't want to end up with an equation that describes an outcome that we're interested in with 30 things on the other side of the equation,
we can't really. It's not an aid. It may be naive to a prediction in some context, but it's not a major understanding.
So we have to try and simplify that.
And there were various techniques in the time of which great regression was one of the most popular and which you essentially
said at least squares or other type of regression equation and penalise your equation by the magnitude of the association.
So if you have two people who buy something in the squares, you penalise a regression coefficients proportional to the square.
What that does is to shrink the. Small coefficients towards zero.
It leaves the big coefficients where they are pushing the small ones not to zero.
But it is. And then some years ago, I put it done.
Yes, 1996 rupture on end in Stanford had this ingenious idea that if you did this system,
that you penalised your regression, not by the sum of squares of the coefficients that you introduced,
but by the sum of the absolute that his so-called L1 norm, then some of the smaller coefficients were actually pushed right to zero.
Not greatly decreased, but still non-zero, but pushed absolutely to zero.
And that's because if you penalise a regression by an allowed that there is light that.
As compared with the relation like that, then the very small, though, is a heavily hit.
So putting it into pure honest formulation, putting a variable into an equation at all incurred a severe cost that would be one way of putting it.
And if you're on his method? Cleverly,
comes the last few have been quite widely used in some genetic studies recently where there may
be a very large number of exposure edibles and the USA can reduce these to a manageable number.
Now, my friend and co-worker added that it's an imperial college.
We've been looking at that from a slightly different point of view and which tips departure on this method is one answer.
It says although there are perhaps 10000 possible explanatory variables.
These five. As well as it's possible now, there may be a totally different five, the state imperceptibly better, slightly better,
but imperceptibly so and what hasn't and I have to do is to have a message that will find not just one model that states, but a set of simple models.
So we can say that the explanation might be this or might be that because there's an important point here.
If they object to the analysis, it's just to predict it is to take the input variables, genetic variables predict the outcome.
Understanding it doesn't matter if some other predictor will do it well, you've got it got the best predictor you can hear it is good luck.
But what it means is somebody else taking a slightly different set of days or a very different
set of related data might perfectly well get a totally and completely different answer.
That also would have fitted the first set of data almost as well.
So what hasn't not tried to do?
It's too ambitious a job to list all the possible models that equally well, but we wind up listing a number, a modest number.
And then it's an interpretive job to say which of these might make sense, which of these is capable of being reproduced?
I think my getting on for time. What time do you want me to stop you from?
Well, I was going to say something about I'm going to do that to move on to something else.
No, I'll move on to something that. This is now a shift back to general remarks.
There are two key ideas connected with the analysis of data.
Not not the only ideas, of course, but two key ones of which the first is are the conclusions generalisable?
Do we have some have we find something that's totally specific to the set of data,
maybe be nationalising or at least finding something more in some sense, more a broader interest, more generalisable.
And there's that issue and then and this may not concern.
LG McCue working so much at the opposite extreme of that is specificity.
I know for that think a of the doctor treat a clinician treating patients.
Here's a patient in front of him or her, and this patient has certain features.
And there are two possible things that the doctor may recommend.
I will be now there have been a let's suppose there's been a beautifully organised, randomised trial.
It's shown that AI is significantly better to be.
That might say of a group of patients, 70 percent do well on a 70 percent do well, I mean, does that mean that every patient does better on a day?
No. And the question facing the clinician.
Is what is best for my patient?
What is the evidence that although I in the randomised trial clearly wins over me, what's the evidence that my patient will do better than me?
Or in a quite different context, if this policy change is introduced, this is being shown in a randomised trial,
essentially in a large study to be based on the average beneficial, will it be beneficial in this particular case that I'm interested in?
And this is the issue of specificity. And in a certain sense, generalisability and specificity at opposite ends of the Pole.
There's generalisability is taking a data outwards into a new context.
The specificity is narrowing it down, not generalising it, but saying saying,
Does it apply here to my particular policy choice or my particular position?
Now. I put the the conditions that.
Enable you to make a generalisation from a particular study, and I think these are fair.
I'm I have to admit I'm thinking of primarily a randomised clinical of clinical trials, but I think the remarks apply much more broader than that.
These are the. Issues, subject matter, understanding that you understand the psychological, social,
biological, physical processes that are going on underneath what you what you see.
You have some deeper understanding.
The size of this age, big effect, particularly is expressed as ratios are much more likely to be stable than smaller effects.
And this was shown in the reference I've given this to there.
It was published in 1959 and reprinted in 2009.
She was given two dates. Connellsville was this was a period when.
Surprisingly. Bradford here and in London had found that.
Smoking seemed to be a cause of lung cancer.
They had expected to find when they did this study that air pollution was,
which was horrific in London in those days, that air pollution was the cause of lung cancer.
But they found that smoking could seem to be the only cause.
And over the next few years, a number of other studies tended to confirm that.
But the three leading statisticians of that time were all sceptical.
All right, Fisher considered that. The effect had about had a genetic base that I mentioned previously, that is kind interest in genetics.
He thought that what was going on was genetic code, which of course has a certain modern nuance to it and.
Name in Berkeley, who could be infectious and that they had had a long and highly acrimonious and destructive conversation
with one another name and also because it wasn't the study of it wasn't verified by a randomised trial.
He was sceptical. And the leading medical statistician of the time, the clinician, Judge Buxton,
who worked at the Mayo Clinic and somewhere in the centre of the United States, backs and also is sceptical because he made the point.
That is the smoking and lung effect on lung cancer is the difference between one small effect and a very, very small effect.
But if you look at the effect of smoking on heart disease, they're kind, superficial effect.
It's the difference between two big numbers and the number of people apparently affected as a difference of rates was much higher.
Now. What Collins did was to analyse all the work that's been done up to that point.
And yes, I'm not going to go into the details of what he showed,
but is the essence of his argument was that there were four totally different lines of argument, all pointing in the same direction.
Laboratory studies on animals, laboratory studies on humans, certain types of trial and and ultimately clinical trials.
And his argument was essentially that while Juan might be able to buy one of these away,
the fact that coming at the problem from four different directions you've got to the same answer was overwhelming,
and it had some more mathematical arguments behind it as well,
who seemed to to do with rights rather than differences, ratios of rights rather than differences.
And that really won the day and confidence.
The paper was accepted by the U.S.
That being said, the general, I suppose, as as as as a definitive statement about the causal relation of smoking and lung cancer.
But an essence of that was that he was working with.
The size of the effect was important. If you can show that you're what you're looking for is repeatable in many environments.
Multiple studies with the ICE, and that's another route to the same conclusion.
In principle? Random sampling of your target population.
Give you the security, you want a security. My sense is virtually always nothing.
Very few serious studies, certainly on the biomedical side, could possibly be justified as a random sample of the targeted population.
Specificity, I'm not going to say too much about it because it's possibly not have so much interest,
but it's to do with the circumstances, as I said, in which.
The conclusion from some broad study, you can say, yes, this really applies here to my particular decision or policy recommendation, or it might be.
And in many ways, the considerations are the same as the generalisability.
Now. It's a dreaded word I have very carefully, I think I've managed to avoid mentioning, and that's causality.
And I think I might decide that I'm going to run out of time and.
When I started statistical work, causality was a forbidden word,
it was philosophy and the serious scientists, mathematicians and physicists didn't want to do that.
So nobody said it quite as good. But uh, and then.
I was going to say quite recently, it was actually about 30 or 40 years ago that fairly suddenly changed when.
It was an important paper was read to the Ross Listicles of society in which it
was pointed out a massive effort had gone into randomised trials clinical trials.
It was time to turn attention to issues that could only be addressed observational.
By observational methods and this data and a change of emphasis,
the formulation of guidelines that would lead to causality and to a massive current literature.
Some of it coming into the field from the computer science, more than statistics.
I'm not going to try and discuss that because it's a lecture more than a lecture on this by somebody else, not by me.
I say three possible definitions.
The one that really the two that really matter is that you have evidence that if this had been different from how it is other things being equal,
then the outcome would have been different. Then you can say this at the beginning is a cause of that.
If you have solid evidence that had this been different from that,
other things being remaining the same, then you have to be very careful what that means.
Then the outcome would have been different.
A third that's the second definition of causality, the third definition is the first is that one, plus some understanding of process.
Why is this going on? Look what biological, social or whatever economic mechanism is in process that produces the change we're talking about.
And so I conclude with. What I suppose I have to confess is the limitations of statistical certainty I tried to present,
I said very little about big data, machine learning and so forth.
Not to be dismissive about it, but because I want to emphasise the aspects of what I see is specifically statistical.
The literature is overwhelmingly about probability probabilistic models that they analyse and fascinating stuff that goes of the mind,
you want to go into that. It's a tiny proportion of what statisticians do, even theoretical statisticians like me.
The formal theory is a fairly important.
It's important, obviously, but it's the intellectual underpinning, but it's not what one primary does.
What it does do is to put an emphasis on trying to get individual studies whose interpretation is as secure as can be.
But not in a way, a limitation, because although it's a good thing clearly to have studies that have got secure individual interpretations.
Nobody can be nobody can not want that.
But to see that as the sole objective is too much to narrow, and the very often the biggest advances come from saying this,
this evidence from here, there's this evidence and there is this evidence and put those together in some way.
They may get an enlightened view and confuse work that I mentioned a few moments ago, in a sense, as an example of that, where he took the very,
very different types of study that could never have been analysed collectively because they were totally different lengths,
different outcomes and so forth. But it is the possible impact of pushing these things all together that made.
So the strong interpretation. Well, I put some references up.
The notes, but not to the think don't. Well, thank you very much, sir.
Thank you. Thanks, Ken.
Thank you very much, David, for these general principles on just how to do research, I think.
Not just not just statistical research, but any project could benefit from, I think, having the kind of outline that you've presented.
We have time now for some questions. So just raise your hands and I can bring the microphone so that we have the voice recorded as well.
Oh, yes, hi, thank you very much for your lecture, it was very interesting.
So I had one question, so there's this one concern that I had, which I discussed with the actually three days before,
which is that I feel like an incremental divide between machine learning and
statistics like machine learning is being developed as it is a separate branch,
completely independent. And now people who are being taught machine learning are not necessarily being taught statistics in the traditional way,
but only going directly to machine learning. And so I wanted to know your thoughts on whether we should be striving for a
combination of strategies between statistics and machine learning for analysing data.
Or is it fine to separate both branches because you're doing just the same?
It's. Well, separations won't be a bad idea.
There is a different explanation, I think comes machine learning is largely come from an engineering background.
The stuff is the most traditional statistical thinking of the mathematical and natural science background.
And. I see machine learning.
The work on machine learning, I see it some of it's very impressive.
And if your object is very short term prediction in a mechanical fashion, it may well be the best, quickest and safest way to go.
Will it lead you to understand the world better? And that I'm less convinced about.
So the reason the strikes by I mean, you see, the work has is a mine that I went through so hastily at the end.
That's the kind of machine learning.
But but it's it's focussed on trying to say it might be that it might be that might be that all these gives a perfectly good fits for the various.
Is there extra evidence somewhere that will tell you which is the right one or what committee includes some from what we see?
And I don't see that there is in the machine learning literature.
And also, so I mean, some of.
Some of the neural net makes very, very clever, but because they're looking for extremely irregular brain dependencies in complex systems, whereas.
Certainly, my feeling would be to start. Start with the simple and make it more complicated if you absolutely have to.
So it was a difference of philosophy that. I may be being unfair to the machine,
but my impression is if they find a new net so that that describes a dependency that very nicely, they don't attempt to say why.
What does this suggest, what are the weaknesses of this, will this apply in some new, slightly different situation?
It don't seem to. I've never seen this, but I'm not trying to be.
Yes. I was just wondering if you could talk a little bit more about the issue of causality in the realm of human action and choice, because I mean,
recently I've been thinking a lot that a lot of our model of causality are really tailored for,
you know, this sort of exogenous shock that happened to people.
And I was thinking that, I mean, in the social world, you know, people make decisions and have, you know, some degrees of free wills.
And I have a lot of troubles bringing really the sort of causal framework into the social aspect.
And if I can just give an example that was thinking the other day, I'm not entirely sure it's the correct example,
but I think it it might touch upon some some of the issues at play.
So for instance, if I shoot Félix and kill him, I caused fairly deaths.
But if Nicola would pull a gun in my head and force me to kill Felix, who kills Felix, did I kill fairly?
So did Nicolo kill Felix? And I feel that in the social world,
we often have these sort of indirect chain that cause triggers and that it's it's unclear what caused the Y and what causing means.
Thanks. Well, one.
I share your considerable caution about using the word at all.
And even so, the various levels because. If you take those out in the sense,
we have reasonable evidence that this group of people had they been randomised to that
treatment rather than this treatment would have done better on the whole better.
We may it may be possible to guess that it's not totally unreasonable to say,
then that this is the treatment, whatever it may be, has caused the outcome, on the other hand.
It's certainly not, and it's a causal explanation at one level.
If you say what was going on in people's minds or what was the biochemistry underlying what they did and so forth.
That's another level. And if you pursue that idea, of course, everything, then.
Goes back to quantum mechanics, molecules.
And if you look at the foundations of quantum mechanics, you find that something goes.
It's a fantastic, beautiful, extraordinary series.
Great. One of the great intellectual triumphs of the 20th century.
But I think I'm right still, if you look. It doesn't rest on its science, doesn't rest on its foundations.
I think is one of the. One of the conclusions, the foundations and the and the superstructure, so to speak,
of mutually supportive now in some situations, the support is almost all in one direction.
But I think it's always an element. The success of one of the reasons the quantum mechanics is so fantastically
successful is that it's predicted extraordinary phenomena with high accuracy.
Not that it's kind of compelling intellectual structure.
Thank you very much for the talk,
and I'd like to especially what you just mentioned that at some point we seemed to hop onto the trains of quantum mechanics all the way in
the lowest layer and then somewhere we just start to track things and we see perhaps predictions of associations with the thought in mind.
I myself was very mind blown when I first got told that a logistic regression works because
the error term is an extreme type value one distribution other than a normal distribution,
because in mathematics, there's some mathematical arguments. Yeah.
So trying to bridge the gap between the machine learning conjecture and the statistics,
which is very heavily indebted to assumptions on the error term.
I was wondering whether you see potential to retain the standard functional forms we've been using
to allow new forms of the arguments is so fundamental to the way classical statistics works.
And what your thoughts are on that, because so what's so fundamental? Let's say you said something is fundamental to her classical statistical work.
Well, in the linear regression framework, I would say the normal distribution on the error term.
But I was wondering how you see different types of because obviously every time we do an or less through our data programming or something like that,
we usually assume this normal distribution. But obviously, we could do different things right now.
If provisionally assume normal distribution, you then get your hands and then you might say either by subsidiary looking at the data or in general,
I'd say, have I made any assumptions in this that might critically affect my answer?
And if if you thought it would be there with the answer,
depended critically on the normality of the errors in the regression and will it have to do something about it?
Quite what I don't know, but something. I mean, it's the.
And I don't see any of these things as stuck in stone, so to speak with which you see worth to try something like a fantastical distribution,
which is so modal and has this kind of the conjecture I was thinking about this is.
Well, if you find a bimodal distribution, you would surely want to know why.
What what are what are the identifiers of of uh, that would be an instance where an intermediate stage of analysis might lead you to revise.
At least some aspects of the question if you started with, and I did emphasise at the beginning that this sequence.
Stages. Is a provisional.
Saying. At any point. The worse, you know, the day before your paper is to go to the journalism and say, Oh my goodness, a better look at that.
You better look at it. I'm understanding.
I don't mean to sound frivolous about it at all. But these men.
All the methods, whether AI, machine learning or.
Driven or more formally statistically based, they're all based on it, they're all they all lie on assumptions.
Those assumptions may be likely to be a bit wrong.
They may be very wrong. That may be so wrong that they're misleading.
And then you have to try and find that in some way or other discover that and rethink.
It's a little flick. I mean, this is the limitation of of textbooks, there's textbooks, particularly textbooks.
Some statistics make it sound also formalised.
That's because the mathematicians have got hold of it in their life and in a way a very good
thing to have very formalised is it's not a very good thing to take them too seriously.
I take them seriously, but not too seriously.
You understand what I mean. Thank you, David.
It's a pleasure to have the opportunity to listen to you,
I wonder if you could comment on the recent trend in various strands of social science towards preregistration,
something with which I have some sympathy with. I'm sorry, but I'm getting a bit death.
I wondered if you could comment on the recent need for registration in the social sciences,
but it's the same thing registering the hypotheses that you wish to test and the expectations you have,
and whether this goes against any of the principles you outlined that have to do with maybe, maybe reassessing or provisional expectations.
Well, to be able to say at the beginning of this study, I have this purpose in mind and I'm going to do this and this.
That's a good thing to say after a year or five years or 10 years or 15 years.
I'm solely stuck with what I said 15 five years ago is an absolute disaster and a travesty
really of what statistical thinking or what scientific thinking really is about it has to be.
So you get the good elements out of these things that but.
Pre-registering, I mean, that people looking in for a research grant, I suppose, you know, thinking they'll probably say what, how?
How are they going to analyse that data, what possibly they might find great?
Of course they should do this stuff. They should be totally tied to it.
Absolutely no. And the longer the time period and the more rapidly involved involving the subject, the more disastrous that would be.
It's a bit like saying wasn't there's that appalling amount of nonsense being written in the statistical literature about of significance tests,
which is based on my own total misunderstanding of what lie about, but we don't go into that so much with the people having to say in advance.
This must be significant and such and such a level before I can report my result.
So they get paid equal 0.05 one disaster hit the five percent level.
And I've actually heard a serious biologist do this.
People point out for nine. All right, we can publish. This is absolute rubbish, of course.
Total nonsense, total misunderstanding of the purpose of these these techniques.
Thank thank you very much. I really enjoyed the talk,
and I would be really interested to know what you think about the possibility to use non probability samples to draw conclusions about certain things,
though in this summer school, we, for example,
heard of this opportunity that the internet or big data provides that if you're interested in a question that, for example,
you don't have data for from a representative survey or maybe a peculiar group of people,
then we could easily go on the internet and collect the data.
Obviously, this wouldn't have the properties of a probabilistic sample that we are used to work with,
and then we learn that there is an opportunity to sort of make the data or stratify and then draw conclusions,
maybe about the proportion of individuals that would vote one way or another way.
But I'm more interested in the relationships between variables.
In the same way, as you mentioned, what how is this thing related to my outcome of interest?
So this was a long winded question.
But basically my question is, do you think that we would be or will be able to use this kind of non probability samples
to draw similar conclusions as to what we are used to based on regression type of analysis?
So what do you think about this in general? Well, probably the samples of of.
A beautiful, fine idea if your response rate is well above the 95, 98 percent.
Fantastic. If your response rate is 40 percent or 50 percent, so yes, it's a probability basis is no formal.
You have to use more delicate methods of analysis to detect well.
Five. How is your sample and not representative of the target population and make some does it affect the answer that you're getting?
Maybe, maybe that has to be explored. But I did have it on one of my slides.
I was going to say outside search in industrial and physical companies,
some context in which sampling is done according to strict rules and implemented 100 percent.
And certain kinds of audit account auditing, for example.
An audit goes to a company, looks it's a large number of accounts that have to be checked,
and he takes a sample is in accordance with clearly doing his job properly with clearly specified sampling rules.
That's that's great. It's very atypical of most, most research, social research situations such as I've seen the.
You have your data. You have the population who have contributed to the data.
What's the clue about? Is that a typical holiday, something that needs alarms?
Maybe not, if you're lucky. I don't think the theory of something.
Is there any help to you? Once the non-response rate dropped below, I'm not going to mention the number, you know, the number of somewhere.
So I to take my take some liberties and ask you that if I may,
one of the things that I thought so you dwell on this and I thought it was very interesting that you
have to deal with uncertainty and think hard about uncertainty in a context where we have big data,
but we also have quality concerns.
So one of the, for example, challenges that we've talked about in the Summer Institute is that often that we have big data sources coming from some,
some from the internet that that give us a lot of numbers.
But at the same time, we don't always understand the data generating process because, you know,
they're not they're not custom made for research in the same way that any survey would be where the researchers designed the instrument and so on.
So in this context of big but generally dubious or unclear or black box,
how do we think about the the measurement of how do we what are the general principles we can adopt to think about uncertainty?
Because you commented on how, of course, standard errors don't work the same way?
What do we do then? Do we do we just use small samples or or should we just a lot of notice,
as some people in this room are very interested in moving into the Bayesian paradigm to address this uncertainty issue?
So I'm interested in hearing your thoughts on how do we think about uncertainty
in the context of big and uncertain and uncertain in the context of quality?
Yeah. And.
And of course. I don't quite know what to say to that.
And. Big day.
Well, first of all, maybe it may be possible to say some parts of this data on reasonably secure and others are not,
and then one try and isolate that or it maybe it's the data has come from many sources that some of the sources are suspect.
I know there are many things that you might do to check on the quality of the data that I don't think in the last analysis,
if the data is MIS is seriously misleading and.
Uncollectible.
The place for it is the wastepaper basket or the modern equivalent of the wastepaper basket not taking out busy people's time analysing it.
And of course, it's a very difficult judgement there as to when that where that threshold is.
I am very sceptical of the idea that this magic number crunching to correct serious biases in data.
That's not a very helpful answer on the first. But.
Divide the data into rational sections. Analyse the section separately if you find conclusions.
Broadly stable across two sections. That's reassuring.
They've got a blast of sitting everything or some stability the.
In other words, don't really rely on.
Assessments of precision that come directly from the stability of the conclusions across what should be comparable situations,
not by internal calculations. In general, the calculations of precision you'd find in the textbooks that lead to the Sigma 10 type of standard,
that these are all based on the independence of individuals within the groups,
it's probably that it's the most critical aspect of most likely to be wrong.
But if you find a meeting apparently meaningful relation, it's stable across.
Different parts of the UK or different countries within Europe or whatever it might be,
if there's some external stability in the conclusions, then of course that that's it.
They may all be wrong, but it's much, much, much less likely.
It's not very helpful, but that good quality data is the message.
Kind of building on that, increasingly a lot of the data that is generated is obviously proprietary to companies,
and the way that we created on outside of the company is mediated by the algorithms that are inside of something.
And so I think at this particular moment,
a lot of us are facing the question of whether we stay outside in academia to try and study and deal with the fact that we can't get into this,
the black boxes to know what the algorithms are,
what the even just the decisions behind the mediation and socialites on these online platforms is versus going
into the companies themselves and working as scholars there where intellectual freedom is often pointed toward.
The aims of profitability or use for the company, there's not as much intellectual freedom, I guess I would say.
I don't know if you have any thoughts on that. But in the aim of trying to get at the good data needing to get inside and therefore leave out the.
Well, that's rather outside my field, I'm afraid the pharmaceutical industry.
The peculiar reputation in many ways, but on the whole.
It seems to me that I have been supportive of good quality research, at least to the probably the bigger companies have anyway.
I live in that's a model of how. My perceived.
Because it's a other than the research councils to mount a clinical trial of the size, it's not so it's necessary to get.
Pretty clear results requires a massive. We saw both of time and I mean,
only my own experience of that is working for 10 years with the management committee of a trial run by a pharmaceutical industry.
And I would say that the. Treatments of the problem seem to be scrupulously honest to me.
I think he said, you mean it's outside of your area to answer, but that's actually a very interesting answer.
I don't think I was thinking the direction of pharmaceuticals, but more in the social media platforms and.
But the answer? It provides a question for all of us in terms of whether the the more social media platforms need to take as seriously,
given the impact that they can have on society, need to take the science.
Is the science more seriously in the way that pharmaceutical companies have to run trials on their drugs to see the effects?
So anyways, the answer was still that since.
So I'm going to ask another one then, because I am interested, so an interesting theme that I thought throughout your talk was sort of there was
implicitly this kind of dichotomy or tension between prediction and explanation.
And and this is a recurring theme also both in the Summer Institute,
but also in the context of wider discussions about statistics versus machine learning.
And and I'm I'm sort of I find this a bit sometimes misleading the dichotomy dichotomies in this way,
because I do think that the purpose of a lot of theorising is implicitly also a prediction.
And so I don't think that it's as sort of separate in the way that often it is made out to be in
this in the machine learning versus sort of classical theoretical or domain specific paradigms.
And and I wondered what what you thought about this? There's difference the explanation versus prediction distinction that is being made and whether
you think that it matters as neatly onto the sort of statistics machine learning divide as well.
Now, that's a very, very important question. And.
I think my reaction would be prediction without understanding is a perilous business.
You may get a beautiful prediction formula that applies.
In the very, very short term or in this particular context.
With unless you understand, unless either you see only essential stability of this prediction formula over quite a wide range of
conditions or you understand the sociology or the physics or whatever might be behind the formula.
It's perilous to predicting the future. And this is seems to me the trouble with much short term economic forecasting.
People can get very. Sometimes very good short term predictions, but.
Then the prediction model changes. So I think.
Even if. Well, the two lots one could take, one can say really, as academics are, the object is to understand the world.
Hopefully make it a better place if it means something totally by.
If you take the other view and say our object is to produce. Good prediction, I did produce good prediction.
Well, in the longer term, my understanding. Not purely by empirical.
Thinking. I think I hope so.
Would you then say that the the the, uh, the machine learning paradigm is then intrinsically geared only for the short term?
On the face of it. I mean, I wouldn't want to accuse.
I'm sure there are people in the machine learning field, a little more broader and more subtle in their thinking than that.
But the machine learning, as I feel, as I look at it from the outside, does seem very,
very focussed on setting this particular set of data, even by highly contrived methods.
The successful methods. And you say this relates to the discussion, as I glossed over the distinction between Cuba, Iran is.
Very influential work on the LSU. Which is a political prediction.
He wasn't concerned to say, Oh, here's here's the formula that predicts beautifully.
Isn't it different for me? It does almost the same job in a totally different interpretation.
I mean, I haven't discussed it with him personally, but. But I don't sense in his writing.
An effort to do that, I may be the answer to.
Compared with signs to say yes, we can predict is a predictor, but there are alternative explanations or,
as it turns out, in the example that Heather and I looked at, some of the features were remarkably stable.
We had to note we ended up with a number of prediction equations that suggested.
Virtually identically, well. But there is a substantial common element of them, so we could say, well,
whatever the final collection we use, we better have this, this and this in there.
And that's a stable that's an element of stability. It seems to me important.
It would say that you see your work with with Heather Batty as as in response to try and
sometimes bridge this prediction explanation because your your the way I understand
that paper is that it's also in some sense trying to see to trying to let the domain
expert adjudicate between different models to see what might be a plausible.
Story to tell,
but so do you do you see that paper is as trying to say that we shouldn't necessarily be thinking of prediction and explanation as a separate goals?
Well, I've always I think they were primarily interested in the explanation.
And misguided predictions. Interesting.
Side issue to that important, but not the prime goal, know how the work with Heather started on, how does work start?
I don't know. So it's after all those years ago, started three years ago and it's a remote first.
Yeah, but just I just wanted to ask you if you think that there should be because I don't I'm not sure I have seen this.
Maybe you know this, but do you think there's a field which should move on the intersection between machine learning and classical statistics?
So let's generate more strategies on combining both.
So, for example, using predictions from machine learning algorithms and more inferential statistics or
just generating sort of like hybrid models that takes to both the best of both worlds.
Well, I I don't quite see it like that. I think we all have to do the best we can.
We all have to do it as broad as we can and recognise the people in related fields to us doing similar things.
We're interested enough and helpful so forth. I regard the division in 10 years time or 20 years time, people.
What's the difference? There may be no difference. I don't think is at the moment, the differences are quite strong, and in fact, in Oxford,
particularly strong because the machine learning scientists statistics in the
statistics department is much bigger than the statistics of statistics thought.
But it's an accident of time. I these things, the emphasis within big statistics is a very broad field ranging from the very, very,
very mathematical to purely descriptive and subject, not purely subject matter based and interests in it, and move even over my lifetime.
They've evolved and changed, some with some periods. This is a focus of most interest.
Sometimes that. A sign of life.
If we don't have any further question, do we OK, we have one more.
Thank you very much for this very nice talk. It's basically actually follow up of what he was asking.
But I wouldn't like to not so much to focus on the distinction between explanation and prediction or the the the computational
field and statistics of maybe what the how do you think your general rules extends to the computational worlds?
For example, is parsimonious ness of models still so important or actually not anymore,
because they only serve us better, comprehend the models or something?
How is it? How important is it still to be super explicit about your uncertainty if you just all you look for is like the best model?
So you are very general, very nice goals for statistics.
Where do you see the commonalities through to the new world?
Oh, that's a very difficult question. They have brutal answers I never think of.
I just I, I just so many working scientists.
One presses on and tries to do something useful when thinking about these general issues is is fantastic,
and that's why it's so good to come and talk to you, hear what you have to say about these things and.
The important. Out of out of diversity comes strength, somebody must have said that and.
The statistical world has changed enormously in my lifetime.
It's very different in different countries and Western Europe.
It's a much more mathematical and it's here. Most British statisticians.
Have a strong interest in monument field of the application. That's not always the case.
It's certainly not the case in the United States, for example.
So there are there are national differences of emphasis and style as between very basic principles,
applying those principles to complicated problems, being involved in real applications and so on.
And these or cede one another. That's a terribly useless answer.
But I don't I can't give a better one. And that's.
All right, I think that takes us to time. I want to thank David once again for a very thought-provoking talk.
And for his comments on, I think, a wide range of issues that we've been discussing throughout the Summer Institute.
Yes. Thank you, David. Thank you.