Hello, everybody. Thank you very much for inviting me. And so this is a sort of I've summarised some of the work we've done,
but look particularly at whether these tools can be used and how they can be implemented in some of the validation work we've done.
So the, the sort of more applied end of the of the picture and these these tools, I mean,
just that, you know, I mean, I was involved in developing them with Tom Fanshawe as well.
And Maria, who's here as well, has been working on them with us as well.
So it's a collaborative effort between psychiatry and primary care sciences.
So I'll start off by telling you a little bit about the tools that I'll tell you a little
bit about some of the challenges we've had with validating them and implementing them.
And that's basically it. And then there's some reflections on all of that.
But I'll sort of introduce a little bit.
So some of that might be overlapping with what you've heard, but I just thought would be useful and show you some that they're serious.
The tools that we've tried to develop for adverse outcomes in mental health and criminal justice and a
lot of a lot of them overlap the mental health and justice adverse outcomes overlap and you'll see why.
What I mean by that and by adverse outcomes are usually meaning things like perpetration
of violence or in some of the tools we're looking at suicide rates as completed suicide.
So these are longer term, very, very serious obviously adverse outcomes.
And one of the things we've been trying to do with these tools is develop them using the best methods.
So large data sets. And where we've gone to for the data is Sweden.
And we've gone to Sweden because Sweden has linked datasets across various different domains,
including health, justice, mortality, but also sociodemographic information, family information.
And it's probably the largest national database that does this as only few countries are able to link across all these different databases.
And Sweden has about a population of about 10 million.
Denmark does it, but it has about half the population in Finland also, and there's one or two Australian states that it's possible to do it.
But again, it doesn't reach the size and I actually think in Sweden that they're very high
quality so that they're very linked because they use a ten digit personal ID number,
which is unique to every resident. And so you don't have these problems that you get, say,
in the UK where you're having to sort of guess someone's name and date of birth and the names and Romeo's date of birth is a bit wrong.
It's sort of to probability matching. You don't get that problem.
So there's very little missing data. So it's very high quality.
And then, I mean, in terms of the methods you've been talking about the last couple of days, I assume, but, you know,
trying to put everything down and specify what we do and use a range of performance measures and
use predictors that are easy to score that require additional interviews as scalable predictors.
And part of the reason for doing that, of course, is that in order to pull predictive predictors out of databases,
you need to use things which are recorded firstly and routinely recorded secondly.
And that means that they tend to be straightforward, relatively speaking.
So it's not for instance, in psychiatry you can ask about a symptom or you can ask, let's say, a predictor of a diagnosis.
And in large scale databases, the symptoms are very inaccurately recorded.
You don't know if they're not recorded, if they're present, because they haven't been asked.
And you don't know even when they're recorded, if it's what threshold people have used for diagnosis,
this sort of agreement, there are classification of diseases, something called the ICD ten, and that's 11.
And there's a you know, there's a consensus diagnosis to be worked out from decades.
And so they're a little bit easier to to score a bit more reliable.
Iterator and rewriter reliability is much better for these type of things.
And then for the tools that we've developed, we try to internally validate them and then externally validate where possible.
So you'll see some of that coming up. And then we've translated them into risk calculators.
And the idea behind these calculators is a bit like Q Risk or Framingham, that they're simple, scalable,
available on the Internet, free to use technical training, and you can see all the coefficients, how they're scored as well.
So you can see how they work. So that's published.
So here are some of the ones we've developed. So the start of results with an ox rack.
So this move is a tool for people with severe mental illness.
With mental illness in this case means people who usually have been to hospital at some point in their life.
It's a diagnosis of what they call schizophrenia spectrum is. So it is an bipolar disorder.
So they're they're on the more extreme end in terms of severity, usually.
And so we developed a tool to look at perpetration of violence in people with severe mental illness.
Any points in the patient pathway? And we did that partly because this is a problem.
I mean, about 10% of people, if you look over a five year period, have showed quite serious violence.
And that's a it's higher than what you would expect for people of similar age and gender in the general population.
Still a minority of patients, but enough to be an important outcome to prevent in this patient population.
And we know it's modifiable. So that's good reason to come up with some method to predict it so you can prevent it and release prisoners.
We developed two clocks REC and this is the first tool we developed and then it's at the end of the prison sentence.
And there again, the idea was that this is a risky period in the first 1 to 2 years after leaving prison for repeat offending.
And a lot of that repeat offending is driven by modifiable factors such as drug and alcohol problems and also in some cases, mental health problems.
So we sought a tool that could predict age and then events, and then ideally the link to an intervention could again help prevent reoffending risk.
And reoffending risk is is much too high in people who leave prison.
So if you look around the world, recidivism rates are sort of in the range of 40 to 60% in two years is quite high, absolute rates reoffending.
And then we have some other tools for specific populations.
And at the bottom, you see there's a tool for suicide risk assessment,
which of course is a very rare outcome in even people's mental illness, fortunately.
But it's nevertheless a very important part of clinical work to evaluate the risk and then try and manage it in some way.
And, and we've just worked very has been involved in quite extensively as is another calculator
to look at suicide risk in people who self-harmed because again that's an important
population with the self-harming rates are very relatively high and they represent a
high risk population for suicides in the subsequent year or so after self-harming.
So if I just look at one of the tools as you see how they've been developed, this is all smooth.
So what we did is we took the whole country and we identified a sample of people geographically, 58,000 of people with severe mental illness.
And then we identified another part of the country which we can use to validate what we developed.
And we when we derive the the tool, we we came up with some candidate risk factors,
some that we thought should be in the tool because of face visibility. In some of that, we weren't sure about that.
We then tested in multivariable models and then we we set up a high risk or elevated risk category at 5%.
So this is a one year risk of violent offending in the population.
And all that was prespecified in the protocol that we published.
And then we tried to present a range of performance measures which I think you've heard about today, measures discrimination, but also calibration.
So the risk factors, the candidate risk factors, this is probably less interesting to you,
but it gives you a sense of the approach we took as we looked at different domains such as demographic, clinical, genetic,
which is just previous violent crime,
and then family factors which are probably proxies for genetic risk or early other early risk factors and outcomes,
violent crime and those in order to select the predictors.
We looked at the literature quite carefully and we looked at those factors that we also thought were possible to extract reliably.
And the outcome, as you can see, is quite a sort of rare outcome actually even in this population.
So it's quite a high threshold too, for that particular outcome.
And so these are some of the risk factors we looked at and here's the prevalence of them.
So the ones in blue we thought should go in the model based on previous literature and face sort of the same ones in number two and three.
We were not sure about two. We thought probably go in the model that we wanted to test and then three we're not sure about at all.
And then when we came to test some of the models, the a couple of the Greens survived and a couple of read survived.
That's quite interesting.
And the externalisation, so in the in the geographical region, which was different to the development region, the two performed quite well.
I mean, the omnibus measure is quite high point 89, so it's suspiciously high school.
And then the here are the other metrics. I mean, the sensitivity is quite important from a population perspective.
And I think the way we write it up also we highlighted the high NPV because it could then act as a screen out too,
because a lot of services want to screen out the large number of referrals they get.
And this is what this could help. Calibration is quite good, so the top one is in the derivation.
One below is in the validation samples. It doesn't address any systematic problem, particularly here's the online calculator.
I mean, it's online, it's got these dropdown menus on my screen to use.
But I mean, this is not actually the online version of my SLAVOJ,
but you just sort of click on that and you can get and you pull that along and you get a different age and then you pull down.
Some of them you can score is unknown and then it gives you a range of probability scores and you end up with a probability score.
And then the pre specified category, elevated or not, was 5%.
So you could see if it's above that. And then we also have some visuals so you can see whether you can visualise it in table or in boxes.
So external validations, I mean, I'll come on to this a bit later, but we've done one,
I suppose, in the Netherlands, which I'll talk about a bit later.
I'll just put it up now to remind me to really talk about it.
And we've done someone else did an external validation rather bizarrely used it in a slightly different population,
an inpatient population of people in a in a psychiatric ward, in a prison, which is this is not really what the tool was developed for me.
Why not? I mean, you can use it for anyone with severe mental illness so they can be inpatients or outpatients.
So it's a very specific population. We've probably a higher base rate.
And then we did a sort of feasibility study of it in a couple of countries, Spain and China,
actually, to see whether just the predictors could be extracted from routine data.
And then we asked clinicians whether, you know, whether it would change their practice in any way,
whether they thought it was usable, practical and useful, actually.
So I mentioned the prison study just very briefly. So we the REC is the prison cohort and then we the numbers are a bit smaller but still quite large.
And you can see the base rate is very different, it's much higher.
And so the cut-off I think for higher risk we have three cut-offs had no medium and high and the Cut-off I think is over 40%.
So it's very different than that. Over 5% which was elevated risk folks move.
But I think it just reflects a very different base rate of the outcome here.
So here are some of the performance measures and that's actually does quite well compared to other tools in criminal justice.
And I'll show you some graphs about that. But it can, I suppose.
I mean, I look at everything ideally.
I mean, in criminal justice, you know, false negatives aren't really tolerated very well.
Yeah, it's very sensitive.
It is important to look at of course you can here change that with the cut cut-off that you use, but nevertheless calibration.
So that's also presented and that's the one and two year plots.
And actually what we decide to see too is we've got a maximum score,
which I think is about 60%, because once you get above that, it becomes a bit unstable.
And we thought that it wasn't accurate and interesting enough, the tool that's used in the UK,
the probation and parole, just assumes that you can go all the way up to 100%.
And so there isn't the cut off threshold.
And that's interesting because I'd you know, I suspect there's a lot of uncertainty at the upper ends of risk, but they haven't factor that in.
And again, you get this dropdown menu and then you can see here the percentage scores at the bottom,
the five in the 8% for this particular individual.
So it's quite low for this particular individual. I know why.
Yes, it's got very little and it's what we call antecedents.
So previous history and this sort of relatively old actually.
Thirty's quite old. Not that old, but it would be not young anyway for criminal justice population.
So the I mean, how does that compare? I mean, so it is a review some time ago of of some of the tools that are used for violence.
These are the tools, I mean, on average.
So the 1820s is a very well-known tool and that on average takes about 14 hours of people time to complete the version two.
So and they'll extract takes about 10 minutes. So you can sort of weigh up the.
Sort of. I mean, some of these tools do a lot and a little bit more.
I mean, identify needs and they go beyond just prediction.
But if you were looking at it as a prediction tool, then it performs quite well to these.
But these these are the sort of more longer tools if you look at the rest of convergence as we just published
last year review and these are tools used in criminal justice that we could find validation studies for.
Some of them are quite well known, so this thing called the LSI EIS is probably the most widely used tool can be just as you can see
the range of and then you report unfortunately these didn't report many performance measures.
The only tool that reported calibration measures was 1399, which is a tool,
quite an old tool which uses age bands to predict risk of sexual re-offending was it's used in people who've committed sex offences,
but some of the other tools are widely used actually in criminal justice and you
can see that the performance is moderate at best based on area under the curve.
But actually the real problem which we identified is they didn't report all the other performance measures.
So you can't actually you know, you didn't you know, the calibrations thought there.
So you just don't know. I mean it could be find it discriminating between different cut-offs but
completely of systematically off when it comes to giving you a probability score.
And the most widely used two is probably this thing called the NSL level of service and entry revised.
And you can see there's a range and you know, I think you could say it's moderate at best and these are independent validation studies.
So what we plan to do when we see when we develop log strength in particular is we we plan to try and validate it in some countries,
which we thought had linked databases and we thought, well, okay, these are the most obvious places.
And I suppose it's quite interesting.
I mean, you start off with these accounts and I can I think I can say that none of them was managed for various reasons,
but I can talk about the reasons if you're interested in this sort of detail of it.
But some of it is due to predictions, some of it's to the outcomes.
We actually spoke to people who, you know, who who manage these cohorts beforehand.
So, you know, we had a good idea that this was possible.
Well, when push comes to shove, it wasn't actually possible in Finland.
We're still hoping I mean, now we're just waiting for one final regulatory hurdle to pass.
But it's taken a long time because of various changes to the system and databases.
But in the other countries, I mean, various problems have emerged.
In Scotland, for instance, we we had difficulty linking health and crime.
It's not linked. And so we approached the agency that does this and basically no one was willing to take to make that decision.
This is what we could really figure out. And so we sort of gave up trying for a couple of years.
And in Australia what seems to be the case is that they had re incarceration data rather than
repeat offending data and then getting the repeat offending data was too much and it was,
it was too administratively burdensome and required a whole new set of ethics approvals.
And it was just too complicated. Too complicated. So the one country where we were able to do a valuation study was the Netherlands.
And it was an interesting experience, partly because they approached us.
And I think there's a lesson there is that, you know, you you can you can approach people, you can get funding to do valuation projects.
But sometimes, you know, there has to be really a sort of need on the ground for something.
And the Dutch approached us because they were actually redesigning their how they did risk assessments in criminal justice.
And they had read about X risk and some some academics had told them and they contacted us.
And so we worked with them to try and validate Tool Netherlands.
And Maree was involved in this project where we, they had data for the whole country actually this was quite interesting project and,
and they were quite interested in how it works in a different population, people on probation, so not even how the tool was designed.
So we, we actually were able to recalibrate the model to reflect that particular population change.
One of the variables give you an average score and then we were able to validate it to the Netherlands,
and that was a sort of part of the implementation pathway,
because then they were able then to pitch it almost to the government or the Ministry
of Justice to say that we think it should be embedded in their routine practice.
So everyone that's on probation and everyone that's assessed by probation should
have it as part of a range of things that happens is not just risk assessment,
they have needs assessment. And so actually it is embedded and so it's now used routinely Netherlands.
And for us it's great because it's an example of where, you know,
you develop a risk police model and then evaluated and implemented, you know, within a few years, which is sort of unusual.
I think the other validation we did was in Tajikistan and that was just sort of coincidentally
a sort of I met actually a couple of people working there who are us academics,
but they had been working in prisons in Tajikistan, particularly around HIV prevention treatment.
And and so we put together a project and were able to actually complete the validation.
Tajikistan The interesting thing about it is the prison population is very different.
I mean, they keep people in prison for much longer. They don't have very many people short sentences.
And so we were a little bit worried. It went very well because almost the prison population we thought might be very different.
And actually it wasn't so validated quite well and it didn't shrink very much.
So the end of the curve was 0.7 and it's quite a large sample.
We prospectively followed up about a thousand people for a year and like I say, it performed quite well.
So interesting that, you know, as long as the effect of predictors doesn't change a lot, you know, the prevalence of them can change.
So the prevalence of the predictions changed because the population of the effect probably doesn't change very much.
And that's probably why it didn't shrink. That's more than we originally thought.
We've done some other validations. So I mentioned the Dutch. One of folks knew that was older.
That's a different one. So that's the Oxford flux move. So the model we've we've, you know,
we've just got a paper done in England and that was quite a difficult project to do
because how in England do you get hold of data across health and criminal justice?
And actually what we did is we went to the police here in Thames Valley and they
have a what they call a flag for mental health and drug and alcohol problems,
which we then looked at quite carefully and figured out what a threshold would be that was about right based on the prevalence of those flags.
And so we used that as a as a as a measure for mental health,
because that's one of the predictors in abstract and then separate measures for drug and alcohol.
And that particular validation also did quite well, didn't shrink very much, and we had to drop a couple of variables they couldn't collect.
So I suppose you can.
Yeah, I mean, when we were speaking about it, Tom said, Well, I think you can get away with dropping one variable and maybe not two or three,
but it seems like if the variables are very thin or very strong predictors, then you probably can get away with two or three.
There was no experience and you just did a large finish of additional measures,
which actually was large in the original study, which is quite interesting as well.
And again, I think is the power of these Nordic registers that you can look at doesn't go across crime register.
It's just within health, but you need to link it to mortality and and sociodemographic information.
And the other thing we've trying to do is look at feasibility. And this asks very different questions.
It asks questions about me, can you use it clinically in your service?
And if you do, what do you think you know?
So we said we the way we do the studies is we we ask people to give us an estimate of risk of ten patients they know.
And then we present to them the score, the risk score and say, well, does that seem right?
Or if not, why do you think there's a difference? Would you do anything differently in your clinical practice?
And you get a sense of whether the the tools, how they would work in practice, what you would link it to, because the tool can only get you so far.
You need to link it to something, an intervention or something or and further assessment, more detailed assessment.
So I think that's part of the pipeline of work is a series of feasibility studies and they're that they're mixed methods.
I mean, they're mostly qualitative. I mean, it's about presenting to clinicians or clinical teams with information from risk models and
seeing what they think and then seeing if you can extract information from clinical records.
So they, you know, they don't they don't attract the best journals in the world.
So it's quite yeah, if you have to sort of take a slightly different approach and just see it as part of a bigger picture.
But it's. Quite interesting and quite informative, and particularly the first one, the Faux Vox, which is a tool for forensic psychiatry.
I mean that was quite good actually when we did the feasibility work.
People then started to use the calculator and into the clinical teams and sort of think, Oh, okay, well that's pretty good, I'm going to use it.
And they sort of. So those are three feasibility studies, a faux vox, one in here in Oxford and books that BMC Psychiatry was in,
in hospital in China and the country was in Sweden and there I think they're all using faux vox regularly now in clinical practice.
So if I was to stand a little bit back and think about the challenges when I think maybe this is three levels,
three things, yes, is developing models,
which isn't really what I've been talking about, but just just I think my impression is it takes ages where it can tell you how to do it properly.
Takes ages. I mean, you can do it and you can do it quickly, but to do it really properly and to figure out,
you know, exactly, you know, getting the protocol right,
what should go into the model and, you know,
getting all the everything put in place and looking at all the various variables, univariate and then multivariable.
That really takes a long time to do it properly.
The validation, I think is is is problematic most of the time, unless you have access to registers of some sort because you just kind of have a scale.
I mean, you may have heard of the rule of thumb of 100 events, you know, to validate the model.
And most outcomes, at least the ones I've been looking at, they're quite rare.
So you you struggle actually, unless you collect the data.
You know, we didn't Tajikistan for a high base rate, which is for offending in that population.
But in most of the time, you know, it's really not possible.
So you sort of almost have to think when you develop a tool to think about predictors that you can validate in registers, you know,
so think about things that are routinely collected because otherwise you may end up developing a tool that you can never validate externally.
And then implementing tools requires collaborators, obviously.
I mean everything because collaborators for this work has external collaborators
usually and and people probably who are looking for something to fill a gap,
a need. And so actually my my impression is that, you know, just telling somebody or an agency, look,
we've got this tool in it, you know, it's really great, you know, and so it's not good work.
I mean, they have to sort of have a need and identify the need among themselves to do that.
And so, of course, you know, that's the bit in the in the pyramid that's very few tools implemented.
You can develop a lot, validate the few more, but implementing them is just the price at the top of the iceberg.
When I think about predictors and outcomes, just to sort of talk you through a few examples,
I think, I mean, one of the challenge with predictors is it has to do with missing information.
You know, something to think about is, you know,
what level of missing this you able to cope with and you know, whether it's at random or not, it's important.
But also when it comes to validation, it's a big problem because like I said, I mean, we are able to cope,
I think in our models where the missing information is for predictors that are not very strong, have strong effects.
But in one of the validations we want to do the missing variable.
Missing predictor was it was the most powerful, it was the previous behaviour.
So whether it's violence or suicidal acts,
it was one of the two and we just thought we couldn't go ahead because I mean the most powerful predictor is missing.
So there there was one missing was powerful. But with the with done in the Thames Valley, I think we had two or three missing.
But they were all very, very they were very powerful predictors.
So actually the model worked fine, didn't shrink very much because they're not very powerful, different definitions.
And I think they're the key thing is to look at the prevalence and see, you know,
whether you can find a prevalence compared to the original development sample, which is similar.
So if you've got I mean, we've had this a lot with mental health variables with some people sort of global school,
it's out of five and is it three or is it two or is it one,
the cut-off that you would use and you would try and base it, in our view,
on what corresponds best to the prevalence and the original sample of the sample nonclinical predictors.
Well, this is the difficulty about using data set. So a lot of datasets within clinical environments.
So you get like in the US you have these insurance datasets, but within health care, if you see what I'm saying.
But if you're gathering information about that. Same educational level or income or employment status, then you're going to get that.
So you have to be aware of that. And it does limit, I think, the power of some tools where those predictors could be quite important, actually.
The previous thing I mentioned and ethical issues are very important to consider because some of the predictors used for some
of these tools are proxies and they're problematic proxies and you have to be careful and there's ways of dealing with it.
You just completely remove anything that might be a proxy or to trying to adjust in some way for the possibility that it's acting as a proxy.
So there's different ways of dealing with it. And I think it's a different question.
It's a different is how it's a whole session of its own because it's such an important area.
Like I say, prevalence in the original episode was worth considering about the direction
and magnitude we need to univariate so in and pre specifying it the outcome.
I mean here's one example of this is a review we did of repeats offending one and two years.
You can see the big range now. I mean, why is there such a big range?
I don't think it's because repeat offending is different things with how you count.
Repeat offending is very different.
And I think that just tells you that, you know, you the how you how you capture the outcomes is going to be quite important,
particularly when you consider if you should recalibrate the model.
So even within one country, you know, you have North Carolina and Oregon and they're probably counting it differently.
I mean, it's the USA and then you have know the Nordic countries, Finland, Norway and they are counting it differently.
So there's ways of counting this particular outcome. For instance,
you don't include fines and fines and or you may not include what they call
someone who's record so someone can be at the end of their prison sentence.
They still have time to serve,
but they can serve in the community and then they can be recalled into prison if they break a rule like not turn up to their appointment on time,
in some cases countenance, repeat, offending, some places don't.
And that will determine part of this variation.
But it's just one example. And I mean, we came across this because, you know, it's one of our outcomes is repeat offending.
And one of the first validation we did, we actually used a non-criminal outcome,
which was any interpersonal violence captured in a very rich phenotype cohort of people with schizophrenia.
But it meant, you know, a culture of recalibration.
And thinking about what to do is not ideal because it's a three year outcome which is different in our paper,
you know, we had a one year outcome to have a three year outcome of a different it was a different outcome.
And and so you need to think quite carefully there about how to how to deal with that.
And like I say, you know, there's different things. This arrest is conviction, this violent conviction and re imprisonment.
I think the problem we had with some of the US data, the US state of Texas,
they only had informational re imprisonment, which is very easy to collect.
I suppose if you particularly have a prison database it's very easy to collect rather than repeat offending,
which doesn't lead to imprisonment, which a lot of it doesn't, particularly if it's less serious than what we call common assault.
And the like I say, I think it requires different cut-offs.
So that's the other thing. So we had prespecified cut-offs but they don't make sense if the outcomes are different, you know, and then, you know,
I think there should be a very clear plan about how you would consider and what you would do to
recalibrate to and experiences that you tend to have to recalibrate tools and new populations and wider,
wider issues and meta issues which may or may not bother you very much, as is when you do it when you develop a model, is who does the validation?
I mean, it's a bit tricky because really no one's got skin in the game.
So it ends up being the person who's developed model, you know, which is which then is not independent, is it?
I mean, because the person who is developed, you know,
is then involved and that has some advantages because they can sort of ensure the some quality, you know, quality control and it's done properly.
But at the same time, there's not truly an independent validation. And that might be too high a threshold to say it should be completely independent,
because why would someone want to validate a particular model unless they have a
particular acute clinical need or practical need for moral issue publications?
I mention that because I think despite what people. To say publishing validations is difficult, so people say it.
So you see journals often saying, well, you know, we've been you know, we look forward to validations of the of these models.
But actually, we found that they're quite difficult to publish.
And it's understandable. I mean, they probably don't attract the same level of citations.
And so journals may be a little bit less inclined towards them, you know, and also, they're not seen as novel because,
you know, the tool is what's novel and it's sort of exciting and there's a new thing out there.
And, you know, at the very least, it's giving you some information about risk factors that probably people didn't really think about.
So there's always something novel, even if it's not going to be interpreted to do something interesting or novel about it.
But the validation is not. And so there is a challenge there.
I mean, there are some journals which say specifically we're interested in validations and you'll see that we published in them.
So they're the ones that we've gone to because they they're more of them.
Our experience has been that they're more interested. Approvals is just a wider issue with doing research is that is that you know,
there are lots of regulatory hurdles and approval hurdles which you have to sort of factor in to your timelines.
So if you want to do a validation study, let's say as part of a project, a master's or Ph.D. project,
you need to be ideally involved in something where the approvals are in place
because getting them up and running and sorting it all out is can be quite a long,
can take quite long, and sometimes it can take longer than the time you have to do the research.
So there's lots of examples of that. I've spoken to people, I mean, you probably know lots as well.
But um, and then just finally just to thank a few of the people involved.
So I mean, early on some of the Swedish collaborators, Susan Mather, who was here now is I see Uclh not at Birmingham.
Tom and obviously Maria have been central to this, the research that we've done and some other people as well.
Thank you very much.