1 00:00:00,030 --> 00:00:06,270 Hello, everybody. Thank you very much for inviting me. And so this is a sort of I've summarised some of the work we've done, 2 00:00:06,540 --> 00:00:14,280 but look particularly at whether these tools can be used and how they can be implemented in some of the validation work we've done. 3 00:00:14,640 --> 00:00:21,299 So the, the sort of more applied end of the of the picture and these these tools, I mean, 4 00:00:21,300 --> 00:00:25,770 just that, you know, I mean, I was involved in developing them with Tom Fanshawe as well. 5 00:00:25,770 --> 00:00:29,400 And Maria, who's here as well, has been working on them with us as well. 6 00:00:30,210 --> 00:00:35,700 So it's a collaborative effort between psychiatry and primary care sciences. 7 00:00:36,240 --> 00:00:39,450 So I'll start off by telling you a little bit about the tools that I'll tell you a little 8 00:00:39,450 --> 00:00:43,740 bit about some of the challenges we've had with validating them and implementing them. 9 00:00:43,860 --> 00:00:47,700 And that's basically it. And then there's some reflections on all of that. 10 00:00:48,000 --> 00:00:49,520 But I'll sort of introduce a little bit. 11 00:00:49,530 --> 00:00:56,009 So some of that might be overlapping with what you've heard, but I just thought would be useful and show you some that they're serious. 12 00:00:56,010 --> 00:01:01,530 The tools that we've tried to develop for adverse outcomes in mental health and criminal justice and a 13 00:01:01,530 --> 00:01:06,780 lot of a lot of them overlap the mental health and justice adverse outcomes overlap and you'll see why. 14 00:01:06,780 --> 00:01:13,319 What I mean by that and by adverse outcomes are usually meaning things like perpetration 15 00:01:13,320 --> 00:01:18,870 of violence or in some of the tools we're looking at suicide rates as completed suicide. 16 00:01:19,350 --> 00:01:23,760 So these are longer term, very, very serious obviously adverse outcomes. 17 00:01:24,210 --> 00:01:30,240 And one of the things we've been trying to do with these tools is develop them using the best methods. 18 00:01:30,480 --> 00:01:35,700 So large data sets. And where we've gone to for the data is Sweden. 19 00:01:36,150 --> 00:01:42,570 And we've gone to Sweden because Sweden has linked datasets across various different domains, 20 00:01:42,570 --> 00:01:50,010 including health, justice, mortality, but also sociodemographic information, family information. 21 00:01:50,550 --> 00:01:58,590 And it's probably the largest national database that does this as only few countries are able to link across all these different databases. 22 00:01:58,920 --> 00:02:01,140 And Sweden has about a population of about 10 million. 23 00:02:01,530 --> 00:02:09,959 Denmark does it, but it has about half the population in Finland also, and there's one or two Australian states that it's possible to do it. 24 00:02:09,960 --> 00:02:16,200 But again, it doesn't reach the size and I actually think in Sweden that they're very high 25 00:02:16,200 --> 00:02:23,969 quality so that they're very linked because they use a ten digit personal ID number, 26 00:02:23,970 --> 00:02:29,210 which is unique to every resident. And so you don't have these problems that you get, say, 27 00:02:29,270 --> 00:02:37,020 in the UK where you're having to sort of guess someone's name and date of birth and the names and Romeo's date of birth is a bit wrong. 28 00:02:37,020 --> 00:02:41,850 It's sort of to probability matching. You don't get that problem. 29 00:02:42,000 --> 00:02:44,550 So there's very little missing data. So it's very high quality. 30 00:02:45,480 --> 00:02:50,880 And then, I mean, in terms of the methods you've been talking about the last couple of days, I assume, but, you know, 31 00:02:51,090 --> 00:02:58,739 trying to put everything down and specify what we do and use a range of performance measures and 32 00:02:58,740 --> 00:03:04,650 use predictors that are easy to score that require additional interviews as scalable predictors. 33 00:03:05,370 --> 00:03:11,549 And part of the reason for doing that, of course, is that in order to pull predictive predictors out of databases, 34 00:03:11,550 --> 00:03:16,320 you need to use things which are recorded firstly and routinely recorded secondly. 35 00:03:17,010 --> 00:03:21,630 And that means that they tend to be straightforward, relatively speaking. 36 00:03:21,900 --> 00:03:30,990 So it's not for instance, in psychiatry you can ask about a symptom or you can ask, let's say, a predictor of a diagnosis. 37 00:03:31,440 --> 00:03:36,450 And in large scale databases, the symptoms are very inaccurately recorded. 38 00:03:36,810 --> 00:03:40,140 You don't know if they're not recorded, if they're present, because they haven't been asked. 39 00:03:40,680 --> 00:03:45,239 And you don't know even when they're recorded, if it's what threshold people have used for diagnosis, 40 00:03:45,240 --> 00:03:53,280 this sort of agreement, there are classification of diseases, something called the ICD ten, and that's 11. 41 00:03:53,790 --> 00:03:58,709 And there's a you know, there's a consensus diagnosis to be worked out from decades. 42 00:03:58,710 --> 00:04:03,330 And so they're a little bit easier to to score a bit more reliable. 43 00:04:03,420 --> 00:04:08,670 Iterator and rewriter reliability is much better for these type of things. 44 00:04:08,970 --> 00:04:14,400 And then for the tools that we've developed, we try to internally validate them and then externally validate where possible. 45 00:04:14,400 --> 00:04:18,840 So you'll see some of that coming up. And then we've translated them into risk calculators. 46 00:04:18,840 --> 00:04:26,250 And the idea behind these calculators is a bit like Q Risk or Framingham, that they're simple, scalable, 47 00:04:26,670 --> 00:04:35,309 available on the Internet, free to use technical training, and you can see all the coefficients, how they're scored as well. 48 00:04:35,310 --> 00:04:38,550 So you can see how they work. So that's published. 49 00:04:38,940 --> 00:04:44,190 So here are some of the ones we've developed. So the start of results with an ox rack. 50 00:04:44,760 --> 00:04:48,930 So this move is a tool for people with severe mental illness. 51 00:04:48,930 --> 00:04:56,760 With mental illness in this case means people who usually have been to hospital at some point in their life. 52 00:04:57,220 --> 00:05:02,110 It's a diagnosis of what they call schizophrenia spectrum is. So it is an bipolar disorder. 53 00:05:02,120 --> 00:05:07,280 So they're they're on the more extreme end in terms of severity, usually. 54 00:05:08,210 --> 00:05:14,780 And so we developed a tool to look at perpetration of violence in people with severe mental illness. 55 00:05:14,810 --> 00:05:20,780 Any points in the patient pathway? And we did that partly because this is a problem. 56 00:05:20,780 --> 00:05:27,790 I mean, about 10% of people, if you look over a five year period, have showed quite serious violence. 57 00:05:27,800 --> 00:05:35,330 And that's a it's higher than what you would expect for people of similar age and gender in the general population. 58 00:05:35,790 --> 00:05:42,829 Still a minority of patients, but enough to be an important outcome to prevent in this patient population. 59 00:05:42,830 --> 00:05:51,470 And we know it's modifiable. So that's good reason to come up with some method to predict it so you can prevent it and release prisoners. 60 00:05:51,980 --> 00:05:59,000 We developed two clocks REC and this is the first tool we developed and then it's at the end of the prison sentence. 61 00:05:59,000 --> 00:06:07,310 And there again, the idea was that this is a risky period in the first 1 to 2 years after leaving prison for repeat offending. 62 00:06:07,730 --> 00:06:17,000 And a lot of that repeat offending is driven by modifiable factors such as drug and alcohol problems and also in some cases, mental health problems. 63 00:06:17,270 --> 00:06:27,860 So we sought a tool that could predict age and then events, and then ideally the link to an intervention could again help prevent reoffending risk. 64 00:06:27,860 --> 00:06:32,240 And reoffending risk is is much too high in people who leave prison. 65 00:06:32,600 --> 00:06:43,670 So if you look around the world, recidivism rates are sort of in the range of 40 to 60% in two years is quite high, absolute rates reoffending. 66 00:06:44,150 --> 00:06:47,390 And then we have some other tools for specific populations. 67 00:06:48,170 --> 00:06:51,890 And at the bottom, you see there's a tool for suicide risk assessment, 68 00:06:52,340 --> 00:06:58,069 which of course is a very rare outcome in even people's mental illness, fortunately. 69 00:06:58,070 --> 00:07:07,850 But it's nevertheless a very important part of clinical work to evaluate the risk and then try and manage it in some way. 70 00:07:08,120 --> 00:07:16,759 And, and we've just worked very has been involved in quite extensively as is another calculator 71 00:07:16,760 --> 00:07:21,020 to look at suicide risk in people who self-harmed because again that's an important 72 00:07:21,020 --> 00:07:27,440 population with the self-harming rates are very relatively high and they represent a 73 00:07:27,440 --> 00:07:32,390 high risk population for suicides in the subsequent year or so after self-harming. 74 00:07:32,900 --> 00:07:37,920 So if I just look at one of the tools as you see how they've been developed, this is all smooth. 75 00:07:37,940 --> 00:07:49,520 So what we did is we took the whole country and we identified a sample of people geographically, 58,000 of people with severe mental illness. 76 00:07:50,330 --> 00:07:56,720 And then we identified another part of the country which we can use to validate what we developed. 77 00:07:57,200 --> 00:08:02,929 And we when we derive the the tool, we we came up with some candidate risk factors, 78 00:08:02,930 --> 00:08:08,270 some that we thought should be in the tool because of face visibility. In some of that, we weren't sure about that. 79 00:08:08,270 --> 00:08:16,069 We then tested in multivariable models and then we we set up a high risk or elevated risk category at 5%. 80 00:08:16,070 --> 00:08:22,190 So this is a one year risk of violent offending in the population. 81 00:08:22,490 --> 00:08:25,340 And all that was prespecified in the protocol that we published. 82 00:08:25,730 --> 00:08:33,890 And then we tried to present a range of performance measures which I think you've heard about today, measures discrimination, but also calibration. 83 00:08:34,550 --> 00:08:38,870 So the risk factors, the candidate risk factors, this is probably less interesting to you, 84 00:08:38,870 --> 00:08:47,419 but it gives you a sense of the approach we took as we looked at different domains such as demographic, clinical, genetic, 85 00:08:47,420 --> 00:08:49,130 which is just previous violent crime, 86 00:08:49,610 --> 00:08:59,659 and then family factors which are probably proxies for genetic risk or early other early risk factors and outcomes, 87 00:08:59,660 --> 00:09:04,580 violent crime and those in order to select the predictors. 88 00:09:04,580 --> 00:09:13,160 We looked at the literature quite carefully and we looked at those factors that we also thought were possible to extract reliably. 89 00:09:13,580 --> 00:09:18,410 And the outcome, as you can see, is quite a sort of rare outcome actually even in this population. 90 00:09:18,800 --> 00:09:21,980 So it's quite a high threshold too, for that particular outcome. 91 00:09:22,400 --> 00:09:26,180 And so these are some of the risk factors we looked at and here's the prevalence of them. 92 00:09:26,750 --> 00:09:34,639 So the ones in blue we thought should go in the model based on previous literature and face sort of the same ones in number two and three. 93 00:09:34,640 --> 00:09:41,210 We were not sure about two. We thought probably go in the model that we wanted to test and then three we're not sure about at all. 94 00:09:41,540 --> 00:09:48,260 And then when we came to test some of the models, the a couple of the Greens survived and a couple of read survived. 95 00:09:48,830 --> 00:09:49,760 That's quite interesting. 96 00:09:49,760 --> 00:09:59,270 And the externalisation, so in the in the geographical region, which was different to the development region, the two performed quite well. 97 00:09:59,760 --> 00:10:04,800 I mean, the omnibus measure is quite high point 89, so it's suspiciously high school. 98 00:10:06,780 --> 00:10:13,200 And then the here are the other metrics. I mean, the sensitivity is quite important from a population perspective. 99 00:10:13,740 --> 00:10:20,250 And I think the way we write it up also we highlighted the high NPV because it could then act as a screen out too, 100 00:10:20,850 --> 00:10:25,860 because a lot of services want to screen out the large number of referrals they get. 101 00:10:26,160 --> 00:10:33,810 And this is what this could help. Calibration is quite good, so the top one is in the derivation. 102 00:10:34,620 --> 00:10:42,779 One below is in the validation samples. It doesn't address any systematic problem, particularly here's the online calculator. 103 00:10:42,780 --> 00:10:47,370 I mean, it's online, it's got these dropdown menus on my screen to use. 104 00:10:47,370 --> 00:10:49,979 But I mean, this is not actually the online version of my SLAVOJ, 105 00:10:49,980 --> 00:10:55,860 but you just sort of click on that and you can get and you pull that along and you get a different age and then you pull down. 106 00:10:55,860 --> 00:11:03,090 Some of them you can score is unknown and then it gives you a range of probability scores and you end up with a probability score. 107 00:11:03,510 --> 00:11:08,559 And then the pre specified category, elevated or not, was 5%. 108 00:11:08,560 --> 00:11:18,420 So you could see if it's above that. And then we also have some visuals so you can see whether you can visualise it in table or in boxes. 109 00:11:19,260 --> 00:11:25,530 So external validations, I mean, I'll come on to this a bit later, but we've done one, 110 00:11:25,530 --> 00:11:30,750 I suppose, in the Netherlands, which I'll talk about a bit later. 111 00:11:31,050 --> 00:11:33,370 I'll just put it up now to remind me to really talk about it. 112 00:11:33,370 --> 00:11:39,180 And we've done someone else did an external validation rather bizarrely used it in a slightly different population, 113 00:11:39,690 --> 00:11:47,320 an inpatient population of people in a in a psychiatric ward, in a prison, which is this is not really what the tool was developed for me. 114 00:11:47,400 --> 00:11:52,020 Why not? I mean, you can use it for anyone with severe mental illness so they can be inpatients or outpatients. 115 00:11:52,320 --> 00:11:56,070 So it's a very specific population. We've probably a higher base rate. 116 00:11:56,700 --> 00:12:01,290 And then we did a sort of feasibility study of it in a couple of countries, Spain and China, 117 00:12:01,290 --> 00:12:08,669 actually, to see whether just the predictors could be extracted from routine data. 118 00:12:08,670 --> 00:12:13,890 And then we asked clinicians whether, you know, whether it would change their practice in any way, 119 00:12:13,980 --> 00:12:17,760 whether they thought it was usable, practical and useful, actually. 120 00:12:18,090 --> 00:12:28,110 So I mentioned the prison study just very briefly. So we the REC is the prison cohort and then we the numbers are a bit smaller but still quite large. 121 00:12:28,620 --> 00:12:31,379 And you can see the base rate is very different, it's much higher. 122 00:12:31,380 --> 00:12:39,120 And so the cut-off I think for higher risk we have three cut-offs had no medium and high and the Cut-off I think is over 40%. 123 00:12:39,120 --> 00:12:44,610 So it's very different than that. Over 5% which was elevated risk folks move. 124 00:12:45,030 --> 00:12:49,140 But I think it just reflects a very different base rate of the outcome here. 125 00:12:49,800 --> 00:12:58,290 So here are some of the performance measures and that's actually does quite well compared to other tools in criminal justice. 126 00:12:58,290 --> 00:13:02,849 And I'll show you some graphs about that. But it can, I suppose. 127 00:13:02,850 --> 00:13:05,879 I mean, I look at everything ideally. 128 00:13:05,880 --> 00:13:11,130 I mean, in criminal justice, you know, false negatives aren't really tolerated very well. 129 00:13:11,820 --> 00:13:13,110 Yeah, it's very sensitive. 130 00:13:13,110 --> 00:13:20,639 It is important to look at of course you can here change that with the cut cut-off that you use, but nevertheless calibration. 131 00:13:20,640 --> 00:13:24,990 So that's also presented and that's the one and two year plots. 132 00:13:24,990 --> 00:13:30,540 And actually what we decide to see too is we've got a maximum score, 133 00:13:30,570 --> 00:13:34,350 which I think is about 60%, because once you get above that, it becomes a bit unstable. 134 00:13:34,980 --> 00:13:41,700 And we thought that it wasn't accurate and interesting enough, the tool that's used in the UK, 135 00:13:42,090 --> 00:13:47,819 the probation and parole, just assumes that you can go all the way up to 100%. 136 00:13:47,820 --> 00:13:51,840 And so there isn't the cut off threshold. 137 00:13:51,840 --> 00:13:58,500 And that's interesting because I'd you know, I suspect there's a lot of uncertainty at the upper ends of risk, but they haven't factor that in. 138 00:13:59,130 --> 00:14:05,610 And again, you get this dropdown menu and then you can see here the percentage scores at the bottom, 139 00:14:05,610 --> 00:14:09,360 the five in the 8% for this particular individual. 140 00:14:09,360 --> 00:14:13,740 So it's quite low for this particular individual. I know why. 141 00:14:14,580 --> 00:14:17,899 Yes, it's got very little and it's what we call antecedents. 142 00:14:17,900 --> 00:14:23,670 So previous history and this sort of relatively old actually. 143 00:14:23,670 --> 00:14:30,690 Thirty's quite old. Not that old, but it would be not young anyway for criminal justice population. 144 00:14:31,110 --> 00:14:40,230 So the I mean, how does that compare? I mean, so it is a review some time ago of of some of the tools that are used for violence. 145 00:14:40,830 --> 00:14:43,500 These are the tools, I mean, on average. 146 00:14:43,650 --> 00:14:53,670 So the 1820s is a very well-known tool and that on average takes about 14 hours of people time to complete the version two. 147 00:14:55,070 --> 00:14:59,430 So and they'll extract takes about 10 minutes. So you can sort of weigh up the. 148 00:14:59,520 --> 00:15:03,800 Sort of. I mean, some of these tools do a lot and a little bit more. 149 00:15:03,810 --> 00:15:07,590 I mean, identify needs and they go beyond just prediction. 150 00:15:07,590 --> 00:15:12,059 But if you were looking at it as a prediction tool, then it performs quite well to these. 151 00:15:12,060 --> 00:15:16,709 But these these are the sort of more longer tools if you look at the rest of convergence as we just published 152 00:15:16,710 --> 00:15:24,600 last year review and these are tools used in criminal justice that we could find validation studies for. 153 00:15:25,620 --> 00:15:32,009 Some of them are quite well known, so this thing called the LSI EIS is probably the most widely used tool can be just as you can see 154 00:15:32,010 --> 00:15:40,860 the range of and then you report unfortunately these didn't report many performance measures. 155 00:15:41,640 --> 00:15:47,430 The only tool that reported calibration measures was 1399, which is a tool, 156 00:15:47,640 --> 00:15:56,100 quite an old tool which uses age bands to predict risk of sexual re-offending was it's used in people who've committed sex offences, 157 00:15:57,270 --> 00:16:00,899 but some of the other tools are widely used actually in criminal justice and you 158 00:16:00,900 --> 00:16:06,240 can see that the performance is moderate at best based on area under the curve. 159 00:16:06,720 --> 00:16:11,610 But actually the real problem which we identified is they didn't report all the other performance measures. 160 00:16:11,610 --> 00:16:15,959 So you can't actually you know, you didn't you know, the calibrations thought there. 161 00:16:15,960 --> 00:16:21,630 So you just don't know. I mean it could be find it discriminating between different cut-offs but 162 00:16:21,630 --> 00:16:27,540 completely of systematically off when it comes to giving you a probability score. 163 00:16:28,320 --> 00:16:34,590 And the most widely used two is probably this thing called the NSL level of service and entry revised. 164 00:16:35,220 --> 00:16:44,370 And you can see there's a range and you know, I think you could say it's moderate at best and these are independent validation studies. 165 00:16:45,360 --> 00:16:54,120 So what we plan to do when we see when we develop log strength in particular is we we plan to try and validate it in some countries, 166 00:16:54,510 --> 00:16:59,520 which we thought had linked databases and we thought, well, okay, these are the most obvious places. 167 00:17:00,140 --> 00:17:01,469 And I suppose it's quite interesting. 168 00:17:01,470 --> 00:17:10,410 I mean, you start off with these accounts and I can I think I can say that none of them was managed for various reasons, 169 00:17:10,950 --> 00:17:14,459 but I can talk about the reasons if you're interested in this sort of detail of it. 170 00:17:14,460 --> 00:17:18,000 But some of it is due to predictions, some of it's to the outcomes. 171 00:17:18,480 --> 00:17:24,600 We actually spoke to people who, you know, who who manage these cohorts beforehand. 172 00:17:24,600 --> 00:17:27,749 So, you know, we had a good idea that this was possible. 173 00:17:27,750 --> 00:17:31,559 Well, when push comes to shove, it wasn't actually possible in Finland. 174 00:17:31,560 --> 00:17:38,129 We're still hoping I mean, now we're just waiting for one final regulatory hurdle to pass. 175 00:17:38,130 --> 00:17:43,140 But it's taken a long time because of various changes to the system and databases. 176 00:17:43,320 --> 00:17:46,890 But in the other countries, I mean, various problems have emerged. 177 00:17:47,220 --> 00:17:51,810 In Scotland, for instance, we we had difficulty linking health and crime. 178 00:17:52,480 --> 00:18:01,800 It's not linked. And so we approached the agency that does this and basically no one was willing to take to make that decision. 179 00:18:01,950 --> 00:18:07,890 This is what we could really figure out. And so we sort of gave up trying for a couple of years. 180 00:18:08,520 --> 00:18:14,639 And in Australia what seems to be the case is that they had re incarceration data rather than 181 00:18:14,640 --> 00:18:19,180 repeat offending data and then getting the repeat offending data was too much and it was, 182 00:18:19,190 --> 00:18:24,809 it was too administratively burdensome and required a whole new set of ethics approvals. 183 00:18:24,810 --> 00:18:33,930 And it was just too complicated. Too complicated. So the one country where we were able to do a valuation study was the Netherlands. 184 00:18:33,930 --> 00:18:39,090 And it was an interesting experience, partly because they approached us. 185 00:18:39,440 --> 00:18:46,739 And I think there's a lesson there is that, you know, you you can you can approach people, you can get funding to do valuation projects. 186 00:18:46,740 --> 00:18:54,270 But sometimes, you know, there has to be really a sort of need on the ground for something. 187 00:18:54,600 --> 00:19:02,190 And the Dutch approached us because they were actually redesigning their how they did risk assessments in criminal justice. 188 00:19:02,640 --> 00:19:09,600 And they had read about X risk and some some academics had told them and they contacted us. 189 00:19:09,600 --> 00:19:13,379 And so we worked with them to try and validate Tool Netherlands. 190 00:19:13,380 --> 00:19:20,510 And Maree was involved in this project where we, they had data for the whole country actually this was quite interesting project and, 191 00:19:20,790 --> 00:19:26,849 and they were quite interested in how it works in a different population, people on probation, so not even how the tool was designed. 192 00:19:26,850 --> 00:19:34,290 So we, we actually were able to recalibrate the model to reflect that particular population change. 193 00:19:34,290 --> 00:19:40,170 One of the variables give you an average score and then we were able to validate it to the Netherlands, 194 00:19:40,830 --> 00:19:45,330 and that was a sort of part of the implementation pathway, 195 00:19:45,330 --> 00:19:52,229 because then they were able then to pitch it almost to the government or the Ministry 196 00:19:52,230 --> 00:19:59,130 of Justice to say that we think it should be embedded in their routine practice. 197 00:19:59,870 --> 00:20:04,879 So everyone that's on probation and everyone that's assessed by probation should 198 00:20:04,880 --> 00:20:10,650 have it as part of a range of things that happens is not just risk assessment, 199 00:20:10,650 --> 00:20:15,920 they have needs assessment. And so actually it is embedded and so it's now used routinely Netherlands. 200 00:20:16,430 --> 00:20:19,610 And for us it's great because it's an example of where, you know, 201 00:20:19,610 --> 00:20:27,169 you develop a risk police model and then evaluated and implemented, you know, within a few years, which is sort of unusual. 202 00:20:27,170 --> 00:20:35,700 I think the other validation we did was in Tajikistan and that was just sort of coincidentally 203 00:20:35,700 --> 00:20:42,290 a sort of I met actually a couple of people working there who are us academics, 204 00:20:42,290 --> 00:20:48,290 but they had been working in prisons in Tajikistan, particularly around HIV prevention treatment. 205 00:20:48,860 --> 00:20:54,919 And and so we put together a project and were able to actually complete the validation. 206 00:20:54,920 --> 00:20:58,639 Tajikistan The interesting thing about it is the prison population is very different. 207 00:20:58,640 --> 00:21:03,710 I mean, they keep people in prison for much longer. They don't have very many people short sentences. 208 00:21:04,370 --> 00:21:11,199 And so we were a little bit worried. It went very well because almost the prison population we thought might be very different. 209 00:21:11,200 --> 00:21:15,229 And actually it wasn't so validated quite well and it didn't shrink very much. 210 00:21:15,230 --> 00:21:19,010 So the end of the curve was 0.7 and it's quite a large sample. 211 00:21:19,010 --> 00:21:27,350 We prospectively followed up about a thousand people for a year and like I say, it performed quite well. 212 00:21:27,350 --> 00:21:38,060 So interesting that, you know, as long as the effect of predictors doesn't change a lot, you know, the prevalence of them can change. 213 00:21:38,210 --> 00:21:43,970 So the prevalence of the predictions changed because the population of the effect probably doesn't change very much. 214 00:21:44,750 --> 00:21:49,700 And that's probably why it didn't shrink. That's more than we originally thought. 215 00:21:50,150 --> 00:21:55,630 We've done some other validations. So I mentioned the Dutch. One of folks knew that was older. 216 00:21:55,640 --> 00:22:01,010 That's a different one. So that's the Oxford flux move. So the model we've we've, you know, 217 00:22:02,210 --> 00:22:09,260 we've just got a paper done in England and that was quite a difficult project to do 218 00:22:09,260 --> 00:22:15,589 because how in England do you get hold of data across health and criminal justice? 219 00:22:15,590 --> 00:22:20,030 And actually what we did is we went to the police here in Thames Valley and they 220 00:22:20,030 --> 00:22:25,280 have a what they call a flag for mental health and drug and alcohol problems, 221 00:22:25,280 --> 00:22:33,920 which we then looked at quite carefully and figured out what a threshold would be that was about right based on the prevalence of those flags. 222 00:22:34,580 --> 00:22:38,630 And so we used that as a as a as a measure for mental health, 223 00:22:38,690 --> 00:22:44,180 because that's one of the predictors in abstract and then separate measures for drug and alcohol. 224 00:22:44,900 --> 00:22:52,310 And that particular validation also did quite well, didn't shrink very much, and we had to drop a couple of variables they couldn't collect. 225 00:22:53,300 --> 00:22:54,620 So I suppose you can. 226 00:22:54,650 --> 00:23:01,340 Yeah, I mean, when we were speaking about it, Tom said, Well, I think you can get away with dropping one variable and maybe not two or three, 227 00:23:01,550 --> 00:23:09,410 but it seems like if the variables are very thin or very strong predictors, then you probably can get away with two or three. 228 00:23:09,470 --> 00:23:13,820 There was no experience and you just did a large finish of additional measures, 229 00:23:14,450 --> 00:23:17,989 which actually was large in the original study, which is quite interesting as well. 230 00:23:17,990 --> 00:23:26,810 And again, I think is the power of these Nordic registers that you can look at doesn't go across crime register. 231 00:23:26,820 --> 00:23:32,750 It's just within health, but you need to link it to mortality and and sociodemographic information. 232 00:23:33,740 --> 00:23:39,080 And the other thing we've trying to do is look at feasibility. And this asks very different questions. 233 00:23:39,080 --> 00:23:45,290 It asks questions about me, can you use it clinically in your service? 234 00:23:46,160 --> 00:23:50,000 And if you do, what do you think you know? 235 00:23:50,000 --> 00:23:59,330 So we said we the way we do the studies is we we ask people to give us an estimate of risk of ten patients they know. 236 00:23:59,660 --> 00:24:04,219 And then we present to them the score, the risk score and say, well, does that seem right? 237 00:24:04,220 --> 00:24:09,440 Or if not, why do you think there's a difference? Would you do anything differently in your clinical practice? 238 00:24:09,920 --> 00:24:18,680 And you get a sense of whether the the tools, how they would work in practice, what you would link it to, because the tool can only get you so far. 239 00:24:18,680 --> 00:24:24,140 You need to link it to something, an intervention or something or and further assessment, more detailed assessment. 240 00:24:24,860 --> 00:24:32,629 So I think that's part of the pipeline of work is a series of feasibility studies and they're that they're mixed methods. 241 00:24:32,630 --> 00:24:41,150 I mean, they're mostly qualitative. I mean, it's about presenting to clinicians or clinical teams with information from risk models and 242 00:24:41,150 --> 00:24:45,979 seeing what they think and then seeing if you can extract information from clinical records. 243 00:24:45,980 --> 00:24:51,560 So they, you know, they don't they don't attract the best journals in the world. 244 00:24:52,160 --> 00:24:57,140 So it's quite yeah, if you have to sort of take a slightly different approach and just see it as part of a bigger picture. 245 00:24:58,160 --> 00:25:04,890 But it's. Quite interesting and quite informative, and particularly the first one, the Faux Vox, which is a tool for forensic psychiatry. 246 00:25:04,900 --> 00:25:08,590 I mean that was quite good actually when we did the feasibility work. 247 00:25:08,830 --> 00:25:17,020 People then started to use the calculator and into the clinical teams and sort of think, Oh, okay, well that's pretty good, I'm going to use it. 248 00:25:17,440 --> 00:25:27,040 And they sort of. So those are three feasibility studies, a faux vox, one in here in Oxford and books that BMC Psychiatry was in, 249 00:25:27,580 --> 00:25:35,830 in hospital in China and the country was in Sweden and there I think they're all using faux vox regularly now in clinical practice. 250 00:25:36,310 --> 00:25:42,910 So if I was to stand a little bit back and think about the challenges when I think maybe this is three levels, 251 00:25:43,210 --> 00:25:45,370 three things, yes, is developing models, 252 00:25:45,370 --> 00:25:55,809 which isn't really what I've been talking about, but just just I think my impression is it takes ages where it can tell you how to do it properly. 253 00:25:55,810 --> 00:26:00,970 Takes ages. I mean, you can do it and you can do it quickly, but to do it really properly and to figure out, 254 00:26:01,390 --> 00:26:05,410 you know, exactly, you know, getting the protocol right, 255 00:26:05,410 --> 00:26:07,150 what should go into the model and, you know, 256 00:26:07,180 --> 00:26:14,010 getting all the everything put in place and looking at all the various variables, univariate and then multivariable. 257 00:26:14,630 --> 00:26:16,840 That really takes a long time to do it properly. 258 00:26:17,620 --> 00:26:27,340 The validation, I think is is is problematic most of the time, unless you have access to registers of some sort because you just kind of have a scale. 259 00:26:27,820 --> 00:26:31,990 I mean, you may have heard of the rule of thumb of 100 events, you know, to validate the model. 260 00:26:31,990 --> 00:26:36,280 And most outcomes, at least the ones I've been looking at, they're quite rare. 261 00:26:36,940 --> 00:26:41,259 So you you struggle actually, unless you collect the data. 262 00:26:41,260 --> 00:26:47,770 You know, we didn't Tajikistan for a high base rate, which is for offending in that population. 263 00:26:47,770 --> 00:26:51,370 But in most of the time, you know, it's really not possible. 264 00:26:51,670 --> 00:27:00,610 So you sort of almost have to think when you develop a tool to think about predictors that you can validate in registers, you know, 265 00:27:00,610 --> 00:27:08,410 so think about things that are routinely collected because otherwise you may end up developing a tool that you can never validate externally. 266 00:27:09,370 --> 00:27:12,460 And then implementing tools requires collaborators, obviously. 267 00:27:12,520 --> 00:27:15,579 I mean everything because collaborators for this work has external collaborators 268 00:27:15,580 --> 00:27:21,649 usually and and people probably who are looking for something to fill a gap, 269 00:27:21,650 --> 00:27:29,070 a need. And so actually my my impression is that, you know, just telling somebody or an agency, look, 270 00:27:29,080 --> 00:27:32,580 we've got this tool in it, you know, it's really great, you know, and so it's not good work. 271 00:27:32,590 --> 00:27:39,250 I mean, they have to sort of have a need and identify the need among themselves to do that. 272 00:27:40,240 --> 00:27:46,180 And so, of course, you know, that's the bit in the in the pyramid that's very few tools implemented. 273 00:27:46,520 --> 00:27:54,910 You can develop a lot, validate the few more, but implementing them is just the price at the top of the iceberg. 274 00:27:56,260 --> 00:28:00,520 When I think about predictors and outcomes, just to sort of talk you through a few examples, 275 00:28:01,930 --> 00:28:08,079 I think, I mean, one of the challenge with predictors is it has to do with missing information. 276 00:28:08,080 --> 00:28:10,070 You know, something to think about is, you know, 277 00:28:10,140 --> 00:28:15,580 what level of missing this you able to cope with and you know, whether it's at random or not, it's important. 278 00:28:16,660 --> 00:28:23,049 But also when it comes to validation, it's a big problem because like I said, I mean, we are able to cope, 279 00:28:23,050 --> 00:28:32,380 I think in our models where the missing information is for predictors that are not very strong, have strong effects. 280 00:28:32,890 --> 00:28:38,320 But in one of the validations we want to do the missing variable. 281 00:28:38,740 --> 00:28:43,390 Missing predictor was it was the most powerful, it was the previous behaviour. 282 00:28:43,450 --> 00:28:46,749 So whether it's violence or suicidal acts, 283 00:28:46,750 --> 00:28:52,230 it was one of the two and we just thought we couldn't go ahead because I mean the most powerful predictor is missing. 284 00:28:52,240 --> 00:28:59,170 So there there was one missing was powerful. But with the with done in the Thames Valley, I think we had two or three missing. 285 00:28:59,170 --> 00:29:02,780 But they were all very, very they were very powerful predictors. 286 00:29:02,780 --> 00:29:09,280 So actually the model worked fine, didn't shrink very much because they're not very powerful, different definitions. 287 00:29:09,280 --> 00:29:13,810 And I think they're the key thing is to look at the prevalence and see, you know, 288 00:29:14,290 --> 00:29:19,590 whether you can find a prevalence compared to the original development sample, which is similar. 289 00:29:19,600 --> 00:29:25,749 So if you've got I mean, we've had this a lot with mental health variables with some people sort of global school, 290 00:29:25,750 --> 00:29:29,559 it's out of five and is it three or is it two or is it one, 291 00:29:29,560 --> 00:29:32,890 the cut-off that you would use and you would try and base it, in our view, 292 00:29:33,310 --> 00:29:41,500 on what corresponds best to the prevalence and the original sample of the sample nonclinical predictors. 293 00:29:41,500 --> 00:29:49,870 Well, this is the difficulty about using data set. So a lot of datasets within clinical environments. 294 00:29:49,870 --> 00:29:55,359 So you get like in the US you have these insurance datasets, but within health care, if you see what I'm saying. 295 00:29:55,360 --> 00:30:05,670 But if you're gathering information about that. Same educational level or income or employment status, then you're going to get that. 296 00:30:05,700 --> 00:30:13,760 So you have to be aware of that. And it does limit, I think, the power of some tools where those predictors could be quite important, actually. 297 00:30:14,580 --> 00:30:22,440 The previous thing I mentioned and ethical issues are very important to consider because some of the predictors used for some 298 00:30:22,440 --> 00:30:29,610 of these tools are proxies and they're problematic proxies and you have to be careful and there's ways of dealing with it. 299 00:30:30,600 --> 00:30:40,830 You just completely remove anything that might be a proxy or to trying to adjust in some way for the possibility that it's acting as a proxy. 300 00:30:40,840 --> 00:30:44,879 So there's different ways of dealing with it. And I think it's a different question. 301 00:30:44,880 --> 00:30:49,770 It's a different is how it's a whole session of its own because it's such an important area. 302 00:30:51,210 --> 00:30:55,570 Like I say, prevalence in the original episode was worth considering about the direction 303 00:30:55,570 --> 00:31:02,640 and magnitude we need to univariate so in and pre specifying it the outcome. 304 00:31:02,970 --> 00:31:09,540 I mean here's one example of this is a review we did of repeats offending one and two years. 305 00:31:09,540 --> 00:31:13,919 You can see the big range now. I mean, why is there such a big range? 306 00:31:13,920 --> 00:31:18,479 I don't think it's because repeat offending is different things with how you count. 307 00:31:18,480 --> 00:31:20,010 Repeat offending is very different. 308 00:31:20,790 --> 00:31:27,930 And I think that just tells you that, you know, you the how you how you capture the outcomes is going to be quite important, 309 00:31:28,110 --> 00:31:31,980 particularly when you consider if you should recalibrate the model. 310 00:31:32,670 --> 00:31:42,479 So even within one country, you know, you have North Carolina and Oregon and they're probably counting it differently. 311 00:31:42,480 --> 00:31:50,010 I mean, it's the USA and then you have know the Nordic countries, Finland, Norway and they are counting it differently. 312 00:31:50,670 --> 00:31:53,879 So there's ways of counting this particular outcome. For instance, 313 00:31:53,880 --> 00:32:02,400 you don't include fines and fines and or you may not include what they call 314 00:32:02,400 --> 00:32:08,430 someone who's record so someone can be at the end of their prison sentence. 315 00:32:08,430 --> 00:32:09,909 They still have time to serve, 316 00:32:09,910 --> 00:32:16,710 but they can serve in the community and then they can be recalled into prison if they break a rule like not turn up to their appointment on time, 317 00:32:17,430 --> 00:32:20,850 in some cases countenance, repeat, offending, some places don't. 318 00:32:21,330 --> 00:32:25,230 And that will determine part of this variation. 319 00:32:25,920 --> 00:32:32,370 But it's just one example. And I mean, we came across this because, you know, it's one of our outcomes is repeat offending. 320 00:32:33,270 --> 00:32:37,500 And one of the first validation we did, we actually used a non-criminal outcome, 321 00:32:37,890 --> 00:32:45,360 which was any interpersonal violence captured in a very rich phenotype cohort of people with schizophrenia. 322 00:32:45,570 --> 00:32:48,760 But it meant, you know, a culture of recalibration. 323 00:32:48,760 --> 00:32:55,560 And thinking about what to do is not ideal because it's a three year outcome which is different in our paper, 324 00:32:55,650 --> 00:33:00,600 you know, we had a one year outcome to have a three year outcome of a different it was a different outcome. 325 00:33:01,170 --> 00:33:06,480 And and so you need to think quite carefully there about how to how to deal with that. 326 00:33:06,780 --> 00:33:13,260 And like I say, you know, there's different things. This arrest is conviction, this violent conviction and re imprisonment. 327 00:33:13,270 --> 00:33:17,489 I think the problem we had with some of the US data, the US state of Texas, 328 00:33:17,490 --> 00:33:21,450 they only had informational re imprisonment, which is very easy to collect. 329 00:33:21,450 --> 00:33:27,419 I suppose if you particularly have a prison database it's very easy to collect rather than repeat offending, 330 00:33:27,420 --> 00:33:35,010 which doesn't lead to imprisonment, which a lot of it doesn't, particularly if it's less serious than what we call common assault. 331 00:33:35,670 --> 00:33:40,200 And the like I say, I think it requires different cut-offs. 332 00:33:40,200 --> 00:33:48,470 So that's the other thing. So we had prespecified cut-offs but they don't make sense if the outcomes are different, you know, and then, you know, 333 00:33:48,540 --> 00:33:53,879 I think there should be a very clear plan about how you would consider and what you would do to 334 00:33:53,880 --> 00:34:03,630 recalibrate to and experiences that you tend to have to recalibrate tools and new populations and wider, 335 00:34:03,630 --> 00:34:13,380 wider issues and meta issues which may or may not bother you very much, as is when you do it when you develop a model, is who does the validation? 336 00:34:13,380 --> 00:34:18,000 I mean, it's a bit tricky because really no one's got skin in the game. 337 00:34:19,740 --> 00:34:25,830 So it ends up being the person who's developed model, you know, which is which then is not independent, is it? 338 00:34:25,830 --> 00:34:28,480 I mean, because the person who is developed, you know, 339 00:34:29,610 --> 00:34:37,620 is then involved and that has some advantages because they can sort of ensure the some quality, you know, quality control and it's done properly. 340 00:34:37,620 --> 00:34:43,889 But at the same time, there's not truly an independent validation. And that might be too high a threshold to say it should be completely independent, 341 00:34:43,890 --> 00:34:48,719 because why would someone want to validate a particular model unless they have a 342 00:34:48,720 --> 00:34:54,690 particular acute clinical need or practical need for moral issue publications? 343 00:34:54,690 --> 00:35:02,510 I mention that because I think despite what people. To say publishing validations is difficult, so people say it. 344 00:35:02,720 --> 00:35:09,110 So you see journals often saying, well, you know, we've been you know, we look forward to validations of the of these models. 345 00:35:09,110 --> 00:35:13,490 But actually, we found that they're quite difficult to publish. 346 00:35:13,850 --> 00:35:17,959 And it's understandable. I mean, they probably don't attract the same level of citations. 347 00:35:17,960 --> 00:35:24,560 And so journals may be a little bit less inclined towards them, you know, and also, they're not seen as novel because, 348 00:35:24,560 --> 00:35:28,549 you know, the tool is what's novel and it's sort of exciting and there's a new thing out there. 349 00:35:28,550 --> 00:35:34,880 And, you know, at the very least, it's giving you some information about risk factors that probably people didn't really think about. 350 00:35:35,510 --> 00:35:39,200 So there's always something novel, even if it's not going to be interpreted to do something interesting or novel about it. 351 00:35:39,200 --> 00:35:43,810 But the validation is not. And so there is a challenge there. 352 00:35:43,820 --> 00:35:50,810 I mean, there are some journals which say specifically we're interested in validations and you'll see that we published in them. 353 00:35:51,380 --> 00:35:57,070 So they're the ones that we've gone to because they they're more of them. 354 00:35:57,530 --> 00:36:05,740 Our experience has been that they're more interested. Approvals is just a wider issue with doing research is that is that you know, 355 00:36:06,200 --> 00:36:12,140 there are lots of regulatory hurdles and approval hurdles which you have to sort of factor in to your timelines. 356 00:36:13,040 --> 00:36:19,730 So if you want to do a validation study, let's say as part of a project, a master's or Ph.D. project, 357 00:36:20,300 --> 00:36:24,890 you need to be ideally involved in something where the approvals are in place 358 00:36:26,390 --> 00:36:31,640 because getting them up and running and sorting it all out is can be quite a long, 359 00:36:32,120 --> 00:36:37,100 can take quite long, and sometimes it can take longer than the time you have to do the research. 360 00:36:37,760 --> 00:36:43,159 So there's lots of examples of that. I've spoken to people, I mean, you probably know lots as well. 361 00:36:43,160 --> 00:36:47,000 But um, and then just finally just to thank a few of the people involved. 362 00:36:47,000 --> 00:36:54,020 So I mean, early on some of the Swedish collaborators, Susan Mather, who was here now is I see Uclh not at Birmingham. 363 00:36:54,530 --> 00:37:02,330 Tom and obviously Maria have been central to this, the research that we've done and some other people as well. 364 00:37:03,200 --> 00:37:03,920 Thank you very much.