Very much for coming to this talk today. I would like to introduce John Trioli who is here from Aberdeen in the Health Services Research Unit.
I know that Sean introduces background and it should take about an hour and there will be questions at the end.
Okay, thank you. So, so I'm from Aberdeen.
A friend of mine,
a colleague of mine was here last week for the Evidence Life Conference and she said that Oxford was hotter than the surface of the sun.
So it's maybe not quite as hot as that today, but it's a lot hotter than it was in Aberdeen.
So great to be here. I'm a trial methodology.
What I'm really interested in is how we can improve the way we do trials.
So I don't have a particular focus on one type of trial or another, but it's how we do trials.
So I'm going to talk a lot about efficiency, things we might do to learn about doing trials more efficiently,
and also raising awareness of some of the issues right now with trials.
And I want to talk about this thing called trial footage. And what I also want to make clear is that we meaning trial forge,
we think everybody who has an interest in trials can contribute to Trial Forge and making trials more efficient.
So we're always looking for collaborators.
One of the key things with trial footage, which I hope by the end of the talk I'll convince you, is absolutely essential,
is that we do more coordinated activity and we do a lot more collaboration all the ways years and years,
decades even can pass without really knowing much more than we did a decade earlier.
And I'd actually point out some clear examples where that is the case.
So coordination and collaboration is what we're about with Trial Forge.
Now, I guess many of you will know that every year there's something called International Clinical Trials Day.
And a couple of years ago it wasn't this year, it was last year.
An Irish methodology research group thought it would be a great idea to get some people
who work in trials to say something about why they thought trials were important.
And this particular one is from Howell Williams, who is the director of the UK's Health Technology Assessment Program.
So that NIH, our Health Technology Assessment program, one of the big public funders of trials here in the UK.
And he said, look, trials are right at the backbone of primary research that here in the UK underpins what we do in the NHS.
They're right there at the centre.
It is really, really important stuff that goes on in clinical trials and what we do in trials finds its way into systematic reviews,
finds its way into clinical guidelines,
and those guidelines influence the care of thousands or millions of people, not only here of course in the UK but around the world.
They are trials very important.
They say here trials do change lives, they are important.
And that makes it odd that people like me.
Trial methodology is not clinical, but a trial methodology is the way that we and our colleagues make decisions about how to design or how to run,
how to analyse, how to disseminate our results.
Those decisions are informed by almost no evidence at all.
So if we think of something at trial that might change the way we manage one aspect of,
say, diabetes care, that thing, that trial could change the care that patients receive.
It's an evidence based component or is a component of our evidence based system.
We triallists do not have an evidence based approach to how we make decisions we do as this person.
This is a great quote. I say show it all the time from somebody called Monica Shah, what she and her colleagues were doing.
They were asking themselves, how could we better select sites for multicenter trials, cardiovascular trials?
And she made this observation, which is absolutely spot on. We do things because we think that is the right way to do that.
We use judgement, we use experience,
we use knowledge and we are forced to use that to a far greater extent than we ought to because there is no alternative.
We do not have a large body of evidence to which we can go to try and improve our decisions.
So if I wanted to. Develop an evidence based trial management approach.
There is not an evidence base that I can go to to do that, and I'll show you that in a moment.
So what Monica is saying here and she's spot on is it's weird.
It's something so important, as we heard Howell say on his slide.
Something that is right at the backbone of the primary research underpinning clinical care here in the UK and elsewhere,
that the way we go about doing much of the activity is far from informed by
evidence and we have not systematically tried to build up that evidence base,
which is one of the things that we are trying to do with Trial Forge.
So one of the things we want to do with this is to try and systematically build up evidence so that those decisions about how best to recruit,
how best to retain, how should we disseminate our results?
Should we send somebody out on a face to face site initiation visit, or is it okay to do it remotely?
We want to try and provide some evidence to support those decisions because to a first approximation,
there is nothing for any of those things right now, almost nothing.
So we want to try and change that and to do that.
That's why we need the coordination and collaboration, because leaving things as they are right now, nothing much has changed for a long time.
So I mentioned efficiency. I want to think about efficiency in two ways.
Think about it as scientific efficiency and process efficiency.
Now, you could cook this in a couple of other ways, I guess, but this is what we think about in Aberdeen.
And scientific efficiency boils down to having an important question and then choosing the right design and process.
Efficiency is about doing things within your trial in a way that is most appropriate given your research question,
your resources, and what you're trying to do. An important thing to remember about both of these, and particularly the scientific efficiency,
is that it is entirely possible to kill the trial on paper before anything happens.
You can do something here which renders the trial irrelevant.
Nobody will care about it. It is fundamentally messed up from the very beginning.
So it's dead. I'm going to show you an example of that process efficiency.
You can do the same there. You might have a great question, chose the right approach up here.
But some of the things you have chosen to do within the trial, perhaps, for example,
the point at which you try to recruit is so spectacularly wrong that the trial fails, even though it's a great question.
You've chosen the right design, but the process is poor, poor to the point of killing it.
So these two things are important, very important.
And it's possible to ruin the trial from the very beginning.
The key thing I think to think about when designing any trial and Clifford and I were talking about this before I started,
was, who are you designing the trial for?
Who is the person or people or groups that you think will gain benefit from the results of your trial?
And three or five or ten years time? Who is that person or persons of what do they need?
What is the information need that they have? Not you. Nobody cares what I I'm a trial meteorologist.
My I don't provide care. I don't meet patients.
What I think is irrelevant. What is important is who is it I'm designing my trial for.
In my own case, it's almost always clinicians and patients hoping to improve their decision making and possibly policymakers,
people who might choose to pay for this in a roll out if it shows itself to be effective.
And therefore the information need has nothing to do with me.
It has everything to do with those clinicians and patients and possibly policymakers.
So there is a trial that I'm involved with right now where we have what we've called a decision package.
And that package of outcomes is relevant for the funder, which in this case is the Scottish Government,
because they are thinking of should we roll this thing out in the future?
What they want to be reassured of is within this package there is enough information for them
to make a decision in four or five years time that we should roll this out across Scotland.
Well, I think is irrelevant. They are the people making the decision.
That is what we should put into that trial. So we are thinking who is the person for whom this trial is?
Designed. So here's a pictorial way of trying to think about this.
Let's imagine that we are designing a trial. And this trial we have a person or group in mind,
and those individuals are thinking what we want is some piece of information that will remove uncertainty about a particular treatment.
When I'm talking, let's say I'm a clinician, when I'm talking to one of my patients, I have a lot of uncertainty about this treatment.
I want you to provide information that can reduce the uncertainty in that decision, and that already has some implications.
That person is thinking about a particular type of patient.
That person almost certainly thinks of where that care will be delivered.
Might be a general practitioner, which means immediately we think about primary care,
that person will have some ideas about the outcomes that would provide reassurance to him or her that this treatment is the most appropriate one,
not this one. That decision package that I was talking about, he or she, together with a patient,
will have some ideas at what point that outcome or outcomes should be measured.
Do I care if this thing is measured after 30 days, or am I really interested in a much more sustained outcome?
That's what affects our decision. That's the thing we need to know. We might be thinking about the resources available to deliver the intervention.
The pieces of kit that I am expected to use.
I don't have access to an MRI. Therefore, what's the point in building an MRI into your protocol?
I don't have access to. It's no good. So let's say overall, if we take those things together,
the information need that that individual has together with his or her patients looks like this piece of vase of flowers.
That is what the trial should look like.
So that's the ideal. This is the trial.
Now, if we think of how long the trial will take, two years.
We could burn our way through a few million pounds, involve five or 600 people.
And at the end of that, what our user is looking for is this.
That's the thing that he or she was thinking about. If we deliver this, it is not quite perfect, but it's is pretty good.
Doing trials is tricky. There are constraints.
There are compromises we have to make. We only have so much money.
We have to recruit within a reasonable amount of time to inform the decisions that those people might want.
There's no point spending 20 years, perhaps, if a decision has been made in three.
So what is delivered is unlikely to be perfect, but that is pretty close.
If if I was one of the decision makers here and I got that, I think, yes,
they they have worked hard to deliver something that I can use in my decisions.
It's not perfect, but I understand why it's not perfect.
They've told me that it's pretty good. I now have a bit less uncertainty when I make my decisions together with my patient.
Not that that's what we would like to see happening.
Sometimes that happens. However, this is what I think happens far too much.
This is the picture that we're aiming for and a trial team spends time doing it.
Trial design runs its trial recruits to 500 people, spend three or four years, burns its way through a couple of million pounds and delivers this.
This is a perfectly respectable salad.
It looks great. It is not flowers.
It's a thing of beauty in its own right. It is, however, completely irrelevant to the decisions that those users have.
They want this. They don't care about that. It might be beautiful to the researchers, but it is not addressing that need.
The information need to reduce the uncertainty that that clinician had together with his or her patient.
Your three or four years of work wasted as far as that person is concerned.
Let's look at an example. This is a a nice and sad piece of work.
In equal measure, this is a group who did a systematic review over a decade ago,
and it was looking for the choice of analgesic for a particular procedure.
And what they did is they did a systematic review. They found about 50 trials.
And the result of those 50 trials was clear.
It showed that one analgesic in particular, lignite cane, was the one to use because it was effective, it was cheap and it was easy to use.
So in that review done over a decade ago, they said lignite came.
This is the one to use. It's cheap, easy to use. It's effective.
Future research should compare whatever it is that's being proposed with ligne k.
That is the clinically relevant comparison because we already have an effective, cheap, easy to use analgesic in the form of ligne k.
Now this particular publication has fast forwarded a decade and they repeated their review of what they were
particularly interested in was how much influence that systematic review the earlier one had on the design of trials.
And they were particularly interested in the choice of comparator because the choice of comparator came out so clearly from that earlier review.
So they had a definition of clinical relevance, which was the thing on the test is compared to ligne k,
why would we not compare it to an effective cheap, easy to use analgesic?
So when they look for the new trials that have been doing that decade,
they found 136 new trials and they concluded that 87 of those trials asked a
clinically irrelevant question because they chose a comparator other than ligne cane.
So eight of them, I think it was eight chose placebo.
So if you imagine a clinician who's wanted to make a decision as to whether the new whizzy thing is better than ligne cane,
which he or she has been using for a decade. And what you say is, well, the new thing is better than placebo.
What he or she is going to say is, I do not use placebo, I use ligne cane.
How does it perform against ligne cane? And the trial team are going to say we have no idea because we didn't compare it.
Telling the cane and the commission is going to say, okay, I'm not interested.
Lovely, bye. By 87, two thirds of all the trial activity in that particular area over a decade wasted, clinically irrelevant.
Almost certainly all of those trials were trying, or most of them trying to inform a direct, immediate clinical decision.
And by making one choice, they have rendered them irrelevant.
So these reviewers here were deeply saddened at the lack of influence that systematic review had on the direction of research over a decade.
And this is one little area of health research. My sense is if you were to do that in many other areas, you would find exactly the same picture.
Indeed, there are a group based here in Oxford who did a survey asking trialists about
relevant systematic reviews and the awareness they had before designing their trials.
And half of them had no idea of the existence of relevant systematic reviews before they designed their trials.
In other words, they designed their trials in complete ignorance of things like this.
Very easy in that situation to make design decisions which render your trial irrelevant to the people you think you are trying to support.
This is not a new revelation. So this is two French statisticians, 1967 said this.
Look, many most therapeutic trials are messed up.
I paraphrase messed up from the very beginning.
The inadequacy is basic.
What they are saying here is that you do not statistic your way out of these errors.
You are stuffed, you've messed up, and you do it from the very beginning.
The way it is formulated is wrong. So you can do that on a piece of paper before you come anywhere close to a participant.
And it doesn't matter what happens from that point on. Messed up that was said in 1967.
Some of you have probably heard of Doug Altman, who is based here in Oxford,
and he wrote a paper, I think it was 1994 called A Scandal of Poor Medical Research.
And he spoke in Oxford last week about that very paper.
And the only regret to, I guess, one regret, he does not think there's been any change since 1994 in the state of health research.
And his only regret is that when he wrote the scandal of poor medical research, that he did not call it the scandal of bad medical research.
Call it what it is. This is bad.
That paper was voted a few years ago, was the most significant paper published in the BMJ in 20 years highlighting this sort of problem.
And it continues and will continue to continue unless we do something a little bit different.
So one of the sort, they start talking about some of the things we might be able to do.
So one of the things I think we can. With the design issue is just be a bit more careful about how we go about our design decision making.
This is one example. A colleague of mine, Kirsty Loudon, so she developed something called Tracey too.
She was a Ph.D. student of mine, so I'm biased. But the point of this tool is to make trial teams think about who am I doing this trial for?
What are the sorts of things that they require from my trial?
And am I doing those things? And it's just a will you work your way around this wheel?
There's a nice description in a paper. There's a website to support it.
If you print out a little crib sheet, there's a sort of eight page crib sheet, four of which is worked example.
You can get the bulk, I think, of value from this having read four pages.
It's easy, I think, to get your head around. But it asks things like Who should be in my trial?
Who say, Where should I get those people from? What sort of outcome should I be measuring?
What should I throw at the trial to make it work? How often should I contact the participants?
Very simple. This is not rocket science. This is not some resistance listicle technique.
This is just thinking through your decision making. Do it on paper.
Involve the whole trial team, which is what Kirsty has done in later work.
Involve the whole trial team. It's quite common to find that people who are members of the same trial team have different views
about this type of trial they are developing and it's good to find that out early rather than late.
So I think that's one sort of thing. It's not rocket science.
It's basically a structured way of thinking through the consequences of our design
decisions before we get stuck into the job of doing the trial and doing the trials.
We're going to talk about next with process, and I'm going to talk about some favourite topics of mine because these are what
we focus on in Aberdeen and the first of these is participants recruitment.
Everybody is terrified about recruitment for good reason. If you don't have any recruits, you are stuffed.
It doesn't matter how wonderful everything else is. If you don't have participants, you have no trial.
Now, if we look to try and think about how many trials there are on the go at any one time a group have.
A few years ago they published an article and they thought something called A Bastion,
and they thought it was about 25,000 new trial publications every year.
To a first approximation, every one of those has to recruit participants.
There might be some odd database ones, but let's say first approximation, 25,000, all of them recruiting Cochrane,
an organisation which many of you are no doubt familiar with, but they try and collate evidence on trials.
In particular, they have a database of things that are called trials types of study.
There's about a million on that.
So let's say there are a million trials, 25,000 new publications every year, 25,000 almost certainly an underestimate.
It's a doubt it includes lots of pharma stuff, but essentially there are a lot of trials.
All of them need to recruit. So you would think that there would be a vast quantity of literature that each of
those trial teams can look to to make their decisions about recruitment better.
This graph is very common. This is from one of my trials.
The red one is projected recruitment. This is how we thought things were going to go.
And the blue one is actual recruitment. And what generally happens is they diverge at some point never to come back together again.
In this trial. It worked out fine in the end, but we got an extension. About half do doesn't always help.
About half of the really best trials hit their targets.
So now people have looked and they tend to look at the best trials. About half of them hit their target.
This is a very common problem. Our projection might be hopeless, entirely possible.
Something goes wrong here, but this is very common for this particular trial.
This is a picture which others here may also recognise.
That trial had five sites and this is a graph of recruitment per site.
So what is obvious is what is happening at sites four and five.
And if we had a graph that plotted energy and tears against site, then that would be reversed.
These would be really high, these would be much lower. So those two at the end soaked up a lot of energy.
And what would be an obvious question to ask is why on earth did we choose those?
Could we have predicted that? An answer is, I don't know.
We don't have any tools really that can help us with that.
I'll talk about something later on. But this prompted but this is very familiar.
And yet trial teams do not have something easy or good that is evidence based or evidence informed, at least to make the choice of, well,
we can not bother with four and five, we need to find some others because they are never going to perform to try and avoid this situation.
Now I look after the Cochrane review of interventions,
things that trialists have tried to improve recruitment with, and I've been doing this since 2006.
So this one picture I show here is from 2010.
We've just submitted an update. It's taken us ages to just submitted it now for this one, the 2010 one.
If we were very kind, we would say that since the time of the Romans,
we go back there as far as we can and we have reasonable evidence for three
things that trial teams could try and reasonably expect to improve recruitment.
And the reality is there's only one of them that could be widely applied.
So if you think a million trials have happened with adding an extra 25,000 at least every year, all of them have to recruit more or less.
And we have one thing in 2010 that might help them recruit, which we can anticipate them being able to apply to their trial.
These other two are a bit odd. One of those things, and I'll mention what it is in a moment, so let's jump forward to this new update.
So it's not through Cochrane yet, so we've just submitted it. This is what it looks like right now.
So in 2017, we have moved from 45 included studies to 68.
They have compared in that bunch of studies, 72 things.
It involves more than 74,000 people.
And if we kind, we have reasonable quality evidence for three things that trialists could use to improve recruitment.
And this this are the same as before.
This one is the one that is the most promising.
And what it boils down to is if you have posted a letter to a potential trial participant and that person has not responded,
why don't you phone them up? That is the pinnacle of trial.
Methodological research with regard to recruitment, phone up people who don't respond to a letter that's as good as it gets, give or take.
This one is about having all trials as being open so we stop blinding trials.
Difficult to think that that will be widely applied, but there is something interesting developing here.
So this is the key thing and it brings us on to collaboration and coordination.
So what this is, is a very involved way of developing participant information, leaflets costs about £10,000, does a lot of PPE, a lot of iterations.
And you get a participant information leaflet that is very is co-produced together with potential participants.
And as a recruitment intervention, it might have a very small benefit of about 1%,
although there is in the confidence interval the risk that it's doing harm. So you might have a small benefit.
The difference between these and that one that I think is most important is that here these studies were both done in 2004.
Here one study was done in 2000 for the user in 2013.
And here these three studies with far more participants were done between 2014 and 2017.
And the reason we have three here in a relatively short period of time is because of coordination and collaboration.
So a group led by Peter Barry in Manchester, coordinated activity on this and then trial teams collaborated.
There are more trials to come because I was involved with one which is not in there yet.
In other words, with collaboration and coordination,
you can get to some real confidence about whether this thing is effective or not in a relatively short period of time.
Whereas here we have a decade pretty much that has passed between the two studies.
They come along every now and again, but not very often less left to their own devices.
There isn't much replication.
And indeed of the 72 comparisons in this review, there are only seven where there is more than one instance of that intervention being tested.
So the majority of them are tested once and it's hard to have confidence in something that has been tested once.
All of this about poor recruitment has a cost. This is a really nice study from the US one medical centre.
They asked how much do all these failing studies cost us? So they looked over a five year period found.
260 trials that they shut down because of poor recruitment and poor recruitment for these with no participants or one.
So we're not even thinking like they recruited 20% of their target.
They recruited zero or one, and then they were shut down 260.
And that cost them $1,000,000, $1,000,000 to just go through contracting, go through ethics, go through site initiation.
And then it was shut down with that most one participant per trial at most.
It's an awful lot of work for no reward.
And in a sense, you can have some sympathy with those trial teams when the only substantial thing to offer
them with regards to an evidence based recruitment technique is if you've posted a letter,
why don't you phone them up if you haven't heard it?
I mean, that's in terms of an evidence informed recruitment strategy, if that's all they had to go on.
Perhaps not surprising that we are burning through large quantities of money with studies that fail to recruit.
That's what we want to try and do better if we work very hard to get people into our trial.
We then want them to stay. So retention is something we are focusing on.
In Aberdeen, it's hard to recruit.
You really do not want your participants to then drop out.
So this group have done a Cochrane review and oddly enough they find three things if we're kind
that would be effective and they all boil down to paying people to send back questionnaires.
So if you were doing a trial that involved a face to face visit, somebody has to come in to your office and give you some data.
There is nothing in that review that's going to say this is something you could use and reasonably expect it to improve your recruitment.
And let's think,
just imagine how many of those 25,000 trials that will be published this year used face to face visits as some component of their trial.
There is no good evidence to support an effective recruitment retention strategy that involves face to face visits.
It really ought to be given how common they are. Now, a little bang on about data collection.
This is me on a step ladder. And the point here is we collect a lot of data in a trial.
So in this particular talk, I'd worked out how tall the pile of paper would be in a particular trial.
We printed out everything that we collected on a double sided paper.
It would be just over three metres, which is slightly higher than I could reach on that step like that.
If you've collected three metres worth of paper, that's a lot of work and you really want to make sure that those data end up out
there in the public domain informing people's decisions if it's in a cupboard.
Well, nobody's health is going to improve from data in a cup. It needs to be out there in one form or another.
This is a nice study, the only one I've seen so far that has done this sort of thing where they asked how
much of our data they did this for their own trials ends up in the public domain.
So all cancer trials, they found eight. They looked at this portfolio of eight and they did something quite simple.
Here's a data collection form. Here are trial publications.
Which items of data on here find their way here on the data collection form?
Is it in the public domain? Does anybody?
And what they did then is work out the proportion, and then they worked out a median across the eight trials,
and then you would want to have a guess at the proportion of data that were not used.
A median proportion. I say 80?
Yeah, 82.
So I don't know whether they collected three metres worth of data, but if you had a three metre pile and 80% of it, you know it's up here somewhere.
Then we're down to this bit that's being used. What is happening to that other 80%?
Now some of the data are never really for publication, so there might be some tracking data.
So we have another project where we're looking at the proportions of different types of data that are collected.
Some of it is tracking, some of it might be processed data saying, yes, we have got informed consent, things like that.
Some of it is safety data, but it's probably not 82% or anything like 80%.
In fact, we know from our or the project is nothing like that.
So if you in the other project, 40% across this is five trials, 40% of the data collected are secondary outcome data, for example.
So there's a lot of outcome data, health economics take a big chunk to this is something I think we should all think about.
Will we be able to get these data out into the public domain?
Because if we cannot, well, why why are we collecting it?
Because that will perhaps causes recruitment problems. It will almost certainly causes retention problems.
Who wants to fill out a huge form anyway? If we can make it shorter, then why wouldn't we?
So this statistic should be in our heads, I think whenever we are designing the data collection for a trial on the same theme,
this is an MSI student doing some work for Trial Forge.
What we've asked him to do is how much time the trial team spend collecting the primary outcome data,
the thing that they themselves have defined as the most important piece of data compared to the secondaries,
the outcomes that they themselves have defined to be of less important than the primary.
So what David Pickles, the MSI student,
has done is to look so far at 20 protocols and figure out how much time the trial team spends on primary outcome data.
And that's, as you can see here, a lot of very short orange bars.
And the reason they're short is because the next slide, I'm going to show you the secondary outcome data, which is this.
And the overwhelming impression here is there's a lot of blue.
So these trials across the PE, not quite every single one, but most of them spend more time and sometimes much,
much more time on the outcomes that they themselves have defined to be of less
importance than the outcome that they really think of as the most important outcome.
This one here, which is the worst offender and is a trial I am personally involved with.
The ratio here is 1 to 32. For every hour they spend collecting primary outcome data, they spend 32 on secondary outcomes.
If you do it across, the piece is 1 to 6, one hour on primary, 6 hours in secondary.
And I guess the point here is not that we should collect always just one outcome.
We should think carefully about the workload distribution within our trial.
I was also saying to Clifford what we tend to find in Aberdeen at any rate, is that if things are going quite badly with response rates,
we very quickly find that actually a one page primary outcome questionnaire suddenly becomes enough.
If we can get that back, we're happy.
And then you might ask yourself, Well, I wonder why we weren't happy with that at the beginning, or something at least similar to it.
We're happy now.
Maybe we could have been happy with something similar to that right the beginning and saved ourselves and everybody else a bit of pain.
So it's just to think what's the distribution?
Is it appropriate for our trial?
This again, is the sort of thing we're trying to fix or attempt to help, at least within trial force raising some of these issues.
Do we really mean to have that sort of distribution?
I don't think we do. We certainly didn't do it mean that in our trial.
So I'm going to try and wind down a bit towards the end next.
So all the things that we are doing, recruitment, as I said, we really focussed on recruitment.
We talked a lot about this review. This is the randomised evaluations, the one where we have,
if we're kind three things where we might expect some sort of benefit with regards to recruitment.
So that's just been submitted the update.
But there are other reviews which we think are equally important and we should stop thinking of this on its own.
So the other one that's up here is to do with factors.
What influences recruitment, what are the things that affect the ease or difficulty of recruitment?
So we have a collaborator. In Galway, Catherine Whorton, who's leading that.
So we work with them on that, but they lead that. What are the things that influence recruitment while you're doing this?
I noticed that we were rejecting a lot of non randomised evaluations of recruitment interventions.
What they tend to be is what my PhD student Heidi and I have now come to call yield studies.
We tried eight things to recruit participants.
This is the number of participants who came in via each technique.
There's an awful lot of those sorts of studies. So we thought we should look at them to see if there's anything useful that we could find.
And I actually was quite optimistic about some types of intervention because I thought they'd be enough.
So you might worry about the the actual design.
The approach is non-randomised, but there would be enough studies in there that we might be able to see something.
We found over a hundred studies Heidi has in that review.
And without wanting to pre-empt too much of what Heidi's results will be, too a first approximation, these have no useful value whatsoever.
They are done spectacularly badly reported in a way that means it's impossible to draw any useful information from them.
In other words, a complete waste of time. Almost all of them.
The worst offender. Just because it amuses me. I'll tell you what it is.
It's a paper. It's four pages long. They used eight techniques, I think it was eight, and they recruited 13 participants.
And then they spent time writing that paper.
Now, if you're designing your recruitment strategy and somebody has recruited 13 participants using eight strategies,
whatever they write is of no use to whatsoever.
It doesn't matter what they say, it's irrelevant. And yet it's in there.
That's the worst offender. But there are an awful lot like that.
It's hard to see what others are to take from those results.
But we want to link these up. So Heidi's wrote the protocol in a pretty amusing way, where one of our possible outcomes was that we would say,
Please stop doing these, and that is likely to be one of our conclusions.
Please stop doing them. But we might punt some over here.
There are some which are done quite well. I basically have them linked.
We should be looking at things perhaps over here.
Maybe we can do randomisation for a particular reason, but it looks promising.
So let's then suggest that we from that review we say we need a randomised evaluation.
We might have randomised evaluations up here which suggest that there are some barriers and facilitators which can be targeted by these.
They seem to be effective against those facilitators to a first approximation.
All of these are not specifically linked to anything in particular with regarding a factor they just dreamed of somebody had to first approximation.
So that linkage is not done.
And we we think this is something we could use for lots of processes within trials, factors, different types of evaluations and link them.
So we're doing that also for retention, and we think we should start thinking much more about these as a trio.
A Trinity is what Heidi calls and of reviews, and that they are linked.
They're explicitly linked. Now,
something that has come through directly from trial footage and this is something which I'm this that you'll
see it's it's a tiny little piece of the trial world but demonstrates something which I think is important.
Remember from that Cochrane review, we had things where a decade passed and nothing much came about to replicate a result.
Now here we are because it came from Aberdeen and Dundee.
So is a Scottish intervention. We wanted to improve response rates to a questionnaire and the innovation that came then here from Dundee.
Debbie Bonetti, in fact, is a psychologist.
What she said was, we should think of returning that questionnaire as a behaviour and our cover letter that goes together with that questionnaire.
We should think of that as a behaviour change intervention.
So this over here is a template that Debbie produced based on psychological theory as a behavioural change intervention,
and they use that in a dental trial and it increased the response rates by 6%.
So an absolute increase of 6% for something which was the cheapest chips to do and you were going to send a letter anyway.
Doesn't make it longer and we're not much.
Maybe a page long, but small useful benefit for essentially no additional cost.
That was promising. But it's one study.
So what we did with the trial for IDEA was can we coordinate some activity around this particular intervention?
Can we get other Triallists to test this? In their trial.
So we asked all of these of we asked lots of people.
These are the ones who said, yes, we're interested. We can help you.
If we have a trial that looks suitable, we'll help. So they were all positive.
And then this is how things stand right now. It's a far from I think it was 2015, 2014, maybe the first trial that Aberdeen won with Dundee.
They had one result and now we have four. And if we look at these sort of pictures here from the ad and the plot,
then this is our overall estimate and it's looking like there's a small benefit, nothing spectacular,
but something small potential possibility that it's doing some harm, I think is there's still grounds to do one or two more of these,
but these were done them to get 2014 to 2017 by a bit of coordination, a bit of collaboration.
Then we end it with this picture, which for what is it?
65 of those interventions in the recruitment review are still waiting for evaluation number two.
So with that coordination, that collaboration, we end it with this picture in a relatively short period of time.
Of course, because this is retention, you also have to wait for the trials to reach that point.
So we can't speed some of these things up beyond that limit.
But this, I think, is the way to go. It's a tiny little piece of the trial pathway.
A couple of more studies I suspect will say, yeah, we've nailed that very small but useful benefit.
Hard to see harm. Just go ahead and do it.
It's one of those options you might want to think about it. Cheapest chips.
You're going to send a letter anyway. Do it like this might improve response rates by 3%.
Why not? Potentially for dental trials in Scotland?
They've already made that decision because they are already enough satisfied enough by that.
But this, I think, is the collaboration that we need to do if we want to improve the situation.
That day Doug Altman was talking about in 1994, his scandal of poor research, this lack of evidence that we have for much of what we do.
And then this is the last one.
So you remember the graph that I had about site selection where there were three bars here and then two flat ones over here,
three sites doing all the recruitment. We had two sites in addition that didn't do very much.
So in Aberdeen we're thinking, can we do that better?
Can we make those decisions in a better way? And the trial managers suggested this idea.
So the trial managers came up with this idea for their project.
And what they're doing essentially is at the point at which they sign off a trial for sorry, a trial site.
This site is now ready to recruit. They made a prediction.
So we weren't stopping any trials going ahead, any sites going ahead?
We were just asking the trial managers, okay, you've now got that site ready to go.
Do you think they will hit their six month target, yes or no?
And why? That's all we asked them to do. We didn't give them any guidance or suggestion.
We said, well, they hit that six month target. Why? And then that ran for a while.
And this is what we got there. Ten trial managers got involved in that across seven trials and they made 39 predictions.
So we got a fair bunch of predictions. And the bottom line is, over here in that bit of the study, if those trial managers said,
yes, this site will recruit, well, then it might or it might not by 50, 50, more or less.
If the trial mind just said no, this site will not recruit.
Well, they were much better at that, so the odds were that was going to be correct.
So where we are right now, remember, there's no guidance given at all.
We just ask, do you think they hit the target or not? But with nothing.
If a trial manager says no, we ought to sit up and listen because chances are they'll be right.
It's not perfect, but the chances are there's something going on there that is worth thinking about up here.
Less so. So the next stage is we got a lot of qualitative work.
And what we're thinking about now is can we give a couple of steering questions?
Will this site hit its target? You might want to think particularly about this and this.
So we have a body of qualitative work that has come from the reasons why, plus some focus group work we did with the trial managers.
So what we're hoping is that by a little bit of steering, we can get the trial managers to reflect on a couple of things.
I don't know what those couple of things are, but that's that's the hope that will hopefully increase both of these.
And the reason I'm I like this idea. There are no complicated forms.
There's no extra data collection. We don't send anything extra to the sites.
We use the trial manager expertise. They're essentially asking the person who is tasked with setting up the sites, what do you think?
No extra work because they're thinking that anyway,
we just might want to give them a little bit of a steer to think about a couple of things in particular just to increase this.
But they hope the idea there was not to increase the burden of sending stuff to
sites because there are some very long forms that sites some some centres do use.
We don't want to do that. We've got this expertise. Let's try and tap into that.
So to finish off to a very large extent within trials, we do things the way we do them because that's how we've always done them.
And really, we're often forced into that because there is no alternative, there is no systematic evidence base available to tap into.
And that's one of the things we want to try and address with Trial Forge Collaboration.
Coordination is absolutely central, I think, to changing that picture.
And if anybody is interested in, okay, we've got an idea at our site or our centre,
then we would be interested because I think it's the ideas that come from the ground which are going to be most useful.
And also to do it collaboratively, collaboratively in a coordinated way means that we might get some answers much faster than we would otherwise do.
So thank you very much for your attention.