[Auto-generated transcript. Edits may have been applied for clarity.] Welcome to you all. So there will be a mixture here of welcome to people on the course that's going this week on the practice of space healthcare.
Um, and also welcome to people across the wider program within the ABC program as well.
So thanks for joining us as well for, uh, this talk, um, which I'm happy very much happy.
It's going ahead. Uh, personal, personal interest and personal conflict.
Uh, I was our seniors, um, one of his default supervisors, uh, and was keen to sort of,
um, share some of his novel work, um, which has now been published.
I'm sure Sonia will let you know where where you can find some of this, uh, information that he's publishing on, uh, that he's talking on today.
Um, also, now, I'll let you do your introductions. It's always nice to hear from the speaker themselves.
And then I think we can, um, we can then get straight in. We'll have some, you know, time for a presentation.
And I'm sure Sony has built in some time for so many questions, uh, at the end as well.
So thank you very much. And I hasten to add that David was my favourite DFL supervisor.
I just have to throw that in there.
Uh uh, so, uh, my name's Presidio, and uh, as you heard, I had just completed last year a DPhil in evidence based healthcare, uh, at Oxford.
So I'm very proud to be back. My background is in physiotherapy, was a physiotherapist for 30 years.
Um, so it was my first DPhil. The other one is in, uh, exercise and physiology research, Alzheimer's disease.
So I've got a long history with clients with, uh, complex interventions of all sorts.
But I want to talk to you about today is, um, something I care a great deal about.
And it's not just the what in the clinical trial, but also the how.
And we're going to talk about intervention,
fidelity and what that means for how we make sense of the results that we get from clinical trials in, in health care.
So I'm sure that many people here come, uh, cook every day. And we've all had the experience of trying to follow a recipe.
One of the things you know is that some recipes are easier to follow than others.
Some recipes might have one ingredient and two steps, and it's surprising how difficult it can be to actually hard boiled egg and not have rubber.
Um, on the other hand, some recipes are a bit more complicated.
They might have significantly more ingredients, like a Charlotte room with 19 ingredients and several technical steps.
Health care interventions can be quite similar.
There are some trials, some treatments that have more ingredients, more procedures and more components than others.
So for example, compared to a pharmaceutical intervention.
And it's not to say that those aren't complex, but in terms of what happens during a trial, you know, the way that they're delivered.
Other interventions tend to be more complex, for example, surgery, rehabilitation, behavioural medicine.
They have multiple components that might influence each other, such as, for example, the skill or the experience of the provider.
When we think about trials, that involves surgery, for example, if it's a new surgery,
whether the surgeons in the trial have experience where they can influence the outcome,
they might get better at performing it over the first 15 and 20 or 30 patients, and so their proficiency over time may influence the outcomes.
You can also have this mixture of the participants characteristics, and then participants will take to the intervention more than others.
There will be a lot of tailoring adaptation you might require for some participants.
Someone, for example, who is very thin, someone who's living with obesity.
Someone who's much older or much younger. You'll have differences in dosages and durations and sometimes even differences in context.
And one of those things that's also very interesting about these complex interventions, these, you know,
um, surgery, rehab, interventional cardiology is that some of the treatments take place over time.
It's not just one treatment, it's one series of treatments 3 or 4 times a week for a few months, for example.
It's one of the reasons I always say surgery is not a treatment. It's a profession.
Physiotherapy is not a treatment, it's a profession. Um, they are made of multiple treatments.
And because these complex interventions have multiple ingredients and multiple steps, you get more scope for variation, um, for unexpected results.
If anyone's a fan of the Great British Baking Show and I'm, I am horribly addicted to it.
You can get the Paul Hollywood stare. Sometimes what you ended up with was not quite what you intended with.
Now, the interesting thing is to try to figure out what went wrong.
We often know whether an intervention succeeded or failed to achieve its outcomes in a trial.
We don't often know why are we don't often investigate the how or the why as much.
Sometimes it might be that there were modifications to, you know, the recipe as if we were cooking.
Sometimes there might be a difference in skill and technique. All we know is that sometimes we have a result.
Someone, for example. I like to liken this to the reviews you see on a recipe online.
Someone said I never felt made something that was worse. It was horrible.
It was just ready for the bin. But when you start digging in, you start seeing that perhaps some of these reviews were followed.
People who didn't follow the directions and some modifications matter less than others.
So, for example, someone saying, well, you know, the only change I made was using two 14 ounce cans instead of one 28 ounce cans.
That's probably not going to make any difference. However, you might also start finding that other things were changed that would make a difference.
Hey, these biscuits were lovely, but they were a bit dry. Then again, I baked them for twice as long as they were meant to be.
Very likely that might have been part of the problem. Or I didn't have any pastry, so I made I use sponge fingers.
I don't like cheese, so I use some fruit and jelly. I thought you should have cream. I said put where cream on it.
In the end, they didn't really taste like cheese straws. They were more like a trifle.
So these are things you see all the time in cooking, but they're also things that can happen in a clinical trial.
So what was it? Was it about recipe? Was it a poor cook? Was there a different recipe?
All of these are the same teleporter that we share in clinical trials.
Again, we know much more about whether they that's the intervention or the trials succeeded to attain the outcomes and why.
All of these things can have an effect. Sometimes we don't know what's the treatment, just ineffective.
Some treatments will just not be effective because of the way that they're designed,
or because they can't target the condition or the function that they're meant to.
Or sometimes it can be ineffective delivery. You've got a wonderful treatment that could succeed, but it wasn't delivered well.
Could it be that they were Co interventions?
You had a wonderful treatment that wasn't delivered well and someone modified it and said, you know, I'd like to add a little this and a little that.
And so you're getting a multitude of treatments mixed together and sometimes it's just the wrong treatment for that patient.
What I want to talk to you about is just that, that intervention, fidelity.
And you might wonder what fidelity is other than a bank intervention.
Fidelity. It's the degree to which an intervention in a trial was delivered as it was intended in the protocol.
You'll see this by many names. It's a rose by many names.
It'll often pop up in the evidence base as treatment integrity or intervention integrity or adherence compliance, concordance quality.
It goes by many names, but but the basic underlying definition is did the intervention deliver it and trial sufficiently resemble what was intended?
When we have a high level of fidelity, we have greater confidence that the outcome we see was due to the intervention,
less likelihood that the outcome we see was due to some form of contamination or some introduction of Co interventions,
some form of bias that was introduced by that.
We have greater confidence that there's a lower likelihood of confounding by introduction of random variability,
because you're adding things into the mix that weren't meant to be there. But this is really important as well.
You have a higher confidence and a higher chance of reproducibility and effective implementation.
One of the biggest things that we have difficulty with in complex interventions is difficulty in reproducing, uh, interventions.
If anyone has ever, uh, read through a paper thoroughly and looked at what was done, it's very difficult sometimes to know exactly what was done.
That's a matter of reporting to some degree, but it also influences fidelity.
It's very possible that if you can't tell what was done by reading the paper,
that somewhere along the way you may not have been able to reproduce it even in the trial.
Now the other wonderful benefit of fidelity. It also limits type one to type three errors.
And you might think type three is this new. This is something in Canada that we don't have in the UK.
We thought there are only two types of errors. So my best way of describing this is think of the boy cry wolf if anyone remembers that,
um, that story if we took the null hypothesis, is there is no wolf.
And we took an alternate hypothesis and said, no, there is a wolf. I'm going to prove to you there's a wolf.
A type one error is that the villagers believe the boy when he says, you know, Wolf, but there's actually no wolf.
They're rejecting the null hypothesis incorrectly.
A type two error is that the villagers do not believe the boy when there actually is a wolf there, rejecting the alternate hypothesis incorrectly.
And you know, there go the poor sheep. The top three areas.
The villages don't believe the boy when there is no wolf, but they do it for the wrong reason.
Like for instance, saying, well, there's no such things that will. This world.
So all of these types of errors are things that can be heightened by having poor intervention.
Fidelity, again, because you don't know what's actually contributing to the outcome, you see.
And throwing spaghetti at the wall. One of the things that we have with port facilities, we just don't know what's stuck to the wall.
We have no we very decreased ability to know what actually contributed to the outcome.
Did the intervention fail to achieve the outcome,
or did they succeed because of the active ingredients that came with the intervention or active ingredients we added by modifying it?
Were the things that were added, things that were left out? We just have decreased confidence in the findings of individual trials.
This really starts to matter that when we can think of systematic reviews and then when we think of meta analysis, when we start compounding,
when we start adding all this variability together in pooled analysis of data, and even more so when we think of evidence based guidelines,
because those are going to depend on the systematic reviews and that analysis.
Now, you know, we keep talking about this delivery of an intervention.
How we follow this recipe is that only by following recipes is it really about how we deliver things.
Well, no, it's not just about how the intervention is. Does this a lot of fidelity also deals with the participants, what participants do.
And most of these interventions and complex interventions involve some active participation, some active steps taken by the participants.
They have to carry out the intervention or enact the intervention, avoid certain behaviours, follow certain behaviours and that's adherence.
And that's we'll talk about how that's a part of fidelity also. But that really tells you what the participants did,
how they follow the directions or the images that were given to them as part of the intervention in the trial, for example, maintaining randomisation.
There are some number of trials that have looked at the outcomes, for example,
of a physical activity promotion program or the outcome of a surgical intervention over time.
And in some of these, what you see is that you have a crossing over.
People who were meant to have only surgery also went to rehabilitation on the side.
People who are meant to only exercise three times a week thought, well, it'll do me good.
I'll exercise seven times a week. This failure to maintain the randomisation breaks the fidelity, and unlike fidelity, it's known by many names.
You'll see compliance, concordance, any number of things in the literature.
So this brings us up to the problems, to the gaps that we have. You know, mind the gap.
We have many definitions in terms of ability and adherence.
It makes it hard to know what we're talking about at times. We have this possible possibility of these overlapping components and concepts.
What what's fidelity. What's adherence. What counts, what doesn't.
Sometimes we just don't know how to conceptualise it. If I can't tell you what it looks like, how do I monitor it in my trial?
Um, and sometimes you have this conflict between the theoretical side of fidelity.
Okay. In theory, it should do this. In theory it should do that. But what does it actually do in a clinical trial?
Well, is there anything I can measure? Anything I can see and feel with my hands?
Um, and what does it do, for example, in these very complex trials that we're looking at,
for example, you know, surgery and physiotherapy and behavioural medicine.
So what I'll tell you is a bit about some of the research that came out of my DPhil.
And my DPhil is called Faithful but Flexible. And it was all about intervention ability in these complex interventions in health care.
And it was a big DPhil that took place between 2018 and 2023.
And really what it did is that it helped to frame fidelity, to come up with an empirically based definition of intervention, fidelity,
and then hearings to come up with a framework for how to how to use it,
how to conceive it or monitor it in these complex interventions and what I would call the physical domain.
So things such as surgery, rehabilitation, physiotherapy, occupational therapy, speech therapy, things that have a physical component to them.
It also came up with an assessment tool that we'll look at later.
An important thing is that to quantify the effect of intervention fidelity on the treatment effect estimates, we get in a randomised controlled trial.
And it actually gave us the direction of bias.
Sometimes we know that something's going to influence something, but we're not certain in one direction.
We'll talk about that coming up next. But we're going to see how this all played out and in the detail.
We also saw how it played out in real time in an ongoing, pragmatic,
randomised controlled trial comparing surgery to rehabilitation for people with a knee ligament deficiency.
So first let's define it. Let's what's what. What? How did we come to define fidelity?
Interestingly, when I'm doing a systematic review and looking at how was defined and conceived, there were 95 different ways of doing this.
When you have 95 different ways of defining something, you might as well have none because it becomes very difficult to have a common language.
So through thematic analysis, through thematic synthesis, what we came up with was the first empirically based definition for it.
And you can really think of it as an umbrella concept, as two distinct but related components in the fidelity of delivery and participant adherence.
And both of them are essential for the clinical trial to be faithful to this protocol.
And both are essential for understanding of how an intervention can influence its outcomes.
And we'll see how. So the first part intervention delivery fidelity.
This focuses mainly on the actions of the intervention as the actions of the practitioner or the provider.
It looks at the quality of the intervention, delivery or the performance of the intervention during the trial.
It reflects the correspondence, how the intervention was done.
Was it similar, sufficiently similar to what was intended in the protocol?
Was it in accordance with the procedures as they were laid out in the study manual or the treatment manual?
Did this thing look in sufficiently as it should have?
So here's an example from a paper literature where you see this, uh, paper about a new Start intervention.
And you can see how they're trying to characterise it as compliance with the new Start intervention.
That you see one of the other terms, it'll be monitored throughout the trial by observations,
regular collections of activity records to assess adherence and to understand whether the
facilitators deliver the intervention in accordance with the training and as it was intended.
Sometimes you'll even see this, um, mentioned not in the trial paperwork itself or the trial application,
but you might see it later when they if they publish a process evaluation.
But one way or the other, what you can see there is this attempt to see was this intervention delivered as we wanted to be.
But the other half of this is now the actions of the participants.
That adherence. And again, that focus is mainly on what?
On their role and how they interact with intervention.
Whether they accepted the randomisation, you know, did they stay in the group that you had randomise into.
Did they initiate the intervention to which they were allocated?
And then how will they follow the prescribed allocated intervention?
So for example, this could be whether they attended the required number of therapy sessions,
whether they performed some activity that was part of the intervention,
a sufficient number of time or uh, or sufficiently similar to how they were intended to.
Um, for example, did the participants also in a surgery arm avoid physiotherapy, avoid exercising, just have surgery.
And you can see an interesting example here in the paper about bariatric surgery.
When they're looking at the rates of adherence to the behavioural recommendations that come along with the surgery.
So did the participants also attend the regular appointments that are meant to attend?
Did they follow the dietary, uh, recommendations, the physical activity recommendations, the vitamin use recommendations.
So they can see already whether the participants were really carrying out the intervention or not.
Because that would tell you a lot, for example, about the success of bariatric surgery.
You can have a fantastic surgery, but if everything the participants do before,
after and around it departs completely from the recommendations, you may not have a successful surgery.
Not because of the physical procedure, but because of everything that happened around it.
But this is all great. So now we know what to call it. But what does it do? What can we say fidelity does?
Can it bias the results of RCTs? And again think of bias.
When we think of bias what we're really looking at are systematic errors or deviations from the truth that result in results or inferences of a trial.
So, for example, flaws in the design or conduct of a trial.
And again, that can, you know, we think a fidelity there can bias that there are treatment effect estimates.
This can lead to an underestimation or an overestimation of the true intervention effect.
So you can have these biases that can either skewing toward the null or away from the null.
For example, we know this is the case with uh, lack of uh, of, of uh, of blinding in clinical trials.
We know that when we have participants who were not blinded, they tend their data tends to exaggerate the effect size in,
uh, in some clinical trials and in systematic reviews of clinical trials.
We know, for example, that, um, observer bias also is exaggerated,
is associated with exaggerated odds of an intervention succeeding or exaggerated odds of an outcome.
Up to, you know, one third, uh, exaggeration of the odds. Sometimes.
We also know, for example, that even, um. You know, study quality, for example, can bias the results of the trial.
So for example, in this paper, um, by Adidas,
we can see that the when you look at whether something was low risk of bias or uncertain risk of bias or high risk of bias,
that also can exaggerate, um, our lead to an overestimation of your treatment effects in a clinical trial.
So what about low fidelity? Well, previously people had just theorised, hey, if you have decreased fidelity,
you're going to have lower treatment effects because you're introducing all these things.
Your participants are being exposed to different levels of the the active ingredient of intervention.
They're getting different doses. Other people had theorised that if, you know, sort of mathematically, that if you have lower fidelity,
you would need a significantly larger sample size to to overcome it.
So if you had a 25% decrease in fidelity, you need a four times larger sample size.
Think of how problematic that is with a lot of our complex intervention trials.
They don't have enormous samples to begin with. You can see a pharmaceutical trial with 10,000 participants.
You're very rarely see a surgical trial with 10,000 participants.
You know, but we didn't know what the actual effect on the treatment effects are in a randomised controlled trial.
It never been shown empirically. We didn't know what direction it would be.
Would it lower the effect? Would it exaggerate the effect? So I want to tell you about this paper.
It's, uh, one of the papers of my DPhil in the Journal of Clinical Epidemiology.
And here we look precisely for that. And we wanted to see how the different levels of intervention fidelity.
Influence the treatment effect estimates derived from these randomised controlled trials.
In complex interventions. And this was a meta epidemiological study.
And by that epidemiological study,
what I mean is a study in which we're looking at the impact of different of characteristics on intervention effect estimates.
Typically you do this from a collection of meta analysis.
So you're contrasting the results of trials with that characteristic with these also trials without the characteristics.
And then you're looking at the estimates of average bias.
Or you're calculating an estimate of average bias that's associated with that characteristic.
And we've seen these um,
use increasingly we've seen them use it about to investigate the effects of poor blinding of a failure to, to, uh, to randomise.
Uh, so the fact sample of sequence generation and sample size, we've seen them for any number of things, but it had not been done for fidelity yet.
So in this study,
our aim was to measure the association between intervention fidelity and the estimate of treatment effects in these complex interventions.
Trials. So how do we do it? Well, actually,
when I'm pumping and cutting and in the cockpit library and did a really exhaustive search
over ten years for meta analysis of randomised controlled trials in these physical,
complex interventions. So these were in physical therapy speech therapy physical activity exercise interventions.
You may wonder this man mentioned surgery before. Why isn't surgery not here.
Surgery wasn't here because there were not a sufficient number of trials of surgery measuring or reporting fidelity, which is really interesting also.
Um, so we had these meta analysis with trials and these, um, these professions or these disciplines with continuous outcomes,
with at least four randomised controlled trials per minute analysis and at least two of them reporting or monitoring fidelity in some way.
Yeah. Um, to, to figure out how many we needed. We as a sample size and power population there are there is one um uh published
by the Sorbonne Paris Centre for Research in Epidemiology that we use for that.
It told us we needed a minimum of 14 meta analysis and a minimum of 140 RCTs.
Of course, we exceeded that just to be safe, but it just gave us a baseline to start with.
To look at the effects of fidelity. You know what difference this. Quantifiable difference that this make in the results of these studies.
We use what's called a meta meta analytic approach. It's kind of like doing meta analysis of meta analysis.
We calculated the standard mean differences between the, uh, trials in which the Arctis in which there was fidelity and in trials.
The Arctis in which we didn't identify any fidelity.
And then we looked at the mean impact of fidelity, um, the, the treatment effect estimates within each meta analysis and then across all of them.
Then we pulled all the standard mean differences, and we wanted to see if they were different in these trials, trials with and on fidelity.
And we did some of analysis. We wanted to see, well, what what influences risk of bias do we have?
Is it something you see more or less in trials a low risk of bias?
Is this something you might see more or less in trials with larger sample sizes compared to the smaller ones?
Because we know sample size, you know, influences treatment effect estimates the same way.
The risk of biases. Then we did something else. We did meta regression.
This is just like, uh,
regression analysis you would do when you want to see how things are related to they have a linear relationship or are they associated.
Can you predict one thing from the other? We did this here to to see how influence, how a change in fidelity may influence the results of a trial,
even accounting for or controlling for their sample size or the risk of bias.
So what do we get? We. Screenprint 427 Mint Analysis.
It's much more work than sounds like. We found that 19 were ultimately eligible.
Eight of them in physical activity or exercise promotions, seven in physiotherapy, four in speech therapy.
And again these you know, speech physio and speech therapy are not in treatment their profession, but they were treatments that fell within that.
Um, we look at 203 randomised controlled trials.
The median we can say is that that they had ten randomised controlled trials per men analysis.
There were 248 separate comparisons between a treatment, an experimental treatment and a control or comparison.
In there it represented the experience of 25,000 participants.
So it's a nice number. And what we found is generally there's a very poor amount of fidelity monitoring or reporting in this body of evidence.
So only 31% of these papers really actually reported that they monitored or maintained fidelity in some way.
And what you see on that graph is that when we looked at the number of papers that didn't report anything was 100, 141,
the large amount as we move, slide the needle toward higher and higher degrees or levels of fidelity, it starts getting smaller and smaller.
In the end. Only a very small number by comparison, really looked at both ability and adherence.
So that already tells you this is a shortcoming, a gap in or in this case.
So then we looked at the standard mean differences, the difference between the effect of the intervention that we're interested in,
the experimental one, the treatment and the effect of it compared to the control.
And when we found that they were very significant systematic differences in these effect estimates,
uh, which was surprising, not what people had predicted.
Basically, when you had fidelity completely absent,
you tended to have a larger treatment effect estimate as you added more degrees of fidelity, more degrees of adherence,
more degrees of closeness to what was intended,
you tended to get a smaller and more precise treatment effect estimate by precise meaning that the confidence intervals were more narrow.
So what this is telling you is that no,
having very poor fidelity is or was associated with an exaggerated and overestimated treatment effect estimate.
Um, and not not a smaller one.
Even interestingly, the heterogeneity of these things was much smaller as fidelity improved, much greater as fidelity decreased.
And this was the same where the we looked at no fidelity versus perfect fidelity or different degrees of fidelity.
So we thought, okay, what about low risk of bias? Will this be the same when you remove all these other things that come with risk of bias,
you know, all these other potential confounders and contamination and all these things?
Then we saw the same result. The more that you move the needle toward a better degree, a higher degree of fidelity and adherence,
the smaller, the more precise the treatment effect estimate and its confidence intervals became.
So again, it gives you the sense that there's an exaggeration,
potentially an overestimation of your effect when people have not monitored or maintained fidelity.
And this was the same way that we looked at only trials with small with a low risk bias or only larger trials, you know,
where you're less likely to have these exaggerate effects that come with smaller trials,
even with larger sample sizes of the same relationship, um, that you can see there.
But I want I want to point out something interesting, um, that will come to you later.
Is that in all of these, when you only looked at adherence, you just wanted to know, did people show up for the right number of sessions?
Did people show up for the right number of of of of of classes.
But you didn't care what they did.
That tended to give you a higher treatment effect, always that when you really monitored how many times they came in, how well they did it.
We'll talk about why that is later, but I want to plant the idea now.
So what was our take on? Poor fidelity may.
Certainly. Bias. In fact, estimates in randomised clinical trials of complex interventions.
Those are very linear association.
Uh, as you again, as you move that needle over from less fidelity to more, you start seeing that the standard mean differences decrease.
Um, by the end of it for when you in from nothing, from no fidelity and adherence to perfect fidelity,
you actually had a 0.24 smaller standard mean, uh, difference and smaller effect size.
And this again is when you adjust for sample size and risk of bias. We'll talk about why that matters.
But just to show you again, even when we looked at, um, you know, the, the when we looked at this in all the trials,
when we looked at this, uh, trials where we considered their sample size,
the risk of bias, um, when we looked at it separately in only trials, the low risk of bias,
it's actually a slightly higher effect, but not statistically significantly higher than in all the trials.
But what you're definitely seeing is, is very linear relationship between the. Why did this happen?
This sounds, you know, sounds counterintuitive, right? You think, hey, wait a minute. If you have better fidelity, shouldn't you have a bigger effect?
One. Smaller one. Well, um, there are a lot of possible reasons to that.
It's kind of hard to answer from, from that study, but one of the things that you may see is that when you have poor fidelity,
what you might be getting is the cumulative effect of a lot of extra goodies being added along the way.
You know, when you're not really carefully monitoring what you're doing.
And I know from my own clinical practice, no matter what someone learns and continuing in, no matter what treatment you see,
you always have this temptation to say, I know, I know, I know, you're supposed to do this.
You know, I don't know. Let's say, uh, somebody has spinal, a spinal cord, you know, sciatica.
And you want to work on something.
I know it says to do it that way, but I find in my own experience that if I do that, but also add something else that works better, or.
I know, I know, but I prefer to do that, and I prefer to do it with, you know, in a hot room instead of in on a table.
You know, all this accumulated addition to little bits here and there is probably adding a significant number of active ingredients.
So partially it could be that the reason you're seeing this larger, you know,
exaggerated request amount and in fact, the poor fidelity, there's more going in the system.
The person's being exposed to additional effect mediating interventions.
Um, there's also an increase in intervention intensity. There's kind of that interesting rule of life.
Sometimes, no matter what you do with a patient, the fact you're doing something, people get better.
You know, it's kind of what we see sometimes, even with some of the the sham and placebo interventions.
Interestingly, you can see how this may be the case.
Also, remember I said, hey, the people who were we only cared about how often they came, you know, but we don't care what they did.
They also had a bigger effect you can consider as well.
They're just being exposed to more and more often. But we don't know what that more is.
So that might be why they're having sort of this potentially exaggerated effect.
So what's the take home message with with this aspect of it? We don't.
Fidelity matters. It's not something that we often talk about. It's something we should definitely, definitely should talk about more.
Something we should plan in to our trials.
The treatment effect estimates may be overestimated or less precise when we don't monitor and maintain fidelity.
And by that I mean, again, how we deliver the intervention, but also what the participants did.
Again, think about what this may mean to mean analysis when we start pooling these errors.
Uh, what it might mean for clinical practice guidelines,
how confident can we be in recommending one treatment over the other if we're not certain what was actually delivered in the trials,
on whose evidence we rely? And are these trials a good basis for evidence or a good evidence source for clinical decision making?
And I'll give you a little, little example of how that worked out.
One of the things we did is we said, okay, if trials with poor fidelity or fidelity tend to have this exaggerated effect,
and trials with much better fidelity tend to have this more precise effect.
What if we went back and either added or took away that, that that that effect from the trials, depending on how their fidelity was?
And what we found is that some of the meta analysis no longer had a statistically significant difference between the
intervention they were they were investigating and the control or the comparison that actually that really matters.
Um, and the other one was that we talked sometimes about the effect the size of a, you know, a relationship with Cohen's d, for example.
Some of them went from a small effect to no effect whatsoever. Some of them went from a moderate effect to a small effect.
And so you can see this here. For example, these are meta analysis about different interventions.
And you're seeing this change in direction. And sometimes the the statistical significance vanishes.
Sometimes the uh, the you know, those are the size of the effect.
Matter vanishes when we want to make a decision or recommendation for patient care on something that may really not have an effect at all.
So could this potentially lead us to make them to make poor decisions?
We have to use caution. If we don't know if this fidelity or if it wasn't monitor reported.
We just have to be a bit Korff caution cautious because it may potentially lead to the adoption of some interventions that are ineffective.
And what about the possibility that we may prematurely reject some interventions that actually could have been effective?
What's the caveats to this? You know, there's always a chicken in it. There's always a caveat.
You might say, well, here's a new. But maybe they did it. They just didn't report it.
Is that a chicken or the egg?
You know, is it possible that you have these randomised trials where they actually reported fidelity because they didn't get the result they want,
and they were looking for somewhere to pin that?
Um, could you have a situation where these randomised controlled trials with the largest effect said, we don't need to bother with this.
Look at how big the effect was. Um, so I can tell you that that was not the case.
We actually looked at that as well. I'm nothing if not thorough. And so that did not pan out.
So there is more there than that. You might say. Also, what about the stage of development of interventions?
Oftentimes we're doing these trials to see if an intervention is does they have promise enough to keep developing it.
That's possible. You know that in an early stage of these interventions, some of them aren't.
They're just not stable. They're not defined enough. We don't yet know what to call them.
We haven't sorted out the parameters yet. So that's why it's hard to deliver them with fidelity.
Um, it's also possible that, you know, you can have a very robust intervention,
and the intervention is so robust that even if you deliver badly, you can still get a treatment effect.
I think that's less likely, but it's possible. And then again, here's interesting question.
This is all focussed on the treatment and the experimental, uh, intervention.
No one's looked at it in the control. What if you have a well delivered experimental intervention and the control is is not delivered well at all.
That's going to throw the treatment effect estimate also off as well.
We may not be comparing things the way we think we are we.
And that's something that hasn't been looked at. Yeah that might be another phase for the future.
But um something else to keep in mind. This is just referring to what we know was done for the for the treatment.
Not first comparison. So now that I'm giving you that, you might say, okay, great, wonderful.
I'm going to go look for this part. How do I do it?
You know, and one of the difficulties we had and we're coming to the end just just to give you the, you know, the wake up warning, uh,
the very little guidance that was available for how to monitor fidelity,
especially in surgery, especially in rehabilitation and all these complex interventions.
We had tools and they were made more for for psychology, for education, for behavioural health.
But they don't always transfer well to surgery or to rehab. They don't transfer well to these physical complex interventions.
So and some of the domains don't really work really well defined. Some of them may not have really also been very, very adaptable for us.
So for example a lot of them have this category called enactment.
And what they want to know is whether a participant uses the skills in daily life.
That may not be as applicable to these interventions, because that doesn't necessarily tell you whether it was delivered well.
Somebody can deliver an intervention beautifully. We see this with smoking cessation.
We see this with even physical activity recommendations. You can deliver that beautifully.
If someone could say, yeah, I don't really care or I know, I know, but it's too hard for me.
You don't understand. So that doesn't mean it wasn't deliver of the fidelity. That's more a, you know, measure of effectiveness or other things.
And most of the tools that were available, they were they were made really to assess fidelity while you were conducting the trial.
But what do you and I do when we just want to know if fidelity was conducted when we're reading a paper?
And so what I want to present to you is a checklist that came out of this thesis of mine,
and it's called the consider checklist and stands for Complex Interventions design, delivery and receipt.
Consider I had tried this with acronym.
If I ever if anyone's ever used the website, you put the key terms in and you you know they'll help you come up with an acronym.
It came up with some fantastic ones. Confound, confuse.
I didn't think those were very those are ideas helpful. So please consider what this framework does is it is looking at different aspects
of intervention fidelity in various parts of a randomised control trial.
So you know what you're looking for at the time. I'm not going to describe it much, but you can but you can find it.
It's a it's available online. It's also available in the epidemiological paper that we discussed.
I'll show you the I'm going to show you the tool in more detail.
But just to give you the overview, it's a multi-dimensional assessment.
It's looking at both fidelity and adherence in the publication, whether it was supported, whether it was monitoring, whether it was reported.
It's got eight items and they're mapped onto the other checklist and tools that we use all the time,
such as Tiger for intervention reporting, survey for reporting of exercise intervention, spirit.
Um, all all of the items are mapped onto those. It makes it easier to find there's an exploration, um, or an explanation and elaboration paper.
And the tool went through a validation study and survey,
and it found that it had a lot of internal consistency and reliability, and the users found that it was easy to use.
So what does it look like?
The first five items are really looking at whether there are things in the paper that you can identify that would support fidelity.
Ask you to say whether they were absent where there were somewhat discernible or they were mostly discernible.
So they'll look at things like materials.
Does it tell you in the paper about whether the therapists or the practitioners or the surgeons had training, whether they had manuals,
whether they had materials they can use to ensure fidelity is looking at things like,
did they describe the amount of leeway that the practitioner had?
Can they adopt things? Could they not adopt things where the intervention was, where they trained?
Did the paper describe even the scheduled duration, the intensity or the dose, enough so that you know that they were delivering things as intended?
So, for example, this is from the elaboration and expectation paper.
What you see here is like here's an example from an actual paper where it says if you want to score, you know,
a score of one would look like the treatment program consisted of this regular model that was manual,
but all three groups were provided with pedometers and logbooks to monitor what they did.
So this is the kind of thing you see in the explanation elaboration paper with a tool.
It's telling you how to decide how to score something.
You know, for example, when it says a score of two, photographic details of the program were distributed to each participant with
a logbook to record the number of days that the exercises were performed.
It's telling you they were really thinking closely, how do we know and how do we monitor what the participants were doing?
The second section is then looking at how fidelity was monitored.
How did they know that things stayed the way they were? Describe the strategies used to monitor or maintain that,
not the participants adherence or the delivery of the or the intervention, and again then scored from zero.
If nothing happened all the way up to three, if if both of them were monitored and defined.
And as an example, you'll see here's a score of three a week from taking from a paper, a weekly therapy log was collected.
The parents were asked to practice this intervention daily, and video recorded at least ten minutes each week for review during the group.
So there was recording and people were actually reviewing them to say, okay, this looks sufficiently like what we intended it to look like.
All the intervention sessions were videotaped. At least two sessions per child were chosen, randomly coded with a checklist.
At least 80% correct on each of six techniques was required to meet fidelity criteria.
They're actually telling you that they met, they match this a certain amount, and he explains you why he was caught up the way you would.
The next section of it, um, is really looking at, you know, did they actually did the report that the report, the modifications.
Because oftentimes you have to modify interventions for particular purposes.
This ask you whether in the paper itself whether you could see that they reported it.
Did they keep track of it. So for example you know, this is an example how one of the items is scored from, um, from an actual paper.
No parent met the fidelity or implementation criteria baseline at weeks 12, 21 and 25.
Find the parents and that group. Um, none of them met fidelity of implementation.
Unfortunately, that's not great news for that particular trial. That's basically saying nobody did what they were intended to do.
But it gives you an example of how to score it. And then very detailed example of why the score.
And finally, the last thing is. Okay. What's your final assessment?
Did they maintain fidelity? Fidelity there? Did they maintain adherence.
Isn't there. And again in the paper what you get is very specific um examples of how to do this and how to score it.
Because what you really want to know is did participants adhere.
Did they do what they wanted to do at least 75% of the time? And was fidelity maintained?
Did things look the way they were meant to do at least 75 to 80% of the time.
So you have very specific guidance there that can be very helpful when looking at this for a paper,
for a systematic review from an analysis, or just to understand what actually happened.
So what's our final take on back to the pastries. What do you put in that box of cupcakes to take home with you?
The take home is fidelity matters again, so it sounds trite, but it really does matter.
Fidelity influences treatment effect estimates. It tells us a bit about the how or the why and why interventions may or may not have worked.
Um. And get something there.
And then conditions and policymakers keep in mind that we all rely on the synthesis of the best available evidence to inform our decision making.
But we have to consider fidelity. And then as well, we need caution when we when we're not certain if fidelity was there or not.
You can see that this is becoming more, um, it's entering the consciousness more even the risk of bias to the ROV to from Cochrane.
It does come up with concepts that are very close to and related to fidelity. It's not quite exactly the same, but it's there.
Um, fidelity can be assessed in trial complications. This can be done.
We should probably. I think we should make this a greater part of quality assessment.
It's not really required now, but I think it should be. And again, there's a framework available.
There are other frameworks available. There is a framework available and checklist there freely available.
And I definitely welcome more development in collaboration. You know I mean it'd be great topic road test and more.
But the take home message is monitor fidelity don't just taken at face value.
Um, if anyone's interested in knowing more. That's me. Um.
And, uh, I'm always happy to answer questions or collaborate in any way.
Thank you. Arsenio. Uh, fantastic. Yes. Do you take that, uh, QR code?
Right. What does that. What does that send me off? If I if I look look up that code now, it's like.
And LinkedIn will connect you with me I do. All right.
Cool. Um, great. Thank you. Um.