I'm going to talk a little bit about this debt watch design, but not an awful lot about this stepwise design,
because one of the important things to think about is what type of study design should we be using.
So I want to make sure that I give you the whole sort of menu of options so you can think about all the different possible options
and then come up at the end to sort of some recommendations for when perhaps the stepwise design might be a good design choice.
So I'm going to talk about health policy evaluation. And that's because this is where the step to design comes into its own.
It helps us to evaluate policies. Or at least that's the intention.
So why evidence based policy? Well, I'm convinced that we need to evaluate different types of policies.
We've been evaluating medicinal products, say, pharmaceutical interventions, for decades.
We all accept that if we're going to use the drug, then we should have robust randomised evidence behind that application to use that drug.
But when it comes to health policy evaluation, we typically accept a much lower quality of evidence.
And I'm going to put up some very poor study designs and I'm going to try to persuade you that these are not the designs that we should be choosing.
But actually the evidence is littered with these very poor study designs.
And if we're using a poor study design, we can't be certain whether this intervention, this policy that we're looking at, works.
So it doesn't work because the study design is littered with these different types of biases.
And this is the sort of work that I'm trying to advocate.
And this is a an editorial in JAMA just from a month ago or so.
And they were also advocating that we should evaluate these health policy interventions
in a robust way rather than just rely on observational case study type of evidence.
So what do we mean by policy interventions?
Well, I'm going to give you some examples, but it's really wide ranging.
So here I've put up some examples.
So one example is something like pay for performance, incentives for healthy behaviour, workplace well-being programs.
Those are the sorts of things that I'm thinking about.
So what I'm going to do in this talk is to, first of all, come up with what I'm saying is the definition of a health policy intervention.
And I am going to try to carefully differentiate that to something called quality improvement,
because I think the literature gets quite messy when it tries to work out what is a policy evaluation of what's quality improvement.
I'm going to give a little bit of a tour around the hierarchy of evidence.
So you know what sort of evidence that you should be aiming to get to.
And then I'm going to think about the suitability of cluster randomisation.
I'm going to look at different types of cluster randomised trials.
I'm going to think about when the stepwise design is appropriate and then I'm going to try to
think about the different types of risks of bias that we get with these different studies.
Because even though I'm talking about randomised designs, there is still the possibility that the designs can be at risk of bias.
This is particularly important with cluster trials. So my objectives then, this really is this is the underlying message of my talk.
Correlation doesn't mean causation.
So just because we have an example, we have a study where they've shown something is correlated with something else.
That doesn't mean causation. We know that. But we mustn't lower our standards when it comes to interpreting these policy based evaluations.
We wouldn't lower our standards if we were looking at a drug intervention.
So why should we do this when we're looking at policy interventions? So every time I'm going to put a case study up,
I'm going to come back to this to think about whether or not we have established correlation, whether we establish causation.
So what's quality improvement them? How does quality improvement differ to a policy intervention?
So there are lots of different definitions in the literature of what quality improvement is and what policy evaluation is.
Some of the definitions talk about whether or not whatever you're trying to do is generalisable.
Some definitions say that if you're doing something that's a local evaluation, you don't need ethics approval.
Now, that's one problem. When we come to determine what sort of evaluation we're doing.
But the other problem is,
is that those people that are doing quality improvement evaluations typically have got less funding and they're using lower quality designs.
So I like to think about what is the objective of what we're doing?
Are we trying to determine whether we can improve outcomes in my hospital or whether this intervention improves the outcome?
And I'm saying that if we're interested in working out whether we can improve outcomes in one hospital, then that's quality improvement.
But if we're trying to make inferences about whether this intervention is improving outcomes, then that's a policy intervention.
Both are going to find different people want to do different things.
You may be working in a hospital and you may be really interested in determining or working out how you can improve things in your hospital,
but also by improving the quality.
So then it's less important then for you to identify what the sort of core component of your intervention is that's making the improvement.
If you are only interested in improving outcomes,
you're not concerned so much with what is the crucial ingredient, but rather are you improving outcomes?
If we're trying to improve outcomes in a hospital, let's call that quality improvement, which starts at a lower level.
We'd make some changes. We'd look to see whether those changes had improved any outcomes.
We'd refine things we'd talk to people would try to do things better.
We'd re-implement what we had at what we had done, we'd make some changes and then we'd look at our outcomes again.
We'd talk to people again. That's quality improvement.
We might try to flag up. If we were looking at a sort of systemwide quality improvement,
we might look at particular hospitals and try to identify which hospitals were
not performing very well and try to differentiate special cause variation.
But that's all quality improvement. That's all about trying to improve outcomes rather than trying to make inferences
about what is the key ingredient that you're doing to improve the outcomes.
So I'm not interested in quality improvement,
but I think it's important to differentiate that because many of these studies that I'm going to be looking at,
they call themselves quality improvement interventions. But I don't think they are quality improvement interventions.
They really are policy evaluation. So they're using quite low quality evidence.
So back to policy evaluation then. So what I'm saying here is that causal inference is the focus.
We really want to know whether this intervention works. It might be that we're interested in knowing whether the intervention works in one hospital.
It might be that we were interested in making more generalisable statements.
But wherever whatever our inference, whether it's about local or generalisable results, it's all about causal inference.
So when we want to evaluate this, these different types of health policy interventions, we've got this hierarchy of study design.
We can choose from some of the low risk of bias and others are at high risk of bias.
Now, of course, we want to choose the design that has the lowest risk of bias.
But sometimes because of the nature of the intervention and the nature of the setting,
we might not have the flexibility to choose the design with the least risk of bias.
And that's because we might only have one cluster available.
Look we've only got one cluster available but we can't do a cluster randomised trial because we only got one thing to randomise.
But we have other options available and these are called Interrupted Time series analysis and I'll show you an example of one of those in a moment.
If we've got many clusters available, then we've got a lot more options open to us.
And then we can start to choose a design that's got a lower risk of bias.
So what I'm really saying here is that depending on the context,
sometimes you might only have the option to choose a study design that's at a higher risk of bias.
But if you're in a situation where you have got many clusters available,
then we really should be trying to choose the study design that has the least risk of bias.
And I'm going to try to share with you my ideas and how we can chase that that design.
So I'm going to run through some different examples.
These examples all evaluating health policy interventions that all are using different types of study designs.
And I've chosen the example so that we're going to work through the hierarchy of evidence,
starting off with a design that is quite low quality and working up to a design where I'm going to
show you that actually is possible to evaluate these interventions using high quality evidence or.
So this is the first example. Now, they call this exam, this study, a before and after study.
So this before and after study, it's something very, very bottom of the hierarchy.
What they wanted to do was to try to improve outcomes in women who were having babies
by making sure that they were monitoring the amount of iron that the women have.
So they introduced a whole heap of different types of resources.
And basically these were educational resources aimed at clinicians.
That's sort of multifaceted.
It was more complicated than just simply education, but it's simply around a wide array raising awareness around this condition.
So three population of interest were pregnant women who were in the delivery clinic and who were in I am in a labour ward.
And then they took data pre and post this intervention phase.
So the data span from around January 2012 to around December 2016,
in one hospital and from the electronic patient record, they gained information on their outcomes.
And these were the rates of ferritin or the rates of iron testing and the proportion of women who were anaemic.
So those were the two outcomes. Now it's very arguable that they could have done this evaluation in more than one hospitals,
but they constrained themselves to evaluate this in one hospital only.
So let's see what they did. So they say it was a before and after study.
So this little diagram here is an example of one. A before and after study typically looks like the light blue represents the control condition.
The dark blue represents the intervention condition. And I'm going to use these same diagrams throughout the talk.
So each row represents a cluster. And a cluster in this situation is the hospital.
So this was carried out in one hospital. So we've got one hospital.
That hospital is observed for a period of time just doing things as they do them.
They then put the intervention in place that comes about here at month 12, and then they change the way that they do things.
They carry out the educational workshops. Then they have a little transition period in here,
a gap in the design where they accept that they're just rolling out the training and then they carry on collecting the post data.
Now I've drawn this diagram here with one cluster, 24 months worth of data collection.
Actually, in this example, they had a little bit more data than this,
and they probably have this sort of transition period in the middle where they implemented this intervention.
So there are different ways of analysing data from a design like this.
You can take all of this data and all of the post data.
You pull all of the pre data and you pull all of the post data and then you're just comparing two numbers.
So the two numbers in this example that they might be comparing, are they the outcomes were.
So the first one were the rates of ferritin testing, and the second one was the proportion of women who were anaemic.
So they might just simply work out the proportion of women anaemic higher proportion women in the post period and then they compare the two.
But that's quite wasteful of the data. It's just pulling everything together.
If you do a pre post analysis like that, you're opening yourself up to a whole risk of different types of biases.
Anything could have happened at this point in time other than your intervention.
Maybe that to date when you rolled out your intervention.
Coincidentally happened to coincide with something else,
perhaps another policy implementation perhaps the government has decided to do, or the local health authority had decided to implement.
Other things could have happened at that same point in time, simply by showing that there's a difference pre and post.
We don't know whether that's because of the intervention that we're trying to
evaluate or whether it's because something else external to our study happened.
Now in this design, in this study here, they called it a before and after study, but it wasn't really a before and after study in that sense.
So that's a worse sort of before and after study we can do pooling of the pre and the post data and just they can get the difference.
They did something a little bit better than that,
although they didn't give themselves credit in the title because they caught it before and after study.
They did the next best thing and that's called an interrupted time.
So as certain Interrupted Time series, he monitor the system.
For a long period of time. You then have your intervention.
You put your intervention in place and then you carry on monitoring the system or what you hope to be able to show.
And this is the data from that study is that when you implemented your intervention at this point in time, something happened to the system.
So in this example, the rates of ferritin testing, this is the rate of ferritin testing for this study and this is the month of the study.
This is the date they implemented their intervention.
They saw here in this example that there was quite a sudden change and a large change in the rate of ferritin testing in this whole population.
The other thing that you can do is you can monitor what we might call the trend here.
So this isn't completely flat. There's some sort of trend going on here.
Things are increasing a little bit anyway, maybe not to a huge amount, but there's some hints here that things are increasing a little bit.
So you might also try to work out whether not only is there a change in shift,
is there a sudden impact of the intervention, but whether it might have changed the trend.
They didn't do this. They didn't look to see whether there was a trend change in trend.
But you could do that. Now in this example, they were lucky.
They saw a huge impact of their intervention.
It's probably quite hard to doubt that this change that they saw was due to the intervention.
But unfortunately, most of the times when we're evaluating these types interventions, we're not so lucky.
We don't see such big changes. And actually smaller changes are probably as important.
May be they had only changed, managed to shift by this amount here.
That would have been harder to detect on that graph, but nonetheless, it could have been clinically important.
So if you use this type of study design and you're in the lucky situation of evaluating an intervention,
that really works, probably going to be all right because you'll see any big shift to change in this type of analysis.
But actually, most of the time we're looking for smaller changes.
And if it's a smaller change, this then becomes hard to differentiate the smaller change from the underlying secular trend.
But the more data you get pre and the more data you get post,
the better you can establish the sort of underlying trends in the system and then that allows you to detect these sorts of changes.
The other issue with this type of design is it's carried out still in one hospital.
So we don't know whether this intervention would work in a different setting.
Maybe the people who instigated this intervention in this one hospital, maybe they were really interested in this area, they really advocated it.
And it was really the people that made this change rather than the intervention.
And if you take it somewhere else,
you don't know whether it's going to have the same effect because you then have to take this sort of person factor away.
So you don't know from this study in one hospital whether that's generalisable.
So that's the other limitation of this type of single centre, pre and post evaluation.
But we can do one better. And that's quite a low support baseline design.
I don't see this very often in the literature, but I think it gets used much at all.
If you've seen, it would be interested to see, to share this, for you, to share your examples.
But essentially what the multiple baseline design does, it replicates this interrupted time series type of analysis in multiple centres.
So this isn't a real example. This is just an artificial example.
But you run your Interrupted Time series in one centre that's called Community at the Top,
and you stagger the times at which you roll your intervention out into practice.
It's not randomised. It just happens. But typically if you're trying to evaluate an intervention and that intervention is just going to be rolled out,
people will do it as and when they choose. So typically you'll find that community a hospital.
Would it a different time to hospital care, etc.
So then you end up with a staggered roll out of the intervention and again trying to collect a large amount of data,
train a large amount of data post. So you really can observe what's happening in the system in absence of the intervention.
You then hopefully are able to infer that every single one of these different centres or different communities here,
you're see an impact around the time the interventions put in place.
Now this is an artificial example.
So again, you'd have to be really lucky that the intervention really was having quite a large impact and that it was consistent across the centres.
But if you do manage to have an intervention that works by a sort of has a quite
a large impact and it's rolled out in several centres or several clusters,
but less provides quite convincing evidence. If you manage to observe an effect.
And how you analyse this type of study is you run your to interrupt a time series analysis in each of these
different centres determine what the shift changes and the trend changes in each of the different centres.
And then you would just summarise that probably narratively. But let's go back to the, um,
the ferritin testing example in this single centre where they were trying to evaluate whether this educational
policy around trying to prevent women having a name at the time of their birth really improved things,
whether this intervention managed to improve the outcomes in the women.
Well, it's difficult. Probably okay in that example because we saw this here is the impact of the intervention.
But in most situations, that type of interrupted,
interrupted time service design would leave us questioning whether the change that we saw
reflected our intervention or whether it was just coincidental that something else had happened.
At the same time, we put our intervention in place.
Now, you might think that it's unlikely that someone else is going to choose to do something just as we're doing our intervention.
I mean, that's not really going to happen, is it? Well, actually,
it does happen because that tends to be things happening in the background and possibly
the very reasons why these investigators decided to evaluate this intervention.
It possibly was not their own idea, but they were actually reacting to something in the larger system.
What people were talking about, this is an issue. So it does happen.
And then the other thing in this example is that, yeah, they saw a big intervention effect.
If they were interested in smaller effects of their intervention,
it's much harder to conclusively say that using this type of study design, that that intervention has caused that impact.
So it's quite low on the study, on the on the hierarchy of evidence.
And I tend to think this is correlation, not causation. It says the only thing you can do, it's probably reasonable,
especially if you did a multiple period baseline design where you replicate the same thing in multiple centres.
So this is case study two. Now, this is a stepwise design.
Again, it's a type of quality improvement, um, educational type of intervention.
They call it quality and a quality improvement package.
I think it's sort of the evaluation of a health policy. So it's a stepwise design.
So first of all, I want to tell you what is a stepwise design? So the stepwise design is a cluster randomised controlled trial.
So that means the cluster is the unit of randomisation.
A cluster might be a hospital, it might be a general practice, it might be a school, but this represents some sort of grouping.
Now each cluster in the scope, if you're choosing a stepwise design,
it's randomly allocated not to intervention A or intervention B, but rather than allocated to a sequence.
These are the sequences in the design. In this design here I've got five sequences.
So I clusters randomly allocated to one of these five sequences.
If they're allocated to the first sequence, they are observed for a period of time, the first period under the control condition.
So light blue again is the control. Then they switched over to the intervention condition in period two and they remain in the intervention
condition for the following duration of the study sequence to say five clusters have been allocated necessary.
Four clusters have been allocated to sequence two. So same thing happens there only they spend two periods in the control condition.
And for Paris in the intervention condition. Sequence, three, three periods in the control, three periods an intervention and so on.
And the number of periods that we monitor things for is just equal to one more than the number of sequences in the design.
And then that means by the end of the study,
every cluster will have been allocated to a sequence at which at least at the very end they get the intervention condition.
So that's what the stepwise design is. So there are lots of different variations.
There's no drunkenness, no riot. So they tend to fit themselves to practice.
So it can be the case that you might have a design where this, like this Pierrot here is missing.
It might be that this period here is missing. There's lots of different ways that they can be implemented.
You can have an unequal number of clusters to the sequences.
You might even have more periods at the end. But this is just the basic format.
Um. The other thing that often happens is you have this transition period somewhere.
So a period of time where we roll out the intervention and that just allows us time to put this intervention in place.
And then if we have a small period of time here where we are rolling out the intervention, we might not include that data in our analysis,
realising actually it's probably doesn't tell us about how the interventions working because they haven't received the intervention just yet.
So that's a basic type of design.
Now every cluster gets the intervention condition and that is an appeal of this study design because sometimes when you're working with clusters,
hospitals, GP practices, there can be a desire from the stakeholders and by stakeholders I mean the people that
you have to engage with in order to get participation of the cluster in your trial.
Now, those stakeholders often have beliefs about whether this intervention is going to work.
Sometimes they might not know anything about the intervention, but they just think that because it's new, it must be good.
So that can often be this desire for everyone to get the intervention.
And this design has the appeal that every cluster gets the intervention, and that's why it's becoming appealing.
But not every individual gets the intervention because depending on how you run your study,
if our study is around women giving birth and having collecting outcomes and whether or not the women are anaemic,
we'll have different women giving birth in each of these different periods.
So women who are giving birth over here, none of these women will have been exposed to the intervention by this time.
If the woman is having her baby in the first four hospitals, she will be exposed to the intervention.
But if the woman is having her birth in any of these hospitals, 17 to 20,
if she is having her baby anywhere over here, she will not be exposed to the intervention.
So will clusters get the intervention? But not all people get the intervention.
So the other appeal, the first appeal is a sort of social appeal that after all, the quest is get the intervention.
But the other appeal is it's not because every cluster is observed under the control condition and then the intervention condition.
We can balance any cluster level characteristics.
So let's suppose that these hospitals at the top in a sequence and maybe teaching hospitals
and these hospitals down here at the bottom are just by chance not teaching hospitals,
this sort of regular community, local hospitals. Now, teaching hospitals and local hospitals tend to have lots of differences between them.
They'll have different types of doctors. They'll have different types of patients.
So if these hospitals down here were allocated to the control condition and only the control condition,
and they happened to be the local hospitals and the hospitals at the top, the teaching hospitals got the intervention condition.
Then we wouldn't know if we observed a difference in their outcomes,
whether it was because of our intervention or whether it is because of this teaching hospital status, which say that that was a confounder.
The appeal of the stepwise design is that every hospital gets observed and to both conditions.
So the teaching hospitals get observed under the control and intervention,
and the local hospitals get observed under the control and the intervention.
So that increases the chance that things balance a little bit.
But the design has a huge caveat, and that is if you look at this dark blue,
that dark blue is all data that we've collected under the intervention condition.
And the light blue is the data that we've collected under the control condition.
What do we notice about the dark blue and the light blue? The time shift?
Yeah. This whole thing has been time shifted.
All this dark blue data, this intervention data, is collected at a systematically later calendar time than this controlled data.
So that means that when we thought about these time confounders in the before and after study and in our Interrupted Time series analysis,
I didn't mention the word time confounder that, but really we were worried about time confounders.
Time confounder is a problem here. This design is confounded on time.
You would never, in an individually randomised control design choose a study design that's confounded with some think by design.
The whole notion of randomisation is to remove confounders.
Yeah, this design has a confounder in it and the confounders time.
So we have to be really careful when we use it because this is a really big limitation of the study.
There are things that we can do because we're collecting data over a long period of time for all of the clusters.
This data here essentially allows us to estimate what the time trend is.
So we use all the light data, light blue data, we estimate the underlying secular trends, and then we can adjust back and find out of our analysis.
So just like in a regular type of design, if you notice something as a counter,
maybe in an observational study, you know, the age is a confounder, you measure age and you adjust for it.
So then you remove its impact. You hope.
You can do the same sort of thing here, but you have to make the assumption that the time trend is the same across all of the clusters.
So you have to make the assumption that whatever is happening in Cluster 20 and this is hospital number 20,
some things are happening in the background that you've got no control over.
We call that the secular trend. In order to adjust out the impact of the secular trend in this design,
you have to make the assumption that that secular trend in cluster 20 is the same as the secular trend in cluster 17,
same as a secular trend in cluster ten and so on. And that's the only way you can manage to adjust for the secular trends.
It's a big assumption when we use that design, we have to make that assumption.
You can't test that assumption really because you don't have enough data to test
it on because if you wanted to test that assumption and cluster one at the top,
you look at cluster one, you've observed it mostly under the intervention condition.
So in Cluster one, you don't have any data on the secular trend.
You only have data on the secular trend down here.
Now maybe you might be able to do something to test whether that secular trend is the same as that secular trend or two is the same as 910.
But then you go into small sample issues. You can't differentiate sampling variation from true differences.
So it's really difficult to test it. So let's go back to the example.
What did they say? Let's just remind ourselves what the example is.
This is the example. The other thing I should say is all of these studies were published in PLOS Medicine.
So very reputable high impact journals. So their population again chosen another women having babies example.
They were women who were at 28, 22 weeks gestation or more, having their baby excluding stillbirths.
They this time looked at 12 hospitals there. Other example, we have just got one hospital now.
We have got 12 large public hospitals. And this is in Nepal.
Started collecting or monitoring the system in April 2017.
They followed and monitored for 18 months. So a short amount of time period.
Then our other another other example. But we've got more hospitals this time.
They have some intervention, which they called a quality improvement package.
They were basically trying to improve mortality. Again, this intervention was around education, so trying to improve health workers competency.
A similar type of intervention package will be at this time trying to target something else.
But they had higher aims. They were trying to influence mortality.
Now, if they can't influence mortality, that impact is going to be much smaller.
Nobody is going to be able to have such high impact on mortality as we could have
on the rate of ferritin testing or on the proportion of women who are anaemic.
I mean, that's just how things work. So they're going to be looking for a much smaller, subtle change in mortality.
So this is their study. On the left hand side, that was a step to design.
So they had four sequences. They called it a wedge in the consort statement on this.
These are these they're not called wedges. They're called sequences.
But sometimes different authors use different terminology,
but they've got four sequences here and 12 clusters so that these three clusters allocated to each of the different four sequences.
Although that diagram doesn't clearly show that there are 363 clusters for each of the four sequences, and then a straightforward stepwise design,
a baseline period where they're all under the control condition, sequential roll out,
and then a slightly elongated step four where they monitor things for a little bit of a longer time period.
And that's their picture taken from that paper. This is a trend in mortality.
So this is week of the study. Not sure why.
It only goes up to 50 weeks when the study lasted for 18 months.
I'm not quite sure. But this is the mortality rate on the X-axis here.
They show this downward trend in mortality and this data have been pulled across these 12 different hospitals.
So there's a bit of noise here. Things are going up and down a little bit, but they've superimposed this line of best fit.
And if you look at that line of best fit looks like things are improving.
So maybe the intervention has had an impact on mortality.
This is a conclusion in their paper here, and they say that the incidence of interpersonal related mortality was 11 per
thousand deaths in the control period and eight in the intervention period.
So what they're saying here is that on average, under this light data here, the control data, the mortality rate was 11 per thousand, 100,000.
And in the intervention, the darker shaded area, the mortality was around eight per 100,000 deaths.
And then they get the adjusted odds ratio. They call it an adjusted odds ratio and I forget exactly what they adjusted for,
but they adjusted for things like age of the mother, socioeconomic status, some things like that.
And then they showed or they reported an adjusted odds ratio of point eight, confidence interval point seven, 2.92.
So sort of conclusive evidence from that statistical analysis that the intervention works.
Do we believe that they've shown us that the intervention works? So they haven't adjusted for the time factor here.
So they didn't adjust for time.
So something else could have been happening in the system that could have caused this change in the mortality rate over time.
And we cannot say with that whether that is because of our intervention or whether that is because of something else happening.
If we had adjusted for time or if they had adjusted for time,
they were willing to openly and make the expectation that actually the secular trends were the same enough clusters
if they were happy to make that assumption and that assumption was transparency and they had still got that result,
then perhaps we could have believed it, but they didn't adjust for time.
And at the beginning, when we looked at this design, we said time was a confounder.
When you look at that picture, you say that the intervention observations are collected systematically later than the control observations.
So therefore, time is a confounder by design. If you don't adjust for it.
We don't know whether this change in mortality is because of the time effect or whether it is because of the intervention effect.
So I'm not convinced that that's anything other than correlation.
But that's not to say that we can't use the step to trial.
I've shown you an example. That's not brilliant. I don't think if we adjust for the secular trends, it moves up a little bit in that hierarchy.
And if we're prepared to make that assumption that these clusters have got the same secular trends,
perhaps because we're working with clusters that are all very geographically similar,
maybe we can convince ourselves that that assumption might be all right if we have got lots of clusters.
This assumption tends to be less important because things just balance themselves out in the randomisation.
So if we've got lots and lots of clusters, this assumption of the same secular trend in every cluster turns out not to be very important.
In reality, we never have enough clusters for that not to matter.
So here we're probably talking about 50 or 100 clusters for that assumption not to matter.
Unfortunately, when we do these studies, we typically have 2010 tops, 30 clusters.
So that assumption does become important. Okay.
So case study three, getting to the top of the hierarchy, I think.
So this is a cluster randomised controlled trial, a parallel design.
And I've chosen this to be at the top of the hierarchy.
They have this taken this quote from the paper and they say to our knowledge this is the first randomised controlled trial
to investigate the effect of repeated rounds of testing of the multiple biological outcomes of chlamydia prevalence,
pi day and ending mitosis. The trial had a pragmatic design.
It reflected the real world roll out of an opportunistic chlamydia screening program.
And I like this example because it's testing a policy.
It's using a robust randomised design to test that policy.
And it's an example of how we can do things if we really want to know whether something works or not.
So what's the design? It's a simple parallel cluster,
randomised controlled trial and all they do in this type of design is randomise
half of the clusters to control half of the clusters to the intervention.
Again, it's cluster randomised. So the unit of randomisation is this cluster, a GP practice, a ward hospital, some big grouping.
Cross this gap control or intervention. The big problem with this design, well, there are two big problems.
And the only one that I'm really mentioning for the time being is this risk of chance imbalance, which we just spoke about a few moments ago.
If you've only got a small number of clusters, you risk allocating clusters to the control condition that are typically
different to those clusters that you randomised to the intervention condition.
So imagine you had a randomised design and you were trying to evaluate some whether some drug a work and you were
going to use individual randomisation and you had 20 people in your trial because I've got 20 clusters in this study.
Imagine you were running a design with 20 people in it.
You would never believe the results because you would be thinking at the back of your mind, Well,
okay, your trial showed me that drug works, but maybe randomisation just wasn't big enough.
You only had 20 people, so maybe these ten people that got the intervention happened to all be younger.
And that can happen if we've only got a small number of RANDOMISATION units.
Things just don't balance. When we look at our baseline table, we'll see an imbalance design.
Exactly the same things happen in cluster randomised designs.
If we don't have enough clusters, things do not balance out between intervention and control.
If we were going to test this intervention many times and then finally accumulate everything into a systematic review,
that chance imbalance wouldn't matter. But we often make inferences not from a systematic review, but from one trial.
So if we want to believe the results from one trial, we need to have a large number of clusters.
If we don't have a large number of clusters, we've got to worry about chance imbalance.
If we know what characteristics are important, we can measure them.
We can use this method of randomisation to try to constrain and ensure that these are not too different to the top.
But then we move away from randomisation, and often we don't know what these characteristics are or we can't measure them.
Or if we don't measure them, we're quite an accurate in our measurements. So that doesn't work that well, unfortunately.
But if we've got a large number of clusters.
So now I'm just going to think about the situation where I've got a large
number of clusters and I think more than 58 is probably just about all right.
Then there's no time effects here that we have to worry about in every other design I have spoken about tonight.
The big caveat has been this issue of a secular trend.
We saw it in the before and after study. We saw it a little bit in the Interrupted Time series.
We definitely saw it in the stroke twice design. But it's not here.
This design is perfectly balanced on time. We don't even think that time is an ingredient in this design.
It is. We don't often think about it, but times still there in the background.
But time is balanced, so time is not a confounder.
So that is the appeal of this design. And it's not really an appeal, I guess.
This was the design everyone was using ten years ago. Then they went backwards and started using the stepped wedge design.
But let's go back to this design because there's no confounder in this design.
So let's get back to the example I put up and this is this Australian cluster randomised controlled trial of a chlamydia screening program.
What did they do? So they took patients, anyone who was the aged between 16 and 29, 29 years, that was their target population.
And they took data from primary care clinics in these 50 rural towns on their intervention package.
Again, sort of educational type based approach. But what they wanted to do was to reduce the prevalence of chlamydia and the entire population.
So they asked 50 rural towns in Australia to participate and then they rolled out this intervention,
educational package, payments for testing, chlamydia, all sorts of different things in their intervention component.
And then they monitored the chlamydia prevalence and they did that before randomisation.
What we might call at baseline or they called in survey one.
Then they put the intervention in place, they allowed the intervention to settle, and then they did another survey at the end of the trial.
They courts that survey too. I forget exactly how much time was between those two surveys.
So this is what they found. So let's first of all, just look at the intervention group.
In the intervention group. We can see the vote on the first day, the results of the survey one.
So in these clusters that were randomised, the 25 tons that went into the intervention on the prevalence started off at about 5%.
So about 5% of people in those rural villages in that age population had got chlamydia at survey two.
That had decreased to about 3.4%. So that's that number.
And that's that number. Say 5%, then 3.4%.
And then we can work out the change. And that's a 1.6% reduction.
Now, I'm not saying this is a good way to analyse the data, it's just how they presented it.
So we might interpret that. It might we might say that that means that any intervention on the proportion of people
with chlamydia in those rural villages reduced by about one and a half percent.
Confidence interval. It's a difference.
It's entirely the one side of the null. So that would be statistically significant.
So that is suggesting that in the intervention arm, things improve by about one and a half percent.
But it's the control. Nothing happens in that control on.
So usual care just continued. There was no educational package, no promotion about chlamydia screening.
Nothing happened in those 25 villages.
So that the committee of Prevalence started off at about four and a half percent on average, a little bit different.
But remember, we don't have many units here to randomise, if any got 50 randomisation units.
So we can't completely be certain that when we randomise these, they're going to be the same as those.
So it's it's expected that we might see a bit of difference there, but are reasonably similar at baseline.
Decreased to 3.4% in the control.
Difference is a bit smaller, but it's a 1.1% reduction.
Now this time it's not quite hitting statistical significance, but we shouldn't pay too much attention to that the whole.
The trend here is that things have reduced. Things have reduced there.
We can't say for certain whether this is statistically significant, but most of the confidence interval is that side of the note.
So probably things are reducing in the control arm too. What does that suggest is happening?
Siri duction here and a reduction there does not tell us the intervention's working or does it tell us something else?
Yeah. Sunday again, exactly the same thing happening, something going on in the wider system.
So things have reduced in the control on things have reduced in the intervention.
Now here they've done a cluster randomised design. So we have this data here in the intervention arm.
I may have the date in the controller. But imagine if they had chosen not to do a cluster randomised design.
Maybe they had just done a before and after study. Like in the first example in the study that was still in PLOS Medicine.
We'd only have that data there. Now that data then would tell us that there was a reduction, but actually we have also corrected the control data.
We know that there is a reduction anyway. So there is a secular trend.
There is something happening in the system. This is just what happens all the time.
Fortunately, things are improving all the time, and so that's why we need to collect this control data so we can compare what
happens in the control in the usual system allowing for that temporal change.
So what happens when we add our intervention into the system,
but only when we have that control data can we allow ourselves or can we have the ability to
differentiate what is the impact of the intervention from the impact of the secular change?
So this is the secular trend, this is the intervention and the secular trend.
And then we can do an analysis where we adjust for the baseline values and we
can get an analysis or an estimate that has adjusted for that baseline values.
And then this then gives us an estimate of the impact of the intervention itself ruling out or taking out that impact of the secular trend.
And that's the intervention effect, much smaller than that impact.
Now, if we had just done a before and after study here, we would probably conclude that this intervention was having some impact here.
We have got a much smaller estimated effect of the intervention.
It's not statistically significant, but then becomes a little bit difficult to interpret what this means because it's not statistically significant.
And we have to be careful that we don't say here that this tells us it doesn't work just because it's not statistically significant.
But we look at the confidence interval. And that confidence interval, what it includes quite a large reduction in the negative direction.
It includes the possibility that things might improve a little bit too.
So it might get worse.
The better to a point estimate is pretty small, but the entire confidence interval perhaps arguably size that it rules out anything but small impacts.
And that was their conclusion. And that conclusion was that it was anything but small impacts.
I slightly dispute whether that is quite consistent with that confidence interval because we might say that a
reduction of two and a half percent is a pretty large impact when we're going from 5% and reducing it by about 50%.
But nonetheless, one thing that's very appealing about this design is we do not have to worry about time effects.
So I then wanted to just sidetrack a little bit and have one word of caution.
Um, about just making sure that we don't use the word policy evaluation when actually
we're evaluating something that's not a policy but an individual level intervention.
So I wanted to quickly show you this example of benzodiazepines.
So population what people have in cardiac surgery. The intervention was perioperative benzodiazepine.
They were in Canadian hospitals and they were trying to reduce delirium after surgery.
Is this a policy intervention? Some people are shaking their heads.
I really agree. Shaking your head. So sometimes cluster randomisation has been used as a means to avoid individual patient consent.
This is a quote from the protocol paper for the study evaluating benzodiazepine in a cluster randomised controlled trial.
And they basically say that although in principle they could evaluate the effect of benzodiazepines
in this setting through an individual patient asked a regular randomised controlled trial.
This is probably not the best approach to address the broad questions of policy.
So what they're saying there really is this is a policy, I think we could disagree with that.
But they then go on to say that consent to participate is therefore obtained from the cluster
rather than the patient ensuring representative sampling and enrolment of all eligible patients.
But these things are true.
They will have a representative sample if they enrol everybody and they will enrol everyone if they do not take that consent.
But justification for cluster randomisation is really important.
Individual randomisation, that is the gold standard approach without any doubt, policy evaluations, we cannot use individual randomisation.
So in all of these examples I've put up today, you couldn't possibly use individual randomisation, even though it's the gold standard.
And that's because your cluster was an entire unit.
But sometimes people have started to manipulate the the question of a policy,
the effect of an intervention into a question of policy just to avoid individual patient consent.
It does make things a lot easier. It's not a good idea.
Firstly, it's ethically inappropriate. And secondly, it can at increased risk of risks of bias.
And I haven't really gone into these increased risks of bias and I don't have time to.
But there are some other risks of bias that I haven't mentioned,
that you do open yourselves up into those if you start to manipulate these questions of.
Drug intervention effects into questions of policy.
Okay. So I think I've run out of time. So very quickly, few minutes back to the stepwise cluster randomised controlled trial.
When should we use that? So when we want to use the cluster randomised design, we have to justify our randomisation of an entire cluster.
So we have to make sure that we've got a get rank justification for randomising an entire cluster.
We randomise an entire cluster just because it avoids us getting individual consent.
So it really has to be a cluster level intervention.
But we additionally have to think about why we have to roll the intervention to all of the clusters, why we need to have this staggered rollout.
The reasons why we have to justify and think carefully about this design, I hope have become evident.
It's a design that's confounded by time.
And so any analysis that we do always has these many caveats.
And at the end of the day, after running a study, we don't want to have to worry about these caveats and about these risks of bias.
So we have to make sure that we really think carefully about what we're going to use it, because it's at increased risk of bias.
Here are some possible justifications.
So we started out at the beginning talking about the second justification here that is appealing to those people that we're going to try to recruit.
So the cluster stakeholders, it has what we might call a social appeal to it.
Sometimes it can increase the feasibility of things.
Sometimes it can be a statistically more powerful design.
And I haven't gone into power. It's you have to work this out on a case by case pace basis.
Sometimes it can be more powerful. It's not always the case.
We also have to consider other things like Is it going to make our study run longer?
How long is it going to take to realise the effect of the intervention?
So there are all these added complications as well that we need to consider.
So I think the real justification that you really don't have to worry about too much is when you
are going to distribute a scarce resource or this intervention is going to be rolled out anyway.
And only very occasionally does that happen.
So maybe what's going to happen in our system is somebody is going to roll out something irrespective of any evaluation.
And perhaps they're going to distribute a scarce resource. Maybe not.
But somebody is going to be rolling out something if in addition, you can persuade them to let you randomise the order of that rollout.
And again, that's very rare. When does that happen?
Not very often. But if you could if those two things held, then maybe that would be a good justification for doing a stepwise trial.
And especially if the alternative was going to be some random non-randomized design like a before and after study.
The study design is definitely higher on the hierarchy than a before and after study.
When does that happen? I think there are very few and far between studies where they really have had good justification for using the stepwise design.
And this example, it's the gun bang stretch cluster, randomised controlled trial.
It's a famous iconic example. They didn't call it stepwise trial design.
So if you search for step twice in the literature, you won't find this.
But it was a stepwise design and in this design it's called the Gumby and Hepatitis Intervention Study.
They were trying to evaluate whether HPV vaccine prevents liver cancer.
I mean, that's the sort of the basic idea behind this study. Does HPV vaccine prevent people from getting liver cancer?
One study was carried out in The Gambia, and there are 17 geographical regions in The Gambia.
They said that universal vaccination in the entire country was impossible for logistical and financial reasons.
Now that's probably very believable. They were going to do this at a country wide scale.
It's very unlikely that an entire country can roll out something new to an entire population at the same time.
So they were going to roll out this HPV vaccination sequentially.
The study design has managed to persuade the stakeholders in the Gambia to allow them to do this randomly.
So these 17 geographical areas were actually exposed to this HPV vaccine.
And by that, what happened was that if a baby was born in any of these villages after the
time after the time at which that cluster was going to get the HPV vaccine,
then the baby was vaccinated. If it was born in one of these villages before the village had crossed over, then it didn't get the HPV vaccine.
They then followed up the babies for 30 years.
Still hasn't reported. I'm going to report very soon, but it hasn't reported just yet.
And then they're going to compare the liver cancer rates between that group and that group.
There will be a big caveat in their analysis that they have to adjust for time, but they will adjust for time.
They also have to make the assumption that the secular trend is the same across all of the clusters.
They don't have very many clusters here, so there will be some concerns about that assumption.
But the alternative here would have probably been a before and after study.
But this is the only example that I can find in the electric chair where it really was the case that
they persuaded the stakeholders to randomise something that was going to be rolled out anyway.
Most of the time people are just using this design because it's convenient or it's fashionable.
So summing up, there are these menus of different types of study designs, and I haven't spoken about some of them.
These ones here alternate between intervention and control, but in policy evaluations, we typically can only go one way.
We can't take away something, but we put it in place.
If you can take away whatever you want and we evaluate these types of designs where you cross backwards and forwards are much better.
Which is the best design. The first thing is it should be low risk of bias.
It should be feasible and that it should be statistically possible.
I think a lot of the literature has emphasised statistical power over bias.
But I think we should be more concerned with bias. Which design is least bias?
Well, the stepwise design requires this model based analysis, so it's not going to be the step wedge trial.
What design is more feasible? Might it be the to trial?
Well, people often say they're using it because it's feasible, but actually it turns out that it has its own complexities.
You have to get all of the clusters ready to start at the same time.
Actually, that turns out to be really difficult. People start at the clusters aren't ready to go, then they don't know what to do.
So it has its own complexities. Is it more statistically powerful?
Well, that depends. Sometimes it can be, but it's not always more statistically powerful.
I guess it really does come down to power. Then you have to work out on a case by case basis which one of these designs is more powerful.
There are some generic statements in the literature that say the stepwise design is more powerful.
It's not. It can be. It's not always. So if you can't randomise, then my advice is to use an interrupted time series type of analysis.
You have to be willing to accept that if you get a big shift, you're probably going to be alright.
But if you're looking for small, subtle changes,
you might miss them in that type of design or you might not be able to unequivocally say that they're because of your intervention.
If you can have many clusters in your Interrupted Time series, then that's going to help.
If you can randomise, the parallel design has fewer risks if there are a small number of clusters.
The stepwise trial is probably going to be your design of choice, but only in the situations where you don't have a large number of clusters.
Because if you have a large number of clusters, if you did a parallel design, you won't have to worry about time trends.
Thank you. I think.