1 00:00:00,030 --> 00:00:03,790 Like to do is take you to Georgia and facts look, focusing on the policy. 2 00:00:03,810 --> 00:00:05,400 It's a tuberculosis policy. 3 00:00:05,400 --> 00:00:14,790 They are still in the process of redefining and it will give us plenty of angles to look at its release evaluation and how you actually do it. 4 00:00:15,390 --> 00:00:23,910 But I focus on the number of challenges. Not all of it, of course, but if things are not clear, if you want to resolve things, as I said, just ask me. 5 00:00:24,360 --> 00:00:32,879 So I don't think I should repeat much this kind of mantras. I think the starting point for our use of healers in Georgia was that there is 6 00:00:32,880 --> 00:00:36,150 this promise of redistribution that you can do it in a very participatory way, 7 00:00:36,570 --> 00:00:42,750 that you can start from a program theory that you develop together with the stakeholders, in this case with policy makers. 8 00:00:43,230 --> 00:00:51,630 And the idea was actually to use the whole released cycle to at the same time, in fact, develop a research program on this policy, 9 00:00:52,080 --> 00:00:58,950 kind of find out how we could check and test effectiveness costs and the processes of implementation, 10 00:00:59,130 --> 00:01:04,170 but also actually informing the policy making process by engaging policymakers in the whole exercise. 11 00:01:04,650 --> 00:01:08,300 And so this kind of a double promise is actually what we started out with here. 12 00:01:08,400 --> 00:01:13,620 And in this project, seven principles or eight or nine. 13 00:01:13,620 --> 00:01:17,099 There's a number of lists out there. I'm again, I'm not going to go very deep in that. 14 00:01:17,100 --> 00:01:24,960 I know you are into this. I think just to say that at some point you will have some questions. 15 00:01:25,050 --> 00:01:31,590 So the configuration analysis part is to make sure that you link up your policy implementation to outcomes. 16 00:01:31,830 --> 00:01:36,270 This black box of mechanisms and contexts and the people in it and that all of that 17 00:01:36,270 --> 00:01:40,349 makes sense and that you do the analysis not in segmentations or by by segments, 18 00:01:40,350 --> 00:01:43,080 but really kind of tied up at the end. 19 00:01:44,880 --> 00:01:52,620 We use an this just kind of a cycle or kind of a diagram to to structure things a bit to make our students see how we go about designing the study, 20 00:01:53,040 --> 00:01:56,280 but also for funding proposals, that kind of stuff. 21 00:01:56,790 --> 00:02:02,310 So the whole thing starts with research questions or your research objectives, your evaluation questions if you want. 22 00:02:02,820 --> 00:02:09,000 You know, by now the programme chief is a starting point, his end point, but also the starting point of any released cycle. 23 00:02:09,420 --> 00:02:16,650 Study design, data collection, data analysis will be in fact decided upon by your actual programme theory, 24 00:02:16,770 --> 00:02:20,370 the programme, which is of course responding to these kind of questions. 25 00:02:20,850 --> 00:02:23,489 And then at the end of the whole empirical cycle, 26 00:02:23,490 --> 00:02:31,810 you try to summarise synthesise kind of pull your CMO so those into the programme trigger and you design that and then you know, 27 00:02:31,860 --> 00:02:37,110 the cycle can go on as long as your funding actually takes. And quite often that's not very long. 28 00:02:37,110 --> 00:02:43,830 Of course, at some point then we'll have recommendations for policymakers or for any kind of commissioners of the evaluation. 29 00:02:44,280 --> 00:02:51,120 So that's basically two steps, and we'll focus a lot now here on the programme to resist for a large extent. 30 00:02:51,360 --> 00:02:53,669 To a large extent, because this and that hasn't been done as well. 31 00:02:53,670 --> 00:02:59,790 But we're not yet in this phase, which we had to introduce, which we'll come back to at the end of the presentation. 32 00:03:01,410 --> 00:03:10,140 So the starting point is this programme theory, which is perhaps easily understood as kind of the assumptions, the ideas that underlie a policy, 33 00:03:10,550 --> 00:03:17,460 and that should explain why that policy would be expected to lead to particular outcomes for the target population. 34 00:03:18,030 --> 00:03:26,939 Because basically the idea here and one of so is not only actually evolution stream using these 35 00:03:26,940 --> 00:03:32,280 kind of principles to reduce evaluation of change largely stems from the same principles, 36 00:03:32,760 --> 00:03:39,060 the idea that if you develop that programme from the beginning, which people involved in the programme or in the intervention, 37 00:03:39,210 --> 00:03:42,240 there's a high chance that you would get a common understanding of what's happening 38 00:03:42,240 --> 00:03:46,980 because you identify critical issues and you might also actually push people to, 39 00:03:47,430 --> 00:03:52,560 to develop or better. Actually, the policy did not develop and we see that actually happening in Georgia. 40 00:03:54,900 --> 00:03:59,160 If you go into the causal pathways from the very start, it allows for a better evaluation design. 41 00:04:00,060 --> 00:04:06,150 And so if you develop this programme theory from the beginning with the policy makers but also with the research team, 42 00:04:06,420 --> 00:04:15,690 and in this case we had economists there, public health people, health systems people, some psychologists and a number of different disciplines. 43 00:04:15,690 --> 00:04:22,380 Basically, if you want them to contribute to your thing, it helps to have something to work on and doesn't disclose the causal pathways, 44 00:04:22,680 --> 00:04:28,649 which then actually leads to a better design of the evaluation because you will see that also you can actually start mapping 45 00:04:28,650 --> 00:04:35,460 out on your program theory the kind of data points you need and the methods you will actually use to collect the data. 46 00:04:39,420 --> 00:04:42,170 By now, this should be non-use nature. 47 00:04:42,190 --> 00:04:49,050 If you the assumption somehow is that we all have ideas about why things work, we don't necessarily make that explicit. 48 00:04:49,320 --> 00:04:50,010 Definitely not. 49 00:04:50,010 --> 00:04:58,830 When you are policy developers, a programme designer, there's a lot of hidden assumptions quite often that it's good to to understand that because. 50 00:05:00,530 --> 00:05:03,740 It still is to a large extent actually what would be implemented and why. Okay. 51 00:05:04,340 --> 00:05:12,990 Now, there are a number of ways to to come to that initial program theory for theories which are programmed to use held by the designers, 52 00:05:13,020 --> 00:05:17,089 the people, beneficiaries, the actors involved. Of course, there's a lot of existing knowledge. 53 00:05:17,090 --> 00:05:21,350 I think probably everything we kind of look at in medicine or in public health has been done somewhere. 54 00:05:22,130 --> 00:05:26,120 There's lots of research that's very relevant for our kind of policy implementation 55 00:05:26,120 --> 00:05:29,600 problems that have been developed in disciplines like public administration, 56 00:05:30,110 --> 00:05:37,590 psychology, economics. So it's helps to to find out how you can get into that knowledge and then if nothing would, exists, exist. 57 00:05:37,700 --> 00:05:40,610 There's always a possibility to start with exploratory research. 58 00:05:40,940 --> 00:05:48,710 So that's kind of basically three categories of who is, let's say, that helps you to to continue your initial program theory. 59 00:05:48,980 --> 00:05:54,050 Now, Georgia, on a bad day, actually, it was quite sunny at that stage, but cloudy. 60 00:05:54,800 --> 00:05:59,450 This is about to me. I don't know if you know where Georgia lies. It's not the American Georgia, of course. 61 00:06:00,140 --> 00:06:04,610 We're into ex-U.S. desert between the Black Sea and the Caspian Sea. 62 00:06:05,450 --> 00:06:09,230 It's in the middle of the Caucasus Mountains. It's a highly political area. 63 00:06:10,820 --> 00:06:14,800 You can see this large chunk of the world, which is the Soviet Russia, 64 00:06:14,810 --> 00:06:19,130 the Soviet Union, Russia, they are still quite involved at the northern borders. 65 00:06:19,580 --> 00:06:26,390 It's a relatively big country, but it has only 6 million people. It's going to have sparsely populated and highly mountainous. 66 00:06:27,020 --> 00:06:36,590 Now from ITB is that it is for it's better now because if you look at the incidents maps, the dark green areas are the worse off. 67 00:06:37,030 --> 00:06:39,799 That's basically to certain parts of Africa. 68 00:06:39,800 --> 00:06:44,720 And that has to a large extent to do with HIV, AIDS infections, with also quite still some in South East Asia. 69 00:06:45,170 --> 00:06:48,210 The excuse is that republics hold quite high levels. 70 00:06:48,740 --> 00:06:52,070 When the USSR broke up, a lot of the systems collapsed. 71 00:06:52,610 --> 00:06:56,600 Georgia kind of picked up, but they're still in the high range here. 72 00:06:56,990 --> 00:06:59,719 So they're kind of this light greenish kind of thing. 73 00:06:59,720 --> 00:07:06,770 But as we see at the upper limit of that, so and so, 92 people out of 100,000 will get TB every year. 74 00:07:08,150 --> 00:07:14,750 The problem is not only that, but actually the success rates are not very good and there's a huge variation geographically in the country. 75 00:07:15,260 --> 00:07:24,079 So in the big cities where the clinics are good, you'll find for drug sensitive routes through sensitive TB patients that has 100% success rates, 76 00:07:24,080 --> 00:07:27,680 but only less than half in a number of other places. 77 00:07:28,460 --> 00:07:32,780 There's quite some multidrug resistant tuberculosis as well, which is quite problematic, of course, 78 00:07:32,780 --> 00:07:38,210 with these people will not get cured and this resistance is also occurring in people never treated. 79 00:07:38,360 --> 00:07:46,610 So in that sense, the multidrug resistant and the and the extreme resistance case is really what concerns policymakers. 80 00:07:46,610 --> 00:07:53,660 And they had to explanations when they looked into the policy in around 2015, 2016. 81 00:07:54,260 --> 00:07:57,290 So one was adherence of patients to the treatment is not good. 82 00:07:57,920 --> 00:08:01,040 And as you know, it's a long treatment. We'll come back to that in a second. 83 00:08:01,610 --> 00:08:03,740 But also a lot of people actually opt out of the treatments. 84 00:08:04,190 --> 00:08:11,329 So when started, they actually had a lot of side effects or didn't feel very, very well cared for and did drop out. 85 00:08:11,330 --> 00:08:12,740 And that was initial analysis. 86 00:08:15,650 --> 00:08:22,400 It has to do to a large extent to the huge wave of privatisation that happens after the collapse of the Soviet Union in 91. 87 00:08:23,090 --> 00:08:28,910 Now it took a bit of time for them to get the rules revolution and then Shevardnadze and a lot of the old apparatchiks were kicked out. 88 00:08:29,750 --> 00:08:36,080 And in that period from in fact already here it was a very wild privatisation going on. 89 00:08:36,290 --> 00:08:49,189 So one of the stories is that they actually gave for symbolic ruble, almost huge hospital complexes in the middle of Tbilisi as a kind of lease, 90 00:08:49,190 --> 00:08:56,810 a five year lease to private companies who wouldn't where they were just supposed to render services for five years for the symbolic one ruble, 91 00:08:57,170 --> 00:09:01,160 and then they could do whatever they wanted to just put them down. And they built shopping malls. 92 00:09:01,970 --> 00:09:07,580 And so it's a bit the spirit of this wild capitalism that really is quite, quite strong in Georgia. 93 00:09:08,360 --> 00:09:16,310 So it led to what we call the provider split, a complete separation of the states functions and the provision of health care. 94 00:09:16,580 --> 00:09:25,370 It was completely privatised. And in fact, for TB, it's the private sector that actually does the queues, the cash, but also preventive activities. 95 00:09:26,030 --> 00:09:29,960 So there's very little left at the centre level. 96 00:09:29,990 --> 00:09:32,930 There's the Russian tuberculosis program. They do the policymaking. 97 00:09:33,680 --> 00:09:40,760 They're supposed to do some in supervision, supervision and guidance over the private practices as well. 98 00:09:41,510 --> 00:09:43,720 The private sector is also very heterogeneous. 99 00:09:43,730 --> 00:09:52,040 There's a small companies running a number of clinics in the few cities that are mega companies running kind of chains all over the country. 100 00:09:52,040 --> 00:09:59,300 They're private practitioners still there, nurses operating small clinics. So it's a very kind of diverse picture, but it's not really priority for. 101 00:09:59,970 --> 00:10:06,330 Because there's not much money in it physically unless we see patients who need to be observed every day. 102 00:10:06,480 --> 00:10:12,060 In fact, when taking the pills or every 2 to 3 days, it's a huge workload for, in fact, very little money. 103 00:10:12,660 --> 00:10:20,490 Now, government foresaw that already, of course, and they had a TB service agreement which forced the private sector to offer a private I mean, 104 00:10:21,360 --> 00:10:27,330 TB programme activities within private companies. But that agreement was supposed to end by 2018. 105 00:10:27,690 --> 00:10:33,810 And so policymakers said you need to to kind of incentivise the private sector now to really take on and continue the treatments. 106 00:10:34,440 --> 00:10:41,219 But besides that, they also saw quite some problems between this primary health care or first care and first level of care, 107 00:10:41,220 --> 00:10:44,330 in fact, and the TB services in the big cities. 108 00:10:44,340 --> 00:10:54,450 So they had a number of problems there. Just to give you an idea, that was a not so directly observed treatment regimen for TB in Georgia, 109 00:10:54,450 --> 00:10:58,350 which is kind of an international protocol it takes. And that's a bit strange. 110 00:10:59,190 --> 00:11:07,830 People had to be hospitalised for six weeks and four months initially, in both cases drug sensitive and drug resistant TB patients. 111 00:11:08,220 --> 00:11:09,450 But it's even takes six months. 112 00:11:10,140 --> 00:11:17,340 And so in some cases, they went to see somebody by the patient would come twice a week to the clinic, take the drugs under supervision, 113 00:11:17,490 --> 00:11:22,170 swallow and actually almost check, and then take some pills home for the next two days, 114 00:11:22,170 --> 00:11:25,080 come back to the clinic to be seen actually taking two tablets. 115 00:11:25,110 --> 00:11:30,690 It's a very from a political perspective, I can see the point, but from a from a human perspective, 116 00:11:30,990 --> 00:11:34,809 this kind of direct observed treatment that I can't understand that you ever did, 117 00:11:34,810 --> 00:11:37,980 that I mean, that you forced people to come and show that they swallow tablets. 118 00:11:38,610 --> 00:11:44,490 It's crazy. And so you can imagine for patients, it's not the nicest of kind of experience. 119 00:11:44,880 --> 00:11:47,550 But for private providers, actually, they couldn't care less almost. 120 00:11:48,510 --> 00:11:54,390 That's not to say that there are not good clinics in Georgia because there is a not all that bad, but it's a problem. 121 00:11:54,900 --> 00:12:02,639 So what they said is, okay, if we need to make sure now that the private providers properly take care of their TV patients and can motivate 122 00:12:02,640 --> 00:12:07,200 them to come every 2 to 3 days and make sure that is sit out the whole treatment we need to incentivise them. 123 00:12:07,950 --> 00:12:11,430 It's a market system. It's a it's a purely private for profit system. 124 00:12:11,730 --> 00:12:17,450 So what you do, you just dangle them a carrot and you hope you have some sticks to beat them if they don't. 125 00:12:17,460 --> 00:12:23,970 But there are not many sticks in Georgia. It's the country that's really the the policy option taken. 126 00:12:24,900 --> 00:12:28,770 So that kind of a decision was taken in around 2015. 127 00:12:29,070 --> 00:12:33,510 Policymakers said we need to do something about the situation. We are going to incentivise them. 128 00:12:34,080 --> 00:12:41,520 And they asked actually a local research centre, the Grassroots Fund International Foundation in Tbilisi, 129 00:12:41,940 --> 00:12:46,830 to set up a program to help them actually to develop the policy and at the same time to evaluate it. 130 00:12:47,340 --> 00:12:53,969 And that was part of the grants they got from the Global Fund to actually roll out a three 131 00:12:53,970 --> 00:12:58,420 year pilot project testing disincentives so that there's a whole set of the projects. 132 00:12:58,920 --> 00:13:03,930 And it got funded at some points by the Health Systems Research Initiative of the MRC. 133 00:13:05,040 --> 00:13:11,970 So what we then did was in fact put together the whole consortium, health economists, health systems, people, some TB guys. 134 00:13:13,380 --> 00:13:17,670 The idea was we have a three pronged approach in terms of evaluation, the impact assessments, 135 00:13:17,680 --> 00:13:25,440 the cost analysis and a process evaluation or a policy implementation assessment, but then based on the release approach. 136 00:13:25,830 --> 00:13:28,200 Now that's kind of common. That's what you see in a lot of situations. 137 00:13:28,200 --> 00:13:33,660 Of course, you have kind of any kind of trial combined with cross evaluations, whatever. 138 00:13:34,080 --> 00:13:41,420 What you did here was to set out to kind of bring all these strands together from the very beginning and based on the program theory. 139 00:13:42,060 --> 00:13:48,780 So here to set up is to try from the very beginning actually to come together with the policymakers, but also the researchers, 140 00:13:49,020 --> 00:13:54,380 and to have one program theory that would help us set up the three other elements of the research 141 00:13:54,390 --> 00:13:59,910 actual data collection to see and to have then one common framework not only to guide the process, 142 00:13:59,910 --> 00:14:08,790 but also to help us with the analysis at the end. So that and at the same time, because we would engage with the policymakers, 143 00:14:09,180 --> 00:14:14,090 the whole monitoring system as well as implementation would be easier to follow up. 144 00:14:14,100 --> 00:14:21,810 That was kind of the promise. So program two here deals with the intended policy will have a combination of designs. 145 00:14:21,900 --> 00:14:29,910 We'll have an integrated data collection tool basically that maps out all kinds of studies that we need or subseries to answer the big questions, 146 00:14:30,390 --> 00:14:35,460 a much more integrated data analysis, a more parsimonious analysis, a more parsimonious data collection also. 147 00:14:35,940 --> 00:14:39,060 And that should help us and to inform the scaling of this. 148 00:14:40,110 --> 00:14:45,120 Now, it's been we had actually been doing that more or less in another project before this one from Health, 149 00:14:45,240 --> 00:14:53,970 which focussed on the exemption of maternal care fees or exemption of fees for maternal health services in West Africa, 150 00:14:54,360 --> 00:14:59,310 which was a huge policy thing like five, ten years ago here in this project. 151 00:14:59,880 --> 00:15:01,380 We did not do it from the very start. 152 00:15:01,380 --> 00:15:09,510 So what we did was kind of halfway tried to figure out how new units of maternal mortality is kind of influenced by biological complications, 153 00:15:09,510 --> 00:15:14,380 but also by a number of health system factors that you then can start mapping out and to colour things. 154 00:15:14,400 --> 00:15:19,620 I'm not going to go in detail. Bit of colour things are basically subsidies were packages that were focusing on 155 00:15:19,620 --> 00:15:24,450 documenting particular elements of the ocean and this was a study done in four countries. 156 00:15:24,870 --> 00:15:33,160 So it helped us to have kind of an overall strategy and some standardisation of tools, but also to make sure that there would not be too much overlap, 157 00:15:33,190 --> 00:15:41,390 not too many gaps, in fact, and a much more integrated analysis that was more or less what we achieved in females. 158 00:15:41,400 --> 00:15:45,990 But here we did not set out to the whole thing. We didn't start from the program theory. 159 00:15:47,220 --> 00:15:57,930 So as we said already, for theorists literally to use it for research that could be used to to formulate your initial program theory here, 160 00:15:58,290 --> 00:16:04,859 we actually did the teachings on top and then the kind of reef system. 161 00:16:04,860 --> 00:16:09,930 If you once we started formulating the initial program, cheering on the basis of policy documents, 162 00:16:10,530 --> 00:16:13,710 of course, had been quite sometimes written on the policy in Georgia as well, 163 00:16:14,190 --> 00:16:19,139 but also internationally, literature reviews on policy implementation, on TB, 164 00:16:19,140 --> 00:16:23,700 programs on the problems of the Caucasus, and then interviews with the program managers. 165 00:16:23,700 --> 00:16:27,270 So that was the very start to kind of a basic starting point. 166 00:16:27,720 --> 00:16:33,480 We took this initial program to read into a stakeholder a kind of a concept mapping workshop, 167 00:16:33,480 --> 00:16:36,780 if you want, or a stakeholder workshop where we brought together a lot of people. 168 00:16:37,320 --> 00:16:44,780 Then that led us to the initial program to number one, there will be holding an additional document review, again, 169 00:16:44,790 --> 00:16:52,080 discussion with policymakers because of political context that changed and then a new kind of program theory that was taken to the second workshop. 170 00:16:53,010 --> 00:16:55,710 And we see there was a slight difference in focus there. 171 00:16:56,070 --> 00:17:03,330 We ended up with the third one and then that one actually was taken up by both researchers and led to a research strategy, 172 00:17:03,330 --> 00:17:06,720 if you want, and with the policy makers to new TB policy. 173 00:17:07,380 --> 00:17:13,380 So we did this process over a period of four months, almost two workshops in similar time. 174 00:17:13,380 --> 00:17:16,590 Some have a few months here and a few months on the outside. 175 00:17:18,330 --> 00:17:23,190 Okay. What is, of course, the importance, if you want to engage stakeholders, is to find the right ones. 176 00:17:24,000 --> 00:17:30,400 And what is quite often not happening is providers and and patients. 177 00:17:30,450 --> 00:17:34,680 In this case we did not have patients, but we had representatives of patients organisations, 178 00:17:35,160 --> 00:17:39,270 but there was a quite good mobilisation of about 45 participants. 179 00:17:39,270 --> 00:17:46,710 The effect of all these levels here. So policymakers, the program managers, of course from the national system, but also the funders, 180 00:17:47,400 --> 00:17:52,170 researchers and then nurses, doctors actually from all regions of Georgia. 181 00:17:52,740 --> 00:17:57,150 Because it's one thing easy, in fact, to get to the people working in the big cities. 182 00:17:57,660 --> 00:18:04,979 You can actually get them and know them. But we had, in fact, quite a number of providers from far away and more much more rural areas. 183 00:18:04,980 --> 00:18:11,790 And that's, as we'll see, actually changed quite, quite strongly the dynamics of the group, but also led to quite some new insights. 184 00:18:12,780 --> 00:18:17,790 Now, if you want to set your program theory, you have this kind of very diverse public. 185 00:18:18,330 --> 00:18:24,270 It helps to have a starting point somehow. And so here this is so an advisor. 186 00:18:24,450 --> 00:18:32,550 Is the health economist involved with one of them? She had been working on the TV programs and cost effectiveness analysis. 187 00:18:32,580 --> 00:18:38,820 They had been working on a kind of reframe. What we did is kind of simply take out the central part of that framework, 188 00:18:39,390 --> 00:18:46,290 which is nothing more than setting out the different steps people would go through from onset of symptoms to being treated. 189 00:18:46,800 --> 00:18:55,020 That's kind of we call that in Antwerp. And operation analysis kind of tool is pure model of TB implementation program evaluation, if you want. 190 00:18:55,620 --> 00:18:58,830 It's nothing new there, but it's helped a lot for this public. 191 00:18:59,190 --> 00:19:07,110 That's a public that's really geared towards TB and program management, doctors and nurses to have something quite tangible and relatively simple. 192 00:19:07,620 --> 00:19:15,870 So we did not use the full blown thing here. We just told them, look, in the first workshop, try to figure out in this kind of a chain, 193 00:19:15,870 --> 00:19:19,320 where are the big problems, where are the bottlenecks, why are people dropping out? 194 00:19:20,220 --> 00:19:27,690 So it's it was used as a first workshop, actually, as a way to to analyse the problem by bringing the different stakeholders together, 195 00:19:28,620 --> 00:19:34,800 but also in the second spheres of that same workshop to make them think about solutions and then prioritise solutions. 196 00:19:35,010 --> 00:19:36,770 Because key policy is about, of course, 197 00:19:36,810 --> 00:19:45,780 priorities and a lot of discussions and interactive small group meetings to try to come to not a consensus, but to actually find rival theories. 198 00:19:45,780 --> 00:19:51,809 So again, the problem with program driven or theory driven evaluation methods is that you 199 00:19:51,810 --> 00:19:55,560 start out with a program theory and that there might be a tendency to confirm that, 200 00:19:55,860 --> 00:20:02,040 as we always do. You have ideas the. My mind is kind of set up, too, to confirm what we think. 201 00:20:02,550 --> 00:20:09,180 To avoid that kind of tunnel vision, you really need to continuously push people to look for alternative explanations. 202 00:20:09,180 --> 00:20:13,620 So that actually happens because we are facilitated, of course, for each of the groups. 203 00:20:15,090 --> 00:20:22,739 So what it actually did was discover not really, of course, because we knew that I mean, the very, very starting point. 204 00:20:22,740 --> 00:20:28,049 Why would you simply incentivise private providers and hope that that would change the situation in Georgia? 205 00:20:28,050 --> 00:20:32,580 Of course, we know there's a whole systemic set of factors that you need to tackle at the same time, 206 00:20:33,180 --> 00:20:38,009 for TB patients actually to be conformable to accept the care and to actually sit out the whole thing, 207 00:20:38,010 --> 00:20:42,780 you know that it is not simply providers actually needing an incentive to change their attitudes, 208 00:20:43,440 --> 00:20:47,580 but the policymakers who are not much convinced of much else. And so they were actually involved in this workshop. 209 00:20:47,910 --> 00:20:51,300 And for them, these kind of contextual factors and system factors, 210 00:20:51,420 --> 00:20:55,920 service related factors and related factors were quite, quite shocking to some extent. 211 00:20:56,400 --> 00:21:01,860 So it led not only to identify these constraints much better, but also to considering, 212 00:21:01,860 --> 00:21:06,390 in fact, a broader package of interventions to be in the policy. 213 00:21:07,290 --> 00:21:11,420 Now, again, no need to reduce your risk. 214 00:21:11,460 --> 00:21:14,580 So this is one way to summarise the findings of that first workshop. 215 00:21:14,940 --> 00:21:18,179 You remember vividly this central pattern of the patient. 216 00:21:18,180 --> 00:21:27,510 We kind of expanded it a bit. Uh, it went up to the organisational settings, this kind of facility level issues, management strategy, 217 00:21:27,780 --> 00:21:32,040 then providers on this side again geographical accessibility, acceptability, 218 00:21:32,040 --> 00:21:35,430 financial accessibility of service related factor that you could factor in. 219 00:21:35,820 --> 00:21:40,229 And then that chain was also developed further with a number of other things. 220 00:21:40,230 --> 00:21:44,820 Don't worry about that. It helped us to somehow keep track of the discussions. 221 00:21:47,060 --> 00:21:52,070 First points for step taken workshops. So we're at this level here now, more or less. 222 00:21:52,760 --> 00:21:56,720 There's also a bit of a narrative, uh, program theory version if you want. 223 00:21:57,020 --> 00:22:03,290 And that was actually done discussed with, again, policymakers. So bit of a strange thing there. 224 00:22:03,440 --> 00:22:06,380 One of the people of the foundation was a member of Parliament, 225 00:22:06,920 --> 00:22:13,240 was the head of the medical commission in or the Health Commission in parliament was closely involved with policymakers, 226 00:22:13,610 --> 00:22:15,090 had a lot of inside information. 227 00:22:15,110 --> 00:22:22,610 Actually, it allowed it allowed for a lot of discussions at high level through this kind of dual position researcher, policy politician, actually. 228 00:22:23,540 --> 00:22:30,260 But these discussions actually were quite important because they led to a number of changes in the policy design, basically. 229 00:22:30,620 --> 00:22:35,639 And then we did some additional document review because what I come back to that later actually started 230 00:22:35,640 --> 00:22:42,350 with this and gets into the policymakers first thing here in a number of these people changed. 231 00:22:42,890 --> 00:22:51,590 And in the process of this program since about 2017, I think we've had three Minister of Health to a different Secretary-General's, 232 00:22:51,950 --> 00:22:58,870 a number of different heads of departments as well. So there's quite a fast turnover of politicians and policy makers there. 233 00:22:58,880 --> 00:23:01,940 That complicates matters. They come. It's different ideas. 234 00:23:01,940 --> 00:23:05,090 They need to be briefed. You need to kind of see what's happening. 235 00:23:05,390 --> 00:23:09,920 We need to understand what they want, actually. So that that was an important phase here. 236 00:23:10,820 --> 00:23:16,700 And what that actually did was where initially the whole policy was about incentives to individual providers. 237 00:23:17,150 --> 00:23:24,080 It was a much more comprehensive package developed. And again, I could have told you that that's what we need from the very start. 238 00:23:24,350 --> 00:23:25,879 But of course it doesn't work that way. 239 00:23:25,880 --> 00:23:32,900 It's not us going to deliver the needs people need to discover what they actually need to do and and the engagement, 240 00:23:32,900 --> 00:23:37,900 the kind of ownership that as a result of this this kind of approach process, I think it was an important outcome. 241 00:23:41,840 --> 00:23:48,500 What you then did in terms of having this discussion with policymakers and the previous response was to go a bit further into 242 00:23:48,530 --> 00:23:57,679 into policy implementation and especially kind of theories and mechanisms which were not identified by the participants. 243 00:23:57,680 --> 00:24:03,499 So when you engage with stakeholders, they are not really they're not looking necessarily for mechanisms. 244 00:24:03,500 --> 00:24:06,860 They might not even have a clue about what the mechanism is for them. 245 00:24:06,860 --> 00:24:13,879 They look at bottlenecks and problems. They see very kind of anecdotal evidence you need to bring them to come to to substantiate their claims. 246 00:24:13,880 --> 00:24:20,060 And that's the process of in the workshops, actually making them argue why they think there's a problem, why that could be a solution. 247 00:24:20,360 --> 00:24:24,740 But they don't go necessarily into the deep, deep analysis and definitely not the level of mechanism. 248 00:24:24,750 --> 00:24:32,750 So at that stage we put in the literature reviews and some other things we had done in the past on adherence. 249 00:24:33,350 --> 00:24:36,050 And then a number of these of course, exist quite soon. 250 00:24:36,440 --> 00:24:43,610 We have taken this information motivation behavioural skills model, which is quite used in HIV research, 251 00:24:44,090 --> 00:24:49,129 where it's used to explain adherence to treatments and self-determination theory, 252 00:24:49,130 --> 00:24:58,490 which is a motivation theory quite well developed since the 1970s and also applied to the motivation of patients in taking drugs. 253 00:24:59,210 --> 00:25:02,420 So it was a quite useful tool that helps us to to model things. 254 00:25:02,420 --> 00:25:09,890 So we had kind of simple models, let's say, to to think through again why people in this case, 255 00:25:10,640 --> 00:25:18,020 women with children who needed drugs would be initiated on the team or not maintain and keep taking the drugs. 256 00:25:18,470 --> 00:25:23,060 And okay, this was a very simple starting point of that, that previous study. 257 00:25:23,070 --> 00:25:30,470 So you can combine, of course, insights from the stakeholders workshops with existing evidence, with the notice that is already out there, 258 00:25:30,860 --> 00:25:36,810 with evaluations from programs and previous studies and kind of accumulate gradually the knowledge and specified. 259 00:25:36,890 --> 00:25:40,210 See whether this explains the situation in Georgia, as you know. 260 00:25:40,220 --> 00:25:46,070 So it's a bit about the possibility of this kind of middle range theory elements, if you want. 261 00:25:47,270 --> 00:25:53,540 We also try to find out a way to summarise these programs using causal diagrams. 262 00:25:53,540 --> 00:25:57,739 We are not going to don't worry, because it's not a terrible thing. 263 00:25:57,740 --> 00:26:02,390 This thing this is the simple one. Let's say there are a number of layers in this thing. 264 00:26:02,660 --> 00:26:08,389 So, of course, loop diagrams allow you graphically to map out relationships between different elements, 265 00:26:08,390 --> 00:26:13,790 if you want of your causal chain is of course, much more advanced than simple drawing program. 266 00:26:13,790 --> 00:26:19,249 You can actually use this to model the whole thing and to run particular changes in the thing. 267 00:26:19,250 --> 00:26:24,080 But we are not into modelling outcomes or predicting policy changes. 268 00:26:24,770 --> 00:26:34,220 This was useful for us to kind of represent in much more detailed way the feedback loops and at some point we are stuck here. 269 00:26:34,340 --> 00:26:41,540 I mean it was not for for a number of reasons, but also because of the the sheer kind of complicated ness of this thing. 270 00:26:41,540 --> 00:26:45,940 Actually. Anyway, just to say that the discussions and that's always the thing. 271 00:26:46,240 --> 00:26:50,920 With modelling, you can believe in modelling or you can be very sceptical. 272 00:26:51,370 --> 00:26:56,739 But develop models actually allows you to again put your hypotheses and assumptions on the 273 00:26:56,740 --> 00:27:01,479 table and to discuss them and to confront people or at least each other's assumptions. 274 00:27:01,480 --> 00:27:05,950 So it helps us to get a much better understanding of the policy as a research group. 275 00:27:07,150 --> 00:27:12,340 And because it made us different kind of assumptions, very so in our group there were people working on PBF, 276 00:27:12,340 --> 00:27:17,830 on performance based financing from health economics perspective, really believing in it, others quite sceptical. 277 00:27:18,130 --> 00:27:22,120 There were people working on tuberculosis, they had lots of knowledge on the TB pathways, 278 00:27:22,390 --> 00:27:26,140 then some epidemiologists from Georgia, some sociologists, some health systems. 279 00:27:26,560 --> 00:27:34,840 Now if you want to kind of have a detailed map, you need to bring all of that together somehow or not, and have a rival explanations. 280 00:27:35,320 --> 00:27:43,870 Okay. But what they did, at least for us, is, again, a see much better which kind of data collection tools we would need to documents, 281 00:27:44,020 --> 00:27:48,460 not the whole map here, but the specific pathways we would be interested in. 282 00:27:50,560 --> 00:27:54,400 And I'll come back to that at the end of its second workshop. 283 00:27:55,300 --> 00:27:58,570 So we had a bit of a program. Now it was a policy decision taken. 284 00:27:59,590 --> 00:28:05,470 We wanted to bring it back now to again the same group of stakeholders to see how far they were actually agreeing with that or not. 285 00:28:05,980 --> 00:28:10,810 Because again, now the politicians or the policy makers had taken back the initiative and they had developed a new package. 286 00:28:11,020 --> 00:28:16,750 I wanted to see an effort that would make sense if discussed with the actual stakeholders. 287 00:28:17,080 --> 00:28:20,139 Here at Fairless participants, about half of the group did not come. 288 00:28:20,140 --> 00:28:26,320 This time we had about 27. But you have the same mix or the same kind of representation of different groups. 289 00:28:28,240 --> 00:28:31,810 So here the idea was, okay, if this is the policy package, what will it achieve? 290 00:28:31,840 --> 00:28:38,290 So you could call it the action model from the policy to the beneficiaries to the expected effects petition side effects. 291 00:28:38,620 --> 00:28:44,830 And in the second step, then we wanted them to think about, okay, but do you think it will work for which reason, why and which kind of conditions? 292 00:28:45,580 --> 00:28:50,350 So action model, causal model is something that's not really used much in realist that comes from program, 293 00:28:50,500 --> 00:28:57,580 from theory driven evaluation, but it basically does the same thing. So the idea was, where are we in the whole process? 294 00:28:58,210 --> 00:29:04,150 We have at some point the policymakers now have an idea that the package should be much larger than they initially thought. 295 00:29:05,350 --> 00:29:09,879 But then and the researchers, of course, kind of steer that to some extent. 296 00:29:09,880 --> 00:29:12,420 And then this politician, researcher guy also. 297 00:29:12,800 --> 00:29:19,240 So at the end, to make to make sure that that's what they would decide and it would make sense for people in the faraway regions. 298 00:29:19,630 --> 00:29:26,050 You said, okay, let's use this opportunity and the dynamics of engaging with the stakeholders a second time to kind of test back. 299 00:29:26,050 --> 00:29:32,680 In fact, the hypothesis underlying the new policy, which are policymakers, of course, did not make very much explicit. 300 00:29:32,680 --> 00:29:39,250 They don't think in terms of program to use, even if they're involved in these workshops, at some point they revert back to type. 301 00:29:39,730 --> 00:29:44,080 So that's quite normal. Of course, it's not their business. So here we said, okay, let's do two things. 302 00:29:44,620 --> 00:29:49,960 Let's first. So the policy was presented to packages and the way the politicians thought it would work. 303 00:29:50,380 --> 00:29:53,020 And then we said, okay, let's now go a bit deeper into that policy. 304 00:29:53,440 --> 00:29:59,710 And as the participants to think about the action model, which is the link between the policy, 305 00:30:00,310 --> 00:30:03,940 how it will play out the intermediate effects and then the ultimate effects, 306 00:30:04,630 --> 00:30:07,390 which is basically health status or adherence to treatment, 307 00:30:08,800 --> 00:30:15,010 but also not forget about potential side effects because politicians always only see positive effects of things, of course. 308 00:30:15,010 --> 00:30:22,270 But we know that a lot of these what is basically systemic interventions will have some negative effects as well. 309 00:30:22,720 --> 00:30:32,170 Okay, second step was, once you have done that kind of action model thinking, explain why you would think it will work or why not to. 310 00:30:32,170 --> 00:30:38,860 Again, kind of find out whether it be not rival theories or rival explanations or better solutions to to the problem. 311 00:30:39,400 --> 00:30:48,460 So it was kind of a test, again, of the hypothesis underlying the policy and that actually did not lead to much results. 312 00:30:48,850 --> 00:30:52,659 So we're not sure. I mean, we didn't go very deep into the analysis. 313 00:30:52,660 --> 00:30:58,740 Is it because the people were tired at some point, because this was no second workshop or because they actually did not see prompts? 314 00:30:59,140 --> 00:31:03,700 But okay, that's where it was. So other is a result of this. 315 00:31:03,700 --> 00:31:10,000 There were very few modifications of the program during the one we had presented to them actually was more realism intense. 316 00:31:10,690 --> 00:31:14,560 But the package of the policy was again refined a bit. 317 00:31:15,160 --> 00:31:22,660 So where it was initially in the very first step, just incentives to individual providers. 318 00:31:23,380 --> 00:31:30,640 After the first workshop, we went to this package. So you will not be incentives to individual doctors, nurses, managers, but to the whole team. 319 00:31:31,570 --> 00:31:35,410 So one of the critiques was that a lot of people are involved in TB care. 320 00:31:35,410 --> 00:31:39,370 You cannot incentivise individuals, you'll get gaming, you get lots of problems. 321 00:31:39,790 --> 00:31:45,700 There will be a lack of team spirit and collaboration. We know, in fact, that is incentivising of individual providers. 322 00:31:46,020 --> 00:31:52,860 Sequential problems if you don't control it. So here too, I said, listen, have the whole team paid in terms of their performance as a team. 323 00:31:53,580 --> 00:31:57,610 And at the same time, make sure that the team can function better. 324 00:31:57,630 --> 00:32:00,600 So it was it was found out in a lot of these private clinics. 325 00:32:01,020 --> 00:32:08,190 The task distribution, the way people coordinated their work was not optimal between laboratory and the clinicians, 326 00:32:08,550 --> 00:32:16,830 between the reception and divisions that required some gaps. And then the other thing which was lacking was an integrated patient management tool. 327 00:32:16,830 --> 00:32:17,640 They call it a plan. 328 00:32:18,330 --> 00:32:24,450 You would call it a patient file, probably, where you keep, in fact, the whole records of the patient, including the treatment plan, 329 00:32:24,810 --> 00:32:32,910 and use that as a kind of integrated data management tool that would help to provide, but also to facilitate, to better manage care for patients. 330 00:32:33,720 --> 00:32:39,360 And something which might not have been super important but which was felt to be important was the side effects. 331 00:32:39,840 --> 00:32:45,600 So the hypothesis was of a lot of providers that because of the side effects, a lot of people drop out. 332 00:32:45,780 --> 00:32:48,810 And of course, TB drugs are not innocent because quite some problems. 333 00:32:49,230 --> 00:32:54,510 So training for specialists to identify the side effects and to do something about it was deemed important. 334 00:32:54,510 --> 00:33:00,480 So that was before the second workshop and the next kind of package did not change much. 335 00:33:01,890 --> 00:33:09,180 They went into a bit of more details now in terms of TB case management and the bonus system also was fine tuned. 336 00:33:09,480 --> 00:33:14,490 Okay. So we are not going to go into evidence of whole TB scores, if you want, if not more. 337 00:33:14,790 --> 00:33:24,120 But the way you are going to make sure that the team actually gets the money and how you are going to decide how to kick is cut up is quite critical. 338 00:33:24,990 --> 00:33:29,010 Who is playing which role into the performance? How much are you going to want to give to the receptionist? 339 00:33:29,640 --> 00:33:34,500 How much to the doctor? How much to the I don't know, the drive. Who takes the samples to the region hospital? 340 00:33:35,250 --> 00:33:38,819 Each of these people plays a role in the whole systemic thing that you need. 341 00:33:38,820 --> 00:33:41,910 But how much do you need to pay them to motivate them to do better? 342 00:33:42,930 --> 00:33:46,620 So it's a quite a messy intervention if you want. 343 00:33:47,400 --> 00:34:00,209 So we were more or less at the end of the second step here where some of the but not so much of the policy package was again, refined and well, 344 00:34:00,210 --> 00:34:04,530 some examples it is that now we need and that has to ensure good in between 345 00:34:04,530 --> 00:34:08,700 because these are kind of dates from last year but they were no more kind of 346 00:34:09,810 --> 00:34:13,290 standard operating procedures let's say or guidelines developed for how an 347 00:34:13,290 --> 00:34:17,400 integrated care team should should function with a clear distribution of the roles, 348 00:34:17,760 --> 00:34:22,680 with some additional training, with guidelines, with supervision tools developed for that. 349 00:34:23,190 --> 00:34:26,670 Okay. They wanted a patient centred approach. Now, that was a huge discussion. 350 00:34:27,180 --> 00:34:32,130 It means 25,000 things to two different people. Of course, I was surprised. 351 00:34:32,550 --> 00:34:39,380 Yeah, okay. But that was an option. I mean, a problem still. And then, okay, as I said to the bonus payment system was also somehow changed. 352 00:34:39,390 --> 00:34:43,260 This is very much technical discussion in performance based financing. 353 00:34:43,830 --> 00:34:52,610 So I kept out of that's what you also did is now because you remember vaguely this huge complex of the the 354 00:34:52,620 --> 00:34:58,860 the loop diagrams that was too difficult to to capture to understand we reduce it to the essential personnel. 355 00:34:59,310 --> 00:35:02,750 And so that's another problem we have, which is really is quite often this. 356 00:35:02,910 --> 00:35:06,120 How do you focus on your complex chain of causality? 357 00:35:06,120 --> 00:35:11,040 Where do you can you capture everything? Then you have to sealed diagrams and of course not. 358 00:35:11,730 --> 00:35:15,570 Our human brain is relatively limited in what it can can deal with. 359 00:35:15,870 --> 00:35:20,219 You need to somehow reduce it and huge. I'll explain that later. 360 00:35:20,220 --> 00:35:27,720 Also, we went into a much more kind of focussed set of hypothesis or a simple computation if you want, 361 00:35:29,790 --> 00:35:35,670 but I'll come to that in seconds and because it's about is basically if you want to focus on this huge, 362 00:35:37,050 --> 00:35:42,720 let's say the chain from people at the political level identifying a problem to the health outcomes, 363 00:35:42,720 --> 00:35:48,629 that basically depends on how patients with TB deal with their disease and how they feel about the disease, 364 00:35:48,630 --> 00:35:53,130 how they perceive it, how they linked up to their community and local situation and then to the health system. 365 00:35:54,180 --> 00:35:57,630 Do we want to identify mechanisms and causal explanations? 366 00:35:58,140 --> 00:36:06,000 At which level? And we talk about out in terms of ladder of mechanisms in released or the number of 367 00:36:06,000 --> 00:36:10,980 ways of conceptualising that you might imagine levels of mechanism at the micro-level. 368 00:36:11,490 --> 00:36:19,590 So at the bottom to see what it is about individuals, people, patients, why would they be motivated now under this policy to take their drugs? 369 00:36:20,010 --> 00:36:24,360 Why would providers actually be more I don't know what difference in their 370 00:36:25,110 --> 00:36:28,170 behaviour towards patients to make sure that people would swallow the drugs. 371 00:36:28,500 --> 00:36:31,260 Why would managers of facilities actually go with this policy? 372 00:36:31,740 --> 00:36:36,450 Why would a supervisor from the TB crew program do that or why with politicians continue to push for funding? 373 00:36:36,780 --> 00:36:44,610 You can imagine in fact mechanisms happening of the number of different places and of course, they all are linked somehow to one to the next to. 374 00:36:46,350 --> 00:36:51,590 There's another example of one of my students now where we look at how, again, that was a feminist example, 375 00:36:51,750 --> 00:36:56,700 a continuation of that, that a particular policy at national level, it could translate into a national program. 376 00:36:57,150 --> 00:37:01,350 It comes down to hospitals. Managers need to adopt it or not. 377 00:37:01,530 --> 00:37:08,520 They will actually have a number of options to to go with the program or not to capture it actually for their own benefits or perhaps to improve it. 378 00:37:08,970 --> 00:37:12,300 So these are kind of street level bureaucracy notions here. 379 00:37:12,780 --> 00:37:18,240 But then it's not because managers decide to go with the program that actually the providers will do so. 380 00:37:18,480 --> 00:37:21,210 You can imagine, in fact, that each level of the system, 381 00:37:21,570 --> 00:37:27,330 a number of hoops and barriers or levers actually need to be kind of taken for the policy to work. 382 00:37:28,110 --> 00:37:33,780 And you cannot just focus probably on just the providers or the facility manager 383 00:37:33,780 --> 00:37:38,939 or a successful policy will need to take into account the whole system, 384 00:37:38,940 --> 00:37:42,840 of course. But okay, that's a discussion you can have at some point. 385 00:37:43,140 --> 00:37:50,850 So where do we want to focus your your efforts? So our way of focusing on things was to say, let's focus back on your initial question. 386 00:37:50,850 --> 00:37:58,709 That is, why are patients not adhering to the policies or how does the policy increase adherence if there is an outcome of better or worse adherence, 387 00:37:58,710 --> 00:38:06,960 how do we explain that by focusing at providers and different kinds of providers, different kinds of private institutions and different logics. 388 00:38:06,960 --> 00:38:12,720 You may have do the facility managers and then also the people at the service level adopted and implemented. 389 00:38:13,020 --> 00:38:16,259 In what kind of context do they do that? And what about patients? 390 00:38:16,260 --> 00:38:19,500 Of course, because they are the people supposed to take the drugs and supposed to get better. 391 00:38:19,620 --> 00:38:24,170 And so that's how we try to kind of to kind of focus the things. 392 00:38:24,180 --> 00:38:31,860 No, there's a slide missing here that what happened to these groups after this is not one of the major problems we have, 393 00:38:31,860 --> 00:38:36,150 and that's not necessarily due to release, is that it's kind of a natural experiment. 394 00:38:36,160 --> 00:38:39,840 This is a policy where you're kind of prospectively involved in the whole thing. 395 00:38:39,840 --> 00:38:43,650 You never know how it's going to end and not even if it's going to continue in the first place. 396 00:38:44,250 --> 00:38:51,360 So one of the major delays we had was changes in the political system due to elections program managers changing, politicians changing. 397 00:38:51,870 --> 00:38:56,909 Initially in the first wave of change, nothing much happened in terms of the policy preference. 398 00:38:56,910 --> 00:39:02,730 They still went for PBF, but the second government came in and they said, well, let's let's drop that idea. 399 00:39:02,970 --> 00:39:07,710 Let's go for primary health care and free care and a kind of a different way of looking at things. 400 00:39:08,280 --> 00:39:13,620 And then the question was very much from a research perspective, okay, I would make my position was let's go with the flow. 401 00:39:13,980 --> 00:39:15,000 So if the policy changes, 402 00:39:15,000 --> 00:39:21,420 let's just adopt the policy question we have and see how that change actually happens and if that new policy will be implemented or not. 403 00:39:22,050 --> 00:39:27,959 But the kind of hardcore economists and the other said, no, no, no, no, we have this program, we have our design, our child is ready. 404 00:39:27,960 --> 00:39:31,920 Let's just do it. And then you have a very strange situation whereby the cross, 405 00:39:31,920 --> 00:39:38,100 your team tries to convince the Global Fund to still get some money there, even if government is not fully in line with. 406 00:39:38,370 --> 00:39:43,110 And it's a quite strange situation. But okay, that explains a bit. 407 00:39:43,320 --> 00:39:51,450 Also again, the delays it's now after, let's say one year and three months of a kind of go between phase. 408 00:39:51,960 --> 00:39:53,910 Don't you begin to really testing the tools now, 409 00:39:53,910 --> 00:39:59,610 starting to data collection piloting and probably implementing the pilot phase in a few months from now. 410 00:40:00,300 --> 00:40:05,850 And it may be that by that stage, again, the political favour will be proposed, but perhaps it will not. 411 00:40:06,600 --> 00:40:10,139 So it's kind of ask, I mean, and that's with any kind of policy research, of course, 412 00:40:10,140 --> 00:40:17,459 to what extent do you need to steer your research in function of politicians and national priorities, 413 00:40:17,460 --> 00:40:21,200 or do you want to stick to your initial fixed design and the finish commitment that whatever? 414 00:40:21,240 --> 00:40:24,270 I yeah, that's a bit dubious. 415 00:40:25,650 --> 00:40:35,219 Okay. So, you know, this was one of the clinics in about to me kind of swamping the office with five, 416 00:40:35,220 --> 00:40:38,790 six, seven people actually asking questions to two nurses. 417 00:40:39,060 --> 00:40:44,550 This was not the way we usually do research. Of course, this was during one of the kind of introduction visits. 418 00:40:46,190 --> 00:40:55,760 Other questions. No, I thank you for forcing here. 419 00:40:55,940 --> 00:40:56,510 Absolutely.