1
00:00:02,520 --> 00:00:06,089
Okay. So it's basically this kind of.

2
00:00:06,090 --> 00:00:10,950
Lefebvre, I got a vial here, but I can't read.

3
00:00:13,060 --> 00:00:23,820
It's. It's it's in English. I can't read this. So I've known Carol 25 years and better, I think the first time in 1993.

4
00:00:23,940 --> 00:00:26,999
I think maybe 94. Long time.

5
00:00:27,000 --> 00:00:31,950
No, but Carol's a founding member of the Kaufman Corporation, as it was then.

6
00:00:32,490 --> 00:00:38,040
And as far as I know, she was the only information officer at the original.

7
00:00:38,580 --> 00:00:46,950
Hmm. Probably. Yep. The meeting and all that started really kind of very precise.

8
00:00:46,950 --> 00:00:50,220
She's incredibly organised and she keeps everybody on their toes.

9
00:00:50,820 --> 00:00:55,530
And this is, I think, a quintessential quality for somebody who does a job.

10
00:00:55,980 --> 00:00:58,110
She Lots of information officer.

11
00:00:58,470 --> 00:01:06,750
Senior Financial Specialist Cochrane said that for 20 years she now does the consultancies and supports people like us.

12
00:01:06,750 --> 00:01:10,560
Strikes me as the best we can with systematic reviews.

13
00:01:11,430 --> 00:01:16,100
Carol and I recently collaborated in that in that Cochran project.

14
00:01:16,120 --> 00:01:17,909
I showed you the digits afternoon.

15
00:01:17,910 --> 00:01:29,430
I showed you the results of the survey showing little very little uptake of regulatory data in everyday Cochran work.

16
00:01:30,330 --> 00:01:36,710
Do forget that you were going to ask Carol about the update of the handbook, huh?

17
00:01:37,290 --> 00:01:42,840
So what a delight. I thought I thought this was an opportunity to get away from the handbook for half an hour, but there you go.

18
00:01:43,500 --> 00:01:47,940
Who was I? Wrong. So thank you very much, Tom.

19
00:01:48,690 --> 00:01:54,690
Thank you for the introduction. Thank you to you and Carmel for inviting me to this evening's event.

20
00:01:55,050 --> 00:01:59,520
And thank you, Natalie, for all your support in the administration.

21
00:02:00,450 --> 00:02:07,590
You'll have seen that today's lecture is entitled The Shifting Evidence Paradigm From Literature to Data.

22
00:02:07,950 --> 00:02:13,829
Can I just check that those of you at the back of the room can hear me? Well, it's never usually a problem, let's put it that way.

23
00:02:13,830 --> 00:02:21,270
Okay, that's good. Okay, well, let me just say a few words about the trials and tribulations of pre-dinner speaking.

24
00:02:22,020 --> 00:02:28,379
We'll get that out of the way. I know that I am. What's brought you out of the bar or away from your drinks outside?

25
00:02:28,380 --> 00:02:33,630
Even on an evening such as this? And I know that I'm all that rest between you and dinner.

26
00:02:33,690 --> 00:02:38,700
I am aware of both of those things, and it's not an auspicious start, it has to say.

27
00:02:39,750 --> 00:02:43,980
But the one thing that Tom didn't say about me is that I always finish on time.

28
00:02:44,370 --> 00:02:51,030
So I have been allocated a certain amount of time for this and we have been allocated time for questions and discussions afterwards.

29
00:02:51,840 --> 00:02:56,250
So please be reassured that you will get to your dinner in time.

30
00:02:57,420 --> 00:03:03,480
I'm going to start with some declarations of interest. So for those of you who do know me or have come across my name or whatever,

31
00:03:03,720 --> 00:03:08,370
you will undoubtedly associate me with Cochrane for the reasons that Tom has suggested.

32
00:03:09,150 --> 00:03:12,420
I have been a founder member of Cochrane since 1992.

33
00:03:12,960 --> 00:03:20,340
When Cochrane started, there were about eight or nine of us and there are now something like 50 to 100000 people involved in Cochrane.

34
00:03:21,210 --> 00:03:27,930
But there was the, the director of the Cochrane Centre three in Chalmers who was an obstetrician,

35
00:03:28,230 --> 00:03:31,620
there was a clinical pharmacologist, Andrew Horkheimer.

36
00:03:31,620 --> 00:03:36,600
You may have come across his name too, and I was the only other professionally qualified member of staff.

37
00:03:36,930 --> 00:03:44,700
So that was how Cochrane was in those days in 92 and what I'm going to be talking about today, I'm going to be talking a bit about how we've moved on.

38
00:03:44,700 --> 00:03:47,280
I'm going to be talking about this shifting paradigm.

39
00:03:47,910 --> 00:03:52,620
I'm also the co-convenor of the Cochrane Information Retrieval Methods Group that advises Cochrane

40
00:03:52,620 --> 00:03:57,599
what to do with regard to searching for studies and a member of the Cochrane Methods Executive,

41
00:03:57,600 --> 00:04:04,319
which advises more widely over methods. I'm the lead author on the searching chapter of the Cochrane Handbook,

42
00:04:04,320 --> 00:04:11,310
which has meanwhile morphed to searching and selecting studies within within the chapter that I am responsible for.

43
00:04:11,640 --> 00:04:19,020
And I'll say a little about that later. And as Thomas mentioned, I'm a co participant in the Cochrane funded Clinical Study Reports project,

44
00:04:19,020 --> 00:04:23,429
of which he was the PI and many, many other hats, I have no doubt.

45
00:04:23,430 --> 00:04:28,950
But I think the key thing is the key elements of this slide as well are what is written on the bottom.

46
00:04:29,160 --> 00:04:32,580
I'm no longer employed by Cochrane and haven't been since 2012.

47
00:04:32,580 --> 00:04:39,360
I am not employed by our publisher either. Wylie So what you hear today is an independent view and an independent voice.

48
00:04:40,860 --> 00:04:43,950
So as you be aware from your programs,

49
00:04:43,950 --> 00:04:52,859
this evening's presentation will address the shift in focus over the last 20 years away from purely literature searching that is only

50
00:04:52,860 --> 00:04:59,820
searching databases such as Medline or PubMed for the published literature such as journal articles and books for identifying stuff.

51
00:05:00,030 --> 00:05:01,440
For evidence synthesis.

52
00:05:01,950 --> 00:05:09,990
She will consider the ever increasing role of unpublished data sources such as trials, registers and regulatory agents through sources.

53
00:05:10,230 --> 00:05:14,520
So that was my brief. So I'm going to start with some definitions, Tom said.

54
00:05:14,520 --> 00:05:18,870
I was very precise, and yes, I do like to be precise.

55
00:05:19,200 --> 00:05:24,420
So starting with some definitions of literature, great literature and data.

56
00:05:24,900 --> 00:05:33,660
So let's start with literature. Literature is defined in the Oxford English Dictionary definition as written works,

57
00:05:33,660 --> 00:05:40,800
especially those considered as superior or lasting artistic merit, books and writings published on a particular subject.

58
00:05:41,160 --> 00:05:47,880
So the bottom line is that literature is published sources, for example, journal articles and books.

59
00:05:49,050 --> 00:05:54,900
But within literature we have different types of literature, one of which is grey literature.

60
00:05:55,650 --> 00:05:59,700
And grey literature therefore is a subset of literature.

61
00:06:00,210 --> 00:06:04,740
And generally when people talk about great literature, they're talking about dissertations,

62
00:06:04,740 --> 00:06:09,090
theses reports and so on, and I won't go through the slides in detail.

63
00:06:09,090 --> 00:06:16,860
I think that Nathalie intends to make the slides available to the participants after the event, so you don't need to take lots of photographs.

64
00:06:16,860 --> 00:06:24,120
You will get the originals. But grey literature is defined as that which is produced on all levels of government,

65
00:06:24,120 --> 00:06:29,670
academics, business, etc., etc., but which is not controlled by commercial publishers.

66
00:06:30,270 --> 00:06:37,200
And then they went on a few years later to amend the definition to say not controlled by commercial publishers,

67
00:06:37,350 --> 00:06:41,640
where publishing is not the main primary activity of the producing body.

68
00:06:42,120 --> 00:06:48,479
So great literature is published, but it's not published by somebody whose main aim is publishing.

69
00:06:48,480 --> 00:06:56,460
So for example, a thesis is published by Oxford University, but the University of Oxford doesn't exist to publish theses or to publish anything else.

70
00:06:57,030 --> 00:07:00,990
So great literature is published. It just simply happens to be grey.

71
00:07:01,530 --> 00:07:06,509
But what about data? What about this shift that there is from literature searching?

72
00:07:06,510 --> 00:07:14,040
The phrase that we all used to use so much, and I'll mention it now, in case I forget, this is a deviation into the handbook.

73
00:07:14,460 --> 00:07:19,560
The Searching for Studies chapter in the handbook runs to about 100 pages and it's

74
00:07:19,560 --> 00:07:23,879
entitled Originally Searching for study is now searching for in Selecting Studies,

75
00:07:23,880 --> 00:07:27,280
but the main focus is searching for studies in that site.

76
00:07:27,330 --> 00:07:31,080
In the whole chapter on searching for studies, running to 100 pages.

77
00:07:31,320 --> 00:07:34,650
How many times do you think we use the word literature searching?

78
00:07:36,060 --> 00:07:40,880
How many people think it might be over 50? Okay.

79
00:07:40,940 --> 00:07:44,180
How many people think it might be? Between 20 and 50.

80
00:07:44,900 --> 00:07:48,290
The phrase literature searching. How many people think it might be?

81
00:07:48,290 --> 00:07:54,500
Between five and 20. How many people think it might be between nought and five?

82
00:07:54,890 --> 00:08:03,830
Yeah. Well done, you guys. It says it's a phrase that I am trying to stamp out because literature searching is such a small part of what we do.

83
00:08:04,100 --> 00:08:07,549
And I'll be going on to say that literature searching is important.

84
00:08:07,550 --> 00:08:11,370
It's still important to search databases such as Medline and Mace and so on.

85
00:08:11,660 --> 00:08:18,080
But literature searching is not the whole story. And we shouldn't just glibly use this phrase literature searching,

86
00:08:18,260 --> 00:08:24,050
when what we're really talking about is identifying studies for systematic reviews and other evidence synthesis.

87
00:08:24,620 --> 00:08:28,160
So that brings us back nicely to data.

88
00:08:28,220 --> 00:08:37,490
Data are data. And the Oxford English Dictionary definition for that is facts and statistics collected together for reference or analysis.

89
00:08:38,540 --> 00:08:42,300
So data are published are available in the published literature.

90
00:08:42,320 --> 00:08:46,610
Obviously, if you do a Medline search and you find a journal article, it may well have data.

91
00:08:47,150 --> 00:08:52,610
Data may be published in someone's thesis or dissertation, in which case it would be data from the grey literature.

92
00:08:52,790 --> 00:08:56,150
But data are also available in unpublished sources,

93
00:08:56,510 --> 00:09:03,709
and the unpublished sources that are probably most important for identifying studies for systematic reviews would be trials,

94
00:09:03,710 --> 00:09:05,660
registers, clinical study reports,

95
00:09:05,660 --> 00:09:12,980
which Tom has started going through with you on the program today and will continue tomorrow and regulatory agency documents.

96
00:09:14,330 --> 00:09:22,850
So looking at this in a Venn diagram format, you have literature and within literature you have published you have published literature.

97
00:09:23,120 --> 00:09:32,120
And within published literature you have great literature. And as we know, there's a vast amount of literature that has no data in it whatsoever.

98
00:09:32,210 --> 00:09:36,890
It doesn't that doesn't have to be any data in literature, but some in the central section there,

99
00:09:37,310 --> 00:09:44,150
some literature will contain data, but the vast amounts of data outside of all of this literature.

100
00:09:44,570 --> 00:09:54,379
So the areas that I was just talking about, the areas of trials registers, clinical study reports and regulatory documents, they are not published.

101
00:09:54,380 --> 00:10:03,620
And it's that that we need to focus more heavily on. But that's not to say that the published literature is not important.

102
00:10:03,620 --> 00:10:11,779
And you have seen from the program that your course is addressing this week that unless the last most recent version of the program is changed,

103
00:10:11,780 --> 00:10:13,850
but the program that I received today,

104
00:10:14,060 --> 00:10:22,580
you were hearing from Tom and Camel reviews of unpublished late data searching for unpublished data and regulatory data.

105
00:10:23,090 --> 00:10:28,550
And then tomorrow, you'll be hearing from Lawyer Roberts about searching for published data.

106
00:10:28,880 --> 00:10:37,430
So this program that you're attending this week is recognising both the published and the unpublished data and,

107
00:10:37,430 --> 00:10:41,690
of course, qualitative evidence syntheses, which I think you were addressing on Thursday.

108
00:10:41,930 --> 00:10:50,419
They may extend beyond what we generally call data into other areas of information which are relevant for qualitative synthesis,

109
00:10:50,420 --> 00:10:53,840
or rather since that synthesis of qualitative data. MORE Exactly.

110
00:10:54,950 --> 00:10:59,600
So what about the shifting evidence paradigm from literature to data?

111
00:10:59,960 --> 00:11:04,910
There has been a shift in focus over the last 20 years from purely literature searching,

112
00:11:05,120 --> 00:11:10,340
that is, searching only databases such as Medline, PubMed and base and so on.

113
00:11:10,580 --> 00:11:17,000
And there has been a considerable shift towards unpublished sources of data in line with their development and access.

114
00:11:17,960 --> 00:11:24,110
And I've started this in 1992 because this is when I started in this area, when I first joined Cochrane.

115
00:11:24,350 --> 00:11:29,540
And in those days there were practically no trials registers, no general trials registers,

116
00:11:29,540 --> 00:11:34,430
no company specific trials registers that were available outside of their own companies.

117
00:11:34,700 --> 00:11:37,700
But there were a few registers within cancer.

118
00:11:38,330 --> 00:11:42,830
And my colleague Ian Chalmers, the director of the UK Cochrane Centre,

119
00:11:43,070 --> 00:11:49,100
worked very closely with the drug company Sharing and eventually in 1997 Sharing

120
00:11:49,100 --> 00:11:54,860
declared that they would provide detailed outline details of 30 of their trials.

121
00:11:55,160 --> 00:12:00,979
And you might think, wow, big news. But I can tell you in 1997 that was enormously big news.

122
00:12:00,980 --> 00:12:04,430
And we wrote an article we couldn't avoid the pun of sharing a sharing too,

123
00:12:05,240 --> 00:12:11,600
but we wrote an article in Cochrane News to announce this what we felt was a monumental event at that time.

124
00:12:13,910 --> 00:12:19,280
GlaxoSmithKline followed suit in 1998 and also in 1998.

125
00:12:19,280 --> 00:12:26,720
The Current Control Trials website was launched, The Better Register was launched, which unfortunately now is under review.

126
00:12:27,050 --> 00:12:32,660
But the first standard numbering system also came in but wasn't through clinicaltrials.gov.

127
00:12:32,660 --> 00:12:36,380
That was through current control trials. It was the esr ctn.

128
00:12:36,620 --> 00:12:43,760
So we eventually had a means of numbering trials. So that we could tell which trial was which in multiple publications.

129
00:12:44,450 --> 00:12:48,200
And it wasn't till 2000 that clinical trials takeoff was launched.

130
00:12:48,470 --> 00:12:54,770
So for many of you, you have always had clinical trials available to you within your professional careers.

131
00:12:55,460 --> 00:12:59,650
You probably don't even remember back to a time prior to clinical trials go.

132
00:12:59,690 --> 00:13:06,049
But believe me, there was a time prior to clinical trials dot gov and it was four years later that the International Committee of Medical

133
00:13:06,050 --> 00:13:12,770
Journal Editors announced their refusal to publish subsequently any reports of trials not prospectively registered.

134
00:13:13,820 --> 00:13:17,270
So that was a major contribution by the Medical Journal editors.

135
00:13:17,780 --> 00:13:20,780
And a few years later, the I CTP,

136
00:13:20,810 --> 00:13:27,110
the International Clinical Trials Registry Platform Search Portal was launched by the

137
00:13:27,110 --> 00:13:31,910
World Health Organisation and that currently brings together 17 separate registries.

138
00:13:33,650 --> 00:13:46,100
In 2008, the federal law came in, it came in with effect from 2008, enacted in 2007, and that included a section on clinical trials databases.

139
00:13:46,100 --> 00:13:55,190
And it expanded the types of clinical trials that increase the number of data elements and also required submission of certain results data.

140
00:13:55,370 --> 00:14:02,360
And it was that that was really important because you know that when you're doing a systematic review, what you want the results, you want the data.

141
00:14:02,600 --> 00:14:06,350
You don't want information about a trial. It's still ongoing necessarily.

142
00:14:06,740 --> 00:14:11,540
It has other uses. I won't say it's not useful, but it doesn't provide data for your matter.

143
00:14:11,570 --> 00:14:20,690
Analysis. In 2011, Cochrane started looking at standards and we declared that searching trials registers.

144
00:14:20,690 --> 00:14:31,010
So to find these unpublished data. We declared that to be mandatory and we said that you had to search both clinicaltrials.gov and the W.H.O. portal.

145
00:14:31,700 --> 00:14:36,590
And we gave the rationale for that, which I'll leave you to look at later when you get the slides.

146
00:14:38,180 --> 00:14:45,350
In January 2013, the All Trials campaign was launched, founded by Ben Goldacre and others,

147
00:14:45,350 --> 00:14:56,450
including the Centre for Evidence based Medicine here in Oxford. And in 2013, GSK established a clinical study data request initiative.

148
00:14:57,830 --> 00:15:04,010
And in December 2013, agreement was reached in principle within the European Parliament that there would

149
00:15:04,010 --> 00:15:10,040
be a European Medicines Agency register of all trials carried out in the EU.

150
00:15:10,430 --> 00:15:14,050
So that was again a very important step forward.

151
00:15:15,790 --> 00:15:22,640
I can. I'll leave you to read the notes of that when you get the get the slides in April 2014.

152
00:15:23,060 --> 00:15:27,620
I was thinking about not including this because obviously Tom has already spoken to you today,

153
00:15:27,620 --> 00:15:33,070
but it seemed a bit ungenerous to not included in the major, you know,

154
00:15:33,170 --> 00:15:37,520
the major steps that have happened, the major developments, the major influences.

155
00:15:37,730 --> 00:15:44,360
But I'm not going to say anything more about this because this is the study that Tom has already talked to you about today,

156
00:15:44,360 --> 00:15:50,270
the Tamiflu Relenza review based solely on clinical study reports.

157
00:15:50,690 --> 00:15:53,780
And there you have the data from that study.

158
00:15:54,530 --> 00:16:00,530
I'm not sure whether Tom gave you the URLs for these other pieces of information about that study,

159
00:16:00,560 --> 00:16:03,860
but that's something that you can follow up later on.

160
00:16:05,240 --> 00:16:14,450
And in September 2016, the Department of Health and Human Services issued the final rule for clinical trials, registration and results information.

161
00:16:14,750 --> 00:16:20,300
And that expanded the registration and results information submission requirements.

162
00:16:21,140 --> 00:16:25,730
And I'm not sure whether you've already looked at it today or whether you will be tomorrow,

163
00:16:26,570 --> 00:16:31,520
but the EMA have started to release clinical data in the form of clinical study reports.

164
00:16:31,850 --> 00:16:40,879
When I first did some slides that was similar to this for something else in May 2017, they only had 20 cases listed.

165
00:16:40,880 --> 00:16:49,490
In October 2017 they had 54, and now in June 2018, when I checked that today there were 109.

166
00:16:49,760 --> 00:16:55,610
So that's an ever increasing resource. But that's not to say that everything's working perfectly.

167
00:16:55,610 --> 00:17:01,909
There are any number of faults that one could find with with the majority, if not all of the things that I've put up.

168
00:17:01,910 --> 00:17:04,010
I'm not saying we now live in a perfect world.

169
00:17:04,280 --> 00:17:15,110
What I'm trying to convey is that the world has changed since, you know, since 1992, when I started work in this area, as Tom says, 25 years ago.

170
00:17:15,860 --> 00:17:21,169
So within Cochrane, how have we changed in this line?

171
00:17:21,170 --> 00:17:27,829
The Cochrane Central Register of Control Trials. Central You're probably familiar with this from when you search the Cochrane Library.

172
00:17:27,830 --> 00:17:30,410
This is the trials register within the Cochrane Library.

173
00:17:30,980 --> 00:17:42,410
So as I say, the UK, the UK CC was founded in 1992 and I was hired as the information specialist and my first task was to build central.

174
00:17:42,920 --> 00:17:46,070
So the immediate focus was on literature searching,

175
00:17:46,310 --> 00:17:54,230
bringing us back to our to our topic of literature versus data that is searching bibliographic databases, indexing the published literature.

176
00:17:54,230 --> 00:17:56,120
So indexing journal articles.

177
00:17:56,600 --> 00:18:06,559
And we were focusing first of all on identifying reports of randomised controlled trials from Medline and at that time Matthias Ecker,

178
00:18:06,560 --> 00:18:14,840
whose name will be familiar to many of you, said that this was clearly the best single source of published trials for inclusion in systematic reviews.

179
00:18:15,980 --> 00:18:23,060
Our second focus then was on identifying reports from Embase again literature searching bibliographic database basis.

180
00:18:23,390 --> 00:18:26,450
But that project has now expanded enormously.

181
00:18:26,450 --> 00:18:32,390
It's now done using crowdsourcing, so individual volunteers, a citizen science initiative,

182
00:18:33,230 --> 00:18:38,660
text mining and indeed taking text mining to the next level of training the machine.

183
00:18:38,720 --> 00:18:43,550
So machine learning to try and optimise the identification of those studies.

184
00:18:44,390 --> 00:18:53,180
But from the point of view of today's topic, Central now also includes unpublished study records from trials registries such as clinicaltrials.gov.

185
00:18:53,540 --> 00:18:57,560
So Central also is moving into this area of unpublished studies.

186
00:18:59,510 --> 00:19:05,180
So what do other organisations recommend with regard to searching for unpublished and regulatory data?

187
00:19:05,570 --> 00:19:14,060
AAC in the US recognises you'll see that their definition doesn't meet my definition, but you know, you can't persuade everybody all of the time.

188
00:19:14,780 --> 00:19:19,040
So apart from the fact that they use grey literature, I think in the wrong way.

189
00:19:19,040 --> 00:19:27,169
I agree with what they say that grey literature x strike the word including so grey literature, regulatory data,

190
00:19:27,170 --> 00:19:33,050
clinical trials, registries and conference abstracts should be searched in addition to bibliographic databases.

191
00:19:33,680 --> 00:19:37,520
The Cochrane guidance that we're working on at the moment that will be published

192
00:19:37,520 --> 00:19:42,290
in 2018 goes into a great deal more detail about regulatory agency data,

193
00:19:42,290 --> 00:19:46,550
clinical study reports, and we will be advocating that all are important.

194
00:19:46,910 --> 00:19:54,860
And the US Institute of Medicine guidance again says that clinical trials dot gov, clinical study results,

195
00:19:55,100 --> 00:20:00,400
current controlled trials and the W.H.O. portal together with the FDA Medical Institute,

196
00:20:00,780 --> 00:20:07,370
Statistical Review Records all have a role in standards for systematic reviews.

197
00:20:08,480 --> 00:20:12,770
So just wanted to look at some common misconceptions about unpublished data.

198
00:20:13,460 --> 00:20:20,340
And let's start with clinical trials. What? Of you can't believe how many times I've heard people say that it's a US database.

199
00:20:20,940 --> 00:20:24,179
It's only got us trials in it. That is just so wrong.

200
00:20:24,180 --> 00:20:28,709
And I'll move on to that in a moment. People also say that it's an ongoing trial.

201
00:20:28,710 --> 00:20:32,950
So I just started again when I finished obliterating the phrase literature search from the universe.

202
00:20:32,970 --> 00:20:36,440
I'm going to be moving on to ongoing trials. Okay.

203
00:20:36,450 --> 00:20:40,390
Ongoing trials exist, but let's not get too hung up about ongoing trials.

204
00:20:40,410 --> 00:20:47,700
Every ongoing trial becomes well, it becomes either either a completed or a stop too early or whatever, but it becomes something else.

205
00:20:47,700 --> 00:20:51,389
Ongoing is an interim word. And as I say,

206
00:20:51,390 --> 00:20:58,950
if you're doing a systematic review you want to know about in ongoing trials in the sense that you're not going to publish your systematic review,

207
00:20:58,950 --> 00:21:04,170
if that's an ongoing trial, that's going to finish in a month's time. When you're going to get the data in three months, you will hold back.

208
00:21:04,440 --> 00:21:09,240
But the whole point about an ongoing trial is that when it's finished, you get the data and you include them in your review.

209
00:21:09,450 --> 00:21:14,519
So it's not ongoing trials that are so fascinating. It's trials that are so fascinating.

210
00:21:14,520 --> 00:21:16,320
And it's the data from those trials,

211
00:21:16,470 --> 00:21:24,930
irrespective of whether they're published in grey or normal published literature or whether they make their data available in other places.

212
00:21:25,350 --> 00:21:36,150
So clinical trials, sculptors not only have data about and have information about ongoing trials, and also people say it doesn't have any results.

213
00:21:36,720 --> 00:21:40,740
Well, it didn't on day one, but it certainly does now.

214
00:21:40,920 --> 00:21:50,940
So taking the first of those, this was taken off the clinicaltrials.gov website earlier today, but they only updated it five days ago.

215
00:21:51,810 --> 00:21:55,920
But you'll see that's the percentage of registered studies by location.

216
00:21:56,220 --> 00:22:02,190
As of June, the 20th was 35% only for the US.

217
00:22:03,060 --> 00:22:09,900
And even if you put both U.S. and non-U.S. in there, you'll still you're still only at 40%.

218
00:22:09,900 --> 00:22:13,800
So you've got 48% of those are non US studies.

219
00:22:14,100 --> 00:22:18,480
So it is absolutely not a register of US studies only.

220
00:22:20,070 --> 00:22:23,040
And the fact that it only has ongoing trials.

221
00:22:23,040 --> 00:22:29,460
No, if you again if you look at the you have the URL there so you can check all my facts should you wish to do so.

222
00:22:30,150 --> 00:22:35,820
You'll see that they have all the different statuses of a study, right from not yet recruiting through recruiting,

223
00:22:35,820 --> 00:22:43,140
enrolling by invitation, etc., etc., right the way through to suspend a terminated completed withdrawal.

224
00:22:43,500 --> 00:22:47,190
So you've got absolutely the whole lifecycle of a study in there.

225
00:22:49,230 --> 00:22:57,450
And what I've included on your slides here is how to find studies with results posted to clinicaltrials.gov.

226
00:22:58,020 --> 00:23:05,940
And today, when I last updated these data, they were 31 and a half thousand studies with results posted.

227
00:23:06,120 --> 00:23:09,630
These are not results from the JAMA or BMJ or whatever.

228
00:23:09,840 --> 00:23:17,700
These are results where the principal investigator or somebody on their behalf has posted simply the raw results.

229
00:23:18,180 --> 00:23:28,170
So it's a really, really important resource. Common misconceptions about the ICC IP, the W.H.O. resource.

230
00:23:29,010 --> 00:23:39,150
You will find all clinicaltrials.gov records by searching the isotope because the isotope is an overall umbrella to all of these 17 registers.

231
00:23:39,420 --> 00:23:46,590
No, that is absolutely not true. The presence of a record within a resource does not equate to its retrieve ability,

232
00:23:47,670 --> 00:23:53,460
and there are a number of reasons for that, though there are fewer data elements in the isotope version,

233
00:23:53,940 --> 00:24:02,460
there are differences in the search functionality, and there is a time lag in the posting of clinicaltrials.gov records in the isotope.

234
00:24:02,880 --> 00:24:06,270
For ICI. For ClinicalTrials.gov, it's only a week.

235
00:24:06,360 --> 00:24:10,800
I think there are probably about five or six of them that are a week. Most of them are four weeks.

236
00:24:11,250 --> 00:24:17,040
So if you search the primary resource, you're much, much further ahead than you would be.

237
00:24:17,520 --> 00:24:21,540
And it's beyond the scope. And I haven't got time to go into it today.

238
00:24:21,540 --> 00:24:29,820
But my colleague Julie Glanville and others published a very important piece in the Journal of the Medical Library Association in 2014,

239
00:24:30,450 --> 00:24:36,240
giving the evidence base as to why you absolutely must search both registers and not rely on just one.

240
00:24:37,950 --> 00:24:48,510
So what have we done in Cochrane about improving the quality in Cochrane reviews with regard to the types of study that you should be looking for,

241
00:24:48,990 --> 00:24:51,030
published and unpublished studies,

242
00:24:51,280 --> 00:24:59,550
that's just a little bit of background about the project called Messier Methodological Expectations in Cochrane Intervention reviews.

243
00:24:59,910 --> 00:25:08,850
The vast majority of the of the the standards were produced in 2011 2012 and then updated in 2016.

244
00:25:09,090 --> 00:25:15,150
And it allows for us to say that such and such is mandatory, which means that if you don't do this in your Cochrane review.

245
00:25:15,180 --> 00:25:23,760
It won't get published. So I've just pulled out a few of the mandatory standards that relate to searching and that relate to sources.

246
00:25:24,090 --> 00:25:30,209
So you'll see we haven't gone completely overboard and said, you no longer need to do literature searching literature,

247
00:25:30,210 --> 00:25:34,710
searching of databases such as Medline and Base and Central.

248
00:25:34,830 --> 00:25:40,110
That's still mandatory for Cochrane reviews of the effectiveness of interventions.

249
00:25:42,180 --> 00:25:47,850
It's also mandatory to address adverse effects, economics and qualitative research should they be relevant?

250
00:25:48,210 --> 00:25:53,010
It's mandatory to search both clinicaltrials.gov and the W.H.O. portal.

251
00:25:53,370 --> 00:25:59,040
It's mandatory to search reference lists of included studies and any relevant systematic reviews.

252
00:25:59,880 --> 00:26:06,840
So you'll see that we still have a broad range of both published and unpublished sources that we consider to be important.

253
00:26:08,250 --> 00:26:13,980
So what are the facilitators for this shifting evidence paradigm from literature to data?

254
00:26:14,400 --> 00:26:19,530
I think it's largely the developments in access to unpublished data, which I've listed above.

255
00:26:20,280 --> 00:26:29,219
I think one very particular facilitator has been the ICI, MJ Journal Editors Declaration, even though it's not always adhered to.

256
00:26:29,220 --> 00:26:34,110
And if you do, if you do a literature search in Medline and or Embase,

257
00:26:34,290 --> 00:26:41,400
you will find records that of studies that talk about where this has not been adhered to.

258
00:26:41,910 --> 00:26:49,230
But I think there is increasing awareness of unpublished sources and there is increasing skills in searching these sources.

259
00:26:50,820 --> 00:26:54,360
But what about the barriers to this shifting evidence paradigm?

260
00:26:55,620 --> 00:27:00,719
Unpublished sources are not peer reviewed. The exclamation mark is for Tom.

261
00:27:00,720 --> 00:27:04,799
I put this in specially for him a number of times.

262
00:27:04,800 --> 00:27:10,920
I've heard people say, Oh, we don't include unpublished sources in our in our systematic reviews because

263
00:27:10,920 --> 00:27:14,340
it hasn't been published in a journal and therefore is not peer reviewed.

264
00:27:16,410 --> 00:27:21,420
Well, you know, I mean, there are advantages to peer review if it's done well.

265
00:27:21,450 --> 00:27:27,750
But peer review usually involves two or three people who have their own axe to grind and their own areas of ignorance.

266
00:27:28,560 --> 00:27:33,690
And on a good day, you might get some helpful comments and they might improve the work.

267
00:27:33,930 --> 00:27:41,750
But peer review is not the answer to everything. And it certainly, in my view, is not a reason to discount using unpublished sources.

268
00:27:41,760 --> 00:27:49,200
But you hear it all the time. People say, I'm not going to use any of these unpublished sources because they have not been subjected to peer review.

269
00:27:49,950 --> 00:27:55,079
They say that trials registries have no usable content for systematic reviews, for example, no results.

270
00:27:55,080 --> 00:27:59,010
And I've just shown you I think it was 31 and a half thousand from just one resource.

271
00:27:59,460 --> 00:28:05,670
Researchers have no access to clinical study reports. Well, if Tom hasn't made that clear today, he will have done by the end of tomorrow.

272
00:28:06,330 --> 00:28:11,340
I don't know whether he's going to get you all to register on the EMR, but I will.

273
00:28:12,090 --> 00:28:20,490
There you go then. Okay. Again, I'm not saying that's a perfect world, but, you know, it's better than it was five or ten years ago.

274
00:28:20,790 --> 00:28:26,160
And then people say, oh, well, regulatory data, they're all secret. You know, you can't get anything out of the AMA and the FDA.

275
00:28:26,340 --> 00:28:30,010
Well, of course you can. And that's improving all the time.

276
00:28:30,900 --> 00:28:39,780
So in summary, I wanted to say that all relevant data are important in evidence syntheses irrespective of their source.

277
00:28:40,230 --> 00:28:44,850
And these data may be derived from published sources such as journal, articles, books,

278
00:28:44,850 --> 00:28:52,650
etc. either published literature or they may be derived from dissertations, theses and reports, great literature.

279
00:28:53,160 --> 00:29:01,470
They may be derived from unpublished sources, for example, trials, registers, clinical study reports and regulatory agency documents.

280
00:29:01,800 --> 00:29:07,500
And all data sources, in my view, have a role to play in evidence synthesis.

281
00:29:08,100 --> 00:29:15,810
Some data sources may be more prone to bias than others, and all eligible studies and their data must be assessed for bias.

282
00:29:16,770 --> 00:29:22,860
So returning to the title of today's talk, the shifting evidence paradigm from literature to data.

283
00:29:23,370 --> 00:29:27,660
Is there a shifting evidence paradigm from literature to data?

284
00:29:28,290 --> 00:29:33,060
Well, I think there is, but not yet a paradigm shift.

285
00:29:33,870 --> 00:29:39,210
As Tom says, I'm very, very precise. But the thing is, you can be part of this shift.

286
00:29:39,960 --> 00:29:45,780
You can when you're doing your systematic reviews back in the workplace, which I assume many of you will be wanting to do,

287
00:29:45,780 --> 00:29:53,370
which is why you're on this course and when you're going to your librarian and you're asking for support in not literature searching,

288
00:29:53,370 --> 00:29:57,540
I hope, but identifying studies, you can be more demanding.

289
00:29:57,540 --> 00:30:04,620
You can say to your librarian, I'd like some help in identifying studies for this systematic review.

290
00:30:04,890 --> 00:30:09,420
And if they say, Oh, yes, I'll do you a Medline search, you can say, okay, that would be very nice.

291
00:30:09,420 --> 00:30:14,970
Thank you. And what else do you propose? And then if they say, Oh, well, no, I'm sure that'll be enough, then you can say, Well, actually.

292
00:30:15,050 --> 00:30:18,920
I'm also interested in unpublished studies. I'm also interested in unpublished data.

293
00:30:19,190 --> 00:30:24,770
And can you give me some hints and tips on this and this? So I think you have a role to play in this to.