Okay. So it's basically this kind of.
Lefebvre, I got a vial here, but I can't read.
It's. It's it's in English. I can't read this. So I've known Carol 25 years and better, I think the first time in 1993.
I think maybe 94. Long time.
No, but Carol's a founding member of the Kaufman Corporation, as it was then.
And as far as I know, she was the only information officer at the original.
Hmm. Probably. Yep. The meeting and all that started really kind of very precise.
She's incredibly organised and she keeps everybody on their toes.
And this is, I think, a quintessential quality for somebody who does a job.
She Lots of information officer.
Senior Financial Specialist Cochrane said that for 20 years she now does the consultancies and supports people like us.
Strikes me as the best we can with systematic reviews.
Carol and I recently collaborated in that in that Cochran project.
I showed you the digits afternoon.
I showed you the results of the survey showing little very little uptake of regulatory data in everyday Cochran work.
Do forget that you were going to ask Carol about the update of the handbook, huh?
So what a delight. I thought I thought this was an opportunity to get away from the handbook for half an hour, but there you go.
Who was I? Wrong. So thank you very much, Tom.
Thank you for the introduction. Thank you to you and Carmel for inviting me to this evening's event.
And thank you, Natalie, for all your support in the administration.
You'll have seen that today's lecture is entitled The Shifting Evidence Paradigm From Literature to Data.
Can I just check that those of you at the back of the room can hear me? Well, it's never usually a problem, let's put it that way.
Okay, that's good. Okay, well, let me just say a few words about the trials and tribulations of pre-dinner speaking.
We'll get that out of the way. I know that I am. What's brought you out of the bar or away from your drinks outside?
Even on an evening such as this? And I know that I'm all that rest between you and dinner.
I am aware of both of those things, and it's not an auspicious start, it has to say.
But the one thing that Tom didn't say about me is that I always finish on time.
So I have been allocated a certain amount of time for this and we have been allocated time for questions and discussions afterwards.
So please be reassured that you will get to your dinner in time.
I'm going to start with some declarations of interest. So for those of you who do know me or have come across my name or whatever,
you will undoubtedly associate me with Cochrane for the reasons that Tom has suggested.
I have been a founder member of Cochrane since 1992.
When Cochrane started, there were about eight or nine of us and there are now something like 50 to 100000 people involved in Cochrane.
But there was the, the director of the Cochrane Centre three in Chalmers who was an obstetrician,
there was a clinical pharmacologist, Andrew Horkheimer.
You may have come across his name too, and I was the only other professionally qualified member of staff.
So that was how Cochrane was in those days in 92 and what I'm going to be talking about today, I'm going to be talking a bit about how we've moved on.
I'm going to be talking about this shifting paradigm.
I'm also the co-convenor of the Cochrane Information Retrieval Methods Group that advises Cochrane
what to do with regard to searching for studies and a member of the Cochrane Methods Executive,
which advises more widely over methods. I'm the lead author on the searching chapter of the Cochrane Handbook,
which has meanwhile morphed to searching and selecting studies within within the chapter that I am responsible for.
And I'll say a little about that later. And as Thomas mentioned, I'm a co participant in the Cochrane funded Clinical Study Reports project,
of which he was the PI and many, many other hats, I have no doubt.
But I think the key thing is the key elements of this slide as well are what is written on the bottom.
I'm no longer employed by Cochrane and haven't been since 2012.
I am not employed by our publisher either. Wylie So what you hear today is an independent view and an independent voice.
So as you be aware from your programs,
this evening's presentation will address the shift in focus over the last 20 years away from purely literature searching that is only
searching databases such as Medline or PubMed for the published literature such as journal articles and books for identifying stuff.
For evidence synthesis.
She will consider the ever increasing role of unpublished data sources such as trials, registers and regulatory agents through sources.
So that was my brief. So I'm going to start with some definitions, Tom said.
I was very precise, and yes, I do like to be precise.
So starting with some definitions of literature, great literature and data.
So let's start with literature. Literature is defined in the Oxford English Dictionary definition as written works,
especially those considered as superior or lasting artistic merit, books and writings published on a particular subject.
So the bottom line is that literature is published sources, for example, journal articles and books.
But within literature we have different types of literature, one of which is grey literature.
And grey literature therefore is a subset of literature.
And generally when people talk about great literature, they're talking about dissertations,
theses reports and so on, and I won't go through the slides in detail.
I think that Nathalie intends to make the slides available to the participants after the event, so you don't need to take lots of photographs.
You will get the originals. But grey literature is defined as that which is produced on all levels of government,
academics, business, etc., etc., but which is not controlled by commercial publishers.
And then they went on a few years later to amend the definition to say not controlled by commercial publishers,
where publishing is not the main primary activity of the producing body.
So great literature is published, but it's not published by somebody whose main aim is publishing.
So for example, a thesis is published by Oxford University, but the University of Oxford doesn't exist to publish theses or to publish anything else.
So great literature is published. It just simply happens to be grey.
But what about data? What about this shift that there is from literature searching?
The phrase that we all used to use so much, and I'll mention it now, in case I forget, this is a deviation into the handbook.
The Searching for Studies chapter in the handbook runs to about 100 pages and it's
entitled Originally Searching for study is now searching for in Selecting Studies,
but the main focus is searching for studies in that site.
In the whole chapter on searching for studies, running to 100 pages.
How many times do you think we use the word literature searching?
How many people think it might be over 50? Okay.
How many people think it might be? Between 20 and 50.
The phrase literature searching. How many people think it might be?
Between five and 20. How many people think it might be between nought and five?
Yeah. Well done, you guys. It says it's a phrase that I am trying to stamp out because literature searching is such a small part of what we do.
And I'll be going on to say that literature searching is important.
It's still important to search databases such as Medline and Mace and so on.
But literature searching is not the whole story. And we shouldn't just glibly use this phrase literature searching,
when what we're really talking about is identifying studies for systematic reviews and other evidence synthesis.
So that brings us back nicely to data.
Data are data. And the Oxford English Dictionary definition for that is facts and statistics collected together for reference or analysis.
So data are published are available in the published literature.
Obviously, if you do a Medline search and you find a journal article, it may well have data.
Data may be published in someone's thesis or dissertation, in which case it would be data from the grey literature.
But data are also available in unpublished sources,
and the unpublished sources that are probably most important for identifying studies for systematic reviews would be trials,
registers, clinical study reports,
which Tom has started going through with you on the program today and will continue tomorrow and regulatory agency documents.
So looking at this in a Venn diagram format, you have literature and within literature you have published you have published literature.
And within published literature you have great literature. And as we know, there's a vast amount of literature that has no data in it whatsoever.
It doesn't that doesn't have to be any data in literature, but some in the central section there,
some literature will contain data, but the vast amounts of data outside of all of this literature.
So the areas that I was just talking about, the areas of trials registers, clinical study reports and regulatory documents, they are not published.
And it's that that we need to focus more heavily on. But that's not to say that the published literature is not important.
And you have seen from the program that your course is addressing this week that unless the last most recent version of the program is changed,
but the program that I received today,
you were hearing from Tom and Camel reviews of unpublished late data searching for unpublished data and regulatory data.
And then tomorrow, you'll be hearing from Lawyer Roberts about searching for published data.
So this program that you're attending this week is recognising both the published and the unpublished data and,
of course, qualitative evidence syntheses, which I think you were addressing on Thursday.
They may extend beyond what we generally call data into other areas of information which are relevant for qualitative synthesis,
or rather since that synthesis of qualitative data. MORE Exactly.
So what about the shifting evidence paradigm from literature to data?
There has been a shift in focus over the last 20 years from purely literature searching,
that is, searching only databases such as Medline, PubMed and base and so on.
And there has been a considerable shift towards unpublished sources of data in line with their development and access.
And I've started this in 1992 because this is when I started in this area, when I first joined Cochrane.
And in those days there were practically no trials registers, no general trials registers,
no company specific trials registers that were available outside of their own companies.
But there were a few registers within cancer.
And my colleague Ian Chalmers, the director of the UK Cochrane Centre,
worked very closely with the drug company Sharing and eventually in 1997 Sharing
declared that they would provide detailed outline details of 30 of their trials.
And you might think, wow, big news. But I can tell you in 1997 that was enormously big news.
And we wrote an article we couldn't avoid the pun of sharing a sharing too,
but we wrote an article in Cochrane News to announce this what we felt was a monumental event at that time.
GlaxoSmithKline followed suit in 1998 and also in 1998.
The Current Control Trials website was launched, The Better Register was launched, which unfortunately now is under review.
But the first standard numbering system also came in but wasn't through clinicaltrials.gov.
That was through current control trials. It was the esr ctn.
So we eventually had a means of numbering trials. So that we could tell which trial was which in multiple publications.
And it wasn't till 2000 that clinical trials takeoff was launched.
So for many of you, you have always had clinical trials available to you within your professional careers.
You probably don't even remember back to a time prior to clinical trials go.
But believe me, there was a time prior to clinical trials dot gov and it was four years later that the International Committee of Medical
Journal Editors announced their refusal to publish subsequently any reports of trials not prospectively registered.
So that was a major contribution by the Medical Journal editors.
And a few years later, the I CTP,
the International Clinical Trials Registry Platform Search Portal was launched by the
World Health Organisation and that currently brings together 17 separate registries.
In 2008, the federal law came in, it came in with effect from 2008, enacted in 2007, and that included a section on clinical trials databases.
And it expanded the types of clinical trials that increase the number of data elements and also required submission of certain results data.
And it was that that was really important because you know that when you're doing a systematic review, what you want the results, you want the data.
You don't want information about a trial. It's still ongoing necessarily.
It has other uses. I won't say it's not useful, but it doesn't provide data for your matter.
Analysis. In 2011, Cochrane started looking at standards and we declared that searching trials registers.
So to find these unpublished data. We declared that to be mandatory and we said that you had to search both clinicaltrials.gov and the W.H.O. portal.
And we gave the rationale for that, which I'll leave you to look at later when you get the slides.
In January 2013, the All Trials campaign was launched, founded by Ben Goldacre and others,
including the Centre for Evidence based Medicine here in Oxford. And in 2013, GSK established a clinical study data request initiative.
And in December 2013, agreement was reached in principle within the European Parliament that there would
be a European Medicines Agency register of all trials carried out in the EU.
So that was again a very important step forward.
I can. I'll leave you to read the notes of that when you get the get the slides in April 2014.
I was thinking about not including this because obviously Tom has already spoken to you today,
but it seemed a bit ungenerous to not included in the major, you know,
the major steps that have happened, the major developments, the major influences.
But I'm not going to say anything more about this because this is the study that Tom has already talked to you about today,
the Tamiflu Relenza review based solely on clinical study reports.
And there you have the data from that study.
I'm not sure whether Tom gave you the URLs for these other pieces of information about that study,
but that's something that you can follow up later on.
And in September 2016, the Department of Health and Human Services issued the final rule for clinical trials, registration and results information.
And that expanded the registration and results information submission requirements.
And I'm not sure whether you've already looked at it today or whether you will be tomorrow,
but the EMA have started to release clinical data in the form of clinical study reports.
When I first did some slides that was similar to this for something else in May 2017, they only had 20 cases listed.
In October 2017 they had 54, and now in June 2018, when I checked that today there were 109.
So that's an ever increasing resource. But that's not to say that everything's working perfectly.
There are any number of faults that one could find with with the majority, if not all of the things that I've put up.
I'm not saying we now live in a perfect world.
What I'm trying to convey is that the world has changed since, you know, since 1992, when I started work in this area, as Tom says, 25 years ago.
So within Cochrane, how have we changed in this line?
The Cochrane Central Register of Control Trials. Central You're probably familiar with this from when you search the Cochrane Library.
This is the trials register within the Cochrane Library.
So as I say, the UK, the UK CC was founded in 1992 and I was hired as the information specialist and my first task was to build central.
So the immediate focus was on literature searching,
bringing us back to our to our topic of literature versus data that is searching bibliographic databases, indexing the published literature.
So indexing journal articles.
And we were focusing first of all on identifying reports of randomised controlled trials from Medline and at that time Matthias Ecker,
whose name will be familiar to many of you, said that this was clearly the best single source of published trials for inclusion in systematic reviews.
Our second focus then was on identifying reports from Embase again literature searching bibliographic database basis.
But that project has now expanded enormously.
It's now done using crowdsourcing, so individual volunteers, a citizen science initiative,
text mining and indeed taking text mining to the next level of training the machine.
So machine learning to try and optimise the identification of those studies.
But from the point of view of today's topic, Central now also includes unpublished study records from trials registries such as clinicaltrials.gov.
So Central also is moving into this area of unpublished studies.
So what do other organisations recommend with regard to searching for unpublished and regulatory data?
AAC in the US recognises you'll see that their definition doesn't meet my definition, but you know, you can't persuade everybody all of the time.
So apart from the fact that they use grey literature, I think in the wrong way.
I agree with what they say that grey literature x strike the word including so grey literature, regulatory data,
clinical trials, registries and conference abstracts should be searched in addition to bibliographic databases.
The Cochrane guidance that we're working on at the moment that will be published
in 2018 goes into a great deal more detail about regulatory agency data,
clinical study reports, and we will be advocating that all are important.
And the US Institute of Medicine guidance again says that clinical trials dot gov, clinical study results,
current controlled trials and the W.H.O. portal together with the FDA Medical Institute,
Statistical Review Records all have a role in standards for systematic reviews.
So just wanted to look at some common misconceptions about unpublished data.
And let's start with clinical trials. What? Of you can't believe how many times I've heard people say that it's a US database.
It's only got us trials in it. That is just so wrong.
And I'll move on to that in a moment. People also say that it's an ongoing trial.
So I just started again when I finished obliterating the phrase literature search from the universe.
I'm going to be moving on to ongoing trials. Okay.
Ongoing trials exist, but let's not get too hung up about ongoing trials.
Every ongoing trial becomes well, it becomes either either a completed or a stop too early or whatever, but it becomes something else.
Ongoing is an interim word. And as I say,
if you're doing a systematic review you want to know about in ongoing trials in the sense that you're not going to publish your systematic review,
if that's an ongoing trial, that's going to finish in a month's time. When you're going to get the data in three months, you will hold back.
But the whole point about an ongoing trial is that when it's finished, you get the data and you include them in your review.
So it's not ongoing trials that are so fascinating. It's trials that are so fascinating.
And it's the data from those trials,
irrespective of whether they're published in grey or normal published literature or whether they make their data available in other places.
So clinical trials, sculptors not only have data about and have information about ongoing trials, and also people say it doesn't have any results.
Well, it didn't on day one, but it certainly does now.
So taking the first of those, this was taken off the clinicaltrials.gov website earlier today, but they only updated it five days ago.
But you'll see that's the percentage of registered studies by location.
As of June, the 20th was 35% only for the US.
And even if you put both U.S. and non-U.S. in there, you'll still you're still only at 40%.
So you've got 48% of those are non US studies.
So it is absolutely not a register of US studies only.
And the fact that it only has ongoing trials.
No, if you again if you look at the you have the URL there so you can check all my facts should you wish to do so.
You'll see that they have all the different statuses of a study, right from not yet recruiting through recruiting,
enrolling by invitation, etc., etc., right the way through to suspend a terminated completed withdrawal.
So you've got absolutely the whole lifecycle of a study in there.
And what I've included on your slides here is how to find studies with results posted to clinicaltrials.gov.
And today, when I last updated these data, they were 31 and a half thousand studies with results posted.
These are not results from the JAMA or BMJ or whatever.
These are results where the principal investigator or somebody on their behalf has posted simply the raw results.
So it's a really, really important resource. Common misconceptions about the ICC IP, the W.H.O. resource.
You will find all clinicaltrials.gov records by searching the isotope because the isotope is an overall umbrella to all of these 17 registers.
No, that is absolutely not true. The presence of a record within a resource does not equate to its retrieve ability,
and there are a number of reasons for that, though there are fewer data elements in the isotope version,
there are differences in the search functionality, and there is a time lag in the posting of clinicaltrials.gov records in the isotope.
For ICI. For ClinicalTrials.gov, it's only a week.
I think there are probably about five or six of them that are a week. Most of them are four weeks.
So if you search the primary resource, you're much, much further ahead than you would be.
And it's beyond the scope. And I haven't got time to go into it today.
But my colleague Julie Glanville and others published a very important piece in the Journal of the Medical Library Association in 2014,
giving the evidence base as to why you absolutely must search both registers and not rely on just one.
So what have we done in Cochrane about improving the quality in Cochrane reviews with regard to the types of study that you should be looking for,
published and unpublished studies,
that's just a little bit of background about the project called Messier Methodological Expectations in Cochrane Intervention reviews.
The vast majority of the of the the standards were produced in 2011 2012 and then updated in 2016.
And it allows for us to say that such and such is mandatory, which means that if you don't do this in your Cochrane review.
It won't get published. So I've just pulled out a few of the mandatory standards that relate to searching and that relate to sources.
So you'll see we haven't gone completely overboard and said, you no longer need to do literature searching literature,
searching of databases such as Medline and Base and Central.
That's still mandatory for Cochrane reviews of the effectiveness of interventions.
It's also mandatory to address adverse effects, economics and qualitative research should they be relevant?
It's mandatory to search both clinicaltrials.gov and the W.H.O. portal.
It's mandatory to search reference lists of included studies and any relevant systematic reviews.
So you'll see that we still have a broad range of both published and unpublished sources that we consider to be important.
So what are the facilitators for this shifting evidence paradigm from literature to data?
I think it's largely the developments in access to unpublished data, which I've listed above.
I think one very particular facilitator has been the ICI, MJ Journal Editors Declaration, even though it's not always adhered to.
And if you do, if you do a literature search in Medline and or Embase,
you will find records that of studies that talk about where this has not been adhered to.
But I think there is increasing awareness of unpublished sources and there is increasing skills in searching these sources.
But what about the barriers to this shifting evidence paradigm?
Unpublished sources are not peer reviewed. The exclamation mark is for Tom.
I put this in specially for him a number of times.
I've heard people say, Oh, we don't include unpublished sources in our in our systematic reviews because
it hasn't been published in a journal and therefore is not peer reviewed.
Well, you know, I mean, there are advantages to peer review if it's done well.
But peer review usually involves two or three people who have their own axe to grind and their own areas of ignorance.
And on a good day, you might get some helpful comments and they might improve the work.
But peer review is not the answer to everything. And it certainly, in my view, is not a reason to discount using unpublished sources.
But you hear it all the time. People say, I'm not going to use any of these unpublished sources because they have not been subjected to peer review.
They say that trials registries have no usable content for systematic reviews, for example, no results.
And I've just shown you I think it was 31 and a half thousand from just one resource.
Researchers have no access to clinical study reports. Well, if Tom hasn't made that clear today, he will have done by the end of tomorrow.
I don't know whether he's going to get you all to register on the EMR, but I will.
There you go then. Okay. Again, I'm not saying that's a perfect world, but, you know, it's better than it was five or ten years ago.
And then people say, oh, well, regulatory data, they're all secret. You know, you can't get anything out of the AMA and the FDA.
Well, of course you can. And that's improving all the time.
So in summary, I wanted to say that all relevant data are important in evidence syntheses irrespective of their source.
And these data may be derived from published sources such as journal, articles, books,
etc. either published literature or they may be derived from dissertations, theses and reports, great literature.
They may be derived from unpublished sources, for example, trials, registers, clinical study reports and regulatory agency documents.
And all data sources, in my view, have a role to play in evidence synthesis.
Some data sources may be more prone to bias than others, and all eligible studies and their data must be assessed for bias.
So returning to the title of today's talk, the shifting evidence paradigm from literature to data.
Is there a shifting evidence paradigm from literature to data?
Well, I think there is, but not yet a paradigm shift.
As Tom says, I'm very, very precise. But the thing is, you can be part of this shift.
You can when you're doing your systematic reviews back in the workplace, which I assume many of you will be wanting to do,
which is why you're on this course and when you're going to your librarian and you're asking for support in not literature searching,
I hope, but identifying studies, you can be more demanding.
You can say to your librarian, I'd like some help in identifying studies for this systematic review.
And if they say, Oh, yes, I'll do you a Medline search, you can say, okay, that would be very nice.
Thank you. And what else do you propose? And then if they say, Oh, well, no, I'm sure that'll be enough, then you can say, Well, actually.
I'm also interested in unpublished studies. I'm also interested in unpublished data.
And can you give me some hints and tips on this and this? So I think you have a role to play in this to.