Welcome on today. You're here at the Centre for Evidence based Medicine.
I have Professor John Barlow from the from Melbourne with me,
who has a great deal of expertise in diagnostic reasoning and has done a number of research studies with over the last five years.
Welcome, John. Okay. Now what we're going to talk about today is we going to set out and look at some
of the evidence over the last 30 or 40 years relating to diagnostic reasoning.
We are both late to some of the interesting papers we've looked at.
We've looked at a paper by Arthur Elford in thinking about diagnostic thinking, a birth year perspective.
What a paper of research and clinical reasoning by Geoffrey Norman and another paper
about educational strategies to promote clinical diagnostic reasoning by Judith Bone.
And what we'll do then is finish up with some of our own thinking around reasoning.
First, we'll just go over to John, and I'm going to pose some issues about the issues of diagnostic reasoning.
So the first paper that with this paper by author of the and who did work in a hypothetical deductive model in some 30,
40 years ago, which you're well aware of.
And he used this sort of system, which maybe you could just describe, which is this thinking aloud type of work.
And if any research on this, you know, Arthur Holstein and his colleagues and the first group to do this systematically looking at how doctors think.
And I left the inheritance, which to the extent that it's still the most important contribution of I suddenly would start.
And what I did, I collected about 30 or so physicians, physicians paying a mixture of general practitioners and generally not specialists.
Yeah, I gave them a and I asked people to participate who were highly regarded in the group.
Yeah. So they would see a lot of experts and they gave them standardised cases like multiple sclerosis and similar that had a complex process.
Mm hmm. Something as a general practitioner of which they call a physician, would not say very often.
Okay. And they asked them to go through thinking aloud, whatever was going through their mind at the time when they asked questions in person.
Patient was giving a it to them.
And what I found was that within a matter of seconds of minutes, all of these doctors had 3 to 5, what they called hypotheses in mind.
Can I bring you up on that? Just. Yes. I mean, read in this 209 report and I, I see the five I mean,
the first thing that jumped out to me with any major findings were experts and non-expert physicians could not be distinguished.
And that's actually is probably the most important finding in that it's hard to believe a
situation which reflects reality that there's no difference between an expert and a non-expert.
So clearly there was a big question mark about some of the things they were finding in relationship to the method,
because at first people used to think that there's a general, a generalised method for clinical reasoning, but there must be more to it.
And there's another finding which I have some friend that is the same expert given different cases,
some of them that they collected and other cases they were not good at it, saying that there was no generalisability and there was more to it.
I just think trying to solve that problem. So if I come back to a way of doing this was to provide artificial cases.
That's right. But the problem with that approach is, is the artificial breadth of the case doesn't necessarily reflect real world practice.
And that presents a problem then in this artificial nature, either it may be too easy or too hard,
and there's no way of distinguishing expert from non-expert.
I agree that that's. Important to note, the fact that it was a standardised case of the real case takes it away from real reality.
Because one thing we do know these days and become down fairly quickly,
there's a great deal of thinking and behaviour is dependent on the environment in the context of the case.
So a expertise if content or case specific.
So in some areas you may be an expert, particularly in certain areas, but not in all areas.
And the other thing that is interesting about these findings, which is that this idea,
which has long pervaded the literature, is a hypothetical deductive method of reasoning,
which is just if you could just briefly explain what that entails and that entails that you recognise something in the patient.
Yeah. And from that you did used that that patient may have a number of other things which make him to have a certain disease.
But we know that this is unlikely to be the case in this case if the patient.
So because it's hard to imagine that you develop a hypothesis in a matter of seconds, it must be a different way of doing it.
Okay. So what I thought were hypotheses developed deduction is probably something else and that's another story later on.
So interestingly so I think there are a couple of interesting things, although think aloud is a useful technique in artificial cases,
it may be somewhat too lacking in content or case specific to be actually elicit true expertise.
And then the second thing is to say in summary, is that potentially the hypothetical deductive model that pervaded for some 30,
40 years is actually in the real world, is not how we operate because you just can't generate you hypotheses and test them.
And it would probably take a really long time if that's how you did practice.
And they use it some in the more complex case. Okay.
And I think it's interesting that even today people talk about hypothetical is the doctor being at all norm and it isn't so interesting.
And then the second thing is, what is don't hear anything is look at the analysis of case reports.
And that's an interesting but one of the cases here says explicitly recalls of who in the era
before the clinical laboratory had developed the urge physicians to listen to their patients.
They are telling you the diagnosis, which is often what we hear.
But is that really helpful, that type of reflection or issue around expert developing expertise?
Well, I think that has a point in that patients often know more than we think.
And I think it's worthwhile listening to the patient. But then you have to check check it by yourself.
Okay. Now, one of the one of the issues here,
which I found the issue with one of the with some of the cases with makes two points relevant to our understanding of diagnose,
diagnostic reasoning, the role of prior clinical experience now in the service of rapid pattern recognition.
And what that goes on in the end.
95 this rapid, poorly verbalised, highly overblown process of intuition or of others prefer it as pattern recognition.
And so within this aspect is that this doesn't clearly fit the hypothetical to this issue
about your prior clinical experience in relation to patterns and what people call intuition.
Yeah, that's an interesting thing to think about because again, people still think about pattern recognition,
which implies that you have looked at a large number of aspects of the case you have passed.
The original so-called hypothesis is usually just dependent on a single look at a single symptom from the patient.
Okay. And I think that is one of the major things to recognise that it is not a person that makes you feel about the idea in the first place,
but is more likely to be a solid feature of a case which you recognised and relationship to experience in experience.
People with medicine still make hypotheses because patients often have hypotheses in medical students.
If you see a patient limping, let's say this patient might have had a stroke because their grandmother's friend had a stroke and she was limping.
So they all have background experiences, but it's a difference of experience from what they expected.
Okay, so that's interesting. So salient features in the role of experience,
but you can have experiences outside of just being trained to be a doctor because they may already be there in the in the background.
Absolutely. Okay. Now, I'm just going to pick out one last point from this paper.
It's a very nice, sort of concise look at some of the work of last 30 years and just come in at the end so
that we know we now know that experienced physicians can and do use all kinds of methods.
So there's more than one method in effect, in current cognitive, very rapid nonverbal,
intuitive cognition is characterised as system one and system two.
We'll come back to that later, if you don't mind, because that theme being important in some of our work.
But it's interesting here and maybe we should tell you this about how do clinicians decide which approach is best suited for the case at hand?
And it leaves a question mark there. That's a question mark.
Yeah. Even today, if they are so unpopular, if that's how they work on the, you know,
theory we're going to talk about that's to shed some light and a few things I but
okay so that's some of the evidence from nice points that we may touch on later.
Then in the middle here, there's a paper by Judith Boyd in the New England Journal of Medicine,
which is educational strategies to promote clinical diagnostic reasoning, which is sort of aimed at teacher.
This paper is a thought review, look at a few cases and so forth.
And what what she does there is is talk about the expert written brought two sets of skills to the encounter
with the patient looking at a case about Gamze I think it was and looking at how an expert with a non-expert,
a bit like a registrar or consultant versus a sort of medical student,
the resident probably formed an early impression, a mental abstraction of the patient's story.
And then the second thing is the expert residents clinical case presentation, presentation with a succinct summary of the findings,
providing the teacher with a clinical picture of the patient as seen through the residents.
I never thought of what she was trying to say with some aspect of expert.
How did that help us in any way or and it helps in the sense that after it was shown that
the type of thinking from the original study was mostly in the course of the answer,
people started thinking about what sort of knowledge is necessary to make diagnostic decisions
and what sort of schemas is the brain stored in the brain to help you with this work?
And so the basis of what she is talking about is that when you have seen a few patients involved in this
particular case and have seen that possible aspect of that and can differentiate it from something else,
like what you have lost to arthritis.
And you do this because you have a picture in your mind, a complex of knowledge that you've built up from seeing a number of cases in contrast.
The first year medical student and a first year intern.
And since he's actually the patient is God is on his list.
So we know from previous studies that a novice tends to diagnose on lists, often not giving priorities because he doesn't know enough enough about it.
In contrast, the person has got a complex knowledge about it and as much work as when it wants to go.
What do you do next to make it better? To make a better diagnosis?
Often they are referred to as an illness illness script.
Yeah, well, we'll come to that later with Jeffrey. Norman may be the only three route so.
So her position is this position is provides a sort of simple model which looks very appealing but themes what we've just heard
previously and from Elstein where the conclusion a bit too simplistic book talks about the first step is based on knowledge,
experience and other important contextual factors, such as you're in the emergency department, you're in a surgery ward.
What time of day it is? What's the patient want? The second step is this data acquisition.
Depending on the setting elements of the history, findings of physical exam and results of the test.
And then another early step is the creation of a mental abstraction or problem representation.
Usually a one sentence summary. Defining the specific case in abstract and patient has pain early in the morning and it's worth an exercise.
Clearly, this is osteoarthritis. Interesting, though, she goes on to this clinicians may have no conscious awareness of this cognitive step.
The problem representation. That's an interesting there's no conscious awareness in this divide.
And how do you see what where does that come into the reasoning process?
And if we're not aware of it, how can you improve that or even understand what you do?
Well, a description is appropriate, practice is accurate up to that point,
but it depends on how much introspection the doctor will exercise and how much,
you know, think about it or just accept it coming out of the blue, assuming it's it would be fine.
It's not really.
It's by no means unusual to diagnose this fight. And one of the questions we have, which we still haven't answered completely.
As we said before, it had to change from acceptance to let's stop here and wait and look at it in more detail.
And I think you want to talk a bit later about this. Yeah.
But I believe that's what this paper is about, is how to improve the knowledge in relationship to that particular disease.
But it's not really clinical reasoning.
Is disease recognition and disease oriented as opposed to condition oriented?
So that's interesting. But she does go go back and just relate a bit to this many methods.
She talks about this both non analytical and analytical reasoning strategies are effective and are used simultaneously in either interactive fashion,
so they must be going at the same time non analytical reasoning as exemplified by pattern recognition.
This is the second time we've seen it come up is essential to diagnostic expertise.
Well, no, not analytical reasoning is a demand problem because not all analytical things happen in your brain.
Yes, but it's not to me. My mind and reasoning process reflect on something, okay.
Something that you have built up over time in many cases over here, if you're an expert.
And the interesting thing about it is that it changes all the time a great deal, which I think we all need to come back to later on.
So does that mean so from a non analytical perspective,
the major way to improve in known all I intellect is through experience and that's how we sometimes define expertise in terms of experience sometimes.
However, then she talks about deliberative, analytical reasoning is a primary strategy when a case is complex or ill defined.
And boy, do you see a lot of them when you're not quite sure what's going on.
The clinical findings are unusual or the physician has had a little clinical experience with the particular disease.
And, you know, there must be then a way of reasoning or working out what's going on that's different.
That's and that's me. That where the skill, which is not related to expertise, starts to come in in some way.
It is it is a skill, but it's not just a skeleton.
It's a way of operating, as it were. Some people are more likely to be want to learn, and I think a lot of they think why.
In other words, explore what you're saying and relating to the previous existing language that you have.
And I think that that to me is the issue here. What's what I've learned.
You are also a physician that you going to see, which will determine what sort of expert you have.
And I'd be okay. And I think as you just finish up with learners with strong diagnostic reasoning skills,
often use multiple abstract qualifiers to discuss the discriminating features of a clinical case in comparing
and contrasting appropriate diagnostic hypotheses and leaking each hypothesis to the finding in the case.
So they use in different things. They may say, Well, here's the pattern.
This is what I think this may be, but I going to think about this test result.
It might make it think this and this ruled it out. So they're using different techniques all at once.
I think that I believe they don't use patterns. Okay. I use critical cases that I think are important.
And that's what's one of the early findings in the first phase of this study is that the patient
could doctors do not collect more data than bad actors and America's correlation between apathy,
collect and the rightness or wrongness of your diagnosis.
Okay, interesting. We'll come back to that and come back to that.
Interestingly, interestingly as well in some of the letters that you're looking at responses,
but some of the criticism and one of the writers said Berlin's review of
educational strategies can be that can be used to promote diagnostic reasoning,
does not sufficiently emphasise the concept of premature closure.
And I thought that was interesting as well. And I'll come back to that when we come throughout some of the work you've done over the last five years.
But let's just before we get there, Jeffrey Norman Hey, I did a paper here in 2005,
which is the research in clinical reasoning, past history and current trends.
I think it's interesting to know of the the 40 million papers published in PubMed every year, there actually isn't much research in this area.
There's there's only a couple of. Hype is out there. Really? And the other problem is it's published all over the place.
Yeah. So it's published from psychology to medicine to surgery to education and so on and so on.
And many in the big journals, I mean, it's quite nice to see the New England Journal in Medicine.
This is 2006, actually don't really push publish much about teaching or reflecting or reasoning around diagnostics, do they?
That's right. Which seems to me a sort of paradox, because this is an important concept of medicine.
If you get the diagnosis right, we can go from there. If you get it wrong.
We're in deep trouble.
Yet nobody wants to emphasise they're more likely to publish a result of a test single test property and to think about how doctors may learn better.
But that's the difference between disease or education and conceptualisation.
So what is just fine? I mean,
there was a number of themes that actually Norman I quite like and I'll maybe I'll take each one and then you could your thoughts on the first one.
There is little evidence that reasoning can be characterised in terms of a general growth variables.
Yeah. Just for the people listening. What does that mean, general process variables?
Well, what it means is that they started with all this.
The idea was that, you know, you've got that system of reasoning and we can teach it to every student for every type of case.
But even the early studies have showed this is not the way it works.
Doctors do it in a very different way, and every doctor will have a different way of doing it himself or herself.
Okay. That's interesting. And that's why is evidence that expertise is associated not with a single basic reputation,
but with multiple coordinated representations in memory from causal mechanisms to prior examples.
And that's what he's trying to get that more than one. Okay.
Different representations may be utilised in different circumstances, but he goes on, this is again.
And there are things that if little is known about the characteristics of a particular situation that led to a change in strategy.
So this is a bit like when we don't quite know which one the presented when why you change the way you do.
Yeah, I have to go back to the beginning and say that they thought that they know very little about this.
There there's some studies recently which addressed a few ideas, but.
We can really divide what's in a concept in the brain into three different categories, again mentioned by Norman Rockwell.
Now, maybe we'll come to that in a minute or come that sort of a nice little narrative through the thought 1780s and
nineties of all things in the crazy how things work and talked about the hypothetical and deductive were interesting.
We talked a little bit in the eighties about the issues about chess and the growing lecture on expertise in other domains.
Particularly, chess was quite influential in the 18th and interesting to note know the single best measure of
expertise in chess was recall of a typical mid-game position where after a five second exposure,
experts would typically recall the exact position of about 80% of the pieces.
It is basically showing the expertise.
Is it based on underlying relation between memory performance and expertise and that characterises all of the thinking around that time?
That's right. Interest in a chess master is remembered about 50,000 game positions to get there.
However, what you say, medicine, there is a little gain from gathering and remembering extensive amount of patient data,
which I think you've already mentioned. Consequently, foreignness is a poor index of expertise.
That's right. So this is where you see some connections there for ten, 15 minutes with the patients and others in their 60 seconds, 90 seconds.
And this concept of some people will write a whole ream of history and other people write two
lines and say there's a diagnosis and in some way the expertise of the ones who come through.
Would you agree more quicker to get into what the nub of the issues in the case was?
Quite okay. You can disagree me if you want.
That's okay. It's been shown that experts with original first thought think of is more likely to be arrived in.
Someone is not an expert. So that's true. On the other hand, just how much data you collect.
Can depends very much on that person's inclination.
Some experts just spend a lot of time and some experts do not have to point it all to risk taking.
Basically, if you make a diagnosis is a matter of seconds yet quite at risk to the outside world and also to yourself,
because you didn't have to to check on your thinking.
In other words, you did not go on to system.
Do as we are okay to check on your system one, but it's a matter of choosing the place that you make and you don't have to do.
This is the issue that you're talking about. Okay, that's interesting.
I'm going to come back to that, but just a bit in this Norman paper that really interested me,
which was research during the 1990s focussed on forms of mental representations going back to
the old pattern recognition and experience and can be divide roughly into broad categories.
And there was a Jack Goldsmith who did a lot of the work and they said and it comes into basic science or causal knowledge, schemas,
scripts and other representations of the relation between signs and symptoms and diagnoses and exemplars based on experience with past cases.
So the example here is that by teaching basic science, you create the ability to have the ability to dip into the exemplar and go back.
And but interestingly, there's a very pertinent quote which is going to read Given the amount of educational time devoted to learning basic sciences,
it is reasonable to assume that expert clinicians exemplify the application of scientific concepts to clinical problems,
yet primarily that expert clinicians make little use of biomedical science in daily reasoning.
Now there's a conflict there in terms of what we teach and what you use in practice.
I think the answer to that is that the basic science knowledge is something which you sometimes isn't used,
and it depends up to the point on the speciality point on the patient, evidently.
It is a concept to be shaped when people talk about encapsulated knowledge of knowledge that's so sticks to the script, for instance, or Schmidt.
And it's the knowledge that you have picked up somewhere else that you stick onto it.
And how much of that is okay is taking place? Because they're very much dependent on the types of cases you are saying.
And so when you say you do not use basic science knowledge, that is not entirely true because you can't,
because you can show that some of it is important to you and some of it is not.
On the other hand, when you look at a student or watch, you have a doctor seeing a patient for the first time.
They use a great deal of basic science.
Yeah, that's how they read things that the expert doesn't have to replace the good doctor with all that somewhere.
I suppose I shouldn't say. Well, some of it. Is there some?
Because certainly learning how many things you've got in your brain that Ossify is not going to help you.
So interesting thought is that I'm going to move forward now because into and we've got
a paper here identifying the early warning signs for diagnostic errors in primary care.
And this is John Barlow and actually myself here and Clare Guide and Matthew Thompson in BMJ Open.
This is actually a paper that looked at 25 experienced general practitioners.
But this is built on a couple of papers that basically try to bring a model together that
says part of this reasoning process and we we heard earlier right beginning from our
Eltham with around the system want to system to rethink you want to just explain what that
is a little bit and then we'll look at how that applies to the process of diagnosis.
Arthur Elston said in this and other paper that we were referring to him that one of the biggest problems they have is
how to thank you going to system two if you need to make what matches you need to as opposed to you don't need to.
So let me just about health system. One reason is the fast intuitive reasoning and system two is useful to the liberal analytical reasoning.
I mean, in some of the books, like I tell you this and they thought to say a system one reasoning so if I say to you to close to
immediately comes to your mind and people listening more you can't stop but go for can you in your mind.
But if I go 35 times 23, then you've got to go into a deliberative, analytical mode to try and work out.
You can still work it out, but you have to think about it. And that's what's going on here up to a point.
So how does that then apply to the diagnostic process?
How do clinicians work in relation to this fast and slow thinking for.
They use mental pictures. I mean, to what's in your knowledge base?
And they have different names for it. The name I like best is Personal Knowledge.
It's okay. And personal knowledge is based on a number of things.
Basic science, in other words, was referred to quite often as the theory of the profession.
This is how she talks about it is what you were talking about before.
Yes. And there's we there.
The second thing that there is the disease oriented schemas that we were talking about that Schmidt and and a number of other things like heuristics,
they're going to be talked about. It used to being sort of rules of thumb, for instance, about if if you do this, you do that.
But there are a couple of other things, and one of which are very important.
And probably the most important is has been provided to things you have seen before.
And you've got a large number of things and then you switch it on to your present knowledge, it becomes your personal knowledge.
And one of the problems we have is we all have different ways of approaching this because we are different people,
so somehow more likely to want to incorporate it and others that do not get much interested in it.
And so what, what I'm talking about is that that is personal knowledge that we have,
which depends on the person himself or herself, as well as the experience and as well as the okay theory.
And as another thing that is very important to recognise is the environment that you doing your work in,
which is you are referring to before that the original cases were not real cases.
Yeah, we are the real world. Yeah.
And one of the most important decisions to make about that I spent time in, as true or not, is coming from the outside world.
And I think you have to think of it as for instance, a typical example is Friday afternoon, everything is closing up and again the next day today.
Is that tomorrow? Because I guess I will be closed tomorrow. Okay.
Yeah, now, now. But when? So in the context of important but we also had this bit earlier which you mentioned and which comes up a bit,
this idea that in this process that there's not so much pattern recognition in, but it's critical queue.
That's right. And what's the difference between these? See the difference between pattern recognition and critical cues,
a better position to make things up, a picture where if you have a patient with a cough,
you have to ask questions about how long you've had it and what time is that cough around the district and so on and so on.
But the experienced doctor will just ask one or two questions about the most complex case.
If you come along to me and you say I had a headache, which was like a thunderclap.
Yeah, you don't ask any more questions then you're more or less not.
That's a single, I think, or a red herring quite often.
Okay. Because she was what's happening. And one of her suggestions will be that certain red herrings that says the opposite red flags,
in other words, suddenly telling you that there's something to look up to, to be worried about.
So let's think a bit harder about this. Okay. But the environment, if you know that you're in there about should your you're ache.
Right. And again the anxious about it, that's the time to make sure that you think about it another way and stop and get back again.
Another one might be and that that is why this work is important that we attach to specific diagnostic not diagnostic readings,
but blues critical views and restaurant red flags and special steps of the diagnostic process.
And if you look at the beginning, the initiation phase, yeah, they keep talking about salient feature.
Maybe I'll just come into this salient feature in this initiation while you're talking then because this paper,
the initiation closure of the cognitive process, that's the beginning in the end are the most exposed to risk of ever error.
Cognitive biases developed early. The pro athletes were at the end.
And so somehow you either are at the beginning of a diagnostic process and this is where that work about heuristics you anchor
in you think it's too early in the or at the end closure because you may be rushing or you're not quite sure and you say,
well the patient's well and actually you should have carried on correcting some information.
How does that help the clinician then think about improving their performance?
And if you have used a certain salient feature for a long time, you know, and based your diagnosis on it,
which we tend to do, then from time to time, you should take time off, time out and look it up on the literature.
So what we're saying is because we know that I know I mean, I've seen that some people develop rule based where they're clearly wrong.
I mean, I've seen one. He can't have pulmonary embolism unless you got a company.
What is that? Somebody that articulated to me. And that's clearly wrong.
There are many of these theories. So we're going to finish up now.
I just want what do you think, all the take home messages?
I think for two types of people. One is for the clinician out there who's trying to improve with reasoning and diagnostics.
And second, for those that try to teach this type of work to medical student,
these issues are the best thing to do would be to work in an environment which supports this in
that I think there's a lot of a lot of writing about this and we know a lot about this as people.
We have it differently in different environments.
And if you work in a supportive environment where you can talk with colleagues and review their activities
with discussing cases and then looking at as to what should be happening and work with a facilitator.
That would be a number one, in my opinion. Okay. Okay.
Okay. Well, I'm going to let conclusion from Alpha Alpha and finish of this conversation.
I thought, this is a nice finish, but if diagnostic errors can never be entirely eliminated.
Human reasoning is not perfect. And so mistakes in interpretation in parents will be made.
Sooner or later situations will be encountered where one's knowledge will be incomplete.
Feedback in this clinical setting setting is spotty.
Second, even if our knowledge were complete and our inference process with perfect clinical evidence is not.
Thank you very much, John Barlow, here today at the Centre for Evidence based Medicine.
It's been a pleasure. Thank you.