Effects of two different instructional formats on

Authors: W E Sjoukje van den Broek Marianne V van Asperen Eugène Custers Gerlof D Valk Olle Th J ten Cate

Publish Date: 2012/08/21

Volume: 1, Issue: 3, Pages: 119-128

PDF Link

Abstract

The script concordance test SCT is designed to assess clinical reasoning by adapting the likelihood of a case diagnosis based on provided new information In the standard instructions students are asked to exclude alternative diagnoses they have in mind when answering the questions but it might be more authentic to include these Fiftynine finalyear medical students completed an SCT Twentynine were asked to take their differential diagnosis into account adapted instructions Thirty students were asked not to consider other diagnoses standard instructions All participants were asked to indicate for each question whether they were confused answering it with the given instructions ‘confusion indication’ Mean score of the test with the adapted instructions was 815 SD 38 and of the test with the standard instructions 829 SD 50 p = 0220 Cronbach’s alpha was 039 for the adapted instructions and 066 for the standard instructions The mean number of confusion indications was 42 SD 44 per student for the adapted instructions and 167 SD 285 for the standard instructions p = 0139 Our attempt to improve SCTs reliability by modifying the instructions did not lead to a higher alpha therefore we do not recommend this change in the instructional formatClinical reasoning is considered a central component of a physician’s medical expertise and there is broad consensus that it should be taught and tested in medical curricula 1 2 One test format that has received considerable attention in the past decade of medical education literature is the script concordance test SCT 3 This test is based on clinical knowledge organised in illness scripts a physician compares the presenting patient to fairly similar cases he or she has encountered in the past and uses these experiences to efficiently make judgments regarding the present case 4The SCT is a relatively recent innovative method to assess clinical reasoning skills 5 Evidence for its validity is available but it still has to be further confirmed 3 6 7 The SCT uses a closedanswer format to assess clinical reasoning and stimulates the test taker to think of realistic clinical scenarios in which candidates are asked to interpret data to make clinical decisions 4 5 In an SCTs clinical scenario a patient’s history the results of physical examination and occasionally the results of diagnostic tests are provided In addition a diagnostic hypothesis is proposed Next the test taker is presented with a new clinical finding that could alter the likelihood of this hypothesis ie making it potentially more or less likely or not affect it The response answer is made on a fivepoint Likert scale ranging from ‘makes the diagnostic hypothesis much more likely’ through ‘does not change the probability of the hypothesis’ to ‘makes the diagnostic hypothesis much less likely’A unique feature of the SCT is that there is no predetermined ‘right’ or ‘wrong’ answer to each question but that the examinee’s response is compared with the average response of a panel of experts which is used to construct the scoring key The score assigned to every response alternative corresponds with the proportion of experts choosing this alternative This scoring system is designed to measure the difference in existing scripts between examinees and this panel 4 In other words the scoring is weighed by the degree of agreement between the experts 8The SCT has been applied in several target groups 6 9 10 11 among which preclinical medical students 12 13 At the University Medical Center Utrecht the SCT was applied among secondyear medical students as a final test of a course in casebased clinical reasoning in 2009–2010 Students’ evaluations of the SCT revealed that they appeared to be confused about the instructional format The issue at hand is that new information in a clinical case may not directly influence the probability of a focal diagnostic hypothesis—for instance it may not bear upon this hypothesis—but may have an indirect influence by making an alternative diagnostic hypothesis more or less likely with a concomitant change in likelihood of the focal hypothesisThe standard instructions for the SCT which we used for the test in 2009–2010 prescribe that candidates should not consider alternative hypotheses when assessing the change in probability of the focal hypothesis as a consequence of the new information 14 Our evaluations indicated that secondyear students have difficulty working with these instructions An example can explain this ‘A 72yearold lady known with rheumatoid arthritis presents at the general practice surgery with a swollen knee Given only this information the suggested hypothesis is inflammation of the joint’ The new information is ‘she fell off her bicycle 2 h ago’ Question how does this new finding affect the likelihood of the suggested hypothesis Respondents must choose one alternative ‘makes it less likely’ ‘makes no difference’ or ‘makes it more likely’ With the standard instructions for the SCT students should answer that falling off the bicycle does not affect the likelihood of an inflammation However it is imaginable that students find it difficult and unnatural to choose this answer as in reality they would now much more readily think of a trauma and therefore find an inflammation less likely To summarize the likelihood of a diagnostic hypothesis as perceived by a respondent may not only be affected by a causal relation to the signs and symptoms in the case but also by its ranking among alternative hypotheses The instructional format of the SCT could alternatively be phrased as ‘keep other diagnoses you find plausible in mind when evaluating the changes in likelihood of the proposed diagnosis’ We would expect that students would find this more natural and would be less confused This could subsequently be reflected in a higher reliability as it could reduce error variance caused by confusion and also yield a higher validity as it resembles the authentic setting better possibly resulting in higher scoresThus in our study we investigate whether the adapted instruction to consider alternative diagnoses when answering the questions in an SCT results in higher scores and increased reliability compared with the standard instruction not to take other diagnoses into account when assessing changes in likelihoodTo examine the effects of instructions on scores and reliability of the SCT a panel of 59 finalyear medical students from a finalyear course and 18 experts general practitioners completed an SCT We invited finalyear medical students because we assumed they could answer the SCT questions without specially preparing for the test whereas junior students would not be able to do this All students and experts participated voluntarily and data were processed anonymously

Keywords: