To a medical student the final examinations are something like DEATH: an unpleasant inevitability to be faced sooner or later.
Richard Gordon
As an internal medicine physician teacher used to a discipline based system of university assessment becoming a university examiner for an integrated system of assessment was a learning experience. What I describe maybe unique to this part of the world and there aren't many published descriptions of what transpires in such a system of examination in spite of its having been in place for quite a few years now.
The written sections of the so-called comprehensive exams are still discipline based. It is only in the clinical examinations, the long and short cases that a candidate is examined comprehensively by a set of examiners together from various clinical disciplines. The OSCE (Objective structured clinical examination) is held on a separate day with all major clinical departments contributing to set up their stations to assess the candidate on simulated procedures and structured patient counseling.
Common ground in an integrated but multidisciplinary system:
On the day of the long and short cases it was interesting to share the table with a gynecologist, surgeon and pediatrician (and reflect on how little perhaps one knew of the other disciplines). Perhaps due to this felt discrepancy amongst each others knowledge about disciplines that were different from our own, there was some informal discussion between some of us on whether each one of us examiners should be examined by the others in the panel, just as we would do the students.
The intent of such an informal examiners assessment may have been to decide what basic knowledge (sort of common ground) a candidate may require to function eventually as a successful specialist (which we assumed ourselves to be)?Or then we just wondered, if the best persons to have an overall multidisciplinary knowledge of all specialties would be general practitioners/ family medicine academicsbut then they had their own post graduate curriculum to look after and could not be classified as sole undergraduate teachers.
Other than this there would be logistic problems of accommodating an examiner assessment (informal as it may be) into an overburdened system. So finally we laid our queries to rest by concluding that yesterday's students (us) may not be able to answer what would be required by our present students to practice in future and so we should shelve the idea of developing a common ground (as far as examiner/assessor knowledge was concerned).
In our existing system of integrated assessment in the university finals the student is assessed on multiple disciplines by four different clinical panels each containing four clinicians from the major disciplines. Each candidate is allotted a long case and three short cases. If a candidate takes a long case in surgery, s/he is allotted a short case in medicine, pediatrics and obstetrics-gynecology. The candidate needs to present her/his case to the panel of examiners, one of who (depending on the discipline to which the case corresponded) takes the lead in asking the maximum number of questions. After finishing their interaction with the candidate and reaching a common consensus, the examiners in the panel decide on the marks to be awarded.
Assessment of clinical competence:
It is human to have bias and much as we would like to drive away this human factor from our assessment strategies it is very much there with some degree of variability in most areas that have to deal with humans and a clinical encounter is a definite example of why it thrives. It is agreed and demonstrated that the reproducibility of long cases is in stark contrast to the objective structured clinical examination (OSCE) that often achieve reproducibility coefficients more near ideal. (1)
Although we would definitely like to objectively structure our thought processes it is not how we normally think. As clinicians our day to day dealing with other humans (our patients) are characterized by cognitive short cuts that we take subconsciously not for lack of time but because we are comfortable with it. As clinicians we may not attempt to understand familiar problems and opportunities using calculative rationalityespecially when things are proceeding normally. We do expect our students to follow a reasonable adherence to taught rules or plans. However a rigid adherence may bring about little situational perception with an added disadvantage of no discretionary judgment. A little practice and experience may enable them to cope with crowdedness and pressure and eventually see actions partly in terms of long-term goals or a wider conceptual framework. (2) All this perhaps requires time and that is one of the reasons our students are expected to become more and more competent only after passing the exams as they get to handle more and more patients. We do expect them to have handled a reasonable number of patients however and gained a suitable level of clinical competence before their exams.
A lot of this would then depend on their formative clinical years when they have the time to spend with their patients, listen to their stories, practice clinical methods on them and eventually develop the skills to integrate the myriad informational inputs history and clinical findings generate for even a single individual patient. We do have a continuous clinical assessment as part of the formative assessment in our clinical curriculum and this forms around 20% of their final university marks for overall assessment. This is based on the impressions of the clinical teacher on a day-to-day basis objectified by dividing them into knowledge, attitude, skills and participation with points (1-5) for each. The problem with this system to my mind lies in its inability to address the detail of work each candidate would have needed to do each day to attain those points awarded. In a present electronic age this system of formative assessment may be improved enormously with a candidates posting of his day to day to clinical activities, particularly the learning points achieved after a critical refection on their days work. This gives the clinical assessor a tangible, persistent document of the candidates formative learning.
Internal examiner bias
This bias is easily taken care of in the theory papers as the candidates are coded and there is no way I can tell who is who while examining their theory papers.
However for the clinical assessment I knew most of the students by sight. As an internal examiner who had interacted with the students for 2 years I had foreknowledge on many of the candidates that I suspected would have influenced my decisions on the candidates. Bias is a human cognitive construct whereby one is predisposed to act on the basis of past social cognitive interactions.
I was an official academic mentor for some of these students and although our social interaction was limited due to time constraints, I wondered if a barbeque party that I had enjoyed with them in the past might influence me to decide positively in their favor? I was thankful to find that the university controller of exams had taken care of it, as I didnt meet any of my mentees on my panel on the day of the assessments.
As I was assessing the students during the long case and short cases I could tell the bias working as I had mentally summed up the number of marks to be allotted to each candidate even before they opened their mouths and was sometimes surprised to see the discrepancy in the amount of marks given by the external examiner and our collective consensus and the one that would have been given if it had been based only on my internal examiners bias irrespective of how the candidate performed on that day. I was disconcerted to see a few candidates doing badly who I knew to be bright and hard working and a lucky few sailing through who in my biased opinion weren't bright enough nor had worked that hard in the wards. In the end I had to acknowledge the ingenuity of the exam system that arranged for this degree of powerlessness to the internal examiners bias. How then do we do justice to the candidates who were bright and had worked well the year round but slipped on the fateful day? Perhaps we could increase the formative assessment weight in deciding the candidates ability to fail or pass the exam.
In search of ideal fool/fail proof exams
We are contemplating a formal quantitative analysis of the data generated by the discipline based exams in the past compared to the comprehensive format at present for the last 5 university finals but one quick approximate look at the data would tell us that there was a dramatic increase in the pass rate of students with the new comprehensive integrated approach in the very first few university finals, which tailed off to a current reasonable stability (although the pass rate is still higher compared to the previous discipline based exams). In spite of the overall average pass rate we still had 20 failures in the last university final that I attended, five of who were taking the exam for the third and last time. They had to leave the course after spending five years in medical school (some of them had spent eight as they had taken their time in the other semesters as well). The psychiatry department had the unenviable task of breaking this bad news to them and at the post exam debriefing there was a heated discussion on ethical and legal issues of this phenomenon. Eventually the clinicians decided that these students should not have been allowed to enter the clinical curricula and called for stricter criteria to pass students in the pre-clinical years.
Failure is a stepping-stone to success for some students who do exceptionally well in their reassessments but for a few this doesnt work and aptitude is held responsible. Aptitude is the ability to respond aptly to challenges such that one is able to successfully able to overcome it. Some of these students most assessors agreed would perhaps do exceedingly well in other disciplines that required a different kind of aptitude. Only it would have been so much better if we had realized that earlier. Again they may have made splendid doctors on the long run but we'll never know.
A last question that was raised in the post exam debriefing was, "How can we expect our students to have an integrated comprehensive approach on just the day of the university finals when the whole year round their training was disciplined based with two to three monthly postings in Medicine, Surgery, Obstetrics-gynecology, Pediatrics etc?" Well to have an ideal integrated training we would need an integrated hospital..."was one conclusion. The question then is how do we really move whole and soul from a discipline based to an integrated system of assessment as long as our practice remains discipline based?
References:
Norcini JJ, The death of the long case? BMJ. 2002. February 16; 324(7334): 408409.
Greenhalgh T, Intuition and evidence--uneasy bedfellows? Br J Gen Pract. 2002 May; 52(478): 395400.
