Multiple choice questions as a ranking tool : a friend or foe ?

Traditionally ,Multiple Choice Questions (MCQs) are used to assess knowledge in Medicine (Jeffrey & John, 2001). In Sri Lanka, a common MCQ assessment was recently introduced to evaluate medical students of all faculties of medicine and to tabulate a ranking system. The employment of medical graduates in the state health sector was initially based on a ranking system in the order of merit statistically calculated from a random choice of theory and clinical assessment marks at the final MBBS examination. This required comparison among 6 medical faculties which had non identical syllabuses and their own assessment systems for their graduates. In addition, the exams are usually held at different times of the year as the commencement and conclusion of the medical courses in different faculties vary.


Introduction
Traditionally ,Multiple Choice Questions (MCQs) are used to assess knowledge in Medicine (Jeffrey & John, 2001).In Sri Lanka, a common MCQ assessment was recently introduced to evaluate medical students of all faculties of medicine and to tabulate a ranking system.The employment of medical graduates in the state health sector was initially based on a ranking system in the order of merit statistically calculated from a random choice of theory and clinical assessment marks at the final MBBS examination.This required comparison among 6 medical faculties which had non identical syllabuses and their own assessment systems for their graduates.In addition, the exams are usually held at different times of the year as the commencement and conclusion of the medical courses in different faculties vary.
This system of assessment appeared to be non transparent and was felt to be unfair by the students as well as the administrators.A need arose to decide on an unbiased method to develop a ranking system to fill the training posts in government hospitals.As a homegrown solution to this, a common MCQ paper was proposed.An appointed panel from each faculty formed a MCQ bank which was to be used for this evaluation exam.As the timing of the exams in the different faculties was different, each faculty received a different assortment of questions from the question bank.
Whenever universities could time the exams together, they received identical papers.However a section of students and staff felt that unless all universities use the same question paper there would still be a degree of bias.This study was thus designed to ascertain the perception of new graduates of the University of Kelaniya, Sri Lanka on the common MCQ as a tool of ranking.

Objectives
The common MCQ for ranking medical graduates at the final MBBS exam was accepted with mixed feelings.Some voiced concerns that a MCQ ranking was not appropriate in deciding the fate of their medical career.The objective of our study was to ascertain the perception of new graduates regarding the common MCQ as a tool of ranking.

Methods
A questionnaire was administered to recent graduates of the Faculty of Medicine, University of Kelaniya.The graduates were requested to voice their opinion on various aspects of the common MCQ, student opinions on the use of one paper for all of the faculties, the use of other components such as structured essay questions and clinical assessments and the proportion of marks that should be allocated for MCQs were analyzed.

Results
The majority of the recent graduates, 96/120 (80%) responded to the questionnaire.A significant number either disagreed or strongly disagreed (n=22; 22.9%) that a common MCQ examination was a suitable tool to determine the order of merit (Table 1).Seventy three (76%) either strongly agreed (n=37; 38.5%) or agreed (n=36; 37.5%) that the common MCQ should be replaced with a structured essay question (SEQ).Fifty six participants (58%) were of the opinion that 40% or less than 40% of the total assessment mark should be allocated for the common MCQ and the rest to be constituted by other forms of assessment such as SEQs and clinical components.Twenty one graduates (22%) thought that MCQs should carry 50% of the total assessment marks.The majority either strongly agreed (n=76; 79%) or agreed (n=12; 12.5%) that all faculties should receive the same question paper.They remarked this was the only rational way to alleviate bias from such a process of assessment.Over one half were of the opinion that a common MCQ alone was not a fair assessor, chiefly because of non uniformity of the "common MCQ" paper.

Discussion
MCQs have been used extensively in all kinds of examinations.It is a time tested method of assessment of knowledge in both undergraduate and postgraduate medical education (Jeffrey & John, 2001;Schuwirth et al., 1996).
Tabulating a common ranking order of merit of the newly qualified doctors has been a controversial issue for a significant period of time.The current practice is to use the marks of MCQs at the final MBBS exam to formulate the order of merit.The opinions of the medical students differ regarding the way they are being tested.
It is interesting to note that a significant proportion of participants felt that this was not the best form of assessment.They believed that other types of formal testing would result in a better chance to display their knowledge.In support of this argument it is understood that according to the assessment matrix MCQs test only recall and problem solving and hardly test the clinical attributes (Table 2).
MCQs sometimes have inherent flaws such as frequent lack of content validity, cueing effect of the options and encouragement of guessing.The low taxonomy level of the items would also have serious implications in examinations of clinical competence.However, it was argued that it is the best way to objectively test a student (Newble & Elmslie, 1979;Bloom, 1956;Premadasa, 1993).This is mainly because there is no examiner bias in the assessment.Although students wish for incorporation of other methods like SEQs and clinical assessment, these methods may be flawed by examiner and institutional bias (Abdel-Hameed et al., 2005).
A common MCQ paper for all the faculties would be the best bench mark where no student could complain of discrimination.However, there will be many obstacles in bringing all universities under one fold of assessment, It is worthwhile to ascertain if the students on their part would agree to wait if required, so that all can sit for an identical exam or whether all would want to beat the other to finish the MBBS race.

Conclusion
It is now "agreed" that a common MCQ paper is the "best way" of objectively testing a medical student for the purpose of ranking in the order of merit in the current Sri Lankan context.While respecting the opinion of these graduates on their views on this crucial matter, it appears fair that their plea for all medical schools to be tested by a single exam be considered by an independent authority.
Medicine, University of Kelaniya, Sri Lanka.Correspondence: C.A.H. Liyanage, Tel: +61450070875 Email: chandikaliyange@hotmail.comIt was important to ascertain the perception of the candidates, as they were the most affected by this change.