‘Blueprinting in Assessment’ – an Online Learning Experience

Objective: Faculty-training on blueprinting is necessary to make medical teachers aware of threats to the attributes of assessment. The experiences and challenges of preparation and moderation of the online ML Web Session are discussed so as to facilitate the readers to conduct such activities. Methods: GSMC-FAIMER Regional Institute, Mumbai, India chose “Blueprinting in Assessment” for Web Discussion in January 2018, as a part of the GSMC-FAIMER Fellowship Programme. 32 participants were involved in an asynchronous manner, using Listserv Online Discussion Forum, from 1st to 31st January 2018. A sequential flow of the activities was aligned with chronological sequence of pre-defined objectives. Reflections were taken at the end of the month. Results: Comparison of Pre-test and Post-test analysis showed statistically significant improvement in percentage scores (p<0.05). The participants appreciated the ML Web Discussion Session activities in their reflections. Conclusion: This ML Web Session achieved the learning objectives, increased the knowledge and led to a deeper understanding of blueprinting in assessment, through comprehensive, though asynchronous, peer-group discussion.


Introduction
Assessment is a very important aspect of the educational spiral. It must be valid, reliable and feasible.
Preparation and assessment of a question paper requires strategy and planning. Aspects like validity, reliability, use of taxonomy, punctuation and grammar used to set questions, all significantly affect the assessment. To ensure a good assessment in either a formative or summative evaluation, blueprinting of the question paper is one important criterion. This is very important for faculty training and development. (Raymond & Grande, 2019;Coderee et al., 2009). Blueprinting represents a detailed plan of action or a map, essential to include and streamline all aspects of curriculum including domains and their appropriate contribution to assessment (Patil et al., 2014). It ensures proper weightage to important topics and aligns questions to the learning objectives (Patel et al., 2016). Errors of Construct Under-Representation and Construct Irrelevant Variance are avoided, to increase the validity of the question paper (Downing & Haladyna, 2004;Sharma et al., 2016). In addition to the selection of topics, mark distribution is another parameter. This could be based on importance or weightage to a topic given according to the time or credit hours

South-East Asian Journal of Medical Education
Vol. 13, no. 2, 2019 allocated in the planned curriculum (Abdellatif & Al-Shahrani, 2019). Making a blueprint however does not ensure that there is alignment between it and the question paper, therefore there is a need to do analysis of the paper made (Eweda et al., 2019).
Blueprinting is seldom taught at graduate or postgraduate level and yet when a faculty joins the academic stream, they are expected to know how to make a valid and reliable question paper. Therefore, a need for faculty training on this important aspect of assessment was recognized and chosen by the participants of two batches of FAIMER participants. Mentorlearner (M-L) web sessions are being practiced in the field of medical education today and online education and mentoring is considered learner centric and useful (Sivakumar, 2017;Walsh, 2016). The aim of the online discussion was to assess the knowledge of the participants on the topic of 'blueprinting', get them to read literature on it, involve them in analysis of question papers already made in their own specialties, get a hands on experience in blueprinting a question paper and to get their feedback on the month long training course.

Methods
The online session was conducted in the period as a part of the GSMC-FAIMER Fellowship Program. ML Web Discussion on the topic of "Blueprinting" was held from 1st to 31st January 2018. There were 32 participants, of batches of year 2016 and 2017. The teaching-learning activities were held in an asynchronous manner, using Listserv online discussion forum under the guidance of faculty. Consent was taken from all participants, for all activities related to the FAIMER course. Ethical approval was not needed since this represents a secondary data analysis of an educational activity.
The activities included a combination of informative activities and hands on experience in preparation of a blueprint. The planning involved the following:

Pre-moderation Preparation
Designing a flawless module for conducting the session was a pivotal step. The learning outcomes and the plan of the activities had to be categorical outlined. Various facets of the topic of discussion were deliberated and the content of all the activities were planned suitably, with repeated inputs and discussions, literature survey and consensus among the team members. The following attributes contributed significantly in designing this session: a. Effective communication.
Proper discussion among the team members regarding designing the activities, was of highest importance. As the team members were placed in different geographical locations, chances of face-to-face discussions and personal meetings to plan sessions were negligible. WhatsApp and Google Group were used for basic asynchronous communication.
b. Defined roles and responsibilities. The roles and responsibilities of moderators, record keepers and the resource faculty were fundamentally delineated to conduct the sessions smoothly. The expectations and opportunities were understood by all team members. The intended outcome of each activity from the participants was also demarcated and explained in the instruction.
c. Proactive strategy. The designing the sessions was intentionally started well in advance so as to keep enough time in hand to collect resources related to sessions and plan activities in line with the objectives.
d. Clear goals. Under the guidance of resource faculty, the goal and learning objectives of the session were defined. The topics chosen for discussion were screened for feasibility to suit in the month's discussion schedule.
e. Vivified milieu. The platforms to float the activities were strategically chosen. The activities were designed keeping in mind the objectives, the time required, whether it ensured active participation and resulted in a suitable outcome. The whole month's session was divided into total of 7 activities as shown in Figure 1.

ML Web design and online moderation (All activities in 4 weeks)
The session was planned with the following objectives: 1. To understand various attributes of effective assessment and threats to those attributes 2. To understand the need and purpose of blueprinting in assessment 3. Weightage calculation and Blueprint preparation 4. Test paper preparation with and without blueprints in participant's own specialty The Following topics were covered during the sessions based on the objectives of the teaching-learning online activity: i. Essential attributes of an effective assessment ii.
Potential threats to an assessment iii.
Setting up one UG and PG question paper each by the participants in their subject iv.
Assessment of the question papers submitted by participants v.
Need and Purpose of Blueprinting in Assessment vi.
General Steps involved in Blueprinting vii.
Strategies to implement Blueprinting viii.
Distribution of Weightage of various content areas in Syllabus, based on available rating scales ix.
Preparation of Blueprint x.
Setting up one UG and PG question paper each by the participants in their subject based on the Blueprint prepared xi.
Assessment of the question papers submitted by participants xii.
Analysis of the improvement in knowledge regarding Blueprinting and its role in Assessment xiii.
Reflections on the learning occurred The weekly distribution of the activities conducted as a part of the online ML Web Session are described below: Week -1, Activity -1 (Pre-test) [1st to 3rd January] To assess prior knowledge of the participants about the topic of discussion a pretest was conducted. It was a designed on a Google Form Platform consisting of validated items.
Week -1, Activity -2 (Basics of Assessment and Its Various Attributes) [4th to 7th January] The objective of this activity was to make the learner mindful of the attributes of an assessment and the threats posed to it. This activity was started with one main thread and four sub-threads (α, β, γ and δ) to ensure optimum learning by utilizing shared resources. In each sub-thread the participants consisted of 8 members (4 from senior batch and 4 from junior batch). Exemplary articles on various aspects of assessment like principles of assessment, evaluating and designing assessments for medical education, reliability and validity in assessment, and threats to validity of assessment, were distributed among subthreads. The participants were asked to submit summary of the articles in the main thread. In this activity, the participants were made aware of the need of blueprinting in assessment.
Week -2, Activity -1 & 2 (Analysis of Theory Question Papers) [8th to 13th January] After making participants to do needsassessment in previous activity, they were made to study and analyze the prevailing practices in assessment in their institutions by doing a hands-on activity. The first task was to find one undergraduate and one postgraduate examination paper of participant's discipline. The second task was to analyze the question papers following the validated guide (Table 1), which was sent to them by mail. This exercise made them introspect the current practices in their institutions and gauge the actual need of blueprinting in assessment.
Week -3, Activity -1 (Blueprinting -Need, Purpose, Steps & Strategies) [14th to 17th January] The awareness of the flaws generally found in question papers and deficiencies in the existing assessment patterns was brought out by the above activity. The need of Blueprinting was well perceived and then this new activity was initiated. The responses were invited on the concept, need, purpose, steps and strategies to implement Blueprinting in Assessment. A task to find a best suitable strategy of blueprinting for their subject, sensitized them to search and study the literature available.

South-East Asian Journal of Medical Education
Vol. 14, no. 2, 2019 Week -3, Activity -2 (Preparation of After going through the literature search on Blueprinting in Assessment, this actual hands-on exercise was designed, with the objective of weightage calculation for content areas in the syllabus of participant's subject. The concept and the purpose of using available rating scales, with merits and demerits of the rating scales in a particular setting were discussed. Excel sheet format was devised for the templates for weightage calculation and blueprint preparation for exam question paper. The blueprint template included the criteria of 'must know, desirable to know and nice to know type of content areas, and selected questions on the basis of levels of cognitive domain, like recall, comprehension and application. Week -4, Activity -1 (Blueprint Preparation and Setting Up Theory Test Paper Based On Blueprint) [23rd to 28th January] After successfully assigning weightage to the course content by the participants in the previous activity, the task of blueprinting of a question paper in participants own subject and setting a test paper based on the prepared blueprint was begun. The same excel sheet format was used for blueprint preparation.
Week -4, Activity -2 (Post-Test) [29th to 31st January] To assess the changes in the level of knowledge about blueprinting in assessment after the ML Web Session, the Post-Test was shared via Moodle Online Learning Platform. The analysis of the Post-Test responses was done.
The activates were initiated in time, clear instructions and prompt responses along with poems and shared stories added to the teaching learning activities. Post discussion report was put on the Web for reinforcement and record.

Dividing the Sessions into Weeks and Two Activities per Week
The week wise distribution of activities in the ML Web Session is shown in Figure 1.

Reflections (Post moderation)
After attempting the post-test, the participants were requested to write their reflections on the whole ML Web Session.

Summarizing the ML Web Discussion
For each week, the following questions were asked in reflections: 1. What did you learn this week? 2. What activities helped you learn? 3. What activities did you find engaging? 4. What questions/ comments did you have for us? 5. How do you rate your knowledge about blueprinting? (asked in first week only) 6. How do you rate your knowledge about blueprinting? (asked in the last week)

Results
The number of participants in pre-test were 31 and in post-test were 26. They were faculty from medical, dental and physiotherapy streams of health sciences.

Analysis of Pre-Test and Post-Test Scores Comparison
The responses of the MCQ's in pre and posttest were compared and paired 't' test analysis of those who completed both (26) was done. The scores (percentage of marks obtained) showed statistically significant improvement knowledge of the technology of blueprinting, as shown in Table 2. Difference in percentage scores of individual participants is depicted in Figure 2.

Response to Online weekly tasks
In the first week 112 mails were exchanged, and 81, 84,131 in the second, third and fourth weeks respectively. There were 26 attempts made for blueprinting and finally 21 papers were made after following the method of blueprinting.

UG paper Analysis
There were 25 responses received. It was reported that 40% of the questions in the paper were long answer type. The range was from15-80%. The weightage of MCQ was 10% (range-0-35%). The comparison according to blooms taxonomy showed that 60% of the questions tested recall component of the cognitive domain. The core syllabus analysis showed that 82% were from the must know areas while 13% and 55 were from desirable to know and nice to know areas. The grading showed 68% to be good, 20% as fair and 12%as poor.

PG paper Analysis
There were 26 responses received. The reported analysis of participants showed that 50% of the questions were long answer type (range 25-100) %. The MCQ were only 2%. Most of the questions, i.e. 60% were the recall type and 74% were from the must know areas while 18% and 8% were from desirable to know and nice to know areas. The grading showed that 30.77% were good and 26.92% were poor.

Table 3: Important Points in Reflections
Reflections in verbatim 1 "I would like to congratulate the moderators who have painstakingly made the excel sheet and were very prompt in resolving any queries raised by fellows and assisting all. Kudos." 2 "The moderators of team blue printing require special appreciation for designing such engaging activities and assisting all of us timely with their excellent comments." 3 "Good selection and distribution of articles for group activity. Good questions in pretest." 4 "Excellent technical hand and use of Moodle platform for post-test." 5 "Very innovative and most effective moderation" 6 "Really extra efforts have been put by the entire team blueprinting. Moderators were just an Email away from us." 7 "Well planned activities have made understanding of the topic easier. Special thanks for the excel sheet for future use." 8 "Practical activity of selecting paper and analysing it. Never done this before. Wanted to learn it and got this opportunity. Also learnt from responses from other fellows who are already into it." 9 "Hands on experience of dissecting the question paper from Medical education angle." 10 "Good job for the accurate, user friendly Excel template and for your attention onto minute details and initiating valuable discussions. Great team work."

11
"It was one of the most exhilarating and enjoyable ML web sessions It is known how difficult and complex it is to embed the various formulas into the excel sheet and then sharing it with all. I have since junked the blues out of blueprinting." 12 "Good feedback given to each individual submission."

Reflections Analysis
The hands-on experience was appreciated. Some of the comments are mentioned in Table  3.

Discussion
The online learning experience enabled all participants to understand the intricacies of making a good theory paper, based on the concept of blueprinting which is the key to an effective assessment

Figure 2: Comparison of Percentage Scores of Pre-Test and Post-Test
The blue print procedure adopted was similar to a study done in the subject of community medicine (Gujarathi et al., 2015). When Analysis of the papers in different subjects was done, it was found that all subtopics and all domains of a subject were not represented. Most papers were testing the recall component of cognitive domain and questions were long answer type. This is similar to a study involving analysis of 40 Anatomy papers. In this study it was reported that proper weightage was not given to all subdivisions of anatomy (Garg et al, 2013). In a previous study in Pharmacology a similar trend was reported. (Khuteta & Saurabh, 2017).
A similar study was carried out in the subject of community medicine where a scoring system of papers was used and it was found that 21% of the papers were good and 42% were fair (Chauhan,2019). The authors reported that the paper was not an appropriate representation of syllabus. Similarly, in a study done on microbiology papers, both over and underrepresentation of many topics was reported (Gill & Sen, 2018). Other studies have also suggested that blueprinting of assessment methods not only increases their reliability and validity, but helps in student's performance result and satisfaction. (Ahmad & Hamed, 2014).
A future scope of our study is to circulate the papers prepared as a result of blueprinting, in the open forum of more faculty who specialize in the subject and to get their feedback. This would increase the validity of these papers and also indirectly propagate the concept of blueprinting. A feedback has been taken in a previous study where blueprinting was done in the subject of psychiatry and the faculty felt that in depth knowledge was tested and the questions were aligned with the objectives (Goyal et al., 2017). A checklist can also be made for this purpose as has been previously suggested (Reddy, 2017). In addition, it would be a good idea to take the students perception regarding the prepared papers and find out whether they are able to perceive a difference between these and papers made earlier. This would help in the students' education, enable them to understand how papers are made and it provide a feedback from our stakeholders (Salih et al, 2018).
Our study reported that online moderation and discussion during FAIMER session resulted in an overall increase in the knowledge of the participants. Participants were sensitized about the utility of blueprints for setting question papers. It was perceived by the participants that blueprinting makes assessment more congruent with the objectives, content area and the curriculum, and can be implemented to improve reliability and content validity of the assessment. The online learning experience enabled all participants to understand the intricacies of making a good theory question paper, based on the blueprint.

South-East Asian Journal of Medical Education
Vol. 13, no. 2, 2019

Conclusion
This ML Web Session could successfully achieve the learning objectives because of the strategic preparation and tremendous support from the motivated participants. Not only the resources but also the peer responses contribute to the total learning of the participants. The online learning experience enabled all participants to understand the intricacies of making a good theory question paper, analyze and make a question paper based on the 'blueprint' prepared by them.

Key Messages
Important topics like blueprinting can be learnt by online web-based discussions. This learning tool not only increases knowledge but can be used to give a hands-on experience and is an important tool for faculty development programs.