Development and Assessment of an Online Computer-Assisted Instruction Method

Objectives : The purposes of this study were to develop an online computer-assisted instruction method and assess its efficiency in terms of propriety, feasibility, accuracy and utility standards. Method : The sample consisted of third-year medical students and critical care nurses, who volunteered to join the study. Research tools included online activities and student assessments, open-ended questionnaires and students' record profiles. Data was analysed using the percentage, mean, standard deviation, correlation coefficient and t-test. Results: A total of 83 out of 116 third-year medical students (71.55%) and 35 out of 56 critical care nurses (62.50%) registered through the website, and completed the learning courseware. Results of the system efficiency assessment through the tests and open-ended questionnaires showed that it was highly appropriate for use in communicating for learning; it was also found that the system was highly feasible in stimulating learners to continuously plan for self-directed learning and for system accuracy. It appeared that the system had objectivity, construct validity, concurrent validity, and internal reliability; and for system utility, it was found that learners gained high learning achievement and growth scores. Online record profiles revealed that 63 medical students and 35 nurses appreciated the significance of the developed online learning system as fair to full. Conclusion: The online computer-assisted instruction method was highly efficient. Through self-directed learning, the students were stimulated to apply problem-solving strategies, individual help seeking strategies, and self-regulation, as well as develop critical thinking with discretion for self-assessment and continue learning with a sense of self-actualization and self-esteem.


Introduction
Medical subject content not formally included in regular class hours is increasing. The diversity and complexity of current health information require medical personnel to constantly strive for self-directed learning even after graduation.
The result is a need for more active search for knowledge, depending on their enthusiasm, interest, and time.
To overcome this learning problem, we planned to focus on promoting an online computer-assisted instruction (CAI). We chose arterial blood gases (ABG) as a prototype subject for third-year medical students and critical care nurses. This was an educational challenge since the medical students had no previous clinical background knowledge while nurses found it hard to comprehend ABG and its applications. The system was based upon the online problem-based learning principles (Peterson, 1997). Our hypothesis was that through this highly efficient method, active participants would retain the content while showing high learning achievement.

Materials and Methods
The Institutional Review Board of Mahidol University approved this study. We launched a communication channel with the students at www.nmac.in.th and requested qualified students to register and express their intention to join this research project. The program was interactive, with personal files for each student on the subject along with other details such as code, password, name, date-month-year, test scores, and log-in periods and time. Students could, on their own, plan their learning of the courseware and they were free to record and copy the articles on the website. For three weeks, there was no log-in limit from any place with connectivity available around the clock.
We developed a set of learning courseware for the specified topic i.e. activities and student assessments related to ABG. The activities covered specified topics whereas the assessments probed the weaknesses of students' understanding, and assessed their learning skills in analytical thinking of patients' problems under assorted circumstances to be faced in clinical practice (Treagust, 1988;Tan, 2003;Heh, 2008;Chittleborough, 2007). The learning sources addressed their misconceptions while avoiding provision of direct answers immediately after each test. Instead they linked the students to a search for the well-prepared content, which contained complete information. In other words, to get the proper answer for each item of the tests, students needed to study the content appearing on the lessons in detail. Each set of courseware contained 3 levels: beginner, intermediate, and advanced. Students were required to complete the tests for each level at least once with a minimum score of 80% before advancing to the next level.
Target groups were third year medical students who had not formally performed patient care, and nurses who had worked in critical care units. Participation was voluntary without any honorarium, and their perceived benefit was clinically applicable knowledge. After registration, all participants first underwent a pre-test. They then studied the website at their leisure for the content designed through activities/ student assessments.
Scores were provided immediately on the completion of each learning courseware and their individual profiles were updated. Each student was free to save, copy or print any information appearing on the website, learn the subject as preferred, and do additional research. Those feeling overwhelmed by the conditions were allowed to withdraw from the project at any time. Once they had passed certain courseware, they were free to review all learning courseware as often as they wished. Those who completed their learning within the specified three weeks then responded to open-ended questions and gave their comments before undergoing a post-test. During this research, no tutorial or additional class notes were given on the subject. We answered any queries about the project procedure through a specified email address, www.link.nmac@gmail.com, while keeping the data about the individual participants confidential.

Validation and reliability of the student assessments
The tests underwent verification for content validity and had their index of item objective congruence (IOC) determined by four clinical experts at the faculty, each with at least 10 years of teaching experience. Deciding whether the tests and behavioural objectives aligned, they set the IOC at 0.79 for pre-tests and 0.73 for post-tests. In addition, 32 nursing students and 21 first year residents in anaesthesiology volunteered to undergo system testing including the operating units, the objectivity of the tools and criterionreferenced tool quality.
The questionnaire scoring was done using a Likert scale that had undergone quality analysis and verification for conciseness, accuracy, and clarity, as well as content and structural validity by three doctorate educational experts with at least a decade of teaching experience. The questionnaires' IOC was 0.79 and had a reliability of 0.82 (Cronbach's Alpha) among the 32 nursing students.

Information analysis
Scores from all tests were analysed by individual question, mean, and standard deviation. Statistical significance was set at .05 (). Post-test scores of 50-60% were set as ‗pass', as similar scores had been previously obtained from the application of an open and distance learning method (Perraton, 2007). Growth of knowledge was analysed through a comparison of pre-test and post-test scores by the formula below (Kanjanawasee, 1989).
Where: S = Percent growth of knowledge F = full test score X = pre-test score Y = post-test score Information from questionnaires yielded means and standard deviation values along with content analysis.
Efficiency assessment including proprietary, feasibility and utility standards were based on an analysis of questionnaires. The accuracy standard was assessed through a comparison of criterion-related validity and concurrent validity of the tests (Stufflebeam, 1971).

Results
One hundred and sixteen preclinical students and 58 critical care nurses registered in the website and took the pre-test. However, only 83 medical students (71.55%) and 35 nurses (62.5%) participated fully in the three-week study and therefore qualified as the final study group.
The system operating units were flexible to the students' interactive behaviour, instruction responsiveness, and network connection. Access to the system through the website provided ready access to the information, was user-friendly, and stable. The tools were objective, covered complete details, and were clear, with elements of the subject content in proper order. Feasibility standard assessment, also based on the same questionnaires, resulted in average scores of 4.68 and 4.70 (highly efficient).
As evident in the website access records, 116 preclinical students logged in to the ABG learning courseware a total of 130 times, with 120 log-ins by 110 students to the first assessment activity/test, 101 log-ins by 98 students to the second, and 91 log-ins by 83 students to the third with scoring more than 80%. For the 56 nurses, a total of 66 log-ins were recorded, with 57 log-ins by 56 nurses to the first assessment activity/test, 45 log-ins by 41 nurses to the second, and 37 log-ins by 35 nurses to the third.

Medical students
The 83 preclinical students consisted of 21 males and 62 females, all aged between 20-25 years. As for the sources used by habitual self-learners, 30 reported that they usually listened to faculty members, more senior students, and classmates; 26 read texts, journals and publications. As for the media used, 12 watched documentaries and demonstrations, nine relied on CD, VCD, and DVD, and 19 relied on the internet, email, web browser, and SMS. As for the time most used for our study, three students chose the morning hours, one the afternoon hours, 12 the evening hours, 41 the night time and 36 indicated that they had no preference. With regards to the total time spent, six spent up to five hours, seven up to six hours, 11 up to seven hours, 45 up to eight hours, 14 more than eight hours. Should this activity become part of the normal curriculum, seven students estimated it would take up to five hours, five estimated up to six hours, two estimated up to seven hours, six estimated up to eight hours, and ten estimated it would take them more than eight hours. Eight students rated this activity under standard, 12 said it was fair, 11 said it was moderately popular, 23 said it was popular, and 29 said it was highly popular. Students suggested that sample questions about real-life patient circumstances should be added to stimulate learning and analytical thinking.

Nurses
Of the 35 nurses, all females, three were aged 26-30, nineteen were aged 31-35, eleven were aged 36-40 and two were aged 41-45. Five had between 1-5 years of experience, 18 had 11-15 years of experience and two had 16-20 year of experience. As for the sources reported by habitual self-learners, 26 reported that they usually listened to faculty members, more senior students, and classmates; 4 read 32

South East Asian Journal of Medical Education
Vol. 8 no.2, 2014 texts, journals, and publications. As for the media used, 14 watched documentaries and demonstrations, 35 relied on CD, VCD, and DVD, and 5 relied on the internet, email, web browser, and SMS. As for the time most used for our study, seven nurses chose the evening hours, 11 the night time and 17 indicated that they had no preference. With regards to the total time spent, four spent up to five hours, 31 more than eight hours. Should this activity become part of the normal curriculum, all 35 nurses estimated it would take them more than eight hours. Seven nurses reported this activity as moderately popular, seven said it was popular, and 21 said it was highly popular.
Accuracy standard: The quality assessment through a comparison of the tests concerning criterion-related validity for difficulty, sensitivity, reliability, content and concurrent validity as well as learning achievement scores (Table 1).

Discussion
On the account of their records of online registration and their lack of scores against expectation, 33 preclinical students and 18 critical care nurses were considered to have dropped out of the study. The remaining 83 students (71.55%) and 35 nurses (62.5%) worked laboriously to show their intention of self-learning the new subject.
Despite the differences in participants' knowledge, their learning achievement was satisfactory, which implied that the criterionreferenced assessment under this study was able to categorize student performance levels for the selected subjects (Jacobson, 2008). This was further confirmed by the capacity to distinguish performance into proficiency of ABG by the post-test scores. It implied that the system provided efficient steps of learning in which students could apply their own methods of study. It can be used by all students, particularly those of the same professional line but with different background knowledge. The

South East Asian Journal of Medical Education
Vol. 8 no.2, 2014 33 system will form a key mechanism for today's learning, which is time-bounded. . This work was supported by the several conceptual frameworks relevant to education. First, it involved an experiential learning approach under a concept laid down by John Dewey in 1940. The chosen subject matter represented the body of knowledge for system development, and challenged preclinical students' learning behaviour, and additionally represented matters of future interest, as they were about to start clinical work. Second, we applied mastery learning as defined by Washburne, (cited by Block, 1971) under the premise that, despite different background knowledge, people could learn with equal success and express their mastery of core knowledge of the subjects. Finally, we applied empowerment evaluation defined by Fetterman in 1996, which took into account stakeholder engagement; the system promoted the learner's role as the decisionmaker and planner, self-assessment designer, and the implementer of the discovered information. The researchers played the role of a ‗good friend', facilitating learning and stimulating referral to databases, while fostering a context conducive to learning.
In our view, the success of the system depended largely on the enthusiasm of the participants, notably their self-learning skills. Learners should be convinced that only they are accountable for what they have learned while being committed to the learning methods and yearning to learn new knowledge. In 2007, Zsiga confirmed the view that those who are able to conduct self-learning tended to be strategic thinkers. Efficient leaders therefore need to be strategic thinkers with high selflearning skills.
It was found that most participants preferred to study in the evening hours through midnight, probably because they felt free from their daily ward rounds and less self-conscious in the privacy of self-learning tuition and online communication. This finding differed from those of Healy (2005) where students favoured day-time for self-learning over the rest of the day.
The system yielded personalized feedback, which did not simply show the actual answers once a test was finished, but instead revealed the answers by linking it to a specific lesson page, where one could find detailed clues. Through this means, students were supported in content understanding, problem-solving strategies, retention of knowledge, and learning success. This idea agrees with a study performed by Tzu-Hua (2008).
Psychometric analysis indicated that the difficulty, discrimination, content, construct, concurrent validity, and the reliability of the tests under this study could be classified as good. This was quite acceptable in a diagnostic and feedback-providing system to reflect the accuracy standard of evaluation and confirm the reliability of measurement. This has been previously confirmed by a study conducted by Ciofalo (2009). The quality of the tests appeared to be valuable tools for formative assessment of the students and charting their course of self-directed learning, the central concept of the study, particularly the concurrent validity, which reflected realtime evaluation of the test.
Torre (2004) and McNulty (2006) identified satisfaction factors in computer based instruction, including user-friendliness, selfdirectedness, technological efficiency, interactivity of learning materials, timely personal feedback, and confidence in overcoming challenging assignments. The ‗anytime, anywhere' convenience of online learningwherever there is availability of the internet and of a home, office or dormitory computer play a crucial role for students. In addition, the developed process must represent a challenging and inspiring pathway for the participants while ensuring enthusiasm, discipline, self-respect and earnest dedication.

Conclusion
The online computer assisted instruction had efficiency in terms of propriety, feasibility, accuracy and utility standard. Through selfdirected learning, the students were stimulated to apply problem-solving strategies, individual help-seeking strategies, and self-regulation, as well as develop critical thinking with discretion for self-assessment and keeping on learning with a sense of self-actualization and selfesteem in education.

Acknowledgments
We acknowledge the voluntary help of nurse students and first-year residents in anaesthesiology, Faculty of Medicine Siriraj Hospital, Mahidol University as well as thirdyear medical students and critical care nurses throughout the country who fully participated in the project.