IMPLEMENTATION AND EVALUATION OF E-LEARNING PROGRAM: A CASE STUDY OF UNIVERSITY TUN ABDUL RAZAK (UNITAR)
MALAYSIA
BY
VARFEE SOKO DUKULY
INTERNATIONAL ISLAMIC UNIVERSITY MALAYSIA
2007
"The root of education is bitter, but the fruits are
sweet"
IMPLEMENTATION AND EVALUATION OF E-LEARNING PROGRAM: A CASE STUDY OF UNIVERSITY TUN ABDUL RAZAK (UNITAR)
MALAYSIA
BY
VARFEE SOKO DUKULY
A dissertation submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy in
Education
Institute of Education
International Islamic University Malaysia
MAY 2007
A B S T R A C T
The fluid nature of the Internet as a communication tool gives rise to online learning.
Colleges and universities are attempting to capitalize on the instant access mindset of people who use the Internet. Success of this new mode of learning cannot be assumed without an evaluation to determine its effectiveness and worthiness. This case study evaluates the e-learning implementation program of the University Tun Abdul Razak (UNITAE.) in order to identify its strengths and weaknesses and possible benefits. The researcher employed the survey method, supported by interviews. Van Slyke et al.'s evaluation model consisting of learner, institution, course and distance-learning characteristics was utilized. Employing stratified random sampling technique.
Questionnaires were used to gather the data from 485 students, 63 administrative staff, and 63 lecturers. Descriptive statistics were used to analyze lecturers' and administrators' data, while multiple regression analysis was employed for student data. In-depth interviews were also conducted with some of the participants. The results of the multiple regression analysis indicated that course information (the course outline), learning flexibility, gender, and year of study (predictors) were principal determinants of student's e-learning success. This showed that the relationships between knowledge as a criterion variable and the predictors were statistically significant. However, the findings showed that students' CGPA was not a significant determinant of e-learning success. Moreover, Confirmatory Factor Analysis (CFA) of student's data provided evidence of a good model as there was no negative variance. Goodness of fit indices exceeded the recommended critical value, with GFT at .93; AGFI .90; IFI .95; TLI .94; NFI .93; CFI .95; and RMSEA .07.
l i
APPROVAL PAGE
This dissertation of Varfee Soko Dukuly has been approved by the following:
Ahmad Marzuki HJ Zainuddin Supervisor
Joharry Othman Co-Supervisor
Daing Zaidah Binti Daing Ibrahim Co-Supervisor
Joharry Othman Internal Examiner
Azizan Asmuni External Examiner
Rahim Hj Ismail Chairman
iv
D E C L A R A T I O N
I hereby declare that this dissertation is the result of my own investigations, except where otherwise stated. I also declare that it has not been previously or concurrently submitted as a whole for any other degrees at HUM or any other institutions.
Varfee Soko Dukuly
Signature Date
v
INTERNATIONAL ISLAMIC UNIVERSITY MALAYSIA
DECLARATION OF COPYRIGHT AND AFFIRMATION OF FAIR USE OF UNPUBLISHED RESEARCH
Copyright © 2007 by Varfee Soko Dukuly. All rights reserved.
IMPLEMENTATION AND EVALUATION OF E-LEARNING PROGRAM:
A CASE STUDY OF ENIVERSITY TUN ABDUL RAZAK (UNITAR) MALAYSIA
No part of this unpublished research may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without prior written permission of the copyright holder except as provided below.
1. Any material contained in or derived from this unpublished research may only be used by others in their writing with due acknowledgement 2. The HUM or its library will have the right to make and transmit copies
(print or electronic) for institutional and academic purpose
3. The HUM library will have the right to make, store in a retrieval system and supply copies of this unpublished research if requested by other universities and research libraries.
Affirmed by Varfee Soko Dukuly
Signature Date
VI
I dedicate this dissertation to my parents. Without their patience, understanding, support, and most of all love, the completion of this work would not have been
possible.
vu
ACKNOWLEDGEMENTS
I would like to express the deepest appreciation to my committee chair, Assoc.
Professor Dr. Ahmad Marzuki HJ Zainuddin, who has the attitude and the substance of thoughtfulness: he continually and convincingly conveyed a spirit of adventure in regard to research and scholarship. Without his guidance and persistent help this dissertation would not have been possible. I would like to thank my committee members, Assoc Professor Dr. Daing Zaidah Binti Daing Ibrahim and Asst. Professor Dr. Joharry Othman for their direction, assistance, guidance, and reading and proof - reading the drafts. Special thanks should be given to my Ph.D student colleagues for sharing their experiences in pursuing doctoral research and who helped me in many ways. My sincere gratitude is also due to administrative staffs of UNITAE. & the Institute of Education (HUM) for their smooth co-operation that led to the successful completion of this dissertation. Words alone cannot express the thanks I owe to my beloved parents, Mr. Soko Dukuly and Mrs. Massa Saibah-Dukuly, for their encouragement, motivation, patience and prayers, and sacrifices, without which the completion of this dissertation would have been unthinkable. Finally, I would like to convey my special thanks and appreciation to my dear Uncle Mr. Mohammed Dukuly, my bothers and sisters, and every individual member of my relatives for their prayers and best wishes.
vui
TABLE OF CONTENTS
Abstract ii Abstract in Arabic iii
Approval Page iv Declaration Page v Copyright Page vi Dedication vii Acknowledgments x
List of Tables xi List of Figures xiv CHAPTER ONE: INTRODUCTION 1
Background of the Study 1
Evaluation Framework 5 Statement of the Problem 6
The Purpose of the Study 8 Research Questions 8 Justification of the Study 9
Significance of the Study 11 Limitations of the Study 12 Definition of Terms 12
CHAPTER TWO: REVIEW OF LITERATURE 15
Introduction 15 Emergence and Development of Distance Education 15
Distance Education: The Malaysian Perspective 16 Implementation Problem of Distance Learning in the Past and Present 19
Summary of the Problems Faced in E-learning Implementation 22 Evolution of E-]^aming and the Importance of the Internet as a Earning Tool.. 22
The Concept of Evaluation 25 Kinds of Evaluation 27 Importance of Evaluation 29 Theoretical Framework of E-learning Evaluation 32
E-learning Clients' Perceptions Regarding their Needs and Expectations... 36
CHAPTER THREE: METHODOLOGY 38
Introduction 38 Site of the Study 38
Respondents of the Study 39 Student Respondents 39 Lecturer Respondents 40 Administrator Respondents 40
Population and Sampling 41 Sampling Procedures 42
IX
Determining the Sample Size 43
Instrumentation 44 Realibility and Validity/ Pilot Testing 44
Results of the Pilot Study 49 Test of Assumptions for Multivariate Analysis 56 Distributional Assumptions of the Data 57
Model Assessment and Modification 57 Assumptions of Normality for Administrators' and Lecturers' Data 60
Interview Methods 60 Data Collection Procedures 62
CHAPTER FOUR: DATA ANALYSIS AND PRESENTATION OF RESULTS 66
Introduction 66 Demographic Variables of Students 66
Demographic Variables of Lecturers 68 Demographic Variables of Administrators 70 Descriptive Analysis for Administrators'Questionnaire 70
The Types of Technologies Used to Support or Interact with Students 76
Descriptive Analysis for Lecturers' Questionnaire 78 Results of Students' E-Learning Evaluation Model Goodness of Fit 85
Assessment Reliability 86 Multiple Regression Analysis 86 Strengths and Weaknesses of UNITAR's E-Learning Program 88
Participant Observations 107 Chapter Summary 108 CHAPTERFIVE: DISCUSSION, CONCLUSION AND RECOMMENDATIONS... 110
Findings 110 Conclusion 124 Implications of the Study 126
Implications of the Findings for Educational Practice 127
Recommendations for Future Research 118
BIBLIOGRAPHY 131 APPENDIX A: STUDENT SELF-ASSESSMENT 138
APPENDIX B: INSTRUCTOR SELF-ASSESSMENT 142 APPENDIX C: ADMINISTRATOR SELF-ASSESSMENT 148 APPENDIX D: ANALYSIS OF INTERVIEW QUESTIONS 151 APPENDIX E: CRONBACH'S ALPHA AND INTER-ITEM CORRELATIONS OF 170
ADMINISTRATORS* EVALUATION INSTRUMENT
APPENDIX F: CRONBACH'S ALPHA AND INTER-ITEM CORRELATIONS OF 171 LECTURERS* EVALUATION INSTRUMENT
APPENDIX G: QUALITATIVE RESEARCH QUESTIONS FOR THE 172 INTERVIEW
x
LIST OF TABLES
Table No. Page No.
3.1 Sample Size 43 3.2 Five-Factor Matrix- for Students' E-learning Evaluation Instrument 53
3.3 Correlation Matrix- for Students' E-learning Evaluation Instrument 55 4.1 Summary of the Distribution of Students' Background Characteristics 68 4.2 Summary of the Distribution of Lecturers' Background Characteristics 70 4.3 Summary of the Distribution of Administrators' Background
Characteristics
4.8 Summary Means and Standard Deviation of the Predictors and Criterion Variables
72
4.4 Administrators' Questionnaire 75 4.5 Summary of the Distribution of Technology Used 78
4.6 Lecturers' Questionnaire 83
4.7 Summary Statistics of the Model-Fit Measurement 85 86
4.9 Model Summary 86 4.10 Correlations among the Variables 87
4.11 Coefficients 87
x i
LIST OF FIGURES
Figure No. Page No.
1.1 Adapted from Van Slyke et al. (1998) model of evaluation 34 . . Confirmatory Factor Analysis on students' e-learning R .
evaluation
xn
CHAPTER ONE INTRODUCTION
BACKGROUND OF THE STUDY
The very fluid nature of the Internet as a communication tool has given rise to online learning program, which today has invaded the realms of the bricks and mortar classrooms of colleges and universities worldwide. But in the midst of an institutional rush toward the adoption of this new educational technology, higher learning institutions have begun to raise questions about the merit and worth of this institutional initiative. As e-learning technologies continue to evolve, evaluation studies have likewise multiplied and have attracted researchers, theorists and practitioners in e-learning to continue in their quest to discover and evaluate the promised diversions of this educational practice.
The way in which individuals live, work and learn has undergone significant changes due to innovations through the use of information and communication technologies (ICT). This new method of delivering information and instruction has removed the obstacles of time and geographical boundaries in education and training.
This transition from traditional classroom to online learning did not come all of sudden, but rather it has evolved from one stage to another. It took a while for the term e-learning to come about, first, around 1995 it was called "Internet based Training," then "Web-based Training" (to distinguish between Inter and Intra-net), then "Online Learning" and finally e-learning, adopting the vogue use of "e" during the dot com boom. The "e" breakthrough enables many industries to raise hundreds of millions of dollars from venture capitalists who would invest in any industry that started with this magic letter (E-learning glossary, 2003).
1
Dongsong (2002) believes that the Internet and the revolution of the learning technologies have changed the way training and education are delivered. This new method of delivering information and instruction has removed the obstacles of time and geographical boundaries in education and training. In addition, this has reinvigorated the concept of "lifelong learning," which does not require employees or individuals to leave their respective places of work or homes or countries. For example, an organization may need to train employees without having them to leave work, and students who missed a lecture may be able to listen to it later at their convenience. These kinds of specialized needs require knowledge acquisition that is personalized, available on-demand, and not restricted to time and place (Dongsong, 2002).
Unlike in traditional education, in lifelong learning the roles of instructor, student, and curriculum are constantly changing. That is, teaching and learning are no longer restricted to instructor-led classrooms, but rather shift toward or become learner-centered by stressing the need for relevancy, personalization (e.g., learning according to an individual's interests, background and style) and flexibility.
Therefore, this new learning method, in addition to becoming personalized, has also become more convenient (Adam et al., 1997).
This new development in conducting learning and training over a distance has created a renaissance in distance education. For decades, institutions of higher learning, big and small, have experimented with various information and communication technologies (ICT) to expand and, in some cases, replace traditional classroom education. Now, many institutions of higher learning are embracing e- learning as a vehicle to capture the vast distance-learning market. For example, in the United States and other countries, a number of colleges, universities and business
2
schools are adopting the e-learning approach in their institutions. These include universities such as Stanford, Phoenix, Massachusetts Institute of Technology, Columbia and Wharton School of Business, just to name a few. In addition to increasing enrollment and cost saving, distance learning also provides higher education institutions the opportunities to create and disperse knowledge where it would not otherwise be available (Emaad & Savitsky, 2001).
In the current knowledge-based economy, which is characterized by industrial change, globalization, increased intensive competition, knowledge sharing and transferring, learning is perhaps the most crucial activity. Globalization requires new methods of delivering education and training, partly to enhance traditional methods of knowledge acquisition and to convey new skills and tools. Moreover, the traditional context of learning is experiencing a radical change. Most institutions of higher learning are going for online learning instead of the face-to-face method (Dongsong, 2002).
Since 1990, a dramatic shift to Internet-based learning has greatly expanded the world of open and distance learning, leading to what has been referred to as electronic learning. According to Bristol University (2003), E-learning is the use of electronic technology and media to deliver support and enhance teaching, learning and assessment. It includes elements of communication within and between communities of learners and teachers, as well as provision of online content, which may be locally generated or developed elsewhere. The elements of communication may include the use of computers by way of CD-ROM, Internet, or Intranet to provide content relevant to learning.
A broad definition of e-learning is using technology to deliver learning and training programs. It is typically used to describe media such as CD-ROM, Internet,
3
Intranet, wireless and mobile learning. Some include Knowledge Management as a form of e-learning (E-learning glossary, 2003).
Electronic learning offers methods such as audio and videotapes, and the more recognized web-based training or computer-aided instruction also known as online courses. These new ways of teaching and learning are decreasing the traditional ways of teaching (Soekartawi et al., 2002). Rosenberg (2001) describes e-learning as the use of internet technologies to deliver a broad array of solutions that enhance knowledge and performance. In fact, he believes that e-learning is a form of distance learning, but distance learning is not e-learning (p. 20).
The current interest of e-learning comes as a realization that universities will save money by delivering education to students that are unable to attend classes because of time or geographical distance. For example, places such as China, Indonesia, and South American countries like Brazil and Argentina have all begun to use distance-learning techniques to reach those that would by any other means be unreachable (Valentine, 2002).
Malaysia, a country in Southeast Asia that has experienced a tremendous economic growth in the past decades, sees distance education as an alternative means of providing knowledge, skills and training to its workforce. This has influenced the educational scenario with far reaching effects for the education sector, and in particular, the higher education segment. One of the initiatives of the Malaysian Government is the establishment of University Tun Abdul Razak (UNITAR), as an e- learning private institution of higher learning to assist government efforts in providing higher education and training, and retraining the workforce with greater efficiency to fulfill the manpower needs of the nation. Being the country's first virtual university, UNITAR fulfils a niche market to provide higher education to many students
4
particularly to adult learners as it decreases the need to be present on campus. The virtual university teaching and learning model offered by UNITAR is designed around the use of interactive multimedia courseware and heavy use of an intranet designed for faculty use in teaching and learning.
The teaching and learning models utilized in UNITAR are the combination of face-to-face (student-lecturer) interaction and futuristic online and multimedia interfaces. Both synchronous and asynchronous modes of communication are employed. There are a number of forces behind the growth of e-learning in Malaysia these include: (1) the need for a competent workforce and professionals who are information technology savvy; (2) lifelong learning; (3) the need to create equitable educational access and services for all. Considering these reasons and the huge financial cost of establishing a conventional university, the Malaysian Government is supportive in facilitating this modern way of providing teaching and learning.
It is believed that the e-learning concept is gaining popularity day by day and this reflects in the growth of e-learning institutions of higher education in many countries, including Malaysia. The potential promise this new system of learning has is that it will reduce costs and make education relatively affordable for many others.
However, institutions undertaking such initiatives need to be properly evaluated to assess whether the objectives for which they were established are achieved.
EVALUATION FRAMEWORK
Evaluation involves quantitative and qualitative judgments about the value of the material and its ability to achieve its given purpose. Judgments can be made using logic and consistency as a guide (internal criteria) or with reference to other works (external criteria) (Belanger & Jordan, 2000 pp.17) The idea that both formative and
5
summative evaluation may be carried out by an internal evaluator, external evaluator or a combination of the two, has gained acceptance over the years (Scriven, 1980).
According to Lewy (1990), formative evaluation aims to improve a program while it is still in progress, when development and implementation of materials and techniques are still taking place. It provides feedback to both learners and teachers about the success or failure of a program. The internal evaluation is carried out by faculty members, students or administrators who are implementing the educational program.
On the other hand, summative evaluation is conducted by an external body without interaction with the internal body (Lewy, 1990). A framework for distance education evaluation proposed by Van Slyke, Kittner and Belanger (1998) suggests that multiple variables must be taken into consideration concurrently when looking at their effects on distance learning. The researcher of this study chose to employ the framework proposed by Van Slyke because of its comprehensiveness. In other words, this model is appropriate because it can compare different variables and their effects on the implementation of e-learning.. This includes the determinants of success (precedent variables) and evaluation criteria (outcome variables). The determinants of success can be grouped into four categories of variables: (a) institutional characteristics; (b) learner characteristics; (c) course characteristics; and (d) distance learning characteristics. The outcome variables include learner, instructor, society and institution.
STATEMENT OF THE PROBLEM
As e-learning technologies continue to evolve, evaluation studies have likewise multiplied and have attracted researchers, theorists and practitioners in e-learning to continue in their quest to find out and explore the promise of this educational practice.
6
Despite the popularity and growth in e-learning institutions of higher education, many things are not understood about this new way of teaching and learning. It is therefore, important to know the perceptions of students, faculty members, administrators and technical support staff about the implementation of the e-learning programs. Their opinions could serve as vital tools which could help e-learning implementers to design and prepare online courses, to provide educators with possible solutions, to conduct relevant research, to assist students in achieving meaningful and successful outcomes and finally, to ensure return on their investment.
E-learning has numerous benefits, such as offering learners useful learning materials, and providing a series of modules and assessments that guide learners through the learning process. Hartly & Robertson (2000) mention that numerous studies in the literature showed the importance and benefits gained from using e- learning in schools and higher education institutes. Some of these studies stated that education is fundamentally concerned with access to resources and communication.
This therefore, means researchers should investigate and find out whether students are benefiting from the e-learning environment. Moreover, implementing e-learning requires trained staff, that can integrate learning, educational design, and the use of the technological devices and tools (Mason, 2000). E-learning also requires strategic planning which this requires attention to both content and process strategies.
Technological infrastructures have to be in place and the human resources that are required. Implementation also requires quality assurance procedures as well as regular surveying of learners' access to technology, such as laptops and the effectiveness of technology based-teaching methods, such as the Internet (Jamlan, 2003). So far these studies are mainly Western-based research. No major study yet about UNITAR that has sought the opinions and perceptions of faculty members, students, and
7
administrators concurrently on the implementation of its e-learning program has yet been done.
THE PURPOSE OF THE STUDY
The main purpose of this study, therefore, was to conduct evaluations of UNITAR's e- learning implementation program to determine the opinions of faculty members, students, and administrators on the successes, weaknesses, and opportunities for improvement. In addition, lecturers' students' and administrators' responses may contribute to the achievement of the University's objectives, and lead to further improvement in its e-learning implementation program. Results of this study may also contribute in future efforts to implement e-learning at UNITAR. The study tries to answer the following questions.
RESEARCH QUESTIONS
There are three sets of questionnaires which were designed to elicit the perceptions and opinions of students, lecturers, and administrators on the e-learning implementation programs at UNITAR. The six research questions are as follows:
1. How do students view the benefits of enrolling in e-learning programs?
2. What are the objectives, delivery mechanisms and support system of the e- learning programs?
3. How are the effectiveness and characteristics of the e-learning programs determined?
4. What are the predictors of student e-learning knowledge?
5. How can e-learning instrument be psychometrically validated?
6. What are the strengths and weaknesses of the e-learning programs?
8
JUSTIFICATION OF THE STUDY
According to Kinley (2001), today's distance education has dramatically shifted toward network-based technologies in general and Internet-based delivery in particular. This new method of teaching and learning requires an assessment and evaluation of its implementation process in order to determine the success of the implementation program. This will enable an institution to assess its strengths or weaknesses and the opportunities for improvement.
Philips (1991) acknowledges that evaluation is undertaken for several purposes such as (1) to improve human resource development, and (2) to decide whether or not to continue with the same style of practice. However, most experts believe that evaluation is a systematic process intended to determine the worth, value or meaning of something (Philips, 1991).
Rossi & Freeman (1993) see evaluation as integral to program development.
Evaluation determines the worth of ongoing programs and the efficiency of modifying the programs, assesses the utility of new programs, improves program management, and satisfies accountability requirements. Maimunah (1990) believes that the importance of evaluation in Malaysia can be seen in the establishment of evaluation agencies and units in public and private organizations. In fact, the Malaysian government has mandated that every public funding program be evaluated. As a Private entity, UNITAR can emulate the example of public organizations by evaluating its activities. In evaluation the use of questionnaires to elicit student opinions of their training and educational courses is widespread.
UNITAR was chosen firstly because an evaluation of such an institution is one way of measuring its past achievements, objectives, and long-term prospects and viability in providing e-learning to its clients locally and internationally. Institutions
9
of higher learning that are hoping to remain competitive and sustainable must evaluation their programs to identify their strengths and weaknesses. Secondly, as far as the literature review is concerned, there have not been major evaluation studies of an e-learning program in the UNITAE. context.
Jenny et al. (2003) admit that the growth of e-learning is accelerating, but what is known about these innovative approaches to training has been limited by the shortage of scientifically credible evaluation. Although recent attention has increased e4earning evaluation, the current research base for evaluating e-learning is inadequate (Jenny et al., 2003). Thirdly, this study will provide both theoretical and practical benefits. Theoretically, it will serve as a source of reference for future researchers in the domain. Practically, the findings of the study will give policy makers some ideas that can help them to make sound decisions on e-learning implementation. Finally, due to the huge initial cost of implementing e-learning programs, it is important to conduct evaluation studies.
According to Karl and Catherine (2001), the developments in communication and information technology and in web-based instruction continue unabated. Although the new technologies offer great promise, important issues need to be considered if this promise is to be realized. It would be helpful for institutions that wish to take advantage of these developments to firstly know what experience their staff members currently have, and what their attitudes toward web-based technology are. Secondly, the institutions concerned need to know what their staff members consider as success or failure in the implementation using these modern information technologies, and most importantly what stakeholders expectations are from the new ways of teaching and learning (Karl & Catherine, 2001).
The goal of most evaluations is to provide "useful feedback" to a variety of audiences including sponsors, donors, client-groups, administrators, staff and other
10
