An Exploratory Study of Adoption of Course Management Software and Accompanying Instructional Changes by Faculty in the Liberal Arts and Sciences
Teresa Yohon, Don Zimmerman and Leslie Keeler, Colorado State University, Fort Collins, USA
yohon@cahs.colostate.edu, don.zimmerman@colostate.edu, lkeeler@cahs.colostate.edu

1.        Introduction

Over the last decade, course management software has become increasingly popular with educational institutions. Course management software provides a wide variety of Web-based teaching tools including e-mail, content and syllabi posting, resource pages, chat groups, form builders, bulletin boards, grade books, online testing, and interactive exercises. Much of the literature (Courtney and Talaong 2002; Dean 2003; Esptein 2003; Lyons 2000; Xu and Sloan 2002; Zhao 1998) reports case studies of how educational institutions introduce course management software but few authors report empirically-based evaluations of the adoption of course management software (Bai, Chuvessiriporn, and Lehmann 2002; Piguet and Peraya 2000).

Despite the apparent widespread adoption of course management software, online teaching methodologies are not mainstream. Lammers and Murphy (2002/2003) reviewed the literature on teaching methodologies and reported that studies over the years still found that lecture was the primary classroom instructional methodology. Lammers and Murphy also conducted an empirical study of classroom methodologies at the University of Central Arkansas, and found that lecture remained the predominate instructional methodology.

2.        Setting

In the spring semester of 1998, our university, a Carnegie Class One Research Institution with more than 20,000 students and more than 50 academic departments, adopted WebCT as the course management software of choice. By late 2003, the university had established a support staff that included a WebCT administrator, a part-time staff member handling policy questions and administrative issues, a part-time multimedia developer, a part-time graphics artist/animator, photo and video production staff to aid in course development, and a half-time network administrator supporting the technical aspects of WebCT. The university has promoted WebCT through annual seminars and workshops. The focus of early workshops and seminars was to demonstrate WebCT’s functions. In the past two years, workshops have provided hands-on, targeted practice on specific WebCT tools such as using the discussion board, using the quiz function, or uploading course content. In recent years, the university’s Office of Instructional Services (OIS) has provided about 60 short courses each year on different WebCT tools.

3.        Research questions

Though WebCT is used in 1300 courses annually, little is known about WebCT’s adoption rates among faculty and the demographics of WebCT adopters versus non-WebCT adopters. Additionally the perceptions of WebCT adopters and non-adopters on the impact of WebCT and of technology in general on teaching have never been explored.

To guide our study, we posed the following research questions: (1) What percentage of the faculty adopted WebCT for classroom usage? (2) How frequently did faculty use the different tools of WebCT? (3) When comparing WebCT adopters and non-adopters, were there significant differences in their perceptions of the impact of information technology on their teaching? (4) Were the demographics of WebCT adopters significantly different from non-WebCT adopters? and (5) Were there significant differences between WebCT adopters and non-adopters on their perceptions of the effects of information technologies on their annual merit evaluations, promotion and tenure evaluations, and post-tenure evaluations?

4.        Methods

4.1         Participants

While the university has nearly 1500 faculty members, a limited budget necessitated that we focus our exploratory study of WebCT adoption. Therefore, we selected faculty in the liberal arts and natural science colleges as our study population. The departments in these colleges include anthropology, art, biochemistry, biology, chemistry, computer sciences, economics, English, foreign languages, history, journalism and technical communication, mathematics, music, philosophy, political sciences, physics, psychology, speech, sociology, and statistics. Faculties in these departments post about 700 courses on WebCT annually. These posted courses are not fully online courses, but use WebCT as a tool to support on-campus classes. They represent 43 percent of the faculty on campus and generate 54 percent of the courses taught on campus.

4.2         Instrument and administration procedure

Using the university campus mail system in the spring of 2003, we surveyed the liberal arts and sciences faculty following Dillman’s (2000) Tailored Design Methodology. The questionnaire was an eight page, 7 by 8.5 inch stapled booklet (printed on 8.5 by 14 inch paper and then folded to create the booklet) with the front cover containing a title, art graphic, organizational address and control number. Half of the back page was blank, should the respondents desire to provide additional comments. The six pages of questions were set in 11 point Times Roman font. The university Office of Human Resources provided a master list (n=657) of tenured, tenure-track, and adjunct faculty members. Adjunct faculty members were either part-time or full-time instructors hired to teach one or more courses during any one academic year. No graduate teaching assistants were included in the study. We then pulled a systematic random sample of 328 faculty names. From the original names sampled, 22 surveys were returned incomplete for a variety of reasons, such as the faculty members were no longer teaching, had left the university, were on sabbatical, had not taught in the previous year, and so forth. This reduced our sample size to n=306.

Three weeks into the spring term, we sent participants a letter notifying them that they would be receiving a survey through campus mail in about a week. A week later they received the initial mailing packet consisting of the cover letter, booklet questionnaire, and addressed envelope so that they could return the survey through campus mail. About two weeks later, we sent participants a reminder postcard asking them to return the survey, if they had not done so. We had planned a third mailing two weeks after spring break, but delayed the third mailing (which included a new cover letter, questionnaire, and return envelope) for another week because a major snowstorm closed the university the week following spring break. We waited for four weeks after the last mailing, then entered the data in SPSS for Windows Release 11.5, verified the data, and ran frequencies, descriptive, and inferential statistics.

5.        Results

5.1         WebCT usage (research questions one and two)

Fifty-nine percent (n=172) of the faculty surveyed (n=306) returned completed questionnaires. Of the respondents, 37% (n=57) reported using WebCT for their classes. Using a 1 to 5 scale whereby 1 = “never” and 5 = “frequently,” the leading usage of WebCT tools, i.e. tools rated above the 2.50 median, included the content and syllabus tools, grade book, e-mail, and publishing Power Point presentations/PDF files (see Table 1).

Table 1: Usage of WebCT Functions among Faculty Members Adopting WebCT

Note: Because of rounding, percentage may be greater than 100%

While respondents reported using WebCT overall, their use of some WebCT tools was low, with two-fifths or more of the participants not publishing Power Point and PDF files to WebCT, using threaded discussion, using chat/private discussions, and using group presentations (see Table 1).

5.2         Impact of technology (Research question three)

Next, we explored the differences between WebCT adopters and non-adopters on the impact of technology on their teaching. In comparing WebCT adopters with non-adopters, significantly more WebCT adopters reported that technology saved them time on their daily tasks and enabled them to improve their teaching (see Table 2). However no significant differences between WebCT adopters and non-adopters adopters emerged on the nine other variables of teaching activity.

Table 2: Perceived impact of technology on teaching*

Scale: 1 = Strongly Disagree to 5 = Strongly Agree

5.3         Demographic differences (research question four)

We then explored demographic factors that might have explained differences between WebCT adopters and non-adopters. No significant differences emerged in the percentage of male and female respondents using WebCT; about the same percentage used WebCT (Females = 30% vs Males = 35%, Chi Square = .392, df = 1, 169; p = .531). Nor were there significant differences between WebCT adopters and non-adopters in the percentage of time spent teaching [57.9% vs 60.0% (t = -.603, df =1, 150)], time spent on research [25.8% vs 24.5% (t = .398, df = 1,144)], or time spent on service activities [15.8% vs 17.72% (t = .909, df = 1,144)]. Likewise we found no significant difference between tenure and tenure-track respondents and their adopting WebCT (41% vs 27.6% , Chi Square =1.533, df= 1, p = .213). Nor did we find a significance difference in the adoption between tenure and tenure-track faculty and adjunct faculty (37% vs 29.9%, Chi square .912, df=1, p =.340).

5.4         Effects of information technology on faculty evaluation and rewards (research question five)

Expectancy factors, i.e. norms and reward structures, can influence technology adoption. Faculty evaluations for annual merit raises, tenure, and promotion may reflect the importance of using technology and course management software. While significantly more WebCT adopters than non-adopters reported that using information technology for teaching influenced their annual merit evaluations, its importance was not high (see Table 3). No significant differences emerged between WebCT adopters and non-adopters on their perception of the importance of using information technology, and they rated its importance low for promotion, tenure, and post-tenure.

Table 3: Comparison of importance of using technology toward faculty evaluations and promotions among WebCT adopters and non-adopters*

Scale: 1 = Not at all to 5 = Very Important

The availability of technology training and support can influence the adoption of technologies (Adams 2003). Respondents were asked to respond to how frequently they took advantage of activities related to technology training and support, using a 1 to 5 scale where 1 = “not at all” to 5 = “very frequently.” WebCT adopters versus non-adopters reported significantly higher frequencies of attending seminars on information technology, talking with other faculty members about information technology, and trying new software programs, but not observing other faculty members (see Table 4).

Table 4: Opportunities to learn about technologies

5.5         Respondents’ additional comments

The back of the questionnaire provided space for comments. Four respondents provided reported usability problems with WebCT. Two others reported technical problems using WebCT during the previous term. One respondent reported her (or his) time was better spent working on articles for progress toward tenure, another pointed out that she (he) had no incentive financial or otherwise to use WebCT, and a third reported WebCT was not worth the effort required.

6.        Discussion

Even though the university has made a concerted effort to encourage faculty to adopt course management software for their classes, less than two-fifths of the faculty in the arts and sciences departments reported using WebCT, and they used only a limited number of the available tools. A closer look at what WebCT tools are used reveals that WebCT’s website publishing tools (such as content page, syllabus, and presentation distribution tools) are the most used, while the interactive tools (such as chat, group presentations, and threaded discussion) were seldom used. As course management software, WebCT seems to be used more to increase faculty members’ productivity versus to increase students’ opportunity for higher order learning or more active student-centered teaching strategies. According to Sandholtz, Ringstaff, and Dwyer (1994), the real benefit of technology use as an instructional tool is as a “medium for thinking, collaborating, composing, and communicating” (p. 1). Garrison (1997) agrees that threaded discussion in particular allows for reflective discussion and as a means of communication, is consistent with higher-order thinking and cognitive development. Faculty choices regarding WebCT have not been to develop collaborative learning environments, but to support information exchange.

The faculty members’ limited use of WebCT may also explain the limited difference between WebCT adopters and non-adopters in terms of the perceived impact of technology on teaching. WebCT adopters reported that using it saved them time on their daily tasks and helped them improve their teaching--impacts that are related to productivity issues, not changes in teaching and learning environments.

Adams (2003) reported that faculty members who integrate technology into their teaching are more likely younger, female and have less teaching experience. However, Schifter (2002) found no statistically significant differences for faculty gender, age range, rank or tenure status in distance education participation at higher education institutions. Our data also reflects this pattern with no significant differences between WebCT adopters and non-adopters on teaching/research/service time distribution, by gender, nor by their status or rank as being tenure, tenure-track, or adjunct instructors.

The value of technology-related projects in tenure and promotion decisions was not found to be significant. Although WebCT adopters thought technology innovation had a slight importance in their annual merit evaluations, the means for all faculty evaluation and promotion questions were low (between 1.00 and 2.00 on a scale of 1 to 5, where 1 was “not at all important”). Our findings appear to align with Young (2002) who reported that technology-based projects often are not recognized as part of the traditional three categories used in promotion: teaching, research, and service. In a study of four Carnegie Class One Research institutions regarding reward systems for distance education, distance education is not identified as an area of professional practice, i.e. it doesn’t serve as a measure of faculty productivity (Wolcott, 1997). However, distance education work by faculty is acknowledged at the department level and during annual performance reviews, which is consistent with our results. According to Gruber (2000), promotion and tenure committees are reluctant to acknowledge or reward a candidate’s technology integration into coursework. These arguments about whether technology in teaching should be counted in tenure and promotion decisions are expressed more at research universities than at institutions with a more teaching focus (Young 2002).

7.        Recommendations

Although the university has offered more than 60 short courses and seminars on WebCT for faculty, and many more courses on technology innovations, the limited use of the different WebCT functions suggests that faculty members are not using WebCT as a tool for changing their instructional methods. In reviewing the WebCT and technology session topics offered by our university, most training sessions are how to use the technology tool versus how to integrate the technology tool to enhance the learning process. We suggest that WebCT training includes pedagogy and student engagement solutions so that faculty members understand how to effectively use WebCT and other technology tools.

According to Allen and Seaman (2003), 11 percent of all United States higher education students took at least one online course in Fall 2002 and that the number of students taking at least one online course is projected to increase by 19.8 percent over a one year period. Because of the increasing need for online methods of instruction, we suggest that universities develop a protocol to reward and acknowledge online creative works as part of the tenure and promotion process. Criteria that could be used are (1) contribution of the project to the field, (2) national and local recognition of the project, and (3) strong research based for the project (Seminoff and Wepner 1997). The consortium entitled Multimedia Educational Resource for Learning and Online Teaching (MERLOT) was formed to look at this issue, and is attempting to establish a peer-review process for electronic teaching materials as is used for journal articles (Young 2002).

8.        Further research recommendations

Continued research is needed in faculty and student adoption of course management software as well as its overall impact on teaching and learning. Research into the factors associated with faculty adoption of course management software includes six areas.

First, research is needed to explore the factors that might explain the differences between those who adopt and do not adopt course management software. Additional social and psychological factors need to be investigated, using both quantitative and qualitative methods, to ascertain the differences in adoption and non-adoption.

Second, research is needed to investigate the usability of WebCT and course management software in general. Such research needs to ascertain the usability and the learning curve required to develop proficiency when using course management software. Usability testing, based conceptually on protocol analysis from cognitive psychology (Ericsson and Simon 1993), has emerged as a major methodology for evaluating software and Web sites (Druin 1999; Nielsen 1993, 2000; Shneiderman 1998; Zimmerman and Akerelrea (forthcoming). Usability testing quickly identifies the major problems in design and structure of software that create difficulties for users. 

Third, research is needed to determine the extent of abandonment of course management software and the factors contributing to its abandonment. The authors currently subscribe to a university course management listserv that provides technical support. Over the three weeks prior to beginning the Fall 2003 term, we noted a series of e-mails from senders asking to be removed from the mailing list, explaining that they were no longer using the course management software.

Fourth, research is needed to ascertain faculty’s perceptions of what kinds of classes for which course management software is most appropriate. Respondents from some departments reported that WebCT was not an appropriate teaching tool for their respective classes, but they did not elaborate on their reasoning.

Fifth, research is needed to investigate how faculty use course management software in institutions where it is required compared to those institutions where its use is optional.

Sixth, research is needed to explore the differences in adoption rates of different course management software across different educational institutions and different academic areas. Comparing the rate of adoption by faculty in research institutions and teaching institutions and the factors influencing the rate of adoption may provide insights into how to encourage the adoption of course management software. Understanding the different factors involved in adoption among various academic areas could also highlight unique factors that influence adoption by content area. Conducting this research across educational institutions and academic departments will add to the generalizability of this research.

Research into students’ perception of course management software needs to focus on three areas. First, research is needed on students’ perception of course management software’s impact on course delivery and its impact on their learning. Second, research is needed to ascertain if using course management software enhances students’ learning where course management software is used as an adjunct to on-campus instruction. Third, research is needed on the usability of the software by students and to identify what kinds of problems they encounter when using course management software.

9.        Conclusion

Clearly, higher education institutions are investing hundreds of thousands of dollars in course management software, building faculty training programs, and maintaining the infrastructure to support course management software. To maximize those investments, higher education institutions needed to invest in empirical research to determine the factors associated with adoption, non-adoption and abandonment of course management software. Identifying such factors will enable the refinement of the course management software functions, potentially make it easier to use, and help guide faculty training programs. Such investments would enable institutions to maximize the return on their investments in online course management software.

References

• Adams, N B ‘Educational computing concerns of postsecondary faculty’ Journal of Research on Technology in Education, Vol 34 No 3 (2003), pp 285-303.

• Allen, I E and Seaman J Sizing the opportunity: The quality and extent of online education in the United States, 2002 and 2003. Sloan-C, Needham, MA (2003) [Online]. Available: http://www.sloan-c.org/resources/sizing_opportunity.pdf

• Bai, H, Chuvessiriporn, S, and Lehman, J D ‘Faculty use of WebCT: Impact of P3T3 faculty development initiatives’ Technology and Teacher Education Annual, Vol 2002 No 3 (2002) pp 1494-1495.

• Courtney, K and Patalong, S ‘Integrated information skills--A case study using the virtual learning environment WebCT’ Education Libraries Journal, Vol 45 No 1 (2002) pp 7-11.

• Dean, D E (2003). ‘The first three years of "Blackboard" at Eastern Washington University: Valuable lessons from a highly successful project’ Technology and Teacher Education Annual, Vol 2003 No 2 (2003) pp 895-901.

• Druin, A The design of children's technology, Morgan Kaufmann, San Francisco, California (1999).

• Epstein, C ‘Using Blackboard for training and communicating with student employees’ College & Undergraduate Libraries, Vol 10 No 1 (2003) pp 21-25.

• Ericsson, K A and Simon, H A Protocol analysis, The MIT Press, Cambridge, Massachusetts (1993).

• Gruber, S ‘Technology and tenure: Creating oppositional discourse in an offline and online world’ Computer and Composition, Vol 17 (2000) pp 41-55.

• Lammers, W J and Murphy, J J ‘A profile of teaching techniques used in the university classroom’ Active Learning in Higher Education, Vol 3 No 1 (2003) pp 54-67.

• Lyons, L Report on adoption of WebCT as instructional management system at the University of Delaware, (September 2000). [Online]. Available: http://www.udel.edu/provost/tltr/webctfall2000.html

• Nielsen, J Designing Web Usability, New Riders, Indianapolis, Indiana (2000).

• Nielsen, J Usability Engineering, Academic Press--Harcourt, Brace, Boston, MA (1993).

• Piguet, A and Peraya, D ‘Creating web-integrating learning environments: An analysis of WebCT authoring tools in respect to usability’ Australian Journal of Educational Technology, Vol 16 No 3 (2000) pp 302-314.

• Sandholtz, J H, Ringstaff, C and Dwyer, D C Student engagement revisited: Views from technology-rich classrooms, (1994) [Online]. Available at: http://a1712.g.akamai.net/7/1712/51/1403b36e54106d/www.apple.com/education/k12/leadership/acot/pdf/rpt21.pdf

• Schifter, C Perception differences about participating in distance education. Online Journal of Distance Learning Administration, Vol 5 No 1 (2002) [Online]. Available at: http://www.westga.edu/~distance/ojdla/spring51/schifter51.html

• Seminoff, N E and Wepner, S B ‘What should we know about technology-based projects for tenure and promotion?’ Journal of Research on Computing in Education, Vol 30 No 1 (1997) pp 67-82.

• Shneiderman, B Designing the user interface, Addison-Wesley, Reading, MA (1998).

• Wolcott, L L Tenure, promotion and distance teaching: A study of faculty rewards and incentives. (1997) Abstract retrieved November 5, 2003 from ERIC Firstsearch database. (ERIC Document Reproduction Service No ED413861).

• Xu, C, Sloan, J and Novikova, A ‘Faculty use of Blackboard for course interaction at two Mid-western universities: A multiple case study’ Technology and Teacher Education Annual, Vol 2002 No 1 (2002) pp 330-334.

• Young, J R ‘Ever so slowly, colleges start to count work with technology in tenure decisions’ The Chronicle of Higher Education, Vol 48 No 24 (2002) [Online]. Available at: http://chroncile.com/prm/weekly/v48/i24/24a02501.htm

• Zhao, Y ‘Design for adoption: The development of an integrated Web-based education environment’ Journal of Research on Computing in Education, Vol 30 No 3 (1998) pp 307-328.

• Zimmerman, D E and Akerelrea, C A ‘Usability testing: An evaluation process for Internet communications’ Internet Encyclopedia. John Wiley, New York, NY (Forthcoming, Submitted July 2002).

EJEL is published by Academic Conferences Limited
Curtis Farm, Kidmore End, Nr Reading RG4 9AY, England
Tel: +44 (0)1189 724148, Fax: +44 (0)1189 724691, Email: anna@ejel.org

Send mail to info@academic-conferences.org with questions or comments about this web site.
Copyright © 2002-2004 Electronic Journal of e-Learning
Last modified: December 15, 2004
ISSN 1479-4403