Performance Based Certification Criteria and Standards for Online Web-Based Instruction and Delivery
A transformation is occurring in teaching and learning. Technology, the Internet, and the World Wide Web are causing us to rethink many of our basic assumptions about learner and instructor roles and responsibilities.
Meeting some form of specific criteria and standards is becoming increasingly important to individuals who are seeking quality online web-based learning experiences. At the same time accreditation agencies are attempting to articulate quality assurance strategies for online web-based instruction. Distance learning associations are also beginning to identify the current best practices in web-based instruction and delivery.
The performance based certification criteria and standards for online web-based instruction and delivery that are currently being established include: determining that the course is constructed effectively for online web-based delivery; determining that the instructor is able to facilitate the learning process with the focus on the learner; and verifying that the appropriate learning has taken place.
Key Online Learning Paradigms
The Council for Higher Education Accreditation (CHEA) commissioned The Institute for Higher Education Policy to investigate the emerging topic of quality assurance in technology-mediated distance learning programs in higher education. In April 1998 CHEA released an Executive Summary of their findings, selected portions of which appear here.
What are the existing or emerging strategies for quality assurance of technology-mediated distance learning programs?
Quality assurance strategies appear to be integrated into the design of most postsecondary education distance learning programs. These strategies look quite similar to those in traditional institutions and programs, focusing on the following four key aspects of quality assurance:
Faculty Credentials, Selection, and Training
Quality assurance in many programs and institutions focuses heavily on review of faculty credentials, selection procedures for new faculty, and faculty training. Often, candidates for a faculty position are required to engage in an intensive training program that includes conducting a mini-lesson, interviewing with current faculty, and being assigned a mentor. In addition to possessing a graduate degree relevant to the field of instruction, faculty are expected to understand the role of technology in a learning environment, be trained in online teaching concepts, and use assessment techniques appropriately.
Time-on-Task Measures
Focus on time-on-task measures, including minimum weeks for courses and monitoring of course "log-ins", is common in distance learning programs. In addition to textbooks, many courses are accompanied by a comprehensive study guide that provides course objectives and key concepts. Students are expected to spend a minimum amount of time per week for study and homework assignments. Interaction with faculty is mandated and, in many cases, students share access to a class group mailbox along with the instructor, which provides a platform for instructor communication at virtually any time.
Student Support Services and Consumer Information
Several of the distance learning providers conduct what appear to be rigorous reviews of student support services as one element of quality control. In fact, focus on adequate student support as an essential element of teaching and learning may be one of the most distinctive features of quality control in distance learning environments.
Goals and Outcomes
Quality assurance in distance learning is distinguished by a strong emphasis on program goals, and assessment of results or outcomes in the context of these goals. The focus on assessment of goals and outcomes does not necessarily mean that distance learning programs have been path-breakers in devising new ways to measure instructional outcomes; in fact, with few exceptions we found that distance learning providers measure learning results in much the same way as do conventional programs.
Comparing "Conventional" and Distance Learning Quality Assurance Strategies
Quality assurance strategies for distance learning tend to be oriented toward institutional assessment activities, and to affirming that the core capacities to assure quality, such as faculty credentials and student support services, are in place. Thus, the review is of resources and inputs more than educational value-added, making it similar to quality assurance in traditional higher education.
What seems to be most strikingly different is the process for quality review in distance learning programs. In distance learning programs there seem to be a greater tendency for the assessment process to be led by the administration instead of the faculty, with greater use of outside consultants and assessment "experts" in lieu of internally-generated peer reviews. The quality assurance process therefore appears to be less process-driven, where there is a high value placed on consultation, consensus building and dialogue, and more oriented to "bottom-line" or market-oriented results.
In addition to the differences in the process for quality assurance, other pertinent differences that emerge in comparing the two include:
Mission and Focus
The major stated commitment in distance learning is to the teaching/learning process, and there is a sharper focus on that as an exclusive goal.
Focus on Client
The student is regarded first as a client of the organization, and the educational activities that the client desires predominate in the design and implementation of programs.
Faculty has Less Control over Curriculum
The tendency to develop or use pre-packaged courses and the preponderance of part-time faculty are characteristics of many distance learning programs.
Less Emphasis on Process
The conventional academic culture relies heavily on process, is substantially consultative, and is consensus-driven. The delivery of distance education programs tends to abandon, at least partially, these traditional quality assurance activities, and appears to be more assessment-driven.
Contracting for Services
Many distance learning programs contract with other entities to provide those administrative and student services not directly related to the teaching/learning process.
Within the context of creating effective web-based learning experiences David Brigham has suggested a focus on process: what influences facilitate and impede student perceived progress toward student course goals, how are these influences related, and what is their relative importance? He, like Richard Clark pointed out that it's not the delivery method that makes the difference—it's the learning strategies employed within the delivery method, and implementation of those strategies, that matter most. J. Olin Campbell added that new delivery vehicles (like computers) can enable strategies like powerful simulations. He also observed that learner and coach self reports can be helpful but are just one cue to causality. He prefers a combination of perceptions through a qualitative study and outside/more objective rating via job performance, projects, tests and expert opinion.
Evaluation of examinations and grades may be too limited. Instead there should be an identifying of specific and meaningful learning outcomes, then comparing student achievement of those outcomes using different methods, according to Edward Neal. That means mastery of higher-order learning rather than simple recall or application of information. He cites Bloom's Taxonomy of Educational Objectives as particularly helpful to identify precise and measurable definitions of learning outcomes. These include recall, comprehension, application, analysis, synthesis, and evaluation (judgment). In particular, Neal is concerned with undergraduate (not adult/professional) education, and suggests that many interchanges that occur in Asynchronous Learning.
Networks do not improve higher order thinking abilities, but are simply sequential position statements.
Kirkpatrick proposes 4 levels of evaluation:
- Learner reaction (e.g., end of course evaluation sheets)
- Learning (e.g., learner performance at end of course)
- Behavior (ability to use what is learned in realistic situation or job)
- Results (impact on the learner's organization)
Evaluation Techniques and Examples
Example A
The list of questions that follow were developed as part of a study, the major purpose of which was to examine the student learning experiences in taking an asynchronous course. To accomplish this, a combination of quantitative and qualitative data was collected. Students were asked to keep logs of their weekly activities for certain periods, that they turned in to the instructor. Students completed satisfaction surveys at the fifth week and again at the end of the course (fourteenth week). At the end of the course, students were also asked to write an evaluation of their experiences with the course addressing specific questions such as:
1. Do you feel that this course was successful for you? Why or why not?
2. Do you feel that your overall learning was less than, about the same, or greater than in a traditional course?
3. If successful, what aspects of the course contributed to its success?
4. If unsuccessful, what aspects of the course proved problematic for you?
5. What suggestions do you have for improving this course?
6. Would you take another asynchronous course? Why or why not?
7. Would you recommend an asynchronous course to other students? Why or why not?
8. Would you recommend this asynchronous course to other students? Why or why not?
9. Provide any other comments that you think would be helpful for this evaluation.
Example B
The following course evaluation for Using the World Wide Web was developed by Dr. Steve Catt, Emporia State University.
1. Approximately how many hours did you spend online to complete the requirements for this course?
2. Was this too much time __, too little time __, or about right __ for this course.
3. How relevant and worthwhile are the skills you acquired in this course?
4. Were you able to work on activities that contribute to your personal or career goals?
Please elaborate.
5. Would you recommend this course to others?Why or why not?
6. How would evaluate your experience in taking this online course?
7. What comments/recommendations do you have for the instructor of this course?
Example C
Please circle your response to each of the following statements:
1. The Objectives of this course were stated clearly.
Strongly Agree.........Agree.........Don't Know.........Disagree.........Strongly Disagree
2. Students were treated fairly in this course.
Strongly Agree.........Agree.........Don't Know.........Disagree.........Strongly Disagree
3. The grading procedures were fair.
Strongly Agree.........Agree.........Don't Know.........Disagree.........Strongly Disagree
4. The instructor is knowledgeable about the subject matter of this course.
Strongly Agree.........Agree.........Don't Know.........Disagree.........Strongly Disagree
5. The instructor was willing to answer questions and give assistance.
Strongly Agree.........Agree.........Don't Know.........Disagree.........Strongly Disagree
6. This course expanded my knowledge and understanding of the subject matter.
Strongly Agree.........Agree.........Don't Know.........Disagree.........Strongly Disagree
7. Overall, the instruction in this course was effective.
Strongly Agree.........Agree.........Don't Know.........Disagree.........Strongly Disagree
Ingredients of a Good Online Web-Based Design
Though it would be difficult to have every Web user agree on what constitutes good online web-based document design, three factors should be given serious consideration by developers: Oftentimes less is more; Consistency; and "Don't waste the user's time."
Oftentimes less is more
The primary feature of the World Wide Web (WWW) that distinguishes it from other Internet services is its ability to provide extensive navigability within a document and between documents. Thus, many developers place with a Web document the largest number of active links possible. Unfortunately, this has opposite the intended effect. When "every fifth word" in a Web page is highlighted, the user tends to ignore the links while scanning the page. This is especially true when the links are presented within paragraphs and are therefore difficult to isolate from the surrounding context. Developers should take full advantage of the power that hypermedia provides for information access. However, to avoid abusing the power inherent in Web documents care must be taken to organize information in an easily comprehensible format that invites, rather than deflects investigation. Such organization includes the use of short paragraphs, lists of bulleted items, and the selection of only main ideas for linking. Web pages should also be limited in length to avoid excessive scrolling. In general, it is better for a site to have many shorter Web pages with few links than to have few long pages with many links.
Consistency and Predictability
What does the homepage look like? All other link pages should have a similar appearance. Paragraphs and lists should have a common format from page to page. Further, each page should be capable of standing alone as a Web document. Each page should include a header, perhaps including a smaller version of the logo used on the main page, as well as contact information and the site's http address. It is a common frustration among WWW users to be several levels into a site, come to a dead-end, and then be forced to use their viewer's "Back" button in order to continue navigating the site. Each link page should include options such as "Next Page," "Previous Page," and "Homepage" or "Main Menu" to enhance navigability and these choices should be in the same place on each page so the user can know exactly where to look for them. Predictability of instructional Web documents is reassuring to the user, indicating to them the dependability of the site and its information.
"Don't waste the user's time"
The ability for anyone to publish almost anything on the WWW is both a strength and a potential pitfall of the Web. Instructors should be encouraged to place course materials on the Web, including their own papers and thoughts as well as those of their students. However, instructors need to take the initiative to carry out a winnowing of contributions, not allowing inappropriate, extraneous or redundant materials to be included on the site. For those materials that are included, it is important to remember that papers and other contributions do not take on added significance when they are converted to multimedia format. Developers must resist an attempt to use every HTML command available to them, such as the use of blinking words and tiled backgrounds, etc. Concentrate on those presentation techniques that will enhance the communication of ideas. In general, if the word "cute" comes to mind when viewing a Web page, the presentation technique being used is probably a waste of the user's time. WWW users have quickly become sophisticated consumers of multimedia and will tend to eschew a site rich in "features" but devoid of content.
Activities Critical to Maintaining a Quality Website
There are many activities that are critical to a successful learning experience of an online web-based delivered course. Four of the most critical are:
1. Conduct regular checks of the links to ensure that the referenced URL is still viable.
Perhaps the most frustrating experience for any Web user is clicking on a highlighted link and receiving an error message such as "URL Not Found." This is especially likely to occur when attempting to link to educational resources offered at other sites. Developers need to conduct regular checks of their links to ensure that the referenced URL is still viable.
2. Being locked out of a site.
Equally frustrating to the user is the surprise of being locked out of a site option because of being physically located off-campus. One wonders whether the site developers realize that the Web is indeed worldwide and therefore their work, especially a site available through the World Lecture Hall, is open to inspection by more than just those on their campus. Certain options and resources can be reserved for students in the class, but users should be warned early of the restrictions. This problem is akin to there being no indication of file size on a link for a large audio or video segment. In both cases the user is forced to choose the link in order to determine whether the action can be completed; this amounts to leaping before looking and is generally unsatisfactory.
3. Avoid large graphics files and items that are slow to download.
Mention of downloading leads one to a consideration of the variety of computing facilities users will be employing in order to access WWW sites. Instructors and developers need to keep in the forefront of their minds that not every user has a T1 high-speed data connection available to them. Thus, the use of visual enhancements such as image maps, rainbow dividers and fancy header graphics merely serve as detractors for many users because of the time required to (constantly) download these images. Developers can mitigate user frustration by including reduced versions of logos on link pages rather than the original full-size logos, eliminating the use of graphics such as rainbow dividers that are all form and no substance, and providing text links instead of image links as often as possible.
4. Don't bury learner work in an obscure link.
Finally, the design of instructional Web sites should reflect the fact that one of the most interesting aspects of visiting these sites is the opportunity to inspect student work. The problem is that many times student projects are buried in an obscure link that requires a great deal of searching to uncover. Student work, the centerpiece of any course, should be prominently displayed in instructional sites. This results both in a shared resource for educators and a showcase for the students of the course.
References
American Council on Education, (1996). Distance Learning Evaluation Guide.
American Council on Education, (1996). Guiding Principles for Distance Learning in a Learning Society.
Bourne, J. R., McMaster, E., Rieger, J., & Campbell, J. O. (1997, August). Paradigms for Online Learning: A Case Study in the Design and Implementation of an Asynchronous Learning Networks (ALN) Course. ALN Magazine, 1(2). [Online]. Available: www.aln.org/alnweb/journal/issue2/assee.htm.
Campbell, J. O. (1997, August). Evaluating ALN: What Worked, Who's Learning? ALN Magazine, 1(2). [Online]. Available: www.aln.org/alnweb/magazine/issue2/campbell_alntalk.htm.
Council for Higher Education Accreditation. (1998, February). Quality Assurance and Distance Education. [Online]. Available: www.chea.org/Events/QA_summary.html.
Council for Higher Education Accreditation. (1998, April). Assuring Quality in Distance Learning. [Online]. Available: www.chea.org/Perspective/assuring.html.
Dasher-Alston, R. M., & Patton, G. W. (1998, Fall). Evaluation Criteria for Distance Learning. Planning for Higher Education, 27, 11-17.
Dereshiwsky, M. (1997, October). Highlights from "Rules Of The Road". ED Journal, 11(10), J21-J23.
Hall, B. (1997). Web-based Training Cookbook, John Wiley & Sons, Inc. New York, 1997, 20-21.
Harris, K. (1998, March). Evaluating Interactive Web-Based Training On The Web. Teleconference, 17(3), 32-35.
Moulds, L. D. (1996, June). Using Distance Learning in the Training of Adult Learners. ED Journal, 10(6), J15-J21.
Moulds, L. D. (1996, October). Cost-Effective Distance Learning Technologies Used By Multiple Location Companies To Train Small Numbers Of Employees Who Are In Widely Dispersed Locations. ED Journal, 10(10), J21-J24.
Picciano, A. G. (1998, March). Developing an Asynchronous Course Model at a Large, Urban University. Journal of Asynchronous Learning Networks, 2(1), [Online]. Available: www.aln.org/alnweb/journal/vol2_issue1/picciano.htm.
Virtual Washington State University. Asynchronous Learning. [Online]. Available: www.wsu.edu/vwsu/RFP9631/WP1101496.html.