Pamela Schneider,
Educational Technology Assessment Coordinator

Office of Educational Technology

Fairleigh Dickinson University

In response to a need for assessing the Core program at Fairleigh Dickinson University, a comprehensive assessment plan for evaluating the first Core course, The Global Challenge, was developed. 

This assessment plan was designed by a committee of faculty members and assessment specialists and implemented in the spring 2003 semester.    The assessment plan involved multiple embedded assessment techniques that were specifically designed to measure the course objectives. 

Assignment for students: One of these objectives focused on the evaluation of websites on the Internet, referred to as the “webliography”.  The webliography assignment directed students to carefully select at least 20 websites during the course of the 14-week semester and evaluate them according to specific criteria: content, currency, reliability, and bias, writing a short paragraph on each.  [If you would like to see a sample of this course, click here.  Within the sample Global Challenge course, under “Assignments” the reader can view the revised Webliography assignment.] 

Assessment of student work: At the semester’s conclusion, a sample of webliographies was randomly selected from all sections of the course.  They were collected, coded and all identification removed.  Faculty volunteers from both the core and non-core faculty met for a full-day session to evaluate each webliography.  The day-long session consisted of a morning orientation to the project followed by a practice session where the individual ratings were compared and discussed using three sample webliographies for evaluation.

A rubric with each of the four assignment criteria was used to evaluate the webliographies. (html version of the rubric; Word version of the rubric] During this session, examples were presented and the faculty had the opportunity to decide on the scores for the three sample webliographies, followed by a discussion of why those scores were given.  This training session helped to minimize the discrepancies between the raters. [Editor's Note: If you would like to learn more about rubrics in general, click here.]

In the afternoon session, each student webliography assignment was evaluated by one Core and one Non-Core faculty member, using the rubric.  A third trained rater evaluated the webliography assignment when discrepancies occurred.  In this assessment activity, particular attention was given to the distribution of total scores, the scores for each of the four criteria, and the reliability of the scoring agreement between each pair of raters. 

Useful findings: What was learned from the webliography assessment activity:

• The use of the analytic rubric helps to bring fairness and clarity to the assessment process.  With training, the percentage of rater agreement in evaluating the student work was impressive. 

• Cathy Kelley, one of the faculty teaching the course, reported that, "As a result of the assessment, the webliography assignment was completely re-designed. Instead of an end-of-semester project, the webliography was assigned early in the term and only five websites were analyzed. Students were asked to provide more detailed information. For example, instead of assessing reliability, students were asked to identify the author of the page and his or her credentials. After receiving feedback on this preliminary webliography, students continued to provide webliographies with every paper submitted during the course of the term, and thus received several opportunities for feedback and improvement. A follow-up assessment has not yet been conducted to determine if the new strategy is working better, but informal reports from instructors suggest that the assignment is far more successful."

Editor's Note: Information literacy, as this article illustrates, is a very different concept from traditional bibliographic instruction. The latter was usually the responsibility of the college library. A librarian could teach an entering student much of what the student needed to know in an hour or two.  In contrast, information literacy includes all the skills students need to do research, including research in their fields, and to analyze and express what they've learned. The TLT Group has been collaborating for years with the Association of College and Research Libraries (ACRL) on fostering information literacy and we are now just beginning to work with the Association of American Colleges and Universities' Greater Expectations program as well. (Click here for information literacy resource page, including online events)