TLT Group Information Literacy Home Page l Matrix Design l Using a Matrix Design to Assess Info Lit Skills l
Using a Matrix Evaluation to Improve Teaching & Learning Activities Intended to Develop Information Literacy l
Why bother with a Matrix Design? l Want to help with Next Steps?

Abstract: This page describes a prototype of a new kind of evaluative tool for faculty to use for assessing and improving the course activities that are intended to develop student information literacy (e.g., assigning a paper that requires use of library databases).

The challenge for gathering evidence from students in order to improve undergraduate information literacy ("closing the loop") is that information literacy isn't a single thing. Nor is a student's information literacy developed by repetition of the same assignment in class after class.  So no one test or evaluation form will be suitable for repeated use in gathering the feedback that faculty, librarians and other staff need.  

Information literacy is composed of a number of component skills.  For example, the Standardized Assessment of Information Literacy Skills (SAILS) includes skills such as these:

1. Developing a research strategy

2. Scholarly Communication Structure of Disciplines

3. Identifying/distinguishing among types of sources

4. Selecting finding tools

5. Selecting search terms

6. Constructing the search

7. Understanding information retrieval systems

8. (etc.)

Some classes involved in promoting information literacy teach one subset of those skills while other course have a different, sometimes overlapping set of emphases.

Even more varied are the course activities (homework, in-class) used to develop those skills, e.g.,

1. Assignment to create an annotated bibliography

2. Assignment to write a paper using library databases

3. Assignment to create a mashup using Google Earth resources

4. Assignment and class activity to work in a team to a) research a question using library and online sources, b) present an argument in class, drawing on those sources to make and support your case, c) have the whole class rate the credibility of the argument anonymously, using some form of student polling.

5. (etc.)

Here again, some instructors use one set of such activities while other faculty members employ others.

So how can one research/evaluation tool provide a) helpful feedback for each instructor, b) helpful feedback for the faculty, librarians and other staff who share responsibility developing undergraduate information literacy?

A matrix design may be ideal for this task.

A team (e.g., a learning community of faculty, librarians and others) could create a matrix tool for research and evaluation that all of its faculty members could use, within an institution or across a number of institutions.  The matrix tool could look something like this:

For example, in course 2, students would be asked to provide evidence about issues B, F and I. ("Issue" could mean either skill or activity.)  An evaluation team could gather such evidence about all the issues by looking across enough courses. For example, this chart illustrates that they've collected student data on issue A from students in five different courses.

A Matrix Design to Assess Skills: If a matrix design were used, in each course students would only be assessed in those skills the faculty were already teaching and testing in that course.  Putting together this matrix of data from the courses, standardized tests, and/or portfolios, faculty and librarians could assess:

• learning in each course;

• how learning in each course relates to learning in other courses,

• whether skills are developing as students move through the curriculum, and

• whether the team of engaged faculty and staff are becoming more effective over the years.

A Matrix Design to Improve Course Activities: Working with Sheril Hook of the University of Toronto, Mississauga, the TLT Group's Flashlight Program has begun developing a prototype matrix evaluation tool for improving activities intended to develop information literacy. You can see examples of two response forms below, each tailored for a different course. As you'll see, the questions we ask students almost all deal with the process of the activity:

• is the activity engaging them and, if not, why not?

• if they haven't been able to participate fully in the activity, what barriers were hindering them?

• how much time did students spend on the activity?

• were the instructions clear? or confusing (perhaps because the student misunderstood the language used in the instructions, misinterpreting a term such as "scholarly resource" or "peer-reviewed."

• what advice do students have about how the activity might be improved, next time the course is offered?

This first response form is tailored for a course whose instructor is concerned about:

• whether students all understood required terminology well enough to understand information literacy assignments:

• an assignment to use library databases to do a research paper;

• a Google Earth mashup assignment; and

• a team activity in which student teams were asked to research a question, present an argument using sources, and then have the credibility of their argument and sources rated anonymously by the whole class (using clickers, for example).

The same prototype matrix survey was also used to create this second response form for a different course, a course whose instructor is concerned about:

• assignment to create an annotated bibliography,

• required terminology (same issue as for the previous course)

• That same activity to do research, create a presentation, and rate the presentations of others. Notice that the name of the topic for the research and presentations is different in this second response form. The first form asked about "Causes of the War of 1812," while this second form asks about research and presentations about "How Austen influenced Dickens?".

Wording of questions is one of many things that Flashlight Online 2.0 can automatically customize. In the example above, each instructor can specify the name of the assignment students are being asked to assess.

Note that, even though response forms for each course are tailored, all these different response forms feed data into one survey. That's true even when data is collected semester by semester over a period of several years The responses all flow to the same database, where they're easy for members of the team to analyze at any time.

Why use a matrix survey? Matrix surveys make it easier to whip up a tailored feedback form to fit faculty and/or librarian needs. Once a community of practice (faculty, librarians and perhpas others) has created the matrix survey, then any faculty member or librarian can specify a course and set of issues, and the matrix survey will automatically create a feedback form tailored to that class.

In some instances, it an also be valuable to the community to pool data. For example, in the evaluation of the database activity, the questions about whether certain previous courses helped prepare the student use databases are much more valuable when many upper division courses use that question and pool data.  When using a matrix survey to study developing skills, pooling of data may help illustrate whether students are developing skills cumulatively as they move through the curriculum.

Want to help with Next Steps? The work described immediately above is a prototype, a simple proof of concept example that we assembled with a  few hours of work. If you and your institution want to explore options for working with the TLT Group and other institutions to develop real matrix approaches to evaluating and improving information literacy programs, please email Steve Ehrmann (ehrmann @ tltgroup.org).

-Stephen C. Ehrmann, Director, The Flashlight Program

TLT Group Information Literacy Home Page l More on Matrix Study Designs l More on Flashlight Online 2.0 (matrix survey tool)

PO Box 5643
Takoma Park, Maryland 20913
Phone: 301.270.8312/Fax: 301.270.8110  

To talk about our work
or our organization
contact:  Sally Gilbert

Contact us | Partners | TLTRs | FridayLive! | Consulting | 7 Principles | LTAs | TLT-SWG | Archives | Site Map |