Evaluating Educational Uses of Technology:

Preparing to Attend a Flashlight Workshop

February 25, 2002

One of the most frequent complaints about Flashlight workshop is, "We could have had more hands-on time if less time were spent lecturing; we should have been asked to read this material in advance."  So here's the reading.  

One of our favorite cartoons shows a parent preparing to assemble a child's toy before the holidays. The parent is peering at a sheet of paper headed "Simple Instructions for Assembly."  Step 1 reads "Get a bachelor's degree in mechanical engineering." There's a footnote. "We said 'simple.' We didn't say 'easy.' "

Evaluation, assessment and research are like that: simple but not easy. The ideas are deceptively simple, but the more you work with them, the more powerful they'll become for you. In other words, no matter how much you study the materials linked to this page, you are in no danger of spoiling the workshop you're preparing to attend. The preparation should simply make the workshop even more rewarding for you.

Ideally you should spend at least a couple of hours in this reading, especially if you need to select or fine-tune a study topic before attending the workshop. 

1. Study at least four of the resources below. 
2. Some workshops are designed to help you work on a topic of your choice during the workshop itself.  If that is true of this workshop and if you don't already know precisely what you might study, these materials should help you select a topic. In any case, send your candidate issue to mailto:Flashlight@tltgroup.org as soon as possible; in the message tell us which workshop you will be attending and, if you know, who will be leading this workshop. If no one has given you a different deadline, please get us this information no less than three weeks before the workshop. We will use the information to select workshop staff and tasks.
3. After you choose a topic, make a final decision about who should come with you to the workshop (should a team work on your study? who would be on the team? who needs the kind of training the workshop offers?) 

Flashlight Background

For a description of Flashlight tools and resources (there are about twenty of them as of February 2002), start from this Web page < http://www.tltgroup.org/programs/ftools.html > 

For a summary of the history of the Flashlight Program and some of the issues facing it, see this Interview by Distance Educator with Steve Ehrmann, Director of the Program (2001)

Concepts, Research, and Cases

Chickering, Arthur and Stephen C. Ehrmann (1996), "Implementing the Seven Principles: Technology as Lever" AAHE Bulletin, October, pp. 3-6. This essay by Chickering and Ehrmann outlines the kinds of technology use that can help faculty and students implement Chickering and Gamson's "Seven Principles of Good Practice in Undergraduate Education."  Flashlight tools for studying learning effectiveness help you focus on how technology can improve outcomes by assisting practices such as these.

Ehrmann, Stephen C., (1995) " Asking the Right Questions: What Does Research Tell Us About Technology and Higher Learning?" in Change. The Magazine of Higher Learning, XXVII:2 (March/April), pp. 20-27. This essay from Change Magazine gives a more brief overview of the evaluation literature on teaching, learning, technology and costs. Some of the key assumptions of the Flashlight Program are developed here.

For prior studies that used Flashlight tools and methods, see the case study section of our Resources Web site.

If your starting point is the need to evaluate a grant-funded project:

Ehrmann, Stephen C. (1998) "How (not) to Evaluate a Grant-Funded Technology Project," in Stephen C. Ehrmann and Robin Etter Zuniga, The Flashlight Evaluation Handbook (1.0), Washington, DC: The TLT Group. This essay lays out some of the most common assumptions underlying evaluations of grant-funded technology projects and argues why all of them need to be rethought.

Ideas for Studies:

Looking for ideas about what to study at the program or institution level? Here's a rather cryptic table. This is a taxonomy of study questions arising from different purposes for using technology to improve education. The first row of those purposes are operational (doing better what we've always done) while row 2 purposes are transformational (altering the character of our institutions). Row 3 concerns studies of issues raised when we make progress on those transformational purposes.

Are you a faculty member looking for a topic to study that might help you improve a course? The hardest thing can be to conceive of a question.  Here's a brief tutorial about the scholarship of teaching that might help you invent your first study question.  Want more help? Here's a draft tutorial that discusses several different types of study.. 

Too many ideas? Here's a list of criteria you or your committee could use to select your first (or next) study. 

"Finding a Great Evaluative Question: The Divining Rod of Emotion." Do all your potential topics seem 'interesting?' Maybe you need to look further. An intense sense of dread or excitement can signal that a topic is important enough, and uncertain enough, to be worth the effort to answer, as the examples in this paper illustrate.