|
Flashlight Handbook Table of Contents
Calling our program "Flashlight" was a
protest against assessments and evaluations that ask vague,
global questions - "fishing expeditions" that try to glimpse
everything and end up proving nothing. If you have written
studies organized around questions such as "What do you like
about this innovation?" and "on a scale of 1-5, how
satisfied are you with the innovation?" please read on.
The process of doing a study is like using a
small, dim flashlight to glimpse what sort of animal might
be in front of you in a huge dark cave. Seeing everything
in the cave is impossible -- it's too large and complex, and
your flashlight (evaluation) is too weak. Furthermore, with
the flashlight of inquiry, pointing it in a new direction
takes more time and money.
The direction of the beam represents the
question you ask. The brightness of the flashlight
represents the power of the inquiry to convince a skeptical
audience of the truth of what you see. Each study design is
the equivalent of picking a flashlight, pointing it in a
direction, and looking down the beam in order to see what
walks into the light.
 |
There's another lesson from this metaphor:
if you have a choice of what to seek, try to illuminate the
most important uncertainty out there. If the cave is
large and dark, and the flashlight is weak, it's easier to
prove you've spotted an elephant than a flea.
You line up the beam six feet or so above the floor of the
cave, and watch what it reveals. If something that looks
like a tusk or trunk seems to pass through your beam, your
guess may well be right: there may well be an elephant out
there. Evaluation is like that: you can never
illuminate the whole cave or even a whole elephant (or
flea). So pick a target that ought to be easy to see,
develop a way of detecting some piece of it, and focus.
|
Imagine, for example, that your program's use
of computers is intended to help students spend more time in
their studies, by increasing their enthusiasm and cutting
time wasted in commuting. That would be an important
achievement: time on task is perhaps the single best
predictor of how much students will learn. But which
administrators, faculty members, staff, or students are in a
position to know whether average student time on task has
increased? That possible change in behavior is "hidden"
because of where it happens: often in private, in hundreds
of dormitory rooms, libraries, apartments, cars,
classrooms.… Each person can estimate how he or she uses
time, but no one knows how everyone uses time. 'How students
study' is an elephant hidden in a dark cave. If you could
illuminate that elephant with a focused study, you could
provide insights that could help faculty teach as well as
helping IT people with plans for infrastructure and faculty
development.
The most important strategies for learning
and teaching are often hidden in plain sight.
These powerful patterns are the sum of hundreds or thousands
of changes in relatively private activities by students and
staff. Each person may know quite well what he or she is
doing differently, but not whether life is changing for most
other people, even in the same course of study. Helping
illuminate such elephants in the dark is an important
mission for the Flashlight Program, and this Handbook.
Next: "How to Find What
You Need in This Handbook"
|
