|
Flashlight
Evaluation Handbook
This draft was
rewritten as a "Viewpoint" column that appeared in the
January-February 2002 issue of
EDUCAUSE Review
From mainframes in the 1960s to
microcomputers in the 1980s, to the Web and Internet II
today, higher education has hoped to use computers to
improve educational outcomes on a large scale. By
“outcomes,” I mean who can learn and what it is they can do
by the time they’re finished with the academic program. By
“large scale,” I mean improvements in outcomes for graduates
of an entire degree program, institution, or nation.
After forty years of trying, the
track record has been uneven, at best. The greatest success
has been achieved in using information technology (IT) to
teach technology-dependent content such as computer science,
computer graphics in the arts, and data-intensive approaches
to political science. A second, growing success has been to
open access to education for students who couldn’t fully
participate on campus before. And we have been able to
enhance productivity in certain areas (e.g., by using
computer spreadsheets instead of paper, pencil, and adding
machine).
But what about the goal of
improving teaching and learning? New purchases of hardware
have often been justified by the claim that they would
improve teaching/learning effectiveness in all fields.
Unfortunately, history has
exhibited a grim cycle. Each time a new technology has come
into vogue, its advocates begged the institution to buy it,
so as not to fall behind. They promised that its use would
improve learning. Yet five years later, a completely new
technology had taken center stage. No large-scale
improvements in the effectiveness of instruction had been
achieved from the last purchase, but this new
technology, advocates promised, would change everything!
Whatever the question, “new technology” always seemed to be
the answer.
Well, I don’t buy that theory
anymore. For almost a quarter century I have been evaluating
new uses of technology, both as a program officer with the
Fund for the Improvement of Postsecondary Education (FIPSE)
and later with Annenberg/CPB, and today as Director of the
Flashlight Program. As early as the 1980s, we funded
pioneers who tried to use “new” technology to improve
outcomes for a degree program. Although we didn’t realize
most of those hopes, I still believe that we could have
succeeded, even in the 1980s, if we’d done a better job of
implementation. Obviously such improvements are possible
today, on an even larger scale. Yet we will surely fail
again if we repeat the “common sense” policies that have
sabotaged so many previous IT investments.
Why IT Investments Often Don’t Improve Educational Outcomes
At least three basic problems
have dogged most attempts to translate technological
investments into improvements in educational outcomes.
1. Surrendering to rapture of the technology.
For a variety of reasons, institutions and programs tend to
focus just on the new technology itself. That’s bad. To put
it metaphorically, you must have yeast to bake bread, but if
you buy only yeast, you’ll never produce bread.
Whether a program’s aim is to use technology to support
learning communities, or better skills or inquiry, or an
internationalized curriculum, the recipe will require more
than hardware. Other expensive
ingredients include staff development and/or new staff; new
assignments and course designs; more books in the library;
altered marketing and advising; changes in roles and
rewards; new organizational partnerships; and new internal
coalitions. In the past, the technology siphoned money and
attention away from the rest of the recipe.
2. Forgetting that the life span of many new
technologies is far shorter than the time it takes to
implement the recipe and improve educational outcomes.
Those complex recipes are not “quick and easy”:
assembling the ingredients takes a long time. Meanwhile
the technology is aging, and losing value. Long before
outcomes have a chance to improve visibly, new technologies
usually distract attention from the “old” improvement
agenda. Over the years,
technology-related interests in improving outcomes such as
programming skill, visualization (in the early days of
videodisc), and collaborative learning (the computer
conferencing systems of the 1980s) have risen, and fallen,
and sometimes risen anew. Over the decades, waves of new
technology have rippled across the surface of education but
large-scale improvements in outcomes almost never had time
to develop.
3. Trying to improve outcomes
and save money by using tutorials and other forms of
self-paced, interactive, branching courseware. This is
one educational recipe for improvement of outcomes that
hasn’t changed. It has been attempted with almost every new
computing technology of the last four decades, from PLATO to
the Web. These kinds of tutorials are always enticing:
research has demonstrated that such courseware can
dramatically improve outcomes, learning speed, and costs.
But the problems of large-scale implementation have always
proven insuperable: the short lifecycle of the courseware;
the expenses of educational debugging of the many pathways;
the hidden costs of altering the curriculum to take
advantage of the courseware; the rigidity of the courseware
in the face of new developments in the discipline and
variations in students; the lack of rewards for authors; and
the expenses of marketing and support.
How to Use IT Investments
to Improve Educational Outcomes
Let’s learn from past
mistakes. Here are five strategies that
should help higher education institutions and programs use
IT to make major improvements in educational outcomes.
1. Begin with a long-term
focus on a few selected outcomes and the educational
activities needed to improve them. Which goals
and strategies are worth pursuing for the seven to ten years
that are needed to make large, visible improvements? Here
are a few candidates: skills of inquiry and research; the
ability to apply learning more successfully in the real
world; skills of working in teams, communities, and
organizations; international and intercultural
understanding; skills of designing, composing, and creative
work.. These outcomes can be relevant in almost any field.
Other, equally valid goals are specific to particular fields
or types of students. If a program succeeds in dramatically
improving even one such outcome, the rewards of enrollment
and financial support could be great.
The simplest way to choose among
candidate goals is to select one that represents both a
current strength and a current concern. For example, an
institution or program might decide to improve skills of
creative work, both because some of its best work is already
being done in that area and because it is worried that
competitors might leapfrog over it.
2. Choose technology that can
contribute incrementally and cumulatively over the long
haul. Suppose that a program has already been using
technology to pursue such a goal for a couple of years.
Now discussions begin about a major
purchase of a new technology that would be used by most
faculty, staff and students. Several
questions can be useful in the choice regarding new
technology: (a) Can the dollars spent on the new technology
help the program make major progress toward improving
the chosen outcomes, compared with other ways of spending
the same money for advancing that educational agenda? (b) Is
the new technology ready for mass use, and are operating
costs, including support and hidden costs, acceptable? (c)
How badly will the adoption of the new technology disrupt
the current educational strategy? For example, how much
curricular material and how many course designs would need
to be discarded in order to take advantage of the new
technology? (d) If this brand or product disappears, will
the curricular material become unusable or will competitive
products also run those same files?
3. Emphasize forms of
instructional material that most faculty members find quick
and easy to create, adapt, and share. Most
courses require materials that instructors or tutors can
inexpensively modify to match the needs of students,
personal styles, and recent changes in their fields. By the
same token, the instructional formats should make it easier
for faculty to organize, edit, and share those incremental
improvements in the materials and course designs so that, as
a field, they can move forward as quickly as possible.
4. Track the progress of the
strategy to get the data (and money) needed to stay on
course. An educational initiative is most
vulnerable two to three years after it begins, as initial
enthusiasm wanes and as other issues begin to distract
attention from the initiative. Evaluation can refocus
attention by charting the implementation of the “recipe”
(e.g., changes in patterns of teaching). Periodic studies
can provide data to help alter and fine-tune the strategy.
For example, this is the time to detect stresses on people’s
time and budget, before they lead to burnout. Regular
reports can help attract energy, money, and less tangible
forms of support (e.g., employer interest). Such reports do
not need to be purely good news: evidence of solvable
problems can be a great way to attract fresh resources.
By the way, the best time to start program evaluation
is immediately, in order to gather baseline data (the
“before” picture) to help map the progress and problems that
emerge in later years.
5. Create coalitions
to ensure that the program has all the ingredients needed in
the recipe for improving outcomes. On July 4, 1776,
Benjamin Franklin remarked, “We must indeed all hang
together, or most assuredly, we shall all hang separately.”
To get the resources needed for the full recipe, build a
coalition of people and interests from inside and outside
the institution, a coalition that focuses on the outcome to
be improved, not (just) the technology. Use that coalition
as a whole to fight for the resources that each element of
the coalition needs. Your institution may have a Teaching,
Learning, and Technology Roundtable (although perhaps not
with that title)—a broad-based group of faculty, staff, and
others that advises the chief academic officer on improving
education through the use of IT. Such a group can provide an
ideal setting in which to debate the above strategies and to
map a path to using investments in information technology to
achieve large-scale improvements in educational outcomes.
Interested in using this article
for strategic planning? You might also be interested in this
worksheet for your planning group. (TLT Group
subscription and username required.)
|
