Improving Outcomes of Higher Education: Learning from Past Mistakes

Productive Assessment l Professional Development l Planning: Visions, Strategies l Boundary Crossing
LTAs - Low Threshold Applications l Nanovation Bookmarks l Individual Members Resources

Flashlight Evaluation Handbook

This draft  was rewritten as a "Viewpoint" column that appeared in the 
January-February 2002 issue of

From mainframes in the 1960s to microcomputers in the 1980s, to the Web and Internet II today, higher education has hoped to use computers to improve educational outcomes on a large scale. By “outcomes,” I mean who can learn and what it is they can do by the time they’re finished with the academic program. By “large scale,” I mean improvements in outcomes for graduates of an entire degree program, institution, or nation.

After forty years of trying, the track record has been uneven, at best. The greatest success has been achieved in using information technology (IT) to teach technology-dependent content such as computer science, computer graphics in the arts, and data-intensive approaches to political science. A second, growing success has been to open access to education for students who couldn’t fully participate on campus before. And we have been able to enhance productivity in certain areas (e.g., by using computer spreadsheets instead of paper, pencil, and adding machine).

But what about the goal of improving teaching and learning? New purchases of hardware have often been justified by the claim that they would improve teaching/learning effectiveness in all fields.

Unfortunately, history has exhibited a grim cycle. Each time a new technology has come into vogue, its advocates begged the institution to buy it, so as not to fall behind. They promised that its use would improve learning. Yet five years later, a completely new technology had taken center stage. No large-scale improvements in the effectiveness of instruction had been achieved from the last purchase, but this new technology, advocates promised, would change everything! Whatever the question, “new technology” always seemed to be the answer.

Well, I don’t buy that theory anymore. For almost a quarter century I have been evaluating new uses of technology, both as a program officer with the Fund for the Improvement of Postsecondary Education (FIPSE) and later with Annenberg/CPB, and today as Director of the Flashlight Program. As early as the 1980s, we funded pioneers who tried to use “new” technology to improve outcomes for a degree program. Although we didn’t realize most of those hopes, I still believe that we could have succeeded, even in the 1980s, if we’d done a better job of implementation. Obviously such improvements are possible today, on an even larger scale. Yet we will surely fail again if we repeat the “common sense” policies that have sabotaged so many previous IT investments.

Why IT Investments Often Don’t Improve Educational Outcomes

At least three basic problems have dogged most attempts to translate technological investments into improvements in educational outcomes.

            1. Surrendering to rapture of the technology. For a variety of reasons, institutions and programs tend to focus just on the new technology itself. That’s bad. To put it metaphorically, you must have yeast to bake bread, but if you buy only yeast, you’ll never produce bread. Whether a program’s aim is to use technology to support learning communities, or better skills or inquiry, or an internationalized curriculum, the recipe will require more than hardware.  Other expensive ingredients include staff development and/or new staff; new assignments and course designs; more books in the library; altered marketing and advising; changes in roles and rewards; new organizational partnerships; and new internal coalitions. In the past, the technology siphoned money and attention away from the rest of the recipe.   

            2. Forgetting that the life span of many new technologies is far shorter than the time it takes to implement the recipe and improve educational outcomes. Those complex recipes are not “quick and easy”: assembling the ingredients takes a long time.  Meanwhile the technology is aging, and losing value. Long before outcomes have a chance to improve visibly, new technologies usually distract attention from the “old” improvement agenda.  Over the years, technology-related interests in improving outcomes such as programming skill, visualization (in the early days of videodisc), and collaborative learning (the computer conferencing systems of the 1980s) have risen, and fallen, and sometimes risen anew. Over the decades, waves of new technology have rippled across the surface of education but large-scale improvements in outcomes almost never had time to develop.

3. Trying to improve outcomes and save money by using tutorials and other forms of self-paced, interactive, branching courseware. This is one educational recipe for improvement of outcomes that hasn’t changed. It has been attempted with almost every new computing technology of the last four decades, from PLATO to the Web. These kinds of tutorials are always enticing: research has demonstrated that such courseware can dramatically improve outcomes, learning speed, and costs. But the problems of large-scale implementation have always proven insuperable: the short lifecycle of the courseware; the expenses of educational debugging of the many pathways; the hidden costs of altering the curriculum to take advantage of the courseware; the rigidity of the courseware in the face of new developments in the discipline and variations in students; the lack of rewards for authors; and the expenses of marketing and support.


How to Use IT Investments to Improve Educational Outcomes

Let’s learn from past mistakes.  Here are five strategies that should help higher education institutions and programs use IT to make major improvements in educational outcomes.

1. Begin with a long-term focus on a few selected outcomes and the educational activities needed to improve them. Which goals and strategies are worth pursuing for the seven to ten years that are needed to make large, visible improvements? Here are a few candidates: skills of inquiry and research; the ability to apply learning more successfully in the real world; skills of working in teams, communities, and organizations; international and intercultural understanding; skills of designing, composing, and creative work.. These outcomes can be relevant in almost any field. Other, equally valid goals are specific to particular fields or types of students. If a program succeeds in dramatically improving even one such outcome, the rewards of enrollment and financial support could be great.

The simplest way to choose among candidate goals is to select one that represents both a current strength and a current concern. For example, an institution or program might decide to improve skills of creative work, both because some of its best work is already being done in that area and because it is worried that competitors might leapfrog over it.

2. Choose technology that can contribute incrementally and cumulatively over the long haul. Suppose that a program has already been using technology to pursue such a goal for a couple of years.  Now discussions begin about a major purchase of a new technology that would be used by most faculty, staff and students.  Several questions can be useful in the choice regarding new technology: (a) Can the dollars spent on the new technology help the program make major progress toward improving the chosen outcomes, compared with other ways of spending the same money for advancing that educational agenda? (b) Is the new technology ready for mass use, and are operating costs, including support and hidden costs, acceptable? (c) How badly will the adoption of the new technology disrupt the current educational strategy? For example, how much curricular material and how many course designs would need to be discarded in order to take advantage of the new technology? (d) If this brand or product disappears, will the curricular material become unusable or will competitive products also run those same files?

3. Emphasize forms of instructional material that most faculty members find quick and easy to create, adapt, and share. Most courses require materials that instructors or tutors can inexpensively modify to match the needs of students, personal styles, and recent changes in their fields. By the same token, the instructional formats should make it easier for faculty to organize, edit, and share those incremental improvements in the materials and course designs so that, as a field, they can move forward as quickly as possible.

4. Track the progress of the strategy to get the data (and money) needed to stay on course. An educational initiative is most vulnerable two to three years after it begins, as initial enthusiasm wanes and as other issues begin to distract attention from the initiative. Evaluation can refocus attention by charting the implementation of the “recipe” (e.g., changes in patterns of teaching). Periodic studies can provide data to help alter and fine-tune the strategy. For example, this is the time to detect stresses on people’s time and budget, before they lead to burnout. Regular reports can help attract energy, money, and less tangible forms of support (e.g., employer interest). Such reports do not need to be purely good news: evidence of solvable problems can be a great way to attract fresh resources.  By the way, the best time to start program evaluation is immediately, in order to gather baseline data (the “before” picture) to help map the progress and problems that emerge in later years.

        5. Create coalitions to ensure that the program has all the ingredients needed in the recipe for improving outcomes. On July 4, 1776, Benjamin Franklin remarked, “We must indeed all hang together, or most assuredly, we shall all hang separately.” To get the resources needed for the full recipe, build a coalition of people and interests from inside and outside the institution, a coalition that focuses on the outcome to be improved, not (just) the technology. Use that coalition as a whole to fight for the resources that each element of the coalition needs. Your institution may have a Teaching, Learning, and Technology Roundtable (although perhaps not with that title)—a broad-based group of faculty, staff, and others that advises the chief academic officer on improving education through the use of IT. Such a group can provide an ideal setting in which to debate the above strategies and to map a path to using investments in information technology to achieve large-scale improvements in educational outcomes.


Interested in using this article for strategic planning? You might also be interested in this worksheet for your planning group. (TLT Group subscription and username required.)

Stephen C. Ehrmann is Vice President of the nonprofit Teaching, Learning and Technology Group and is Director of its Flashlight Program for the Study and Improvement of Educational Uses of Technology. This column appeared in the January 2002 issue of EDUCAUSE Review.  It is derived from a longer article available at Outcomes.html.  


Some Rights Reserved:  "Share it Forward" Creative Commons License by the TLT Group, a Non-Profit Corp.

PO Box 5643,
Takoma Park, Maryland 20913
: 301.270.8312/Fax: 301.270.8110  

To talk about our work
or our organization
contact:  Sally Gilbert

Search TLT

Contact us | Partners | TLTRs | FridayLive! | Consulting | 7 Principles | LTAs | TLT-SWG | Archives | Site Map |