Annotated Bibliography of Flashlight and other Relevant Assessment Articles

Productive Assessment l Professional Development l Planning: Visions, Strategies l Boundary Crossing
LTAs - Low Threshold Applications l Nanovation Bookmarks l Individual Members Resources

This page is an annotated bibliography of free articles and Web sites on Flashlight and on the larger subject of studying educational uses of technology. 
You may also want to look at our collection of Flashlight-related case studies.

Table of Contents for This Page


I. Summarizing Flashlight


II. Flashlight Articles on Methods


III. Case studies (F-LIGHT back issues)


IV. Resistance to Evaluation - Dealing With


V. Applying Flashlight Thinking to Specific Teaching and Institutional Policy Issues


VI. Other Evaluation-Related Sites Useful for Studies of Teaching, Learning, and Technology

+Other Examples of Studies Whose Findings Improved Local Educational Uses of Technology

+Interesting Evaluations of Educational Uses of Technology

+Methods and Tools for Evaluation of Courseware

+Studies of Costs; Methodology of Cost Studies

+Computer-Aided Evaluation and Classroom Research

+Computer-Aided Assessment of Learning

+Comparisons of Distance Education versus Campus-Bound Programs

+Other Resources Related to the Evaluation of Educational Uses of Technology

About the Flashlight Program and Its Approach to Studying Technology Use and Learning

Interview by Distance Educator with Steve Ehrmann, describing the history, achievements, and prospects of the Flashlight Program. (2001)

(NEW) "On the Necessity of Grassroots Evaluation of Educational Technology" (2000). The Flashlight Program's goal is to equip as many faculty members, administrators and students as possible with the skills and tools to guide their own uses of technology in support of learning. This article, originally published in Technology Source and recently added to the TLT Group's web site, explains why.

TLT/Flashlight Web site on learning space design and evaluation - This web site, created by Steve Ehrmann, summarizes the Flashlight approach to evaluating physical, blended, and virtual learning spaces (e.g., classrooms, course management systems, libraries, ...) .(2004- )

TLT Flashlight Web site on formative evaluation of ePortfolio initiatives. (2004- ).  Sample of subscriber materials, created by Steve Ehrmann. The Flashlight approach begins by identifying the different activities which the use of the technology is intended to improve. The center from which the Flashlight inquiry spreads is that set of activities: are they changing? where? why or why not? with what consequences? What are the strengths and weaknesses of eportfolios for carrying out these specific activities? how does the use of eportfolios alter the costs and stresses associated with these activities? and so on?

TLT/Flashlight web site on evaluation of institutional portals (2003- ). Sample of subscriber materials, created by Steve Ehrmann. Uses the same model described above for eportfolios, but around activities which the use of institutional portals is supposed to improve.

Ehrmann, Stephen C. (1999), "Studying Teaching, Learning and Technology: a Tool Kit from the Flashlight Program," Revision of article originally published in Active Learning IX (December 1998), pp. 38-42. This somewhat technical essay shows how to design Flashlight-style studies whose findings can show a program how its technology can foster better educational outcomes. The essay is illustrated with questions drawn from the Flashlight Current Student Inventory. The article explains key Flashlight concepts such as 'blob, 'triad,' and scenario. 

Ehrmann, Stephen C. (2000), "Studying and Improving the Use of Technology to Support Collaborative Learning: An Illustration of Flashlight Methods and Tools".  This essay tells the story of a study team at a fictional college that is developing a study to track and improve the use of technology to support collaborative learning by students and educational outcomes for graduates. It was written to illustrate Flashlight thinking in action and includes a sample survey created with Flashlight Online. This article may also be of special interest to Teaching, Learning, and Technology Roundtables, to teams preparing for accreditation self-studies, and to others interested in analyzing the use of technology to support large-scale patterns of educational improvement. 

Ehrmann, Stephen C. (1997) "The Flashlight Project: Spotting an Elephant in the Dark" . This article gives a general description of the design of the Flashlight Program, arguing that a tight focus is essential to seeing anything when studying teaching and learning with technology. The essay summarizes the history of the Flashlight Program and briefly describes the content and organization of the Flashlight Current Student Inventory.

Ehrmann, Stephen C. and Robin Etter Zuniga (1994), The Flashlight Project Planning Grant. Final Report This report on the original planning grant from the Fund for the Improvement of Postsecondary Education (FIPSE) describes the roots of the Flashlight Program's tools in more detail.

Frequently Asked Questions About Flashlight

Flashlight Articles on Methods

Ehrmann, Stephen C. (2002), "Evaluating (and Improving)  Benefits of Educational Uses of Technology." This draft chapter for a volume on cost analysis methods serves as a primer of issues to consider when evaluating the benefits of educational uses of technology.

Ehrmann, Stephen C. (2000) - narrated slideshow explaining the idea of a triad.   "Does [technology X] improve learning?" That's one of the most frequently asked questions about evaluation. It's also fallacious -- there is no direct relationship between any materials or tools of learning (computers, paper, etc.) and learning outcomes, or costs. This narrated slideshow explains why an investigation that seeks to link technology and outcomes must also study at least one more thing: the activity that makes use of the technology in order to produce the outcome. This combination of technology-activity-outcome is called a triad.

Chickering, Arthur and Stephen C. Ehrmann (1996), "Implementing the Seven Principles: Technology as Lever" AAHE Bulletin, October, pp. 3-6. This essay by Chickering and Ehrmann outlines the kinds of technology use that can help faculty and students implement Chickering and Gamson's "Seven Principles of Good Practice in Undergraduate Education." The Flashlight Current Student Inventory helps focus attention on technology uses for supporting good instructional practice.

Ehrmann, Stephen C., (1995) " Asking the Right Questions: What Does Research Tell Us About Technology and Higher Learning?" in Change. The Magazine of Higher Learning, XXVII:2 (March/April), pp. 20-27. This essay from Change Magazine gives a more brief overview of the evaluation literature on teaching, learning, technology and costs. Some of the key assumptions of the Flashlight Program are developed here.

Ehrmann, Stephen C. (1998) "How (not) to Evaluate a Grant-Funded Technology Project," in Stephen C. Ehrmann and Robin Etter Zuniga, The Flashlight Evaluation Handbook (1.0), Washington, DC: The TLT Group. This essay lays out some of the most common assumptions underlying evaluations of grant-funded technology projects and argues why all of them need to be rethought.

Ehrmann, Stephen C. (2000), "Finding a Great Evaluative Question: The Divining Rod of Emotion." Do all your potential topics seem (merely) 'interesting?' Maybe you need to look further. An intense sense of dread or excitement can signal that a topic is important enough, and uncertain enough, to be worth the effort to answer, as the examples in this paper illustrate. 

Ehrmann, Stephen C. (1998) "What Outcomes Assessment Misses" Many people assume that, to evaluate a technology-based innovation in education, it is necessary and sufficient to use uniform testing to discover gains in student learning. This recent essay argues that attending to outcomes is desirable but it is not always possible and it is almost never sufficient. The essay points more attention to outcomes that uniform tests miss, to the processes that produce the outcomes, and to the problems that programs encounter, some of which can be deeply revealing about the program's goals and strategies.

Ehrmann, Stephen C. (1997) "The Student as Co-Investigator," from Stephen C. Ehrmann and Robin E. Zuniga, The Flashlight Evaluation Handbook, Washington, DC: The TLT Group. Educators usually assume that the student's role in program evaluation is to answer questions. The reasons why the questions are being asked aren't explained for fear of biasing the answer. This essay argues that students should be involved with the design of evaluations of teaching, learning, and technology and that their own theories of using technologies for learning should be brought to the surface and put to the test. 

"Increasing Student Participation in Studies." A brief essay by Steve Ehrmann on how to increase response rates to surveys and interviews by demonstrating that the study is actually worth people's time and thought.

Other Resources for Studying Educational Uses of Technology

Resistance to Evaluation - Dealing With

Diagnosing and Responding to Resistance to Evaluation. These notes, drawn from a panel at the American Evaluation Association meeting in 2001, summarize some thoughts about uncovering and dealing with the natural distrust between the evaluator and the rest of the world.

"Frequently Made Objections (FMO's) to Evaluation, and Some Suggested Responses." The ideas in this list were drawn from participants in the Flashlight Leadership Workshop in July 2001 held at Syllabus/TLT Group Summer Institute in Santa Clara, California.

Applying Flashlight Thinking to Specific Teaching and Policy Issues
Most of our guides and sample materials are available only to the faculty, staff and students of subscribing institutions. The following materials are publicly available.

(NEW!) Distance and Distributed Learning: Examples of Flashlight Studies That Can Be Used to Improve Program Effectiveness

Getting More Value from Course Management Systems

Evaluating Institutional Portals (using data to get more educational value from portals). Free sample of TLT/Flashlight material.

"Using Technology to Make Large-Scale Improvements in
The Outcomes of Higher Education: Learning From Past Mistakes
," by Stephen C. Ehrmann. Observers have been expecting an imminent computer-enabled transformation of teaching and learning in higher education ever for almost 40 years.  Dr. Ehrmann argues that past effort have often been frustrated inappropriate strategies, not the technology itself. This brief article outlines a five part strategy for institutions, systems, and nations that want to use technology to make valuable, visible improvements in the outcomes of higher learning. This column was published in the January-February 2002 issue of Educause Review

Ehrmann, Stephen C. (2000), (Draft) "An Evaluation Plan for the Year".  This essay suggests a two-pronged strategy for helping an institution make best use of Flashlight methods and tools in order to improve education. Comments welcome!

For applications of Flashlight thinking and tools, see our collection of case studies

Other Evaluation-Related Sites Useful for Studies of Teaching, Learning, and Technology

We are continually expanding our site. Please send us URL's to include here, along with a brief description of the site.


Other Examples of Studies Whose Findings Improved Local Educational Uses of Technology

Ehrmann, Stephen C., (1999), "Asking the Hard Questions About Technology Use and Education". This article describes a set of evaluative questions -- study designs -- that institutions have used to good effect in studying the role of technology in the performance and costs of their academic programs. A revised version was published in the March/April 1999 issue of Change Magazine, pp. 24-29).

The goal of Teachers 2000 at Gloucester County College is to infuse technology into the education of future teachers. The program is divided into four student cohorts (learning communities), each mentored by a different faculty member. Evaluation has helped them improve learning communities, drop a telecourse that wasn't sufficiently helpful, and demonstrate that the program was helping to improve learning.

Penn State's Stat 200 course has been restructured to emphasize guided self-study by individuals and groups, improved assessment, and fewer but more interactive lectures. Early evaluation work is helping them fix a problem with the assessment process and pinpoint early cost savings.

This recent study of web-based distance education helped improve faculty development for distance learning.  Like Flashlight's tools, their survey focused on the seven principles of good practice in undergraduate education.


Interesting Evaluations of Educational Uses of Technology

The preceding citations are studies that report both on findings and on what influence the findings had for practice. The following evaluations strike us as worth reading for various reasons. You will probably find both ideas you can use and things you think you can do better. Please let us know if there are studies you think should also appear on this page.

Gerald Schutte of the California State University Northridge wrote this widely-discussed evaluation of his own 1996 statistics course taught both on campus and online. His report does not say whether the findings affected his teaching, but I suspect that they did. He found that the online students did better on exams, apparently because their study environment did a better job of encouraging collaboration than did the on-campus environment. The online students didn't like being left so much alone but the lack of faculty attention appears to have encouraged students to rely more upon one another.

This 1996 article by D. R. Newman, Chris Johnson, Clive Cochrane and Brian Webb at Queens University, Belfast, reports on a seminar split in two: half discussed face-to-face while the other half conducted their seminar online.  The authors document in their course what some other instructors have reported anecdotally: students doing more critical thinking in their online exchanges than the students talking face-to-face.

A study that compared two chemistry courses, one using collaborative inquiry and the other organized around interactive lectures, and the use of faculty from other departments to assess student learning:  J. C. Wright, S. B. Millar, S. A. Kosciuk, D. L. Penberthy, P. H. Williams, and B. E. Wampold (1998) A Novel Strategy for Assessing the Effects of Curriculum Reform on Student Competence, Journal of Chemical Education, v. LXXV, pp. 986-992 (August). Abstract on the Web at .


Evaluation of Courseware

    Georgia Tech has a set of generic tools for evaluating courseware projects (part of an even larger library of tools for managing the development of courseware projects. The evaluation tools can be found here . We especially liked the evaluation matrix .

    The United Kingdom's TILT Project has reached its end but the team left behind some good articles and references. During the three years of the TILT project, the Evaluation Group performed about 20 evaluation studies of teaching software in Glasgow University across a very wide range of subject disciplines. Some exercises have been single episode field trials, others have looked at computer aided learning material in use within a full degree course over two or more years. The Group employed a variety of instruments, three of the most important in recent practice being student confidence logs, short quizzes and learning resource questionnaires.

    Another older but still valuable reference is not itself on the Web. It's a book by Paul Morris, Stephen C. Ehrmann, Randi Goldsmith, Kevin Howat, and Vijay Kumar (1994) Valuable, Viable Software in Education: Case Studies and Analysis. New York: Primis Division of McGraw-Hill. This study described the kinds courseware and other nationally distributed software that have had a widespread, long term influence on the curriculum, and why. One of the key findings: the destructive impact of the rapid progress in computers and operating systems. A summary of some of the findings is on the Web.


Studies of Costs; Methodology of Cost Studies

    The Flashlight Program has its own Cost Analysis Handbook, which includes instruction on methods, examples, and seven detailed case studies.  By cost study, we mean an analysis of how educators use time, money, space, and other scarce resources as they try to educate people; the aim of the studies of most interest to us is to help the people involve "unstretch" those resources - to make key processes easier, more fulfilling and more rewarding.  One of the most overlooked, and most important, reasons to do such studies is to prevent burnout.  The Flashlight Cost Analysis Handbook teaches its readers how to analyze their own use of resources in particular activities, in order to improve those activities; we also periodically offer online workshops on how to do such studies.  Check our calendar to see if one is coming up.

    Good cost studies are rare and their methods are often similar.  We work closely, and want to work more closely, with others doing cost studies, and developing such methodologies, all over the world.  Here is our list so far, in the order we found these resources.

    The Western Cooperative for Educational Telecommunications, with support from the Fund for the Improvement of Postsecondary Education, is now at work on the second edition of its Technology Costing Methodology (TCM) project, which extends previous work on costing done by NCHEMS. TCM includes a handbook, case book, and tools; TCM focuses mainly on comparing the costs of two or more types of instructional strategies.

    SCALE at the University of Illinois, Urbana, recently published evidence from nine "Efficiency Projects" that were SCALE’s focus in the 1997-98 academic year. The Efficiency Projects were specifically aimed at using asynchronous learning networks (ALNs) to achieve higher student/faculty ratios, without sacrificing instructional quality. The study concentrates on data amassed for the fall 1997 semester. Evidence was collected on the cost side, for ALN development and delivery, and the performance/attitude side, from both student and faculty perspectives. The study supports the view that when a sensible pedagogic approach is embraced that affords the students with avenues to communicate about their learning, ALN can produce real efficiency gains in courses without sacrificing the quality of instruction.

    The Pew-funded Center for Academic Transformation supported a large number of projects in the United States; each project redesigned a large (often introductory) course in ways that trimmed costs while often improving outcomes. Their Web site contains links to a growing number of reports from their projects.  For example, Carol Twigg's monograph,"Innovations in Online Learning: Moving Beyond No Significant Difference," summarizes many of the lessons of their projects for cutting costs while maintaining or improving outcomes.

    The work on costs is becoming increasingly international.  Flashlight is working closely with efforts in other countries as well. One such program is the project at Sheffield Hallam University in the United Kingdom that has been developing activity-based methods of estimating the costs of network learning. Click here for their Web page of other efforts on estimating the costs of technology use.

    To repeat, we would like to add other references and sites to this list, including studies, methodologies and resource sites in other countries. Please send your suggestions to


Computer-Aided Evaluation and Classroom Research

    The Student Assessment of Learning Gains site at the University of Wisconsin helps faculty craft their own online studies of the effectiveness of different elements of their courses.

Computer-aided assessment

    This site in Great Britain is one source of information on the use of computers for assessing student learning (where "assessment" refers to gauging what a student has learned, as opposed to "evaluation," by which we mean studies of how well educational program is working. Assessment data is one useful input for evaluations.)

    In a March 2000 article in Education Policy Analysis Archives, Russell and Haney argue that, for school children accustomed to using computers, state exams limited to paper and pencil seem to underestimate achievement. Their research raises interesting questions for course-based college level assessment.

Comparisons of Distance Education versus Campus-bound Programs

    North Carolina State completed this study ("Project25") comparing Web-based courses and campus courses in early 1998.

    Tom Russell has assembled abstracts of some 300 studies to refute the often-heard contention that "distance learning can't possibly be as good as campus programs." He was looking for studies that showed, at worst for distance education, 'no significant difference' in quality of outcomes when compared to programs on-campus. As you'll see, he found hundreds of such studies.

Other Resources Related to the Evaluation of Educational Uses of Technology

Web Center - Learning Networks Effectiveness Research. Developed in conjunction with the Sloan-funded Asynchronous Learning Networks projects.

Evaluation results - Program in Course Redesign: summary of Pew self-study of 30 projects redesigning large enrollment courses. Findings on costs and effectiveness.

The Coalition for Networked Information (CNI) has a long-term project on assessing the academic networked environment.  The McClure-Lopata volume and the case studies deserve a special look.

Other Resources on Assessment and Evaluation

Internet Resources for Higher Education Outcomes Assessment (North Carolina State University)

Field-tested Learning Assessment Guide (FLAG) - an indexed collection of assessment tools for faculty (especially faculty in math, science, engineering, and technology

Nine Principles of Good Practice for Assessing Student Learning (AAHE)


Some Rights Reserved:  "Share it Forward" Creative Commons License by the TLT Group, a Non-Profit Corp.

PO Box 5643,
Takoma Park, Maryland 20913
: 301.270.8312/Fax: 301.270.8110  

To talk about our work
or our organization
contact:  Sally Gilbert

Search TLT

Contact us | Partners | TLTRs | FridayLive! | Consulting | 7 Principles | LTAs | TLT-SWG | Archives | Site Map |