F-LIGHT

E-Newsletter for the Flashlight Program

• FIPSE grant funds a Flashlight approach to course evaluation; Position opening for Research Coordinator
• Improving Information Literacy: Flashlight's Evaluation of Project JSTOR
• Research ideas: Validating surveys and interviews about activities
• Upcoming Flashlight-related Webcasts and Conferences  
• New ways to use benefits of Flashlight Network membership
• New Flashlight subscribers
• Learning More About Flashlight( free demo accounts), The TLT Group and F-LIGHT

FIPSE Grant funds BETA: Flashlight Course Evaluation Project; Position Opening for Research Coordinator 

The US Department of Education's Fund for the Improvement of Postsecondary Education (FIPSE) recently announced a major grant to The TLT Group to support development of model software, templates, survey items, training materials, and policies.  One of the major goals: to create evaluation processes that can simultaneously: 

• enable faculty to ask good, hard questions about their own teaching practices and other elements of a course that affect its quality, 
• enable the institution to gather such information about groups of courses;
• enable the institution to gather data that can be used in promotion and tenure decisions.

The project will test processes of inquiry that an institution can use to develop survey templates that correspond to its faculty's ideas about good practice, while also factoring in research findings on the kinds of courses that most often help students learn well. 

The grant was just received and a search is on for a Research Coordinator to help lead the project.

Project JSTOR was a three-year grant initiative from 1999-2002, supporting 35 public and private colleges and universities in Minnesota, North Dakota and South Dakota. Its goals: strengthen digital library use and scholarly research, particularly through the acquisition and use of the JSTOR digital library collection. Through the program, 20 colleges and universities became participating JSTOR members, joining a network of 15 other member institutions in the region.  The Flashlight Program conducted an external evaluation which included extensive interviews and surveys of program participants.

Perhaps the most significant finding was the power of Faculty-Librarian Instructional Partnership (FLIP) grants. Small grants helped at least one faculty member and a librarian at an institution to team up and improve a course's ability to develop information literacy among students. The grants seemed to help advance a new working relationship between faculty and librarians, while triggering substantial institutional increases in the use of the JSTOR collection and other online resources.

Click here for links to a fuller description of Project JSTOR and the full text of the evaluation. 

Validating Surveys of Activities

Knowing what people actually do with technology is key to a) understanding whether technology investment is helping improve learning outcomes, b) understanding how technology investment is affecting total program costs. 

The typical methods for discovering what people are doing, and why, are surveys and interviews.  In other words, we usually rely completely on surveys and/or interviews to estimate both the educational outcomes and program costs resulting from investments in technology.  But can we really trust these data?

The first issue is honesty of respondents. Flashlight tools such as Flashlight Online surveys and the Cost Analysis Handbook rely heavily on these approaches because the typical study is about educational practices, not people. Because the typical respondent is not being asked how well the person's teacher has been doing, it should be obvious that the student's grade won't be influenced. The typical cost study is designed to make the respondent's work more rewarding, not to consider whether the respondent's job will be eliminated.  So we usually assume that the respondent will be honest enough that the average response is trustworthy.  True?

And, if we can trust respondents' honesty, is that enough? What does current literature say about the accuracy of honest responses to these kinds of questions? 

If the literature isn't yet helpful, perhaps some new methodological studies need to be done.  Are there ways to validate such survey and interview results in studies of education and/or costs? Are there other strategies that could complement or replace surveys or interviews for estimating the frequency of key activities (e.g., frequency of, or time spent in, student-faculty communication). For example, suppose students in a course are surveyed about the frequency of student-student interaction, and those same students are also observed (video? participant observation in informal settings? records of e-mail)  Might we use such supplementary observation to test various ways of framing survey or interview questions? or at least to check on the value of the survey or interview question?  

This kind of attention is even more important in cost studies than in studies of educational outcomes, simply because cost studies have received less such attention. The typical activity-based cost model uses no statistical tools to estimate the validity of respondent estimates of time spent on activities, perhaps because such studies are usually done by people with no statistical training.

Steve Ehrmann

PS Check out our growing list of ideas for dissertations and grant proposals.

Upcoming Flashlight-Related Webcasts and Conferences

Webcast on Cost Analysis and the new Flashlight Cost Analysis Handbook (co-sponsored by NACUBO), Thursday, Dec. 5, 2002, 3 pm ET on Horizonlive

For many people "cost analysis" equals "threat": someone is thinking about cutting a budget or terminating programs and wants an excuse.  Flashlight's approach is tailored to institutions and programs that are trying to reduce the risk of staff burnout and budget over-runs by analyzing how they are currently using people's time, budgets, and space. 

On Dec. 5, this brief webcast (30-45 minutes) will discuss the basics of activity-based cost models and summarize key ideas in the second edition of the Flashlight Cost Analysis Handbook, being published in early December.  

The webcast, co-sponsored by the National Association of College and University Business Officers (NACUBO) will feature an interview with Susan Tucker, author with Jamie Kirkley of "Mainstreaming Cost Analysis into the Development and Evaluation of Distance Education Products ," a chapter in the new Handbook. Click here for information on registering for this free event.

National Learning Infrastructure Initiative Annual Meeting, New Orleans, January 26-28, 2003

The NLII is dedicated to fostering use of technology to make fundamental improvements in higher education. As part of an ongoing collaboration among NLII, the Coalition for Networked Information and the Flashlight Program, Steve Ehrmann will take part in a featured session on transformative assessment.

14th Annual International Conference on College Teaching and Learning, Jacksonville FL, April 1-15, 2003

Steve Gilbert will be a featured speaker at the conference.  The TLT Group is also helping to manage one track of the conference on mass engagement.

The TLT Group is also doing a full-day preconference workshop, "Teaching and Learning with Technology for Almost Everyone: Strategies and Materials for Mass Engagement and Institution-Wide Improvement."

Teaching Well Using Technology , Notre Dame University, South Bend, IN, July 14-17, 2003

This Institute, co-sponsored by Notre Dame and The TLT Group, will help a select group of faculty developers and technology consultants learn and implement a seven step workshop model that can help faculty take advantage of technology to make fundamental improvements in courses.  Gathering data is key both for faculty improving courses and for staff running these kinds of workshops. Steve Ehrmann will lead the session on assessment.

For details on this and other Flashlight and TLT Group events, both face to face and online, keep an eye on The TLT Group calendar. 

New ways to use benefits of Flashlight Network Membership

There are many ways in which institutions can use the benefits of a Network subscription. Even when the subscription was initially purchased by just one office or grant-funded project, for example, the entire institution is entitled use the benefits: 

• faculty and students can use the free survey system;
• the business office can do cost studies;
• self-study teams can use the item banks;
• consultants can be engaged to plan external evaluations for grant proposals;
• templates can be used to gather information to help the CAO to make budget decisions;
• Tech support units can do needs analyses and service evaluations;
• Course management systems and portals can be evaluated; 
• avoiding staff burnout and budget over-runs;
• people doing assessments can find collaborators at other institutions and share surveys and even data...

We've assembled some of the more important ways to use Network membership here. (Most of the ideas also apply to Flashlight Tool Series subscriptions, too.)  Think of it as a menu that could help you decide whether to subscribe or, if you're already a subscriber, how to use the benefits. If you have other ideas (or questions), please send e-mail to flashlight@tltgroup.org 

Flashlight Subscribers  

Flashlight has continued its dramatic growth. Five years ago, Flashlight had 5 subscribing institutions.  Currently, over 280 institutions and projects subscribe annually to Flashlight tools/services. Among recent subscribers: Central Baptist College, Furman University, Oakwood College, Rider College, and the University of Southern Colorado. 

Our Web site has fallen behind again, but, if you'd like to see if your institution is one of approximately 440 institutions and projects around the world that are subscribers or licensees of Flashlight tools, please visit our list of participating institutions.

Top of Page

The Flashlight Program for the Study and Improvement of Educational Uses of Technology is part of the non-profit TLT Group, Inc. Flashlight was created by Annenberg/CPB in 1993. The TLT Group is headquartered in Washington DC (but moving to the Maryland suburbs on January 1, 2003) with additional staff in Texas, and Senior Associates around the world. Our thanks to Washington State University for their many ways of supporting Flashlight, including developing and administering Flashlight Online and providing the listproc for distribution of F-LIGHT notices.  We are also grateful to St. Edward's University for extensive support for Flashlight; to the corporate sponsors of The TLT Group; and to funders whose dedication to higher education has aided the TLT Group's work, including Annenberg/CPB,  Atlantic Philanthropic Service, the Andrew W. Mellon Foundation, the Fund for the Improvement of Postsecondary Education (FIPSE), and the National Science Foundation.

If your institution needs to get a better look at Flashlight Online, the best way is for someone at your institution to request a temporary, free demonstration account.  Send e-mail to Flashlight@tltgroup.org with the header "Free Demo Account" to ask for details. One account per institution, please.

If you know someone else who would like to be alerted to new issues of F-LIGHT, please suggest that they send e-mail to LISTPROC@LISTPROC.WSU.EDU with the one line message
   SUBSCRIBE F-LIGHT (the subscriber's first and last name)

Do the same for yourself if you have changed e-mail addresses.

To stop receiving the bulletin about F-LIGHT, please send e-mail to LISTPROC@LISTPROC.WSU.EDU with the one line message
   SIGNOFF F-LIGHT

Top of Page

Number of visits to this page: