Introduction l Choosing a Topic l Concepts l Methodology l Flashlight Tools
Topical Guides l Culture of Evidence l Resources l Password

These materials are for use only by institutions that subscribe to The TLT Group, to participants in TLT Group workshops, and to invited guests. The TLT Group is a non-profit whose existence is made possible by their subscription and registration fees. if your institution is not yet among our subscribers, we invite you to join us, use these materials, and help us continue to improve them!  If you have questions about your rights to use, adapt or share these materials, please ask us (info @ tltgroup.org).

This Handbook is continually being enriched in collaboration with our subscribers. We would love to incorporate your case studies, training materials and favorite resources.  Please also send us suggestions for how to make this material clearer and more useful for you and your colleagues. This Handbook has been created with support from TLT/Flashlight subscriber institutions and is for the use of staff and students at those institutions. That's why many of the sections require a username and password. To see if your institution is a subscriber and to get your institution's username/password, consult our list of current subscribers.

1. Introduction

1. Acknowledgements and Preface

2. Why Use Inquiry to Improve Practice?

   • Self-test: "Scholarship of Teaching, Part I"

3. Inquiry as a part of a Counter-Intuitive Strategy for Improving Program Outcomes

4. The Flashlight Approach to Evaluation: A Summary (pdf for workshop handout)

5. How to find what you need in this Handbook

6. Flashlight Oath

2. Choosing a Topic Worth Studying. If you haven't decided what to study yet, one or more of the following might help you choose or focus your study. 

   1. Why Focus (and why is this called the "Flashlight" Program?)

   2. Focus on What You Fear?

   3. Focus on Students' Objections to How You Are Asking them to Learn

   4. Topics for Studies (ideas for proposals, dissertations, and grant proposals)

   5. Levels of Analysis
       

3. Concepts to organize your study:  It's not possible to understand or improve the influence of technology on outcomes without studying how the technology is used (the "activities") and why those activities are happening as they do. Click here for an introduction to triads.

1. Types of outcomes and their assessment - some basic categories (including "uniform impact" and "unique uses" as important categories for conceptualizing and assessing outcomes)

2. Activity-based study designs:

   • Using Chickering and Gamson's Seven Principles of Good Practice to Evaluate (and Improve) Instruction Across Disciplines, Modes

   • Scanning studies

   • How well does the technology support the activity?

   • Proof (cause and effect) and Tracking Studies

   • Diagnostic studies - finding barriers to success in order to lower them

   • Tracking studies

3. Technology studies (e.g., debugging, usability studies) - this section has not yet been written.

4. Methodology. Here are some hints about doing studies. All of them should be useful, but sections C-F are more useful for program evaluations and other studies done by teams.

   1. Methods for gathering data (surveys, interviews, etc.)

      • Matrix surveys

      • Rubrics

   2. How to "Compare Apples and Oranges" - an essential step in summative evaluations of educational uses of technology

   3. Increasing response rates to your inquiries (e.g., surveys) (free sample of subscriber material!)

   4. Pretesting Your Surveys and Your Final Report

   5. Getting the Right People Involved in Your Study

   6. Dealing with resistance to your study (free sample of subscriber material!)

   7. Dangerous Discussions about "Assessment" - using surveys and other techniques to shine a flashlight on confusors and objections that can sometimes fester beneath the surface of discussions about assessment (for people involved with accreditation, assessment training, student course evaluation, faculty tenure discussions, and related topics)

      Frequently Asked Questions (and Frequently Made Objections) regarding evaluation and assessment (and some responses)

      • Confusors that can derail discussions of an evaluation

   8. Tricks of the trade (under construction)

      • Embedding video clips in your Flashlight Online survey - why and how

5. General Purpose Tools for Inquiry

   1. Flashlight Online and (coming soon) Flashlight Online 2.0 (includes a searchable version of the Flashlight Current Student Inventory)

      Flashlight Current Student Inventory (item bank included in Flashlight Online; also available separately)

      • About the CSI

      • Table of Contents of the CSI

      • CSI item bank and site license

      • Validity

   2. Flashlight Faculty Inventory

   3. Flashlight Cost Analysis Handbook
       

6. Topical Guides for Studies. Each of these chapters embodies the Flashlight approach of starting by focusing on a) how people choose to use technology to produce outcomes (desirable and undesirable), and b) why they make those choices.  The more of these sections that you read, the better idea you should get of the "Flashlight approach" to productive evaluation and assessment.

   1. Accreditation Self-studies 

   2. Blended courses (see "Hybrid courses" below)

   3. Clickers (see "Student Response Systems" below

   4. Course Management Systems

   5. Course Assessment Techniques (CATs) ("action research" in courses).  Great material for workshops to introduce faculty members to uses of student feedback to improve learning in their courses.

   6. Digital Writing Across the Curriculum

   7. Distance Learning (see Online/Distance Learning below)

   8. Electronic portfolios - 1) planning and formative evaluation of ePortfolio initiatives; 2) how instructors can use student feedback to figure out how to get even more value from the use of ePortfolios in their courses.

   9. Faculty Development/Support Services (i.e., services faculty can use to improve their teaching with technology - workshops, consultations, LTAs, mini-grants, etc.)

   10. Grant-funded Projects, How (not) to Evaluate Them

   11. Hybrid courses

   12. Information Literacy - slideshow on assessment of information literacy programs

   13. Laptop Initiatives - Guide describing how to use formative evaluation to increase your chances of success, reduce risks

   14. Learning spaces (e.g., high-tech classrooms, libraries, and other campus facilities; virtual learning spaces)

   15. Online discussions and team projects in courses - using student feedback to remove barriers to 100% participation rates in online discussion, collaboration

   16. Online/distance learning programs

   17. Pilot tests of new technologies and new services. This evaluation of a pilot test of a new technology for possible enterprise deployment illustrates the Flashlight method.

   18. Portals

   19. Presentations, Feedback to Improve

   20. Scholarship of Teaching and Learning (see also "Teaching")

   21. Science, Technology, Engineering and Math (STEM) Learning, Using Evaluation to Improve Technology Use in. The examples in this tutorial are specific to math and science, but the concepts are generic.

   22. Second Life (see "Virtual Worlds" below)

   23. Student Course Evaluation

   24. Student Response Systems (also known as "clickers" "button boxes")

   25. Time and money - using data to avoid putting too much stress on staff and budgets; cost modeling (Flashlight Cost Analysis Handbook)

   26. Virtual Worlds such as Second Life -formative evaluation of academic experiments, programs and services in a virtual world

   27. Writing (see "Digital Writing Across the Curriculum," above)

7. Strengthening an Institution's Culture of Evidence. What steps can institutions take to help faculty members and other staff gather and use evidence in order to improve practice?

   • Introduction

   • Case studies

8. Techniques for promoting use of Flashlight Online to support a culture of evidence

9. Resources

   • How to help more people at your institution to use Flashlight tools and ideas to do productive studiesl

   • Educational materials (workshop designs, slides, eClips) from The TLT Group for helping faculty and staff learn about how to do productive studies, especially with Flashlight tools (under construction)

   • Frequently-Asked Questions about Flashlight

   • Flashlight Online templates (model surveys to modify and use)

   • Anthology of related articles on methods (being assembled)

     • Asking the Right Question - What Does Research Tell Us About Technology and Higher Education

   • Selected Web resources for assessment and evaluation

   • Evaluation humor -cartoons and stories useful in lightening the mood and raising important questions in evaluation and assessment workshops.

PO Box 5643
Takoma Park, Maryland 20913
Phone: 301.270.8312/Fax: 301.270.8110  

To talk about our work
or our organization
contact:  Sally Gilbert

Contact us | Partners | TLTRs | FridayLive! | Consulting | 7 Principles | LTAs | TLT-SWG | Archives | Site Map |