|
Flashlight Evaluation Handbook Table of Contents
l
Faculty
Development/Support
Subscriber Materials
These
materials are for use only by institutions that subscribe to
The TLT Group, to participants in TLT Group workshops, and
to invited guests. The TLT Group is a non-profit whose
existence is made possible by their subscription and registration
fees. if your institution is not yet among
our subscribers,
we invite you to
join us, use these materials, and help us
continue to improve them! If you have questions about
your rights to use, adapt or share these materials, please
ask us (info @ tltgroup.org).
What kinds of evidence can help guide improvement of
faculty development/support services whose goal (in whole or
in part) is to improve teaching and learning with
technologies? Here are some suggestions, guided by the
Flashlight Approach.
-
Ideas and Concepts for using data to improve faculty
development/support
-
Cases - Examples of useful studies of faculty
development/support
-
Flashlight tools and services
I. Ideas and Concepts
for Using Data to Evaluate and Improve Faculty
Development/Support
If you're going to evaluate a program that
helps faculty use technology to improve teaching and
learning, a few of the following ideas might help you design
your study. The most overwhelming fact about designing such
a study is how many things, potentially, you might possibly
study and how few you can actually afford to take a look at.
This section of the Flashlight Guide is intended to help you
consider some of your options.
1.1 Are the desired outcomes the same
for all faculty? do all faculty use the service in the same
ways?
For faculty development and support
programs usually have two faces, each of which requires its
own approach to evaluation (using data to improve the
program's effectiveness)
- Uniform impact: to some
degree, all participating faculty are being helped for
the same purpose with the same kinds of expected
outcomes, e.g., satisfaction, skill at using a course
management system. And they all use the service in the
same way (e.g., they all attend a workshop on how to
create syllabi using the course management system in
order to learn how to do that). Evaluation begins by
developing a way to assess the impact or value-added,
e.g., how good are they at using the CMS to create a
syllabus? or how much better have they become as a
result of the training?
- Unique uses: to some degree,
each faculty member is being helped to do different
things (e.g., some faculty are improving presentation
skills while others are learning how to facilitate
online discussion and still others are working on topics
of their own choosing). Evaluation begins with case
studies of individual faculty to discover how that
individual has benefited from the support. Only after
each case has been documented does the evaluation look
for patterns across subjects.
If the program's emphasis is split half
and half between these two emphases, then the evaluation
effort ought to be split half and half as well. [For more
on these ideas and their implications for assessment of
impact and evaluation of process, subscribers can consult
the
Flashlight Evaluation Handbook.]
1.2 What's important to study?
Issues likely to be pertinent to
evaluation of a faculty development or support program
depend on its goals. Some are relatively generic; they could
be featured in the evaluation of almost any program:
- In what ways has your service helped
faculty teach more "effectively?" What does
"effectiveness" mean? One way to become more specific
about that is to use the
seven principles of good practice (or ideas of
comparable general importance in evaluating quality of
instruction). Flashlight Online contains several
hundred questions for students about these kinds of uses
of technology. The Flashlight Faculty Inventory contains
parallel questions aimed at faculty.
- Does the scale of your service fit
the scale of the needs? or is the service providing
service to only a small fraction of the faculty who want
and need it?
- Why are some faculty not using your
service in a given year? (The reasons may be different
for different people, and provide clues about how to
increase the level of use, if that's one of your goals).
- How effective is your service
relative to other services at other institutions with
comparable resources and comparable needs?
Other potential foci for your study might
be more specific to your service's intent and context.
- If, for example, the program uses
student technology assistants, part of the inquiry might
well focus on whether students and faculty are learning
in distinctively valuable ways because of their work
together
- If the program is focusing on
'broadcasting' low threshold applications and activities
to large numbers of faculty, part of the evaluation
would focus on what fraction of the messages are
reaching various members of the faculty, and the issues
that affect whether each message is heard and acted
upon.
- If your program has a specific goal
for improving teaching/learning in a specific way (e.g.,
online learning; writing across the curriculum; student
and faculty research), then you'll want to focus a
significant part of your effort on those outcomes and
how you hope to be achieving them.
Make sure your study design also addresses any concerns
that you, or your stakeholders may have about the program.
- For example, does the program attract mainly faculty
who are already excellent teachers, without affecting
the skills of typical instructors?
- Does the program have any influence on the quality
of the teaching and its likely outcomes? or is it just
helping faculty teach in the old ways (probably with
unchanged outcomes) but using new and expensive tools to
do so (e.g., putting their yellowing course notes onto
PowerPoint slides)?
One way to make sure that your study addresses such
concerns is to work closely with your stakeholders in
designing the study.
1.3 Time Frame
- Most evaluations focus on the past,
sometimes several years worth of the past, in order to
learn lessons for the future.
- Some evaluations might focus just on
the present, gathering data that can be used as a
baseline and aid to future planning. Such a study might
focus on how all faculty have recently been improving
their teaching (regardless of whether they have used
your service). Such findings could be used both to
adjust your strategy and as a point of comparison when
you do a similar study in a year or two. Such
evaluations can also function as needs analyses, to help
create the case for fresh budgets and new energy to be
applied to current problems.
1.4 What Else do You Need to Know? Why?
- Think hard about why you're doing the
study. For example, you might decide that it's
important, in order to justify a budget, to get a vivid
sense of how much faculty have learned to value your
service. If you suspect they do value it, you might
offer a survey that briefly names and describers your
services and then asks the respondent a question or two
like these:
- If you were trying to persuade
someone to consider a faculty position at this
institution, would you mention this service? What
would you say about its value?
- The budget provided for this
service has been reasonably stable for the past few
years. But imagine for a moment that there was a
real question about doubling it, keeping it the
same, or eliminating it. The provost has asked a
few faculty, including you, for an opinion. What
would you say and what examples from your own
experience with our service would you mention to
make your case about the future budget for this
service?
We're collecting successful formative
evaluations of programs that help faculty use technology in
their teaching. We're especially interested in uses of data
that guide efforts to improve faculty development or
support, and/or help it become more cost-effective. If you
know of a study that might be linked here, please e-mail
ehrmann@tltgroup.org
)
Instructional Design at Washington
State is helping improve distance learning courses while
cutting costs
Some institutions see "instructional
design" as an expense that may need to be cut in times
of tight budgets. But Tom Henderson, Gary Brown, and
Carrie Meyers found in a series of studies at Washington
State University that up-front instructional design both
improved teaching-learning practices in courses and also
helped control development and delivery costs. These
findings are helping define policy and practice at WSU,
one of the founding institutions of the Flashlight
Program.
Click here to see what they discovered.
Faculty-Librarian Partnerships
Effective in Developing Information Literacy in Minnesota,
Dakotas
Project
JSTOR was a three-year grant initiative from 1999-2002,
supporting 35 public and private colleges and
universities in Minnesota, North Dakota and South
Dakota. Its goals: strengthen digital library use and
scholarly research, particularly through the acquisition
and use of the JSTOR
digital library collection. Through the program, 20
colleges and universities became participating JSTOR
members, joining a network of 15 other member
institutions in the region. The Flashlight Program
conducted an external evaluation which included
extensive interviews and surveys of program
participants.
Perhaps
the most significant finding was the power of
Faculty-Librarian Instructional Partnership (FLIP)
grants. Small grants helped at least one faculty member
and a librarian at an institution to team up and improve
a course's ability to develop information literacy among
students. The grants seemed to help advance a new
working relationship between faculty and librarians,
while triggering substantial institutional increases in
the use of the JSTOR collection and other online
resources.
Click here for links to a fuller
description of Project JSTOR and the full text of the
evaluation.
One
role of faculty support at SE Missouri State is to
prepare faculty to teach online. David Starrett and
Michael Rodgers studied who was being served by their
institution's online courses. The University's
investment in helping faculty use technology had been
justified in large part by the hope that the resulting
courses would serve students across the University's
service area, students not close to campus. Their data
indicated that online courses were serving precisely
these students.
Click here to see a summary of their study, and an
e-mail address to get more data.
University of Missouri, St. Louis case
study
Cheryl Bielema reports that
these reports (done annually) are used as fodder for
planning faculty development.
Relevant Flashlight tools and resources
that come with TLT Group subscriptions include
Flashlight Online,
the Flashlight
Faculty Inventory, the
Flashlight Evaluation
Handbook, and the
Flashlight Cost Analysis Handbook. We also design
special purpose assessment tools, and offer subscribers the
opportunity to peer review one other's instruments and study
designs, and then publish them to other subscribers.
Flashlight consultants also do external
evaluations of faculty development and support programs, and
provide suggestions and help for internal evaluations. If
you'd like to chat about the possibilities, please send
e-mail to
online@tltgroup.org. We're especially interested in
working with systems, consortia and institutional
associations; we'd help develop rubrics and surveys to that
the benefits, challenges, methods and costs of each program
could be anlyzed in relation to other programs.
|
