|
Flashlight
Evaluation Handbook Table of Contents
These
materials are for use only by institutions that subscribe to
The TLT Group, to participants in TLT Group workshops, and
to invited guests. The TLT Group is a non-profit whose
existence is made possible by their subscription and registration
fees. if your institution is not yet among
our subscribers,
we invite you to
join us, use these materials, and help us
continue to improve them! If you have questions about
your rights to use, adapt or share these materials, please
ask us (info @ tltgroup.org).
Second Life
is a virtual world that some academics are using, often
experimentally, for academic purposes: meetings, student
projects, and other applications. We're applying the
Flashlight approach to developing questions that
subscribing institutions can use to guide and shape their
academic experiments and programs in Second Life: needs
assessment, formative and summative evaluations.
The Flashlight approach focuses on the
activities for which you are using Second Life. (Click
here for some notes on possible uses of Second Life,
some of which are developed in the sample survey below). For
each activity you want to study, ask yourself these
questions:
-
If you wanted to carry out this
activity, and weren't using Second Life, what learning
space would you use (e.g., face-to-face interaction?
conference call? web conference? software for creating
simulations? web page with photos or streaming video?).
-
Is it possible and useful to compare the
activity when carried out in Second Life (either
directly, or by asking your Second Life users to compare
this experience with their experiences in the other
learning space)?
-
What are your best hopes or goals for
this use of Second Life? What worries you or others
about this use of Second Life?
-
What incentives, disincentives,
experiences, or barriers might help or hinder people
from using Second Life for this purpose? For example, is
it important to discover whether the people you want to
include in Second Life already use it extensively? Have
already tried it and rejected it? How long it takes
novices to become sufficiently adept to use it for your
purposes?
-
Remember in developing your survey or
interview questions that the activities are, to some
degree, separable from the facility. For example, a
student project done in Second Life may be intensely
engaging because it was a great project. Would it have
been more, or less, engaging if it been created in a
different facility or with different tools? More or less
difficult? More or less costly?
-
Start evaluating now, even if you aren't
yet using Second Life. Sound crazy? Not if you're
focusing on the activity rather than just on the
facility in which it's to be carried out. No matter what
you're using, or considering using, Second Life for,
chances are that you or someone else is already doing
this. Or that you've tried to do it. By beginning
your study immediately, you'll learn more about what can
be rewarding about the activity, and barriers that make
it difficult. Many academic activities fail on a new
technology for the same reasons that they fail with
older technology. By studying the nature of success and
failure of this activity, you're laying a foundation for
a more successful use of Second Life, and for greater
success with this activity.
Needs Assessment and Formative Evaluation
As we implied in #6 above, needs assessment
and formative evaluation are closely related. For
example, if you're considering using Second Life as a
facility where students can create designs for buildings,
and then walk around in them, you could begin a needs
assessment by looking into existing learning spaces
(facilities/software) that make that activity possible. What
do faculty and students see as the strengths and weaknesses
of those facilities? Second, look for people at other
institutions who have already tried using Second Life for
your activity. If they haven't already done an evaluation,
offer to work with them to help devise surveys, interview
guides or other ways of gathering data from their users. Or
do a pilot experiment yourself. In either case, ask users
how suitable Second Life is as a facility for this activity,
and how it compares with some other learning space
(facility/software) that could be used for a similar
purpose. Costs can be studied in the same way. For an
example of an evaluation of such a pilot study (of Acrobat
Connect Pro, by Purdue and The TLT Group), see
this chapter of the Flashlight
Evaluation Handbook on evaluating pilot tests of new
technology. For more on modeling of costs
(including uses of people's time), see
The Flashlight Cost Analysis Handbook,
another benefit of your subscription.
Using the Flashlight Online Template
We've created a simple item bank in
Flashlight Online. It's template ZS61936. (For more on how
to use Flashlight Online templates,
click here.) If your institution is an active
subscriber, you also have permission to use these questions
in other survey systems. If your subscribing institution
does not have a Flashlight Online account and you would like
a copy of these items, please e-mail
flashlight@tltgroup.org.
This template is really a small item bank;
only a fraction of the questions will be relevant to any
given survey, so delete the others, rewrite these as needed,
and add items of your own.
The introduction field in the template is
blank. A good introduction educates the respondent about why
it's important to think about the questions, and supplies
other motivation to complete the survey (e.g., we will let
you know whether and how we change our use of Second Life as
a result of what we've learned from this study). Here's a
sample introduction:
"We are currently using "Second Life", a
web-based environment, to offer [name of program or
experience]. We need your help to decide whether and, if
so, how to continue offering this program in Second
Life. This survey should less than [x] minutes to
complete. If you give us your name at the end, we will
send you a report on our findings, and on how we've used
those findings to make decisions about this program. We
appreciate you giving us the benefit of your experience.
Thanks!"
Case Studies Needed
We'd love to upgrade this chapter of the
Flashlight Evaluation Handbook by interweaving examples from
these kinds of studies of academic uses of Second Life,
perhaps studies that you do, alone or in collaboration with
us. (If you're interested in working with us, one good way
is through a TLT/Flashlight
Network
membership, which includes two days of
consulting/training on topics of your choice.) Whether you
or your institution are TLT Group subscribers or not, we
would welcome information about studies you've done that
have yielded useful information, information valuable enough
to justify the work you put into discovering it. We can
publish it in
F-LIGHT and include it in a rewrite of this chapter.
Thanks for thinking about working together!
|
