|
FAQs about
Flashlight Program l
FAQs about Flashlight Online 1.0 l
FAQs
about Flashlight Online 2.0
Frequently Asked Questions (FAQs)
about the Flashlight Program
1.
What is Flashlight? Is it
for-profit?
2.
What are your
findings?
3. Are the items in your tool kit validated? Can I see them?
4.
What
are the "seven principles"? Where can I learn about the research
behind them?
5. Do you offer workshops?
6.
What does "activity centered evaluation" mean?
7.
Why is it called
"Flashlight?"
8.
The TLT Group site license limits use
of its materials to the institution that subscribes. What is an
"institution?" If there is a main site and several branches or outreach
sites, does each site need its own license in order for its people to use
Flashlight Onilne?
Flashlight Online Questions
1)
What is the
Flashlight Program in a nutshell? Is it a research project? A for-profit company?
Since 1992, the Flashlight Program has been helping educators and their institutions study and improve educational
uses of technology. More recently, our methods, tools, training and
consulting have been used for a wider variety of needs in the
scholarship of teaching and learning, student course evaluation, program
evaluation, accreditation support, and other needs. Since 1998,
Flashlight has been a program of the non-profit TLT Group.
Click
here for an interview that describes the development and work of the
Flashlight Program.
RETURN TO
TOP
2)
What does
the national Flashlight study show?
Trick question, but one that we have
often been asked.
There is no national
Flashlight study although Flashlight methods and items could be used for such a study.
We provide tools, tool kits, training and consulting to help users do
the studies they need. User data is private; not even we
know what they (or you) may find. We do try to help users publish their findings.
In near future, we will begin to offer study packages; these will
include standardized, online surveys that will enable interested users
to share data; we will also develop national reports aggregating data
from interested users. The first study packages will deal with uses of
the Web in nursing education and uses of Web Course Management Systems
(e.g., WebCT, Blackboard).
RETURN TO
TOP
3)
Are the
questions normed and validated? Can I see them before our institution buys a site license?
Our focus has been on
research and content validity. The issues selected for study are those raised by research
as the most promising potential benefits and the most worrisome potential problems. Most
items in the Flashlight Current Student Inventory
(the items tapped by Flashlight Online)
have been subjected to trial use and
then to follow-up focus groups to make sure that the wording is clear to many types of
respondents.
The CSI is an item bank. As such it can't have true validity and
reliability since both depend so much on the order of the questions in a
static instrument. However, the items in the CSI have content validity
because a) many of them are based on the seven principles of good
practice in higher education and b) different versions were reviewed by
experts from five pilot institutions.
We then tested items for face validity by having more than 40
different surveys created from the item bank at these 5 pilot
institutions. Approximately 2,000 respondents completed these surveys.
We then organized focus groups with the respondents, and with the
faculty and administrators responsible for interpreting the results. All
of the teaching and learning items were able to be tested in this way
(not all of the demographic items were used, but most of these are of a
standard format that have been well-tested elsewhere).
Since the CSI was released, we have created a number of standard
templates that are based on the item bank. One of these is the
Evaluating Educational Uses of the Web in Nursing (EEUWIN benchmarking
survey). It was pilot tested at 3 Nursing programs. This survey has been
tested for validity and for internal reliability. Over three years it
has demonstrated a consistent Cronbach's alpha of .85-.90. We are
continuing to work on other templates that test other areas of
subscriber concern and will make those reliabilities available when they
are generated.
One way to "see before buy" for institutions that are seriously
interested is to request a two week guest account for Flashlight Online
(which includes access to database of CSI items as well as to survey
templates on different topics). Send e-mail to
flashlight@tltgroup.org to
request an account for your institution. Only one account per
institution, please.
4)
What
are the 'seven principles'? Where can I learn about the research behind
them?
In
1987 Arthur Chickering and Zelda Gamson answered the question,
"According to educational research, what practices
tend to produce better educational outcomes?" by listing
"seven principles of good practice in undergraduate
education." They pointed to the following characteristics of
teaching-learning as being especially valuable for improving learning
outcomes (i.e., if a course or institution increases what it does in
these areas, learning outcomes of many sorts are likely to improve):
-
Encourages
student-faculty contact.
-
Encourages
cooperation among students.
-
Encourages
active learning.
-
Gives
prompt feedback.
-
Emphasizes
time on task.
-
Communicates
high expectations.
-
Respects
diverse talents and ways of learning.
The
Web usually has a number of sites that provide more detail on each of
the seven principles but the URLs seems change yearly, sometimes
monthly. So instead of recording some of those current URLs here and
frustrating you when the links break, we suggest using a search engine
and doing a search with terms such as "seven principles"
"Gamson" and "Chickering." For a book on the
education research behind the seven principles, Zelda Gamson recommends Applying
The Seven Principles For Good Practice In Undergraduate Education: New
Directions in Teaching and Learning, #47, published in 1991 by
Jossey-Bass; it's now out of print but it may be in your library's
collection.
The chapter by Mary Deane Sorcinelli reviews the research literature.
Many
of the items in the Flashlight Current Student Inventory (and thus also
in Flashlight Online) ask students about how often these practices
occur, how often technology is used to carry them out, and how
appropriate available technology is for carrying them out.
For much more on the seven principles and uses of technology to
advance them, see http://www.tltgroup.org/seven/home.htm
RETURN TO
TOP
5)
What
happens at the workshops? How often and where are they offered? How much do they cost?
The
workshops are tailored to your needs. A talk (not a workshop) might run an hour or so and
provide a briefing on Flashlight's capability. A two-hour workshop
would guide participants through creating a simple
survey while also introducing participants to principles
for designing truly useful studies. Advanced workshops provide intensive training in Flashlight
use, and are especially useful to help teams devise shared evaluation strategies.
All Flashlight workshops are available in face-to-face,
online, and hybrid forms. Please contact
Flashlight@tltgroup.org
for more information.
6) I've heard
people call Flashlight an activity-centered
approach to evaluation. What does that
mean?
The
purpose of evaluation is to help people make decisions. Knowing that technology was
used and that outcomes improved (or stayed the same, or got worse) doesn't help anyone
decide anything. To make improvements you need to know how the technology was used (the
activity that used it). Focusing your attention not just on technology and on outcomes but
also on activities yields some surprising dividends. For example, suppose your
institution invested in e-mail in part so that students would learn better. How is e-mail
supposed to help? Let's suppose that an important activity is students doing homework
together. The focus on the activity raises some important questions for your study.
For example, do students have access to e-mail from the places they do homework? Do
faculty encourage, or discourage, students working together on homework. Do students
believe collaboration of this sort is helpful? a waste of time? cheating? The
answers to these kinds of questions help determine how helpful e-mail will be for such
collaboration and thus whether the e-mail (indirectly) helps students learn the material.
RETURN TO
TOP
8)
Why is the program called "Flashlight?"
Three reasons:
a) Someone says to you, "Let's evaluate
this educational program." Where do you pay attention? There
are an almost infinite number of possible studies that could be done of
even one assignment. A successful study almost always focuses on
one particular question (while remaining open to the
unexpected).
b) Why "Flashlight" and not
"Floodlight" or "Spotlight?" These studies,
especially those that can be done by faculty, staff, departments or
institutions about their own operations, are not methodologically
powerful. So focus and humility are required: a pocket flashlight is a
more metaphor than a gigantic spotlight.
c) Imagine yourself trying to teach, or run an
innovative program, in the dark. Someone hands you a flashlight. How
would you use it? You'd think first about the direction you'd like to
go, and shine it in that direction a little way ahead to see what you're
getting into. The assessment equivalent are studies of need, of
similar practices at other institutions, and some focused environmental
scanning. If you're smart you'll also then pivot and point the
flashlight immediately behind you: how quickly have you been moving
forward? what problems have you encountered? how might you respond to
them the next time? That's the image we have in mind when we say,
"Flashlight."
RETURN TO
TOP
9) The TLT Group site license limits use of Flashlight Online and
TLT/Flashlight materials to the institution that subscribes. What is an
"institution?" If there is a main site and several branches or outreach
sites, does each site need its own license?
The US Department of Education has a data bank about
higher education called IPEDS, and The TLT Group uses that reference for
US institutions. If two or sites share the same UNITID in the IPEDS
system, we consider them to be different sites of the same institution and
only one subscription is required. If, on the other hand, if they each
have a different UNITID then we consider them different institutions. We
are happy to negotiate a single discounted fee to cover all these related
institutions, however. For information about all this, and for non-US
institutions, please send your inquiry to
info at tltgroup.org.
RETURN TO
TOP
I've heard Flashlight Online called a collaborative tool. Can it be used
to help survey authors work together? even if they're at different
institutions?
Flashlight Online runs on
computers at Washington State University. Each subscriber institution can create
accounts for authors; each author gets a unique username and password.
Any author can make any other author (at any other
institution) a co-owner of a survey and, from then on,
they each can edit the same online survey, rubric, item
bank or other online form.
Flashlight Online also makes it easy for one person to
let any other author see such an online form, copy it or
copy elements of it, and then incorporate that material
into a new form. The Flashlight Program, for example,
makes many folders of surveys, item banks and other
online forms available to all the TLT Group's Network
and Comprehensive Collection institutional subscribers.
Will Flashlight Online 2.0
someday be available as
open source?
Yes, that's our plan.
The code will not be available until Washington State University is confident
that the system is reasonably stable and performing
well; at that time they will begin documenting the code
and preparing it for download. At this writing
(April 2010) there is no timeline or plan for conditions
under which the code would be made available.
What are survey meta-data?
Authors and
administrators will be able to 'tag' information with descriptors that
can be used later to search for, and find, that information. In
Flashlight 1.0 many items in the Flashlight Current Student Inventory
have such tags, which is how a user can call up dozens of items that
relate to 'collaborative learning' even though none of those items
include the word 'collaborative.'
Flashlight Online 2.0
will allow local administrators and users to tag items, surveys, and
item banks. Meta-data are essential for administering complex course
evaluation systems - information about particular authors, courses,
surveys, items, item banks, and the like.
Is the Current
Student Inventory (CSI) available in Flashlight Online
2.0? Yes. In
fact CSI 2.0 is gradually appearing in Flashlight Online
2.0. When we considered simply copying CSI 1.0
into Flashlight Online 2.0, we realized we should update
the CSI to a) take advantage of matrix survey
capabilities, and b) updat the content. For more,
click here.
When I set times
for Flashlight Online 2.0 surveys to turn on or off,
what time/calendar is used?
Flashlight Online 2.0 runs on Pacific
Time in the US. We have users around the world To
see what day and time it is currently (for Flashlight
Online), look at the lower left hand corner of your
browser.
RETURN TO
TOP
How secure is Flashlight
Online?
No one from another
institution can see your surveys unless you give them authorization to be
in a "member group" that is under your local administrative group.
Only the author of a
survey (the person who started the survey), or someone else that the
author explicitly designates, can download the raw data at any time.
Recently Charles Ansorge of the University
of Nebraska wrote, "I submitted an IRB (Institutional Review Board) proposal
using a Flashlight Survey as the tool and they want to know whether the data
will be sent to a secure server, whether the data will be encrypted while
in transit, and will it collect IP addresses? I am unclear how much of this
applies to how Flashlight works. For my study, the survey is anonymous - so
no id number is given to the participants."
We replied:
-
Secure server?— Yes,
we take every measure to protect the security of the data on our server.
It is behind a secure firewall and all security updates are applied to
the server in a timely fashion. It is also physically secured in the
server room which includes video surveillance.
-
Encrypted in transit?
— No, we do not encrypt data passed in transit.
-
Collect IP numbers? —
While no IP numbers are stored with the survey responses we do collect
IP numbers in the server log. The server log is replaced on a rotation
schedule and there is no link that can definitely be made between any
respondent and the IP number they took the survey from. If a survey has
respondent IDs this is also the case. Those IDs are in no way tied to
the server log of IP numbers.
When
happened to Flashlight Online 1.0? I need a copy
of one of my old surveys or some of my old data that's
still in the system.
The Flashlight Online 1.0 server was shut down and the
data archived at the end of March 2010. A substantial fee will be charged to
retrieve any of the old surveys or data. For more
information contact flashlight at tltgroup.org.
Are
there any limits or extra charges on large numbers of authoring accounts,
or surveys, or respondents for Flashlight Online?
No. An institution with a subscription to the
Comprehensive Collection or to the Network can give (if it
wants) authoring accounts to all its faculty, staff and students. Each
author can create an unlimited number of surveys, and each survey can have
an unlimited number of respondents.
However, there
are some limitations: 1)
authoring accounts can only be given to members of the institution (its
own faculty, staff and students); 2) the surveys must be for institutional
purposes (e.g., faculty or staff can't use the institution's site license
to do surveys for private consulting unconnected with the institution)
without a separate license from The TLT Group; and 3) the system cannot be
used for institutionwide student course evaluation without
express written permission from the TLT Group; an extra fee
will be charged for this use of Flashlight Online because of
the extra demands it poses for both servers and support. For
more information on limitations on uses of Flashlight Online
posed by Washington State University, see the 'terms of
service' linked to the
Skylight log-in page.
If you've read
this and have other questions, please send them to
Flashlight@tltgroup.org.
|
