|
This document is both for people submitting
surveys and documentation for potential publication as part of
the Flashlight Collection and for peer reviewers who are judging
those submission. The first part of the document is mainly for
those preparing submissions, and the second part mainly for
reviewers, but both groups should read the whole document.
Subscribers Can Submit Survey Packages for Publication in
the TLT/Flashlight Collection
“Templates” are surveys that new authors can use
either “as is” or as a first draft for writing their own
surveys. A survey package includes a template, documentation on
why and how to use it, and an example of how data from the
survey has been used by the author or others in the past. (More
on the content of survey packages can be found
below.)
We invite you to submit a survey package (a
survey template plus documentation) for peer review and possible
publication if you think it would be useful faculty, staff and
student authors at some of the 130+ TLT/ Flashlight subscribing
institutions around the world.
Who Can Submit a Survey Package for Potential
Publication?
Anyone associated with a TLT/Flashlight
subscriber institutions may submit a survey package:
-
a student who has used the instrument in a
dissertation,
-
a faculty member who has used the instrument to
study his or her own course,
-
an administrative unit that has done a needs
assessment
-
an accreditation self-study team that has
studied achievement
-
a vice-president who has collected data for a
cost-analysis
-
....
Surveys need not deal with educational uses of
technology..
What to Submit, and How
Submissions should include:
-
a statement of purpose for the
study, description of respondents (e.g., students),
-
method(s) of conducting the study,
-
summary of findings,
-
brief description of how the
findings were worth the effort.
-
assurance that you have the legal
right to allow The TLT Group and its subscribers to use,
modify, and distribute the materials you have submitted.
Optional: include in your submission a brief
summary of the validity and reliability tests of the survey plus
any relevant references.
These descriptive materials would be used, along
with the survey itself, to inform potential users.
Topics: Surveys should be useful for improving
education at any level (college, school, corporate training;
course, institution, national, international).
NOT required: The survey need not have been
created with Flashlight tools or methods. The survey does not
need to deal with educational uses of technology.
Survey packages and questions about the review
process should be e-mailed to Steve Ehrmann, Director of the
Flashlight Program <
ehrmann@tltgroup.org >.
Review Process
Each survey will be reviewed by at least two
reviewers. Some surveys may be tested at the reviewers’
institutions. See below for rubric to be
used in evaluating surveys for inclusion in Flashlight. Each
package will be reviewed independently on four criteria
(purpose, plan, validation/reliability, report) as shown in the
scoring sheet below.
After independent review, reviewers will confer
and agree on a final rating:
1) Not appropriate for publication to Flashlight
users. Author will receive feedback.
2) Promising; returned to the author for
improvement (or modified by reviewers, with revisions returned
to the author for approval)
3) Appropriate for publication as a survey
package.
Publication
Survey packages will be published in at least two
media:
Invitation to Serve as a Peer Reviewer
We also invite you to serve as a peer reviewer,
to make this process of user-to-user sharing work. If you’re
interested in serving as a member of the peer review committee
in this publication process or if you have questions, please
send e-mail to Steve Ehrmann at
Ehrmann@tltgroup.org.
RUBRIC FOR EVALUATING SURVEYS FOR
INCLUSION IN FLASHLIGHT
|
|
Unacceptable (one rating of
“unacceptable” requires rejection of survey) |
Acceptable (1) |
Superlative (2) |
|
Purpose |
Not a survey; or data would not be
useful, directly or indirectly, in improving education |
Data from this survey could be useful,
directly or indirectly, for improving education.
(“Indirectly” includes administrative surveys). |
Addresses an important need; would
significantly increase the value of Flashlight for its
subscribers.
Surveys already used by more than one
institution for benchmarking are of special interest. |
|
Plan (sound evaluation practices,
including sample or population description, confidentiality
statement) |
Methodology flawed; results may be
misleading |
Methodology appropriate to the purpose
and scope of the study |
Methodology itself should be useful for
other authors and other surveys |
|
Validation; reliability of survey |
No internal review or pretest |
Reports on how the survey was
validated, and whether it is statistically reliable |
Report demonstrates validity and/or
high level of reliability |
|
Report on results |
No report is included and reviewers
conclude that the survey has little likelihood of producing
beneficial findings for its future users.
|
Report on what was done with findings
is included. |
Based on the report survey results had
an important influence or benefit; and/or reviewers believe
that the findings would be of great use to future users. |
|