|
Flashlight Evaluation
Handbook Table of Contents
One of the most frequently
asked questions is "How can I get more people to respond to
my study?" Response rates are often so low and so uneven as
to invalidate findings. People less often ask, "How can I
get respondents to think more carefully about their
responses?" but that's a challenge, too. And
still less often do authors ask, "How can I design a
study will make decision makers sit up and take notice
(in instances where the author is not the (only) decision
maker).
The answers to all these
questions are similar: a) design a study that's likely to
benefit its subjects (respondents) and also its decision
makers, not matter what you find. And then make sure that
both groups learn what's found, and what's done as a result.
This section focuses on increasing engagement by potential
subjects; much of this Handbook is designed to help you
engage decision makers, especially the next two sections.
But the two issues are tightly linked: one way to engage
respondents is by making sure they know that decision makers
will be paying attention to the findings.
How to increase
response rates: Survey methods books suggest
many answers to that query. (Flashlight users tell me that
an especially good book of this type is Don A. Dillman's
Mail and Internet Surveys: The Tailored Design Method,
Wiley: 2000). In this short essay, let's focus on how the
content and context of the study affect response rates.
Let's start with some
truth-telling. Most of us have learned that surveys are a
theft of time, an exploitation of the potential respondent.
So we ignore surveys because experience tells us that the
survey will certainly take time and effort with, in the end,
little or no reward. The survey creator might not even ever
analyze the data!
I'd like to propose a dozen
principles of good practice that use the substance and
process of the study to increase response rates: intrinsic
motivation. The first three principles below relate to the
culture of the institution. The next five relate to the
qualities of the study itself. The final four deal with how
the study is publicized.
The Culture of the
Institution
- Studies that students
have already seen at the institution should have
followed principles of good practice such as those
described below so that students are already mostly
convinced that this new study, like the ones they
responded to last year, will be worth the effort to
answer thoughtfully and completely.
- Students should have
learned enough about surveys (and inquiry) as part of
their general education, and as part of their life at
the institution, to realize the importance of good
response rates.
- Students should have
become part of a larger community of learners. By
"community" I mean a group with a tradition and
expectation of mutual help. In such a community,
students will be more likely to respond to a study that
promises to help other students, even if it isn't likely
to benefit them personally.
The Study Itself
- Explain how study
results might be linked to subsequent action that
matters to the respondents. For example, if the study
shows A, it would be a reason to do X but if the study
shows B, it would be a reason to do Y. Won't that
bias the responses? That depends on the study
design. It is often possible to design a study so that
one can honestly tell respondents what is at stake while
not biasing the outcomes. When complete honesty is not
possible, then tell respondents so ("we have to keep
some aspects of this a secret for now but ultimately
we'll tell you the results of your participation). If
the study really isn't worth their time, give them other
reasons to participate ("participating in the study
won't directly help you or your peers, so we're going to
pay you for your time.) However, the problem with
extrinsic motivation such as payments, raffles, adding
points to student grades and so on is that external
rewards usually don't encourage thoughtful
participation. And you may bias your sample toward
people who need your bribe! The easiest way to make sure
that students think carefully and answer honestly is for
the study to be worth their attention.
- Create studies aimed at
felt needs and carry them out in language that students
can easily understand. This (and designing studies that
really do benefit students) can be aided by using
students to help design or critique the study.
- If students discover
that the study is thought-provoking and educational, at
least some will be more likely to think and to go on to
the end.
- If the study is not
intrinsically worth student time to respond or if there
is some reason why the value of the study has to be
hidden from students, offer an extrinsic reward. One
radical notion: pay them for their time at going rates.
If the study really isn't worth their time, what is the
justification of asking them for their time?
- If the survey is a course evaluation
survey (student feedback form),
click here for more ideas from Flashlight and its BETA
project.
Explaining the Study
- Faculty often play an
important intermediary role in increasing response
rates, even if the faculty member isn't the primary
author of the study. If faculty aren't convinced of the
value of the study, they are unlikely to be effective
advocates, explaining and reminding students to respond.
So make sure that the study is going to be of value to
faculty as well as students, and make sure that as many
faculty as possible have bought in before making a final
commitment to the study.
- Explain at the beginning
how students will be helped if most or all students
respond thoughtfully to the survey; this, like some of
the other steps, increases the chances that students
will encourage one another to respond.
- Promise to report on
changes later that were influenced by the findings.
- If the study is of major
importance, seek co-sponsorship of the study by student
organizations before the design is too far advanced;
these co-sponsors may also play a constructive role in
its focus and content.
- If the study is done
repeatedly, use testimony from last year's students
about the value of the outcomes of last year's version
of this study.
In other words, if the
culture and the investigator make a legitimate case that
studies like this, and this study in particular, are worth
the respondent's time, respondents are more likely to answer
and to answer thoughtfully.
Of the last dozen studies
you've seen at your own institution, how many of them met
more than one or two of these criteria? If most didn't,
that may help explain why response rates are often so low.
Certainly it takes time to
follow these principles of good practice, more time than
some investigators have. Instead of putting a little effort
into each of many studies that together poison the well for
future studies, let's teach our students about good practice
in professional research, and justify their faith in their
institutions, by putting more time into each of a smaller
number of studies that really are worthwhile.
For some more thoughts on the
student role in designing studies, see the essay "Student
as Co-Investigator."
|
