|
Flashlight Evaluation
Handbook
Some notes from
Steve Ehrmann from American Evaluation Association '01
Updated March 22, 2003 and March 26, 2007
I’ve just
returned from the annual meeting of the American Evaluation
Association (AEA) in St. Louis. The session I’m summarizing
here was facilitated by Molly Engle, Jean King, Michael Q.
Paton, and Sharon Rallis. If you are
doing a study (e.g., studying your own students; working
with a program as an external evaluator), you might find
some of these notes useful.
Several
dimensions were suggested for diagnosing resistance to
evalution by clients and subjects (in what follows the
evaluator or researcher is "you," and the clients or
respondents are "they"):
-
Overt (what you can see them
do to resist evaluation) versus covert (what they do
behind your back)
-
Active (they resist by doing
something) versus passive (they resist by not
participating)
-
Intended (they intend to
resist) versus unintended (their actions, taken for
other reasons, have the side effect of obstructing what
you’re doing; perhaps they’re trying to help you but
don’t realize what your study really needs)
-
Rational (they can be seen
as justified in resisting) for versus irrational (if
they knew more, or were a different kind of person, they
would embrace the study)
Reasons for
resisting (from a UN study in 1991, cited by Jean King)
-
“Our project is unique”
-
The program ‘aint broken’
-
Evaluation could trigger
threatening political consequences; a trap; a hassle
-
The study is seen as
impossible. (not realistic; can’t be objective; it’s the
rainy season; it’s a national holiday)
-
It’s not in my job to
participate; let someone else do it
-
What’s in it for me; other
personal factors (give me the funds; I’ll do it; I’m on
leave then)
-
Not enough money to do a
good job on the study
-
Fear of failure (can’t find
the original work plans; project plan is too vague)
-
Staff or leadership has
changed; staff turnover; someone else was in charge when
the project began
-
Been there, done that, we
don’t need to do it again
-
Not invented here; might
work elsewhere but never here
-
Opening the door – let one
evaluator in and others will follow
Many of
these reasons are well-founded: "they" often resist because
they understand the issues all too well, perhaps better than
the evaluator does! Reasons for resisting that are most
often found in education, in my experience:
-
People assume that "evaluation" or "assessment"
equals grading. They assume it's done when a program
or performance is over, for the purpose of (someone
powerful) deciding whether the program or
performance deserves praise or punishment. In
otherwise, participating in an assessment is to
surrender power to someone else and open one's self
for possible penalties.
-
The perceived
interests of the evaluator (and/or the parties who
will get the evaluation report) may differ from the
perceived interests of the person or people being
evaluated. For example, the evaluation may be seen
as influencing future budgets, and a project
director may feel more confident in influencing
budgets for the project without the evaluation.
(I've been repeating the word 'perceived' because
you need to deal with the fears of the people being
evaluated, but those fears may not be well-founded;
for example, their budget might be helped, even by
an evaluation that demonstrates a problem with their
program, because the data can be used to justify
budget money to fix the problem.)
-
People have had so little exposure to the idea that
data can be used to improve performance and reduce
risk that they literally can't understand that
rationale. For more on this possibility, see this
essay on the
scholarship of teaching.
Deception
When an
evaluator deals with clients or subjects, their silence or
agreement can be deceiving. The
clients/subjects may not feel it’s OK to talk about their
resistance or about feeling insulted by the evaluator’s
inference that there’s a problem
-
One
clue: Listen for a lack of constructive questions and
suggestions from the client; if the client is simply
enthusiastic, you may be getting a ‘line’ from a
skeptical client who doesn’t want to be seen as
obstructive when a higher up has asked for this to be
done.
-
One
response: start by finding out what they’ve already
learned, what they see as the problems that deserve
inquiry. Start by asking them about what they see as
problems or reasons for not taking part. And then
respond to those issues one by one.
Start by Acknowledging the
Reasons for Distrust?
When an
evaluator first meets a client (e.g., an evaluator of a
funded project, paid by the funder, meeting the project
director of that funded project), there are often reasons
for distrust on both sides, before they have ever exchanged
a word:
-
The project director may
have valid reason to distrust the evaluator, perhaps
because the project director has seen mostly studies
that did no good and/or that were flawed, and perhaps
because the most likely consequences of the study appear
to be either 'no change' or 'loss of funding';
-
The evaluator may have valid
reason to begin by distrusting the project director.
After all the project may be doing badly, it may be seen
as the project director's fault, and the project
director may be doing his or her best to hide all that
because of the expected consequences of the truth coming
out.
What are
the possible outcomes of the evaluator (or the project
director) beginning the conversation by acknowledging that
they may be starting from mistrust, with the task of earning
trust? Perhaps the evaluator should announce in his or her
first communication that “I realize it’s my responsibility
to prove that this study is likely to well worth your
effort, almost no matter what we find; I don't expect you to
take that for granted.” The evaluator may also need to help
the project director speak about these issues, too.
1.
The distrust continues, perhaps because the interests
of the evaluator and the PD are really clashing and the
conversation has invented no win-win study designs (for one
hypothetical example of developing a win-win design in the
face of stress, see "Finding
a Great Evaluative Question: The Divining Rod of Emotion.")
The evaluation then may have to proceed despite the overt or
covert, active or passive, resistance of the project
director. However, action or inaction by the distrustful
project director may restrict the flow of data, and increase
the chance of invalid data sneaking in, there's more of a
chance of an erroneous or unpersuasive report.
2.
On the other hand, the parties may be able to work
through a process and develop trust around a win-win design.
Another, parallel tack: in the AEA session, Michael Paton
talked about an evaluator might teach clients and
respondents how to resist evaluation, how to subvert and
alter the study to help make sure it either helps them or is
stopped in its tracks. In either case, the evaluative data
are likely to be more varied, rich and valid.
This is
no panacea. But, if it’s successful, in many cases such
an approach could increase the energetic participation of
project staff in the design and implementation of the study
design.
You may
also want to look at the related web site on "Frequently
Made Objections to Evaluation (and Some Responses)."
Flashlight Evaluation
Handbook
|
