How often you hear sentiments like these from faculty,
staff and students?
"Before I (or we, or you) decide what to
do next, the next step is to unearth some additional
facts, in order to make a better choice."
If the answer is 'frequently," that's one hint that your
program has what some people would call a healthy "culture of evidence."
(some people might call it a 'culture of assessment,' or
'culture of inquiry' while
others might reject the term
'culture.')
Another hint that your institution systematically values and supports
the gathering of information before making decisions: you
would have heard from colleagues, over the years, many
stories of how pausing to ask some questions first, rather
than acting on impulse or experience, had led to surprising results,
unexpected success, avoidance of disaster, or just saving of
time and avoiding stress. In other words institutional
folklore is conveying the importance of inquiry from one
generation to the next.
That sharing of stories is one reason to use the word
'culture' in this context: That passing along of
folklore is part of the informal process that makes an
organization organized.
Another hint that an institution has a strong culture of evidence:
when people get surveys or requests to be interviewed, they
respond favorably because past experiences have led them to
associate inquiry with empowerment. "Someone is asking
me what I've seen, or think, or need and, if I respond
thoughtfully, I'm likely to gain more control of what's
happening to me."
Another hint of a healthy culture of evidence: faculty
see course evaluation surveys as a way to teach students (by
example) about what good applied research looks like -- part
of general education at the institution. The faculty take
pride in the validity of the inquiry and the responsibility
with which data is analyzed and used, and in the way the
researchers remain accountable to stakeholders (including
the students who have invested time in providing feedback.)
How can an institution strengthen its "culture of evidence"? Among
the
formal and informal elements of support:
- Lots of education available for faculty,
staff and students in how to 'ask the right question,'
especially peer-to-peer training in how to ask
important, answerable questions about one's own
practices.
-
Providing assessment and other research tools: A large and growing
fraction of staff have access to, and know how to wisely
use, tools for assessment (e.g., tools for creating
authentic assessments), inquiry (e.g.,
Flashlight Online),
and analysis.
Perhaps the most relevant feature of Flashlight Online:
it makes it easy for people to find surveys created by
others, and then either use them "as is" or adapt them.
So it can support the development of a culture of
evidence, while it's also providing useful tools for
individuals. [Note to TLT Group subscribers:
we can
help you craft and offer such workshops. For example,
for faculty learning how to use student feedback to
improve teaching and learning in their own courses, our collection of workshop materials
simultaneously help participants understand why to ask
such questions, how to ask them, and how to use relevant
research tools. And the workshops are organized into
chunks of 15 minutes or so each, making them easy to
insert as agenda items in departmental meetings.]
- Healthy networks for sharing problems, methods and
findings: staff and faculty frequently
interact with peers, inside and outside the institution,
peers who are also gathering information to improve practice. They
can share ideas, techniques, findings and occasionally work
as a coalition to get more support for this work (by
'support' we refer to all the other bullets in this
list.)
- Organized institutional data collection that
can be tapped by individual investigators and tailored
(e.g., faculty can add their own items to student course
evaluation surveys).
- Expert assistance can be, and is, used by
them to help with such studies (e.g., statistical help;
help in designing good surveys; help in 'mining'
institutional data)
- A shared sense of track record in this area -
post or broadcast examples of studies that helped improve learning, save
time or money, and the like are widely known. Potential
respondents to surveys, interview requests, etc. know
from past experience that this is usually a productive
way to use their time.
- Hiring and retention of staff: When faculty
and staff are hired and promoted, attention is paid to
their track records of using evidence to improve
practice and outcomes.
- Gradually developing among students a sense
of individual and collective responsibility for
education, so that responding to surveys and other
research seems both a way to gain control over their
environment and the fulfillment of a responsibility to
others
- Institution/program goals and priorities for improving
outcomes,
including the use of evidence to guide and accelerate
those improvements (e.g., an institution that has a
multi-year plan to improve distance learning, or
learning communities, or international study; and then
does institutional studies to support this effort while
also supporting studies on the topic by individual
faculty and staff with regard to their own contributions
to the effort).
- Priorities/budgets/persistence: A shared
sense that the administration values such practices of
inquiry and consistently invests in them
- Related policies and practices: does the
Institutional Review Board (IRB) have quick turnaround
on requests to gather data? do union contracts support
or interfere with efforts by faculty and staff to use
data to improve teaching?
- Support and rewards from outside the institution
for these practices and their results (e.g.,
regional accreditors who value such practices and reward
institutions that are especially good at them).
In 2003, The TLT Group, Washington State University, the
Coalition for Networked Information, and EDUCAUSE's National
Learning Infrastructure Initiative (now EDUCAUSE Learning
Initiative) collaborated in a project on "Transformative
Assessment."
This rubric, developed by Gary Brown of WSU as part of
this collaboration, is a useful tool for assessing
institutional progress toward using data to improve
practice. We hope to expand and improve this rubric soon.
This 2001 article on the efforts of three institutions
to create a culture of inquiry was another contribution to
that effort.
*Some people use the phrase
'culture of assessment.' As we define terms, a
culture of assessment makes assessment of learning an
everyday activity. A culture of evidence supports not
only assessment but also other inquiry by faculty, staff and
students intended to help them gather facts in order to make
choices.
Note to TLT
Group subscribing institutions: one way to use your TLT
Group consulting time: our consultants can help you assess the factors that support
(or are currently failing to support) a culture of evidence
at your institution, and then suggest steps that could help
you move forward. This kind of assistance might be
especially timely in the years before your institution's
next accreditation or program review.
