|
Definition l Matrix Surveys
& Student Course Evaluation
l
Examples of Matrix Surveys l History
l
Ask Us
to Develop Your First Matrix Survey l
Flashlight Online 2.0
l
Creating a Matrix Survey with Flashlight Online 2.0
What is a Matrix Survey?
Why
are Matrix Surveys so Valuable?
Matrix Survey: different pools of
respondents can receive questions chosen and worded
specifically for them.
For example, imagine
gathering student feedback about staff teaching their
courses. Some courses have one instructor. But in other
courses, there may be a series of lecturers, a section
leader, a lab assistant, and a student assistant. A
matrix survey can automatically tailor the feedback form so
that it asks students in each course about the right number
of instructional staff, asks questions appropriate to the
role of each one, and identifies each staff member by name.
In this way, matrix surveys enable
authors to ask respondents fewer, more precise, less
ambiguous questions.
Click here
to see more examples of how matrix surveys can transform
your research or evaluation.
Reduce survey
fatigue and increase response rates: Two or more authors, or
offices, can meld their inquiries into a single,
shared matrix survey. By doing so, they can:
-
increase
response rates, by pooling their efforts to
publicize their inquiries and reward response
-
save money
in implementation
-
decrease survey
fatigue (fewer surveys, fewer total questions),
and
-
save time.
For example, the Writing Program may have
questions only for writing-intensive courses, the
Information Technology services may only want to question
students who used the discussion board in the course
management system, and the Library might want to question
students with JSTOR accounts. Those three units could
collaborate on a matrix survey: some students would see only
questions from the Writing Program, some might see the
Writing Program questions, and the discussion board
questions, and the JSTOR questions. And so on. The authors
could share the data from some questions (e.g., demographic
information).
Example: Using Matrix
Surveys for Evaluating Courses or Workshops
Matrix surveys can be used in different
formats for gathering feedback from participants in courses
or workshops:
-
Ask the same questions of participants
in each course/workshop. Analyze the data course by
course, in subgroups, or in one pool.
-
Ask questions of participants of
workshops that occur all at once, or workshops that
happen one, or a few, at a time. analyze the data as it
comes in to see trends.
-
Alter the wording for each course or
workshop, in modest ways (e.g., insert the name of the
workshop, its date, and the name of the leader in text
and questions, but otherwise keep questions the same)
-
Create one or more groups of special
purpose questions that are each posed for only certain
courses.
-
Involve more than one stakeholder in
gathering feedback (e.g., invite instructors to select
teaching/learning activities from a menu, so that
participants are asked questions specifically about
those activities; invite the writing program to ask
questions only of students in writing intensive courses;
enable the Technology Department to ask questions about
course management system only in classes that used it)
We call it a matrix survey
because we visualize each group of people who get the same
items as a row in a matrix, while each author is represented
by a column. The students in each row get the questions
marked with an "X".
|
Question Groups
Respondent Pools |
Prof. Smith's questions
for the 10 am Calculus
class |
Group of questions for all
lecture courses, from the Assessment Program |
Questions for
writing intensive courses from the Writing Program |
Questions
for courses meeting in computer classrooms, from the
IT department |
(More
questions from other units) |
| Prof. Smith's 10 am Wednesday
Calculus section |
X |
X |
|
X |
|
| Prof. Smith's 2 pm Thursday Calculus
section |
|
X |
|
X |
|
| Prof. Black's chemistry
laboratory |
|
|
X |
|
|
| Prof. Green's dance studio |
|
|
|
|
|
| (more) |
|
|
|
X |
|
If you're a Flashlight Online
author, here
is a more detailed explanation of how to create a matrix
survey.
Other Applications of
Matrix Surveys
Matrix surveys can be used to
study any phenomenon, product, or idea that is predictably used differently, or is
seen differently, by different groups of people.
This page
displays a variety of matrix surveys: for formative
evaluation of ePortfolio use, information literacy
development, student response systems, classroom
technologies, ....
Matrix surveys can also be
used by faculty and by students for survey research in their
disciplines, on topics ranging from market research to
health care. Matrix surveys provide a new flexible, focused
tool that should open new research frontiers in several
disciplines.
And because matrix surveys offer the
option of collaboration among multiple authors, still more
options open. For example, let's imagine a study of
healthcare statewide, with response forms tailored to the
location of the respondent. Some of the questions
could come from the health care programs themselves as well
as from relevant government programs and public interest
groups, giving them all more of a stake in the study and its
data.
History
The concept of matrix surveys
emerged from "Better Teaching through Assessment (BeTA)," a
FIPSE grant to the TLT Group to rethink student course
evaluation; Flashlight Online 2.0 was developed in part with
support from this grant. The concept and name, "matrix
survey," was primarily the work of Nils Peterson, Washington
State University, and Steve Ehrmann of The TLT Group.
Conditional Questions
- A Complement to Matrix Surveys
Flashlight
Online 2.0 will also offer conditional questions -
questions which a respondent sees only if earlier questions in the
survey were answered in a certain way. "Branching" and "skip
patterns" are types of conditional questions. For
example, if an earlier question asks, "Have you ever used X"
and the person responds "yes", and if the person has also
responded that he or she is over 21, then (and only then)
would the respondent see questions about X.
In general, authors should
use a conditional question for issues where the respondent
is the best judge of the relevance of the conditional
question. In contrast, matrix surveys are appropriate
when it's more appropriate for someone else to make that
judgment (e.g., in a primary election where only registered
Republicans are allowed to vote for Republican candidates),
a matrix design is more appropriate than asking in the
survey, "Are you a registered Republican?" Matrix
surveys are especially valuable when an inappropriate
respondent is unlikely to understand the diagnostic question
(e.g., "do you use ePortfolios to support reflection?")
Shall We Develop
a Matrix Survey Together?
|
