|
Matrix Surveys
& Student Course Evaluation
l Using a Survey to Create a Matrix Survey
l Why Create a Matrix Survey? l
How to
Create a Matrix Survey with Flashlight Online 2.0 l
Matrix Surveys & ePortfolio Initiatives l
Matrix Surveys &
Student Response System (e.g., "clicker") Initiatives l
Flashlight Online 2.0
In creating surveys and other online forms,
authors usually face two barriers:
-
All respondents must receive the same
("lowest common denominator") questions. That's one
reason why respondents often see questions as rather
bland and sometimes also off-target, confusing, or
(seemingly) stupid.
-
The survey must be designed by a single
person or team, from top to bottom.
That all
seems inevitable. But consider a modern election, especially
a primary, statewide election. Ballots (surveys) are created
for all voters based on their home address and, in a
primary, also on their political party (metadata about the
voter). Voters who live in the same area and are registered
in the same party get identical ballots. The questions
(candidates, referenda) are chosen by different stakeholders
(party organizations in specific districts, governmental
units). The questions on the ballot are those (and
only those) that are relevant to each group of respondents.
But it is all one election: the ballots are analyzed
statewide, and reports are created for all those stakeholder
groups. We call this a matrix survey. And,
using
Flashlight Online 2.0, it's to create and administer a
matrix survey.
In a matrix survey:
-
Different subgroups of
respondents can receive different sets of questions,
questions chosen and worded for that particular pool of
respondents.
(For example, in a study of student response systems
where one course uses devices that students call
'clickers' while students in another class refer to
"polling systems", the wording can be tailored for each
class. If one set of courses uses clickers for peer
instruction and another set does not, only the first set
of courses will get questions about peer instruction.
-
If several groups of
stakeholders need data from the same large population,
they can each contribute questions, using metadata to
assure that their questions only go to that subset of
the total population that will understand and
respond. (For
example, in a survey given out at the end of all courses
at a university, the writing program may need data
(only) from writing intensive courses, the IT department
may have some questions (only) for courses using
document cameras, and another set of questions only for
courses known to be using the threaded conferencing
utility of the course management system.)
Matrix
surveys open up new paradigms for survey research and
program evaluation.
Example: Using Matrix
Surveys for Student Course Evaluation
Consider the example of student
course evaluation. Imagine a
student course evaluation survey for Prof. Smith's 2 pm
political science course. Automatically created with
Flashlight 2.0, it includes 3 questions from the Writing
Across the Curriculum program (because this course has been
designated as writing intensive), 4 questions from a space
planning team (because it's taught in a room that has been
recently renovated), 3 questions from a faculty community of
inquiry that includes Prof. Smith which is studying
techniques for moderating student discussion online, 1
question for the promotion and tenure process that is
included in all end-of-course forms, and 4 questions from
Prof. Smith written specifically for this section.
Data from the last questions can only be seen by Prof.
Smith, which helps her feel free to ask some otherwise risky
questions about some elements of the course that had clearly
challenged the students in this section.
Matrix
surveys like this can include questions from many authors
(the faculty, the Writing Across the Curriculum program,
...) and can be tailored for each section (respondent pool). All
that data, even data from surveys administered at different
dates can stream into one database for analysis and, if you
wish, can stream out privately to the various authors (so
that, if you like, they can't see one another's data.
We call it a matrix survey
because we visualize each group of people who get the same
items as a row in a matrix, while each author is represented
by a column. The students in each row get the questions
marked with an "X".
|
Question Groups
Respondent Pools |
Prof. Smith's questions
for the 10 am Calculus
class |
Group of questions for all
lecture courses, from the Assessment Program |
Questions for
writing intensive courses from the Writing Program |
Questions
for courses meeting in computer classrooms, from the
IT department |
(More
questions from other units) |
| Prof. Smith's 10 am Wednesday
Calculus section |
X |
X |
|
X |
|
| Prof. Smith's 2 pm Thursday Calculus
section |
|
X |
|
X |
|
| Prof. Black's chemistry
laboratory |
|
|
X |
|
|
| Prof. Green's dance studio |
|
|
|
|
|
| (more) |
|
|
|
X |
|
In the past, end-of-course evaluations have been created for
top-down control -- the instrument would be created or adopted centrally, and
imposed on all courses so that administrators in departments
and at the institution can make thumbs up orthumbs down
decisions about faculty, e.g., raises, promotion, retention.
Such uniform instruments can be used
(and used easily) with Flashlight Online 2.0. But Flashlight
Online's real power lies in creating a
more pluralistic approach -- a culture of inquiry and
evidence -- in which faculty, departments, offices, even
student organizations could all be given permission to
gather evidence in order to improve their own work. Because
of this ability to provide different forms of the survey to
different groups of respondents, and to easily provide
tailored reports to different authors, a single survey of
students can provide evidence to help:
-
faculty improve teaching,
-
students to learn which
ways of studying are more successful,
-
the IT organization to
improve services,
-
the writing program to
enhance writing across the curriculum,
-
the facilities office to
evaluate and improve classrooms,
-
the library to upgrade
information literacy, and
-
the distance learning
program to assure quality in that program.
If you're a Flashlight Online
author, here
is a more detailed explanation of how to create a matrix
survey.
Using a Survey of faculty to Create a Matrix Survey of
their Students
In the example described
above, the author of a survey already knows which respondent pools
should receive each type of question (e.g., allocating
certain questions about writing only to courses that have
been designated writing intensive in the catalogue.)
It's also possible to
give the instructor a role in deciding which questions
should be asked of students in that section. The author can
prepare an earlier survey (online form) for instructors,
offering a menu of teaching/learning activities (e.g.,
lecturing, facilitating seminars and other forms of
classroom interaction, facilitating online interaction,
designing and running laboratory experiences, coaching dance
and other physical performances, ...). The instructor
would be asked to identify 3-5 such activities are of
special importance to this course. The instructor's
choices would determine which questions about
teaching/learning activities would appear in the response
form for that course. For example, if the instructor had
checked "lecturing," the students in that response section
might be asked to judge the degree to which the lectures had
motivated them to study and also asked for suggestions for
how the lectures might be improved in future courses.
Why Use
a Matrix Survey?
Let's look beyond student
course evaluation. Why use matrix surveys?
-
Study of something that
varies in ways you understand across the people you'd
like to study. For this reason, you need to ask
different questions of different subgroups of the
population. For example, suppose you were doing a survey
of military retirees and you wanted to ask somewhat
different questions of the veterans depending on when
they served, their branch of service, and their
specialty (all information that you have about them
before sending them the survey). Or suppose, as an IT
services department, you were studying the use of a new
technology; you might first survey faculty to see how
they were using a technology, and then use a matrix
survey to gather data from their students, asking mostly
about the ways that the faculty report using that
technology.
-
Different stakeholder
all want to ask their own questions of a population, or
subgroups of a population, and want to study not only
their questions but also questions asked by others. Take
our student course evaluation example above: the writing
program may want to ask questions about writing
assignments and feedback in writing intensive courses,
and relate that data to data collected by other authors
from those same courses (e.g., teaching style, use of
technology).
-
What uses do you see in
your discipline? in education? business? health care?
sociology? political science? criminal justice? women's
studies? area studies? geography? ...
Other Applications of
Matrix Surveys
Matrix surveys can be used to
study any phenomenon, product, or idea that is predictably used differently, or is
seen differently, by different groups of people. For
example, The TLT Group is developing a
matrix survey of students to help guide
ePortfolio initiatives (because ePortfolios are used
differently in different courses, programs, and
institutions). And we've also been developing a
prototype matrix
survey that institutions can use to help faculty get
feedback on their various uses of Student Response Systems
(e.g. clickers).
Matrix surveys can also be
used by faculty and by students for survey research in their
disciplines, on topics ranging from market research to
health care. Matrix surveys provide a new flexible, focused
tool that should open new research frontiers in several
disciplines.
And because matrix surveys offer the
option of collaboration among multiple authors, still more
options open. For example, let's imagine a study of
healthcare statewide, with response forms tailored to the
location of the respondent. Some of the questions
could come from the health care programs themselves as well
as from relevant government programs and public interest
groups, giving them all more of a stake in the study and its
data.
Conditional Questions
- A Complement to Matrix Surveys
Flashlight
Online 2.0 will also offer conditional questions -
questions which a respondent sees only if earlier questions in the
survey were answered in a certain way. "Branching" and "skip
patterns" are types of conditional questions. For
example, if an earlier question asks, "Have you ever used X"
and the person responds "yes", and if the person has also
responded that he or she is over 21, then (and only then)
would the respondent see questions about X.
In general, authors should
use a conditional question for issues where the respondent
is the best judge of the relevance of the conditional
question. In contrast, matrix surveys are appropriate
when it's more appropriate for someone else to make that
judgment (e.g., in a primary election where only registered
Republicans are allowed to vote for Republican candidates),
a matrix design is more appropriate than asking in the
survey, "Are you a registered Republican?" Matrix
surveys are especially valuable when an inappropriate
respondent is unlikely to understand the diagnostic question
(e.g., "do you use ePortfolios to support reflection?")
Your Role?
We are eager to work with our
Network members to
develop research strategies that use a matrix design (and
conditional questions). We can use that experience to do
some real pioneering, while also further improving
Flashlight Online 2.0. If your institution is a subscriber
(or would like to be) and you would like to help us develop
this line of work, please contact Steve Ehrmann
(ehrmann@tltgroup.org).
|
