Matrix Surveys

 

Flashlight Online log-in l About Flashlight Online l Handbook and Other Materials l ARQ l
F-LIGHT l Training, Consulting & External Eval.l Student Course Evaluation l FAQ

 Matrix Surveys & Student Course Evaluation l Using a Survey to Create a Matrix Survey l Why Create a Matrix Survey? l
How to Create a Matrix Survey with Flashlight Online 2.0 l Matrix Surveys & ePortfolio Initiatives l
Matrix Surveys & Student Response System (e.g., "clicker") Initiatives l Flashlight Online 2.0

In creating surveys and other online forms, authors usually face two barriers:

  1. All respondents must receive the same ("lowest common denominator") questions. That's one reason why respondents often see questions as rather bland and sometimes also off-target, confusing, or (seemingly) stupid.

  2. The survey must be designed by a single person or team, from top to bottom.

That all seems inevitable. But consider a modern election, especially a primary, statewide election. Ballots (surveys) are created for all voters based on their home address and, in a primary, also on their political party (metadata about the voter). Voters who live in the same area and are registered in the same party get identical ballots. The questions (candidates, referenda) are chosen by different stakeholders (party organizations in specific districts, governmental units).  The questions on the ballot are those (and only those) that are relevant to each group of respondents. But it is all one election: the ballots are analyzed statewide, and reports are created for all those stakeholder groups. We call this a matrix survey.  And, using Flashlight Online 2.0, it's to create and administer a matrix survey.

In a matrix survey:

  1. Different subgroups of respondents can receive different sets of questions, questions chosen and worded for that particular pool of respondents. (For example, in a study of student response systems where one course uses devices that students call 'clickers' while students in another class refer to "polling systems", the wording can be tailored for each class. If one set of courses uses clickers for peer instruction and another set does not, only the first set of courses will get questions about peer instruction.

  2. If several groups of stakeholders need data from the same large population, they can each contribute questions, using metadata to assure that their questions only go to that subset of the total population that will  understand and respond. (For example, in a survey given out at the end of all courses at a university, the writing program may need data (only) from writing intensive courses, the IT department may have some questions (only) for courses using document cameras, and another set of questions only for courses known to be using the threaded conferencing utility of the course management system.)

Matrix surveys open up new paradigms for survey research and program evaluation.

Example: Using Matrix Surveys for Student Course Evaluation

Consider the example of student course evaluation. Imagine a student course evaluation survey for Prof. Smith's 2 pm political science course. Automatically created with Flashlight 2.0, it includes 3 questions from the Writing Across the Curriculum program (because this course has been designated as writing intensive), 4 questions from a space planning team (because it's taught in a room that has been recently renovated), 3 questions from a faculty community of inquiry that includes Prof. Smith which is studying techniques for moderating student discussion online, 1 question for the promotion and tenure process that is included in all end-of-course forms, and 4 questions from Prof. Smith written specifically for this section.  Data from the last questions can only be seen by Prof. Smith, which helps her feel free to ask some otherwise risky questions about some elements of the course that had clearly challenged the students in this section. 

Matrix surveys like this can include questions from many authors (the faculty, the Writing Across the Curriculum program, ...) and can be tailored for each section (respondent pool). All that data, even data from surveys administered at different dates can stream into one database for analysis and, if you wish, can stream out privately to the various authors (so that, if you like, they can't see one another's data.

We call it a matrix survey because we visualize each group of people who get the same items as a row in a matrix, while each author is represented by a column. The students in each row get the questions marked with an "X".

Question Groups

 

Respondent Pools

Prof. Smith's questions
for the 10 am Calculus
class
Group of questions for all
lecture courses, from the Assessment Program
Questions for
writing intensive courses from the Writing Program
Questions for courses meeting in computer classrooms, from the IT department (More questions from other units)
Prof. Smith's 10 am Wednesday Calculus section X X   X  
Prof. Smith's 2 pm Thursday Calculus section   X   X  
Prof. Black's  chemistry laboratory     X    
Prof. Green's dance studio          
(more)       X  

In the past, end-of-course evaluations have been created for top-down control -- the instrument would be created or adopted centrally, and imposed on all courses so that administrators in departments and at the institution can make thumbs up orthumbs down decisions about faculty, e.g., raises, promotion, retention.

Such uniform instruments can be used (and used easily) with Flashlight Online 2.0. But Flashlight Online's  real power lies in creating a more pluralistic approach -- a culture of inquiry and evidence -- in which faculty, departments, offices, even student organizations could all be given permission to gather evidence in order to improve their own work.  Because of this ability to provide different forms of the survey to different groups of respondents, and to easily provide tailored reports to different authors, a single survey of students can provide evidence to help:

  • faculty improve teaching,

  • students to learn which ways of studying are more successful,

  • the IT organization to improve services,

  • the writing program to enhance writing across the curriculum,

  • the facilities office to evaluate and improve classrooms,

  • the library to upgrade information literacy, and

  • the distance learning program to assure quality in that program.

If you're a Flashlight Online author, here is a more detailed explanation of how to create a matrix survey.

 

Using a Survey of faculty to Create a Matrix Survey of their Students

 In the example described above, the author of a survey already knows which respondent pools should receive each type of question (e.g., allocating certain questions about writing only to courses that have been designated writing intensive in the catalogue.)

 

It's also possible to give the instructor a role in deciding which questions should be asked of students in that section. The author can prepare an earlier survey (online form) for instructors, offering a menu of teaching/learning activities (e.g., lecturing, facilitating seminars and other forms of classroom interaction, facilitating online interaction, designing and running laboratory experiences, coaching dance and other physical performances, ...).  The instructor would be asked to identify 3-5 such activities are of special importance to this course.  The instructor's choices would determine which questions about teaching/learning activities would appear in the response form for that course. For example, if the instructor had checked "lecturing," the students in that response section might be asked to judge the degree to which the lectures had motivated them to study and also asked for suggestions for how the lectures might be improved in future courses.

 

Why Use a Matrix Survey?

Let's look beyond student course evaluation. Why use matrix surveys?

  1. Study of something that varies in ways you understand across the people you'd like to study. For this reason, you need to ask different questions of different subgroups of the population. For example, suppose you were doing a survey of military retirees and you wanted to ask somewhat different questions of the veterans depending on when they served, their branch of service, and their specialty (all information that you have about them before sending them the survey). Or suppose, as an IT services department, you were studying the use of a new technology; you might first survey faculty to see how they were using a technology, and then use a matrix survey to gather data from their students, asking mostly about the ways that the faculty report using that technology.

  2. Different stakeholder all want to ask their own questions of a population, or subgroups of a population, and want to study not only their questions but also questions asked by others. Take our student course evaluation example above: the writing program may want to ask questions about writing assignments and feedback in writing intensive courses, and relate that data to data collected by other authors from those same courses (e.g., teaching style, use of technology).

  3. What uses do you see in your discipline? in education? business? health care? sociology? political science? criminal justice? women's studies? area studies? geography? ...

 

Other Applications of Matrix Surveys

Matrix surveys can be used to study any phenomenon, product, or idea that is predictably used differently, or is seen differently, by different groups of people. For example, The TLT Group is developing a matrix survey of students to help guide ePortfolio initiatives (because ePortfolios are used differently in different courses, programs, and institutions). And we've also been developing a prototype matrix survey that institutions can use to help faculty get feedback on their various uses of Student Response Systems (e.g. clickers).

 

Matrix surveys can also be used by faculty and by students for survey research in their disciplines, on topics ranging from market research to health care. Matrix surveys provide a new flexible, focused tool that should open new research frontiers in several disciplines. 

 

And because matrix surveys offer the option of collaboration among multiple authors, still more options open. For example, let's imagine a study of healthcare statewide, with response forms tailored to the location of the respondent.  Some of the questions could come from the health care programs themselves as well as from relevant government programs and public interest groups, giving them all more of a stake in the study and its data.

 

Conditional Questions - A Complement to Matrix Surveys

 Flashlight Online 2.0 will also offer conditional questions - questions which a respondent sees only if earlier questions in the survey were answered in a certain way. "Branching" and "skip patterns" are types of conditional questions.  For example, if an earlier question asks, "Have you ever used X" and the person responds "yes", and if the person has also responded that he or she is over 21, then (and only then) would the respondent see questions about X.

 

In general, authors should use a conditional question for issues where the respondent is the best judge of the relevance of the conditional question.  In contrast, matrix surveys are appropriate when it's more appropriate for someone else to make that judgment (e.g., in a primary election where only registered Republicans are allowed to vote for Republican candidates), a matrix design is more appropriate than asking in the survey, "Are you a registered Republican?"  Matrix surveys are especially valuable when an inappropriate respondent is unlikely to understand the diagnostic question (e.g., "do you use ePortfolios to support reflection?")

 

Your Role?

We are eager to work with our Network members to develop research strategies that use a matrix design (and conditional questions). We can use that experience to do some real pioneering, while also further improving Flashlight Online 2.0. If your institution is a subscriber (or would like to be) and you would like to help us develop this line of work, please contact Steve Ehrmann (ehrmann@tltgroup.org).

 

One Columbia Avenue,
Takoma Park, Maryland 20912
Phone
: 301.270.8312/Fax: 301.270.8110  

To talk about our work
or our organization
contact:  Sally Gilbert

Search TLT Group.org

Contact us | Partners | TLTRs | FridayLive! | Consulting | 7 Principles | LTAs | TLT-SWG | Archives | Site Map |