|
The common objection, "You can't compare apples
and oranges," is obviously mistaken. People make choices
about what fruit to eat all the time. It's equally
possible to compare two programs whose outcomes are
qualitatively different. When evaluation a
programmatic change that takes advantage of technology,
comparing "before" and "after", or "experiment" and
"control" almost always requires comparing apples and
oranges.
That's because some of the most important uses of computing
involve qualitative change in outcomes. Some of these
comparisons are relatively subtle, e.g., two political science
courses, one of which uses Geographic Information Systems to
help students learn more visual, iterative skills of
investigating data and the other which asks students to use
census data from paper books. The kinds of skills engendered
by the first course might differ from the second in ways
analogous to the difference between sculpting in clay versus
sculpting in stone -- the first can involve many experiments
and revisions, while the second the student may try the
analysis only one way.
Some of these qualitative comparisons are far
more complex: for example, distance program 1 (e.g.,
a correspondence program) involves a certain number of
students of various characteristics, with certain retention
rates and certain learning outcomes. In contrast, program 2
(e.g., web-based) attracts somewhat different kinds of
students in different numbers, has different retention rates,
and different learning outcomes for those students). Those
learning outcomes differ qualitatively for many reasons, e.g.,
students learning skills of online discourse in the web-based
courses; students doing more web-based research in the
web-based courses.
Here is a "simple" study design for deciding
which of two such programs or courses produces more valuable
outcomes (or if these two designs come in sequence, is the
second improve outcomes, relative to the first).
1. Is there real disagreement about which of
two courses of study (course designs/curricular designs)
produces more valuable outcomes, when one of those courses of
study has qualitatively different goals than the other? If so,
proceed with this design.
2. Can the people who disagree (or their
representatives) agree on a pool of judges who will act for
them in the assessment? If so, proceed with this design
3. Give the judges the design of the two
programs and assessment data about the outcomes of each (i.e.,
how well did each one achieve its own aims).
4. Ask the judges which course of study
produces more valuable outcomes.
Make sense? I haven't seen this done
rigorously but it is done informally all the time.
|
