|
As we revamp our web site in 2007, I
took a fresh look at this otherwise unpublished essay and
realized, to my surprise, that it's still useful reading.
Steve Ehrmann
1. Simple summative questions l
2. Is technology magic? l 3.
Is your investment helping the people who most need help?
l
4. How does your use of technology affect
costs? l 5. Is technology helping a
department or school create widespread improvements in
practice? l Conclusions
In my
visits to colleges and universities, I sometimes ask folks
what studies or evaluations of teaching, learning, and
technology have been useful for them. "The studies need not
have been published or rigorous," I assure them. "I'm just
interested in studies whose findings were useful to you. The
research may have reported a success or failure, a trend, or
a general tendency -- some important finding with relevance
to teaching, learning, and technology." After taking some
time to think about the question, they usually report that
they know of few such studies, if any.
Everyone
wishes they had good data about teaching, learning, and
technology but few institutions are doing the work to get
it. That's dangerous. Technology changes quickly and
unpredictably, technology budgets are large and getting
larger, money is tight, and the higher education world is
turbulent and unpredictable. Faculty and administrators are
making large investments of time and money with their eyes
closed. In such a world it is important to get some
information so institutions can see what they're doing, fix
problems, and document achievements.
The good
news is that good studies are being done. Local evaluative
studies have produced important insights that have reduced
risks, guided policy and shaped practice. Provosts and chief
information officers just don't hear about these studies at
other institutions because their normal information channels
don't bring them this kind of information.
It's
worth spreading the word, however. These successful studies
demonstrate there are certain evaluative questions about
teaching, learning, and technology that more institutions
need to ask for themselves.
1. Simple
summative questions
Some
study questions focus on results. When Rensselaer
experimented with offering computer-intensive studio courses
for first-year students instead of the classic
lecture-discussion-laboratory combination, for example, an
early sign of success was a sharp increase in attendance.
This was especially dramatic during the middle of the term
when lecture attendance had traditionally been low. RPI also
asked students in these courses questions such as
-
"Was
this course so good that it would justify recommending
to a high school senior that he or she attend this
university?" and
-
"At
Thanksgiving, what would you tell your friends at home
about the courses at Rensselaer?"
Because
RPI has been asking such questions for years about various
courses, the substantially higher scores for the studio
courses also provided some early signals that the innovative
courses were on the right track.
Ramapo
College's Teaching, Learning, and Technology Roundtable
recently surveyed faculty about their uses of technology in
teaching; about half the full-time faculty responded. They
asked some pointed questions such as "Do you agree that
student competence with technology is not essential for the
courses you teach?" and got some equally dramatic answers
(63% disagreed - student competence with technology had
become essential for their courses.). Similarly 75% of the
respondents agreed that their uses of technology were
helping students understand course content. Findings such as
these help to provide hints about whether technology use has
become an integral element of the academic program.
Top of Page
2. Is
technology magic?
Technologies such as computers (or paper) don't have
predetermined impacts. It's the uses of such
technologies that influence outcomes. The statement seems
obvious, but many institutions act as though they believed
that the mere existence of technology would improve
learning. They use computers to teach the same things in the
same ways as before, yet they expect learning outcomes to
improve. It's an easy mistake to make, especially if your
attention is distracted.
Good
evaluative questions, on the other hand, can help focus
attention where it matters. A couple of years ago, for
example, Prof. Susanmarie Harrington and her colleagues in
the composition program at Indiana University Purdue
University Indianapolis (IUPUI) took a fresh look at their
program's use of technology in the classroom. Working with
our Flashlight Program, she and her colleagues
studied
the role that technology played in the quality and cost of
lower division composition courses. They compared sections
taught in computer classrooms with other sections taught in
traditional classrooms.
Not
surprisingly, it was more expensive to teach composition in
the computer classrooms. Not only was the equipment
expensive. The classes were smaller (due to the capacity of
the rooms); thus the cost of staff per student was higher.
Other factors tended to make the sections in computer
classrooms cheaper (e.g., staff teaching in these rooms
reported spending a bit less time in preparation and
teaching than did their colleagues teaching larger classes
in traditional classrooms). On balance, however, IUPUI found
that it was more expensive to teach composition in these
classrooms.
Were
those classes better than those taught in traditional rooms?
Apparently not. Using surveys designed with the Flashlight
Current Student Inventory, Harrington and her colleagues
discovered that the teaching and learning practices in the
computer classrooms were essentially the same as in the
traditional classrooms. This had been done on purpose. The
computer classrooms and traditional classrooms housed
different sections of the same course. Faculty wanted to be
fair to all students so they taught all students in the same
way. Why use expensive facilities while carefully not taking
advantage of them? The issue of cost had not been of direct
concern to faculty members, perhaps because the composition
program was not charged for the facilities it used.
Flashlight findings reminded faculty that was a conflict
between their efforts to use technology wisely and
efficiently and their desire to treat students fairly.
Prof.
Harrington concluded,
"As a result of
the Flashlight inquiry, we are reconsidering whether the
[effort] to standardize across sections affects the
possibilities for the computer-assisted sections, and
whether we should explore ways in which the
computer-assisted sections could achieve program goals
by different means. At the time the data were collected,
however, the common syllabi were a powerful influence on
student experiences; given that there were no
significant differences between curricula, it is not
surprising that student learning practices were so
similar across sections."
So your
department may also wish to consider the IUPUI question: if
large courses are being taught in multiple sections, some of
which use more technologies than others, are faculty and
students able to take advantage of the technology? Have
they, for example, been able to change the content or to
make pervasive changes in teaching and learning practices?
Or are concerns for equity or other factors insuring that
costs increase but that outcomes do not improve?
Top of Page
3.
Is your investment helping the people who most need help?
Another
thing that Harrington found was that some IUPUI composition
students faced multiple disadvantages. Students with the
poorest computer skills also had lower GPAs. Perhaps because
of their poor skills and poor access to technology, these
students also spent fewer hours per week using e-mail and
word processing for their writing courses. (Poor computer
skills were not a gender issue, by the way, even though the
students with the best computer skills were mostly male.)
Harrington concluded that because "students with low
technical skills are so few in number, they can easily
become an invisible group, scattered among sections.
Teachers must be more vigilant about identifying these
students early, and providing assistance early and often."
Are
students with poor technical skills actually getting the
technical support they need? Mount Royal College, a two-year
institution in Calgary, Canada, asked that question. Patti
Harvey, a consultant, took the lead in designing the
study
. Mount Royal faculty had assumed that students with the
least computer experience would be the most frequent users
of technology support services such as Student Tutor And
Resource Technicians (START). For a set of five composition
courses early in 1998, Mount Royal's Flashlight-based survey
found the opposite: students new to technology were least
likely to know about START and other helping resources and
least likely to use them. Students with more than ten years
of computer experience were also under-using the technology
support services: most of them were over-confident. It was
students with a moderate amount of experience who seemed
most knowledgeable about helping resources, and most
efficient in selecting the help they needed. Armed with
these findings, Mount Royal instructors explained to
different types of students the ways in which the helping
resources could be useful to them. Some months later, the
evaluation was repeated. This time, data showed that
students were using the training and support materials,
whether they had just a little computer experience or a lot.
As a result, students now reported few problems or issues
with instructional technology applications. Mount Royal's
studies are instructive in two ways:
-
revealing what is probably a common problem (computer
novices and experts making too little use of important
helping resources) and
-
demonstrating how evaluation can both lead to a solution
and then document that the problem has indeed been
solved.
Are your
institution's uses of technology unfair to some students? If
so, which technologies discriminate against certain
students: the old technologies or the new ones? In a study
done in the late 1980s, Roxanne Hiltz of the New Jersey
Institute of Technology compared courses taught in virtual
classrooms with courses taught in traditional classrooms.
(It was Hiltz and her husband Murray Turoff who coined the
term "virtual classroom.") In the traditional classrooms,
she discovered, students whose native language was not
English got lower grades than did native speakers. However,
in courses taught in virtual space in that same year and in
the same institution, the grades for the two groups of
students were the same. In the virtual classroom, students
had more time to think about what has been said and to
decide how to reply. The virtual classroom did not
discriminate against non-native speakers. The traditional
classroom facility did.
Are some
types of students facing barriers at your institution? In
technology-intensive courses? In courses taught in more
traditional ways in more traditional facilities? If you can
identify the barriers, perhaps you can lower them.
Top of Page
4.
How does your use of technology affect costs?
Some of
the toughest questions deal with costs. The high stakes make
it harder to collect data. And framing the study is
difficult. What standard of judgment should be used to
decide whether an activity has cost "too much" or whether it
has "saved money" (compared with what?). Which expenditures
should count in the model? Today's operating expenses are
always included, but what about yesterday's capital
spending? Spending by students? The value of faculty time?
Student time? Staff time? Despite the difficulties, one can
make some sensible assumptions, build some models of how
time and money are being "spent," and draw some important
conclusions.
At
Washington State University, for example, Tom Henderson
analyzed the costs of three different approaches to creating
technology-based course materials and experiences. Using
Flashlight Program's approach to cost modeling, Henderson
studied:
-
Development of InterFon, an instructional module to help
students learn the sounds associated with phonemes in
foreign languages. The student can click on a phoneme to
hear the sound. Quizzes and writing assignments about
phonemes can be integrated into InterFon as part of the
program.
-
Translation by a faculty member of his course material
into an online format (combining content with strategic
email response boxes); and
-
Development of a "template" web forum for organizing
threaded discussions that could be used for many
courses.
Henderson
and his colleagues summarized their findings in two ways:
total costs of each program, and costs per student for each
program:
-
Total costs: Creation of the template for a Web
forum was the most expensive activity. InterFon was
moderately expensive and the translation of course
material to an online format was the least expensive
activity.
-
Costs per student: However, when costs were divided
by weeks of student use, the modular approach
(development of InterFon) cost $14,000/student week. It
was by far the most expensive. The other two approaches
were both relatively inexpensive to develop due to the
large volume of subsequent student use (each cost about
$8/student week to develop).
Henderson
and his colleagues concluded that tailored modules need to
have very specific and justifiable goals (e.g., opening an
instructional bottleneck with substantial benefits for
learning in later courses) in order to justify their steep
costs per student. The cost of putting a course online is
far more affordable but requires advanced faculty skills.
The Web template placed some limitations on teaching and
learning approaches but combined acceptable costs with a
lower threshold of required faculty skill in using
technology. The question for your institution: how do
different methods of improving courses compare if you
compare the cost per week of student use of that
improvement?
If your
institution has problems with attrition, another
illuminating indicator is the cost per student passing a
course or the costs per graduate. For example, Baruch
College started an experimental course called "College
Literacy" designed for students with poor skills in reading
and writing. A section that did not use computers suffered a
failure rate of almost 50%. Doing better were students in an
experimental course that featured considerable use of a
real-time computer conferencing system called Daedalus. The
instructor's assessment of the exit exams (each of which was
graded by three independent readers) was that "the students
in the computer-enhanced section consistently wrote longer
...essays rich in ideas and details, organizationally
complex, remarkable in their fluency." Pass rates for the
experimental course were also much better: about 75%.
Although
most costs were the same (space, faculty time) for the two
courses, the conferencing software added about 7% to the
cost per student. However, so many more students passed this
course that Baruch estimated that the costs per passing
student were 29% less than the costs per student
passing the course that did not use computers.
Time is
money in academe: most of our budgets go to the time of a
set of people who are on long term contracts. Thus the
problem of cost control is closely linked to the challenge
of helping staff make rewarding and efficient use of their
time. The Rochester Institute of Technology applied the
fledgling Flashlight cost analysis approach to several
different methods for teaching distance learning courses.
They found something quite surprising. When experienced
instructors were first interviewed, they all immediately
remarked that teaching at a distance was more time-consuming
than teaching on-campus. Then they were asked to break down
their activities into different functional categories, both
for courses taught on-campus and off-campus. Then a
dramatically different picture emerged. About one third of
the faculty had indeed spent more time teaching their course
for off-campus students. But another third had the opposite
experience: it was their on-campus courses that were
more time-consuming. And the remaining third found that they
were spending equal time for on- and off-campus courses.
Perhaps first impressions about one's own time use can be
deceiving. If so, studies like this may help faculty and
staff members make more rewarding use of their time while
helping the institution improve education with its current
staff and budget.
Top of Page
5.
Is technology helping a department or school create
widespread improvements in practice?
Very few
institutions are asking whether their uses of technology are
fostering institution-wide changes in teaching and learning
practice. It's easy to understand why: teaching and learning
practices are the province of individual faculty teaching
isolated courses. It's hard to see what's going on in more
than one class at a time. But we need to be able to see
across the curriculum. Educational outcomes for graduates
are largely determined by patterns of teaching and learning
that transcend individual courses. For example, if students
graduate knowing how to write well, it's the fruit of their
whole education, not a single course. The same is true for
other key competencies such as critical thinking or their
command of their major subject. So it's important to
understand whether technology is enabling changes in those
patterns of teaching and learning. For example, if an
institution values collaborative learning and the
development of skills of teamwork, it's important to
discover whether e-mail is being used to foster that kind of
learning across the curriculum. It's also crucial to monitor
whether the institution is learning to take advantage to its
technologies to make such improvements in teaching and
learning year by year.
One of
the few institutions to study the role of technology in
fostering large-scale patterns of change in teaching is
Valley City State College in North Dakota. This institution
requires student ownership of notebook computers. Kathryn
Holleque and her colleagues asked students many questions
about their uses of computing. For example, students were
asked whether using the notebook computer helped them
organize information in personally meaningful ways; most
agreed that it did. The Valley City State study also looked
directly at some of the fears about computing. For example,
responding to a 1997 survey, only a few students reported
that they had become even slightly "addicted" to the use of
computers. Slightly larger numbers of students resonated at
least a little to questions about information overload,
sleep deprivation, and social isolation. However, the only
negative hypothesis about universal ownership of laptops
that pops out from the crowd is the 30% of respondents who
agreed "very much" that the laptops distracted them from
what was being covered in class.' I wonder whether that
figure has dropped as students and faculty members have
become more accustomed to the laptops. Studies like these
become even more powerful when done on a regular basis so
that institutions can discover whether good practices are on
increase and whether problems seem on their way to being
solved.
These
days, most institutions offer a stunning variety of
information utilities for students, faculty members, and
staff, including publicly available computers, Internet
connections to dorms and offices, e-mail accounts, and the
like. Each element of this technology infrastructure can be
used for many educationally important activities. Each
activity can potentially be supported by many elements of
the infrastructure. Is it possible to say anything coherent
and concrete about whether and how this technology is being
used to improve the academic program?
The first
step in creating such a study is to select an educational
focus. Identify a few absolutely crucial educational
activities, practices or processes that technologies should
already be making easier and productive. For example, a
commuter campus might choose to look closely at how students
are collaborating on their homework while off-campus. When
you are selecting such educational foci for a self-study,
consider activities that fall within Chickering and Gamson's
Seven Principles of Good Practice in Undergraduate Education
(e.g., student-student interaction, student-faculty contact,
active learningÖ) These are the kinds of activities that
(researchers tell us) often foster better learning outcomes.
So if you find that technology is being used to foster such
good practices, you're part way toward showing that it's
being used in ways that improve quality and access.
Step two,
equally focused, is to select just a few technology
utilities that are crucial for carrying out those
educational activities. For example, if student
collaboration on homework and projects has been selected,
the study might focus on e-mail and on the Web-as-library.
Notice that we're now talking about a very small, very
important slice of the institution ñ not all of the
educational program, not all of the technology ñ just a
slice that is small enough to study and important enough to
matter.
Step
three: study the extent how, how much, and how well those
particular technologies are being used in those particular
educational activities. If technologies aren't being used to
expand and enrich those activities, why not? Are the
activities on the upswing each year thanks in part to
better, wiser use of the technologies?
Such
studies will probably produce a mix of good news and bad
news. The bad news is key to making improvements. For
example, if students aren't collaborating on their homework,
perhaps instructors are discouraging the practice or are
failing to help students learn how to do it. Some barriers
may be technology-related: for example, do students have
access to the technology they need? Are the support services
giving them help when they need it?
Armed
with such findings, faculty and staff will be in a much
better position to make improvements. And the more faculty
and staff have been involved in selecting the focus of the
study, the more are likely to act on its findings.
Some
Conclusion
Since the
mid-1970s, I've been involved with evaluation. Since my
first days doing educational research at MIT and The
Evergreen State College, I've known that educational
innovators, like nocturnal creatures, do most of their work
in the dark. Technology -- with its unpredictable changes
and vast expenditures -- has made flying blind much more
dangerous than it used to be. The good news is that some
institutions are learning how to see.
Stephen C. Ehrmann, January 19, 1999
1. Simple summative questions l
2. Is technology magic? l 3.
Is your investment helping the people who most need help?
l
4. How does your use of technology affect
costs? l 5. Is technology helping a
department or school create widespread improvements in
practice? l
Conclusion |
