|
Other
institutional profiles
l
Flashlight
Evaluation Handbook
Johnson C. Smith is a small college near
Charlotte, NC. Thanks to years of hard work there, it's become more
common for faculty and administrators to use data to improve
what they're doing. For example, over half of the faculty
are doing surveys of their own courses to help improve their
teaching. Many of these surveys focus on their uses of
technology.
Among the factors that have helped create
this "culture of
evidence" (the likelihood that people can
and will use data to monitor and improve what they're
doing):
- Role of Accreditors: As they
have with many other institutions, accreditors such as
SACS and NCATE have emphasized the importance of
assessment to JCSU.
- Use of
Mini-grants : The College Fund/UNCF and
Bush-Hewlett grants have provided mini-grants to faculty
for different kinds of course improvement. JCSU staff
designed those grant proposals to UNCF, Bush and other
sources so that faculty recipients would be required to
do surveys to evaluate their progress, and then write
about what they'd learned from the surveys. The
University provides training (often led by other
faculty) in how to use Flashlight Online and other data
to do such evaluations. Faculty do not receive final
payments until their reports are submitted. This
process of action and reflection has helped faculty
learn how to create more powerful, useful studies. The
reports can be included in promotion/tenure portfolios.
- Shared effort: In one of those
grant programs, on learning communities, faculty use
some items in common so that their data can be pooled.
Two of the mini-grant programs help recipients work
together, which has (among other results) helped them
improve their assessment skills.
- Training by peers: Training
is offered frequently to faculty on how to use a
survey tool that's designed for this kind of study (Flashlight
Online). Faculty typically lead these workshops,
which focus on ideas, not just techniques. Specifically,
faculty learn how to think of teaching and learning in
terms of the seven principles of good practice, and how
data can be used to improve those practices and thereby
improve outcomes. They also learn how technology can be
used to improve practice, and how data can be used to
fine-tune the use of technology for that purpose.
Another good feature of the training is that faculty
learn about the ideas by seeing how they are embodied in
real surveys.
- There has been strong academic
leadership to develop a culture of assessment,
especially from JCSU's forceful president, Dorothy
Cowser Yancy.
- Coordination: The head of
faculty development (Phyllis Worthy Dawkins) and the
Director of Information Services (Frank Parker) work
together quite closely to help faculty, sharing the work
and budgets, writing grant proposals together. They have
also worked closely with the head of institutional
research.
- Persistence: this effort has
been going on steadily for about four years.
Administration have been patient and persistent in
supporting the work, and faculty have been persistent,
too. Even when (as often happens) faculty realized that
their first surveys had not been focused enough to
produce useful results, they tried again, often
producing more useful findings the second time.
Sample Studies
- English Prof. Don Mager does regular
end-of-term studies to get feedback on course web sites
and on the uses of PowerPoint in class (including
student presentations using PowerPoint). Mager's studies
also helped him realize that students were not clear
about the interdisciplinary objectives of one of his
courses, so he's made some changes. Mager also has
gotten feedback on the quality of his comments in online
discussion areas, which have helped him change his
approach. "I learned that my prompt had to be focused,
but not lead to a single simple answer." Yet another
study confirmed that students were finding it easy and
useful to record their Chaucerian English using digital
audio.
- Historian Gene Hermitte did a study
of his students' use of concept maps. "I needed to do
this survey. When you're face to face with them, they
don't want to criticize you. This is anonymous and they
can express themselves honestly and objectively. If
they'd come across negatively, I would have dropped the
concept mapping or revised it."
- Profs. Joe Turner and Maria
Papanikolaou (English) are studying how their students
evaluate information they discover with computers;
they're looking for clues about how to help students
learn to think more critically about source materials.
- Asst. Prof. Karen Butler (Health
Education) has, like many other faculty at JCSU, been
studying the pairing of courses with collaborating
faculty (learning communities). "We're asking whether
they feel a sense of community, whether they're
participating in outside activity, and whether all this
is deepening their understanding in distinctive ways.
The survey data helped us see the importance of
budgeting time for these activities, for example." We
asked her what she had learned about doing surveys.
"It's not as easy as it looks! Choosing the right
questions to ask is hard. That's one of the beauties of
using Flashlight Online. It's easy to draft a survey,
get feedback, revise and re-revise before you put it out
there."
- A number of studies have been done of
learning communities. Among other things, findings have
led to an increase in tutorial support for freshmen, in
order to improve retention.
|
