|
1. Select Activities l
2. What other ingredients needed? l
3. Monitor Activities l
4. Debug Activities l
5. Diagnose Barriers to Participation
l 6. Control Costs l Summary l
Attachment: List of Activities
l
Part II: Using Student Feedback to Improve ePortfolio
Activities l Flashlight Evaluation Handbook Table of Contents
These
materials are for use only by institutions that subscribe to
The TLT Group, to participants in TLT Group workshops that
feature this particular material, and
to invited guests. The TLT Group is a non-profit whose
existence is made possible by subscription and registration
fees. if you or your institution are not yet among
our subscribers,
we invite you to
join us, use these materials, help us
continue to improve them, and, through your subscription,
help us develop new materials! If you have questions
about your rights to use, adapt or share these materials,
please ask us (info @ tltgroup.org).
Revised
August 22, 2008
"Formative Evaluation"
If you know what we mean by this term, you can skip this
section. "Formative evaluation" of ePortfolio use is
an intentional inquiry designed to produce insights that
will enable authors and other stakeholders to get more value
from the ePortfolio-support activity. In contrast,
"summative evaluation" is information to help stakeholders
decide what the benefits and costs of that activity have
been: "how are we doing?" and "how did we do" (summative) in
contrast to "how can we do better?" (formative)
For example, imagine a professor using ePortfolios to
encourage students to think more consciously about what they
have been learning in order to deepen that learning and plan
for future learning. The faculty member calls this activity
"reflection."
- A formative question might be to ask students to
describe their own definition of reflection and to
describe how reflection has influenced their own
thinking about what to study next.
- Another formative question might be to ask students
if they've had trouble uploading their reflections into
the ePortfolio.
Student answers to each of those questions could help the
faculty member decide what to do next in the course, or how
to help individual students to do better in the course.
Student Surveys for Formative Evaluation -
Feedback for the Instructor
Step 1 for a program is to encourage and help instructors
use student feedback to figure out how to fine tune their
use of ePortfolios in a course. (See below for some
ideas for questions.) Once there's some base of instructors
who have found it useful to get student feedback, you may be
ready to go to the next stage: a collaboration among leaders
of the ePortfolio initiative with individual faculty,
designed to help both groups get valuable feedback.
Student Surveys for Formative
Evaluation - Feedback for the Instructor AND for the
ePortfolio Initiative
Overview: We suggest a
pluralistic approach to formative evaluation: one that gives
participating faculty and administrators to ability to a)
contribute questions to the total inquiry, b) ask students
only those questions relevant to what that student has
actually been doing with ePortfolios, c) give people reports
on only on those questions that are most useful for them.
This approach uses a new kind of feedback system called a
"matrix survey."
For this discussion, let's consider five
major activities for which ePortfolios are used and for
which student feedback could be informative:
A. Deepen learning via
reflection (e.g., reflection on how the work itself,
sometimes in combination with other artifacts, provides
evidence of capability; reflection on development of a
capability)
B. Deepen learning via students describing their own goals
and capabilities.
C. Deepen learning, and relationships, by getting more kinds
of people to assess the student’s learning.
D. Deepen learning via use of online projects, some visible
to real audiences. [It presumably makes it easier to shift
toward an emphasis on student projects, many of them
published online to be visible to and perhaps used by
audiences other than the instructor, if ePortfolio software
is used.]
E. Selecting work (and assessments), as evidence of personal
capability, for job and grad school applications.
No one course at a university would use an
ePortfolio to support all five activities. In fact most
courses only focus on a couple such activities.
Normally that would create a quandary. How
could you create a formative evaluation survey that asks,
among other things, about deepening learning through
reflection if some of the students to be surveyed would have
no clue about what you're asking about, because their
courses don't use ePortfolios that way. The answer:
use a matrix survey.
What's a Matrix Survey?
Think of the following grid (matrix) as a
set of notes about the ways in which 11 courses use
ePortfolios. The columns represent the five activities
(A-E). The "x"s represent the activities important for
each course. Course 1 uses ePortfolios for students to
set their own goals, and then reflect on their learning
(Activities A &B). That's why the "x's" in the first row are
in those two columns.
| |
A |
B |
C |
D |
E |
| Course1 |
x |
x |
|
|
|
| Course2 |
|
|
x |
|
x |
| Course3 |
|
|
x |
|
x |
| Course4 |
x |
|
|
|
|
| Course5 |
x |
x |
|
|
|
| Course6 |
|
x |
|
x |
x |
| Course7 |
x |
|
|
|
|
| Course8 |
x |
|
|
|
|
| Course9 |
x |
|
|
|
|
| Course10 |
x |
|
|
|
|
| Course11 |
x |
|
|
|
|
Now imagine that each of those "x"s represents some
questions that students could be asked about that activity,
questions whose answers might help someone figure out how to
improve that activity.
* For example, suppose that activity C is giving the student
audiences or users whose opinions and assessment matter to
the student more than the faculty member alone (e.g., having
professionals outside the university assess the student's
ePortfolio and give feedback on the student's skills). The
faculty member is using the ePortfolio this way and hopes to
make this use even better. That instructor might find it
useful to ask students
questions such as these.
* In courses where ePortfolios are being used to get
students to reflect on what they've been learning, student
answers to
questions like these might provide the faculty member
with guidance.
With a matrix survey, those online feedback forms can do
double duty. The faculty member gets a report on student
feedback all the activities that the instructor has chosen.
For example, in course 4, the students are asked only about
Activity A. In course 5, the students are asked about
Activities A and B. But all these course forms feed
into a single database, so the faculty running the program
get reports on seven activities, with student responses
drawn from all the participating courses.
Let's add just one more element to this matrix: a column
representing questions we need to ask no matter what
activity we're talking about (e.g., did you post any work in
your ePortfolio this term). Column H in the matrix
below represents those core questions; that's why. there's
an "x" in every cell in that column.
| |
A |
B |
C |
D |
E |
H |
| Course1 |
|
x |
|
|
|
x |
| Course2 |
|
|
x |
|
x |
x |
| Course3 |
|
|
x |
|
x |
x |
| Course4 |
x |
|
|
|
|
x |
| Course5 |
x |
x |
|
|
|
x |
| Course6 |
|
x |
|
x |
x |
x |
| Course7 |
x |
|
|
|
|
x |
| Course8 |
x |
|
|
|
|
x |
| Course9 |
x |
|
|
|
|
x |
| Course10 |
x |
|
|
|
|
x |
| Course11 |
x |
|
|
|
|
x |
Here's our
first draft of those core questions. As with the other
questions, our criterion is simple: is it likely that
faculty could use this evidence from students, in
combination with other evidence, to improve the course?
Could program leaders use this evidence, in combination with
other evidence, to improve the program? If "yes", then we'd
like to include the question.Procedure
Step 1 is to ask
faculty in pilot test departments whether it would be worth
their time to gather such information from their students.
We'd show them the seven activities and the core questions,
plus examples of the kinds of surveys we could easily
develop for their students, once they select their issues.
-
Mockup of online form for faculty to select
from eight issues
-
Sample
student survey focusing on use of ePortfolio for
reflection and as a means of getting assessment from
more kinds of people, such as peers, other faculty and
outside experts -- assessors whose judgment has
distinctive value and whose judgment is likely to be
meaningful and motivating for the author of the
portfolio. Notice that some questions are designed to
provide direct, useful feedback for use by the
instructor. Other questions are designed primarily for
people supporting and leading the ePortfolio initiative.
We'll discuss the activities and decide together whether
to modify that list. Once the list is ready, we'll ask each
participating faculty member:
- to identify a course which uses ePortfolios,
- to select 1-3 activities -- the ePortfolio uses that
are most emphasized in that particular course,
- to imagine how their students might respond the
draft questions for each of those activities,
- to imagine what he or she could do with that report
on student responses.
- Then we'd ask them to imagine that their students
had responded in unexpected ways: would they still find
the responses helpful?
Building on that foundation, we'd rewrite the items about
each activity. By this time we will have come to trust, or
distrust, our most important working assumption: that
faculty using ePortfolios in the same way will also usually
want to ask their students the same questions about that
activity.
At this point the evaluation tool kit will be ready for a
pilot test in courses at the university.
Using the tool kit, we can easily build real surveys for
each course. Then we will help collect data, provide the
faculty and the program with reports, and then watch to see
whether and how the faculty and the program use the data.
Thanks to Flashlight Online 2.0, all of this is easy.
In fact, once we're ready, the whole process of creating,
administering and analyzing surveys can be automated. All
the participating faculty will need to do is pick issues
from a list, remind their students to respond, and wait a
few days for the report. If faculty wish to add their own
questions, perhaps different questions for each class or
section, that can be easily done as well.
Note: Flashlight Online 2.0
The tool kit itself is being created with
Flashlight
Online 2.0. Flashlight Online 2.0 is based on a
new matrix survey software system being developed with The
TLT Group by Washington State University. Nils
Peterson, the lead designer at WSU, has been an important
collaborator in developing this approach to formative
evaluation.
Flashlight Online 2.0 offers us many options
for this kind of tool kit, including (for example):
-
Faculty selection of issues can
automatically create surveys for their students which
can then be administered online.
-
In the early phases of this work we
could include other institutions in this study, and
analyze the growing pool of data by institution, by
activity, by discipline, etc.
-
The different courses could be surveyed
at different times, over a period of years, and yet the
data would still flow to a single database, and reports
can be generated showing change in response patterns
over time
-
Students can be prompted if they haven't
yet responded, without compromising anonymity (if the
surveys are anonymous)
|
