|
1. Select Activities l
2. What other ingredients needed? l
3. Monitor Activities l
4. Debug Activities l
5. Diagnose Barriers to Participation
l 6. Control Costs l Summary l
Attachment: List of Activities
l
Part II: Using Student Feedback to Improve ePortfolio
Activities l Flashlight Evaluation Handbook Table of Contents
These
materials are for use only by institutions that subscribe to
The TLT Group, to participants in TLT Group workshops that
feature this particular material, and
to invited guests. The TLT Group is a non-profit whose
existence is made possible by subscription and registration
fees. if you or your institution are not yet among
our subscribers,
we invite you to
join us, use these materials, help us
continue to improve them, and, through your subscription,
help us develop new materials! If you have questions
about your rights to use, adapt or share these materials,
please ask us (info @ tltgroup.org).
Revised
November 2, 2008
"Formative Evaluation"
If you know what we mean by 'formative section', you can
skip to the next
section. "Formative evaluation" of ePortfolio use is
an intentional inquiry designed to produce insights that
will enable authors and other stakeholders to get more value
from the ePortfolio-support activity. In contrast,
"summative evaluation" is information to help stakeholders
decide what the benefits and costs of that activity have
been: "how are we doing?" and "how did we do" (summative) in
contrast to "how can we do better?" (formative)
For example, imagine a professor using ePortfolios to
encourage students to think more consciously about what they
have been learning in order to deepen that learning and plan
for future learning. The faculty member calls this activity
"reflection."
- A formative question might be to ask students to
describe their own definition of reflection and to
describe how reflection has influenced their own
thinking about what to study next.
- Another formative question might be to ask students
if they've had trouble uploading their reflections into
the ePortfolio.
Student answers to each of those questions could help the
faculty member decide what to do next in the course, or how
to help individual students to do better in the course.
Step 1 for a program is to encourage and help instructors
use student feedback to figure out how to fine tune their
use of ePortfolios in a course. (See below for some
ideas for questions.) Once there's some base of instructors
who have found it useful to get student feedback, you may be
ready to go to the next stage: a collaboration among leaders
of the ePortfolio initiative with individual faculty,
designed to help both groups get valuable feedback.
Student Surveys for Formative
Evaluation - Feedback for the Instructor AND for the
ePortfolio Initiative
Overview: We suggest a
pluralistic approach to formative evaluation: one that gives
participating faculty and administrators to ability to a)
contribute questions to the total inquiry, b) ask students
only those questions relevant to what that student has
actually been doing with ePortfolios, c) give people reports
on only on those questions that are most useful for them.
This approach uses a new kind of feedback system called a
"matrix survey."
For this discussion, let's consider just
three of the
major activities for which ePortfolios are sometimes used and for
which student feedback could be informative.
-
Deepen learning via
reflection (e.g., reflection on how the work itself,
sometimes in combination with other artifacts, provides
evidence of capability; reflection on development of a
capability)
-
Deepen learning, and relationships, by getting more kinds
of people to assess the student’s learning.
-
Selecting work (and assessments), as evidence of personal
capability, for job and grad school applications.
Maybe no one course at a university would use an
ePortfolio to support all three of those activities (and
there are, obviously, others in addition to these three). In fact most
courses only use ePortfolios for a couple such activities.
So can you create a formative evaluation survey
that serves both the individual instructors and also the
staff who support many such instructors. One answer:
use a matrix survey.
What's a Matrix Survey?
Think of the following grid (matrix) as a
responses from faculty who are describing courses in which
they use ePortfolios and for which they would like feedback
from their students. Column B contains an X in each row
where faculty have reported that ePortfolios are used for
reflection. Column C is checked for each course where
faculty reported that ePortfolios were used as a vehicle to
receive feedback from peers, professionals in the field, or
others ('multiple sources'). And Column D is checked if
faculty want student feedback on their efforts to use
ePortfolios to apply for jobs or further schooling.
With a matrix survey created with
Flashlight
Online 2.0, students in each class can receive a
different response form, with a different mix of questions.
Column A represents a group of questions that could be asked
of all courses, regardless of how ePortfolios are used
there.
| |
A |
B |
C |
D |
| Course1 |
x |
x |
|
|
| Course2 |
x |
|
x |
|
| Course3 |
x |
|
x |
|
| Course4 |
x |
|
|
x |
| Course5 |
x |
x |
|
|
| Course6 |
x |
x |
|
x |
| Course7 |
x |
|
|
|
| Course8 |
x |
x |
|
x |
| Course9 |
x |
|
x |
|
| Course10 |
x |
|
x |
|
| Course11 |
x |
x |
|
x |
A Prototype Matrix Survey on ePortfolio Use
by Students
We have created a small proof of concept survey with four
groups of questions. Matrix surveys created with Flashlight
Online 2.0 have other advantages for this kind of survey;
for example, it's quite possible that ePortfolios are called
by different names in different courses. When interested
faculty fill out a form to request a student survey for a
course, we can also asked them what they call ePortfolios in
that course.
To show you how different response forms can be for
the same survey:
- Here's the Flashlight
response form for one respondent pool, an English
class that calls its portfolio "TaskStream", where the
students get the core group of questions, the reflection
group of questions, and job/school application group of
questions.
- Here's the Flashlight
response form for another class, a Civil Engineering
class that calls its portfolio "iWebFolio," where
students see the core questions and the questions on
feedback from multiple sources.
Reports: The faculty member would be sent a report
about student
feedback from that class.
But all these student responses are also flowing into a
single database (even semester after semester, if you like,
so that trends can easily be analyzed). (Matrix surveys are
analogous to a statewide election; people in different
locations may get ballots -- surveys -- with different
candidates -- questions -- on them, but the responses all
flow to one place for centralized analysis.)
So the people running the ePortfolio initiative will be
to analyze student responses, activity by activity, across
courses (and across institutions if this initiative is a
collaborative developed by several institutions). "How
are we doing at using ePortfolios to foster reflection" (in
the courses where that's being tried) and what are the
barriers we should be trying to lower?" "In classes
where both job applications and reflection are goals, do the
two goals reinforce each other, or interfere with each
other?"
Return to home page for
ePortfolio Evaluation,
Flashlight Evaluation Handbook |
