A Principle-Based Approach to Assessing General Education Through the Majors
Sharon J. Hamilton
Chancellor’s Professor of
English
and
Associate Dean of the
Faculties
IUPUI
shamilto@iupui.edu
The Office for Integrating
Learning
IUPUI
755 West Michigan, Suite
1140
Indianapolis, IN 46202
A Principle-Based Approach to Assessing General Education Through the Majors
Abstract
The learning matrix of the IUPUI student electronic
portfolio (ePort) will be pilot tested during the fall of 2003. Based on our
Principles of Undergraduate Learning, it is intended not only to document and
assess both improvement and achievement in these discipline-transcendent skills
and ways of knowing, but also to serve as a catalyst for deeper, more insightful,
and more connected learning. To students and faculty, both at IUPUI and at
other institutions of higher learning across the country, ePort will appear as
an attractive and effective product. This article unpacks the processes,
dilemmas, and decision-making that went into the development of this ePort
learning matrix.
Institutional Context
The institution and its students
IUPUI (Indiana University Purdue University Indianapolis) is an urban public research-extensive institution located in downtown Indianapolis. Combining both Indiana University and Purdue University programs, it houses twenty-two academic and professional schools, with more than 1600 faculty. A majority of its more than 29,000 students come from the city and surrounding counties; several of its professional and graduate programs, however, such as Medicine, Dentistry, and Engineering, attract a high proportion of international students. Since 1996, more than 50% of our undergraduates attend full time. On the other hand, while more of our students attend full time than ever before, they also work, on average, more than 20 hours a week, creating challenges of time management and opportunities for extended intellectual engagement.
Three years ago, the state of Indiana instituted a community college system, a move that enabled IUPUI to redefine its mission, establish higher admission standards, and expand its honors program. It is consequently in the process of transition from a “default choice” for students who could not get into their first choice of college into a first choice for many students in area high schools. As a result, our retention figures have begun to improve over the past two years, although they are still below the norm for our peer institutions.
The institution and general education
With so many academic and professional schools, some originally proprietary, some following Purdue traditions, and most with roots in Indiana University, the overall approach to general education was, to say the least, haphazard. In 1991, in preparation for the 1992 NCA accreditation visit, a newly-formed Council on Undergraduate Learning at IUPUI, composed primarily of deans of academic units, and the Academic Affairs Committee of the Faculty Council established a Commission on General Education to oversee development of a centrally coordinated approach to general education for undergraduates at IUPUI. At the time, general education was the responsibility of each school, and followed, primarily, a distributive model, wherein each school defined required areas, such as humanities, sciences, and social sciences, and then specified particular requirements within those defined areas. The 1992 NCA Accreditation team noted a need within this distributive approach to identify “desired outcomes for general education…amenable to meaningful assessment.” Bearing in mind both their initial charge to develop a centrally coordinated approach and the NCA mandate to develop specific learning outcomes for general education, several members of the Commission on General Education attended the 1993 Lilly Endowment Workshop on the Liberal Arts. Out of that workshop, and in conjunction with several other campus committee conversations, the Commission initiated “a process approach” to general education. They set up a series of multi-disciplinary committees, day-long retreats, and town halls to explore fundamental values associated with general education.
This process culminated in the IUPUI Principles of Undergraduate Learning (PULs).
The simplicity of that previous sentence belies the complex, often contentious turf-related negotiations, passionate disagreements, and entrenched attitudes that threatened the process. With twenty-two different academic and professional schools carefully guarding their tuition dollars in a responsibility-centered budgeting system, consensus was not just elusive; it appeared at many times to be downright unattainable. Yet more than 200 faculty persevered in trying first to come to some agreement and second to convince their colleagues that a set of common learning outcomes would provide not only a shared intellectual foundation but also a coherent path for IUPUI students through the morass of school-specific and program-specific requirements.
These PULs are significant not in their uniqueness – they are very similar to the undergraduate learning values in almost any institution of learning – but rather in the fact that they are intended to permeate the undergraduate curriculum instead of being a set of courses or skills concentrated in a student’s first two years of college. Students are expected not only to improve their level of competence in each of the PULs during their first and sophomore years, but also to continue to improve their level of competence throughout their undergraduate learning experiences.
Identification of Learning Outcomes to be Assessed
The following outcomes for undergraduate learning were approved by the IUPUI Faculty Council in March of 1998, after considerable discussion, and a winnowing down from a set of first eight and then nine principles originally developed during the 1992 discussions. The six faculty-approved Principles of Undergraduate Learning at IUPUI are:
- Core Communication and Quantitative Skills, which involve the ability of students to write, read, speak and listen, perform quantitative analysis, and use information resources and technology.
- Critical Thinking, which involves the ability of students to analyze carefully and logically information and ideas from multiple perspectives.
- Integration and Application of Knowledge, which involves the ability of students to use information and concepts from studies in multiple disciplines in their intellectual, professional, and community lives.
- Intellectual Depth, Breadth, and Adaptiveness, which involves the ability of students to examine and organized discipline-specific ways of knowing and apply them to specific issues and problems.
- Understanding Society and Culture, which involves the ability of students to recognize their own cultural traditions and to understand and appreciate the diversity of the human experience, both within the United States and internationally.
- Values and Ethics, which involves the ability of students to make judgments with respect to individual conduct, citizenship, and aesthetics.
The Office of Planning and Institutional Improvement (PAII), through its establishment of the Council for Undergraduate Learning and the Commission on General Education, played a significant role in the development and approval process of the PULs, which involved several hundred faculty from all IUPUI academic and professional programs. That same office continued its stewardship of the PULs through another committee, the Program Review and Assessment Committee (PRAC), comprised of two faculty members, one of them generally serving also in some administrative capacity, from each of the twenty-two schools. While the 1998 approval of the PULs resulted in their acceptance both as a significant part of the undergraduate curriculum and as our approach to general education, the approval included no specified mechanism for assessing student growth or achievement in the PULs. Integrating the PULs into the curriculum was implicitly accepted as the responsibility of all faculty, but explicitly stated as the responsibility of no specific faculty. Informal surveys of graduating seniors carried out in some capstone classes between 1998 and 2000 indicated that most students had not encountered the PULs explicitly in any of their courses. Annual assessment reports, submitted to PAII and reported on to PRAC, indicated only sketchy and sporadic integration of the PULs into the curriculum, although professional schools whose accrediting agencies required similar kinds of skills and knowledge were able to integrate the PULs into their curriculum much more readily.
The year 1998 saw not only the approval of the PULs, but also the beginning of the IUPUI Institutional Electronic Portfolio (I-Port), developed as one of the first generation of institutional portfolios as part of the Pew-funded, AAHE–sponsored Urban Universities Portfolio Project. Our portfolio focused on evaluating the processes and evidence related to achieving our mission of providing to our constituents excellence in teaching and learning, research and creative activity, and civic engagement. One of the key challenges of developing the portfolio was how to demonstrate student learning using authentic evidence of learning, not just aggregates of grades and surveys. With so many different academic and professional programs, the obvious point of entry seemed to be the PULs. Additionally, with the impending 2002 NCA accreditation visit, we knew we would need to demonstrate the ways in which the PULs contributed to and interacted with learning in our academic majors, not only as the key component of our general education program but also as principles that permeate the undergraduate learning experience. In other words, if IUPUI defines general education with a set of principles that are intended to permeate the undergraduate learning experience, we need to be able to demonstrate what students know and are able to do in relation to the PULs at both the “general education” level (within the first 56 credit hours) and at the senior level. We need to be able to show both improvement and achievement, and not just in the PULs alone, but as they are integrated into the major and professional programs in ways that enhance student learning and add value to the undergraduate experience.
Concurrent with the approval of the PULs and the development of I-Port was the establishment of the Committee on Liberal Arts and Sciences, charged with exploring the possibility of a common core curriculum. Labelled “The Principled Curriculum,” this common core, which took almost three years to develop and become approved, is based upon the PULs, and has played a key role in bringing the PULs to the attention of those faculty who had not been directly involved in the process of their development and approval. One consequence of the Principled Curriculum was a requirement for every course syllabus to include the PULs. Unfortunately, in higher education, such a mandate results more often in perfunctory compliance than in enthusiastic intellectual engagement with the reason behind the mandate, and many course syllabi simply listed the PULs, with no mention of how they were integrated into the course and no further mention during the course. Inclusion of the PULs in course syllabi without further explicit integration in a number of disciplinary areas reinforced the need to determine the extent to which the PULs were playing a significant role in the undergraduate learning experiences of our students.
In 2000, the Dean of the Faculties provided funding for three faculty associates, headed by the Director of Campus Writing, to ascertain the extent to which the PULs were being explicitly integrated into the curriculum. This group met with the academic deans of every academic and professional school, pored through syllabi, and conferred with faculty. The results were, to put the best possible face on it, spotty. Some schools paid no explicit attention at all to the PULs and had no direct evidence to determine whether their students were improving in writing, critical thinking, or understanding of society and culture. Many faculty articulated an assumption that students improved inherently in these areas as a result of their courses in the school, and that passing grades in courses such as freshman writing and oral communication indicated corresponding competence and understanding transferable to other disciplinary areas. Others, particularly the professional schools, had explicitly integrated the PULs into the coursework, and could present a corpus of evidence demonstrating the growth and achievement of their students in these areas. After presenting our findings to PRAC, in a document entitled Phase 1 of a Study of Student Learning at IUPUI: A working document for the campus, we decided to present the information on the institutional portfolio in three ways:
a. First we uploaded the narrative discussion of the study, providing an overview of how the PULs were taught, learned, and assessed in each of the schools;
b. Second, we summarized the narrative in the form of a matrix that showed, for each school and PUL, how the PUL was taught and assessed, how it was integrated into student work and learning outcomes, what the school learned from its assessment of the PUL, and how that assessment influenced curricular and pedagogical decisions [ADD EXAMPLE].
c. Third, we took advantage of the electronic feature of the portfolio to develop an interactive matrix, whereby the visitor to the portfolio could identify which school(s) and which PUL(s) he or she wanted information about. For example, if someone keyed in Liberal Arts and Critical Thinking, they would see how critical thinking was taught and assessed, what forms it took in Liberal Arts, and how Liberal Arts had used its assessment to improve teaching and learning.
This interactive matrix quickly became one of the more demonstrated and visited areas of our institutional portfolio. Whether at national conferences or in campus-level committees, people wanted to see what was occurring in relation to these PULs in certain schools, and any gaps or blanks were blatantly apparent. In particular, the last two columns – What have you learned from your assessment of the PULs? and How have these discoveries influenced curriculum and pedagogy? -- were either left blank or barely begun by several schools. The interactive matrix became a powerful catalyst for schools to make the integration of the PULs into their academic work much more explicit, in order to be able to complete their part of the matrix. The most important lesson we learned from this endeavor was that the PULs were not being explicitly or intentionally taught or assessed in any consistent manner across the campus. While it may have been true that students were improving in their ability to think critically or communicate more effectively, we had insufficent evidence to support that claim, or to demonstrate that the PULs were indeed providing a coherent pathway through the undergraduate experience at IUPUI.
I write “insufficient,” because we do actually have an accumulating corpus of indirect evidence, through self-reporting on NSSE, on our own first-year and graduating senior surveys, and on reflective writing done concerning the PULs by graduating seniors in the School of Liberal Arts and the School of Science. But we needed to develop a system that would provide direct and authentic evidence of improvement and achievement in learning of these PULs in relation to learning in the major. We needed this system not only for accreditation purposes, in order to demonstrate that our general education program provides an effective foundation for learning, but also for our own purposes, to demonstrate to ourselves and to our constituents that the PULs provide a coherent curricular basis for undergraduate learning at IUPUI, and actually enhance learning in the major. Out of this need grew the impetus for the IUPUI electronic student portfolio (ePort).
Overview of Assessment Process and Method
The Assessment Method
While the mature ePort will have many components, including a resume-building function and a knowledge-mapping function, we are beginning with two features: a learner profile and a learning matrix. The learning matrix is at the heart of the assessment function of ePort, and will, in essence, capture the entire undergraduate learning experience on one screen that will look something like this:
Learning Matrix of the IUPUI Student Electronic Portfolio
PUL Pre-Survey:
Principle of UndergraduateLearning |
Introductory |
Intermediate |
Advanced |
Experiential |
|
1a Core
Skills: Written Communication |
Add/Edit Help Reflection |
Add/Edit
Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
1b Core Skills: Analyzing Texts |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
1c Core Skills: Oral Communication |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit
Help Reflection |
Add/Edit Help Reflection |
|
1d Core Skills: Quantitative Problem Solving |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
1e Core Skills: Information Literacy |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
2. Critical Thinking |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
3. Integration and Application of Knowledge |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
4. Intellectual Depth, Breadth, and Adaptiveness |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
5. Understanding Society and Culture |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
|
6. Values and Ethics |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Add/Edit Help Reflection |
Intermediate PUL Survey Senior PUL Survey
In the matrix, Introductory Level captures evidence of student learning based on campus consensus of what all students should know and be able to do, regardless of major or professional program, after 26 credit hours at IUPUI. Intermediate Level captures evidence after 56 credit hours. The Senior Level captures evidence of student learning of the PULs as they have been integrated into the academic major or professional program. The Experiential Level cuts through all three of the previous levels, and involves evidence of student learning of the PULs in co-curricular and extra-curricular experiences. To date, we are focusing on the Introductory and Intermediate levels.
The development of campus consensus about what all students should know and be able to do in relation to each of the PULs, regardless of academic major or professional program, was a year-long process involving more than a hundred faculty working on multi-disciplinary committees. It is currently in its final phases, as faculty work to develop “scenarios” for the portfolio. Here is one, for example, that was just drafted for PUL 6: Values and Ethics:
Scenario for students uploading
documents directly into matrix
1.
Student enters PUL Matrix
2.
Student clicks on PUL 6: Values and Ethics:
3.
Student sees:
a.
Values and Ethics
b.
Aesthetics
4.
Student clicks on a. Values and Ethics
5.
Student sees:
a.
Students are able to make judgments with respect to
individual conduct and citizenship.
b.
Introductory - Students
demonstrate an understanding of how ones values influence personal ethics
and conduct.
c.
Intermediate - Students demonstrate an
understanding of and respect for the values of others in contrast to their own
and are aware of how decisions and conclusions may vary based on different
perspectives.
Values provide the
foundation for making personal and ethical decisions related to what a person
considers to be good or bad, right or wrong.
They stem from multiple sources as someone matures (e. g. family, local
community, religious affiliations, professional organizations, educational
institutions, national forums, etc.).
Values are developed through making choices, experiencing challenges,
and taking action in accordance with personal beliefs. They are reflected in the judgments people
make and the solutions they choose.
Values are categorized as moral when they pertain to the human
interactions and non-moral when related to inanimate objects. A gun is non-moral but if used by a human,
the act may be judged as moral or immoral in regards to the act itself or the
motivation for the conduct.
7. If students click on ethics,
students see:
Ethics refers to
the study of moral values held by individuals or groups. A person or an act is considered ethical or
moral when it is judged consistent with the values of the group or society
(morality). Ethics also establishes the
degree of rightness or wrongness for moral conduct or decisions.
8. If students click on Introductory, students
see:
To demonstrate your understanding of this PUL at the
Introductory level, the documents you upload and your reflection should show the
following:
a)
you can explain the relationship between personal
values and the choices a person makes.
b)
you can articulate the values that are important to
you in making personal choices about conduct and citizenship.
(May include a situation in which
your values influenced your personal ethics and resulting conduct.)
The documents you choose to illustrate competence at the
introductory level may be from 100 – 200
level courses or from documents you have written related to out of class
experiences (campus organizations, religious affiliation, employment,
etc.).
Not every paper needs to show every element, but every
element should be represented in the totality of documents you upload in this
section or else accounted for in your reflection.
Do you wish to upload a document
now? Y N
9. If student click Y, student sees a browse/upload
screen
10. IF students click on Intermediate, they see:
To demonstrate your understanding of the PUL at the
Intermediate level, the documents you upload and your reflection should show
the following:
a.
you can analyze situations and foresee how decisions
or conclusions may vary when values, within yourself or between individuals, are conflicting.
The documents you choose to illustrate competence at the
intermediate level may be from 200+
level courses or from documents you have written related to out of class
experiences (campus organizations, religious affiliation, employment,
etc.).
Not every paper needs to show every element, but every
element should be represented in the totality of documents you upload in this
section or else accounted for in your reflection.
Do you wish to upload a document
now? Y N
11. If student click Y, student sees a browse/upload
screen.
For PUL 6b
12.
Student enters PUL Matrix
13.
Student clicks on PUL 6:
14.
Student sees:
a, Values
and Ethics
b. Aesthetics
15.
Student clicks on PUL 6 (b) Aesthetics
16.
Student sees:
c.
Intermediate - Students explain the role of
aesthetics in society.
Choose
documents that demonstrate your aesthetic awareness.
17.
If student clicks on aesthetics, student sees:
Aesthetics is the study of
cognition and emotions in relation to beauty and meaning in life. Meaning is attained through
intense perceptual, intellectual, and emotional experiences that provide
personal insight into the human condition. These
aesthetic experiences influence ones personal values and promote active
inquiry and reflection.
18.
If student clicks on Introductory: student sees:
To
demonstrate your understanding of this PUL at the Introductory level, the
documents you upload and your reflection should show the following:
a)
you can explain how aesthetics influences decisions
you make in your life.
b)
you can evaluate an aesthetic experience and how this
strengthened or changed your valuing or understanding of the human condition or
culture.
The documents you choose to illustrate competence at the
introductory level may be from 100 - 200
level courses. OR From documents you have written related to
out of class experiences.
Not every paper needs to show every element, but every
element should be represented in the totality of documents you upload in this
section or else accounted for in your reflection.
Do you wish to upload a document
now? Y N
19. If student click Y, student sees a browse/upload
screen
.
20. If student clicks on Intermediate: student sees
To demonstrate your
understanding of this PUL at the Intermediate level, the documents you upload
and your reflection should show the
following:
a. you can explain the unique contributions
of aesthetic experience to human life and culture.
b. you can analyze how your aesthetic
awareness has been broadened through your studies at IUPUI and the effect of
this awareness on your personal development.
The documents you choose to illustrate competence at the
intermediate level may be from 200+
level courses OR from documents you have written related to out of class
experiences.
Not every paper needs to show every element, but every
element should be represented in the totality of documents you upload in this
section or else accounted for in your reflection.
Do you wish to upload a document
now? Y N
21. If student clicks Y, student sees a browse/upload
screen
Students complete the matrix by submitting assignments from their academic and professional classes. These assignments will already have been graded by the professor in each class for content knowledge, and, in some cases but not all, for the particular PUL in the matrix where the student has chosen to upload it. As is evident in the above scenario, students will need to upload several (3-5) documents, generally from different courses or classes, to complete a PUL cell in the matrix at any given level. When students determine that a cell is complete, they click on “Reflection,” and will be prompted to write a reflective essay making the case that the uploaded documents do indeed demonstrate the level of competence as specified.
This reflective piece is what takes the portfolio beyond the function of providing authentic evidence of student learning, important as that is in itself. The reflection is intended to catalyze deeper learning, to capture the connections between disparate skills and information from diverse courses, and to move students into an awareness of meaning that transcends discipline-specific knowledge. Simply writing a reflection, however, does not guarantee profound or insightful metacognitive thinking or enhanced understanding [REFERENCE ARTICLE]. Through workshops offered by national experts, such as Marcia Baxter Magolda, in the development of intellectual understanding, faculty committees are being guided to write prompts for reflective writing for each of the PULs at the Introductory and Intermediate levels. These prompts will be accessible to students through the portfolio infrastructure, in the form of a ‘prompt wizard’. The intention is that the “just-in-time” prompts will lead students to the kind of intellectual probing required for connective understanding beneath and beyond course content and will provide the catalyst for deeper, more meaningful learning.
These reflections will then be electronically sent to and read by members of the Senior Academy (retired faculty wanting to remain intellectually connected to IUPUI) and by alumni volunteers who will be trained to read and evaluate the reflections in relation to the campus expectations for learning for each PUL. By drawing upon the expertise of our retired faculty and our alumni, we intend ePort to contribute significantly to a community of learning that extends beyond the immediate campus. These alumni and members of the Senior Academy will send a written response to the students and a 1, 2, or 3 designation, 1 being that the work exceeds expectations; 2 being that it meets expectations; and 3 being that it does not meet expectations. The written response, which can be chosen from a selection of pre-written responses or can be composed by the trained volunteer, will comment on the strengths and weaknesses of the reflection, and the appropriateness of the documents selected by the student to demonstrate the specified PUL, and will include suggestions for writing more effective reflections for the next cell they complete. The 1s, 2s, and 3s will be automatically aggregated according to a wide range of demographic information, so that deans, chairs, and campus administrators will not only have a richer picture of how the PULs are influencing student learning on the campus, but will also be able to pinpoint areas where more attention – and possibly more resources – might be needed to generate improvement. This information will be reinforced by a PUL survey that students will complete during their first month at IUPUI, after they complete 56 credit hours, and during their senior capstone. The survey contains three questions for each PUL, two quantitative in nature, and one qualitative. Example:
|
1. How important do you think written communication will be in your education at IUPUI? Not at all Slighty Moderately Important Very important important important important □ □ □ □ □ How would you rate your
knowledge level or competence in relation to written communication? Very low
Somewhat low
Moderate
Somewhat high Very high □ □ □ □ □ What does written communication mean to you and what role will it play in your education?
Students enter their
open-ended response to this question in an expanding word box. Please limit your
response to 300 words. |
