Introduction

One of the most attractive features of information technology is its adaptability. For example, Flashlight Online was originally designed to support “evaluation procedures that a postsecondary institution or department could use to periodically assess its evolving educational strategies”. There was nothing in Flashlight Online’s design, however, that prevented it from be adapted to other uses. This report summarizes such a use: managing a complex, self-paced course in order to accelerate student learning.

The Course

This introductory programming course is required for undergraduate MIS majors at the University of South Florida. One of the principal challenges of the course is its dual purpose:

        to provide all MIS majors with sufficient grounding in representation and algorithmic thinking so that they can perform adequately in subsequent technical non-programming courses (e.g., databases, data communications) and

        to provide future professional programmers  with a strong foundation.

While the latter group is not large (perhaps 15-20% of all majors), it contains many of the major’s best students—individuals who are also substantially more motivated than those students who have already explicitly rejected programming as a potential career. Further complicating this diversity of interests is the fact that about half the students have had one or more previous programming courses while the others have had no prior exposure to programming whatsoever.

Over the years, to accommodate this diversity of goals and backgrounds, faculty gradually developed a self-paced design. Rather than have all students attempt to keep up with a pace intended to support the needs of “programming career track” majors, students instead accumulate points by completing assignments (validated by exams taken after each assignment is handed in) and their grade is based entirely on the total number of points accumulated.

To enable the self-paced design, technology has been used extensively. Specifically:

• all course materials except the textbook are made available in electronic form

• all course lectures were made available 24/7 online, as multimedia Flash and streaming Windows media (wmv) files

• online discussion boards are established for each assignment

• online practice exams are available for many of the assignments

• a course web site clearly links available resource materials to specific assignments

Once online versions of lectures were made available, attendance at live lectures dropped precipitously. Two reasons for this were cited by students.

1. Online lectures were more convenient and allowed complex explanations to be replayed.

2. Students not keeping up with the “career track” pace quickly found that weekly lectures were “out of synch” with the assignments they were working on.

The result: in a typical semester, the last month of lecture was attended by 1 or 2 students (out of a section of 35). As this became the norm, lectures were eliminated entirely, with course resources (i.e., instructor and teaching assistant time) being redirected to increased support of online activities.

The Challenge

One unintended side effect of the elimination of live lectures was a growing disconnect between instructional staff and students. Even though TAs offered a total of over 30 office hours a week, students rarely showed up, except to take exams. The instructor saw even fewer students. This created a serious obstacle to the assessment of course status. One summer, for example, with 10 days left in the 10 week semester, only about 6% of the class had yet achieved the minimum passing grade. Furthermore, despite repeated requests for status, students simply would not tell the instructional staff what problems they were encountering. As it turned out, things were not as bleak as they appeared to the staff—as a result of a last minute push on the part of the students, the course pass rate actually exceeded that of previous summer. The outcome, however, as not entirely satisfactory either, with many students receiving C grades who could have done much better had they managed their time more effectively.

After the disappointing results from the summer semester, the instructor decided to institute policies intended to provide the staff with better ongoing feedback with respect to course status. The use of Flashlight Online was instrumental towards accomplishing this end.

Using Flashlight Online to Track Student Status

The key to applying Flashlight Online to enhance the self-paced programming course was realizing that—although designed as an assessment tool—it also possessed the capabilities needed to act as a workflow management tool. In particular, Flashlight-designed web forms could be used:

• To collect background information about students

• As a means whereby students could submit progress reports

• As a means whereby TAs could submit reports on student status

During its first semester of use, three different forms were developed for use in tracking student progress.

Figure1: Check in Form

Check-In Form

The first form developed for the course was a check in form (Figure 1) that students completed during the first week of the class. This form provided a means of consolidating variety of information needed to manage the course—the student’s name, preferred email address (often different from the address provided by the university), what TA was assigned to monitor the student’s progress, the ID they were intending to use for their web log (students were provided extra credit for making entries to a LiveJournal blog that each of them had created), how they planned to meet with their TA, and their target grade. The last of these was critical in assessing student progress, as a realistic pace of assignment completion for a ‘C’ was very different from that required for an ‘A’.

From time to time, students changed information (e.g., target grade, contact information) over the course of the semester. These students simply submitted the check in form an additional time. The instructor kept track of the number of forms submitted (using instant analysis feature of Flashlight Online) and downloaded the data whenever new submissions were made.

Weekly Forms

Two similar forms were used for weekly progress reports, one for use by TAs, one for use by students (student version of the form is shown in Figure 2). At least one of these two forms a generated each week for a given student. If the student met, in person, with his or her TA, the TA submitted a form. If, however, the student did not meet with the TA, the student could submit a form directly. Students could also submit forms at any time to report a problem.

To help motivate submission, 100 points (out of 1000 available course points) were allocated to class participation. Each weekly meeting with a TA or student submission of a weekly report was awarded 5 points (80 points over the whole semester) with remaining points available for Blackboard participation, LiveJournal postings and completing course check in.

Figure 2: Student Weekly Progress Report

Weekly Feedback

The Flashlight Online forms were key inputs to a feedback process that continued throughout the semester. As shown in Figure 3 below, each week, the instructor downloaded the TA and student weekly reports from Flashlight Online (along with updated check in forms, if the count changed during the week). In addition, current grades were downloaded from Blackboard.

These forms were then fed into a simple program developed by the instructor called IsmClient. This program used the data sources (all of which were in text format) and combined them to create reports (in HTML format) for each student detailing their cumulative progress in the course. In addition, consolidated reports containing information for all of the students were sent out to teaching assistants—who could use them to assess the progress of their assigned students (and to identify students who were not checking in or reporting their progress). This sample weekly report illustrates the feedback; it has been disguised to preserve student privacy.

Figure 3: Weekly Reporting Process

In principle, there is nothing that appears in the weekly reports sent out to students that they should not already know. We believe, however, that the minor task (~15 minutes) of preparing them each week is justified by: a) the fact that students can use it to verify their grades and participation, and b) the clear message they send to students that the staff is paying careful attention to their progress.

Results

The initial results of the using the Flashlight-driven workflow system were almost immediately visible. As noted at the outset of the paper, 9 weeks into a 10 week summer semester, only 6% of the class had met the requirements for a grade of ‘C’.  The subsequent semester—with precisely the same workload in terms of number and size of assignments—roughly 40% of the class had met the requirements for a ‘C’ 9 weeks into a 15 week semester. Furthermore, by the 9 week point, the percentage number of students who had accumulated enough points for an ‘A’ or a ‘B’ already exceeded the percentage that were awarded those grades during the summer.

Discussion and Future Directions

The use of Flashlight Online as a means of collecting information to manage a self-paced course has proved to be very convenient. While the development of the IsmClient program has also been instrumental in tying the information from various sources together, many of the same capabilities (excluding sending weekly emails with web attachments) could have been achieved using a database program such as MS Access. Indeed, the IsmClient program included the capability of writing its data to a format (xml) that Access could import—allowing the excellent report writing facility provided by Access to be used in generating custom reports.

Currently, we are preparing a proposal to create a more user-friendly interface for the IsmClient program, making it accessible to faculty having no programming background. It is also our intention to increase the number of different data sources the program can import. This could allow it, for example, to bring in data from online discussion groups, existing data bases or even static web pages. At the present time, however, we continue to use the Flashlight Online-IsmClient combination every week—and will continue to do so as long as evidence that it is effective in helping students better manage their time continues to accumulate.