Ehrmann notes on Cohen Case

This was my second step in designing a study: moving from my notes on the study itself to paraphrasing and arraying them as triads. I’ve listed the students reading the lectures first because that seems first in sequence. Prof. Cohen’s major interest lies in the use of the Bulletin Board System (BBS) within WebCT; that material appears second, followed by some other triads that might be of importance. I’ve not translated all of my notes to this form, just those that seemed likely to be helpful in designing a study. For example, although the gradebook is online (and although some other faculty have used online grade books as an instructional tool), Prof. Cohen’s essay does not discuss any activities and outcomes that seemed important to him, so it does not appear in this chart.

Triads
Technology or other important resources in use	Activity- something educationally important that is done with the technology and other resources (often, in order to achieve the outcome)	Likely outcome or objective of the action (may also be a goal or a negative outcome)
Written lectures posted on the system each week	Students read the lectures	Students have reasonable understanding of what they’ve read in the lectures
Assignment posted each week Bulletin board space	o * Students first had to learn to use the BBS o Students respond to one assignment question a week (Students answer challenging questions based on the reading ) (250 words) o *Students see assignments as challenging (and worthwhile) o *Student pose one question a week (ideas and problems they develop themselves) o Student answer one student question a week o *Students read ideas of all other students, see their thinking processes (“visible knowledge”) o *Students interact with one another in solving historical questions, as in a traditional seminar; o *Students take more time to think than they would in classroom o *Students can speak w/o worrying about interrupting others o *“Weaker” students more likely to see and imitate what “stronger” students do o *Stronger students help weaker students o *Student meet deadlines	Immediate objectives o Students’ comments more reasoned o *More students participated in the conversation (by end of course than would in a f2f course this large) o *Students learned to ask questions they develop themselves o *Student don’t hate the online format (Even so, Cohen predicts, students are likely to experience the course as difficult) o *“Weak” students got help (more than in traditional classroom) Outcomes re historical understanding? Other?
	Instructor answers questions of fact Instructor manages interaction
class email
Optional Wednesday classroom meeting	Students who didn’t understand online material or who had questions could come and engage in historical discussion with faculty and other students about anything that interested them;	Students feel less isolated than if course purely online Actual outcome: lack of understanding of the nature of the Q&A
		Overall outcome: attrition v. retention

Fine tuning the triad

Because I’m doing this as an exercise, there has not been a conversation with Prof. Cohen, who is a continent away. If I on a college staff and working with a faculty member, we’d have used the triad above as the object of discussion, and would almost certainly have altered it to better reflect the faculty member’s most important concerns about the course.

Creating a Theory-testing survey

As I begin thinking about potential studies, I consider a number of possibilities for this one (again, in the real world, I would be talking with Prof. Cohen about this: about his concerns, about other users of the data and what they might need to know, etc.) I’m considering several possible issues and designs::

• Do I suggest a study that looks at all (or most of) Cohen’s triad or focus on some smaller elements of it? In trying to answer that question, I consider the importance of what might be learned and balance it against the costs (not just for me and the faculty member but for the students whose time we’re using).
• I’d focus on at least some ‘early’ elements in the cause/effect chain, some of which are implicit in Cohen’s paper and in my table, such as “(All) students have sufficiently convenient computer and Internet Service Providers (is that why some dropped out almost immediately, because they didn’t?” “(All) Students do the reading” “(all) Students do post questions” and “once they learn to use Cohen’s system of readings and posts, it’s easy enough to do.” It’s important to check things like this because, if some students are blocked at this early point, they’re unlikely to benefit in the way that the professor hopes; for example, if they don’t converse online, they aren’t going to benefit from the conversation.
• There are other factors that would influence my choice about which factors to study. Aaron Cohen is part of the Visible Knowledge Project, which suggests that whether this design functioned effectively to give students insights into one another’s thinking, and whether that process helped them, might be of special importance.
• Some things I’d suggest dropping from the study design because the payoff from the study is not likely to have justified the difficulty of collecting the data. For example, I might decide to ignore the outcome having to do with students’ skills of reasoning: it seems unlikely to me that this different degrees of WebCT use would produce noticeable differences in student reasoning by the end of the course, and finding or creating a quantitative measure of their reasoning is likely to be difficult. On the other hand, the issue of who can participate
  
  
• Should I focus on:

1. how this one class function in this term – did the triad work for it? The first step for Prof. Cohen in deciding whether to do this again could be to discover whether he and his class had really done it the first time. I’d talk with him about some draft questions (see next section) to get a sense of whether this data would be sufficiently valuable or whether he already has a good enough sense of what happened in the course)

2. how this course (and others like it) are changing over time? Are faculty learning over several terms to use the BBS more effectively for the activity? As the faculty member learns more about the activity and the outcome, is the activity more likely to cause the outcome (as nearly as we can detect)? Are their triads themselves changing (different ways of defining the activity? Different sense of what the important outcomes might be?)

3. why some students followed the triad more closely than others. For example, some students probably posted more than others. Why? Some were less likely to read other students’ posts or respond to them? Why?

Data Collection plans – first drafts (partial) as thinking exercises

how did the triad work this term? (same questions could work for #1 and #2 above)

Extant data – what already exists

The most obvious source of extant data are transcripts from WebCT. It should be possible to chart who said what to whom, when, and about what, in order to describe the online interactions that Prof. Cohen cares about. I can wait and define later just what kind of analysis should be done. I’d need to find out what kind of data gathering and analysis capabilities that Cal State Sacramento’s WebCT has.

I’m not sure if other data will be useful in studying this triad. Unless we decide to study student reasoning, we probably have no need to analyze student papers, for example (and, even if we do look at reasoning, we might assess it only as reflected in their postings).

New data

Survey? Interviews? Focus groups? I think surveys may be sufficient for study.

Sample questions for assessing whether students carried out the activity, and how useful or problematic WebCT and the online environment were for those purposes