• Study strategies for using technology to make large courses more participatory, active, and engaging for diverse learners. When people think about transforming higher education, they are often rejecting an image of large lecture halls half-full of dosing students (and the other half not showing up at all). The National Center for Academic Transformation has made a good start in supporting redesign of such courses. It's time for a good study to create a taxonomy of such redesign strategies. What are the strengths of each? How do they relate to each other? What problems often occur when each such strategy is implemented?
• Develop TLT case studies to help faculty learn to handle the rough issues that technology can raise.  When academic staff begin using technology to improve teaching and learning in their courses, they can easily experience frustration or even serious difficulty. Some of these problems are solvable (e.g., bring a spare copy of your slides, in case your first version is unexpectedly unavailable); know the help number to dial. But many such problems are not so easy to solve, especially when they arise from the teaching/learning process itself. What can go wrong when courses shift the balance away from lecture and recitation, plug-and-chug, toward active learning, collaborative learning, and creative work? 
  When using a TLT case study, faculty in a discussion group begin by reading a  "trigger case" which tells the story of a faculty member and how he or she came to face a common problem that can arise when instructors begin using technology to a. First individually and then as a group, the participating staff analyze the problem and discuss different ways to address it.
  Developing such TLT case studies is a) pragmatically useful for faculty development, and b) a distinctive approach to the scholarship of teaching and learning. (See this TLT case study, for example).
  Take a look at these first, brief TLT case studies. Try them out. We're sure you can do better, and we'll add  your cases to this Collection!
• Are common uses of word processing helping colleges educate graduates who are more adept at developing and critiquing complex arguments?  The answer might be "yes" because (we hypothesize) rethinking one's work while rewriting can help students master a process of developing a complex, internally consistent argument. But, so far as we know, no one has done the study even though the data is out there, waiting to be analyzed. Click here to see more. 
• Digital writing raises many interesting questions for study. For example, it's been claimed that, when students know that they can continue to develop a piece of writing after it's been submitted for a grade, a) at least some students will continue to do so and will benefit from that continuation, b) at least some students (more than the first group) will see the initial assignment as "real" rather than "schoolwork" and will therefore be more motivated to put thought and energy into the writing. This conjecture is closely linked to a conjecture about audience: that students will put more energy into writing when they know that an audience that means something to them will be seeing and reacting to the work. True?
• How much "collaborative learning" is there in a typical course? How much "active learning?" How much faculty-student contact?  These abstract 'umbrella' terms cover a variety of student and faculty activities; they are also some of the more common ways in which faculty use technology to improve teaching and learning.  The Flashlight Current Student Inventory, Faculty Inventory, and Flashlight Online provide survey tools for estimating how much of this kind of activity goes on, and how helpful technology is for increasing or enriching it.  But these tools can be improved and augmented by other ways of observing teaching-learning activity.  Also lacking (so far) is any kind of norm to which individual courses, departments or institutions might be compared. These are big projects and perhaps worthy of a grant proposal.  You could tackle it on your own or partner with us...
• Diagnostic tools for faculty.  This is a priority for Flashlight but you might take on part of it, alone or with us. Here's an example: items for gathering student feedback to faculty use of PowerPoint (working with Mount Royal College, we've already developed that)   
  What's next? The example about which we have talked the most is online collaboration among students. We've been talking with faculty for some time about barriers that can hinder at least a few students; the list is now over 60 barriers. In other words, in a class of 30 students, some may find no problems collaborating online, but others may face one or more of a rather large set of problems. That's the bad news. The other part of the bad news is that, without help, the faculty member can only see that participation is disappointingly low, but not why.   The good news is that, if the right person knew what problem each student was facing, almost all the problems are pretty easy to fix. Diagnostic tools: a survey and test for the institution to administer regularly, plus another survey or two for the faculty member to administer a week or two into the course.  These kinds of data might result in much better collaboration. That's the kind of tool we're talking about.  
• Study how faculty learn to use technology in their teaching. For example, imagine a set of faculty who know relatively little about facilitating collaboration among students (as in a seminar or in team work on problems) and relatively little about using a system such as WebCT or Blackboard for that purpose.  What are some of the more efficient ways to help them learn? How does the availability of "learning objects" (e.g., from Merlot) affect faculty progress? For more on the nature of this problem, see the June 2002 issue of F-LIGHT.
• Study competences faculty need to teach well online. The idea, in brief, is to identify two groups of faculty - one set of faculty that (by some measure) teaches superlatively and a set of other faculty who do not teach superlatively.  Compare how they each think about what they do? Are all the superlative teachers alike? how about the controls? If there are differences in how they think, do those differences lie in trainable skills? in deeply held values and assumptions?  For more on how such a study might be designed, click here.
• Develop a measure of faculty approaches to teaching. The Schneider, Klemp, and Kastendiek (SKK) study described in the link above suggests there may be a dimension of approaches to teaching, characterized at one extreme by faculty (widely seen as excellent teachers) who believe all their students can become engaged, that students are different from one another, and that it's important to use continual trial and error to see what it takes to get each student engaged. At the other end of the scale are faculty (widely seen as normal teachers) who believe that only some students will become engaged, and that their job is to teach (explain) while it is the students' task to learn (study). "My job is to put it out there, their job is to learn it," a member of this normal group might say.
  The SKK model was derived from a small sample of faculty in a specialized type of program in 1980-81.  The research challenge is to create a way of measuring faculty approaches to teaching (survey? observation?) to see whether a) the model does cleanly describe today how different faculty approach teaching, and how many faculty fall into each category (if there are indeed just two categories). (See also the item immediately below on this page).                
• Is there a relationship between how faculty use technology and what they believe about "assessment"?  If the "Balancing Act" measure described above can be developed, I predict this measure will be useful for predicting faculty uses of technology and faculty attitudes toward assessment.  (Click here to read more.)        
• Do faculty approaches to teaching (as described above) change from 'normal' to 'excellent'? Are there techniques that universities can use to help more faculty become 'excellent'?  The SKK study above suggested that faculty widely regarded as 'excellent' believe that all students can become engaged, and try to help each student attain that, while 'normal' faculty believe that many students will never become engaged and do not focus on helping students as individuals. Over the years, how often do faculty shift from one belief set to the other? Do any university practices, or collegial practices, make it more likely that faculty will over time shift from a 'normal' approach to an 'excellent' approach?
• (NEW!)  Under what circumstances have institutions teamed up to create and offer stable, successful, shared courses of study?  I'm talking about two or more institutions that each contribute some of the teaching in order to offer a set of courses for which any of their students can register. For example, five institutions collaborate to offer the OneMBA Executive Degree Program in Global Management. Tiny physics departments at a number of public institutions in Texas collaborate to offer a physics major open to all their students. The advantages of such programs are compelling, but the barriers are daunting.  Under what circumstances do such programs thrive? What does that suggest about future policies, technology platforms, and practices?  Click here for a longer description of this topic. 
• Is your institution improving? evolving?  Some people suggest that the basic character of higher education is being transformed (e.g., "Access and/or Quality" essay; "vision and mission" of National Learning Infrastructure Initiative).  How might such transformational change be measured? This table summarizes a variety of research and evaluative questions - the top row contains questions raised by improvement, the second row by transformation, and the third row by efforts to guide or control that transformation.
• Validating survey and interview estimates of activities.  In order to estimate the educational value of technology investments, we have to ask people what they've done with technology (e.g., how much time did faculty spend using e-mail to discuss assignments with students). We have to ask similar questions to estimate changes in educational costs resulting from technology investments (e.g., how much time did faculty spend using e-mail to discuss assignments with students). But are the responses to such survey or interview questions accurate enough to make policy? Are there better ways to frame the survey or interview questions? Are there other kinds of inquiry that could complement or replace the survey or interview? For a bit more on this topic, see this article in F-LIGHT.
• Estimating the costs of current and alternative ways of supporting faculty, staff and student use of technology. Most institutions face some kind of support service crisis: their budgets for support staff fall far short of need.  Relatively little research has been done conceptualizing different methods of support and then estimating the monetary and human costs (and rewards) of each. Costs are so local in nature that it is probably more appropriate to do these studies within institution in order to guide local policies and practices. In the long term, however, new approaches to providing support on a large scale may emerge. For a bit more on this topic, see this article in F-LIGHT.
• What kinds of virtual programs have the best chance in the future of gaining financial support from their alumni? This essay suggests that the building of current relationships around academic work (and its affective side) is key, and suggests a study that attends to teaching/learning practices and the details of the virtual workplace itself.
• Research on assessment's role in the use of technology to help transform academic programs.  This essay sketches ways in which assessment (feedback) could be used to help guide and accelerate institutional improvements and outcomes, especially where technology is used to provide leverage. Part II of the paper describes some research questions whose answers could help institutions around the world make better use of such assessment.
• What are some of the design issues and policy questions raised by infrastructure for integrated access?  "Virtual universities" that don't teach courses but do offer courses taught by others, along with advising.  Programs that bank credits, or offer credit by examination. Companies that proctor exams offered by online degree programs. These are just a few examples of infrastructure that mediates between online instructors/institutions and online learners. What are we learning about designs that do, and don't work?  What policy questions are raised about subsidies? regulation?  Click here to read more background information.

Return to Flashlight Evaluation Handbook Table of Contents