Evaluation models can be used to help you define the parameters of an evaluation, what concepts to study, and the processes or methods needed to extract critical data.

Virginia Tech is a member of the Flashlight Network of higher-education institutions dedicated to improving evaluation techniques on local campuses. We recommend the Flashlight "Triad" Model described below to help you focus and carry out technology-oriented evaluations. If the model does not seem appropriate to your needs, several additional evaluation models are listed at the bottom of this page.

Flashlight Triad Model

The Flashlight Model covers five distinct steps:

1. Overview and Confronting the Blob. This step involves brainstorming and "bringing to the table" all of the various elements that feed into and flow from a lesson, a course, a program, a Web module, etc. For example, faculty development and preparation, student prerequisites, budget, assistance with course materials development, student attitudes and satisfaction, etc. To generate a comprehensive list, you may wish to work with a colleague or instructors who teach similar sections of the same course.

2. From Blob to Issue. To move from the confusing, large-scale blob, to a manageable evaluation, consider the purpose for the evaluation. What matters most to you and others? Who is the intended audience for your completed evaluation? Answer the following questions, then select one or two top issues to study and forget about the rest (e.g., student retention and attrition, student motivation, active learning).

  • What are the three most crucial things you fear might happen as a result of using the technology?
  • What are thre three most crucial things you hope will occur?

3. From Issue to Triad. For each issue selected, create several "triads." A triad consists of the type of technology you might employ (T), a specific activity that the technology enables (A), and outcomes expected from that type of activity (O). Some sample triads are provided in the figure below. After generating several triads, select the triads you wish to focus on.

4. From Triad to Data. For the triads you have selected, you can now generate questions to gather data about the triads. These questions can be delivered as surveys, interviews, focus groups, etc. They are independent of method. At least five types of questions may be generated, as depicted in the figure above:

  • Question Type 1: Technology. These questions are related to the technical infrastructure, its capacity, its functioning, or the degree to which students/faculty were appropriately trained to use/manage it.
  • Question Type 2: Interaction of Technology and Activity. These questions are related to the functions and features inherent in various technical systems and their ability to support or enable a specific activity (i.e., "Did X technology enable you to do Y?").
  • Question Type 3: Activity. These questions are related to the nature of student tasks, or what students are asked to do.
  • Question Type 4: Interaction of Activity and Outcomes. These questions are related to the capability for specific technology-based activities to generate predicted outcomes (i.e., "Did activity Z help you to accomplish S?").
  • Question Type 5: Outcomes. These questions are related to what students learned, how satisfied they were with a lesson or course, and what changes they would recommend.

**Note, the question types listed above can be generated on your own, or you can utilize the pre-existing database of 500 question items created for the Flashlight Network. Flashlight items are organized by sub-scales for faculty to investigate triads focused on: active learning, collaborative learning, faculty-student interaction, time management, time on task, high expectations, engagement, technology "addiction", cognitive and creative outcomes, feedback, respect for diversity, and the infusion of real-world problems into the curriculum. The Flashlight Online tool is used to generate online surveys.

5. From Data to Next Steps. Based on the evaluation data, you will make decisions about your unit or course (e.g., modify how you use technology, scrap an instructional strategy, improve support networks of training/scaffolding).

General Evaluation Models

  • Connoisseurship Evaluation (Eisner) involves a connoisseur or expert in a field of study estimating the worth of a new innovation. Obvious biases and threats to validity exist. (See Program Evaluation, Venedam).
  • Goals-Oriented/Objectives-Based (Tyler, 1949) describes whether or not students have met their goals, with the results informing how to handle a new instructional strategy (i.e., revise, adopt, reject). One weakness is the evaluator may overlook unexpected outcomes or benefits of instruction beyond original goals. (See Program Evaluation, Venedam).
  • Goals-Free Evaluation (Scriven) supplements inherent weaknesses in a goals-oriented approach by providing an unbiased perspective of ongoing events. (See Critique of Accreditation).
  • Judicial/Adversary Evaluation focuses on comparing or describing all sides of an innovation, both positive and negative. Analogous to the defense and prosecution of a court room. (See Evaluation Exploration, Crawford).
  • Kirkpatrick's 4-Level Model describes student "reactions" to and "learning" from an innovation, as well as "behavior" changes in real job performance, and other potpourri "results." (See Evaluating Training, Nickols, 1999; and Instructional Systems Evaluation, Clark, 1997).
  • Situated Evaluation describes the characteristics of varying contexts that cause innovations to fail or succeed differently. Proponents of situated evaluation argue that educational innovations are situated within their context of use. (See Situated Evaluation of Project Lincoln, Levin et al., 1998).
  • Stufflebeams' CIPP Model describes the "context" in which an innovation occurs, the "inputs" of the innovation, the formative "processes" occurring, and the summative "products" or outcomes. (See A Design for Evaluation, Nova; and Evaluation, Iowa State).