Evaluation of ELIXR, a FIPSE Funded Project

Transcription

BbK
Broad-based Knowledge, LLC
Evaluating Technology Innovations in
Education
Table of Contents
Executive Summary ............................................................................................................... 2
I
Project Background, Context and Implementation ............................................................ 4
II
The Evaluation Process ................................................................................................... 5
II.A. Evaluation Approach and Constraints ............................................................................................................... 5
II.A.1 Formative Overview ....................................................................................................................................................5
II.A.2 Summative Overview...................................................................................................................................................7
II.B Evaluation Questions ................................................................................................................................................. 7
II.C Data Collection Methods and Instruments ....................................................................................................... 8
II.D Evaluation Studies .................................................................................................................................................... 11
II.D.1 Use Statistics ................................................................................................................................................................ 11
II.D.2 ELIXR Fellows Evaluation Study Overview .................................................................................................... 11
II.D.3 ELIXR User Study Overview .................................................................................................................................. 12
II.D.4 SERC User Study Overview .................................................................................................................................... 13
II.D.5 ELIXR CSTLT Study Overview .............................................................................................................................. 14
III Evaluation Findings, Conclusions and Recommendations .............................................. 14
III.A Intended Outputs ..................................................................................................................................................... 14
III.A.1 Case Stories ................................................................................................................................................................. 14
III.A.2 Faculty Development Workshops ..................................................................................................................... 16
III.A.3 ELIXR Online Teacher Community ................................................................................................................... 17
III.B Intended Outcomes ................................................................................................................................................. 18
III.B.1 Improved Faculty Development Programs .................................................................................................. 19
III.B.2 Enhanced Teaching Knowledge and Practices .......................................................................................... 20
III.B.4 Improved Access and Contributions to Faculty Teaching Communities ........................................ 21
III.B.5 Sustainable Organizational Model .................................................................................................................. 22
Appendix A: ELIXR Logic Model ............................................................................................ 25
Appendix B: ELIXR Data Collection Plan................................................................................ 27
Appendix C: ELIXR Use Statistics .......................................................................................... 29
Appendix D ELIXR Fellows Evaluation Study ......................................................................... 35
Appendix E: ELIXR User Study .............................................................................................. 63
Appendix F: SERC User Study ............................................................................................... 98
Appendix G: ELIXR CSTLT Study .......................................................................................... 116
5935 Orchard Avenue
[email protected]
Richmond, California 94804
Phone: 510-967-5327
Email:
Evaluation of ELIXR, a FIPSE-supported project
Executive Summary
Sponsored by the California State University System, the ELIXR faculty development
project has grown to involve 28 institutions of higher education in the US and four
Canadian institutions. It also collaborated with two repositories of online learning
materials, MERLOT (http://www.merlot.org) and SERC (http://serc.carleton.edu) to
engage faculty members in adopting and using exemplary teaching practices. Faculty
developers, instructional technologists and faculty members from partnering institutions
collaboratively created multimedia online materials describing the use of an exemplary
practice.
The goals as outlined in the original proposal to FIPSE were:
GOAL A: To provide discipline-based examples of exemplary teaching practices.
GOAL B: To leverage faculty identity as members of both institutional and disciplinary
communities of teachers.
GOAL C: To develop new ways for faculty development centers and online discipline
repositories to collaborate on engaging faculty w/ exemplary teaching practices.
GOAL D: To develop, disseminate and sustain social and technology infrastructure for
collaboration amongst faculty development offices and discipline repositories.
The focus of the evaluation activities was on faculty members as learners. Formative
evaluation activities were given priority in this project because much of the project focused
on the development of learning materials (case stories) for this audience. These efforts
included evaluations to improve the case stories themselves in addition to evaluating the
training workshops in which the case stories were used. The ELIXR Fellowship program
was established in order to learn about use of the case stories in situ. ELIXR Fellows used
the cases in a variety of ways (online, ‘homework’ prior to workshops, demonstrations
during workshops, etc.) and evaluated their impact on the participants learning and
application of the innovation in teaching situations. In addition to these efforts, several user
studies were performed involving faculty members, faculty developers and instructional
designers.
The findings from these evaluation studies indicate that the ELIXR model was most
successful in terms of its collaborative model for developing media rich case stories. It
successfully recruited partners who in turn created its growing library of cases. This
collaborative model has resulted in a nascent community of developers. Such communities
are essential for the continued development of case stories (and other video learning
resources) and their use in professional development situations.
The media richness of the cases is extremely attractive to potential users, especially faculty
developers. The cases do require that faculty development users learn how to use them
effectively. This is a potential barrier to their continued use. Analysis of the site’s usage
indicates a growing number of unique visitors and number of visits, indicating that regular,
ongoing dissemination efforts are effective. ELIXR’s sustainability model relies upon
5935 Orchard Avenue
[email protected]
Richmond, California 94804
Phone: 510-967-5327
Email:
sustaining strong partnerships with influential faculty development and teaching
organizations; relationships with POD, TLT and SERC suggest that such partnerships are
possible.
As the project’s funding winds down, it is clear that the main challenge it will face is finding
successful strategies to keep the momentum going around the development and use of the
case stories. The project level plan called for reducing supporting funds for individual
projects as the amount of methods and tools for case story design, implementation, review
and application increased. This worked well at the project level, however, the central
coordination effort did not decline to the extent expected. Although some organizations did
develop this capability, in particular regional faculty development networks, the take-up by
discipline associations and topic interest groups has not developed at the same pace.
Exemplary case stories are available and in use, illustrating the benefits of the ELIXR
approach; the tools, methods and collaborative processes are documented and available for
continued application. However, in terms of organizational capability to stimulate and
coordinate application, the sustainability of the project now relies largely upon the small
but dedicated community of developers and users in regional faculty development
networks. To continue to grow the community beyond these groups, the ELIXR leadership
team and others will need to continue their work with partner organizations such as POD
and TLT Group, encouraging them to take on the role of promoting the materials as faculty
development resources and promoting the methods, tools and processes for creating and
applying new digital case stories. Other organizations such as NMC – the New Media
Consortium will be critical to bringing developers together to encourage continued
development of online faculty development materials. Continued support and collaboration
with MERLOT will ensure an effective ‘passive’ dissemination strategy (i.e., the materials
will be available online through the repository); ‘active’ dissemination, that is the
necessary workshops, educational seminars and marketing to encourage the use of the
innovative materials, will remain a challenge.
3
I Project Background, Context and Implementation
Sponsored by the California State University System, the ELIXR faculty development
project has grown to involve 28 institutions of higher education in the US and four
Canadian institutions. It also collaborated with two repositories of online learning
materials, MERLOT (http://www.merlot.org) and SERC (http://serc.carleton.edu) to
engage faculty members in adopting and using exemplary teaching practices. Faculty
developers, instructional technologists and faculty members from partnering institutions
collaboratively created multimedia online materials describing the use of an exemplary
practice.
The goals as outlined in the original proposal to FIPSE were:
GOAL A: To provide discipline-based examples of exemplary teaching practices.
GOAL B: To leverage faculty identity as members of both institutional and disciplinary
communities of teachers.
GOAL C: To develop new ways for faculty development centers and online discipline
repositories to collaborate on engaging faculty w/ exemplary teaching practices.
GOAL D: To develop, disseminate and sustain social and technology infrastructure for
collaboration amongst faculty development offices and discipline repositories.
Building on the pedagogical technique of case studies, the multimedia learning materials
were designed as case stories combining elements of case studies and digital storytelling.
Multimedia case stories are a relatively new genre within the realm of adult learning. Case
stories are real or realistic stories about teaching. They are not lessons, presentations, or
even reflections (although a good story may contain all or some of these elements)
presented through a digital medium. The key characteristic of a good case story is that it
draws in audience members to imaginatively relate to the story. A good story about
teaching helps other teachers experience what the storyteller has done and feel what the
storyteller has felt. Case stories are organized for learning. Teachers in higher education
are the intended audience for ELIXR multimedia case stories and so the case stories reflect
upon questions faculty members might ask when considering a new pedagogical approach.
Each ELIXR multimedia case story is unique, reflecting a particular teaching method, the
unique qualities of the instructor who is the center of the story and the style of the team
that developed it. However, most contain the following elements:
A short video overview that previews the content and affective dimensions of the
case.
A story narrative that outlines, in video and text, why the practice was adopted, the
resulting changes, how students responded to these changes in teaching, and
evidence that these changes made a difference in student learning. The authorship
of the story is personalized to reinforce the notion that the faculty member is the
author/storyteller. Critical to the story are the successes and difficulties the
author/storyteller encountered.
4
An author’s personal reflection provides the viewer with a more personal account of
the changes the innovator experienced in teaching and learning.
More detail on this story includes resources such as course syllabi, assignments,
links to outside resources, and assessment data such as actual examples of student
work so that viewers can more fully understand the context and environment
surrounding the implementation of the innovation.
Similarly many include a faculty development resources section containing resources
and tools intended for faculty developers to use when incorporating the case stories
into faculty development programs.
Case stores are developed and produced by campus-based teams of faculty members,
faculty developers, instructional technology staff members and videographers. These
teams are referred to throughout this document as Case Story Theme Leadership Teams or
CSTLTs for short. The ELIXR website (http://elixr.merlot.org) is an online resource
providing access to the cases and their development. Other resources that track the
development of the project are also available.
II The Evaluation Process
II.A.
Evaluation Approach and Constraints
The ELIXR Logic Model (see Appendix A) illustrates the intended outputs and outcomes
that guided the ELIXR project implementation and evaluation. The focus of the evaluative
activities, given the three-year span of the project, was primarily on faculty members as
learners as described by the outputs (i.e., faculty development workshops, the case stories
themselves and online teacher community spaces) and outcomes (i.e., improved faculty
development programs, changes in faculty members’ perceptions and practices in regards
to teaching, changed teaching practices, and so on). It was generally agreed that long term
impacts of the project, e.g., improved student learning, could not be measured given the
relatively short time-span of the project.
The project implementation plan was complex, relying upon collaborations within and
across institutional boundaries. Central to the collaborative effort were the CSTLTs who
worked together to create, design and produce the case stories. The focus on both the
development of the multimedia case stories and then upon their use in faculty development
settings at multiple sites throughout the US required multiple evaluation approaches,
which are described in the following sections of this report.
II.A.1 Formative Overview
Much of the effort expended by project participants was on the following activities:
creation and production of the multimedia case stories,
integration of the stories into existing faculty development programs or the creation of
new programs, and
development and nurturing of online teaching communities.
5
These developmental activities meant that there was a very high need for formative
evaluation related to the projects’ outputs. Evaluation capacity, expertise and resources
differed by each campus. With 28 campuses eventually involved in the project, it very
quickly became apparent that ELIXR resources would need to be developed to support
evaluation at the ‘local’ level for both the case story developers and the faculty developers
using them. As a result, midway through the project, part of the evaluation activities shifted
towards evaluation resource development and education. The ELIXR Data Collection plan
of December 2007 (Appendix B) lays out the details of how the original evaluation plan
was modified to meet the growing needs of project participants. Ultimately, the plan
eventually evolved to reflect the efforts showing in Tables 1 and 2.
Formative evaluation efforts focused on improvement of case stories, faculty development
workshops (i.e., the effective use of the case stories) and improving the organization of the
project as a whole. This information was gathered through regular debriefings held with
the project staff, ELIXR Fellows, the CSTLTs and regular meetings with the External
Evaluator. The outcomes from the formative efforts are summarized below.
As the development of the case stories progressed, peer reviews were conducted by: the
ELIXR Leadership Team, potential faculty development users, and several of the ELIXR
Fellows. The CSTLTs used the results of these reviews to improve or refine the case stories.
The formative evaluation efforts resulted in a number of design documents available for
new case story authors, or those interested in creating case stories. These documents may
be found at: http://elixr.merlot.org/creating-stories3?noCache=506:1268669316#facultydevelopers. (Forty-seven case stories have been submitted to MERLOT; these peer reviews
will be more summative in nature as they will be done on completed case stories.)
Formative evaluation also focused on improving the use of case stories in actual faculty
development events. ELIXR Leadership Team members observed workshops using case
stories in addition to implementing workshops themselves. Feedback from these
experiences was fed back to the developers of the case stories as well, This feedback was
collected and communicated to other faculty developers via the blog for faculty developers,
and in several cases the feedback resulted in revisions to the development and application
processes and tools . The blog included resources and tips for using the case stories in
faculty development programs. It can be found at: http://elixr.merlot.org/facultydevelopment-resources?noCache=399:1268669327.
Meetings between the ELIXR Leadership team and the External Evaluator were conducted
at regular points throughout the project. These meetings focused on progress towards
accomplishing goals, project organization, and revisions of project strategies and tactics to
meet changing and evolving needs. The outcomes of these meetings included for example,
revising evaluation activities to include a focus on evaluation capacity building for
participating faculty developers and institutions, streamlining of the case story
development process to focus on teams, and implementation of the ELIXR Fellows
program.
6
II.A.2 Summative Overview
The purpose of the summative evaluation efforts was to examine the outcomes associated
with the use of multimedia online case stories in faculty development efforts, both formal
and informal. Formal methods being special workshops or seminars; informal methods
associated primarily with an individual faculty member’s view of a case story accessed via a
teaching community or other website. Evaluation efforts associated with formal faculty
development centered on examining the workshop participants’ use of the materials, their
opinions about the workshops and materials and changes in their instructional strategies
or pedagogical approaches resulting from their interaction with these materials.
While the case stories are available to anyone to use, for the purposes of this grant,
evaluation of formal faculty development was limited to the participants in the ELIXR
Fellows program. (See the 2009 ELIXR Annual Report for more details of this program.)
ELIXR Fellows agreed to integrate multimedia case stories into their faculty development
programs over the course of a semester and to survey the faculty members who
participated in the workshops. The surveys were designed by the ELIXR evaluator in
conjunction with the ELIXR management team and the CSTLTs of several of the case
stories. (More details on this aspect of the evaluation can be found in the Methods section
of this report)
SERC, the Science Education Resource Center of Carleton College in Northfield Minnesota
studied the informal use of the case stories. This study focused on examining how to
integrate case stories into a discipline specific site Pedagogy in Action
(http://serc.carleton.edu). This site is devoted to improving teaching and learning in the
earth sciences. To that end, case stories were integrated into the site’s content. SERC
conducted a set of in-depth interviews with faculty users of its site to determine the
potential effectiveness of the cases as faculty learning modules.
Last, a study was conducted with faculty developers attending the MERLOT annual
conference held in August 2009. The purpose of this study was to determine faculty
developers’ impressions about the usability and usefulness of the case stories. These
participants were contacted in October of 2009 to complete a follow-up survey regarding
use of the materials in faculty development programs.
Table 1 (page 9) and Table 2 (page 10) summarize the relationships between the project
activities and evaluation approaches.
II.B
Evaluation Questions
The goal of the evaluation was to answer the following evaluation questions:
1. Was ELIXR implemented as planned, i.e., did ELIXR accomplish its’ goals and
objectives?
2. What was the quality of the implementation, i.e., did the participants experience
high quality case stories and workshops?
3. Was ELIXR disseminated as planned, i.e., were its products adopted or used by the
participating institutions and others, as well as individuals?
7
4. What are the promising practices associated with long-term sustainability and
institutionalization?
5. What factors are necessary to ensure adoption by faculty developers, institutions
and individual faculty members?
6. What was the effect of ELIXR on program participants, i.e., how did participants’
attitudes, skills and knowledge change as a result of participation in ELIXR?
a. What impacts, if any do participants report on with regards to their students
and academic communities?
II.C
Data Collection Methods and Instruments
In the following pages, Tables 1 and 2 below lay out the relationships among the project’s
intended outputs and outcomes, project activities, evaluation questions, data collectors,
data collection processes and uses of the data.
8
Table 1: Evaluation of PROGRAM OUTPUTS
Evaluation Use or Question
Project Activity
Development
of Online Case
Stories
Intended Result
Output: Set of
multimedia online case
stories
Data Collection
ELIXR
Leadership Team
ELIXR
Leadership
Team, Volunteer
Case Story Peer
Reviewers &
MERLOT Faculty
Development
Editorial Board
ELIXR Fellows
Online
Teaching
Community
(ELIXR
Website)
Faculty
Development
Workshops
Output: Case stories
integrated in online
teacher community
space
Output: Workshops
developed and offered
using case stories
Data Collection:
Instruments & Processes
Number of Cases
Developed & Deployed
Peer Review of case
stories
ELIXR
Leadership Team
ELIXR Fellows
Evaluation: Workshop
survey
Webstats on usage of
ELIXR.MERLOT.org
ELIXR
Leadership Team
# of Themes
implemented
ELIXR Fellows
# CSTLTs creating case
stories
End of Workshop
Questionnaire
# of workshops
# of participants of
workshops
9
Formative
ELIXR Leadership Team
uses to determine needs
for new cases or Case
Story Theme Leadership
Teams (CSTLTs)
CSTLTs use reviews to
improve case content
and presentation
Summative
EQ 1: Did project meet goals &
objectives?
EQ 2: What was the quality of
implementation, i.e., did the
participants experience high
quality case stories and
workshops?
ELIXR Fellows use to
improve workshops
Website developer
tracks usage to improve
usability of site
ELIXR Leadership Team
uses to determine needs
for new cases or CSTLTs
EQ 3: Was ELIXR disseminated
as planned? What is the level of
use of products by others?
objectives?
ELIXR management
team develops
promising practices
resources and
disseminated via
teaching community
sites.
objectives?
EQ 2: What was the quality of
implementation, i.e., did the
participants experience high
quality case stories and
workshops?
Table 2: Evaluation of PROGRAM OUTCOMES
Evaluation Question/Use
Project Activity
Faculty
Development
workshops and
case stories
Intended Result
Outcome: Improved
faculty development
programs through use of
Case Stories
Outcome: enhanced
teaching knowledge,
practices, etc.
Data Collection
ELIXR Fellows
Data Collection:
Instruments & Processes
ELIXR Fellows Evaluation:
One-on-one interviews
with ELIXR Fellows
ELIXR Fellows
Post-workshop survey
CSTLTs
Survey of CSTLTs
ELIXR Fellows
Pre-Workshop
Questionnaire
Formative
EQ 4: What are the promising
practices associated with long-term
sustainability and
ELIXR management team
develops promising
practices resources for
faculty development and
disseminated via teaching
community sites.
EQ 5: What factors are necessary to
ensure successful adoption by
faculty developers, institutions and
individuals?
ELIXR Fellows revise and
refine faculty development
workshops
EQ 6: What was the effect on
workshop participants?
Post-Workshop Follow-up
Questionnaire
Other local evaluation data
collection methods
Online Teaching
Communities
supporting Case
Stories
Summative
ELIXR management team
develops promising
practices resources for
faculty development and
disseminated via teaching
community sites.
individuals?
EQ 6a: What impacts, if any do
participants report on with regards
to their students and academic
communities?
Outcome: Improved
faculty development
programs through use of
Case Stories
Faculty
Development
Professionals
User study: observation,
interviews & surveys
ELIXR website & cases
refined & improved based
on feedback.
individuals?
Outcome: Improved
faculty access &
contributions to teaching
communities
SERC
User Study: observation &
interviews of case stories
integrated into
serc.carleton.edu
ELIXR & SERC management
teams refine and improve
websites & cases based on
feedback
EQ 4: What are the promising
practices associated with long-term
sustainability and
individuals?
10
II.D
Evaluation Studies
Five evaluation studies were conducted to answer the evaluation questions. They are
briefly described below. More detailed reports on the methods and summaries of the data
from these studies may be found in the appendices.
II.D.1 Use Statistics
Google Analytics were used to learn about how users interact with the ELIXR online
teaching community website focusing in particular on access to the case stories. (The
detailed report on analytics can be found in Appendix C.) The use statistics were examined
from April 1, 2009, the launch of www.elixr.merlot.org and March 15, 2010. The analytics
looked at the number of visits and visitors during that time frame, as well as visitor loyalty,
geographic location of visitors, how they discover the ELIXR site and the use patterns
associated with the case stories.
Visits to the site increase dramatically once the case stories were made available and
outreach regarding them began in earnest. The number of visits rose steadily during the
summer of 2009, peaking in August. (It is important to note that during August the site was
heavily promoted during the MERLOT International Conference.) Visits declined slightly
thereafter until the start of the spring 2010 term, when the downward trend appears to
have reversed possibly as a result of online webinars conducted with other organizations
such as the Teaching and Learning Technology Group (TLT). Because tracking can only be
analyzed for this time frame, it is impossible to identify with any certainty trends. However,
it does appear that usage is higher before the beginning of a term, which may indicate that
faculty developers and instructors are visiting the site in preparation for workshops or
implementing new teaching activities.
The popularity of case stories is also dependent upon time, in that the case stories that
were made available early in the project appear to be the most popular. This is most likely
only because they have been more heavily promoted. It is interesting to note that
regardless of the popularity of a case study (popularity is measured by the number of page
views a case has) users spend a great deal of time viewing them. The average time spent on
a case page is 3 minutes, 50 seconds. When examining the average times for each case,
most hover around that 3:50 mark, or extend beyond.
II.D.2 ELIXR Fellows Evaluation Study Overview
The ELIXR Fellows program was developed as a means to examine the use of the case
stories in situ. Led by Tasha Souza (CSU – Humboldt) four faculty developers from three of
the partner states were selected to participate: Roberta Ambrosino (University of Texas
Health Science Center – San Antonio), Kathy Ross (Indiana University at Kokomo), Joe
Grimes (CSU – San Luis Obispo) and Kirin Dosanjh Zucker (CSU – Northridge). Each of the
faculty developers selected a case story to use in a face-to-face faculty development
workshop setting (in one case, an online version of the workshop was also conducted.) As
part of their responsibilities as a Fellow, they agreed to gather evaluation data using the
workshop survey instruments developed for the project. (The summary report on the
study may be found in Appendix D).
11
Each instance of the workshops was unique; not only did they use different case studies
(with some overlap), they also used the case stories in different ways. This variety in usage
mirrors how it is anticipated that the broader community of faculty developers might use
the case stories in faculty development situations.
The workshops were conducted and data collected between July 2008 and November 2009.
Approximately 60 participants attended the workshops, 44 workshop evaluations and 38
pre and post surveys were returned. (In one situation, the Fellow used a locally developed
instrument to evaluate the workshop and because of privacy concerns was not able to
share actual results of evaluation.) The pre and post survey instruments were designed to
gauge participants’ familiarity with the topic prior to the workshop and to identify changes
they made as a result of attending a workshop. They were also asked to complete a
questionnaire regarding their satisfaction with the workshop. All instruments were
designed to provide the workshop leaders with formative information on how they might
improve the workshop as well as to serve the project’s need to gather evaluation data.
Findings from the study indicate that participants in the workshops not only self-reported
learning about the workshop topic, if appears that a number of the participants also
changed their teaching practice in some way as a result of attending the workshop.
Participants reported that they liked the case story videos and some wished there had been
more cases to review as examples. Findings from this study must be interpreted with care
as each evaluation was conducted with a relatively small number of respondents.
Each of the ELIXR Fellows was also interviewed as a part of this study. The interview study
focused primarily on identifying how case stories might best be used in faculty
development situations and barriers to their use. The Fellows reported that they were able
to use the case stories in face-to-face workshops to encourage interactive learning, to jump
start discussions and to give participants more examples beyond those discussed in the
workshop. The cases were also used in online workshops and in faculty learning
community settings. The Fellows reported challenges and possible barriers as being:
the time necessary to learn the case story and determine which aspects of it can be used
for a particular workshop
lack of flexibility in terms of the case story online environment (Pachyderm) with
regards to, for example, projecting the case story or ability to cue up a specific video clip
to illustrate particular points
lack of bandwidth for users working from home (less so if used within an institution)
II.D.3 ELIXR User Study Overview
Effective dissemination of innovation is often measured by its adoption or adaption by the
target audience. To learn more about dissemination to faculty developers an observation
study was conducted with faculty developers who attended the August 2009 MERLOT
International Conference held in San Jose, California. Attendees were invited to review a
case story and respond to a survey evaluating it, or to participate in a one-on-one
interview. Approximately 40 participants viewed a case story and completed the survey; 11
were interviewed. In October, all of these participants were invited to complete a follow-up
12
survey regarding their use of the case stories. 17 (36% response rate) responded to the
follow-up survey.
In general, all participants reported being very impressed with the potential value of the
case stories for faculty development to increase the adoption of exemplary teaching
practices. (The summary report on the study may be found in Appendix E) They felt they
would be extremely useful in introducing faculty members to a new topic and they felt they
would be particularly useful to new faculty. Most reported that they were not influenced by
the case stories to adopt the innovated teaching practice presented in the case story. Most
likely this was because they already practiced it, or the one they selected for the purposes
of the study did not meet their teaching needs in terms of content, type of course or course
level. This is consistent with innovative nature of those participating in the study for
attendees of the conference tended to be experienced instructors who already practice
innovative teaching techniques. They reported that they would help to disseminate the
ELIXR project by sharing information about it with their colleagues or others on their
campus.
The follow up survey results showed that of those responding, few had had the opportunity
to share information about the case stories in the manner they had intended, and none had
used the case stories in their faculty development efforts. Timing may have had an impact
on the results, i.e., at the time of the study, faculty developers may have had the sessions
already planned for the fall term, and thus were not able to integrate a case story into their
sessions because they did not match the topics of the workshops. One of the purposes of
the follow up survey was to determine the optimum amount of interaction with the site
necessary to encourage use by a novice. The results indicated that one chance to explore a
resource was not enough to encourage use in a formal program. Respondents also
indicated that they needed frequent reminders and repeated interactions with a resource
in order to put more priority on integrating something new into their activities.
II.D.4 SERC User Study Overview
The SERC study looked at the specific instance of incorporating ELIXR case stories into
existing SERC modules in the Starting Point collection of teaching materials for the
geosciences. Links to ELIXR case story videos went live on November 8, 2008 within the
‘First Day of Class” and the “Just in Time Teaching” modules. The purpose of this evaluation
was to assess the value and utility of incorporating the cases into existing web-based
pedagogic modules, and to get a sense of how they might be used by visitors to the site.
SERC surveyed 11 Geoscience faculty members who were experienced SERC users after
they have viewed a module. They also examined web statistics to determine the paths of
users and conduced several in-person walk-throughs.
Findings from this study indicated that the case stories enhanced the SERC modules by
providing additional examples and illustrations. They added another medium for
communicating how-to implement the techniques discussed at the SERC site. (The
summary report on the study may be found in Appendix F.)
13
II.D.5 ELIXR CSTLT Study Overview
Essential to the development of the case stories were the Case Story Theme Leadership
Teams (CSTLTs). In order to identify promising practices and resources for
creating/disseminating cases, 29 of the CSTLTs from 24 different campuses were surveyed.
These participants were also key to providing insight into dissemination and adoption of
the cases. 13 of the CSTLTs responded to the survey (45% response rate).
The ELIXR website and management team provided the designers with online resources
and at least in the initial stages, significant one-on-one support. The designers confirmed
the helpfulness of the personal contact as well as the review website. Respondents
reported that the guides and informative materials were moderately helpful. The CSTLTs
also reported that the collaborative nature of working with the faculty members and the
rest of the team generally went well, but they noted that it could be challenging. The time
consuming nature of the project was also mentioned as being surprising to them, especially
the time it took to work in a collaborative manner. The respondents reported that they had
already used and disseminated their products, mainly on their own campuses. Some
reported sharing their story with other institutions. About half reported use in faculty
development programs; those who had not reported that use was in the planning stages.
(The summary report on the study may be found in Appendix G.)
III
Evaluation Findings, Conclusions and Recommendations
This section of the report discusses the findings of the various components of the ELIXR
evaluation in relation to each evaluation question. Tables 1 and 2 (see pages 9 and 10)
illustrate the linkages between the evaluation questions and the project’s intended outputs
and outcomes. The discussion that follows is organized by the program’s intended outputs
and intended outcomes . The relevant evaluation questions guide the discussion of findings
and recommendations.
III.A
Intended Outputs
The intended outputs for the project focus on the three main activities of the project:
creating and development of case stories, creation and development of faculty development
workshops that incorporate the case stories and the development of the ELIXR online
teacher community space.
III.A.1 Case Stories
The original target for the number of case stories to be created in this project was 96. At the
end of this three year period, 70 case stories representing 19 themes have been completed
and are currently available at the ELIXR website (16 case stories from our Canadian
institutions are still in production). Twenty-eight campuses and 30 CSTLTs have been
involved in creating the case stories.
Outputs as they are used in this project’s logic model are the products developed, e.g. case stories,
workshops, etc. Outcomes are a change in behavior by the intended audience using the products, e.g., faculty
developers who use case stories, faculty members, etc.
14
The ELIXR strategy for developing case stories relied upon collaborations among partner
institutions. Small groups of interested faculty developers, faculty members and
instructional technologists would decide upon a theme, e.g., Integrative Learning. Once a
theme was established, the group would create multiple case stories centered at their home
campus. As the project evolved, it has become clear that this strategy was effective,
especially if the teams were geographically close to one another. And, while funding was
used to help support the CSTLTs efforts, the home campuses absorbed the actual cost of the
production. Interestingly, none of the respondents in the ELIXR CSTLT study commented
on cost being a barrier or challenge to overcome in creating the case stories.
The production of case stories began slowly. In its initial stages the project required
significant start up time to educate potential partners. The notion of using multi-media is
new to faculty development – there were few examples to turn to and the collaborative
nature of the endeavor required upfront organization. However, once the first case story
was created and resources developed to support the creation (e.g., Lou’s Zweier’s blog on
how to create case stories) and a more systematic support system for those who had not
create a case story previously, the CSTLTs gained momentum and production increased. It
is important to note that the ELIXR Leadership Team in the early stages of the project spent
a great deal of time providing the CSTLTs with individual, one-on-one advice on how to
produce the case stories. As the project evolved this expertise was codified and made
available via the tools on the website (see http://elixr.merlot.org/creatingstories3?noCache=707:1270160460 ) So, while the project has not yet reached its final
target, it has shown that the process for developing them was sound.
It is not enough to simply develop case stories; the project was also dedicated to the
creation of quality case stories. Case stories are developed as learning objects for faculty
members. As such, just like online curriculum materials, their content must be correct, well
organized and must ‘ring true’ to the viewer or user. To ensure the cases were of the
highest quality, the ELIXR Leadership Team turned to the MERLOT Faculty Development
Editorial Board to conduct peer reviews of the case stories. This review would serve
several purposes: it would provide a summative review for the quality of the case story,
and it would also work as a mechanism for increasing dissemination of the cases via the
MERLOT site. However, this peer review process, like professional journals can be time
consuming.
To date, all of the case stories have been submitted but no formal review has been posted.
(User comments have been posted, which provide yet another form of review.) Another
important element in the review process was a formative review that could provide much
needed feedback while the case story was still in production. During the project, the ELIXR
Leadership Team created an informal review team made up of members of the Leadership
Team, members of CSTLTs, the ELIXR Fellows and volunteer faculty development
professionals interested in the project. This group have provided developers with much
needed feedback on their case stories. While no formal criteria were used, the reviewers
were experienced case developers or users and so were able to provide the developers
feedback along those lines. The results from the user study were also helpful to these
reviewers.
15
Recommendations
Members of the CSTLTs recommended the development of a case story on how to
create a case story. This resource would be an excellent model for those interested in
pursuing the development of cases and it would free the leadership team from the time
consuming and burdensome consulting role.
Evaluation or review criteria should be codified and published as a resource for
developers.
Development of criteria for acceptance or ingestion of cases into the ELIXR collection
should be published and shared to ensure that only the highest quality cases are shared.
Survey results from the CSTLT survey showed that learning and using Pachyderm was
difficult and time consuming. Since the initiation of the project, social media
environments, particularly YouTube and Facebook have become increasingly popular.
While Pachyderm has the benefit of creating a learning object that links multimedia
with traditional text resources, it may be overly restrictive. The ELIXR Leadership Team
and its partners should explore how they might decouple the strengths of the case
stories (i.e., the videos and resources materials) yet keep the integrity of the learning
experience using less burdensome social networking environments.
The collaborative model is an effective strategy for building the cases but requires
significant coordination from (in this case) the ELIXR Leadership Team, especially for
collaborations that cut across institutional or geographic boundaries. ELIXR should
explore how collaborations might be supported via other professional development
venues, such as special topic groups (TIGS) associated with professional societies such
as the POD Network or the New Media Center (NMC).
III.A.2 Faculty Development Workshops
Integration into existing or new faculty development workshops was one of the major uses
envisioned for the case stories. The only workshop developed at the ELIXR Leadership
Team level was designed for the 1st Day of Class case story. The materials associated with
that plan were included as resources with the case story and were instrumental in
determining the design for the workshop evaluation instruments. The 1st Day of Class
workshop materials proved to be an effective model for faculty developers to test the use of
the case stories. At least one ELIXR Fellow was recruited into the program because she
discovered the materials, liked them, and used them as a pattern for her faculty
development workshops. Others who examined the case stories also noted the resources
available in the 1st Day of Class model and commented that they would appreciate it if more
resources, such as those associated with planning and implementing workshops were
included.
This model was not however, repeated for other case stories, there was no underlying or
centralized vision for how the case stories related to workshops and unlike the case stories,
there were no specific targets set. Instead, the workshops were seen to be more in control
of the individual faculty development professionals and so case stories were to be designed
to be flexible enough to be used in multiple ways. The ELIXR website section on faculty
development provides potential users with a space to learn more about how to use the case
stories and the experiences of the ELIXR Fellows is a helpful tool especially for those who
wish to use the 1st Day of Class case story.
16
Since the priority in the first half of the project’s lifespan was to create case stories, their
use in workshops did not begin until the 2nd half of the project. Over the lifetime of the
project ELIXR estimates that 10 workshops were conducted using case stories . The ELIXR
Fellows used them in five different workshop formats. Members of the ELIXR Leadership
team or CLTLTs conducted the remaining workshops. Approximately 75 faculty members
participated in the ELIXR Fellows’ workshops and it is estimated that a similar number
participated in the other workshops. Evaluations of the workshops (See Appendix D)
showed that the participants felt they had learned from the workshops and the case stories
rated comparably to other aspects of the workshops, e.g., presentations, discussions, etc.
The respondents in the CSTLT survey (See Appendix G) indicated that the case stories they
had created were being integrated into their campuses’ faculty development efforts.
Recommendations
The lack of a coherent plan associated with the use of case stories in workshops was a
weakness in the ELIXR project. The identification of best practices in the use of case
stories for faculty development would have been more efficient had the design of the
project included a set of testable workshop models. The lack of connection to a theory
of change or model makes evaluation of the materials difficult for they cannot be
separated from the androgogic methods used in the trainings. In the future, care should
be taken to develop (based on the best practices discussed later in this report)
workshop models that highlight effective uses of the case stories.
Workshop models, materials and references should be included in each case story.
Case story developers should carefully consider the learning outcomes associated with
each case story. The outcomes should be articulated within model workshops described
in the resource sections.
III.A.3 ELIXR Online Teacher Community
The ELIXR Online Teacher Community (www.elixr.merlot.org) has recently been updated
and refined. It holds a collection of over 70 case stories and more stories are under
development. Individual case stories are cataloged in MERLOT and are currently under
review. ELIXR resources are also available through the SERC site as well as POD.
ELIXR’s dissemination goals have been met in terms of creating links with online partners
and in terms of developing partnerships with developers and users. The growth of the
visitors to the ELIXR site as measured by Google Analytics (See Appendix C) shows that a
growing number of users are visiting the site. Jumps in use tend to coincide with
dissemination efforts such as presentations at conferences or online workshops. The level
of use by faculty developers remains unknown. While the web statistics indicate users are
coming to the site and viewing the stories, there is no way to know what they actually do
with the stories. In other words, a user might view a story for their own personal learning,
recommend it to another for viewing, they might show it (or part of it) to a group of faculty
The open nature of the ELIXR website (users are not required to register to download case stories) makes it
impossible to track the exact number of cases used and in what ways. ELIXR only knows if a case story has
been used if the user of the case informs ELIXR. Thus this estimate was based on the number of faculty
developers who consulted worked with the ELIXR Leadership Team in how best to use the cases.
17
members, there is simply no way to telling how it is used. Web statistics can only measure
that it has been used, not how.
Members of the ELIXR Leadership team have presented at a number of conferences, e.g.,
POD, MERLOT International Conference, the New Media Center (NMC) annual conference,
as well as presented at a number of online workshops sponsored by the TLT Group and
CATS of the CSU system. Recently, the ELIXR Leadership Team authored a chapter to be
included in the coming issue of To Improve the Academy. (A list of the presentations,
papers etc. can be found in the project’s final report.)
The Survey of CSTLTs indicates that they have also taken an active role in disseminating
their materials through informal means (i.e., recommending a story to a colleague) as well
as more formal means such as writing announcements about the case story for online or
print publications such as newsletters. The ELIXR Fellows also actively promoted the case
stories and workshops at professional conferences and meetings.
Recommendations
The ELIXR online community site needs to be more visible via MERLOT. Currently the
ELIXR site is cataloged at the same level as the case stories. This makes it difficult to
find at the ELIXR site. To raise its visibility, ELIXR should work closely with the relevant
MERLOT editorial boards to move it onto the various discipline community sites.
To spread the word about ELIXR beyond MERLOT, consider making more links to
campus faculty development sites and other sites devoted to teaching. The links to
SERC and POD are good examples of dissemination partners; other sites might include
system level sites (e.g., the CSU Faculty Development Site) or Teaching and Learning
Centers at other campuses.
NMC provides a link into the instructional technology community. Faculty development,
as a growing field is found on many campuses in both the more traditional teaching
centers as well as instructional technology departments. NMC is a potential partner that
can lead to continued support and development of case stories as well as another
avenue to follow to involved faculty members.
The OER movement focuses on making educational materials widely available,
especially to under developed countries. It lacks resources with strong grounding in
effective pedagogy. This gap may be an opportunity for building new partners and
meeting the needs of a growing international audience of faculty members.
III.B
Intended Outcomes
The intended outcomes for the project focus on anticipated improvements in the outputs of
the project and changes in the participants as a result of using or interacting with the
products. Use of the case stories were intended to lead to improved faculty development
programs, enhanced teaching knowledge and practices on the part of the faculty members
participating in the workshops and improved access and contributions to faculty teaching
communities. Development of a sustainable organizational model to support growth in
resources and in use was the overarching outcome for the project.
18
III.B.1 Improved Faculty Development Programs
The ELIXR Fellows and several of the ELIXR Leadership Team members tested a number of
the case studies on their campuses. Each case study was used in the unique faculty
development environment of a particular campus. The 1st Day of Class being the most
frequently used case story in multiple locations and in within different contexts. The
workshop evaluations all pointed to successful integration of the case story, regardless of
the case story used or the way in which it was used, i.e., face-to-face workshop or online
workshop.
Use of the case stories did require that the faculty development professionals (ELIXR
Fellows) leading the sessions develop new competencies namely, how to use multi-media
resources effectively. This was not an insurmountable barrier for this group of users. They
did however, benefit from one another’s experiences. The group held several conference
calls over the course of the project and used the Faculty Development blog to share their
experiences and discuss what worked and where they had problems.
The workshops evaluations and interviews with the ELIXR Fellows identified a set of
practices in the use of the case stories that indicate that they added value to the workshops.
For the most part, in the face-face workshops, the entire cases were not used. Instead,
video clips were selected to illustrate a particular concept, to demonstrate a particular
technique or aspect of a practice, or to expand the set of disciplinary examples. Both the
Fellows and the workshop participants commented that this was particularly effective. Use
of the entire case was employed when faculty developers wanted workshop participants to
be introduced to the topic prior to participating in a face-to-face workshop or when
conducting an online version of the workshop.
The Fellows tended to focus on embedding the videos into their workshops, not the entire
case. Several of the Fellows in fact, used only the video sections, using downloaded versions
of the video instead of those available through the website. This type of use suggests that
the website itself, while a good dissemination vehicle for use by individuals may not be
effective when projecting for larger groups.
Recommendations
The blog for faculty developers and resources with tips and best practices for using the
case stories is an effective online set of resources for potential users. This is true as long
as it is easily discoverable. Faculty developers have been relatively slow to adopt online
technology as a means of building community and communication so it will be
important to increase the visibility of the ELIXR website and resources among this
group.
As the use of the case stories expands, it is likely that they will be used in new and
unique ways. Support of these users can lead to the development of a community of
users who can help disseminate their practices. Existing partners (e.g., MERLOT) might
The tendency to avoid technology is changing as the recently redesigned website for POD illustrates. POD
has depended upon its vibrant listserve to build online community, this is currently being migrated onto their
website (http://podnetwork.org/index.htm)and the new wikiPODia site.
(http://sites.google.com/site/podnetwork/)
19
be tapped to support this community; other supporters that might take up this effort
include POD and the TLT Group, both of which focus on the professional development
of faculty as well as the professional developers themselves.
Faculty development programs have varying levels of expertise and resources available
to evaluate their programs. ELIXR invested in the development of evaluation tools and
processes, which are freely available to those who wish to use the case stories. These
users should be encouraged to not only evaluate their workshops, but also to share the
results of the evaluation via the ELIXR website and other venues.
Perhaps the most significant factor in improving faculty development via the use of case
studies is ensuring that faculty developers actually adopt the case studies as ‘go to’
resources. The results of the follow up aspect of the user study indicate that this may be
difficult. ELIXR, like other educational innovations, must confront the challenges
associated with moving beyond the innovators. The follow up study suggests that even
when given the chance to explore the resource in depth, the participants had not
incorporated the cases into their repertoire of potential resources for faculty
development. Participants indicated a number of reasons for this, focusing in on the
need to be reminded about the resource.
One of the unsettling findings of the ELIXR User Study was the lack of use following an
intense introduction to an ELIXR case story. The lack of use of the case stories, even
though the potential users found them to have high potential, points out the need for
systematic and concentrated outreach efforts on the part of ELIXR and its partners.
Regular outreach must be made to this community to not only inform them about the
availability of the resource but also to model effective use the resource. The CSTLTs
may be able to play an important role for they have indicated that they have been using
the case stories for faculty development on their own campuses. They should be
encouraged to take their efforts to the next phase, that of outreach to the larger
community.
III.B.2 Enhanced Teaching Knowledge and Practices
The ELIXR project was predicated on the desire to make exemplary teaching practices (that
is those that have already been shown to be effective through practice and research) more
available to faculty members through new faculty development approaches involving web
technologies. Conducting a conclusive study regarding the impact of the case stories on
teaching knowledge and practices would require a significantly more in depth study than
was possible to accomplish within the timeframe of this grant. Taken together, however,
the evaluation conducted by the ELIXR Fellows along with the SERC study suggest that
faculty members who participated in the workshops using the case stories and those who
viewed them on the SERC site did learn about the topic and make changes in their teaching
practices.
Both of the studies found that faculty members even though experiencing the cases
differently, i.e., some via the SERC website, others via a faculty development workshop,
found them valuable tools for communicating how to implement a teaching technique.
Follow-up surveys with the workshop attendees indicated that they had, when possible,
20
included one or more of the activities associated with a particular workshop into their class
activities or they were likely to do so in the future.
The samples in both studies were similar in that they were for the most part, very
accomplished instructors . Both groups indicated that they were not likely to return to the
videos to improve their own teaching practices. They found that the videos reinforced what
they already know and had learned from experience. They concluded that they were
valuable, but that they would be most valuable for less experienced instructors.
Recommendation
While the studies regarding enhanced teaching knowledge and practices indicate
changes by faculty members who interacted with the case studies in both the website
setting and workshop settings, more research in this area is necessary. These initial
studies were conducted when only a few of the case stories had been completed. Now
that there is a richer compendium of resources, it is possible to conduct a larger scale
study.
III.B.4 Improved Access and Contributions to Faculty Teaching Communities
Incorporating the ELIXR case stories into the resources found at the SERC Pedagogies in
Action website (http://serc.carleton.edu/sp/index.html) represents one means for
integrating these resources into websites that promote good teaching practices. The
Pedagogies in Action digital library of pedagogic methods and collection of activities was
designed to support instructors in exploring new pedagogies. ELIXR case studies were
integrated into pages of the Pedagogies in Action site dealing with the 1st Day of Class, Just
in Time Teaching (JITT) and ConcepTests. The Pedagogies in Action site is a well-known
and respected online teaching resource within the Geosciences community. ELIXR
partnered with SERC in part to test the effectiveness in linking case studies with discipline
specific online communities.
SERC conducted an evaluation of the integration of case stories into the Pedagogies in
Action website (see Appendix F). This evaluation found that the case stories enhanced the
modules for this set of experienced users of the Pedagogies in Action site. The case stories
and most especially the videos, were found to add value to the existing materials and
information found at the site. Viewers found that they validated the techniques they were
using and they believed they would give viewers who were not currently using the
pedagogical technique more confidence in trying them out. Participants felt that one of the
strengths of the videos was that they provided a number of examples for implementing
teaching methods in diverse settings.
The results of this partnership indicate that members of this discipline specific community
who were regular users of their site, found the case studies to be a valuable addition to the
resources already available to them. In this instance, SERC carefully embedded the case
stories into relevant content areas, meaning the link to the case study was not simply
The faculty developers in the user study conducted during the MERLOT International Conference also
tended to be instructors for whom faculty development was an additional duty for them. The faculty
members in the SERC study were not associated with faculty development in any formal way.
21
‘tacked on’ to a page on the site. This care to the design and implementation, to
contextualization of the case, may be key to effective integration.
Partnering with discipline specific resources such as SERC appears to be an effective
strategy for increasing access to cases by individual faculty members. The case stories
illustrated the techniques and pedagogies that had only previously been described in text.
They also provided a wider variety of examples of use, especially across different
disciplines, which seemed to be attractive to the viewers. This partnership showed was
then, an effective means for contributing to and expanding access to the materials within a
specific faculty teaching community. Members of that community found it to contribute to
it.
Recommendation
Continue to explore partnerships with discipline specific online faculty communities
such as SERC. ComPADRE (www.comPADRE.com) for example (also a SERC partner)
serves the physics community. Having 10,000 registered users, a partnership here
would extend access into a large and vibrant community. Other online discipline
specific communities with similar resources include: Biology (www.biosciednet.org),
Sociology (www.teachingwithdata.org), Mathematics (www.mathdl.org), Engineering
(www.needs.org).
III.B.5 Sustainable Organizational Model
ELIXR was successful in establishing 28 partners and 30 some development teams within
the US in a two-year period. This model proved to be successful as a means for supporting
the development of the case stories. At the point of time at which this report is being
written, it appears that as planned, a community is beginning to coalesce around the cases.
The collaborations benefited from the support of small mini-grants in Year 1 and the
extension of that model through a Request for Proposals (RFP) in the second year. Each
development team received minimal support. While funding may have been necessary to
jump start a community of developers and support the development of this first set of case
stories, it is unclear if funding is necessary for continued development of case stories. In
interviews with the ELIXR fellows, the two who also participated in creating a case story,
indicated that they were continuing to create case stories and work in this milieu,
suggesting that continued funding while helpful, was not crucial to their continued interest
in the format nor to creating more case stories. Likewise, in the results from the survey of
theme leaders, cost was not mentioned as being a barrier to the creation of the case stories.
These findings suggest that case stories may become more commonly used in faculty
development situations once faculty developers and instructional technologists have
created one and learned what resources and skills are needed.
However, growing and sustaining the community of users (i.e., faculty development
professionals, individual faculty members, etc.) will require some level of financial support
necessary until the community reaches the critical mass necessary for self-sustainability. In
February 2010 the ELIXR team laid out a series of strategies for sustaining the project
beyond the initial funding period. The strategies included: ensuring the availability of the
content, providing continuing training and professional development in terms of creating
22
case stories, conducting marketing and outreach to promote the use of the case stories for
faculty development, providing technology support for Pachyderm, the ELIXR website and
the videos and lastly the management of the project. Many of these strategies are being
acted on as the funded portion of the project is winding down, For example, the CSU-CDL
has taken over the maintenance for the ELIXR website and is willing to provide technical
support for the site and Pachyderm (or those interested in developing a case story) until
2015.
ELIXR Leadership team members have also been actively promoting the project through
conference sessions and online webcasts hosted by POD, TLT and NMC. Team members are
contacting disciplinary associations and teaching and learning organizations (including
those already mentioned) to develop new partnerships as well as strengthen existing in an
effort to expand outreach efforts and enlarge the existing community of users around the
case stories. Web statistics show increased visits after each of these kinds of activities
indicating suggestion they are effective in introducing potential users to the site and the
case stories.
Still, ELIXR faces a major challenge with regards to building and maintaining the kind of
community that will become self-generating (i.e., create new and expand existing uses of
cases as well as develop more sophisticated uses of case stories, videos and multi media for
the purposes of faculty development.). Successful online community builders such as SERC,
MERLOT, the Math and Science Partnership - MSP (http://hub.mspnet.org/index.cfm),
ComPADRE, etc. have been relatively well funded for 10 years (or more in some cases) and
still there is some question as to whether their participants have extended beyond
innovators into the early adopters group. Funding has allowed these groups to combine
face-to face-community building activities with their online efforts, which have led to
growth in user communities.
ELIXR was not formed as an independent community. Unlike the others listed above, it is
not a portal to a variety of materials, instead it builds on Web 2.0 thinking meaning, it relies
upon linking to other aspects of the Web to build its community. ELIXR has shown progress
when linking with online communities able to fully integrate the case stories into their
resources such as they did in collaboration with SERC. Such integration into MERLOT has
not been achieved in part because of the design and requirements associated with
MERLOT’s online environment. This raises a number of questions about ELIXR’s eventual
integration with MERLOT, such as: How do the two distinct ELIXR and MERLOT members
combine into one, unified community? By what processes will MERLOT encourage its
members to ‘own’ the project to the same or similar extent that others ‘own’ MERLOT
services?
Perhaps the most critical aspect associated with ownership is finding ELIXR a champion
within MERLOT. One likely candidate is the MERLOT Faculty Development Editorial Board.
Everett Rogers’ seminal work on diffusion of innovations claims that no more than three percent of adopters
are innovators. They are the people who first try out an innovation and adopt it. Early adopters make up
approximately another 14%. Adoption of an innovation must move into this second group for an innovation
to be to successfully adopted.
23
Again however, several questions must be resolved for this to be successful. How can the
Editorial Board give priority for review to ELIXR products over other resources? How
might ELIXR resources be highlighted on the Faculty Development Community page
(http://facultydevelopment.merlot.org/index.html.); similarly, how might other MERLOT
discipline communities highlight or promote resources related to their discipline? How will
the MERLOT administration support the project when and if, ELIXR’s principle players no
longer hold leadership roles within MERLOT?
In sum, There is a small but growing community of users and developers associated with
ELIXR. ELIXR has created and made sharable a large number of potentially useful case
stories. These stories have the potential to significantly impact faculty development
practices by making them available to a much wider audience via the web. Currently, there
is no plan to continue ELIXR as a formal organization. Instead it is depending entirely upon
the growth of the virtual community and the support of a core of volunteers who ‘own’ the
case stories and the individual projects.
Recommendations
While commitment levels of volunteers remains high, continue to encourage case story
developers (and other ELIXR participants) to share their experiences at professional
society meetings (POD, ELI, ISOTL), discipline-based educational conferences (AAAS,
AAPT, AGU, etc.) and virtually, via websites associated with the organizations
previously listed and other social networking sites such as Facebook and YouTube, and
so forth.
Continue to militate for higher profile and exposure on the MERLOT site. As a MERLOT
project, work with the MERLOT management team to find ways to integrate the
coordination and promotion of the case stories within their everyday outreach efforts.
When possible, continue to link case stories to websites frequently visited by faculty
developers and individual faculty members alike. In this recommendation I am
particularly interested in encouraging ELIXR to devise strategies to take advantage of
the power of social networking instead of the waning influence or utility of portal based
communities.
Should the ELIXR community coalesce and grow as an entity, it should seek funding for
linking more explicitly to projects associated with the Hewlett-funded Open
Courseware Consortium (http://www.ocwconsortium.org/). This group, while making
great strides in increasing access to online learning materials (especially
internationally) lacks linkages to faculty development and pedagogical resources. The
case stories are an extremely effective tool that would fill this gap.
24
Appendix A: ELIXR Logic Model
25
ELIXR Logic Model
Planned Work
Inputs
Faculty Dev.
Specialists
Intended Results
Activities
Develop Workshops
using case stories
Project Staff (Lou,
Cynthia, etc)
Create online case
stories
Outputs
Workshops
for faculty,
using
discipline
case stories
re: innovative
pedagogies
Set of online
case stories
Faculty Authors of
Case Stories
Repository Staff
(MERLOT, SERC)
Outcomes
Improved faculty
development
programs
Case stories
integrated in
online teacher
community
spaces
Enhanced faculty
teaching knowledge
Enhanced faculty
knowledge & attitude
re resource re-use
Faculty access &
contribute to cases
Sustainable organizational
model to support growth in
resources and in use
26
Improved teaching
(faculty gains vis a vis
knowledge, skills,
attitudes, use of
resources & networks
Faculty perception of
value from cases
Enhanced faculty
teaching practice
Develop online
teaching
communities to
support case stories
Impact
Improved
Student
Learning &
curricula
Enhanced
faculty
communities
Appendix B: ELIXR Data Collection Plan
27
28
Appendix C: ELIXR Use Statistics
29
Google Analytics Report - ELIXR
April 1, 2009 – March 15, 2010
Prepared by: Flora McMartin
30
Introduction
This report examines usage of the ELIXR website (www.elixr.merlot.org) and its contents
(multimedia case stories) from April 1, 2009 to March 15, 2010. The April date is the
launch of the official website. This report covers basic metrics useful for examining the
ELIXR project goals, growth in use of the site as well as its content. The results will provide
a baseline for future comparisons. Google Analytics is a well-known and trusted free
service that tracks this type of information. In the future analysis of the analytics will
provide deeper insight into user profiles and activities at the site. These data will be
important for determining the usability of the site, impact of outreach efforts and potential
for new services.
Glossary of terms
Page Views: A page view is counted every time a page is loaded in a user’s browser.
Visits: A visit is a sequence of consecutive page views without a 30 minute break.
Visitor: A visitor is a user performing a sequence of consecutive page views without a 30
minute break.
Return Visitor: A return visitor is a visitor visiting a website more than once.
Visitors
During the time frame of this report, 3,496 unique visitors came to the site. As can be seen
in the graph above, the largest increase in usage came during the summer months of 2009.
The height of the peak is seen in August. This peak coincides with the outreach efforts
made during the MERLOT International Conference, which was held in August. Usage
decreased towards the end of the year, but the trend appears to be reversing during March
2010.
The following chart lays out other important metrics to consider when examining usage.
Google Analytics allows for analysis in more detail, for example week-by-week or month by
month. For the purposes of this report however, the long-term approach was selected.
Total
visits
6,640
Unique Visitors
3,496
Page views
20,314
Average Page
views
3.1
Time on Site
3:23
New Visits
52%
User Loyalty - Return Visitors
User Loyalty is measured by return visitors to the site. Over this period returning visitors
are almost on par with new visitors, with return visitors making up about 48% or the visits.
31
The upward trending seen in March of 2010 seems to indicate a slight shift in that more
new visitors (57%) having viewed the site in comparison to returning visitors.
New visitors
Returning visitors
Total visits
3,479
3,161
Pages/Visit
2.7
3.5
Avg. Time on Site
2.36
4.14
The majority of users (52%) visited the site once, almost 25% have visited it between two
and nine times. Then, as usage becomes quite high, that is people who visit anywhere
between 10 and over a 100 times, the usage begins to rise again indicating that frequent
users go back to the site quite regularly. The table below illustrates this usage pattern.
Number of visits
1
2
3
4
5
6
7
8
9 - 14
15 -25
26 – 50
51 – 100
101 – 200
201 +
Visitor’s nth visit
3,479
645
338
217
159
118
80
72
259
286
302
211
176
288
% of all visits
52.39
9.71
5.09
3.27
2.39
1.78
1.36
1.08
3.90
4.31
4.55
3.18
2.65
4.34
Interestingly, the depth of visit indicates that visitors for the most part go no deeper than
eight pages in to the site. The rate of people viewing one page and leaving is consistent with
other usage statistics in that about 53% of the visitors viewed only one page.
Page views in the visit
1
2
3
4
5
6
7
8
9 - 14
15 -19
20 +
Visits with this many
page views
3,533
1,144
587
375
231
167
121
77
248
62
95
% of all visits
53.29
17.23
8.84
5.66
3.48
2.52
1.82
1.16
3.74
0.94
1.43
Analysis: Visitor Profiles
The geographic location of visitors tends to represent states with large enrollments, with
California of courses, leading the way. Note that ELIXR Fellows were located in those states
32
in bold type. Analysis note: time on site is double edged – on one hand it might indicate a
user has spent a portion of time in one area, it may also mean the that person had to look
around for what he or she wished to access. Therefore, short site visits might indicate that
the user was able to access, find and view what they wished to look at very quickly. The
following table lists the top 10 states with the most visits.
California
Texas
Missouri
Pennsylvania
New York
Virginia
Indiana
Ohio
Connecticut
Colorado
Visits
2,657
540
239
215
167
165
135
135
108
107
Pages/Visit
3.74
2.82
2.87
2.06
2.56
3.24
2.15
2.65
2.39
2.15
Avg. Time on Site
4:20
3:22
3:11
2:00
3:03
4:59
1:56
1:46
2:38
2:17
Referrals: How Users Discover ELIXR
Users find ELIXR mainly by knowing about the project. While visits came to the site via 164
sources, 40% of the visits came from going directly to the URL. Google was the second way
visitors found the site (20%), with pachyderm.cdl.edu directing about 9% of the traffic to
the site. This indicates that the site has not been promoted heavily at this stage in its life
cycle. It is encouraging however, that users can find the site easily via google.
Most Popular ELIXR Case Stories
The most popular case stories are judged so based on the number of unique pageviews. The
top 5 case stories are listed in the table below. Average time spent on the page is also listed.
For all case stories, the number of unique pageviews was 8,,849 and the average time spent
on page was 3:50.
Case Story Page
st
1 Day of Class
Geoscience – JIT
Active learning - Chemistry
Faculty Learning Communities
Integrative-learning - Global Climate Change
Unique Page views
2,127
828
630
402
448
Avg. Time on Page
3:50
3:50
4:26
3:17
2:50
The bottom 5 case stories are listed in the following table. Differences in popularity are in
part driven by when the case stories were finished and posted to the site. Lower numbers
may in fact only indicate that the stories are newer. Of interest however, is the length of
time users stayed on the pages. The average time spent by all visitors was 3 minutes and 50
seconds. Clearly, given the time spent on all of the pages, regardless of their popularity, the
user is engaged in what is being presented.
33
Case Story Page
Student Engagement – Electronic Communication
Audience Response System – Medical School
Student Engagement
Audience Response System – Student Readiness
Audience Response System – Theme module
Unique Page views
9
5
8
4
4
Avg. Time on Page
2:59
3:03
7:13
5:39
4:50
Conclusions and Limitations
This report gives an overview of the use of the www.elixr.merlot.org website. These data
may be used in the future for determining use trends. Given that the report covers the
developmental phase of the project, care must however, be used in making comparisons.
Critical events can make a difference in interpreting the data. For example, the website was
revised significantly in February 2010. The changes made were made in part in response to
the feedback obtained in various user studies. Timing is also an important factor to
consider in viewing the statistics associated with the case stories. Use of the case stories
will vary significantly based on when they were posted to the site. Currently, the usage
statistics as skewed towards higher use of those cases that have been posted for a longer
period of time.
The data clearly show that visits to the site increases as a result of outreach efforts. The
‘bump’ seen in August 2009 is clearly related to efforts made during the MERLOT
International Conference. An effect (albeit smaller) is also observed during the October
period when ELIXR presentations were made at the annual POD conference. Usage
increases are also observed geographically, with increases observed from both Indiana and
Texas; these are most likely the result of the site being used for the professional
development efforts of the ELIXR Fellows located in those states.
Time spent on the site and page-views are metrics of particular interest. It is encouraging
to note that the time spent on newly posted cases is similar to that of cases posted earlier in
the lifespan of the project. That the average time spent on the site is over two minutes
suggests that viewers at minimum are watching the introductory video for a case before
moving on in the site. This speaks to the value of the introduction being a good ‘hook’.
The ration of new visitors to returning visitors is almost equal. This suggests that the site
has good ‘sticking’ power; meaning that viewers returned to the site at least one more time
over the duration of the report. This is a trend that should be watched carefully. It may be
the result of the type of outreach conducted (e.g., hands on experiences vs. one shot
presentations or demonstrations) or that most users have been introduced to the site at
faculty development workshops.
34
Appendix D ELIXR Fellows Evaluation Study
35
BbK
Education
Report of the Use of ELIXR Case Stories in Faculty
Development Programs
[Author Name]
36
Introduction
Overview
ELIXR case stories were designed in part to be used as curriculum materials for use in
faculty development programs. Case stories were designed for use by any faculty
development professional who wishes to use them. The cases are accessible from the ELIXR
website; also accessible are evaluation instruments that can be used to evaluate them. It is
impossible to determine if, when or how faculty developers use the materials because the
materials are freely available via the site and no registration is required. ELIXR established
the ELIXR Fellows program to support faculty developers in the use of case stories in
faculty development workshops at a variety of campuses to test the case stories with
regards to usability, and effect on workshop participants.1 As Fellows, they agreed to use a
case story in at least one professional development activity, to evaluate it and report out on
the findings of the evaluation.
Four Fellows participated in the program: Roberta Ambrosino (University of Texas Health
Science Center, San Antonio) Joseph Grimes (California State University – San Luis Obispo)
Kathryn Ross (Indiana University, Kokomo) and Kiren Dosanjh Zucker (California State
University – Riverside). Each Fellow selected a case story to use in a workshop or seminar
setting during the later half of 2009. Each workshop setting and curriculum consisted of a
unique implementation, though two of the Fellows used the same case story. The case
stories tested included: The 1st Day of Class, Universal Design for Learning (UDL) and Just
In Time Teaching (JITT).
Methods
The first part of this report summarizes the activities of each Fellow and reports on the
results of their evaluation efforts. Each Fellow conducted a workshop during 2009 using a
case story. When possible, prior to the workshop they administered an online pre-survey to
determine the participants’ awareness of the workshop topic. At the completion of the
workshop, attendees completed an online workshop evaluation that asked about their
impressions of the workshop, their satisfaction with the workshop and beliefs about what
they had learned and how they might apply it in their teaching. Several fellows also
administered an online post survey to determine how the workshop impacted the teaching
practices of the participants.
Each of the Fellows’ workshops were designed and implemented to meet the unique needs
of their faculty; they also used different evaluation methods and instruments. The reports
on these efforts (see the following section of this report) constitute a set of case studies on
the use of the case stories. The survey instruments used by the Fellows were centrally
Several other faculty developers who were not Fellows have used and evaluated the use of case stories. The
results of their use is reported on in a forthcoming chapter of To Improve the Academy, the pre-eminent
research publication of the POD Network – Souza, T.J., Carey, T., McMartin, F., Ambrosino, R., & Grimes, J. (in
press). Using multimedia case stories of exemplary teaching for faculty development. In L.B. Nilson & J.E.
Miller (Eds.), To Improve the Academy: Resources for Faculty, Instructional, and Organizational Development,
29. San Francisco, CA: Jossey-Bass.
1
37
designed to ensure focus on the areas of interest. The goal of providing centralized
instruments was to relieve faculty developers from the burden of creating their own
evaluation tools if they did not have in-house evaluation resources or expertise. The pre
and post instruments were designed with a set of questions common to all users, but
included a section modifiable to reflect the workshops’ intended outcomes. The
instruments were also designed to provide the user with formative evaluation information
(e.g., how to improve workshops) as well as to provide more summative information (e.g.,
impact on teaching). (Examples of the instruments are freely available at the ELIXR website
(www.elixr.merlot.org for anyone wishing to use them.) For the purposes of the Fellows’
program, the survey instruments were made available to them via Survey Monkey. Each
Fellow was given a unique survey URL for their campus so that they could administer the
instruments; results were returned to them on their request. Survey instruments are
included in Appendix A of this report.
In the second part of this report focuses on interviews with the Fellows, which were
conducted in February of 2010. The purpose of the interviews was to learn more about
their perceptions regarding the use of the case stories in their faculty development efforts,
the factors important to adoption and use of case stories and recommendations for
dissemination. See Appendix B for the interview protocol.
Evaluation – Survey Results
University of Texas Health Science Center, San Antonio
The University of Texas Health Science Center used the 1st Day of Class case story in two
face-to-face workshops with a total of 30 participants from various disciplines, different
teaching environments, and varying years of experience. Seasoned faculty facilitated the
workshops. Case stories focused on framing course content and motivating students and
were used by the facilitators as ‘jumping off points’ to launch discussions and reflections
based on their own experiences.
Workshop Evaluation
Participants were invited to complete an evaluation of the workshop after completing it.
The workshop evaluation (See Appendix A) focuses views and opinions regarding the
effectiveness of the workshop with regards to learning new ideas and pedagogies related to
teaching as well as the effectiveness of different aspects of the workshops including the
case stories.
The participants felt that the workshop contributed to their knowledge about the topic.
Prior to attending the workshop, almost 75% indicated that they were not expert on the
topic with only 20% indicating they knew a great deal about the topic. At the end of the
workshop 57% of the participants indicated that the workshop had contributed a great
deal to what they now know. With regards to effectiveness of various aspects of the
workshop, the top rated (based on a 5 point scale where 1 = poor and 5 = excellent) aspects
were: presentations by workshop leaders (mean rating: 4.43), discussions with peers
(mean rating: 4.41) and the case stories (mean rating 4.00). Sixty percent of the
38
participants rated the workshop as valuable while 30% found it extremely valuable and
most of the participants (90%) were satisfied or extremely satisfied with it.
While the open-ended questions did not focus specifically on the case stories, many of the
respondents used language associated with the case stories, indicating that they had had an
impact on learning. When asked about what worked especially well in the workshop or
how to improve the workshop, several participants commented that they enjoyed the
examples from the case story videos and would have appreciated seeing more. The
workshop organizer reported that the workshop facilitators felt that the story format of the
ELIXR resources promoted a story-sharing atmosphere, which led to the participants to
openly identify and compare their differences in disciplines and teaching environments.
Knowledge About the Topic
Prior to the workshop, 12 of the participants completed the 1st Day of Class pre-workshop
survey (See Appendix A). The purpose of this survey was to provide the workshop
facilitators with insight into the participants’ previous experience with the topic, as well as
sense of why they were attending the workshop. After the workshop, participants were
again invited to complete a similar survey in order to gain insight into the impact of the
workshop on teaching behavior, as well as to identify barriers to making such changes.
Seven participants completed the follow-up post survey. The respondents who completed
both were similar with regards to teaching experience; 58% of those completing the pre
survey had over seven years teaching experience while 57% of those completing the post
survey had over seven years of teaching experience.
In the pre-survey, respondents were asked to rate the importance of a series of 1st day of
class activities. In the post survey, respondents were asked to indicate if they had included
any of these activities in their first day of class or were planning to do so in future classes.
The table below summarizes the top box score (that is those who strongly agreed with the
activity) for the pre survey, with the responses from the post survey.
39
a. Motivate students to encourage their interest in the course.
b. Communicate my academic expectations regarding for
example, academic honesty, rigor, critical thinking, etc.
c. Communicate my social expectations regarding for example,
attendance, participation in class discussions, respect for
one another, etc
d. Determine the students‟ prior experience or understanding
of course related concepts.
e. Get acquainted with the students.
f. Share my philosophy of teaching.
g. Explain course requirements, including: course materials,
technology requirements, etc.
h. Do something attention getting to pique students‟ interest in
the course.
i. Share learning outcomes for the course.
j. Provide a course syllabus (or refer to online version at
course Website.)
k. Provide a course timeline or outline (or refer to online
version at course Website.)
l. Frame the course in relation to other requirements in a
major, or to a general education.
m. Model my own expectations, e.g., start and finish on time.
n. Introduce course tools, e.g., learning management system,
computer courseware specific to course, web resources,
etc.
o. Create a comfortable learning environment.
% Strongly
Agreed
% Who
included
activity in 1st
Class
% Reporting Very or
Extremely Likely to
Include in Future Class
73
100
100
80
100
100
80
71
71
53
57
43
27
33
57
86
86
71
80
86
71
67
57
86
67
100
86
73
100
100
67
86
86
53
71
57
73
86
86
60
71
86
67
86
86
These responses indicate that a number of the participants put into practice some activities
identified in the workshop as important to accomplish the first day of class. The strong
agreement and the high reports of doing an activity may indicate that these are the
activities that participants have done in the past and will continue to do so. Those areas
where the level of agreement regarding importance differs with the level of reported
activity may indicate a behavioral change. For example, respondents rated getting
acquainted with students (27%) and sharing their philosophy of teaching (33%) relatively
low in importance as compared to the other activities. However, their reported behavior
and anticipated behavior suggest that their attitude may have changed as a result of the
workshop resulting in adding that activity to their first day of class. Less dramatic changes
are also evident, for example, modeling expectations and framing the course in relation to
other requirements.
The data however, must be interpreted with care, as this version of the survey did not ask
participants to report if they already did this activity during a first day of class. It is also
interesting to note the variation between integration of the activity into the participants’
course in the semester following the workshop and the plans to do so in future terms. For
some activities, participants’ responses dip, suggesting that they will wait to see the impact
of including these activities in the current term. Alternatively, the differing scores may
reflect that some of the participants were not teaching in the current term, but will be
doing so in the next term.
40
In order to determine the impact of the case stories on the participants, if any, they were
also asked in the post survey to identify activity(ies) during the workshop that they found
particularly informative. Most respondents discussed a particular activity to include in a
course, though several comments noted that the stories from other faculty were
particularly useful. However, because the comments were not specific in nature, it is not
possible to determine if they were referring to the case stories or stories told by other
faculty members during the workshop.
Indiana University, Kokomo
Workshop Evaluation
The case story, The 1st Day of Class was used multiple times at Indiana University, Kokomo,
both in a face-to face-workshop and in an online setting. Two workshops were run in the
online version. For the purposes of this report, the data regarding the online versions of the
workshop are of most interest. Each online workshop spanned a four-week period and
enrollment was limited to seven participants. In each offering, four participants completed
the session. Seven of the participants completed the workshop evaluation.
The participants varied in terms of teaching experience with 30% of those starting having
taught between three and six years; the remaining having taught for over seven years. Two
of the participants were adjunct or visiting professors, four were assistant professors and
the remaining were evenly split between full or associate professors. The ‘home’ discipline
of the attendees also varied with participants from the Health Sciences, Science and
Technology, Social Sciences and Professional schools represented. (See Appendix A for the
workshop evaluation instrument.) The majority of the participants (71%) reported that
they knew “some” about the topic and 100% reported that the workshop contributed to
their knowledge of the topic. The mean ratings regarding the extent to which certain
aspects of the workshop contributed to their learning were similar: Discussions with Peers
mean rating 4.3 (1 = poor; 5 = excellent) Presentations by Workshop Leaders’ mean rating
4.2 and the mean rating for Video Case Stories was 4.1.
When asked what worked especially for them, well one participant commented: “Video
clips for illustrating points”, and when asked how the workshop might be improved several
participants recommended including more video clips from other disciplines.
In this version of the 1st Day of Class pre-survey, a question was added asking participants
if they already use the activity listed during a first day of class. They were also asked to rate
the importance of the activity. In the post-test, participants reported if they implemented
the activity into their teaching in the coming semester, and to rate the likelihood that they
will continue to conduct the activity in the future. The results are listed in the following
table. Seven participants completed the pre-survey and four completed the post survey.
The responses indicate that the participants changed their behavior, with a number of
them adding new activities for the first day of class. (Given the small number of
respondents care must be taken to not over interpret the responses or generalize to a
larger population.) The responses point to a possible upward trend that once introduced to
41
the activity and why it is important to do it, the participants included it. For example, none
of the participants reported that they framed the course in relation to other requirements
in a major or to general education and only 14% felt that this was important. A quarter of
the participants reported that they had included this in the subsequent term and would do
so in the future. In one case however, there is a downward trend related to determining
students’ prior experience or understanding of course related concepts. In this case, 71%
reported that they already did this, while on 14% feeling it was important to do in the first
class. Unlike the other activities listed, it appears that after attending the workshop,
participants are less like to undertake this activity in the future.
% Strongly
Agreeing
with
Important
to 1st Class
% Who
included
Activity in
1st Class
After the
Workshop
% Reporting Very or
Extremely Likely to
Include in Future
Class
71
57
75
75
71
71
75
75
86
86
100
100
71
14
25
25
86
43
71
59
100
50
75
50
86
71
100
100
43
43
75
75
57
43
75
75
86
86
75
75
86
57
75
50
0
14
25
25
57
43
75
75
71
57
50
50
71
57
100
100
% Who
Currently
include
Activity in
Class
a.
Motivate students to encourage their interest in the
course.
b. Communicate my academic expectations
regarding for example, academic honesty, rigor,
critical thinking, etc.
c. Communicate my social expectations regarding for
example, attendance, participation in class
discussions, respect for one another, etc
d. Determine the students‟ prior experience or
understanding of course related concepts.
g. Explain course requirements, including: course
materials, technology requirements, etc.
h. Do something attention getting to pique students‟
interest in the course.
j. Provide a course syllabus (or refer to online
k. Provide a course timeline or outline (or refer to
online version at course Website.)
l. Frame the course in relation to other requirements
in a major, or to a general education.
m. Model my own expectations, e.g., start and finish
on time.
n. Introduce course tools, e.g., learning management
system, computer courseware specific to course,
web resources, etc.
Few of the participants made comments regarding barriers to implementing the activities
listed in the table above; several noted which specific activities they did try and that
students responded positively to them, e.g. “they are willing to spend more time on my
course” and “I handed out numbered puzzle pieces which placed the students into 9
groups. They then had 2 group activities. The students were having more interactions with
each other than in previous courses. The students asked more specific questions regarding
the course assignments after I reviewed them.”
42
California State University – Riverside).
The case story used in faculty development at CSU – Riverside was Just in Time Teaching
(JITT). This workshop was held face-to-face. The timing of this workshop was difficult as it
was conducted late in the fall quarter, just two weeks prior to finals. Eight faculty members
attended; five completed the pre-survey and four completed the post survey. In this
instance, only the JITT pre and post surveys were administered; the workshop evaluation
form was not used.
The pre-survey created for this workshop asked participants to describe to what level they
included the set of class activities associated with JITT in their teaching. The scale was: Not
Implemented, Partially or Occasionally Implemented or Fully and Consistently
Implemented. The following table shows the changes associated between reported
behavior prior to the workshop and very shortly after the workshop. Interestingly, the
ratings for none of the activities received higher than 50% rating in terms of importance to
include in classes. The follow-up low ratings for inclusion of the activities is most likely due
to the timing of workshop leaving little or no time for the participants to revise their
existing courses. The likelihood of including these activities in a future course suggests that
the participants are interested and willing to include more JITT activities.
% Strongly
Agreeing
with
Important
to 1st Class
% Who
included
Activity in
1st Class
After the
Workshop
% Reporting Very or
Extremely Likely to
Include in Future
Class
13
25
40
100
38
25
60
80
0
50
100
100
13
13
60
60
13
28
80
100
% Who did
not include
Activity in
Class
a.
b.
c.
d.
e.
Use in-class exercises such as in-class surveys,
“warm up” questions, etc.
Use exercises to gauge students‟ understanding of
course materials prior to a class session or
sessions.
Engage students as active participants in
constructing knowledge.
Address students‟ misconceptions of the course
materials in class.
Use interactive in-class exercises (small group
discussions, reflective exercises, etc.)
California State University – San Luis Obispo
The CSU San Luis Obispo workshop used the Universal Design for Learning in Information
Systems (UDL) case story as part of a UDL Faculty Learning Community. This group met
five times. Over the course of this time frame the participants reviewed and discussed
materials regarding the fundamentals of UDL and planned how to transform a course
following UDL principles. The participants viewed the UDL case story at the end of the
project so as to not influence their course designs. The case story focused on the value of
UDL, illustrated a variety of UDL techniques (e.g., small group discussions, guided notes,
use of online multimedia) and included sample assessment techniques. Viewing the case
after the course redesign provided the participants with an opportunity to check to see if
their redesign met UDL standards.
43
In this evaluation, the standard CSU San Luis Obispo faculty development workshop
evaluation form was used and no pre or post survey was conducted. The Fellows also
collected informally, verbal feedback from the participants on a regular basis on what
worked about the FLC and what needed improvement. Because of privacy restrictions, the
actual results of the evaluation cannot be shared. However, the Fellow was able to share
the evaluation findings, they are: FLC participants felt the two main benefits of case story
use were that seeing UDL happen was better than just talking about or reading about how
UDL should happen and that interdisciplinary discussions of the case stories allowed
different insights to emerge. All FLC participants felt the case story was valuable for them
and all followed through on the FLC by implementing UDL in revised courses.
Evaluation – Summary in Findings from Interviews
In February of 2010, informal, one-on-one telephone interviews were conducted with each
ELIXR Fellow. The purpose of the interview was to learn from them what they liked about
using case stories in faculty development situations, the challenges associated with their
use, what recommendations they had for developers of the case stories, and how the ELIXR
project might change or improve dissemination of the cases.
The following section summarizes the interview responses; the interview protocol can be
found in Appendix B.
What works
Respondents commented on all of the different ways they employed the case stories in
their faculty development activities. (Each of the Fellows has used a number of cases in
different situations; their comments are not restricted to the activities that were a part of
the ELIXR Fellows evaluation effort.) The ways they were used included:
Face to face workshops
Demonstration of a particular concept followed by short discussions with participants.
URLS to cases given to faculty members to review prior to the workshop session
Review of a full case during the workshop
Review of a case using the suggested guidelines to evaluate course re-design
Integrate viewing case stories into ‘think-pair-share” activity
Alternative to lecturing on a topic
Other settings
URL given to workshop facilitators as a starting point from which to structure the
workshop
One on one consultations with faculty
Two of the Fellows were also developers of case stories, while two were not. All however,
gave examples and illustrations of how they were using cases outside of a formal
workshop, seminar or faculty learning community setting. The informal use, e.g, working
with faculty one-on-one, was one example where the Fellows felt the cases were extremely
effective and easy to use.
44
Improving Case Stories – Recommendations for Developers
The Fellows agreed that the main improvement that would benefit the case stories was
creating or supporting a mechanism to help someone find and cue up a particular part of a
video clip within the case study environment. Currently access to the video is only via the
Web within the Pachyderm environment. This makes it relatively difficult to manage for
several reasons: download time is long and it is difficult to edit (cue up, mainly) a video to
show a specific example. Some of the Fellows have access to the actual video clips in house
(because they were involved in making the case) so they tend to use that video instead of
that which is available online.
Also mentioned was the need for more resources linked to the cases. Several of the Fellows
discussed how they appreciated the resources (sample agendas, activities and tools) that
were attached to the 1st Day of Class case story. They wished that all the cases had these
types of materials available.
Barriers to Using Case Stories in Faculty Development
The barriers discussed by the Fellows tended to vary by the needs associated with their
campuses. For example, one Fellow reflected on how the faculty at her campus wanted to
be passive learners and the cases allowed for that. The challenge in her case was to use the
videos in a way to engage the participants interactively. Others commented on the
problems associated with technology, e.g., browser speed, no broadband connection, etc.
(this was especially problematic when having participants view case stories from their
homes in online workshops or as homework for a face to face workshop.) When using the
case stories in face-to-face workshops, Fellows discussed problems associated with
projecting them, e.g., small font size) as well as issues associated with lack of editing ability
because of the online environment and nature of the cases.
Impressions of the ELIXR Website
The Fellows all rated the value of the ELIXR website (www.elixr.merlot.org) as extremely
valuable, and most were very satisfied with it. Even though rating it highly, they had
suggestions, including:
Annotate the list of cases to improve ‘findability’ of the cases
Reduce the text – too text heavy
Bog is useful – reading the posts really helpful, but takes a lot of time.
Improve search function.
Encouraging Use by Faculty Developers
Suggestions included:
Get linked to POD website.
Make more visible on MERLOT.
Include MERLOT-like reviews with the cases.
Show more about how to make a case story.
Do more presentations and workshops on the topic.
45
Offer online support materials for workshops, e.g., agenda, exercises, announcements,
etc.
Other Comments
Fellows concluded with several observations about their participation. They noted an
appreciation for participating in the Fellows program because it enabled them to try out
and test a new way of conducting faculty development on their campuses. They noted that
the conversation and sharing among the Fellows via conference calls and the website blog
was useful to them. Several noted that because of the fellowship, they had the opportunity
to present at national and regional conferences about their work. The presentations had
resulted in a number of people following up with them on how to use the cases on their
own campuses or on how to make a case story on their campuses.
Conclusions
The purpose of this evaluation was to identify ‘best practices’ in terms of use of case stories
in faculty development situations, determine impact of faculty members’ teaching practices
as a result of participation and to identify potential barriers or challenges in terms of
disseminating cases to faculty developers.
The ELIXR Fellows program was an effective means for studying the use of case stories ‘in
situ.’ Two of the Fellows had also been developers of case stories; neither of these
evaluated the cases they worked. Two had not been involved in this aspect of the project.
Cross campus comparisons cannot be made with regards to use of case because no single
workshop model was followed by the Fellows, nor was only once case used. Fellows did use
similar instruments to evaluated the effectiveness of the workshops (the San Luis Obispo
instrument covered topics similar to the ELIXR evaluation instrument); the pre and post
survey instruments varied only with regards to measurement of the workshop outcomes.
The pre-survey instrument was refined as a result of this testing adding a question to
determine the extent to which a technique was used prior to attending the workshop.
Best Practices
The results from each individual campus in conjunction to the results from the interviews
with the Fellows indicate that use of the cases was successful in the following faculty
development workshop models: face-to-face workshops facilitated by expert faculty
members, workshops conducted online, and faculty learning communities. The interviews
(along with informal communication with the external evaluator when refining the
evaluation instruments to meet the individual Fellow’s needs) suggest that the use of the
case stories can be quite time consuming for faculty developers because they must learn
how to use the case story as well as the technology that supports the cases.
The Fellows were highly motivated2 and so may have been more likely to spend the time
necessary to integrate the cases fully into their work than novices might be. However, they
2
Fellows were paid a $500.00 stipend to participate, received technical and evaluation support and were
encouraged to publish their findings.
46
also integrated the case stories in ways in which they were comfortable, which is consistent
with how people tend to integrate technology into personal practice. They used a case to
illustrate a point, they used a case to jump start discussion or to precede interactive
exercises. When in a group setting the Fellows tended to use only the video portions of the
case stories and they used them to introduce a topic to the participants. Only in the online
workshop did participants view entire cases, which was consistent with the workshop
design.
Impact on Teaching Practices
The results from the pre and post survey indicate that the faculty who participated in the
workshops learned from them and did make initial attempts to change their practice.
Faculty development activities by their nature tend to attract relatively small numbers of
participants3 so generalizing the results from this study alone is not possible. However, this
study does indicate that in the short term, some change may have occurred. Participants
self reported that they had or were in the process of trying a particular technique learned
during a workshop in their teaching practice. For example, as a result of attending the 1st
Day of Class workshop, it appears that more participants (regardless of which workshop
they attended) will now: share their philosophy of teaching, get acquainted with the
students, share learning outcomes and frame the course in relation to other requirements
in a major or general education; all aspects that were covered in the case stories and
reinforced in the case story resource materials.
It is difficult to separate out the impact of the case stories on learning from the impact of
the rest of the workshop curriculum. However, in the workshop evaluations, participants
rated the case stories highly and being comparable to Discussions with Peers and
Presentations by Presenters. These ratings were all higher than other aspects of the
workshops such as handouts. Unprompted, several participants reported that they liked
the video parts of the workshops (this is generally taken to mean the case stories since no
other video resources were used.) and several wished that there were more video
examples to view.
Barriers and Challenges to Dissemination
Barriers to use and challenges to dissemination tended to fall into three categories, those
that related to:
individual use of a case story by a faculty developer
challenges associated with the technical environment
access to the case stories.
Individual use by a faculty developer: Based on the experience of the ELIXR Fellows,
effective use by a faculty developer may require significant time and learning on the part of
that professional. Use requires that in the case of face-to-face workshops, faculty
developers: become very familiar with an entire case story to determine the aspects of it
they wish to use, learn how to cue up the video portion effectively, determine how to use it
3
The faculty development sessions reported on in this report had no more than 15 participants and in the
online case, participation was limited to seven per session.
47
in existing workshops (if appropriate) or develop a new workshop entirely. Online
workshops may require similar learning on the part of the faculty developer, but because of
the format, it might be easier for participants to view an entire case, eliminated some of the
technical learning on the part of the faculty developer. (This assumes of course that
conducting an online workshop is not a new activity for the faculty developer.)
Technical environment: The fact that the case story is build in the online software
environment, Pachyderm, and that the cases are only available online may make it difficult
to use in some situations. For example, projecting a case story may be difficult to do since
the text has not been optimized for viewing in this manner while the videos themselves
may not be a problem to view. Because the cases are in The Pachyderm environment (and
web-based access) require navigation. Segments cannot be easily de-coupled making it
difficult to use different sections for different purposes.
Access to case stories: Discoverability of the cases is problematic. To date, the ELIXR
website has not been extensively promoted. Outreach activities have only begun in the last
stages of the project. The ELIXR website is the main source for accessing the cases and the
Fellows, even though they are intimate participants in the project, reported difficulties in
finding the resources and learning about them.
Limitations
This study is limited by a number of factors requiring that conclusions be drawn with care.
Viewed within the methodological framework of a case study (the Fellows did not test the
same case story, nor did they use a consistent workshop model), the results and findings
may inform use in similar situations. The span of the study was relatively short (six
months) so it is difficult to determine if participants changed their teaching practices for
the long term. For example, the results suggest that faculty were willing to try a new
practice but were still somewhat ambivalent about using it in future classes. This may be a
result of the timing of the follow up survey. More longitudinal study is necessary to know
the long-term impact on teaching practice. Impact was measured only in terms of selfreports by participants. Triangulation of self reports with observations or analysis of
course materials would be necessary to confirm actual changes to teaching; unfortunately,
this was beyond the scope of this study.
Recommendations
Access to Case Stories/Dissemination
Faculty developers (and individual faculty members) must become aware of the case
stories in order to use them. ELIXR’s focus at the time of this study has been on the
development and refinement of both the cases and the ELIXR website. The participants in
this study expressed excitement and enthusiasm in the use of the cases for faculty
development purposes. They also expressed that they knew about the cases only because
they were involved in the project.
Recommendations
48
The Fellows, because they are experienced in the use of case stories can be effective in
conducting outreach to their peers. As a part of the Fellowship, they presented their
findings at various conferences (cite) and several co-authored a chapter describing
their research in the forthcoming issue of To Improve the Academy. These types of
activities should be encouraged and supported when and where possible.
Others within the ELIXR project share the characteristics of the Fellows, they too should
be tapped to encourage sharing their case stories and experiences within their
professional networks.
Improve the discoverability of the ELIXR case stories.
o work with MERLOT to link more closely too, or highlight the cases amongst
members of their faculty development community.
o Link ELIXR website more closely to the POD network website and other websites
associated with faculty development.
o Develop a searchable website.
Use in Faculty Development
Use of web-based, multi-media case stories for faculty development is an innovation for the
majority of faculty developers. Like other innovations, faculty developers will require
orientation and training to make the most effective use of these materials.
Recommendations
Develop learning materials specifically for faculty developers on how to use a case story
effectively. Consider creating a case story for faculty developers on how to use case
stories.
Encourage the Theme Development Teams to include resources for use in faculty
development in each case story. The Fellows highly valued the appended materials
found in the Resource section of the 1st Day of Class case story that focused on how to
use the case in faculty development.
Technical Environment
A number of the technical problems experienced by the Fellows are not within the purview
of the project, e.g., bandwidth, however, it is important for the case story designers to
consider these factors when designing the cases.
Recommendations
Consider making videos from the case stories available via YouTube to improve the
flexibility of use in face-to-face situations.
Develop specifications or design requirements for developers of case stories to guide
them in creation of stories that can be more easily accessed.
Review the structure of the cases (technical and environment) to increase their re-use,
for example consider developing re-use guidelines that establish appropriate
granularity for modules within a case story.
49
Appendix A – Evaluation Instruments
Note: All surveys were conducted using the online Flashlight survey system developed by
the TLT Group and Washington State University. The forms reproduced below do not
reflect the actual layout of the instrument
ELIXR Workshop Evaluation Form
Survey Consent
The following questionnaire is a part of our efforts to evaluate the program in which you have recently
participated. This program is a partner with the ELIXR research program, which is funded by the
Department of Education‟s Fund for the Improvement of Post-Secondary Education (FIPSE). The ELIXR
program goal is to improve the effectiveness of faculty development programs through application of
digital stories of exemplary teaching and the adoption of innovative teaching practices.
All of the information you provide us will be anonymous and our use of the data for research purposes will
be limited to aggregated results. Our Institutional Research Board has approved the collection of these
data for research purposes.
When you provide us with information on the Evaluation Forms, your responses will be included in this
research; you may, of course, choose to not complete the Evaluation Forms and thus be removed from
any applications of the resulting data for research purposes.
Please indicate your willingness to participate in this study below. Selecting 'yes' means you will allow
your responses to be used in the evaluation of this program and that you understand that your responses
will be recorded but reported only in aggregate with other responses.
 Yes
 No
Workshop Evaluation
Your candid and frank responses to this evaluation will help us learn more about your expectations for
this workshop and your ideas on how it might be improved or strengthened. We will also use the results to
assess the impact of conducting this and similar kinds of workshops in the future.
Workshop Title: ________________________________________________________
Institution/Organization Name: ____________________________________________
50
About You
Faculty Rank
 Full Prof.
 Adjunct or Visiting Prof.
 Assc. Prof.
 Graduate Student
 Asst. Prof.
 Emeritus Prof.
 Lecturer
 Other (please describe)
Discipline _________________________________
Teaching Experience (Please include years as teaching assistant, adjunct faculty member, teaching at
another level of school, e.g., community college, high school, graduate school, etc.)
 No teaching experience
 Over 7 years
 1 – 2 years
 I am not an instructor or teacher
 3 – 6 years
Opinions about the Workshop
Little/None
Some
A Great Deal
1
3
5
To what extend do you currently feel knowledgeable about
the topic of this workshop?



To what extent did this workshop contribute to what you
now know about the topic?



Please rate the effectiveness of the following aspects of the workshop in contributing to your learning.
Little or
None
Some
A Great
Deal
Don’t
Know,
1
2
3
4
5
NA
Presentations by the workshop leaders






Handouts






Video‟s






Video case stories






Discussions with peers






“Homework” prior to workshop






Other (please describe)






51
Please circle the response that best reflects your rating for this workshop as to its overall value to you and
your satisfaction with it.
Value of the Workshop to you
Not at
all
Valuable
1
Valuable
2
3
4
Satisfaction with the Workshop
Extremely
Valuable
Not at
all
Satisfied
5
1
Extremely
Satisfied
Satisfied
2
3
4
5
Comments about your ratings for value and/or satisfaction:
What are the two or three most important things that you learned during this workshop?
As a result of attending this workshop, what one or two methods, ideas, techniques, knowledge, etc. do
you foresee applying to your teaching?
What if anything, do you anticipate might get in the way of you applying your learning from this workshop
to your teaching?
What worked especially well in the workshop?
52
If offered again, how might the workshop be improved?
Other comments you wish to make regarding the workshop.
53
ELIXR 1st Day of Class Pre-Survey Form
st
This quick survey will help us tailor the 1 Day of Class workshop to meet your needs. It will help us
determine what the participants‟ know about and wish to learn more about. We will also use the results to
assess the impact of conducting this and similar kinds of workshops in the future.
Survey Consent
 Yes
 No
About You
Faculty Rank
 Full Prof.
 Assc. Prof.
 Asst. Prof.
 Emeritus Prof.
 Lecturer
Discipline _________________________________
 Over 7 years
 1 – 2 years
 3 – 6 years
Institution Name _________________________________
54
About the Workshop
Please tell us about how you currently run your typical first class meeting and then rate the following
items in terms of importance to the first class meeting.
During the first
class meeting, I
Importance in a first class meeting
Strongly
Agree
Tend to
Agree
Neutral
Tend to
Disagree
Strongly
Disagree
No
Yes
NA
5
4
3
2
1
















































g. Explain course requirements,
including: course materials,
technology requirements, etc.
h. Do something attention getting to
pique students‟ interest in the course.
i. Share learning outcomes for the
course.
j. Provide a course syllabus (or refer to
online version at course Website.)
k. Provide a course timeline or outline
(or refer to online version at course
Website.)
l. Frame the course in relation to other
requirements in a major, or to a
general education.
m. Model my own expectations, e.g.,
start and finish on time.
n. Introduce course tools, e.g., learning
management system, computer
courseware specific to course, web
resources, etc.
o. Create a comfortable learning
environment.








































































a. Motivate students to encourage their
interest in the course.
b. Communicate my academic
expectations regarding for example,
academic honesty, rigor, critical
thinking, etc.
c. Communicate my social expectations
regarding for example, attendance,
participation in class discussions,
respect for one another, etc
d. Determine the students‟ prior
experience or understanding of
course related concepts.
During this workshop, you will have the opportunity to design a first day of class for one of your courses.
What concerns do you have about implementing this plan?
What will make this workshop a „total success‟ for you?
55
ELIXR Post-Survey – 1st Day of Class
This quick survey will help us better understand the impact of participating in the workshop regarding the
st
1 Day of Class. We will also use the results to assess the impact of conducting this and similar kinds of
workshops in the future. The surveys are anonymous, so please be candid in your responses – your
frankness can only help us improve the workshop.
Survey Consent
 Yes
 No
About You
Faculty Rank
 Full Prof.
 Assc. Prof.
 Asst. Prof.
 Emeritus Prof.
 Lecturer
 Other
Discipline _________________________________
 Over 7 years
 1 – 2 years
 I am not an Instructor or Teacher
 3 – 6 years
56
Implementing Your Plan for the 1st Day of Class
st
With regards to the following goals for the 1 day of class, please indicate if you included an activity or
st
action to accomplish the goal in the 1 class session, then rate the likelihood that you will continue to do
so in future course or courses.
Included in 1st
Class Session
Likelihood of Including in Future Course(s)
No
Yes
NA
Not at
all likely
Likely
Very
Likely
Extremely
Likely
NA
a. Motivate students to encourage their interest
in the course.
b. Communicate my academic expectations
regarding for example, academic honesty,
rigor, critical thinking, etc.
c. Communicate my social expectations
regarding for example, attendance,
participation in class discussions, respect for
one another, etc
d. Determine the students‟ prior experience or
understanding of course related concepts.
















































g. Explain course requirements, including:
course materials, technology requirements,
etc.
h. Do something attention getting to pique
students‟ interest in the course.
























j. Provide a course syllabus (or refer to online
k. Provide a course timeline or outline (or refer
to online version at course Website.)
l. Frame the course in relation to other
requirements in a major, or to a general
education.
m.
Model my own expectations, e.g., start
and finish on time.
n. Introduce course tools, e.g., learning
management system, computer courseware
specific to course, web resources, etc.
















































As a result of including the activity(ies) or action(s) described above, what changes (if any) did you
observe in your students‟ learning, opinions towards the course, motivation, etc. (to the course in
general)?
st
What problems of challenges, if any, did you encounter that made implementing your 1 Day of Class
plan difficult?
st
What activity, resources or experiences from the 1 Day of Class workshop did you find especially helpful
or useful? Why?
57
ELIXR Pre-Survey - JITT
This quick survey will help us tailor the Just in Time Teaching (JITT) workshop to meet your
needs. It will help us determine what the participants‟ know about and wish to learn more about.
We will also use the results to assess the impact of conducting this and similar kinds of
workshops in the future.
Survey Consent
 Yes
 No
About You
Faculty Rank
 Full Prof.
 Assc. Prof.
 Asst. Prof.
 Emeritus Prof.
 Lecturer
Discipline _________________________________
 Over 7 years
 1 – 2 years
 3 – 6 years
58
About the Workshop
Prior to the JITT workshop, we would like to learn your opinions about the importance of different
elements when designing a course. For the elements listed below, please think about a course you are
currently teaching, or will teach in the coming term and for each element, please: 1. Indicate how the
course is currently designed and implemented, and 2. Rate it in terms of importance to JITT.
Current Course
Design or
Implementation
Importance to Course Design
In my courses and classes, I:
No
Yes
NA
Strongly
Agree
Tend to
Agree
Neutral
Tend to
Disagree
Strongly
Disagree
a. Use in-class exercises such as in-class
surveys, “warm-up” questions, etc.
b. Use exercises to gauge students‟
understanding of course materials prior
to a class session or sessions.
c. Engage students as active participants
in constructing knowledge.
d. Address students‟ misconceptions of
the course materials in class.
e. Use interactive in-class exercises
(small group discussions, reflective
exercises, etc.)








































During this workshop, you will have the opportunity to consider how to design one of your courses using
JITT. What concerns, if any, do you have about implementing this plan?
What will make this workshop a „total success‟ for you?
59
ELIZR Post-Survey – JITT
This quick survey helps us better understand the impact of participating in the workshop: Just in Time
Teaching (JITT).
Survey Consent
 Yes
 No
About You
Faculty Rank
 Full Prof.
 Assc. Prof.
 Asst. Prof.
 Emeritus Prof.
 Lecturer
Discipline _________________________________
 Over 7 years
 1 – 2 years
 3 – 6 years
60
Implementing your JITT Course Plan
Please indicate if you included a JITT activity or action in your redesigned course (or in other courses). It
is important to remember that these elements must be considered within the context of your specific
subject area and are not intended to alter or reduce your academic expectations. They are not intended
as a rigid checklist or prescription for instruction but rather serve as a framework for planning and
delivering instruction. As such, there are no „right‟ answers in this question – please respond as candidly
as possible.
Included in
Current Class
Session
Likelihood of Including in a Future Course(s)
Somewhat
Likely
Very
Likely
Extremel
y Likely
Don’t’
Know,
NA
In my courses and classes, I:
No
Yes
NA
Not at all
Likely
a. Use in-class exercises such as in-class
surveys, “warm-up” questions, etc.
b. Use exercises to gauge students‟
understanding of course materials prior
to a class session or sessions.
c. Engage students as active participants
in constructing knowledge.
d. Address students‟ misconceptions of
the course materials in class.
e. Use interactive in-class exercises
(small group discussions, reflective
exercises, etc.)








































As a result of including the activity(ies) or action(s) described above, what changes (if any) did you
observe in your students‟ learning, opinions towards the course, motivation, etc. (to the course in
general)?
What problems of challenges, if any, did you encounter that made implementing JITT strategies difficult?
What activity, resources or experience from the workshop on JITT did you find especially helpful or
useful? Why?
61
Appendix B – Interview Protocol – ELIXR Fellows
Fellow ________________________________________
Date of Interview ________________________________
1.
In your opinion, what worked especially well when using care stories for faculty
development activities?
2.
How might case stories be improved?
3.
What are the barriers, if any to using case stories in faculty development
situations?
4.
What is your impression of the ELIXR website? How might we improve it as a
resource to faculty developers?
5.
Please rate the ELIXR website as to its overall value to you and your satisfaction
with it.
Value of the ELIXR website
Not at all
Valuable
1
Valuable
2
3
4
Satisfaction with the ELIXR website
Extremely
Valuable
Not at all
Satisfied
5
1
Extremely
Satisfied
Satisfied
2
3
4
5
6.
What recommendations to you have about how we might encourage use by other
faculty developers?
7.
Other comments
62
Appendix E: ELIXR User Study
63
BbK
Education
Report on the Value and Usability of ELIXR Case
Stories
[Autho
64
Table of Contents
Executive Summary ............................................................................................................. 67
Study Method .......................................................................................................................................................................... 67
Results ........................................................................................................................................................................................ 67
Organization and Usability.................................................................................................................................................... 67
Content and Video Clips........................................................................................................................................................... 67
Value to Teaching ...................................................................................................................................................................... 67
Value to Others on Their Campus ....................................................................................................................................... 68
Adaption/adoption for Future Use .................................................................................................................................... 68
Recommendations...................................................................................................................................................................... 68
Summary of Responses - Survey .......................................................................................... 69
About the Respondents ....................................................................................................................................................... 69
Survey Results......................................................................................................................................................................... 70
Q1: To what extent has this case story piqued your interest in trying out this teaching method? ...... 71
Q2: How likely are you to return to this case story to help you with your teaching practice?............... 71
Q3: How likely are you to let other colleagues or faculty members know about these case stories as a
resource to help them with their teaching? ................................................................................................................... 71
Q4: As a result of examining this case story how likely are you to integrate this teaching practice in
a future course or class?.......................................................................................................................................................... 71
Q5: How much did the case story contribute to what you now know about the case story topic? ...... 72
Results from Open-Ended Questions ............................................................................................................................ 72
Q6: Why did you select this case story for review? ..................................................................................................... 72
Q7: How might the case story be improved?.................................................................................................................. 73
Q8: Other comments you may have about case stories or this survey. .............................................................. 76
Summary of Responses -Interviews ..................................................................................... 78
I. About the Respondents .................................................................................................................................................. 78
II. Case Selection and Review ........................................................................................................................................... 78
Reason for Selecting Case Story for Review ................................................................................................................... 79
Relationship Between Prior Knowledge, Time Spent Reviewing Case Story and Reason for Stopping
............................................................................................................................................................................................................. 79
III. Value of the Case Story to Teaching ........................................................................................................................... 79
Value of Case Stories to Other College Level Instructors ......................................................................................... 80
Value of Case Stories to Other Colleagues, e.g. Faculty Developers .................................................................... 80
IV. Quality of the Content, Organization and Usability of the Case Story ....................................................... 80
Organization of Case Story .................................................................................................................................................... 80
Presentation Style ...................................................................................................................................................................... 80
Ease of Use ..................................................................................................................................................................................... 80
V. Quality of Resource Materials ..................................................................................................................................... 80
Adequacy of Details Supplied on Implementation ...................................................................................................... 81
VI. Dissemination, Impact or Potential Use of Case Story ..................................................................................... 81
Effectiveness of Case Story in Raising Interest in the Topic ................................................................................... 81
Use of Story for Personal Professional Development ................................................................................................. 81
Dissemination of Case Story .................................................................................................................................................. 81
65
Contribution to Knowledge About the Topic ................................................................................................................. 81
VII. Suggestions for Improvement of Case Stories ................................................................................................... 81
Suggested Improvements to the Cases ............................................................................................................................. 81
Other Comments ......................................................................................................................................................................... 81
Summary of Responses – Follow-up Survey ......................................................................... 83
About the Respondents ....................................................................................................................................................... 83
Survey Results......................................................................................................................................................................... 83
Q 1: While attending the MERLOT International Conference, you viewed an ELIXR case story. Since
the conference, have you revisited that case story? ................................................................................................... 83
Q 2: Since the conference, have you visited www.elixr.merlot.org to view any of the case stories? .... 83
Please explain your response to Q2.................................................................................................................................... 83
Q3: Please indicate how you used any of the case stories, if at all. (Select all that apply.) ...................... 84
Q4: Did you recommend the case stories or the ELIXR site to your colleague(s) other instructor(s) or
faculty developer(s) on your campus or another campus?..................................................................................... 84
Please describe your response......................................................................................................................................... 84
Q5: Please make other comments about the case stories or ELIXR here.......................................................... 85
Limitations .......................................................................................................................... 85
Appendix A – Instruments ................................................................................................... 86
Survey Instrument ................................................................................................................................................................ 86
Interview Protocol ................................................................................................................................................................ 91
Follow-up Survey................................................................................................................................................................... 92
Appendix F: SERC User Study ............................................................................................... 98
Appendix G: ELIXR CSTLT Study .......................................................................................... 100
66
Executive Summary
Study Method
Attendees of the August 2009 MERLOT International Conference were invited to view a
case story of their choice and respond to a survey or participate in an interview regarding
their impressions of the stories. 42 participants responded to the survey and nine
participants were interviewed. In October 2009 a follow-up survey of these same
participants was conducted. 17 participants responded for a response rate of 36%. Most of
the participants were faculty (49). Several of participants held dual positions, e.g., faculty
developer, administrator, instructional designer. Only a few respondents were not actively
teaching, they tended to be consultants in the fields of faculty development or educational
technology.
Results
Organization and Usability
Most of the participants (over 75%) found the case stories easy to use, clearly presented
and well organized. Some participants had difficulty with the site navigation. This might be
because of the unusual appearance of the navigation buttons, which were different than the
“usual” menu/web page buttons (the current ELIXR case story buttons are screenshot
images rather than icons.) Participants also felt the stories needed more organization,
suggesting that an outline of the whole story might be helpful. While they did not mention
titles specifically, we noticed that titles did make a difference in what people chose to view.
More creative titles seemed to be more appealing.
Content and Video Clips
Overall, participants greatly appreciated the multi-media aspect of the case stories: they
found the videos very compelling. We observed that participants tended to watch each
video to its end and were very engaged when watching. Participants’ suggestions for video
clips: more student classroom shots and less “talking-head” interview shots. They also
want to see more “illustrations or examples”. Most participants stated that they want to see
more details about the content both in video clips and documents.
Value to Teaching
Over 70% of the participants found the case stories valuable either to their or to other
instructors’ and colleagues’ teaching. Around 60% of the participants felt that the case
story piqued their interest in trying out that teaching strategy or integrating it into their
future courses/classes. However only 35% thought that the case story contributed to what
they already know about the case story topic. It should be noted that some of the users
selected a topic that they knew quite well, while others selected topics they knew nothing
about. Experts viewed the stories to ‘test’ the accuracy and validity of the stories, while
novices viewed them to learn more about the topic.
67
The case stories then, for novices may be good in terms of inspiring people but they, may
not add too much to what people already know. Another interpretation might be that the
case stories need to include more details to add more to what people already know. Case
stories might be inspirational, but may need some more detailed information to satisfy the
expert user.
Value to Others on Their Campus
Generally, survey respondents and interviewees felt that the case stories would be of great
value to other faculty or administrators on their campuses. They thought they were good
introductions to topics of interest and that less experienced faculty would find them most
useful. (The participants in this study were all very experienced instructors.) Case studies
need to go beyond an introduction to a topic in order to attract the attention of more
experienced instructors.
Adaption/adoption for Future Use
The findings from the follow up survey suggest that adoption of the case stories for use will
require strong follow-up with potential users. Even though the participants spent a fair
amount of time (usually around 15 – 20 minutes) with the case story they were examining,
that amount of exposure was insufficient to ensure use in a different setting. Respondents
indicated that they felt the materials were good and potentially useful, but once back on
their campuses, they were overwhelmed with other responsibilities. This is not an unusual
finding with regards to adoption of innovative teaching or learning materials.
Recommendations
The usability problems, e.g., lack of recognition of the photo navigation as icons should
be addressed as soon as possible.
Suggestions regarding how content is presented, e.g., hearing more from students or
reducing ‘talking heads’ should be codified and shared with those developing new case
stories
Outreach strategies need to be implemented that more strongly focus on working with
potential users, e.g., workshops on use in faculty development situations. More hands
on training, web seminars, etc. For potential users to adopt the use of the cases, they
must be encouraged on a regular basis. More outreach must be done. This is
particularly difficult as the development activities of the project wind down. However,
there is a core set of developers of the materials who are active participants in
professional organizations such as POD and NMC. They should be encouraged to
develop and present workshops in these environments.
68
Summary of Responses - Survey
About the Respondents
Of the 42 respondents survey respondents, 31 were faculty members, 5 were faculty
developers, 6 were administrators, and 6 were support staff and/or instructional
designers. Most faculty members held multiple roles, e.g., they were also faculty developers
Demographics
Role
Faculty Developer
Faculty Member
Administrator
Other
Discipline
Arts & Humanities
Life Sciences
Health Sciences
Science & Technology
Social Sciences
Professional Schools
Teaching Experience
None
1 – 2 years
3 – 6 years
> 7 years
Not a teacher
Institution Type
Community College
Comprehensive
Research
Other
# of Responses
24
2
4
15 (consultant, instructional
designer, etc.)
7
1
2
6
9
15
1
1
8
29
3
11
13
7
9
69
Survey Results
Participants rated the selected case story using a 7 – point scale (1 = very poor, 7 =
excellent). Note that participants were not viewing the case studies from the website,
therefore, the resources were not available to them.
Table 1: % Responses Value and Usability
Very
Poor
Poor
Fair
Neutral
Good
Very
Good
Excellent
5
2
14
7
24
31
14
0
10
2
7
31
29
19
2
2
2
2
5
10
7
5
26
24
21
21
31
31
0
7
7
10
24
21
26
0
7
7
19
33
14
14
0
2
14
2
21
24
33
2
10
19
10
26
17
12
2
2
12
33
19
17
2
Value to my
teaching
Value to collegelevel instructors
Clarity
Organization
Presentation
Style
Value to my
colleagues
Ease of Use
Adequate details
for
implementation
Quality of
resources
Table 2: Mean Ratings: Value and Usability
Evaluation on Value and Usability
7.00
6.00
5.53
5.27
5.00
5.54
5.45
5.30
4.98
4.88
Mean (7-rate scale)
4.53
4.41
4.00
3.00
2.00
1.00
0.00
Value to my
teaching
Value to
instructors
Clarity
Organization
Presentation
style
Value to my
colleagues
Evaluation Criteria
70
Ease of use
Adequacy of
details
Quality of
resources
Q1: To what extent has this case story piqued your interest in trying out this teaching
method?
Rating
Percentage
1. Little or Not at all
5%
2. Some
12%
3. Neutral
19%
4. A Good Deal
45%
5. A Great Deal
17%
Mean: 3.57 (on a 5 point scale)
Q2: How likely are you to return to this case story to help you with your teaching practice?
Rating
Percentage
1. Very unlikely
24%
2
5%
3
2%
4. Neither unlikely nor likely
7%
5
12%
6
14%
7. Very likely
29%
Q3: How likely are you to let other colleagues or faculty members know about these case
stories as a resource to help them with their teaching?
Rating
Percentage
1. Very unlikely
5%
2
7%
3
2%
14%
5
10%
6
17%
7. Very likely
43%
Q4: As a result of examining this case story how likely are you to integrate this teaching
practice in a future course or class?
Rating
Percentage
1. Very unlikely
2%
2
5%
3
0%
21%
5
19%
6
12%
7. Very likely
33%
71
Q5: How much did the case story contribute to what you now know about the case story
topic?
Rating
Percentage
1. Little or Not at all
19%
2. Some
2%
3. Neutral
36%
4. A Good Deal
21%
5. A Great Deal
14%
Results from Open-Ended Questions
Note: See the Appendix for case story codes
Q6: Why did you select this case story for review?
CS1
I'm a CIS instructor
I am working with our Accounting department to add online components into their courses.
This case study was perfect since the first course I am working on is an AIS course.
I did not know anything about Flowcharts or Business Accounting.
CS3
I had viewed several of the case stories before this and had not viewed this one. I am also
interested in Universal Design and in writing.
CS6
Topic relates to similar course I teach.
random decision.
CS8
I have experience using media from mimeographs to podcasts.
I was interested in the podcast content because I have heard about it but never used it. I
wanted to know more.
CS9
Somewhat at random, although I am interested in issues of remedial courses.
CS10
It was topical, engaging to students. The summary indicated that student learning occurred
I am interested in engaging students in active learning and was curious to see the Debate
assignment structure/action
I'm interested in Global Warming
CS11
interest in intercultural materials
I am recently moving into intercultural learning research and academic enhancement
(professional development) and wanted to learn more about this area.
I reviewed several. In all honesty, none was really on a topic that interested me.
CS12
i am interested in and work with students interdisciplinary studies.
My undergrad degree is in Art Education and use Visual Graphics facilitation tools. Love digital
72
media and storytelling.
CS13
Achievement Gap reference
CS14
To gain a fuller understanding of a how to-csl. Involved with community: university, faculty
and many other communities within the greater, regional community.
I work with hospital staff and nursing students I was interested in looking at the service
learning model. I know little about it
Not highly familiar with topic BUT somewhat interested and my daughter will be beginning a
job at Montgomery College about service learning
CS15
I teach French and use WIMBA
CS16
I have taught College Algebra in the past and have an interest in that topic
CS17
1- same as my doctoral program; 2 - type of course I may teach as part-timer
I work with community college admin
It seemed to be something "different," yet related to my own need for course redesign.
I work in community colleges
I teach mathematics full time at a community college and occasionally teach research methods
in an Ed.D. program.
CS18
First impressions are critical in establishing classroom success between faculty and students.
Felt it was a crucial element and one that will capture the student if done properly
closer to my subject
I ant to adapt my first day of class
Relevant - getting ready to start classes next week
first dAY IS CRUCIAL AND HAS ALWAYS BEEN A challenge. need tips!
CS19
I was not very familiar with the subject area
CS20
I was interested in active learning groups
CS21
I was interested in the concept of "just in time" but it was different than I expected.
CS22
Interest and some exerience with the topic
I am interested in learning more about the creative process.
Topic interested me
CS23
I will be undergoing this kind of process at my institution starting in ..... 2 weeks or so
Q7: How might the case story be improved?
CS1
73
Organization needs improvement . State the goal and the problem it suppose to address at the
start.
include the materials that were being discussed so the audience can also view the material
It seems to me that one needs to know what the issues in the field are in order to orient
oneself to a Q & A time discussion such as the one in the video.
CS3
this one felt somewhat less than complete, at less when compared to some of the other case
stories. I may have overlooked it, but I also didn't notice any additional resources.
CS6
No suggestions
CS8
Cut video segments in half; offer view to beginners and then to more experienced users
I wasn't given information about how to access ELIXR and how it interfaces with MERLOT. I
am new to MERLOT and so all of this is new to me.
CS9
There is a need for more reference materials and links to more information. If the goal is to
encourage change in instructor practices, they will want more details when/if they decide to
work on the changes.
CS10
It was an excellent summary. The videos and sidebars were well designed and contributed
much to my understanding of this method
My screen did not link to the "Assessment" documents. I was interested in viewing these, but
not matter the click, it did not take me to those materials. this would be an easy gliche to fix:0
There is a lot of talking-head video, which most of us find un-engaging. Voiceover of
materials, meetings, etc. is more engaging.
CS11
Give some examples of intercultural specifics.
I chose this case study because the title was "intercultural learning." The content did not have
anything to do with intercultural learning. There was 'talking head' content about "integrative
learning" but no definitions or resources to introduce the concept. The interviewed academic
was articulate and dynamic, but there was no context for me to relate to what she was saying.
I would highly recommend that the videos move beyond 'talking head' to showing clips of what
she was talking about. I was excited to see the link at the very head (not intuitive to get to)
called something like - the faculty member shares her journey. However, when I clicked on the
circles on the diagram nothing happened. There was no content linked to the circles. Further,
the interface did not work well. Rather than a scroll bar, there was a simple 'next' arrow. When
I clicked on it, it went too far, and I found that I missed a section of text in between. In
summary, I thought I was going to learn about "intercultural learning," but instead viewed
micro content about "integrative learning" without being introduced to what this concept
means.
Again, I looked at several. They need to be either discipline-focused or very general - for me,
there was nothing discipline appropriate - and the general ones didn't grab me.
CS12
Wanted to hear more about how class discussions on media were handled, how they could
have been improved (from both instructor and student viewpoints)
1. Provide transcripts for all areas. 2. Break the videos into shorter segments, name them,
provide hyperlinks to the segments. 3. Provide visual images of the student work and let the
users see examples. (I kept looking for the results of the work the students created in these
74
assighments. 4. For time conscious people (me :-) providing a ticker, e.g. a counter that
shows how much time left or an actual clock would be great. The time the instructor spent
talking was a bit long. 5. While I recognize the focus is on teaching practice, student voices,
and outcomes such as visual illustrations of student work would be great.
CS13
Wondering if examples of the student work would be useful?
CS14
It looked a bit like a marketing video for the program but it's understandable. I would have
liked to see the interviews with project participants (the population served)
Very specific examples of service learning projects at the beginning - making it much clearer
what the beginning item(s) is (are)
CS15
It is really very good right now.
CS16
Would have been helpful to hear from the instructors with regard to how easy/difficult it was to
obtain administrative approval to embark on this new program and how they went about
obtaining it.
CS17
illustrations of examples instead of talking head
It was too short. Only one recommendation that was good for training faculty--take an online
course first.
Use a wider variety of people in the case study. Using the same individuals was too repetitive.
Speaker too slow, need someone with more topics to add
Discussion of the syllabus elements and letter sent to the students.
CS18
I would like to see more student interaction with the professor.
Some dialog on problems, and/or fiascos that might have occured presented in a warm or
humorous light.
Additional disciplines added in the examples.
more disciplines, like history
CS20
I accidently increased the size of the text and noticed some of it was lost or words were broken
into two lines
CS21
The instructor had much expertise, and I would have liked much more detail, links to support
materials, lesson plans, ways to get more details about planning, preparation, implementation,
and assessment. Also what if any support may be needed. The instructor had been doing this
for 10 years, how long before she felt accomplished with the techniques?
CS22
Perhaps more clarity or overview of how it is organized
I would like to see a scroll bar on the videos so that I could move through parts that are not
relevant to me. I would also like a counter to see where I am in the video.
The one on using real clients did not have a lot of practical suggestions for how to go about
doing it in class.
CS23
75
More hard facts and information. Tangible results. An outline or description of methodologies.
see previoius written comment. Basically, it needs a more analytical frame.
Q8: Other comments you may have about case stories or this survey.
CS1
Needs tooltips for the navigational buttons.
I wondered if I was part of a trick experiment.
CS3
I really like the approach and the visual unity of this study.
CS6
Interesting from a language as well as cultural aspect
CS8
I have already used vodcasts and podcasts.
CS9
On the interface, the beeps when one clicks on new parts of the story are a bit annoying. I
would also suggest that there be some indication on the videos of the progress of the videos.
Also it would be good to have the videos go blank (or something like that) at the end so you
don't end up with pictures of the speaker with their mouths hanging open.
CS10
This is a fantastic tool/resource. Kudos to those who are developing these videos. Very
intriguing... you just need to get the word out to faculty across the universities so we can have
access to learn from these. I never would have learned about this had it not been for the
MERLOT conference. So many people can benefit, please please share *widely*.
Technically very good, easy to use.
CS11
There seemed to be a lot of jargon with little specific information other than students should
connect assignments. i couldn't see how it was intercultural.
I love the idea of this resource and was very excited about the list of topics. I hope that I
happened to click on a case story that does not parallel the quality of the rest of the case
stories. I hope that this resource is developed further. If so, I will definitely use it.
The navigation needs to be more apparent. All case studies should have the same navigation.
Beginning by linking out to a website is bizarre. Audio levels should be consistent - I had to
turn it up for one. I like the ones that started with music - the 3 that had music all had music
that set a mood - which was interesting. But I kind of then wanted what I was seeing to be
more clever. Short clips - perfect.
CS12
Loved the idea of project-based work incorporating hands on learning, peer-teaching and that
incorporated a lot of instructor and peer feedback opportunities
The instructor talks about iMovie, the Visible Knowledge Project, and the work that Berkley
does in digital storytelling. Links or additional resources wrapped up for users would be a nice
addition. Also, transcripts for people who like to read, and captioning would be valuable.
CS14
very interesting module.....relevant points were raised...the best way to learn something is to
do something like this
I felt strong impulses in 2 directions often 1. Wanted some kind of skimming/browsing option
- like a counter/slider i could use when viewing a single clip 2. Wanted some option for
76
responding WHILE i was reading or listening - to submit a different view, alternative
interprettion, suggest a correction, or ask for some other kind of clarification
CS15
This is an effective way to teach about new strategies for redesigning FL environments.
CS16
Very nicely put together!
CS17
- presenters expertise was a plus; - take advantage of strengths of visual medium; more than
talking head
Suggest more dynamic speakers
CS18
None.
Enjoyed the wide variety of disciplines, gender and breadth fo experience of the instructors.
Very interesting way to present informaton
CS19
- Bottom icons should have mouse-overs to say what they do - It took me a while to figure
out how to get back to the starting point - I really wanted to have a timeline for the videos
so that I could see how far through I was, and to skip or replay easily
CS20
I like the way the responses were organized into student and teacher changes, benefits and
especially challenges. It wasn't showing only the successful aspects of the work
CS21
A line of text repeated on the second page of one segment
CS22
Wonderful interviews!
CS23
The scrolling of text was slightly disconcerting. I wasn't sure where to begin reading again at
first. Animating the scroll slighly would have helped.
77
Summary of Responses -Interviews
I. About the Respondents
Demographics
Role
Faculty Developer
Faculty Member
Administrator
Other
Discipline
Arts & Humanities
Life Sciences
Health Sciences
Social Sciences
Teaching Experience
None
1 – 2 years
3 – 6 years
> 7 years
Not a teacher
Institution Type
Community College
Comprehensive
Research
Other
# of Responses
2
7
3
1 (consultant)
1
0
0
1
2
5
0
0
0
9
0
3
6
0
0
II. Case Selection and Review
Day one of the study, respondents were asked to select something of interest to them, in the second
day, they were asked to select something they didn’t know about. Respondents who selected cases
because they knew something about the topic tended to want to ‘check’ the case for accurate
content and/or look for ideas on how the topic was presented and discussed. In other words the
later group might have been ‘shopping for new ideas’.
78
Reason for Selecting Case Story for Review
Case Story
# Viewers
UDL
1
5. Rhythm and Pitch: Introduction
to Music Class
1
6. Understanding Abstract
Theories by Connecting to
Common Issues
Reimagining Learning Spaces
16. College Algebra Redesign for
Greater Student Success
Course Preparation and Design
8. Mimeographs to Podcasts
Reason Case was Selected
" music mystifies me - thought I'd learn
something - really unfamiliar territory”
similar to course she teaches - subject matter was
business english. Respondent is very experienced,
over 40 years teaching, so she looked at the case as
there was not much new she could learn - she
selected it based on topic and closeness of topic to
what she does and the multi-cultural aspect
1
teaching area
1
FD component, wanted to see what sort of message
it had about podcasts - going to be starting to use
iTouch technology in all our grad programs – so
they can learnt to integrate into their teaching
student success is really important to her, so she
thought that learning about this might be a good
thing - Gardner's modular (?) intelligence -
9. Increasing Student Success
1
Community Service Learning
14. Building a Community Service
Learning Project
2
has an interest in it - teaches a class - each class has
a SL component
works closely w/ community based learning and
active member of American Democracy project. comments that she will definitely have to show it to
our director.
Teaching Strategies
22. Fostering Creativity
1
Prior to teaching, respondent was a professional
photographer and so is very visual. The creativity
aspect was what attracted him.
Developing Instructional Expertise
23. Supporting Course Redesign
1
having a lot of course redesign issues on campus so
thought it would be practical to review
Relationship Between Prior Knowledge, Time Spent Reviewing Case Story and Reason for
Stopping
There seems to be no relationship between the amount of prior knowledge about the topic and
reason for stopping viewing the case. Those who spent less time on the case reported more
problems with usability – those who spent over 20 minutes made no comment on usability in
this section of the interview. Those who reported knowing a great deal or a lot about the topic
might be less patient in terms of examining the content.
III. Value of the Case Story to Teaching
In general and regardless that respondents viewed different case stories they rated the
value of the case stories very highly (x = 3.5, on 5 point scale where 5 is excellent). This
79
may indicate unwillingness on their part to be too critical (or provide realistic ratings) to
the interviewer.
Few respondents were able to articulate a particular value to their own teaching. While
they described positive things about the case stories, they did not discuss how they might
use the information. This may be a factor that for many, they were already using the
technique or pedagogy and there fore saw no need to apply it specifically to their own
teaching.
Value of Case Stories to Other College Level Instructors
Consistent with SERC research regarding the case stories, faculty felt that the case stories
would be very useful for others, especially new faculty or faculty interesting in the topic.
Value of Case Stories to Other Colleagues, e.g. Faculty Developers
Those who rated this most highly tended to equate value to colleagues with value to faculty
developers. They felt the case stories would be very useful in for faculty development
purposes – this is consistent with their thoughts about use by other faculty.
IV. Quality of the Content, Organization and Usability of the Case Story
For the most part, respondents appreciated the way the stories were constructed, the
visual nature of them, their short length and the closed captioning. Some respondents
reported the navigation difficulties made it difficult to sort through.
Organization of Case Story
Comments revealed appreciation of the flexibility of how to access the content, but also
brought out some usability issues. People who wanted linearity were somewhat frustrated:
the same can be said for several who appreciated the non-linearity aspects of the layout.
Presentation Style
Variability in comments and ratings most likely reflect differences in how each case story is
built. However, each comment reflects something to do with differences in learning styles.
This may mean that the case stories are particularly valuable in that they address learning
styles: some may be stronger than others in achieving this as comments with the lower
ratings indicate.
Ease of Use
By now in the interview most respondents felt they had already addressed this question in
their previous responses.
V. Quality of Resource Materials
Respondents did not access the case stories from the ELIXR website. Instead, the case
stories were stored locally, on the computers used for the study. As a result, it was not
possible to download and view the pdf’s and other resources attached in the ‘Resource’
section of a case story (when such a section exists). Interestingly, it appears that a number
of the respondents felt they had viewed the available resources, or thought they had
viewed all possible resources when they had not. For some, this is a function of the short
duration of time they spent on the site.
80
Adequacy of Details Supplied on Implementation
Respondents felt that they had been adequately introduced to the topic, but for the most
part felt (even though rating it highly) that more information on how to implement would
be helpful.
VI. Dissemination, Impact or Potential Use of Case Story
Effectiveness of Case Story in Raising Interest in the Topic
The case stories were an excellent way of introducing someone to the area or topic. They
each seemed to have strengths, thought not always the same one, e.g., students telling their
stories in one case, faculty doing the same in a different case. Even if respondent felt he or
she was quite adept in the area covered by the case story, they could identify someone (in
addition to themselves) who could benefit from some part of the story.
Use of Story for Personal Professional Development
This question produced the most variability of any question asked. Respondents each saw
different ways in which they might use it or recommend its use to others. Several, who felt
they were experts in the area, did not feel they could learn more from it by returning to it.
Dissemination of Case Story
Almost universally, respondents felt they could and would recommend the case story they
reviewed (or another) to others who might find it useful.
Contribution to Knowledge About the Topic
Respondents who did not identify themselves as experts in the field reported that they had
gained knowledge. Either they learned something completely new, or learned a new way to
present something they already knew about
VII. Suggestions for Improvement of Case Stories
Suggested Improvements to the Cases
Would have liked to hear more from the students - especially about their success
Felt like it was 2 case stories - length of time was fairly long she went through everything
thoroughly. Wonders about seat time - people's attention span. Had idea in her head, how
would I use this? Greater focus and not so much about UDL - there's that clip w/ the FD
center person -that clip didn't work. But her thought was that person's discussion wasn't
enlightening - she was more interested to hear teacher. That's where the power of them is it looses power and energy when it goes to: here's some more info for you.
Knowing how busy people are - try to chunk the 20 minute videos down into shorter bits
Giving where to go to contact or find out about ALEKS - guidance on getting started and how
you know you are done.
Didn't capitalize on the visual - even for their art and design division - it’s a different skill.
Didn't see anything in particular - interview was well done, clear response –
One page that explains the icons - or a tutorial,
Less of the administrators talking and more students
Other Comments
overall very well done - those currently looking to revise courses - there were good ideas
81
"Super Happy that I discovered ELIXR - opened up a way to have a conversation w/
colleagues about how to use Tech effectively to do PD - a new way - something to point to
that's here is an example."
enjoyed the process
nicely done
saw where this would be useful for topic as well as culture - to see someone doing ESL - that
would be helpful to educators and students.
82
Summary of Responses – Follow-up Survey
About the Respondents
Demographics
Role
Faculty Developer
Faculty Member
Administrator
Other
Discipline
Arts & Humanities
Life Sciences
Health Sciences
Social Sciences
Other
Teaching Experience
None
1 – 2 years
3 – 6 years
> 7 years
Not a teacher
Institution Type
Community College
Comprehensive
Research
Other
# of Responses
11
2
4
2 (consultant)
4
0
0
3
2
5
3
0
0
3
12
2
5
6
6
0
Survey Results
Q 1: While attending the MERLOT International Conference, you viewed an ELIXR case story.
Since the conference, have you revisited that case story?
All respondents selected NO
Q 2: Since the conference, have you visited www.elixr.merlot.org to view any of the case
stories?
All respondents selected NO
Please explain your response to Q2.
During the conference I viewed several of the case studies and haven’t had TIME to look
at them again
Busy teaching and not currently developing course material
83
While I have not visited the sites, I have sent people there to look at the sties. Would
also like to be contacted about how some of our state faculty might connect with the
case stories. We have teacher awards that create DVD materials similar to the case
studies.
At my age, I need an email follow up to be reminded!
I haven’t felt a need or compelling reason to revisit
I forgot
Just too busy. I was finishing a book while teaching four classes and have had no time to
do anything. But I do intent to revisit the site
Probably a combination of explanations – time constraints make revisiting a lower
priority is one. And ‘out of sight out of mind’ is likely another
It slipped my mind
I do not have easy access to the links, I would otherwise
Have not had opportunity or reason
Since the merlot conference, I returned to fall startup and I have not touched the
materials from the conference
The one I viewed was essentially a talking head without examples, and I was not
interested in seeing more
Q3: Please indicate how you used any of the case stories, if at all. (Select all that apply.)
Reason
# of
Responses
I used one or two of the techniques or innovations described in the case
0
story in my own classroom and teaching
I recommended the case story to fellow instructors who I know are
1
interested in this topic or teaching
I recommended the case story to faculty development professionals on my
1
campus
I used (or am planning on using) a case story for a faculty development
0
workshop
I recommended the case story to campus administrators
0
I have thought about how I might integrate some of the ideas or topic of
0
the case story, but I have not yet done so
other
2
Q4: Did you recommend the case stories or the ELIXR site to your colleague(s) other
instructor(s) or faculty developer(s) on your campus or another campus?
Six responded yes
Please describe your response
I have discussed the case studies with various administrators and talked about them at
a faculty round table
My colleagues and I who attended MERLOT wrote a piece for our CTL newsletter and
discussed the conference and the sessions we attended and chaired with fellow faculty
84
Q5: Please make other comments about the case stories or ELIXR here.
This is good work and I hope it can continue
Enjoyed working with the case studies and look forward to your results and
conclusions of your evaluation
I need a reminder for my responsibility
I think they are definitely useful and will continue to study as time permits
Make the links easier to find
I think the idea is good, but I didn’t feel that the program I watched would be of a great
deal of help to other instructors.
Limitations
The subjects of this study were all attendees of the MERLOT International Conference,
which tends to draw participants who are educational innovators. Therefore, the sample is
not representative of faculty members as a whole. The sample is biases towards people
who value innovative approaches to learning and who are themselves innovative
practitioners. The study therefore, cannot address how a novice to teaching or faculty
member who is not pre-disposed to a particular pedagogy might react to these learning
materials.
The sample, while including some faculty developers, for the most part were not people
who plan and implement faculty development on their campuses. Therefore, their
responses to questions regarding faculty development must be interpreted with care. For
example, while they may believe the resources might be excellent for such training, they do
not know what it takes to plan and implement trainings, therefore, they may not be able to
identify areas for improvement.
The purpose of the study was to learn more about the resource potential as learning
materials for faculty. The follow up survey provided much needed information about what
it takes to encourage use of materials. One research question that was of interest was to
what extent hands-on exploration of a resource impacted a user’s future use of a resource.
In this case, we found that even though they rated it highly, few revisited it. And, while
others reported that they would forward information about ELIXR (and the case stories) on
to others on their campuses, few in our follow-up survey did so. Lack of positive action in
this regard may be because, as mentioned above, they may not have had any reason to do
so, for example, they were not faculty developers or administrators who would benefit
from doing so.
85
Appendix A – Instruments
Survey Instrument
86
87
88
89
90
Interview Protocol
91
Follow-up Survey
92
93
94
95
Appendix B – List of Case Stories
Case story codes for the user study, numbers of participants for each case story and URLs
CS1: Flowcharting in Business Education [3 participants]
http://pachyderm.cdl.edu/elixr-stories/udl-business/
CS2: Integrating Technology into Instruction - American Sign Language [0]
CS3: Multiple Delivery Methods of Course Content - Technical Writing [1 participant]
http://pachyderm.cdl.edu/elixr-stories/udl-technical-writing/
CS4: Small Group Discussion - Information Systems [0]
CS5: Rhythm and Pitch: Introduction to Music Class – Music [0]
CS6: Understanding Abstract Theories by connecting to Common Issues - English Education [2 participants]
http://pachyderm.cdl.edu/elixr-stories/udl-english/
CS7: Transforming Business Math – Business Math [0]
CS8: Mimeographs to Podcasts – Education [2 participants]
http://pachyderm.cdl.edu/elixr-stories/tcd-elementary-education/
CS9: Increasing Student Success – Math [1 participant]
http://pachyderm.cdl.edu/elixr-stories/tcd-dev-math/
CS10: Global Climate Change and Debate – Interdisciplinary [3 participants]
http://pachyderm.cdl.edu/elixr-stories/integrative-learning-communication/
CS11: Intercultural Learning – Communication [3 participants]
http://pachyderm.cdl.edu/elixr-stories/integrative-learning-diversity/
CS12: Teaching Visual Media Analysis, Writing and Computer Skills – Interdisciplinary [2 participants]
http://pachyderm.cdl.edu/elixr-stories/integrative-visual-sociology/
CS13: Bridging the Gap for Student Success – Marketing [1 participant]
http://pachyderm.cdl.edu/elixr-stories/comm-service-marketing/
CS14: Building a Community Service Learning Project – Interdisciplinary [3 participants]
http://pachyderm.cdl.edu/elixr-stories/comm-service-uthscsa/
CS15: College Algebra Redesign for Greater Student Success – Mathematics and Computer Science [1 participant]
http://pachyderm.cdl.edu/elixr-stories/reimagining-college-algebra/
CS16: Community College Leadership Program Goes Online – Community College Leadership [1 participant]
http://pachyderm.cdl.edu/elixr-stories/reimagining-college-leadership/
CS17: Making Your 1st Class Session Really First Class – Interdisciplinary [5 participants]
http://pachyderm.cdl.edu/elixr-stories/1stday-slo/
CS18: Using ConcepTests – Geology [6 participants]
http://pachyderm.cdl.edu/elixr-stories/serc-geology/
CS19: Active Learning Groups – Organic Chemistry [1 participant]
http://pachyderm.cdl.edu/elixr-stories/active-learning-chemistry/
96
CS20: Just in Time Teaching – Geosciences [1 participant]
http://pachyderm.cdl.edu/elixr-stories/serc-geoscience/
CS21: Fostering Creativity – Computer Arts and New Media, Interior Architecture Design, Sculpture,
Video Editing and Writing [1 participant]
http://pachyderm.cdl.edu/elixr-stories/fostering-creativity/
CS22: Supporting Course Redesign – Multidisciplinary [3 participants]
http://pachyderm.cdl.edu/elixr-stories/flc-ssu/
CS23: Foreign Language Redesign for Oral Proficiency – Foreign Language [2 participants]
http://pachyderm.cdl.edu/elixr-stories/reimagining-foreign-language/
97
Appendix F: SERC User Study
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
Appendix G: ELIXR CSTLT Study
116
[Document Title]
February 2010
Flora McMartin
117
118
119
120
121
122

Evaluation of ELIXR, a FIPSE Funded Project

Transcription

Similar documents

merlot - Cheval Quancard

flyer - MyGrandmontRosedale.org

Untitled

2012 Marilyn Merlot Nova1.3

Campiano Ontaneta Spedale

Fellows and Longtime Members Meet at ACC

Vikatan 2x

With a Taste of MERLOT

Response to Intervention (RTI)