University of South Alabama College of
Outcomes Assessment Methodology
University of South Alabama College of Medicine(USA COM) employs a number
of techniques to assess the outcomes of our educational activities. The type of
technique we use will depend on the educational format of a continuing medical
education (CME) activity and the level of outcome that we are aiming to assess.
University of South Alabama College of Medicine espouses the model
described by Donald Moore, Jr., PhD, Joseph Green, PhD, and Harry Gallis, MD[i],[ii] in developing our outcomes-based
evaluations. We work closely with our partners to determine which techniques
will yield the best results.
The Moore, Green, and Gallis model describes 7 outcome levels as follows:
- Level 1 Participation
- Level 2 Satisfaction
- Level 3A Learning: Declarative Knowledge (Knows)
- Level 3B Learning: Procedural Knowledge (Knows How)
- Level 4 Learning: Competence (Shows How)
- Level 5 Performance (Does)
- Level 6 Patient Health
- Level 7 Community Health
All of our activities are assessed for Levels 1, 2 and 3 by our
registration data (Level 1) and our standard activity evaluation which asks
participants to rate their level of satisfaction with the activity (Level 2)
and the degree to which they believe the learning objectives were met (Level
Measurement of objectives achieved
Activity participants are tested based on the behavioral learning
objectives established for a CME activity. For example, one of the objectives
might be – “At the conclusion of this activity, participants will be able to
list three of the currently approved statin drugs.” As part of the evaluation
form, participants would be asked to actually list three of the currently
approved statin drugs. In the absence of a pre-test, there is no guarantee that
the learning occurred as a result of the activity. Yet, this process
demonstrates whether or not the objectives were achieved – important
information for the faculty and CME staff. This method can be employed with
live meetings, enduring and Internet-based CME activities.
University of South Alabama College of Medicine currently uses the
following types of outcomes assessments to measure Level 3B, Level 4, and Level
5 outcomes, ie. procedural learning, competence, and performance based changes.
Pre- and post-tests
Activity participants complete multiple choice questions concerning
activity content before and immediately after a CME activity. This method
measures learning that occurred as a result of the activity. The benefit of
this type of measurement is that the participants, the faculty and the CME
staff have immediate feedback regarding what learning has occurred (Level 3B
measurement). This method may not necessarily predict retention of the learning
or change in performance. Pre- and post-tests can be used in conjunction with
live meetings, printed enduring materials and Internet-based CME activities.
Commitment to Change
Participants of live and enduring material activities are asked to write
one to three changes that they plan to make a change as a result of our
activities (Level 4 measurement). Jocelyn Lockyer and her associates have found
that a commitment to change (CTC) predicts actual change in practice[iii]. According to Lockyer, et. al.,
“Three quarters of CTCs were fully or partially implemented” in her study (p.
76). A summary of these reveals the immediate impact of the CME activity,
providing useful needs assessment data for planning future activities.
Post activity surveys (“fax-back”
Post activity surveys go further in measuring change by venturing into
performance based change – the Level 5 outcome. Participants are asked, at the
conclusion of a CME activity, to list three changes that they intend to make as
a result of the activity.
Within one to three months of the CME activity, the USA COM staff will fax
or email our CME activity participants and ask them if they have fully
implemented, partially implemented or were unable to implement the changes they
intended to make.
The limitation of this data is that it is self-reported. However, in the
absence of actual observation of a physician’s performance in practice, this
information serves as a surrogate marker that, according to Lockyer’s research,
is indicative of actual change.
Case based assessment
In a comparison of chart audits, standardized patients (where actors take
on the role of patients and physicians are evaluated on their interactions with
the “patients”), and case vignettes, case vignettes were found to be as
effective as the other two methods in determining outcomes[iv]. Aimed at measuring Level 3B and Level
4 outcomes, we have asked physicians in live meetings to answer key multiple
choice questions in response to a case presentation. The cases and questions
are presented before and after the CME activity to measure learning. Case
vignettes can also be administered to a control group, ie. a group of physicians
who share a professional profile with the activity participants but who did not
participate in the activity.
Performance Improvement Initiatives –
Following the AMA’s guidelines on awarding AMA PRA Category 1 Credit™
for performance improvement, University of South Alabama College of MedicineCME
staff will work with physicians to undertake customized practice-based
performance improvement initiatives, facilitating the measurement of change
through chart reviews. Participants audit a set of charts using a data
collection survey created by USA COM faculty. The data from the charts are
reviewed by USA COM faculty and a live educational intervention is developed
based gaps in care revealed through the chart reviews. As a part of the
educational intervention, the participants are asked to submit Performance
Improvement Plans based on the results of the chart reviews. Members of USA COM
will review and approve proposed plans, offering suggestions whenever
necessary. Three months after the participants implement their Performance
Improvement Plans, they are asked to audit a different set of charts to see if
the intervention has had a positive effect on their practices and in turn on
the health of their patients. 20 AMA PRA Category 1 Credits™ are awarded
to participants for completing the initiative. Performance improvement
activities will lead to Level 5 outcomes assessment data. Level 6 data can be
attained through longitudinal studies, if resources allow.
Performance Improvement Initiatives –
Through a unique design, University of South Alabama College of Medicine develops
Virtual Communities of Practice working with physicians and allied health
practitioners from varied locations to implement practice improvement
initiatives. Activities are supported through a custom designed website, email
University of South Alabama College of Medicine will continue to seek new
ways to obtain outcomes assessment data for our CME activities. In keeping with
the charge set forth by Moore, Green, and Gallis, we will strive, in
particular, to incorporate formative assessment processes into our activities
that provide participants with opportunities for practice and feedback.
[i] Barnes, B, Davis, D, Fox, R. Continuing
Professional Development of Physicians: From Research to Practice. Chicago.
AMA Press, 2003: 249 – 274.
[ii] Moore, D, Green, J., Gallis, H. Achieving
Desired Results and Improved Outcomes: Integrating Planning and Assessment
Throughout Learning Activities. Journal of Continuing Education in the
Health Professions. Winter 2009; 29: 1 – 15.
[iii] Lockyer, J, Fidler, H, Hogan, D, Pereles,
L, Wright, B, Lebus, C, Gerritsen, C. Assessing Outcomes Through Congruence of
Course Objectives in Reflective Work. JCHEP 2005; 25: 76-86.
[iv] Peabody, J, Luck, J, Glassman, P,
Dresselhaus, T, Lee, M. Comparison of Vignettes, Standardized Patients, and
Chart Abstraction: A Prospective Validation Study of 3 Methods for Measuring
Quality. JAMA 2000; 283: 1715 – 1722.