Chapter 1 • Experimental and Quasi-experimental
Evaluations
S. Bartholomew Craig and Kelly Hannum
Chapter
2 • Leading
with Theory: Using a Theory of Change Approach for Leadership Development
Evaluations
Manuel Gutiérrez and Tania Tasse
Chapter 3 • EvaluLEAD: An Open Systems Perspective on Evaluating Leadership
Development
John Grove, Barry Kibel and Taylor Haas
Chapter
4 • Make Evaluation Work for the Greater Good: Fueling Generative
Visions of Provocative Possibility and Responsive Praxis in Leadership Development
Hazel Symonette
Chapter
5 • Measuring ROI in Leadership Development
Jack J. Phillips, Ph.D. and Patricia P. Phillips, Ph.D.
Part Two • Part Three • Book Home • Order
Experimental
and Quasi-experimental Evaluations
S. Bartholomew Craig and Kelly Hannum
Experimental and quasi-experimental approaches to evaluation are the
focus of this chapter, and they provide a structured means to think about
designing
evaluations. The chapter addresses a noticeable lack of effort to apply
experimental procedures to leadership evaluation, and highlights some
of the difficulties
and benefits of doing so. The chapter presents experimental and quasi-experimental
procedures as an effective way of measuring changes in leadership and leadership
outcomes, and determining the relationship between the leadership development
initiative in question and the changes measured.
Chapter 1 Resources
Buros Institute
of Mental Measurements
http://www.unl.edu/buros/bimm
Existing measures
can be identified and evaluated using information provided by the Buros
Institute of Mental Measurements, which has a searchable web-based
Test Directory.
Helpful online calculators for estimating sample size requirements:
http://www.surveysystem.com/sscalc.htm
http://www.stat.uiowa.edu/~rlenth/Power
http://www.isixsigma.com/library/content/c000709.asp
• • • return to top
Chapter 2
Leading with Theory: Using a Theory of Change Approach for Leadership
Development Evaluations
Manuel Gutiérrez and Tania Tasse
When faced with the task of designing evaluations of leadership development
programs, evaluators are likely to wonder what types of evidence they
should look for in order to measure program success. This chapter outlines
the
implantation and utilization of pathway mapping as a first step in
measuring leadership
development. The article describes multiple uses of pathway mapping
and describes in detail its effect on evaluation.
Chapter 2 Resources
Aspen Institute
Roundtable on Community Change
http://www.aspeninstitute.org/site/c.huLWJeMRKpH/b.612045/k.4BA8/Roundtable_on_Community_Change.htm
This resource describes the work of the Roundtable and includes
links to resources concerning theory of change and community-building topics.
Eisenhower Fellowships
http://eisenhowerfellowships.org
This
source describes the mission and goals of the Eisenhower Fellowships
as well
as program activities.
Leadership for a Changing World (LCW)
http://leadershipforchange.org
This resource describes the LCW program and includes announcements
about new and
current award winners and their social justice work and accomplishments.
Pathways Mapping Initiative
http://www.pathwaystooutcomes.org
This resource provides information about
strategies that have been effective,
appear to be
working, or are promising, in community initiatives taking
place in particular locations. It also discusses how pathways are
designed
to
guide choices
about investments, programs, and policies made by multiple
stakeholders.
Next Generation Leadership
http://www.nglnet.org
This resource
includes information about the background of the Next Generation
Leadership
program, and the program’s
alumni network.
W. K. Kellogg Foundation Logic Model Development Guide
http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf
• • • return to top
Chapter 3
EvaluLEAD: An Open Systems Perspective on Evaluating Leadership
Development
John Grove, Barry Kibel and Taylor Haas
One of the major challenges of leadership evaluation lies
in assessing the value added by a leadership development
program
to those directly
served. This
chapter introduces EvaluLEAD as a method to evaluate links
between program actions and organizational or systems-level
results.
Subjects included:
history/evolution of evaluLEAD, implementation of evaluLEAD,
possible problems with the method.
Chapter 3
Resources
EvaluLEAD
Grove, J., Kibel, B., and Haas, T. EvaluLEAD:
A Guide for Shaping and Evaluating Leadership Development Programs.
W.K. Kellogg
Foundation and Public Health
Institute, 2005. http://www.wkkf.org/DesktopModules/WKF_DmaItem/ViewDoc.aspx?LanguageID=0&CID=281&ListID=28&ItemID=2813740&fld=PDFFile This
guide provides detailed information about and guidance
for the EvaluLEAD process.
EvaluLEAD at http://www.evalulead.net provides access to the guide
as well as contact information for the authors and other
practitioners
who
can work
with organizations
to implement EvaluLEAD and related open systems evaluation
solutions.
Jacobs, J. The Nature of Economies. New York: Random
House. (2000).
Senge, P., Jaworski, J., Scharmer C. O., & Flowers B.
S. Presence: Human purpose and the field of the future.
Cambridge, MA: Society for
Organizational Learning. (2004).
Wheatley M. J., & Kellner-Rogers, M. A simpler way. San
Francisco: Berrett-Koehler. (1996).
• • • return to top
Chapter 4
Make Evaluation Work for the Greater Good: Fueling Generative
Visions of Provocative Possibility and Responsive Praxis
in Leadership Development
Hazel Symonette
This chapter focuses on evaluation as a development resource
for excellence through processes that intentionally
blur the lines
between evaluator,
program developer, and participant. The author explores
the potential (and positive)
ramifications of a successful leadership program and
also the implementation of a more “natural” form
of evaluation.
Chapter
4 Resources
Evaluation Training Institute, Howard University
http://www.howard.edu/schooleducation/eti
This
program offers professional development to expand the number
and
capacity of mathematics
and science
project evaluators who can plan and implement evaluations
that are technically sound,
contextually relevant, and have increased utility.
Reina, D., & Reina, M. Building trust in the workplace.
Access at http://www.trustinworkplace.com
Other Resources
Johnson, A. Privilege, power and difference. Mountain View,
CA: Mayfield, 2001.
Style, E. Curriculum as window and mirror. Social Science
Record. Fall 1996, 35-42.
Thomas, V. "Building a contextually responsive evaluation
framework." V.G. Thomas & F.I. Stevens (eds).
Co-constructing a contextually responsive evaluation
framework: The Talent
Development
Model of
school reform. New
Directions in Evaluation 101. San Francisco, CA: Jossey
Bass, 2004.
• • • return to top
Chapter 5
Measuring ROI in Leadership Development
Jack J. Phillips, Ph.D. and Patricia P. Phillips, Ph.D.
Increased emphasis on leadership development programs
in the business, non-profit, and governmental sectors
has
lead to many
executives
questioning the value
of individual development programs on their organizations.
This chapter offers methods to better quantify the return
on investment
(ROI) of
leadership programs.
A framework for measuring the ROI of development programs
is explored, along with a discussion of the many variables
that
can impede accurate
evaluation
of a program.
Chapter
5 Resources
Return on Investment
Each of the following organizations has members interested
in ROI.
The American
Society for Training and Development
http://www.astd.org
The ROI Institute
http://www.roiinstitute.net
The
Society for Human Resource
Managers
http://www.shrm.org
The ROI Network
http://www.astd.org
This network
has been
organized to share information. Case studies are available.
Other Resources
Phillips, J. Investing in Your Company’s Human
Capital: Strategies to Avoid Spending Too Little or Too
Much. New
York, NY: AMACOM,
2005.
Phillips, J. Proving the Value of HR: How and Why to
Measure ROI. Alexandria, VA: SHRM, 2005.
Phillips J. Return on Investment in Training and Performance
Improvement Programs. 2nd Edition. Woburn, MA: Butterworth-Heinemann,
2003.
The second edition of
this bestselling book guides you through a proven, results-based
approach to calculating the Return on Investment in training
and performance
improvement programs.
Phillips, J. and Schmidt, L. The Leadership Scorecard.
Woburn, MA: Butterworth-Heinemann, 2004.
This book expands
and discusses
best practice
leadership development
methods, incorporates ROI measurement and evaluation
methodology, sets out a step-by-step process, presents
case studies,
and provides proven
measurement
and evaluation techniques.
Phillips, J., Phillips, P., and Hodges, T. Make Training
Evaluation Work. Alexandria, VA: ASTD Press, 2004. This
book provides
the learning professional—newcomer
or veteran—practical and specific ways to show
value and communicate results, select the right model
and find
resources, and get management
buy-in and overcome resistance.
Phillips, J., Stone, R., and Phillips, P. The Human Resources
Scorecard: Measuring the Return on Investment. Woburn,
MA: Butterworth-Heinemann, 2001. This book
provides a comprehensive, step-by-step guide for measuring
the impact of human resources programs and includes seven
detailed case studies.
This
book is essential
for human resource executives, professionals, CEOs, CFOs,
consultants, professors and other managers concerned
with their business’s
bottom lines.
Phillips, P. The Bottomline on ROI. Atlanta, GA: Center
for
Effective Performance, 2002.
This book offers the business case for ROI. It provides
the basics, benefits, and barriers to measuring training
and performance improvement programs.
Phillips, J. and Phillips, P. ROI at Work: Best-Practice
Case Studies from the Real World. Alexandria, VA: ASTD
Press, 2005.
This book
presents best-practices
case studies from the real world of measuring return
on investment, including a five case studies about leadership
development
programs. Case studies
come from the government sector as well as a range of
industries
and provide valuable
lessons for professionals working to contribute to the
strategic goals of their organizations.
Phillips, P., Ed. Phillips, J., Series Ed. In Action:
Measuring Return on Investment, Volume 3. Alexandria,
VA: ASTD Press,
2001.
This book
has 11 cases from a variety
of industries including telecommunications, computer
and technology, retail stores, automotive, and the government
sector.
Phillips, P. and Phillips, J. ROI Basics. Alexandria,
VA: ASTD Press, 2005.
This book provides the fundamental
steps
in developing
a comprehensive
evaluation, offering the reader basic skills in ROI,
tools for selecting appropriate programs
for ROI evaluation, and the ability to develop a strategy
to integrate ROI as part of the ongoing learning process.
• • • return to top
Project Sponsors
