Building a Strategic Approach to Learning Evaluation

13 Nov Building a Strategic Approach to Learning Evaluation

By Marjorie Derven – astd.org

This five-step strategy serves as a vehicle to promote shared responsibility with key clients and obtain their commitment up front to engage in action planning for positive change.

Are your learning initiatives adding value to the organization? That is a simple question, but addressing it is not always easy. Faced with tight budgets, limited resources, and ongoing change, today’s organizations expect answers.

Building a strategic approach to learning evaluation is a solution that demonstrates value, deepens partnerships with clients, and promotes collaboration with multiple stakeholders and HR colleagues—and is a priority for integrated talent management.

The goal for learning should be to enable performance; this objective should likewise drive evaluation initiatives. Considering that formal training accounts for only a small component of learning (Michael Lombardo and Robert Eichinger estimate that 70 percent of learning is obtained from assignments and on-the-job experiences; about 20 percent is developed through relationships, networks, and feedback; and only about 10 percent is derived from formal training processes), evaluation initiatives must reflect the overall environment and context in which learning takes place.

Since training is only a small part of the equation that drives performance, evaluation must incorporate a more integrated, systematic inquiry. A holistic view of performance—the context, barriers, and supports in which the performance must be realized—is essential.

Too often, evaluation and metrics are based more on talk than action. Measuring impact continues to be one of the most challenging aspects of the function, according to Kevin Oakes, founder and CEO of the Institute for Corporate Productivity, which studies best practices in learning. Other research from Bersin & Associates shows that fewer than 7 percent of organizations regularly measure learning impact.

Learning evaluation often is limited to Kirkpatrick’s Level 1, also known as the “smile sheet.” Although Level 1 is easy to do, these reaction-level data are rarely used in any systematic fashion such as determining how the data will be used, with whom it will be shared, and what actions, if any, will be taken as a result.

This somewhat dismal state of learning evaluation is not surprising, given multiple barriers that include

• an evaluation study without a sponsor or a clear picture of how data will be used
• concern about obtaining negative results
• business interruption or costs
• short-term focus
• lack of skill or knowledge about how to conduct evaluation studies
• expectations and buy-in of stakeholders that are not established in advance.

However, the benefits of conducting evaluation and demonstrating value more than outweigh these formidable barriers. These include

• creating accountability—what gets measured gets done
• establishing a baseline for ongoing improvements
• developing momentum and support for learning solutions
• enhancing the learning function brand
• identifying success stories and best practices to be used in future organizational change and training initiatives.

The solution to this dilemma is to build a strategic approach to learning evaluation. A learning evaluation strategy (LES) is a guide for designing and implementing a learning evaluation that consists of a plan of action that integrates learning with business goals and values (see “Expected Outcomes” below). LES is based on five principles:

1. Focus on high-priority learning areas.
2. Address evaluation requirements of multiple stakeholders.
3. Foster shared responsibility for performance improvement.
4. Collect data and use resources efficiently.
5. Conduct action planning.

Focus on high-priority learning areas

Aside from compliance training, which often mandates the demonstration of learning transfer, only a small percentage of training programs warrant conducting the more involved levels of evaluation, Levels 3-5 (see “Five Levels of Evaluation” below). Due to the expense, time, and business interruption that learning evaluation involves, differential investment is essential for other training initiatives.

When selecting the high-priority training initiatives for learning evaluation, consider the following questions.

• Is this training initiative addressing a key strategic imperative?
• Does it involve a large number of participants?
• Is it a large expense to the organization?
• Has a formal evaluation been requested by the client or sponsor?

Kevin Engholm, director of executive development at Citibank, says the rationale behind selecting a program targeted for managing directors for an evaluation effort was that these individuals “are seen as being our ‘culture carriers.'” Therefore, the program was made a priority for learning evaluation investment.

Engholm says the managing directors “are pivotal to help transform our organization. Our goal is moving the needle on key execution priorities and through this program, which is taught almost exclusively by our senior leaders, to create a forum for businesses and geographies to form a more connected global organization.”

Irene T. Boland, manager of global learning for GE Aviation, emphasizes that while “learning evaluation is not optional,” she weighs a number of factors to select learning evaluation priorities. “It is important to pick and choose to make a worthwhile investment of evaluation dollars, based on criticality to the business, such as impact on products or customers,” Boland says. “Be able to address the question, ‘To what extent are our people prepared to deliver against customer requirements?'”

Address evaluation requirements of multiple stakeholders

When training professionals fail to fully engage their clients in the evaluation process, the risks include collecting data without a sponsor and using metrics that only matter to the learning function. Determine what different stakeholders view as meaningful evidence of value, and incorporate these diverse perspectives into your evaluation design.

How do you determine your key stakeholders? Typically, they are cross-functional groups with a vested interest in outcomes, accountable for performance addressed by learning initiatives, and key to long-term success (see sidebar below).

You want to involve stakeholders to gain buy-in early on for cross-functional action planning, obtain a big picture perspective, and build a broader base of organizational support.

Jim Gillece is senior vice president of AlliedBarton Security Services and serves as the human capital management officer and chief people officer. He provides an example of an integrated perspective for evaluation efforts. “We evaluate effectiveness of our leadership efforts through tracking historical trends, looking at factors that include turnover, client retention, and organizational climate surveys,” he explains. “We have established an internal leadership index, which is a compilation of behaviors based on the observations and perceptions of peers, supervisors, and direct reports.”

Boland emphasizes that working with stakeholders is a necessity. “Involving stakeholders is always important and should extend beyond the release of training,” she says. “You want to understand how leaders evaluate success using the technique of asking multiple ‘why’s’ until you get to the root of training—What problem are we trying to solve? Is it a pain point or an opportunity for the business?”

An initiative under way to standardize measures of success is the Talent Development Reporting Principles, which recommend a set of commonly accepted principles that would be analogous to the Generally Accepted Accounting Principles adopted in the accounting profession. As described in the Talent Development Reporting Principles whitepaper, three foundational reporting statements focus on outcomes, effectiveness, and efficiency, which can be used as a framework that can help structure stakeholder metric requirements.

In addition to enlisting their desired metrics for evaluation initiatives, a broad coalition of stakeholders often will be needed for access to data or people to conduct the evaluation study.

Foster shared responsibility for performance improvement

You need others to buy into your evaluation studies, and to obtain their commitment to take action based on data. Using a strategic approach to learning evaluation facilitates actionable information about factors that influence performance, which include rewards and recognition, management direction and priorities, time to demonstrate desired skills, access to resources, and motivation.

If training is evaluated in isolation, and fails to include work environment, barriers, and supports, the ability to improve performance going forward is severely limited. The LES serves as a vehicle to promote shared responsibility with key clients, and obtain their commitment up front to engage in action planning.

Addressing the importance of integrated talent management and involving multiple levels and players is critical, according to Nataliya Lomakina, manager of leadership development at Depository Trust & Clearing Corporation. “Since our leadership development efforts are integrated into our talent review process and strategy, our board of directors is involved in the overall decision making about approach and then later debriefed on the outcomes. We also recognize that the participants’ managers play a critical role in helping us develop people by providing ongoing feedback, real-time coaching, and guidance,” she says.

A recent Hudson Research & Consulting engagement study illustrates this well. When learning evaluation results showed that a well-designed program focused on collaboration didn’t fully hit the goal to change desired behaviors, internal compensation experts were engaged to better align incentives and IT professionals were charged with enhancing networking tools. The result of this cross-functional effort is that the learning initiative has been supported organizationally and collaboration has increased, which produced improved business outcomes.

Recognizing the relationship between training outcomes and business performance, Gillece makes this explicit. “We ascribe to the linkage that good leaders lead to engaged employees, which lead to happy clients, which lead to better business results—so we track leadership as a key metric.”

Use resources efficiently

The fourth LES principle addresses today’s “do more with less” business environments. With limited appetite for activities that take people away from key responsibilities—and there simply is no way to collect evaluation data without interviews, surveys, or observation of people at work—a strategic integrated approach allows you to partner with HR and other colleagues to streamline data collection and avoid needlessly asking the same respondents similar information. LES promotes the examination of organizational needs and sequencing evaluation studies to use people’s time effectively and helps select the best fit among multiple evaluation methodologies.

Engholm is cognizant of extreme time constraints faced by managing directors, and this is a significant factor driving the evaluation approach. “We know this senior-level population is quite busy, so we want to create a balance of setting a strong expectation about their follow-through, but not be punitive,” Engholm says. “In many ways, we are trying to reset the notion of what it means to be a managing director at Citi, and delivering on the cascade [an explicit expectation that managing directors will take key messages and cascade these to their direct reports within 30 days] is one of their responsibilities as senior leaders.” Indeed, finding the “sweet spot” that balances rigorous metrics with practical business requirements is a key to successful evaluation.

Once you have engaged key sponsors up front and defined what they consider to be meaningful evidence of value (which may be different from what you anticipated), you can collect the right data.

As business priorities shift, so will evaluation metrics. “We are continuously working on refining our evaluation approach,” says Lomakina. “The goal is to obtain the right metrics without overcomplicating it. It requires ongoing calibration and integration of various forms of input.”

Conduct action planning

An LES helps engage key stakeholders in action planning so meaningful change can be executed. At Depository Trust & Clearing Corporation, “organizational action planning involves linking business challenges to the program design, then measuring the outcomes,” says Katie Sterrett, senior director of leadership development and talent.

Solutions to improve performance are multifaceted and require collaborative effort. “We conduct after-action reviews with our clients and ensure that what is expected is delivered. In these reviews, we want to ensure that key learning and behaviors are being sustained,” says Gillece. “The biggest factor in making this happen is leadership. The participant’s direct supervisor plays a large role in ensuring that the learning sticks and is reinforced on the job.”

Tell the evaluation story during action planning in a compelling way, so that there is momentum and excitement to drive change. Tailor it as needed for different audiences. When delivering data to senior executives, for example, summarize key takeaways succinctly, and offer detailed data only when requested. A one-page infograph can illustrate your story to senior leaders at a glance and provide metric highlights that matter most to this busy audience.

Engage your stakeholders during action planning to decide together what actions to take. For example:

• Define what to stop, start, and continue.
• Improve learning transfer and line involvement.
• Make investment decisions based on data.
• Improve the value proposition for the learning function.

Going forward, business needs will change, learning modalities will shift, and evaluation priorities will continue to be transformed. Gillece emphasizes the need for agility: “As the composition of the workforce is changing, so will the learning approaches. People learn best by doing, and learning will take many shapes going forward. Traditional and e-learning will change, and mobile learning, informal learning, and new delivery methods will become the norm. The corresponding metrics will need to evolve to adapt to these changes.”

Regardless of the pace of change, implementing the LES provides a virtuous cycle based on enduring principles that will guide these efforts going forward.

Please submit your comments here…

Marjorie Derven

Marjorie Derven is managing partner of Hudson Research & Consulting, which specializes in strategic learning evaluation, learning design, and leadership development.