Leveraging Analytical Tools to Transform L&D

The single biggest challenge facing learning leaders today is evaluating the effectiveness of their learning interventions. Data Analytics is all the rage today. There is a vast array of Analytical models, tools, and data science expertise, which are all helping transform businesses.

So, how do we leverage the latter to enable the former, and give L&D organizations some real tools to measure the effectiveness of their learning programs and use that data for continuous improvement and transformation?

The Challenges Faced by CLOs and Their Teams

L&D professionals understand that learning is crucial to business performance. However, proving the value of L&D to other parts of the organization, including business leaders and decision-makers, is an uphill task. Why do CLOs and their teams struggle in this area? The answer lies in the practical challenges they face in measuring the effectiveness of learning programs, and hence, the challenge in showing impact to business imperatives or performance.

There are frameworks to enable measuring learning effectiveness, with the most notable being the Kirkpatrick's Model. However, where organizations run into obstacles in leveraging these frameworks is the lack of real tools to put the framework to use. At each level of the Kirkpatrick Model, the first step would be to collect data, which in itself is a difficult task for beyond Level 2 for several organizations. As per a State of the ROI of Learning survey report by Udemy, most companies still rely on metrics at Level 1 of the Kirkpatrick Model. The majority of companies were measuring training satisfaction and completion rates. By relying only on Level-1 data, organizations lose out on measuring the impact of L&D on employee behaviour and KPIs, as well as on business-critical metrics.

Even with the availability of data, how does one go about analyzing the mass of information and generating valuable insights, and enabling decisions? That is the bigger challenge.

The Kirkpatrick Model

LEVEL 1: Reaction: Reaction of learners, typically captured through feedback forms or ‘smile sheets’; helps understand how much participants find the learning intervention engaging and relevant to their job

LEVEL 2: Learning: The extent to which learners acquire the intended knowledge, skill, attitude, from the learning intervention; typically captured through assessment scores

LEVEL 3:Behavior: The degree to which learners apply their learning from the learning intervention to their jobs; supervisor, mentor, and 360-degree feedback are some ways to capture this information

LEVEL 4: Results: The degree to which targeted outcomes occur as a result of the learning intervention; the impact to key metrics

This is where Data Mining and Analytics come into picture. The key objective is to start a thought process on how to address the challenge around measuring learning effectiveness by leveraging cutting-edge, new-age technology. How can organizations leverage Analytics to evaluate the effectiveness of their L&D interventions and drive continuous improvement?

Data mining and Analytics in L&D

Data mining is the process of finding anomalies, patterns, and correlations within large data sets to predict outcomes (https://www.sas.com/en_us/insights/analytics/data-mining.html).  Big Data analytics is the process of collecting, organizing and analyzing large sets of data (called Big Data) to discover patterns and other useful information. (https://www.webopedia.com/TERM/B/big_data_analytics.html )

So, how can L&D professionals leverage data mining and analytics? Figure 1 is a depiction of what analytics tools can be leveraged and at what stages of the Kirkpatrick's level.

Let us delve into a little more detail, with the Kirkpatrick's levels as the reference points.

Reaction

Organizations typically collect learner feedback either through physical or online survey forms. In the case of eLearning hosted on Learning Management Systems, feedback forms are often built into the eLearning.

The net result is a host of data (questions that ask learners to score parameters, for example, relevance of course content, ease of understanding of concepts, interactivity or engagement, etc. on a 5- or 10- point scale) and/or a lot of unstructured information through comments on various parameters in the questionnaire or survey forms.

For this kind of unstructured data, two analytics tools in tandem are extremely useful: Text Mining and Sentiment Analysis. Imagine a Python Sentiment Analyzer that takes input from the various feedback sources, the algorithm generates sentiments from the required or specified surveys, throws up key themes, and generates output that is continuously appended to the previous analysis for the same training program. Then, it generates dashboards with meaningful, insightful data, and also shows trends over time.

Let us take an example. Let us say a large global organization conducts a New Manager Induction program for people who have been newly promoted to managerial or supervisory roles from individual contributor roles. Feedback is gathered and run through the Python sentiment analyser every time the training is conducted. From each session, the training team gets instant insights into the key themes, what worked well, what needs improvement, etc. Insights can be generated in numerous ways, such as geographically and evaluation parameter-wise. So, for example, course content might be working well across geographies. Logistics might be working well in one geography, but might be a pain in another. Learner engagement might be a pain across.
The text mining reads each sentence in the verbatims from the feedback forms, breaks, chunks, and organizes the information from them, and throws up key themes in visual formats, such as pain areas in red and successes in green

This, now, is something that L&D teams can actually work with! They now know specifically what aspects are working well, what are not, say, which geography/group/department is doing better and what can others imbibe from them, and a host of other meaningful insights. This helps them make continuous improvements and with time.

Learning

Learning Management Systems (LMS) today are able to generate a host of useful insights from Learner-generated data. They are able to generate individual as well as group proficiency dashboards through processing pre-and post-assessment scores of learners or assigned target score levels and achieved score levels. Reports can also be generated to compare scores manager-wise, department or group-wise, location-wise, or trending of scores over time. There are also Artificial Intelligence (AI) powered LMS’ that scan the host of data and visually categorize scores, depict trends, and flag anomalies.

This empowers L&D teams much more than traditional LMS that give insights only into time spent by learners over a course, individual learner scores, and most popular courses. Thus, L&D teams can quantify the ‘learning’ that has taken place in the organization as a result of L&D interventions.

Behavior and Result

This is where Artificial Intelligence and Predictive Analytics come in!
Data on behavior and results over time are typically derived from outside the LMS, from systems such as HR Management Systems (HRMS) or performance evaluation, customer surveys, 360 surveys, etc. Today, even external data sources can be plugged into the learning system to provide a more accurate employee profile. Systems such as Adobe Captivate Prime have features that enable automatically importing user details from HRMS or other applications, such as SalesForce Dot Com (SFDC), into Adobe Captivate Prime.  Thus, the system recommends learning paths based on a combination of assessment scores and performance data. This is the essence of Predictive Learning.

Predictive analytics is embedded in several learning platforms today. Platforms like EdCast and several others apply Artificial Intelligence (AI) and Machine Learning (ML) algorithms, then provide corporate learners with the online training resources they require. The resources are pulled from the LMS as well as external sources, such as YouTube videos, TED talks, and learning libraries on the web. As a result, every member of your team is able to pursue personalized online training paths to bridge gaps and improve workplace performance. Learners are given recommendations based on their performance on key metrics, assessment scores, what courses they are viewing already, what courses others with similar roles in the organization, similar learning needs, or similar learning interests are taking, etc.

The result is wholesome learning, integrated with performance and behavioral aspects. Behavioral and performance changes can be input back via the manager evaluation and other HRMS components and training paths re-adjusted.

What do organizations need to kickstart and sustain leveraging analytics for learning?

L&D must not shy away from asking for support. They will need to work closely with Big Data or Analytics experts to see where they are in terms of the quantity and quality of learning data, what improvements need to be made and the time and effort that will entail. It is the data that is key to showing results. They will also need to work with technology experts to understand what exists in the organization, to evaluate what features their LMS supports (assuming there is an LMS in place) and whether they are optimally utilizing the LMS, and also evaluate whether a change in LMS implementation is required. Having an LMS that integrates with organizational HRMS, curates and aggregates content from external sources, and employs AI and ML for personalized learning is the way forward.

All this would ideally be followed up with an implementation roadmap, associated budgets, and relevant stakeholders will need to be enlisted for approvals. Central to this is ensuring the L&D team is committed to driving business impact by understanding the value of training through effective training evaluation practices.

Benefits of leveraging data mining and analytics in L&D

Having the right data and accurate insights into learning effectiveness will help drive continuous improvement and performance enhancement across the business. Through these tangible contributions to the business, L&D can establish itself as not just a cost center but a key business partner.

Insights on skill levels, alignment of employee skills to business needs, and the impact of learning on key organizational metrics can enable more informed decision-making. L&D can more effectively determine what courses need to be designed and delivered, as well as larger decisions on hiring, staffing, and competency development.

By leveraging data mining and analytics in L&D, business stakeholders can see how the learning organization is addressing and impacting key business imperatives, including operational efficiency, employee performance, and ultimately business results.

Note: This article first appeared in the August edition of the Training Industry magazine. Link: https://trainingindustry.com/magazine/jul-aug-2019/leveraging-analytical-tools-to-transform-ld/