Estimated reading time: 4 minutes
Last week, I wrote about completion rates and whether they tell us anything meaningful about learning. I also considered a middle ground with “engagement metrics”. Time spent in modules, quiz attempts, discussion participation, and resource downloads. These feel more sophisticated than simple completion tracking.
But are we still measuring the wrong things?
The Engagement Mirage
Turning to engagement isn’t random; it’s rooted in decades of education research. Student engagement theory has established clear connections between engagement and learning outcomes. Students who participate in high-impact practices, interact with faculty, collaborate with peers, and dedicate time to academic activities exhibit better learning gains and higher retention rates.
These findings are robust and have shaped our understanding of effective learning environments. The problem isn’t the theory – it’s how we’ve translated engagement from education to corporate learning analytics.
Student engagement research measures meaningful participation, including contributing to class discussions, collaborating on projects, seeking help from instructors, and connecting coursework to real-world experiences. Corporate learning analytics measure platform interactions, including clicks, timestamps, module completions, and quiz attempts.
We kept the concept of engagement but changed what we’re actually measuring.
Engagement analytics reveal what people do with our content, rather than what our content does for them. Someone can spend twenty minutes on a page, attempt a quiz three times, and download every resource, but if they can’t apply the concepts next Tuesday when they need them, was that engagement meaningful?
This raises questions about what high engagement actually indicates. Are learners deeply processing content, or are they struggling with confusing materials? When someone spends minimal time accessing specific resources and moving on, does that signal disengagement or efficiency?
The disconnect between engagement metrics and learning outcomes suggests we might be measuring the wrong things entirely.
What Analytics Predict
The appeal of engagement metrics is theoretically sound. Self-Determination Theory suggests that intrinsic motivation (genuine interest and engagement) yields better learning outcomes than external compliance. Time-on-task research in education has shown correlations between the time spent and achievement. So measuring engagement feels like we’re measuring motivation and effort.
But we’re conflating different types of engagement. There are three types of engagement: cognitive engagement (deep thinking about the content), behavioral engagement (interacting with materials), and emotional engagement (interest and enjoyment). Most learning analytics capture behavioral engagement, such as clicks, scrolls, and submissions, while assuming it indicates the cognitive engagement that drives learning.
Most engagement metrics predict user experience satisfaction, not learning transfer. High time-on-task might indicate content that’s genuinely compelling, or content that’s confusing and requires multiple passes to understand. Quiz retakes could demonstrate determination to master the material, or they could reveal poorly written questions that trick people into incorrect answers.
Participation in discussions often correlates with personality traits and comfort with interaction (especially online) rather than learning depth. Some people think out loud; others process internally. We’re measuring communication preferences, not comprehension.
Return visits are more promising, especially when people revisit specific resources rather than browsing randomly. That suggests they’ve identified something helpful enough to reference again. But even then, we don’t know if they’re applying what they’re accessing.
What’s Missing from the Data
Cognitive Load Theory adds another wrinkle: high engagement might indicate cognitive overload. When learners struggle with extraneous cognitive load, such as navigating confusing interfaces, processing unnecessary information, or managing complex systems, they exhibit high behavioral engagement; yet, their actual learning suffers.
The most efficient learning often looks disengaged in our analytics. Someone who quickly finds the specific information they need, applies it successfully, and moves on appears to have low engagement. Meanwhile, someone wrestling with poorly designed content shows high engagement metrics while learning less effectively.
Engagement analytics capture everything except the thing we care most about: whether someone can do something differently because of the learning experience. They tell us about platform behavior, not performance change.
Transfer theory, how learning moves from the training context to the work context, is what we should be measuring. However, transfer is difficult to track, occurs over time, and depends on factors beyond the training itself. So, we default to engagement metrics because they’re available and immediate, even though they measure the wrong thing.
The gap between engagement and application is where most learning efforts fail. Someone can engage deeply with conflict resolution content but still avoid difficult conversations. They can master project management frameworks in theory, but continue missing deadlines in practice. The analytics will show successful engagement; the work will show no improvement.
Objective evidence of learning occurs in contexts that we rarely measure: the decision someone makes differently, the question they ask that they wouldn’t have thought to ask before, or the moment they recognize a pattern and respond more effectively than they would have previously.
Better Questions to Ask
Instead of asking “Are people engaging with our content?” what if we asked “Are people performing differently because of our content?”
Instead of measuring time spent, what if we tracked time saved, the moments when someone solved a problem faster because of something they learned?
These measures aren’t perfect either, but they point us toward outcomes rather than activities. They’re harder to capture in dashboards, but they might reveal more about whether our learning experiences truly matter.
What gaps have you noticed between engagement scores and actual performance change? I’m curious about the times when your analytics looked promising but the results felt disappointing – or when the data suggested low engagement but you saw real improvement in people’s work.
