When implementing a digital learning initiative in a corporate setting collecting and analyzing the digital footprints of users could answer several questions to inform future design and development decisions. In my original post, I looked at David Cormier’s experience with change in an education setting and considered the validity of his recommendations in a corporate learning setting. Once again, I will borrow from the field of education’s experience with learner analytics and consider the potential benefits for my corporate training setting.
Data provided from our current learning management system (LMS) has proven useful. After implementation of a new online offering, tracking how many users logged in and how many users logged in and completed the course is data that we use to gauge accessibility and digital capabilities. This data allows us to reach out to the users who do not log in to find out why and create supports. Some users have no access to devices, some are intimidated by technology use and, some do not receive the email communication. As a result, we can offer individual tech support as needed and look for new ways of communicating including social media and bulletin boards. This is just one way of analyzing and interpreting the data provided by the LMS activity to help future planning.
In higher education, learner analytics are used to bring awareness to students at risk of dropping out or passively failing a course (Sclater et al., 2016) and in a corporate setting, this data might bring the same benefit to increasing employee retention rates if there is indeed a correlation between employee engagement in learning and job satisfaction. According to Gallup’s State of the American Workplace report “Employees who are engaged are more likely to stay with their organization, reducing overall turnover and the costs associated with it. They feel a stronger bond to their organization’s mission and purpose, making them more effective brand ambassadors…” (Eagle’s Flight, n.d.) So, planning digital learning initiatives that support engagement and collecting and analyzing data that shows engagement levels may be an impactful part of an overall employee retention strategy.
Analyzing the data further, we can determine how many users completed the first offering and did not complete any further. Did they find it too difficult to navigate? Did they not find the content relevant or valuable? This data will also be helpful in building a business case for a more accessible, user-centered learning management system or a different approach to the digital learning environment completely. Sclater et al. (2016) discussed the potential of learning analytics to inform program development and improved practices. Data collected through learner analytics that illustrate user behavior such as time spent interacting, resources viewed, and assessment results may be complementary to the traditional survey style feedback our organization uses to gauge employee satisfaction of online learning offerings.
From an ethical and privacy standpoint, how the data is used and disseminated needs to be considered. For example, if grades or attempts are tracked and provided to a user’s manager, could the data be misinterpreted and create judgments or biases against an individual? If a manager sees a user who attempted an assessment four times and failed judge the competence of that employee? Perhaps there was a technical or accessibility issue that caused the results. Also, it must be considered, how, where, and, to what extent employees are being expected to interact in the digital learning environment. In our organization (6000 employees, the majority are drivers and do not work in offices), is the use of personal email addresses and personal devices an acceptable expectation? Company policy regarding data use must be carefully considered, and the policy must be clearly communicated with the employees. I believe it would be beneficial to an organization to create an in-depth employee perception survey to gather data about how and to what extent employees feel comfortable with the collection and data use associated with learner analytics in their organizational context.
References
Sclater, N., Peasgood, A, & Mullan, J. (2016). Learning analytics in higher education: A review of UK and international practice. Jisc. https://www.jisc.ac.uk/sites/default/files/learning-analytics-in-he-v3.pdf
The Link Between Employee Engagement and Staff Retention. (n.d.). Eagles Flight. Â https://www.eaglesflight.com/resource/the-link-between-employee-engagement-and-staff-retention/
Hi Melissa,
Some really good insights about how analytics could be used to help improve training and programming and also employee engagement. You also raise some excellent points, that I think are particularly salient about privacy and possible surveillance within work learning settings. How is the data used, and who can use it? How can an organization ensure that it is not misinterpreted, or misused, as it only provides one small window into a user’s actual experience with that technology. There was a lot of talk early in the pandemic about Microsoft’s “productivity” scores, that would provide various metrics on how often employees were doing certain tasks (working on documents, collaborating et cetera) – and how intrusive that could be (employees learned to also work around it). I think this links to many concerns with the amount of data being collected as we engage in many of the tools we use for both work and study – and how organizations may need to consider policies and procedures around their uses. My colleague Brenna wrote a long post about it and the use of student data in various forms. I will post the link below if you want to delve further.
https://digitaldetox.trubox.ca/digital-detox-4-habits-data-and-things-that-go-bump-in-the-night-microsoft-for-education/
After reading about how leaders can leverage data for decision-making, I better understand how data can be used to support decision-making from a teaching perspective. The following post discusses my interpretation of data used in a higher education context. Like your post, I also look at data from a current course learning management system (LMS). The data represented one day in one course (LNRT525) and was provided by the instructor (Michelle Harrison). My interpretations provide similar insights to your discussion of an LMS in a corporate setting. First, learning analytics can help inform how modules are taught and assessed within an LMS (The Open University, 2015). Data analytics needs to start with a good question in mind (KelloggInsight, 2015). For example, data analytics might answer questions about the effective timing for instructor engagement compared to when learners most often frequent the LMS. The data may also answer questions about what learners are working on at a given time. In the LNRT525 LMS example, the data suggests that learners are working on a team learning activity, with 1006 views by 21 users on October 24, 2022 – the highest number of views and users in the learning community report. By contrast, the teamwork resources folder had 18 views by seven users. So, although users work on a team assignment, learners may not be using the team resources. Third, data can provide actionable information to inform responsive decisions (Marsh et al., 2006). In the LNRT525 example, the data may prompt the instructor to orient students to the team resources in the current and future courses. Alternatively, if students were not accessing the team assignment folder, send a reminder or offer support. LMS data may also help to identify individual students who are at risk because they are not engaging in the course (Prinsloo & Slade, 2014). However, data analysis relies on group selection and comparison (KelloggInsight, 2015). There are many unknowns in the LNRT525 example, and I have questions about the data collection timing, duration, and the complexity of how the groups access the LMS collectively.
Lastly, the LMS data provides insight into how to enhance the effectiveness of learning technology during and after a course to support student learning. However, the data represents a glimpse of the course learning activities, representing one day, and may not account for group interaction rather than individual users. In sum, decision-making must consider the intricacies that may not be interpreted from the LMS data alone (Prinsloo & Slade, 2014).
References
KelloggInsight. (2015, May 1). A leader’s guide to data analysis: A working knowledge of data science can help you lead with confidence. [Blog post]. https://insight.kellogg.northwestern.edu/article/a-leaders-guide-to-data-analytics/
Marsh, J., Pane, J., & Hamilton, L., (2006). Making sense of data-driven decision making in education: Evidence from recent RAND research. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/occasional_papers/OP170.html
Prinsloo, P., & Slade, S. (2014). Educational triage in open distance learning: Walking a moral tightrope. International Review of Research in Open and Distance Learning, 15(4), 306–331. https://doi.org/10.19173/irrodl.v15i4.1881
The Open University. (2015). Using information to support student learning. The Open University. http://www.open.ac.uk/students/charter/sites/www.open.ac.uk.students.charter/files/files/ecms/web-content/using-information-to-support-student-learning.pdf