A member of my family works for a large semi-industrial company that expends extraordinary effort to ensuring worker safety. They have a system where any sort of safety issue, right down to a paper cut, is logged into a central system and plans are developed to prevent the issue’s reoccurrence. On the surface this sounds great, a semi-government semi-industrial company that encourages employee safety over all else. Unfortunately the implementation of their safety strategy has two serious flaws.
- The divisional managers of this company are responsible for ensuring every issue relating to safety is logged into the central safety system.
- One of the key performance indicators (KPI) of these divisional managers is their safety record which is calculated from the number of issues logged into the central safety system.
There was a minor incident at this workplace recently that, while not leading to an injury, demonstrated a potential safety hazard that need to be rectified. As the manager responsible for this area has a KPI linked to their safety record, the incident did not get logged with the central system. This is a classic example of task corruption and Goodhart’s law, both of which, I think, are going to be issues associated with the burgeoning interest in academic analytics.
At our institution, the Indicators project has recently attracted the attention of upper management who would like us to do a series of presentations to staff showing the link between student behaviours within the learning management system (LMS) and the student’s resulting grade. In one respect this a good thing, as I firmly believe that the path to better learning and teaching starts with informing and improving the teaching staff. However it is the potential of task corruption and Goodhart’s law that worries me when management takes an interest in these things.
The predictive value of academic analytics is not a precision instrument. Comparing the current term’s student behaviours with previous terms is really only useful as a guide as it disregards context to some extent. For example every student will use the LMS in different ways and even the same student will use the LMS differently depending on the type of course, type of assessment and what is happening in their lives at that particular point in the term. Another example is a student who is completely lost in a particular course will make a large number of clicks within this course, equivalent to a high-achieving student and still fail even though their ‘measurable’ effort indicates a much higher grade. This is why we espouse academic analytics as a tool for the teacher and, perhaps a tool for the students, so that the information they are provided with, can be contextualized by them into their situation. However there is potential for task corruption and Goodhart’s law to become an issue in this situation.
Given that one of the correlations identified by the Indicators project shows that there appears to be a relationship between the teacher’s level of engagement in the LMS course and the student’s level of engagement in the same course, I worry about a KPI being introduced around a teacher’s level of engagement in online courses. In such a circumstance, the pragmatic educator would spend a short while making random clicks within the system, whereby achieving the metric based on their LMS activity, and then getting on with their work for the rest of the term having satisfied the compliance police. Task corruption 101. Ok. This is extreme and is probably not (I hope) a realistic scenario, but it does highlight the dangers of taking complex, context dependent data and using it as a performance metric.
One of the dangers as I see it, is that while academic analytics provides some great data for educators on what is happening in learning situations facilitated by an LMS, there may be a temptation to use it as some sort of performance measurement. Dave Snowden talks about complex and complicated systems where complicated systems are receptive to the concept of good practice whereas complex systems are not due to their ever-changing context. The point here is that academic analytics is data that results from activity occurring in a complex system and not a complicated system. Therefore ‘best practice’ and a one-size-fits-all approach to measuring learning using academic analytics is not likely to work.