I appreciate the indication from Marsh, Pane, and Hamilton’s research (2006) that anecdotal or quasi-qualitative data such as observations from “learning walks” and interviews may provide valuable information alongside quantitative data such as summative testing. I believe that gathering feedback from stakeholders throughout a change process is the key to initial planning plus iterations of prototype solutions. Change leaders must consider stakeholder perspectives in order to define the problem and goals and then gather data; however, data may be useless if gathered without intent or analyzed and interpreted without the application of knowledge (Zettelmeyer, 2015).
A hypothetical example that applies these principles might be a situation where participants are struggling with and/or submitting uncharacteristic feedback about one particular online course within a program. Data from participant surveys, LMS interactions, email communication, and interviews with representatives of all stakeholder groups might be used to gain empathy and define the problem and/or goals then ideate solutions. The organization could develop prototype modifications to the course and gather feedback to refine the prototype changes then test the changes while continuing to gather data via formal and informal feedback loops.
One challenge that I have experienced in processes like this is that we often seem to be “building the plane in flight” as it is often difficult or impossible to pause activity within an organization for the necessary duration to complete a change project; however, if sufficient attention is given to the initial stages, there may be less disruption.
References
Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006). Making Sense of Data-Driven Decision Making in Education. Evidence from Recent RAND Research. https://doi.org/10.7249/OP170
Zettlemeyer, F. (2015). A Leader’s Guide to Data Analytics. https://insight.kellogg.northwestern.edu/article/a-leaders-guide-to-data-analytics/
Hi Alisha,
I liked your comparison of many change initiatives to “we are building a plane in flight”…I think that is so true. We don’t pause – for many reasons, and that leads us to overlook key data, and to also not really plan our data collection that well. Considering analytics, we often just adopt what is provided within our systems (LMS or other), rather than build thoughtful data collection tools that might actually answer some of the questions we would really like answers to. In your example of trying to make curricular changes, we often might need some very specific data – but likely don’t have those tools at the ready and it can be challenging to even build in informal feedback loops. I wonder how we do build in that needed time at the beginning of the process? Thanks for your post!