I am beginning to think “going to meetings” is my drug of choice! Here I am, yet again on an airplane heading home from another really exciting meeting that has my mind going in all sorts of directions. In this case, The CAPTURE Project partnered with Kerry Robinson and her crew in the Knowledge Development and Exchange unit at the Public Health Agency of Canada and Barb Riley from the Propel Centre for Population Health, to put on a meeting focused on Creating a Culture of Evaluation for Learning and Improvement in Chronic Disease Prevention and Health Promotion.
I didn’t have anything to do with the invites, so I can say (without patting myself on the back), what a great group of people the team pulled together. I really enjoyed the perspective and ideas provided by many of the speakers and participants. For example, Dana Vocisano of the McConnell Family Foundation talked about their efforts to completely separate evaluation for learning from evaluation for accountability. This led through subsequent discussion to an idea for changing the feedback loop of evaluation in a way that has potential to help create a real paradigm shift. What if a funder sets the conditions of program refunding not on whether you did what you said you were going to do, but on what problems you identified and solved during the running of your program? Imagine having your success judged based on your ability to identify what went wrong and then fix it rather than on your ability to forecast what you will achieve long before you get started!
While I could easily go on talking about all the great ideas that came up during the meeting, I am sure we will be posting the meeting report when it is ready so I best get on to the actual point of this blog and that is what does systems thinking have to do with evaluation? Marla has been bugging me for some time about the topic and about how systems thinking applies to CAPTURE, but mostly my head space has been on systems thinking and other stuff. On the flight out I started to read what folks in evaluation like Meg Hargreaves and Bob Wilson have to say about systems thinking and evaluation. I think this reading primed the pump so that after my presentation on “What is Systems Thinking and Why is it Important?” a conversation with Jamie Gamble of Imprint Consulting Inc., caused the light bulb to finally go on!
One of my favourite systems thinkers is Donella Meadows. Her work on 12 places to intervene in complex systems has had a huge impact on the work of my research group, the Chronic Disease Systems Modeling Lab. We recently adapted this work into an intervention level framework with a more manageable 5 levels.
Intervention Level Framework
- Paradigm (System’s deepest beliefs)
- Goals and Rules (What the system is trying to achieve)
- Structure as a whole (Connectivity, information flows, resources, trust)
- Feedback and delays (Self-regulation, self-reinforcement, adaptation)
- Structural Elements (Subsystems, actors, and the physical structure of the system)
Up until now we have been using the framework to sort recommendations for action to address changes needed in food systems, obesity and chronic disease prevention. We used it prior to the conference to sort actions suggested by participants in a pre-meeting survey on changes needed to shift the culture towards evaluation for learning. The frequency of recommendations at each of these levels has us thinking about the optimal degree of effort at each of these levels to promote systems change.
Jamie’s interest in using this framework in his own evaluation work made me realize that when one approaches the evaluation of a complex system, be it a developmental, formative or utilization focussed evaluation one should consider the evaluation of the system at each of these intervention levels. Mostly we focus on the structural elements level since this is easier to get our heads around. But this framework for systems thinking suggests we need to also consider whether an intervention has changed any of the feedback loops that exist or caused new ones to appear. Evaluations might also benefit from a focus on system level variables such as connectivity and trust. Meadows points out that the goals of a system are not necessarily what people say they are, rather one needs to observe the system in action to discern its goals. Maybe evaluations should consider how stated and actual goals match up and surely an evaluation of a complex system would benefit from understanding the deepest held beliefs under which the system operates.
Okay Marla I finally get it, yes this framework probably could serve as a useful guide to evaluation of interventions, as well as a guide to planning interventions in complex systems…..now that my light has turned what’s next?
