I have been thinking a lot lately about what we mean by the use of the term “real-world evidence” in the CAPTURE acronym. Is this about the collection of evidence from the “messy” real-world or the use of evidence in the “messy” real world?
In the beginning, when the acronym was being developed and our ideas were just coming together, we used this as a way to evoke the notion of “practice-based evidence” a term coined by Larry Green. As he points out, most evidence is not very practice-based and the evidence given the most credence comes from highly controlled trials that are ineffective for taking into consideration the diverse circumstances of public health practice.
What do people who work in the real-world of public health practice think about the term “real-world evidence”? I was treated late last year to an opportunity to engage with the wonderful folks who work in public health in the Peel Region. Thanks to Megan Ward and David Mowatt I was able to engage with about 75 of the Peel Region staff on the topic of evidence, the barriers to using evidence and the kinds of support they need to collect evidence from their own practice. When asked “What does the term real-world evidence mean?” I heard things like:
- Experiential information
- Information from the actual population he/she is working with
- Evidence that relates to the specific program being implemented
- Local knowledge
- Contextual knowledge
- Qualitative data
- Evidence that has been applied to practice
- Non-academic knowledge
These answers suggest that public health practitioners are thinking about evidence they can use to inform their own practice in the real world they work in. They seem to want evidence that is collected in their “messy” real-world and they implied this would be useful to inform their real world practice. It was interesting to note that they distinguished this from “academic” knowledge. But when asked “what are their barriers to collecting, sharing and using evidence?”, they mostly spoke about the challenges they had in finding the evidence in the academic literature. It seemed to me there was a disconnect between the desire for “real-world” evidence, what they look for when seeking evidence and the recognition that to have the kinds of real-world evidence they think would be useful they need to be collecting it themselves.
One challenge in overcoming this barrier to having real-world evidence will certainly be capacity both in terms of time and skill sets, but another major issue is the perception that real-world evidence is inferior. Most evidence hierarchies put this kind of evidence in the middle or at the bottom of the pyramid, and the rigorous criteria for inclusion of data in systematic reviews means that very little of the evidence that might be generated in real-world situations is collected into syntheses or serves as the basis for recommended actions.
Of course no one wants to be using bad evidence or evidence that points us in the wrong direction, but what should be the criteria for validity when we are trying to build a database which will enhance our understanding of what works, for whom and under what conditions…. in the real-world?
It seems to me we need to have a broad perspective on the value of different kinds of real-world evidence. Since this means CAPTURE might collect and share evidence from a variety of sources we will need multiple ways to tag evidence to inform users about the “validity” or “utility” of data they might retrieve or that we might use for the automated reports that get generated. If you know of a taxonomy or set of criteria for evidence that would help us with this challenge we would love to hear about it.
I keep thinking about Bernard Choi’s comment during our October consultation that someday people might say “Why Google when you can CAPTURE?” I love this idea, but it suggests to me that people will want to get the widest possible set of answers sorted in a way that they will find the most useful to their purpose. What does this mean to you in terms of the nature of the evidence we want to have in the CAPTURE database and perceived utility of CAPTURE as a resource?
