How to choose outcomes measures: an example for mental health

Many care teams are asking what outcomes they should measure in order to paint an accurate picture of the quality of their care. This often leads to heated debate. Should they track process metrics? Clinical or patient-reported outcomes? Mandatory government measures? Measuring everything is simply impossible and would not make it easier to get to insights. Keeping 80/20 in mind, a handful of measures probably predicts the biggest part of the resulting quality.

The risks of choosing too few outcomes

One place they could look is the academic world. Unfortunately, publications in medical research are often limited to just a few meaningful endpoints on the outcomes of interest for the intervention in the study. This leads to an endless parade of usual suspects such as survival, response rate and more recently, Quality of Life.

But real life is messier than a research study. There are more stakeholders and factors at play than in a controlled setting. Focusing on too few outcomes has two main consequences.

The first is a high risk to miss out on important aspects of quality. Only measuring survival says nothing about what it cost to get to those survival rates. Focusing on 90 day readmission fails to paint the picture how the patient is doing long term. It’s hard to claim you meet quality standards if you have big blind spots on critical areas of the result.

The second consequence is related to human tendency to optimize what is measured. Measuring the quality of a procedure will inevitably lead to optimizing everything that surrounds that procedure to improve quality. But what if a big % of the procedures was not appropriate in the first place? Measuring the appropriateness of a procedure and its outcomes should go hand in hand.

Similar insights led to the creation of the Balanced Scorecard in business in the late 1980s. The focus on short term financial metrics as the single measure of success had led to underinvestment in the areas that mattered in the longer term for many companies. The parallel with healthcare is easy to make. To measure something as complex as quality of care, we should come up with a “balanced scorecard” of outcomes to track. A set of outcomes broad enough to make sure there are no blind spots on critical areas and balanced in such a way that optimizing for the metrics on that scorecard will not lead to detrimental effects.

The outcomes hierarchy

To help care teams with this exercise, the father of Value-Based Healthcare Michael Porter came up with the outcomes hierarchy in the early 2000s.

It is a simple framework to define a balanced set of outcomes measures for any given medical condition. Because we recently had several conversations with mental health focused digital health companies and virtual care providers about how to measure outcomes, we’ll take depression as an example. Please note that this is a made up example for illustration purposes; we’ll link to an actual set of depression outcome measures below.

Building further on the work of Michael Porter, there is an organization called the International Consortium for Health Outcomes Measurement (ICHOM) that recognized this need to define balanced sets of outcomes measures per medical condition. For several years now, they have brought together global teams of experts and gave them the assignment to come up with a consensus set for the medical condition of their expertise.

This has resulted in more than 40 of such sets of which several in mental health (Depression & Anxiety for both adults and children, Personality disorders, Psychotic disorders, Addiction) with many more in development. The teams compiling these sets get the explicit request of defining outcome measures that are easy to capture in routine care, often making use of common validated questionnaires such as the PHQ-9 and the GAD-7 for depression and anxiety respectively.

The goal of publishing these sets is to be able to benchmark quality of care and the underlying approaches. If just a handful of teams measure the same outcomes at the same time in exactly the same way, the data becomes comparable. When this happens we can study variation in outcomes and the causes of that variation in the care delivery process being the treatments, therapies and pathways followed.

Conclusion

Chances are high others have answered similar questions and came up with an answer already. And if not, frameworks exist to have a productive conversation with peers around what outcomes to measure.

‍