Monitoring and evaluation of digital health

Monitoring and evaluation of a digital health system answer two different questions. Monitoring asks whether the intervention is working as intended, while evaluation assesses the changes the intervention brings about. A hospital, an insurer or a start-up with a system in daily use needs both.

The World Health Organization's practical guide of 2016 gives step-wise guidance for both, for digital health interventions also known as mHealth or eHealth. It also ties what to measure to the stage a system has reached, from prototype to national scale.

Knowing that stage, and the four things to monitor at each, keeps the effort aimed at what the system needs next.

Monitoring and evaluation told apart

In WHO's guide, process monitoring is the continuous collection and analysis of data to compare how well an intervention is being implemented against expected results. For digital health, that means the routine collection, review and analysis of data, either generated by the digital system or collected on purpose. These data measure implementation fidelity and progress towards the intervention's objectives.

Evaluation, by contrast, is the systematic and objective assessment of an ongoing or completed intervention. It aims to determine whether objectives were met, and the intervention's efficiency, effectiveness, impact and sustainability.

In short, monitoring measures properties inside the digital health system, while evaluation looks at metrics outside it. Ideally, evaluation then shows that changes in those outside metrics occurred as a result of the intervention.

Monitoring is also iterative. It leads to adjustments in the intervention's activities, so as to maintain or improve the quality and consistency of the deployment.

Monitoring and evaluation across six stages of maturity

Digital health interventions evolve through several stages of maturity, and their monitoring and evaluation needs change rapidly with them. WHO's guide sets out six stages.

1. Stages 1 and 2, pre-prototype and prototype. Here the questions focus on the technology itself.
2. Stage 3, pilot. The questions now relate to the interface between the end user and the technology.
3. Stage 4, demonstration. Limited deployments aim to measure attributable impact on specific processes or outcomes, usually in controlled environments.
4. Stages 5 and 6, scale-up, then integration and sustainability. Larger deployments test effectiveness in non-research settings, without tight controls on delivery. They aim to measure cost and cost-effectiveness, or to identify the challenges of scale-up in policy or organisational change.

The four components of monitoring

The guide defines four major components of process monitoring for a digital health intervention. Each answers its own question.

• Functionality: the ability of the digital health system to support the desired intervention. It asks whether the system operates as intended.
• Stability: the ability of the system to remain functional under both normal and anticipated peak data loads. It asks whether the system consistently operates as intended.
• Fidelity: whether the intervention is delivered as intended. It asks whether the realities of field implementation alter the system's functionality and stability.
• Quality: seen from both the user's side and the side of the intervention's content. It asks whether the content and its delivery are of high enough quality to yield the intended outcomes.

The focus of process monitoring shifts as the intervention matures. Early on, monitoring centres on stability and functionality. As the intervention grows towards national scale, it moves to fidelity and quality.

Richnovex's fourth phase covers training staff, monitoring performance metrics, and handling post-go-live maintenance when needed, as its Optimisation & Support phase sets out. After go-live, the work turns to the people who run the solution: training staff, monitoring performance against the goals set in discovery, and maintaining the system when needed.

Objectives, indicators and data quality

For monitoring and evaluation, objectives in WHO's guide state results that are Specific, Measurable, Attainable, Relevant and Time-bound, known as SMART. Outcomes, in turn, should be measurable using indicators. The guide also asks for outcomes to be defined so that they produce the evidence that key decisions require.

Reviewing the data sources is critical as well. Poor-quality data, the guide warns, can undermine both monitoring and evaluation.

Monitoring and evaluation follow the system as it grows

Monitoring checks whether a digital health intervention works as intended, and evaluation assesses what it changes. Both shift with the stage the system has reached, from prototype towards national scale.

Sources

1. World Health Organization, Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment (2016): who.int, the guide