Summative Evaluation

A holistic vision on eHealth is essential: technology, context and stakeholders are intertwined. Consequently, it makes sense to conduct an evaluation that involves all of these concepts. Among other things, it is important to determine if the added value that was identified in the value specification was achieved, and if and what kind of influence the eHealth technology had on the broad issue described in the contextual inquiry. The role that the technology played in changing these kinds of matters has to be determined as well, and the perspective of the stakeholders should again be taken into account. Evaluation of an eHealth technology should focus on the actual effects on health, the influence on healthcare or related processes, the role and use of technology, and the perspectives of stakeholders.

Thus, summative eHealth evaluation should not just focus on its impact on health, healthcare and people, but also on the uptake of the technology [1]. The evaluation of the impact of an eHealth technology focuses on the values that were determined in the value specification phase. It aims to measure whether the intended effects in clinical (e.g., health status is improved), organizational (e.g., the hospital saves costs), and behavioural terms (e.g., due to an improved medication prescription protocol, MRSA is reduced) have been reached at a given point in time. Besides impact, summative evaluation also looks at the uptake of an eHealth technology in practice, which refers to the evaluation of how people and organizations have been using the technology. Was the technology used as was determined in the design phase, and was it implemented and used as described in the operationalization phase?

What is the aim of a summative evaluation? 

The summative evaluation has two main objectives:

• Determining the impact of the technology on the context and stakeholders, based on the predetermined values.
• Analysing the uptake of an eHealth technology in terms of adoption or use of the technology by predetermined users and implementation, and use within the intended context.

What are the outcomes of a summative evaluation?

Besides these formative evaluations, evaluation is also performed to determine what has been achieved at a specific point in time, mostly after the technology has been operationalized in practice for a while. In the CeHRes Roadmap, this is referred to as summative evaluation, but other terms have been used as well. Such evaluations focus on various outcomes:

• Clinical outcomes: Indicate improvements in the health status of the users, for example, blood pressure, quality of life, depressive symptoms or blood glucose levels. 
• Behavioural outcomes: Indicate changes in behaviour, such as prescription behaviour or levels of physical activity.
• Organizational outcomes: Indicate changes in the context in which the technology is used, for example cost-reductions in healthcare or more efficient working routines.

Such (backward) evaluations are conducted to assess whether the intended objectives of an eHealth technology have been realized. 

Because of the use of multiple methods and perspectives, eHealth evaluation can be seen as a contextual inquiry of the new situation that includes the technology. Its main purpose is determining whether the intended values have been achieved, if other unexpected effects arose, and what could be further improved. The exact nature of the outcomes of the summative evaluation depends on the research questions and accompanying methods. However, one should always look at both the impact and uptake of the technology in a holistic manner, involving health, behavioural, and organization in the evaluation. In the contextual inquiry, the development team should already identify the desired impact and uptake. 

The results of the evaluation methods can be used for several purposes. First of all, claims can be made about the effects of an eHealth technology. If the added value of a technology is apparent and objective, it will be easier to implement it on a larger scale. Also, results can be used to make changes to the technology itself. If it becomes apparent that some features of the technology aren’t used at all, or if users are dissatisfied with a specific element of the technology, this should be adapted. eHealth development is never finished. Furthermore, evaluation results can be used to form theory about eHealth, for example about the added value of persuasive features, or predictors of dropout. Finally, evaluation can also provide the development team with insight into their process: it might very well be that after evaluating, it becomes clear that certain activities should have been conducted in an earlier or later stage, that a suboptimal decision was made along the way, or that important information was overlooked. The development team can learn from these lessons. To conclude, evaluation is an essential part of eHealth development and should, just as the other phases, focus on the technology, stakeholders, and context.

References

[1] Van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011). A holistic framework to improve the uptake and impact of eHealth technologies. Journal of Medical Internet Research, 13(4): e111. 

What is determining the impact of an eHealth technology?

Several examples of research topics that determine the impact of eHealth technology are:

• What impact does the eHealth technology have on patient or consumer health and wellbeing?
• Improved clinical values.
• Improved quality of life, social functioning, or general and mental health and wellbeing.
• Improved lifestyle, self-care and self-monitoring.
• Improved attitude and increased knowledge, behavioural intention and actual behaviour.
• Improved compliance with advice or treatment adherence.

What is the impact of the eHealth technology on healthcare delivery?

• Increased availability of healthcare, independent of place and time.
• Increased efficiency: reduced utilization of (unnecessary) healthcare services, time savings.
• Increased safety by the reduction errors by healthcare professionals.
• Improved interaction between caregivers and patients, among caregivers or among patients.

What is the aim of determining the impact of an eHealth technology?

The results of the evaluation methods can be used for several purposes. First of all, claims can be made about the effects of an eHealth technology. If the added value of a technology is apparent and objective, it will be easier to implement it on a larger scale. Also, results can be used to make changes to the technology itself. If it becomes apparent that some features of the technology aren’t used at all, or if users are dissatisfied with a specific element of the technology, this should be adapted. 

What is determining the uptake of eHealth technology?

Several examples of research questions that can determine the uptake of eHealth technology are:

How is the eHealth technology being used over a sustained period of time?

• How are the system’s features and functions used in terms of frequency?
• How are the system’s features and functions used in terms of usage patterns?

Who is motivated and capable of using the eHealth technology?

• Who are the ‘hardcore’ users?
• Who are the ‘drop-outs’?

What factors explain the usage behaviour?

• What are reasons for hardcore (sustained) use, or dropout? How do the features and functions of the technology support the use of the technology and the satisfaction with it?
• What elements of the technology contributed to the effects that were found?
• Did the users notice the technology’s persuasive elements? And did these persuasive elements work?

As mentioned before, multiple methods can be used, depending on the goal of the evaluation. Sometimes, the same method is used for both impact and uptake purposes.

What is the aim of determining the uptake of an eHealth technology?

The results of this sub phase may provide insight into the effects of an eHealth technology. Also, results can be used to make changes to the technology itself. If it becomes apparent that some features of the technology aren't used at all then this should be adapted. Furthermore, evaluatation can be used to form theory about eHealth, for example, about the added value of persuasive featres or predictors of dropout.