What is a summative evaluation?
A holistic vision on eHealth is essential: technology, context and stakeholders are intertwined. Consequently, it makes sense to conduct an evaluation that involves all of these concepts. Among other things, it is important to determine if the added value that was identified in the value specification was achieved, and if and what kind of influence the eHealth technology had on the broad issue described in the contextual inquiry. The role that the technology played in changing these kinds of matters has to be determined as well, and the perspective of the stakeholders should again be taken into account. Evaluation of an eHealth technology should focus on the actual effects on health, the influence on healthcare or related processes, the role and use of technology, and the perspectives of stakeholders.
Thus, summative eHealth evaluation should not just focus on its impact on health, healthcare and people, but also on the uptake of the technology [1]. The evaluation of the impact of an eHealth technology focuses on the values that were determined in the value specification phase. It aims to measure whether the intended effects in clinical (e.g., health status is improved), organizational (e.g., the hospital saves costs), and behavioural terms (e.g., due to an improved medication prescription protocol, MRSA is reduced) have been reached at a given point in time. Besides impact, summative evaluation also looks at the uptake of an eHealth technology in practice, which refers to the evaluation of how people and organizations have been using the technology. Was the technology used as was determined in the design phase, and was it implemented and used as described in the operationalization phase?
What is the aim of a summative evaluation?
The summative evaluation has two main objectives:
- Determining the impact of the technology on the context and stakeholders, based on the predetermined values.
- Analysing the uptake of an eHealth technology in terms of adoption or use of the technology by predetermined users and implementation, and use within the intended context.
What are the outcomes of a summative evaluation?
Besides these formative evaluations, evaluation is also performed to determine what has been achieved at a specific point in time, mostly after the technology has been operationalized in practice for a while. In the CeHRes Roadmap, this is referred to as summative evaluation, but other terms have been used as well. Such evaluations focus on various outcomes:
- Clinical outcomes: Indicate improvements in the health status of the users, for example, blood pressure, quality of life, depressive symptoms or blood glucose levels.
- Behavioural outcomes: Indicate changes in behaviour, such as prescription behaviour or levels of physical activity.
- Organizational outcomes: Indicate changes in the context in which the technology is used, for example cost-reductions in healthcare or more efficient working routines.
Such (backward) evaluations are conducted to assess whether the intended objectives of an eHealth technology have been realized.
Because of the use of multiple methods and perspectives, eHealth evaluation can be seen as a contextual inquiry of the new situation that includes the technology. Its main purpose is determining whether the intended values have been achieved, if other unexpected effects arose, and what could be further improved. The exact nature of the outcomes of the summative evaluation depends on the research questions and accompanying methods. However, one should always look at both the impact and uptake of the technology in a holistic manner, involving health, behavioural, and organization in the evaluation. In the contextual inquiry, the development team should already identify the desired impact and uptake.
The results of the evaluation methods can be used for several purposes. First of all, claims can be made about the effects of an eHealth technology. If the added value of a technology is apparent and objective, it will be easier to implement it on a larger scale. Also, results can be used to make changes to the technology itself. If it becomes apparent that some features of the technology aren’t used at all, or if users are dissatisfied with a specific element of the technology, this should be adapted. eHealth development is never finished. Furthermore, evaluation results can be used to form theory about eHealth, for example about the added value of persuasive features, or predictors of dropout. Finally, evaluation can also provide the development team with insight into their process: it might very well be that after evaluating, it becomes clear that certain activities should have been conducted in an earlier or later stage, that a suboptimal decision was made along the way, or that important information was overlooked. The development team can learn from these lessons. To conclude, evaluation is an essential part of eHealth development and should, just as the other phases, focus on the technology, stakeholders, and context.
References
[1] Van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011). A holistic framework to improve the uptake and impact of eHealth technologies. Journal of Medical Internet Research, 13(4): e111.