Formative Evaluation

What is a formative evaluation?

Formative evaluation is not a separate development phase, but a principle that is intertwined throughout the eHealth development process. Within the CeHRes Roadmap, it is visualized as the cycles that are connected to all five phases. The eHealth development process is not linear, but iterative and dynamic, and a formative evaluation represents this. In this, all development activities are interrelated and outcomes and decisions can constantly be adapted. The formative evaluation can be focused on including stakeholders' perspectives, and on checking the outcomes of the previous phases.

The basic presumption of formative evaluation is that ongoing information on how to improve the process and eHealth technology is collected. This information assists the development team in ensuring that there is a constant focus on the context and people involved. In this way, formative evaluation can be seen as creating by evaluating. Using input gathered by different methods to improve the development and design process is the key.

To achieve this, formative evaluation is twofold. On the one hand, it can be used between development phases. At the end of a phase, the development team has to check whether the outcomes of a previous phase have been accounted for and haven’t been forgotten along the way. For example, in the design phase, the development team has to make sure that the designed technology incorporates the requirements and addresses the values. Besides this, the development team also has to decide on whether changes in the outcomes of previous phase are required. This might be the case when new information arises during the value specification which has implications on, for example, the issue or target group that was specified in the contextual inquiry. This is part of the iterative nature of eHealth development.

The other use of formative evaluation is within a development phase. This refers to activities that support the development team in ensuring that there is a fit between their activities or output and the perspective of the stakeholders and context. An example of this is the constant involvement of stakeholders in decision-making, or interviewing potential end-users to elicit requirements instead of the development team autocratically coming up with them. These kinds of activities should be conducted in every phase of eHealth development.

What is the aim of a formative evaluation?

The development of eHealth is an iterative process consisting of several intertwined development, implementation and evaluation phases. Within this process, eHealth development requires continuous evaluation cycles, both 1) during eHealth development, and 2) at the end of the development and implementation, which is often referred to as formative evaluation. 

The formative evaluation has two main goals, both related to the development process:

What are the outcomes of a formative evaluation?

These continuous, formative evaluations during eHealth development can provide information on how to improve the design and the implementation of the technology. This type of evaluation has two main goals. First, it can be conducted as a backward evaluation, which occurs at the end of a specific development phase to check whether the goals as set at the start of that phase have been reached and have been accounted for. Second, it provides information on the current state of the development process and how to make improvements before moving on to the next phase (forward evaluation). The results of these evaluations can be used to improve the technology’s design, activities, or performance, and to make sure that the technology fits the users, other stakeholders, and the context. By these means, the best possible technology for prospective users in a certain context can be created. Therefore, evaluation during eHealth development and operationalization is a way of ‘creating by evaluating’.

The formative evaluation of eHealth is iterative and is the backbone of the development of eHealth technology. Formative evaluation aims to monitor whether a project is still on track, detect problems as early as possible, solve these problems as soon as possible, and to make sure that the relationship between the technology, stakeholders and context is guaranteed. In this way, formative evaluation can be used to identify potential implementation problems and to tackle them in early phases of development. Because of this broad scope, there are no concrete outcomes, since formative evaluation has no fixed endpoint and is part of the development, implementation and evaluation of eHealth.