Formative Evaluation

Formative evaluation is not a separate development phase, but a principle that is intertwined throughout the eHealth development process. Within the CeHRes Roadmap, it is visualized as the cycles that are connected to all five phases. The eHealth development process is not linear, but iterative and dynamic, and a formative evaluation represents this. In this, all development activities are interrelated and outcomes and decisions can constantly be adapted. The formative evaluation can be focused on including stakeholders' perspectives, and on checking the outcomes of the previous phases.

The basic presumption of formative evaluation is that ongoing information on how to improve the process and eHealth technology is collected. This information assists the development team in ensuring that there is a constant focus on the context and people involved. In this way, formative evaluation can be seen as creating by evaluating. Using input gathered by different methods to improve the development and design process is the key.

To achieve this, formative evaluation is twofold. On the one hand, it can be used between development phases. At the end of a phase, the development team has to check whether the outcomes of a previous phase have been accounted for and haven’t been forgotten along the way. For example, in the design phase, the development team has to make sure that the designed technology incorporates the requirements and addresses the values. Besides this, the development team also has to decide on whether changes in the outcomes of previous phase are required. This might be the case when new information arises during the value specification which has implications on, for example, the issue or target group that was specified in the contextual inquiry. This is part of the iterative nature of eHealth development.

The other use of formative evaluation is within a development phase. This refers to activities that support the development team in ensuring that there is a fit between their activities or output and the perspective of the stakeholders and context. An example of this is the constant involvement of stakeholders in decision-making, or interviewing potential end-users to elicit requirements instead of the development team autocratically coming up with them. These kinds of activities should be conducted in every phase of eHealth development.

What is the aim of a formative evaluation?

The development of eHealth is an iterative process consisting of several intertwined development, implementation and evaluation phases. Within this process, eHealth development requires continuous evaluation cycles, both 1) during eHealth development, and 2) at the end of the development and implementation, which is often referred to as formative evaluation. 

The formative evaluation has two main goals, both related to the development process:

• Checking whether the outcomes of previous phases have been accounted for in the current phase, and that the outcomes of all phases are related to each other.
• Using methods to gather information from the stakeholders and context to continuously include their perspectives in the activities and outcomes of the development, implementation and evaluation.

What are the outcomes of a formative evaluation?

These continuous, formative evaluations during eHealth development can provide information on how to improve the design and the implementation of the technology. This type of evaluation has two main goals. First, it can be conducted as a backward evaluation, which occurs at the end of a specific development phase to check whether the goals as set at the start of that phase have been reached and have been accounted for. Second, it provides information on the current state of the development process and how to make improvements before moving on to the next phase (forward evaluation). The results of these evaluations can be used to improve the technology’s design, activities, or performance, and to make sure that the technology fits the users, other stakeholders, and the context. By these means, the best possible technology for prospective users in a certain context can be created. Therefore, evaluation during eHealth development and operationalization is a way of ‘creating by evaluating’.

The formative evaluation of eHealth is iterative and is the backbone of the development of eHealth technology. Formative evaluation aims to monitor whether a project is still on track, detect problems as early as possible, solve these problems as soon as possible, and to make sure that the relationship between the technology, stakeholders and context is guaranteed. In this way, formative evaluation can be used to identify potential implementation problems and to tackle them in early phases of development. Because of this broad scope, there are no concrete outcomes, since formative evaluation has no fixed endpoint and is part of the development, implementation and evaluation of eHealth.

Check Whether the Outcomes of Previous Phases Have Been Accounted For

What is checking the outcomes of previous phases?

This aspect of the formative evaluation mainly focuses on ensuring that there is a clear relationship between the content and output of the separate phases. The responsibility for this lies with the development team. No concrete methods are prescribed for this, it depends on the nature of the project and the preferences of the project team. However, it is essential that every member of the development team is aware of the decisions that are being made. Another requirement is that in every eHealth development project, a thorough, transparent documentation of the activities and outcomes of each phase should be created for a proper formative evaluation. If the documentation isn’t clear, it is hard to retrieve what the main outcomes of previous phases were, and what the grounds for specific decisions were. Deciding on whether the outcomes of the current phase match those of previous phases can be achieved by means of methods like focus groups or meetings with the development team and/or stakeholders.

What is the aim of checking the outcomes of previous phases?

The aim of this sub-phase is to continuously check the outcomes of the distinct phases by using a number of methods. This allows to adjust the development process while the technology is still being developed.

What is including stakeholders' perspectives?

The formative evaluation does not have separate methods since it is an overarching term that refers to principles behind the methods used during development. To illustrate and further clarify this, some practical examples of methods that convey the principles from formative evaluation are provided below.

• In the contextual inquiry phase, snowball sampling is used to ask existing stakeholders to identify missing stakeholders. This is an example of formative evaluation, since it assists in validating whether the list of stakeholders is complete and reflects the actual context.
• In the value specification phase, lists of requirements are often verified by stakeholders once they are drawn up. This assists the team in ensuring that the requirements they elicited from focus groups and interviews are still relevant and make sense to the stakeholders.
• In the design phase of the technology, a straightforward example of formative evaluation is usability testing among users via the think-aloud method. This method aims to make sure the technology fits the user’s opinion.
• During the operationalization phase, the involvement of stakeholders in finishing up the business model and making a plan on how to implement the model is an example of formative evaluation. Involving stakeholders in the implementation increases the chances of the implementation strategies fitting the context. 

What is the aim of including stakeholders' perspectives?

In the previous sub phase, it was monitored whether a project is still on track to detect problems as early as posible and solve these problems as soon as possible. In this sub phase, the aim is to make sure that the relationship between the technology, takeholder and context is guaranteed. In this way, formative evaluation can be used to identify potential implementation problems and tackle them in early phases of development.