UTFacultiesBMSEventsFULLY DIGITAL - NO PUBLIC : PhD Defence Marjan Faber | Effects of digital formative assessment tools on teaching quality and student achievement

FULLY DIGITAL - NO PUBLIC : PhD Defence Marjan Faber | Effects of digital formative assessment tools on teaching quality and student achievement

Effects of digital formative assessment tools on teaching quality and student achievement

Due to the COVID-19 crisis measurements the PhD defence of Marjan Faber will take place online without the presence of an audience.

The recording of this defence will be added to the video overview of recent defences

Marjan Faber is a PhD student in the department ELAN Teacher Development. Her supervisor is prof.dr. A.J. Visscher from the Faculty of Behavioural, Management and Social Sciences.

In this dissertation it was investigated whether the use of data from student assessments can help accomplish higher levels of student achievement. More specifically, the question was whether teachers use digital systems for analyzing student assessments, and take instructional decisions based on the student assessment data, resulting in positive effects on student achievement. Teachers can use information from student assessments as feedback about the extent to which their instruction matches students’ instructional needs. In this dissertation the term digital formative assessment tool (DFAT) is used for digital tools which are used for obtaining, storing, organizing, and analyzing student assessment data, that support teachers during instruction planning by providing feedback to teachers that is based on student assessments.

In the first study of the dissertation feedback to teachers was given based on daily, non-standardized student assessments.

Study one: the effects of a digital formative assessment tool

In this investigation the effects were studied of a DFAT called Snappet on student achievement. The following hypotheses were tested:

·       The DFAT has a positive effect on mathematics and spelling achievement.

·       The DFAT is more effective when students and teachers use the tool to a greater extent.

·       The effect of the DFAT differs between low-performing, average, and high-performing students.

A randomized experiment was conducted to examine the effects of Snappet on mathematics achievement (1808 students) and on spelling achievement (1605 students) in grade three of primary education. The findings of the study indicate that a DFAT can have a positive impact on mathematics achievement and student motivation. However, a positive effect on spelling achievement was not found. To investigate these effect differences more in depth students’ intensity of data use (i.e., the total number of completed assignments and the percentage of adaptive assignments of all completed assignments) for spelling were compared with students’ intensity of data use for mathematics. This showed that students overall completed fewer Snappet assignments for spelling, than for mathematics. The percentage of adaptive assignments was also somewhat lower for spelling, than for mathematics. The difference in the number of completed assignments by students in the experimental groups compared to the number of completed assignments by students in the control group was presumably larger for mathematics, than for spelling, based on these students’ intensity of use data. Since we did not have data on the number of completed assignments in the control group, we could not test this assumption. Furthermore, high-performing students completed more assignments, than lower performing students and this difference was somewhat larger for mathematics, than for spelling.

Our study findings also indicate that the DFAT was most effective for high-performing students. Students’ intensity of use measurements show that experimental students who used the tool to a greater extent (i.e., the total number of assignments completed) performed better, than experimental students who used the tool to a lesser extent (both for spelling and for mathematics). We do not know whether students who completed more Snappet assignments performed better on the posttests because they completed more assignments, or that higher performing students simply completed more assignments. This could imply that the mathematics achievement effect was mainly caused by the fact that experimental students completed more assessments, than students in the control group. Still, all students showed a higher learning gain in the experimental condition for  mathematics, than the students in the control condition and the positive difference in the mean achievement growth is highest for the 20% best performing students.

The effects of the feedback to teachers are most important for this dissertation. Teachers use information from a DFAT as feedback to improve instruction and student achievement is supposed to improve as a consequence. The relationship between teachers’ classroom observation scores and student achievement levels in the first study suggests that teachers use information from a DFAT as feedback, and that the effectiveness of a DFAT depends of how the feedback from DFATs are used by teachers.

Study two: differentiated instruction

In the second study the relationship between differentiated instruction (DI) and student achievement was investigated. In this study we tested the following hypotheses:

·       Students’ achievement levels are higher in classrooms of teachers who differentiate their instruction more.

·       Students’ achievement levels are higher in classrooms of teachers who pre-plan differentiated instruction more.

·       Students from different ability groups do not benefit to the same degree from a teacher who differentiates his/her instruction.

Two characteristics of DI were distinguished and measured. Firstly, DI should be planned and instructional decisions should be based on the analysis of student data. Secondly, what makes DI observable in the classroom is the variation in learning goals, instruction content, instruction time, assignments, and learning materials aimed at addressing varying learning needs. The first characteristic was measured by means of an analysis of teachers’ instructional plans (n=89), the second by means of classroom observations of teacher behavior (n=144), using three items from the International Comparative Analysis of Learning and Teaching (ICALT) instrument.

In the second study no positive effects of DI on student achievement were found. So, the effect of teachers using feedback from a DFAT to teachers in study one was not confirmed. An important difference between the DFAT in study one and the DFAT used in the second study is the frequency (high or low) and timing (immediately or delayed) of the feedback to teachers. The differences between the findings in the first and the second study could indicate that teachers make a better connection between the feedback they receive, the instruction they gave, and the instruction they are planning to give, if the time span between delivered instruction and received feedback is short. The DFAT in the first study and the DFAT in the second study also differed in terms of the content of the feedback provided by the DFATs. The DFAT in the second study offered teachers more options for complex analyses (e.g., information on students’ scores for specific subject matter components such as ‘dividing’ or ‘subtraction’ in mathematics) compared to the analysis options of the DFAT in study one. However, no effect of teachers using feedback from a DFAT on student achievement was found in the second study, whereas in the first study the teacher observation score seemed to indicate an effect on student achievement.

The findings from the second study also indicate that students from different ability groups do not profit from DI to the same extent. This finding is in line with previous research, showing that ability grouping can have a negative impact on the achievement of students in the low-ability groups. It can be effective for students in the average-ability groups and have no impact on students in the high-ability groups.

In the second study teachers by means of the DFAT analyzed students’ assessment scores on a standardized, summative test that is taken twice a school year. Teachers were trained to use the results, among other things, for determining the composition of the student ability groups they work with in their classrooms. Using the DFAT for composing student ability groups was an important training component, however, perhaps not enough attention was given to the importance of using ability groups flexibly (changing the composition of the groups if, based on new information, there is reason to do so) and to using multiple data sources for composing the ability groups. Overall, the findings from the two studies seem to point to the importance of immediate feedback to teachers. This may also stimulate teachers to compose student ability groups more flexibly.

In the third study a meta-analysis was conducted to further study the critical characteristics of DFATs.

Study three: DFAT features and intervention features

The aim of the this study was to investigate whether the results of high quality empirical studies in primary and secondary education confirm that the use of DFATs by teachers is effective. The meta-analysis of the studies answered the following two questions:

·       Is there an effect of teachers using a DFAT on student achievement?

·       Which DFAT features, and which DFAT intervention features influence the effect on student achievement?

After a literature search 91 studies were screened by using seven eligibility and methodological inclusion criteria. Seventy-seven studies did not meet all inclusion criteria, the remaining 14 studies were included in the meta-analyses. The 14 studies were coded on four DFAT feedback features (i.e., feedback frequency, feedback content, feedback level, predictive feedback) and three intervention features (i.e., intervention frequency, intervention content, intervention target). The coding resulted in three groups of DFATs and interventions (A, B and C).

The findings of this third study indicate that the use of a DFAT has a small positive effect on student achievement. More specifically, positive effects were found for DFATs in primary education, for mathematics in primary and secondary education, and for DFATs in group C. DFATs in group C can be distinguished from DFATs in groups A and B on three important characteristics. First, the feedback frequency in group C was high compared to the frequency of the feedback of the DFATs in groups A and B. Second, the effectiveness of DFATs in group C might be explained by the fact that the DFATs and interventions in group C were more suitable for, and more targeted at those who are actually responsible for changing instruction (i.e., teachers) instead of those who are further away from the instructional process (i.e., school principals and members of the school board). DFATs were also coded on the intensity of the intervention and on the content of the interventions meant to support the implementation of the DFAT. Surprisingly, intervention intensity and intervention content for the DFATs in group C were often less comprehensive than most of the interventions categorized in group B. A potential explanation for this finding might be that the complexity of the feedback content, the feedback level and the absence of predictive feedback in DFATs categorized in group C made the feedback more accessible and easier to digest for teachers. The feedback content of the DFATs in group C was often easy to interpret and detailed (i.e., information on students’ mastery of specific instructional content). Relatively simple feedback content may make a more comprehensive intervention in terms of intensity and content less necessary.

General conclusions

DFATs can have a positive effect on student achievement. The results of the classroom observations in study one seem to indicate that teachers use information from a DFAT as feedback to evaluate the effects of their instruction, and as feedback for planning follow-up adaptive instruction. However, the results of the second and third study show that positive effects on student achievement are not always accomplished. The results of the three studies together give indications of what characteristics of effective DFATs might be;

·       frequent and immediate feedback to teachers, that is also

·       detailed and concrete, and

·       accessible and easy to interpret for teachers.

These characteristics may make a DFAT suitable for improving teachers’ daily instructional planning and make it easier to integrate a DFAT into teachers’ teaching practices.