Effects of digital formative assessment tools on teaching quality and student achievement
Due to the COVID-19 crisis measurements the PhD defence of Marjan Faber will take place online without the presence of an audience.
The recording of this defence will be added to the video overview of recent defences.
Marjan Faber is a PhD student in the department ELAN Teacher Development. Her supervisor is prof.dr. A.J. Visscher from the Faculty of Behavioural, Management and Social Sciences.
In this dissertation it was investigated whether the use of data from student assessments can help accomplish higher levels of student achievement. More speciﬁcally, the question was whether teachers use digital systems for analyzing student assessments, and take instructional decisions based on the student assessment data, resulting in positive eﬀects on student achievement. Teachers can use information from student assessments as feedback about the extent to which their instruction matches students’ instructional needs. In this dissertation the term digital formative assessment tool (DFAT) is used for digital tools which are used for obtaining, storing, organizing, and analyzing student assessment data, that support teachers during instruction planning by providing feedback to teachers that is based on student assessments.
In the ﬁrst study of the dissertation feedback to teachers was given based on daily, non-standardized student assessments.
Study one: the eﬀects of a digital formative assessment tool
In this investigation the eﬀects were studied of a DFAT called Snappet on student achievement. The following hypotheses were tested:
· The DFAT has a positive eﬀect on mathematics and spelling achievement.
· The DFAT is more eﬀective when students and teachers use the tool to a greater extent.
· The eﬀect of the DFAT diﬀers between low-performing, average, and high-performing students.
A randomized experiment was conducted to examine the eﬀects of Snappet on mathematics achievement (1808 students) and on spelling achievement (1605 students) in grade three of primary education. The ﬁndings of the study indicate that a DFAT can have a positive impact on mathematics achievement and student motivation. However, a positive eﬀect on spelling achievement was not found. To investigate these eﬀect diﬀerences more in depth students’ intensity of data use (i.e., the total number of completed assignments and the percentage of adaptive assignments of all completed assignments) for spelling were compared with students’ intensity of data use for mathematics. This showed that students overall completed fewer Snappet assignments for spelling, than for mathematics. The percentage of adaptive assignments was also somewhat lower for spelling, than for mathematics. The diﬀerence in the number of completed assignments by students in the experimental groups compared to the number of completed assignments by students in the control group was presumably larger for mathematics, than for spelling, based on these students’ intensity of use data. Since we did not have data on the number of completed assignments in the control group, we could not test this assumption. Furthermore, high-performing students completed more assignments, than lower performing students and this diﬀerence was somewhat larger for mathematics, than for spelling.
Our study ﬁndings also indicate that the DFAT was most eﬀective for high-performing students. Students’ intensity of use measurements show that experimental students who used the tool to a greater extent (i.e., the total number of assignments completed) performed better, than experimental students who used the tool to a lesser extent (both for spelling and for mathematics). We do not know whether students who completed more Snappet assignments performed better on the posttests because they completed more assignments, or that higher performing students simply completed more assignments. This could imply that the mathematics achievement eﬀect was mainly caused by the fact that experimental students completed more assessments, than students in the control group. Still, all students showed a higher learning gain in the experimental condition for mathematics, than the students in the control condition and the positive diﬀerence in the mean achievement growth is highest for the 20% best performing students.
The eﬀects of the feedback to teachers are most important for this dissertation. Teachers use information from a DFAT as feedback to improve instruction and student achievement is supposed to improve as a consequence. The relationship between teachers’ classroom observation scores and student achievement levels in the ﬁrst study suggests that teachers use information from a DFAT as feedback, and that the eﬀectiveness of a DFAT depends of how the feedback from DFATs are used by teachers.
Study two: diﬀerentiated instruction
In the second study the relationship between diﬀerentiated instruction (DI) and student achievement was investigated. In this study we tested the following hypotheses:
· Students’ achievement levels are higher in classrooms of teachers who diﬀerentiate their instruction more.
· Students’ achievement levels are higher in classrooms of teachers who pre-plan diﬀerentiated instruction more.
· Students from diﬀerent ability groups do not beneﬁt to the same degree from a teacher who diﬀerentiates his/her instruction.
Two characteristics of DI were distinguished and measured. Firstly, DI should be planned and instructional decisions should be based on the analysis of student data. Secondly, what makes DI observable in the classroom is the variation in learning goals, instruction content, instruction time, assignments, and learning materials aimed at addressing varying learning needs. The ﬁrst characteristic was measured by means of an analysis of teachers’ instructional plans (n=89), the second by means of classroom observations of teacher behavior (n=144), using three items from the International Comparative Analysis of Learning and Teaching (ICALT) instrument.
In the second study no positive eﬀects of DI on student achievement were found. So, the eﬀect of teachers using feedback from a DFAT to teachers in study one was not conﬁrmed. An important diﬀerence between the DFAT in study one and the DFAT used in the second study is the frequency (high or low) and timing (immediately or delayed) of the feedback to teachers. The diﬀerences between the ﬁndings in the ﬁrst and the second study could indicate that teachers make a better connection between the feedback they receive, the instruction they gave, and the instruction they are planning to give, if the time span between delivered instruction and received feedback is short. The DFAT in the ﬁrst study and the DFAT in the second study also diﬀered in terms of the content of the feedback provided by the DFATs. The DFAT in the second study oﬀered teachers more options for complex analyses (e.g., information on students’ scores for speciﬁc subject matter components such as ‘dividing’ or ‘subtraction’ in mathematics) compared to the analysis options of the DFAT in study one. However, no eﬀect of teachers using feedback from a DFAT on student achievement was found in the second study, whereas in the ﬁrst study the teacher observation score seemed to indicate an eﬀect on student achievement.
The ﬁndings from the second study also indicate that students from diﬀerent ability groups do not proﬁt from DI to the same extent. This ﬁnding is in line with previous research, showing that ability grouping can have a negative impact on the achievement of students in the low-ability groups. It can be eﬀective for students in the average-ability groups and have no impact on students in the high-ability groups.
In the second study teachers by means of the DFAT analyzed students’ assessment scores on a standardized, summative test that is taken twice a school year. Teachers were trained to use the results, among other things, for determining the composition of the student ability groups they work with in their classrooms. Using the DFAT for composing student ability groups was an important training component, however, perhaps not enough attention was given to the importance of using ability groups ﬂexibly (changing the composition of the groups if, based on new information, there is reason to do so) and to using multiple data sources for composing the ability groups. Overall, the ﬁndings from the two studies seem to point to the importance of immediate feedback to teachers. This may also stimulate teachers to compose student ability groups more ﬂexibly.
In the third study a meta-analysis was conducted to further study the critical characteristics of DFATs.
Study three: DFAT features and intervention features
The aim of the this study was to investigate whether the results of high quality empirical studies in primary and secondary education conﬁrm that the use of DFATs by teachers is eﬀective. The meta-analysis of the studies answered the following two questions:
· Is there an eﬀect of teachers using a DFAT on student achievement?
· Which DFAT features, and which DFAT intervention features inﬂuence the eﬀect on student achievement?
After a literature search 91 studies were screened by using seven eligibility and methodological inclusion criteria. Seventy-seven studies did not meet all inclusion criteria, the remaining 14 studies were included in the meta-analyses. The 14 studies were coded on four DFAT feedback features (i.e., feedback frequency, feedback content, feedback level, predictive feedback) and three intervention features (i.e., intervention frequency, intervention content, intervention target). The coding resulted in three groups of DFATs and interventions (A, B and C).
The ﬁndings of this third study indicate that the use of a DFAT has a small positive eﬀect on student achievement. More speciﬁcally, positive eﬀects were found for DFATs in primary education, for mathematics in primary and secondary education, and for DFATs in group C. DFATs in group C can be distinguished from DFATs in groups A and B on three important characteristics. First, the feedback frequency in group C was high compared to the frequency of the feedback of the DFATs in groups A and B. Second, the eﬀectiveness of DFATs in group C might be explained by the fact that the DFATs and interventions in group C were more suitable for, and more targeted at those who are actually responsible for changing instruction (i.e., teachers) instead of those who are further away from the instructional process (i.e., school principals and members of the school board). DFATs were also coded on the intensity of the intervention and on the content of the interventions meant to support the implementation of the DFAT. Surprisingly, intervention intensity and intervention content for the DFATs in group C were often less comprehensive than most of the interventions categorized in group B. A potential explanation for this ﬁnding might be that the complexity of the feedback content, the feedback level and the absence of predictive feedback in DFATs categorized in group C made the feedback more accessible and easier to digest for teachers. The feedback content of the DFATs in group C was often easy to interpret and detailed (i.e., information on students’ mastery of speciﬁc instructional content). Relatively simple feedback content may make a more comprehensive intervention in terms of intensity and content less necessary.
DFATs can have a positive eﬀect on student achievement. The results of the classroom observations in study one seem to indicate that teachers use information from a DFAT as feedback to evaluate the eﬀects of their instruction, and as feedback for planning follow-up adaptive instruction. However, the results of the second and third study show that positive eﬀects on student achievement are not always accomplished. The results of the three studies together give indications of what characteristics of eﬀective DFATs might be;
· frequent and immediate feedback to teachers, that is also
· detailed and concrete, and
· accessible and easy to interpret for teachers.
These characteristics may make a DFAT suitable for improving teachers’ daily instructional planning and make it easier to integrate a DFAT into teachers’ teaching practices.