National add-on study on mode effects in PISA 2018
The project examines mode effects in PISA. This means that differences between paper-based and computer-based testing are investigated. In particular, questions arise about the psychometric comparability of both test forms. In addition, the project will examine the effects of mode change on the measured trend in Germany.
PISA 2015 brought about an important change for test takers: the switch from paper-based to computer-based testing. In doing so, they made use of the much greater possibilities for test and task design on the computer. In this context, however, numerous questions arose regarding the comparability of the two test forms, which will be examined in more detail in a national add-on study to PISA 2018.
On the one hand, the aim is to check whether the response behaviour when working on paper tasks differs from that on the computer. Some of the test items for processing on the computer differ significantly in their design from those on paper. This gives rise to questions as to whether the changeover places different demands on the participants and thus changes with regard to the competence measured (mode effect on construct interpretation), whether the tasks can distinguish in the same way between more or less competent students (mode effect on item discrimination) and, furthermore, whether the difficulty of the tasks changes to the same extent for all persons or groups (moderated mode effect on difficulty). It is also important to investigate whether mode effects differ between schools and by which school characteristics (e.g. computer equipment or use of computers in class) they can be explained.
A second important point is the comparability of the study results of previous PISA cycles. In addition to describing the results of the current PISA study, the national PISA report always analyses and interprets the development (trend). With the switch to computer-based testing, comparison is no longer possible without further ado. In analyses and comparisons, the mode effects must always be taken into account. The add-on study therefore also investigates how pronounced such mode effects are and to what extent they affect comparability, in order to be able to make reliable statements about national trends in the future.
First results of the add-on study provide evidence for construct equivalence between paper-based and computer-based testing. Furthermore, the data of the add-on study indicate that computer-based items are on average somewhat more difficult than paper-based items. With regard to the changes between 2015 and 2018, there is a high degree of agreement between the internationally reported (original) and national (marginal) trends. The changes between 2009 and 2018 are somewhat more favourable overall for the national trend, which is based solely on paper-based measurements, than for the original trend.
Selected Publications
Goldhammer, F., Harrison, S., Bürger, S., Kroehne, U., Lüdtke, O., Robitzsch, A., Köller, O., Heine, J.-H & Mang, M. (2019). Vertiefende Analysen zur Umstellung des Modus von Papier auf Computer. In K. Reiss, M. Weis, E. Klieme & O. Köller (Hrsg.), PISA 2018 Grundbildung im internationalen Vergleich (S. 163-186). Waxmann. http://zib.education/fileadmin/user_upload/Dokumente/Artikel/Moduseffekt_Studie_PISA_2018.pdf
Funding: Centre for International Student Assessment (ZIB)
Duration: 2017 - 2022
Staus: running
Contact: Scott Harrison