PISA 2012 NaBe

Today, computers leave a mark upon both everyday and working life. Consequently, knowledge and skills are increasingly assessed by using computers, as it is the case in the assessment of 15-year-old students in PISA 2012. Computer-based testing is not only cost-effective and effort-saving, but also, compared to conventional pen and paper-based formats, computer-based assessment (CBA) allows new test designs and response types. Hence, the national add-on study PISA 2012 in Germany aims to find solutions for the possible computer applicability particularly for assessing reading and mathematics comprehensions.

Objectives
The add-on study focuses on conducting computer-based assessment (CBA) in the context of PISA. In particular, research-oriented answers shall be given to the following areas of investigation:

1. Computer-based measurement of reading hypertexts (Digital Reading) 

On the contrary to conventional linear texts, digital hypertext formats own certain characteristics that request special demands of the reader. Gathering information from a website does not only mean reading text passages. Moreover, the user has to decide, which additional websites to access via hyperlinks and in what order. Other criteria need to be taken into account when choosing a website, e.g. regarding the relevance and trustworthiness of the pages. In the case of capturing digital texts, the computer is thus not only an output medium, but it furthermore enables the simulation of hypertexts. In the context of the national add-on study to PISA the survey investigates empirically whether linear text reading competency can be distinguished from hypertext reading competency, and to what extent differences can be explained concerning individual skills, such as computer literacy. 

Examples from NaBe

This is how the implementation of digital reading tasks looks like:

The tasks are constructed similarly to homepages. Links can be clicked and refer to further pages, of course.

2. Computer-based testing of mathematics (CBA of Mathematics)

Compared to the questions regarding digital text reading, similar ones will be addressed by the introduction of computer-based assessment of mathematics in PISA 2012. But unlike the case of reading, we do not assume the assessment of mathematics to be fundamentally changed by the computer as a medium of delivery. Rather, computers are considered merely as a distribution system assessing the same – regarding convergent validity – mathematical competencies as the pen-and paper formats.

3. Investigation of mode effects

Further aspects that shall be addressed by the PISA add-on study relate to effects that might occur by transferring linear paper test formats to computers. To maintain comparability of linear reading between the two different modes, paper vs. computer, particular attention was paid to a preferable high congruence of paper-based with computer-based format. Beside technical transferability per se, it is of particular interest, if the change of mode affects the psychometric characteristics of tasks, e.g. difficulty of solution, and, if so, how occurring changes might be explained. The investigation focuses on the question of how mode effects depend on particular response types, e.g. free text writing or multiple choice, of the reading tasks.

Examples from NaBe

This is how the tasks for the investigation of mode effects look like:

The tasks should be as similar as possible compared to the paper-based version. Therefore the tasks do not include hyperlinks in the text, for example.

4. Mode effects on motivation

The given performance in the competence fields in PISA is not only depending on cognitive skills of students, but also on their present mood and motivation. Hence, the PISA add-on study questions to what extent the modes, paper- vs. computer-based presentation, affect the actual motivation – and therefore maybe the performance likewise.

Methodologic approach

Data assessment of the CBA add-on study took place as the second day of the 2012 PISA main assessment. For taking part in the add-on study 77 of 212 PISA schools were acquired, i.e., data of nearly 900 students can be analysed. Mainly, the students answered  tasks delivered by a computer. Beside a test battery verifying their computer skills, working memory capacity and basic reading skills, they were working on reading comprehension tasks and questions collecting their respective stances. Some students answered tasks in a test book in paper format. Furthermore, research questions have been addressed by integrating findings from reading, mathematics and problem solving tasks that were administered computer-based on the first day of the PISA study.

 

Funding: Centre for International Student Assessment (ZIB)

Cooperation: TUM, DPC

Duration: 12/2011 – 12/2015

Project manager: Frank Goldhammer

Contact: Carolin Hahnel, Ulf Kröhne

Literature:

- Hahnel, C., Goldhammer, F., Naumann, J., & Kröhne, U. (2014, September). Validierung von Digital Reading: Befunde zum Erklärungsbeitrag ICT-bezogener Komponenten [Validating Digital Reading: Findings about the contribution in explanation of ICT-related components]. Talk presented at the 49. Kongress der Deutschen Gesellschaft für Psychologie (DGPs), Bochum, Germany, September 21-25, 2014.

- Hahnel, C., Kröhne, U., Naumann, J., & Goldhammer, F. (2013, August). Explaining hyperlink selection by PISA reading and cognitive component skills. Talk presented at the 15th Biennal Conference of the European Association for Research on Learning and Instruction (EARLI), Munich, Germany, August 27-31, 2013.

- Kröhne, U., Hahnel, C., Schiepe-Tiska, A., & Goldhammer, F. (2013, August). Analyzing Mode Effects of PISA Print Reading Including a Comparison of Time-related Information. Talk presented at the 15th Biennal Conference of the European Association for Research on Learning and Instruction (EARLI), Munich, Germany, August 27-31, 2013.