PISA 2012 NaBe

Today, computers leave a mark upon both everyday and working life. Consequently, knowledge and skills are increasingly assessed by using computers, as it is the case in the assessment of 15-year-old students in PISA 2012. Computer-based testing is not only cost-effective and effort-saving, but also, compared to conventional pen and paper-based formats, computer-based assessment (CBA) allows new test designs and response types. Hence, the national add-on study PISA 2012 in Germany aims to find solutions for the possible computer applicability particularly for assessing reading  comprehensions.

Objectives

The add-on study focuses on conducting computer-based assessment (CBA) in the context of PISA. At DIPF, primarily, research-oriented answers shall be given to the following areas of investigation:

1. Computer-based measurement of reading hypertexts (Digital Reading) 

On the contrary to conventional linear texts, digital hypertext formats own certain characteristics that request special demands of the reader. Gathering information from a website does not only mean reading text passages. Moreover, the user has to decide, which additional websites to access via hyperlinks and in what order. Other criteria need to be taken into account, e.g. regarding the relevance and trustworthiness of the web pages. In the case of capturing digital texts, the computer is thus not only an output medium, but it furthermore enables the simulation of hypertexts. In the context of the national add-on study to PISA the survey investigates empirically whether linear text reading competency can be distinguished from hypertext reading competency, and to what extent differences can be explained concerning individual skills, such as computer literacy. 

Examples from NaBe

This is how the implementation of digital reading tasks looks like:

The tasks are constructed similarly to homepages. Links can be clicked and refer to further pages, of course.

2. Investigation of administration mode

Further aspects that shall be addressed by the PISA add-on study relate to effects that might occur by transferring linear paper test formats to computers. To maintain comparability of linear reading between the two different modes that is, paper vs. computer, particular attention was paid to a preferable high congruence of paper-based with computer-based format. Beside technical transferability per se, it is of particular interest, if the change of mode affects the psychometric characteristics of tasks, e.g. difficulty of solution, and, if so, how occurring changes might be explained. The investigation focuses on the question of how such mode effects depend on particular response types, e.g. free text writing or multiple choice, of the reading tasks.

Examples from NaBe

This is how the tasks for the investigation of mode effects look like:

The tasks should be as similar as possible compared to the paper-based version. Therefore the tasks do not include hyperlinks in the text, for example.

 

Methodologic approach

Data assessment of the CBA add-on study took place as the second day of the 2012 PISA main assessment. For taking part in the add-on study 77 of 212 PISA schools were acquired, i.e., data of nearly 900 students can be analysed. Mainly, the students answered  tasks delivered by a computer. Beside a test battery verifying their computer skills, working memory capacity and basic reading skills, they were working on reading comprehension tasks and questions collecting their respective stances. Some students answered tasks in a test book in paper format. Furthermore, research questions have been addressed by integrating findings from reading tasks that were administered computer-based on the first day of the PISA study.

Publications

Hahnel, C., Goldhammer, F., Kröhne, U., & Naumann, J. (2018). The role of reading skills for the evaluation of online information. Computers in Human Behavior, 78, 223–234.  http://doi.org/10.1016/j.chb.2017.10.004

Hahnel. C. (2017). Demands and Cognitive Processes in Reading Digital Text (Dissertationsschrift). Goethe Universität Frankfurt am Main.

Hahnel, C., Goldhammer, F., Kröhne, U., & Naumann, J. (2017). Reading digital text involves working memory updating based on task characteristics and reader behavior. Learning and Individual Differences, 59, 149–157.  http://doi.org/10.1016/j.lindif.2017.09.001

Hahnel, C., Goldhammer, F., Naumann, J., & Kröhne, U. (2016). Effects of Reading Comprehension, Basic Computer Skills, Evaluating Online Information, and Navigation on Reading Digital Text. Computers in Human Behavior, 55, 486-500. http://doi.org/10.1016/j.chb.2015.09.042

Selected papers

Kröhne, U., Goldhammer, F., & Hahnel, C. (2014, 04. Juli). Benefits of Process-related Information for Explaining Differences Between Administration Modes. Vortrag auf der 9th Conference of the International Test Commission (ITC) in San Sebastian, ES.

Kröhne, U., Hahnel, C., Schiepe-Tiska, A., & Goldhammer, F. (2013, 27. August). Analyzing Mode Effects of PISA Print Reading Including a Comparison of Time-related Information. Vortrag auf der 15th Biennal Conference of the European Association for Research on Learning and Instruction (EARLI) in München, DE.

 

Funding: Centre for International Student Assessment (ZIB)

Cooperation: TUM, DPC

Duration: 12/2011 – 12/2016

Project manager: Frank Goldhammer

Contact: Carolin Hahnel, Ulf Kröhne