ProCom

Measurement and modelling of competencies generally focus on outcome indicators, i.e. response patterns in tasks serve as indicators of the underlying competence levels. In this approach, although behavioural differences occur during task processing, these differences are not integrated into competence modelling in addition to individual task results.
The interdisciplinary project assesses the use of such process data, focusing on time spent on tasks, theoretically grounded process measures and exploratively constructed process models on the interaction and navigation behaviour in complex computer-based task formats.

Objectives

The main objective consists in an extended modelling of competencies, by using process data. Levels of competencies that are normally only defined in terms of outcomes will be described in a more comprehensive manner regarding the three sub-targets outlined below, by subjecting process data (time on task, sequences of time spent on tasks)  to different methodological approaches (psychometrics, computer sciences).

1. Processing time: Integrated measurement models for responses and response times (IRT-RT models) serve to empirically determine the relation between domain specific skills and speed constructs.
2. Processing steps (theory-based): For the domains of literacy and problem solving, theory-based process measures are constructed describing the quality of a navigation path and serving as predictors of successful outcomes as well as competence indicators.
3. Processing steps (exploratory): The recorded sequences of events are also to be analysed automatically by means of process mining techniques, to explore and determine from empirical processing steps an underlying process model of actual processing behaviour.

Method

National field test data from the PIAAC study provide the empirical basis for this study. In the intended project, enhanced and advanced analyses shall be carried out, which are not part of the regular analysis plan of the PIAAC study. The sample from the national PIAAC field test (FT) comprises ca. N=1500 participants (household sample). Data from the computer-based assessment of literacy, numeracy and problem solving will be used.
Statistical modelling encompasses measurement models for categorial and continuous indicators (IRT, CFA), explanatory models, taking into account task and person characteristics and their interaction, as well as explorative process mining techniques.

Publications

Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., Klieme, E. (2014). The Time on Task Effect in Reading and Problem Solving Is Moderated by Task Difficulty and Skill: Insights From a Computer-Based Large-Scale Assessment. Journal of Educational Psychology. doi: 10.1037/a0034716

Goldhammer, F., Rauch, W., Schweizer, K., Moosbrugger, H. (2010). Differential effects of intelligence, perceptual speed and age on growth in attentional speed and accuracy. Intelligence, 38, 83-92.

 

Funding: German Research Association

Cooperation international: R.H. Klein Entink (NL), M.L. Lennon (USA), L. Salmerón, E. Vidal-Abarca (ES), W. van der Aalst (NL)

Cooperation national: J. Funke, J. Hartig, D. Moldt, GESIS

Duration: 10/2011 - 06/2016

Status: completed

Project manager: Frank Goldhammer, Johannes Naumann

Contact: Annette Stelter, Frank Goldhammer