NEPS TBT: work package Technology-Based Assessment

NEPS: National Educational Panel

The National Educational Panel Study (NEPS) is a multi-cohort longitudinal study. Since 2009, NEPS has traced individual educational and competency development of 60,000 children, adolescents and adults in six starting cohorts in a longitude. With a focus on the measurement of competencies, NEPS provides a database for the assessment of educational opportunities and transitions in Germany. To this end, interviews and surveys are conducted at regular intervals in different domains. In the first project phase (2009 until 2013), NEPS was funded by the Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung, BMBF). Since 2014, the National Educational Panel has been run by the Leibniz Institute for Educational Trajectories (LIfBi) at the Otto-Friedrich University Bamberg. DIPF | Leibniz Institute for Research and Information in Education is a consortium partner of NEPS and a founding member of LIfBi.

Project description

The work package “Technology-Based Testing” (TBT) is part of the NEPS method group and is located at DIPF in the interdisciplinary Centre for Technology Based Assessment (TBA), scientifically managed by Dr. Ulf Kröhne. NEPS-TBT collaborates closely with the Leibniz Institute for Educational Trajectories (LIfBi) and deals with innovative assessment and survey methods, e.g. computer and internet-based measurement of competencies. The TBT work package is operatively managed by Dr. Lena Engelhardt; it offers scientifically based services, project-specific software adaptations and accompanying research to support the introduction and implementation of technology-based testing in NEPS. After successfully introducing technology-based assessment in extra-institutional assessment contexts and a growing implementation in institutional contexts, the research and development focus of NEPS is on using characteristics of technology-based testing for further developing and optimizing assessments. A current focus concerns the increase of measurement efficiency and usage of log and process data.

Project objectives for the project phase 2018-2022

The superordinate aim of NEPS-TBT concerns the operation of scientifically grounded technology-based assessments that can connect to interna¬tional standards. Five central innovation aspects will contribute to this objective: (1) update of software components step by step, (2) transfer of assessment innovations (e.g. innovative item formats & increase of measurement efficiency) in panel studies, (3) cross-mode linking also for heterogeneous assessment hardware (tablets, touch entry), (4) data processing of all TBT data via log data, (5) automated software testing and quality assurance. These foci of innovation are specifically implemented in the following work packages:

1. A strategy will be developed for the testing and quality assurance of study-specific TBT-modules. By means of automated testing, complete data storage checks will be enabled. Automated testing will furthermore serve the quality assurance of fixed test assembly and allow for checking adaptive test assembly.
2. The development of a standardized editor should enable automated checking of codebooks and test definition for multistage tests.
3. A generic, study-independent concept will be developed for coding missing responses taking into account indicators from log data.
4. Prerequisites for implementing psychometrically sophisticated test designs will be prepared, such as adaptive algorithms. The TBA Centre is thus developing an infrastructure to configure CAT algorithms for test development from R. These should be tested in simulation studies which are operatively integrated into the delivery software.
5. Following the paradigm of economy, result data and log data should not be processed in parallel. Instead, result data will be processed on the basis of log data. To this end, criteria have been developed for defining the completeness of log data (cf. Kroehne & Goldhammer, 2018). These developments will be used for creating generic tools, to enable reproducible and transparent data processing.

Completed project phase 2014-2017

Completed project phase 2009-2013

Research topics

Services

 

Funding: BMBF

Project partners: Leibniz Institute for Educational Trajectories e.V. (LIfBi), Leibniz Institute for Science and Mathematics Education at the University of Kiel (IPN), software development intern and CBA ItemBuilder (Nagarro GmbH)

Duration: 2018 - 2022

Status: running

Project management

Scientific management: Ulf Kröhne
Operative management: Lena Engelhardt

Project team: Tobias Deribo, Marisa Herrmann, Christina Röper, Felix Wagner

Contact: Ulf Kröhne

 

To the website of NEPS at LIfBI