NEPS TBT: work package Technology-Based Assessment
DIPF | Leibniz Institute for Research and Information in Education is a consortium partner to the National Education Panel Study (NEPS), and thus contributes to its planning and operation. One of the project’s priorities is on the work package technology-based testing (TBT).
NEPS: National Educational Panel
The National Educational Panel Study (NEPS) is a multi-cohort longitudinal study. Since 2009, NEPS has traced individual educational and competency development of 60,000 children, adolescents and adults in six starting cohorts in a longitude. With a focus on the measurement of competencies, NEPS provides a database for the assessment of educational opportunities and transitions in Germany. To this end, interviews and surveys are conducted at regular intervals in different domains. In the first project phase (2009 until 2013), NEPS was funded by the Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung, BMBF). Since 2014, the National Educational Panel has been run by the Leibniz Institute for Educational Trajectories (LIfBi) at the Otto-Friedrich University Bamberg. DIPF | Leibniz Institute for Research and Information in Education is a consortium partner of NEPS and a founding member of LIfBi.
The work package “Technology-Based Testing” (TBT) is part of the NEPS method group and is located at DIPF in the interdisciplinary Centre for Technology Based Assessment (TBA), scientifically managed by Dr. Ulf Kröhne. NEPS-TBT collaborates closely with the Leibniz Institute for Educational Trajectories (LIfBi) and deals with innovative assessment and survey methods, e.g. computer and internet-based measurement of competencies. The TBT work package is operatively managed by Dr. Lena Engelhardt; it offers scientifically based services, project-specific software adaptations and accompanying research to support the introduction and implementation of technology-based testing in NEPS. After successfully introducing technology-based assessment in extra-institutional assessment contexts and a growing implementation in institutional contexts, the research and development focus of NEPS is on using characteristics of technology-based testing for further developing and optimizing assessments. A current focus concerns the increase of measurement efficiency and usage of log and process data.
Project objectives for the project phase 2018-2022
The superordinate aim of NEPS-TBT concerns the operation of scientifically grounded technology-based assessments that can connect to interna¬tional standards. Five central innovation aspects will contribute to this objective: (1) update of software components step by step, (2) transfer of assessment innovations (e.g. innovative item formats & increase of measurement efficiency) in panel studies, (3) cross-mode linking also for heterogeneous assessment hardware (tablets, touch entry), (4) data processing of all TBT data via log data, (5) automated software testing and quality assurance. These foci of innovation are specifically implemented in the following work packages:
- A strategy will be developed for the testing and quality assurance of study-specific TBT-modules. By means of automated testing, complete data storage checks will be enabled. Automated testing will furthermore serve the quality assurance of fixed test assembly and allow for checking adaptive test assembly.
- The development of a standardized editor should enable automated checking of codebooks and test definition for multistage tests.
- A generic, study-independent concept will be developed for coding missing responses taking into account indicators from log data.
- Prerequisites for implementing psychometrically sophisticated test designs will be prepared, such as adaptive algorithms. The TBA Centre is thus developing an infrastructure to configure CAT algorithms for test development from R. These should be tested in simulation studies which are operatively integrated into the delivery software.
- Following the paradigm of economy, result data and log data should not be processed in parallel. Instead, result data will be processed on the basis of log data. To this end, criteria have been developed for defining the completeness of log data (cf. Kroehne & Goldhammer, 2018). These developments will be used for creating generic tools, to enable reproducible and transparent data processing.
Project partners: Leibniz Institute for Educational Trajectories e.V. (LIfBi), Leibniz Institute for Science and Mathematics Education at the University of Kiel), software development intern and CBA ItemBuilder (Nagarro GmbH)
Duration: 2018 - 2022
Contact: Ulf Kröhne
To the LIfBI