National Educational Panel Study (NEPS)

The National Educational Panel (NEPS) is a multi-cohort longitudinal sample that has been examining individual educational and competency development of 60.000 children, teenagers and adults for more than 5 years now. With a focus on measurement of competencies NEPS provides the data basis for the examination of educational chances and passages in Germany. Therefore, besides interviews and interrogations competence tests are administered in different domains on a regular basis. In the first project period (2009 till 2013) NEPS was supported by the Federal Ministry for Education and Research (BMBF). Since 2014 the National Educational Panel is continued in the Leibniz Institute for Educational Trajectories (LIfBi) at Otto-Friedrich-University, Bamberg. DIPF is a consortium’s member of NEPS and founding member of LIfBi. Currently several NEPS-core areas are located at DIPF:

•  “Learning Environments” (pillar 2, Prof. Eckhard Klieme)
• “Upper Secondary Education”  (stage 5, Kai Maaz)
• “Meta data” (Prof. Marc Rittberger)
• “Technology Based Testing” (TBT, Prof. Frank Goldhammer)

Work package Technology Based Testing (TBT) 2014-2016

The working package Technology Based Testing (TBT) at NEPS belongs to the methods group of NEPS. TBT at DIPF is located within the interdisciplinary TBA (Technology Based Assessment)-group and is under the scientific direction of Prof. Frank Goldhammer. TBT closely co-operates with LIfBi and is engaged in innovative interview techniques and testing methods, for example, computer- and internet-based competency assessment.
The working package TBT supports the introduction and realisation of technology-based testing in NEPS with scientifically based services, project specific adaptations of software products and accompanying research. For the domains reading, math, science and ICT Literacy, which are tested several times in the longitudinal study of NEPS, changes concerning measurement instruments due to the computerisation are explored psychometrically based on the combined mode effect and linking studies as well as based on the help of experimental modus variation. Therefore, procedures of quantification and correction of mode effects are investigated and applied precisely that enable an introduction of computer based competency measurement in NEPS.
After successfully introducing technology based testing in the extra-institutional and institutional survey context of NEPS the scientific and development focus is on the use of characteristics of technology-based testing. This aims at the enhancement and optimisation of assessments in regard to validity (for example, simulation based assessment of ICT Literacy, e.g. TECCI or CavE-ICT Pisa) as well as the increase of measurement efficiency (see, for example, HEUREKO and MATE) and the use of log and processing data (see, for example, PROKOM).

Fields of research (psychometrics):

• Effects of test administration / mode effects/ test equivalence
• Online assessment and effects of test setting
• Measurement efficiency and test set / adaptive testing
• Innovative item and response formats for computer-based testing
• Use and analysis of log and process data
• Capturing and modelling of response times

Fields of research (informatics):

• Response time measurement and analysis of timely accuracy of assessment systems
• Workflow support for the implementation of computer-based tasks (ItemManagement Portal)
• Mobile assessment/Online assessment with mobile devices
• Digital preservation of computer-based test instruments
• Test security and test delivery on existing hardware for computer-based testing in institutional contexts

Services:

• Requirement analysis and transfer planning for NEPS assessments with embedded computer-based testing in individual and group assessment
• Support of domain experts and item authors while they transfer computer-based items with the authoring tool CBA ItemBuilder (development of item templates, counseling, advanced trainings)
• Prototypical programming of innovative and new item formats as well as computer-based assessments
• Co-ordination of requirements for the development of the authoring tool CBA ItemBuilder for the use in NEPS
• Development of NEPS TBT modules for computer-based testing and computer-based interviews (self-respond) in individual testing or notebook-based group testing with TBA Tools
• 2nd level support for assessment institutions during the realisation of studies with computer-based tests
• Load testing of online-delivery, development of testing plans for computer-based tests
• Programming of specific instruments and prototypes as diagnostic modules for computer-based assessment in NEPS

Project team:

 

• Scientific management: Prof. Dr. Frank Goldhammer
• Operativ management: Dr. Ulf Kröhne
• Co-ordination software-development: Elham Müller
• Sarah Bürger (phd-student, psychometrics)
• Marisa Herrmann (grafic design, user support)
• Saba Sabrin (software development)
• Alexander Bogumil (software development)
• Gabriel Mateescu (software development)

Associated staff:

 

• Roland Johannes (response time testing)
• Felix Wagner (software development)
• Yvonne Keßel (phd-student, psychometrics)
• Björn Buchal (system administrator)

Selected talks and publications:

- Buerger, S., Kroehne, U. & Goldhammer, F. (2014). An experimental investigation of mode effects in the National Educational Panel Study (NEPS). Talk presented at the VI European Congress of Methodology (EAM), Utrecht, Netherlands, July 23 – 25, 2014.
- Buerger, S., Kroehne, U. & Goldhammer, F., (2014). Gibt es Unterschiede zwischen der papier- und computerbasierten Administration eines Leseverständnistests?  - Eine Analyse von Mode Effekten im NEPS. Beitrag auf dem Kongress der Deutsche Gesellschaft für Psychologie (DGPs), Bochum, 21.-25.9.2014.
- Kroehne, U., Buerger, S., Weiss, T. & Goldhammer, F. (2014). Trennung von Mode- und Kontext-Effekten einer Online-Kompetenztestung im Rahmen des Nationalen Bildungspanels, Deutsche Gesellschaft für Psychologie (DGPs), Bochum, 21.-25.9.2014.
- Kroehne, U., Buerger, S, Goldhammer F. (2014). Disentangle Mode Effects and Context Effects for Online Assessment in the National Educational Panel Study (NEPS) in Germany. Talk presented at the VI European Congress of Methodology (EAM), Utrecht, Netherlands, July 23 – 25, 2014.
- Goldhammer, F., & Pfaff, Y. (2011, September). Measuring Individual Differences In High School Students’ (grade 12) ICT Literacy: Accessing And Evaluating Information. Talk presented at the European Conference on Educational Research (ECER), Berlin, Germany, September 13 – 16, 2011.
- Goldhammer, F., Kroehne, U., Keßel, Y., Senkbeil, M., & Ihme, J.M. (2014). Diagnostik von ICT-Literacy: Multiple-Choice- vs. simulationsbasierte Aufgaben. Diagnostica, 60, 10-21.
- Kroehne, U. & Martens, T. (2011). Computer-Based Competence Tests in the National Educational Panel Study: The Challenge of Mode Effects. Zeitschrift für Erziehungswissenschaften, 14, 169-186.
- Goldhammer, F., Naumann, J. & Keßel, Y. (2013). Assessing Individual differences in Basic Computer Skills: Psychometric characteristics of an interactive performance measure. European Journal of Psychological Assessment, 29, 263-275.

Finished project phase 2009-2013

In the project phase 2009 until 2013, preparatory to the work package Technology Based Testing (TBT) DIPF was responsible for the following tasks:
AP 13 B

 

• The software development for a data warehouse has been located at TBA; the data warehouse ensures a data access as fast as possible while considering privacy data protection.
• The development of a data warehouse was supposed to guarantee a central data stock for the entire NEPS study and to provide appropriate tools for filtering as well as report production.
• Data Warehouse: Three proceeding processes of parallel software developed during the project duration: (1) implementation, optimisation and development of data banks (2) implementation, optimisation and further development of ETL- and reporting tools (3) implementation, optimisation and further development of a web portal.
• With the Data Warehouse the data from the four waves of assessment as well as the tools for filtering and report production should be accessible for the whole scientific community subsequently.

 

AP 13 C

 

• For the preparation of the electronic test assessment empirical assessments had been conveyed to identify possible differences between paper-based and computer-based testing (quantification of mode effects) and to investigate computer-based testing results (cross mode linking).
• Mode effect studies (equivalence studies combined with linking studies of NEPS) have been conducted to prepare a test execution on a technological basis. The mode effect studies aimed at testing the equivalence of paper-based assessments (PBA) and computer-based assessment (CBA) by means of different criteria.  The organisation and execution of mode effect studies was carried out with pillar 1 (Competence development in the life course).

 

Further information

To the website of NEPS at LIfBI