In the context of a co-operative project an instrument will be developed for the computer-based, adaptive and behaviour related PISA assessment of ICT-skills of 15-year old students.
Owing to the relevance of computers in daily life, competent handling of information and communication technologies (ICT) is a prerequisite for a successful participation in modern knowledge-based societies and thus an important goal in education. In the context of a co-operative project, a computer-based instrument will be developed aiming to test ICT skills of 15-year old students. In this context, a valid and efficient measurement of ICT skills is possible because authentic and adaptive (i.e. fitting individual competence levels) items are presented in behavior-related environments.
1. Definition of a theoretical framework
The first partial goal aims at the preparation of a theoretical framework concept to define and organise the domain of ICT –related skills. Based on PISA questionnaires concerning familiarity with computers ('ICT Familiarity Questionnaire‘) and existing theoretical concepts, new and so far neglected aspects such as co-operative types of collaboration shall be taken into account in the theoretical framework. Assisted by psychological theories and models, the internal structure of ICT skills will be described. The framework concept defines prototypical ICT tasks, e.g. internet-based queries for information, and describes the therefore relevant ICT skills and abilities. Furthermore, aspects defining constructs and difficulties of such ICT tasks will be identified with regard to task development.
2. Item development and field testing
Based on the theoretical framework concept, the second objective concerns the development of a survey concept, whereby representative tasks will be systematically allocated to the formulated skills and abilities. Task types can be categorised and ordered according to different characteristics (e.g. characteristics defining difficulty, context, mode of reaction), preparing the basis for a systematic task development. Accordingly, trained item developers will construct items containing simulations of ICT environments (e.g. a web browser), implementing them by using the CBA ItemBuilder, an authoring tool. Cognitive Laboratory Studies will subsequently be conducted to put these items to a first qualitative test, leading to an optimisation and implementation in a large sample in the PISA 2015 Field Test.
Planned steps (1-5) of the project are oriented towards the conceptual developmental process that is typically pursued in international Large-Scale Assessments. In a first step, a theoretical framework is created (1), defining and organizing the domain of ICT-related skills and serving as a foundation for test development (2). Data for scaling development and validation (3) of the measurement instruments will be subject to a national supplementary study to the PISA 2015 Field Test and will serve as a basis for the development of the adaptive test (4). The developed instrument can thus be applied to test the role of ICT skills in computer-based assessments (5).
Project phases and steps will all be conducted in close cooperation of all project partners involved. Here, project partners at the will be particularly responsible for preparing the theoretical framework, deducting the assessment concept, developing and delivering the software for the item development (CBA ItemBuilder), technical delivery of the assessment instrument and formulating the validation approach.
The project has been completed in the meantime but further analyses concerning data evaluation will be carried out. The final report is available online at the the German National Library of Science and Technology of the University Library Hanover (project funding reference number 1LSA010). The following link leads directly to the final report: Project CavE-ICT-PISA: (November 2015)
Funding: Federal Ministry of Education and Research (BMBF)
Co-operation: Prof. Dr. Holger Horz, Goethe-University of Frankfurt am Main (Project co-ordination), Prof. Dr. Andreas Frey, Friedrich-Schiller-University of Jena
Duration: 4/2012 – 3/2015
Staus: completed (see above "Project close-out")