LogDataAnalyzer

The LogDataAnalyzer reads out data consisting of unstructured process data (log files) and tabulates them. These tables can be further processed with any given spreadsheet or statistical package. Problem Solving Items of PIAAC („Programme for the International Assessment of Adult Competencies“) are analysed that require different actions and several processing steps. Single actions or whole sequences of actions can be selected with the LogDataAnalyzer, as far as they correspond to defined actions or sequences of actions in the log files. For the extraction, specific actions that are relevant for the individual research question are searched for in the log files.

What is LogDataAnalyzer?

The LogDataAnalyzer has been developed in co-operation with German Social Science Infrastructure Services (GESIS) and serves for data processing. The application of this tool does not need extensive training and can be used easily.
Especially, the LogDataAnalyzer is useful for data extraction and tabulation of large samples. One possible application of the LogDataAnalyzer is the reconstruction of processing in achievement test items. Especially for complex items in achievement tests the solving process usually needs different sequential steps. Computer based assessment in educational research allows a reconstruction of solving processes supported by the LogDataAnalyzer: First, this can help to explain the results in individual test items and, second, it can be used for the measurement of competency.

How does LogDataAnalyzer work?

Log data represent different types of interactions between test taker and test surrounding as well as internal actions (also events), that are listed together with a time stamp and description:

The time stamp shows the time (in milliseconds) that passed since the start of the solving process. Additionally, the description gives information about the underlying action (i.e., which link has been clicked within a hypertext surrounding).
The LogDataAnalyzer can extract summative information (for example, number of [specific] actions) as well as information of sequences (for example, the time for a specific action) and saves them in a so-called CSV-file. Currently, only log data from the Organisation for Economic Cooperation and Development (OECD), more specific PIAAC R1, are legible.  Further development of the LogDataAnalyzer is planned.
An example: In the PIAAC-item ‘jobsearch’ (see figure) test takers have to choose hits from a result list of a search engine according to specific criteria and to bookmark them:

In this task, for example, setting a bookmark is a relevant action for the solving process:

The LogDataAnalyzer can give information for each test taker and his or her bookmarking activities. For example, this includes the number of the set bookmarks, the time for the setting of the single bookmarks as well as the web page for which a bookmark has been set.

Where is LogDataAnalyzer used?

The LogDataAnalyzer was first applied in the DFG project Process Data for Competence Modelling, in which data of the field test are analysed. The extracted data are used amongst others to describe the relations between the result of the process of technology based problem solving items and characteristics of the processing.
The time needed for the solving process of items (Goldhammer et al., in press), the number of interactions between test takers and test surrounding (Naumann, Goldhammer, Rölke & Stelter, under revision) and the time needed for elementary steps of item processing proved to be explanatory predictors.

 

Funding: own funds, OECD

Cooperation: GESIS

Duration: 06/2013 - 06/2016

Status: completed

Projekt management: Frank Goldhammer, Johannes Naumann

Contact: Annette Stelter