ReCo: Automatically Processing Text Responses from Large-Scale Assessments

The project Automatic Response Coding (ReCo) centers around text responses in tests. The computer program ReCo automatically assesses whether a text response is correct, for example to the PISA test, such as “The author aims at saving the trees.” Moreover, ReCo extracts further features, for example whether a student adds knowledge to their response beyond the explicit information in the text.

The computer program ReCo was originally developed at the Technical University of Munich and in the Centre for International Student Assessment (ZIB). In cooperation with TUM and ZIB, the TBA Centre at DIPF is now developing the software package further. The project comprises the general software development as well as scientific studies employing ReCo. For this, TBA is responsible for the conceptual and technical development on the one hand and acts as the project manager for projects such as ReCo-Multi on the other hand. Also, TBA supports external research groups using the program.

Enhancing Educational Measurement

When constructing tests, research groups are often faced with the question which response format should be implemented. If they offer response options such as in the multiple choice format, responses can be easily processed. The downside is that this closed format entails guessing and other undesirable effects. But if the respondents can write their response into an open text field, the processing is often significantly complicated. At the same time, open response formats are better suited to assessing deeper understanding than closed ones are for most domains, because the respondents are required to produce a response themselves rather than just recognizing the correct one.

Therefore, the OECD already for PISA 2000 decided to include some tasks with open response format. The resulting effort for processing the text responses is particularly challenging for large-scale studies like PISA. Each participating country needs to train a team of coders diligently in order to evaluate the large amount of text responses from more than 500,000 students in total in the recent rounds. The coding team has guidelines for this task so that the coding takes place as objectively and comparable across countries as possible.

Because human beings have subjective perspectives and cultural backgrounds, the risk of inconsistent coding needs to be considered. Owing to their lack of stamina, humans are also prone to errors when being exposed to such a large volume of data. Computer programs such as ReCo can improve consistency and, in turn, objectivity within as well as across countries. Also, the large volume of the data is not an obstacle for the computer guaranteeing further consistency at a large scale. First studies with German data showed that ReCo is capable of such automatic coding to a promising extent.

ReCo combines technologies from Natural Language Processing and Machine Learning. Among others, the program conducts a Latent Semantic Analysis on a text corpus specifically designed for this purpose. The meanings of words can thus be compared to each other. This way, it can identify different (semantic) response types in the data and connect them through Machine Learning with, for example, response correctness. In addition, ReCo can extract further response features, such as whether a student included information beyond the original text in their response or whether the student simply repeated the text.

ReCo-Multi: ReCo Learns Multiple Languages

For the purpose of extending ReCo’s scope to more languages, several PISA National Centers have joined the cooperation project ReCo-Multi. They share their text data comprising multiple test languages, further develop software components, and evaluate whether the satisfying findings of the German study can be replicated with other languages. Together with the German PISA National Center in the ZIB, TBA is in charge of managing and carrying out this international cooperation project.

Free Program

ReCo can be used upon request. Currently, the usage is constrained to German text data only. Over the course of the project, TBA will develop a graphical user interface for the program and both will be freely available for download.

 

Publications

Zehner, F. (2016). Automatic processing of text responses in large-scale assessments (Dissertation). Technische Universität München, München. doi:10.13140/RG.2.2.26846.84800

Zehner, F., Sälzer, C., & Goldhammer, F. (2016). Automatic coding of short text responses via clustering in educational assessment. Educational and Psychological Measurement, 76(2), 280–303. doi:10.1177/0013164415590022

Also: Best poster award ReCo-App shinyReCoR - check out the poster session.


Cooperations: Technical University of Munich, Centre for International Student Assessment (ZIB) e.V.

Duration: since 04/2017

Status: running

Contact: Fabian Zehner, Frank Goldhammer