Multimodal Immersive Learning with Artificial Intelligence for Psychomotor Skills

The COVID-19 pandemic has shown that many teaching/learning activities can be performed without physical presence. This is hardly true for psychomotor skills: their development, as required in many disciplines (e.g., medicine, engineering, chemistry, artistic activities, sports), requires hands-on practice, direct feedback, and reflection. In order to achieve the desired learning successes, personnel support and material input are therefore indispensable. Both of these factors increase costs and limit the scalability of the courses concerned: experts are rare and expensive, and the use of materials causes further costs.

Current technological developments are changing this situation:

• mixed, augmented and virtual reality make it possible to create immersive learning and practice spaces.
• Modern sensor technologies can track and record fine-granular movements.
• Big Data methods and their application in learning analy-tics can analyze and evaluate large amounts of data, which is especially indispensable for data-intensive learning, such as real-time analysis of psychomotor skills.
• Machine learning (e.g., reinforcement/deep learning) and generative artificial intelligence (e.g., generative adversarial networks) techniques can inter-pret and infer large data sets and generate individualized feedback.

To date, these technologies have largely been considered separately. MILKI-PSY aims to create AI-supported, data-intensive, multimodal, immersive learning environments for independent learning of psychomotor skills. In doing so, a cross-domain approach is emerging that enables multimodal recording of expert activities and the use of these recordings as blueprints for learners. With the help of artificial intelligence and automated error detection, the learning progress is analyzed and individual feedback is generated. This creates holistic, innovative learning environments for learning psychomotor skills, in which personalized, AI-supported learning support enables individual learning processes based on complex data analyses.

Funding: BMBF

Co-operation: TH Köln – University of Applied Sciences (Projektkoordination), Deutsches Forschungszentrum für künstliche Intelligenz, Rheinisch Westfälische Technische Hochschule, Deutsche Sporthochschule Köln, Institut für Produktentwicklung und Konstruktionstechnik, TH Köln

Duration: 07/21 - 06/24

Status: ongoing

Projektteam: Jan Schneider, Fernando Cardenas-Hernandez, Gianluca Romano

Kontakt: Jan Schneider