AVATAR

Language is our most important means of communication. According to current knowledge, thousands of years ago, prehistoric humans began to gradually form an increasingly complex structure from simple sounds until the first language finally emerged. It is unclear exactly when this was, but what we do know today is the first written language: the Sumerian script was developed around 3300 BC. Today, around 6000 different languages are spoken in the world, all of which presumably developed from a common original language. This example alone shows that language is something that defines people.

This makes it all the more important to be able to articulate oneself correctly and make oneself understood to others. Children have to learn this process, from their first sounds to their first words to complete sentences. It is not uncommon for children to have problems - they struggle with certain syllables, have an incorrect tongue position and pronunciation, or have a lisp. At a certain point, it is important to work with an expert so that minor problems do not develop into permanent articulation disorders.

Working with a speech therapist requires regular appointments on site and lots of practice at home. And this is exactly what the AVATAR research project wants to make better and smarter - and has developed a speech therapy assistant with ERDF funding. An interdisciplinary consortium worked on playful therapy exercises as an app for mobile devices such as smartphones, tablets and laptops. "Alongside therapy with a speech therapist, training at home is the decisive factor for successful learning," explains Prof. Thiem from the Biomedical Engineering research focus at Fachhochschule Dortmund. "This is exactly where the innovation comes in. We are creating a child-friendly tool with which speech exercises can be significantly improved". The hoped-for research goal is to increase the quality of the therapy and at the same time shorten the therapy time.

Five players from science and the IT industry worked together to achieve this: Fachhochschule Dortmund, Ruhr University Bochum, Bochum University of Health, Phoenix Software GmbH in Bonn and PuckPresse GmbH in Cologne. In order to identify individual speech and language deficits in children, particularly those between the ages of four and six, cameras are used to analyze not only microphone signals but also mouth movements on the child's face. In a nutshell, the project consists of audiovisual speech recognition for speech therapy exercises in the form of an app. Distinctive features are extracted from the recorded audio and video signal using special signal processing software. The system then combines the audio and video signals using artificial neural networks to recognize pronunciation errors.

AVATAR automatically recognizes whether the child is forming the sounds correctly and offers child-friendly assistance to improve pronunciation. Prof. Thiem: "The aim is not to draw attention to the errors in an educational way, but to encourage and motivate the child's perception through the exercises and the feedback they generate in a playful way." The speech therapists involved in the therapy should follow the child's progress and adjust the home therapy exercises in the app accordingly. During the course of the project, the prototype developed was trained with data sets that were created at great expense thanks to cooperating children and parents. The software can now also be improved beyond the project using these data sets.