Department of Media Technology and Design - Capturing of sign language gestures and visualization through real-time 3D computer animations

Communication with deaf people can be improved by offering hearing people the opportunity to learn sign language on a larger scale. For this purpose, an Austrian multimedia company is developing a CD-ROM-based training program for the Austrian Sign Language.

Up to now, single signs are shown through video sequences within the multimedia product. Many advantages, however, can be provided by visualizing signs through 3D animations. This has lead to a common research project with the Department of Media Technology and Design at the Hagenberg Polytechnic University, titled "3D Techniques for Computer-Based Training in Sign Language".

Making use of a pair of CyberGlove Systems 3D CyberGloves, Polhemus Fastrak sensors, and the 3D animation program Softimage 3D, a motion capture system has been developed to record signs with parallel real-time visualization. With the used sensor constellation, the movement of the hands, arms, upper body, and head can be captured. CyberGlove has proved to be sufficiently accurate for the reproduction of hand and finger movement, while also providing appropriate calibration and visualization software.

For the playback of captured 3D signs in real-time, an animation player has been developed for flexible 3D interaction. The user can, for example, orbit around the 3D character while the animation is looping, choose an arbitrary speed while the full frame rate is maintained, or drag the sign animation to an arbitrary time. Thus the learning of signs is facilitated considerably and the intuitive 3D interface is convenient to use. In the near future, this animation player will be integrated into the sign language training product of the cooperation partner.