Sophisticated 3D Measurement Technology Permits Gesture-Based Human-Machine Interaction in Real Time

3D sensor technology
3D sensor technology: two high-speed cameras and a gobo projector capture the 3D data of the moving person. © Fraunhofer IOF

Humans and machines will increasingly support each other in the workplace. For processes to be efficient, the machine must respond to the human worker without any time delay. Thanks to sophisticated high-speed 3D measurement and sensor technology, researchers at the Fraunhofer Institute for Applied Optics and Precision Engineering IOF are making this real-time interaction possible. They will be demonstrating how this works in practice at the Hannover Messe trade show from April 1 to 5, 2019 using the example of an interactive wall of spheres that reacts immediately, contact-free and in three dimensions to every movement of a person standing in front of it (Hall 2, Booth C22).

As gesture control represents a seamless interface between humans and machines, more and more machines, robots and devices are able to respond to gestural cues. Researchers at Fraunhofer IOF in Jena are raising human-machine interaction to a new level: the high-speed 3D measurement and sensor technology developed in the 3D-LivingLab research project (see “3D-LivingLab” box) enables them to capture and interprete even complex movements – and does so in real time. At Hannover Messe 2019, the research team is demonstrating its gesture-based human-machine interaction technology using the example of a wall made up of 150 spheres, which copies in 3D every head, arm and hand movement of a person standing in front of it. The wall of spheres effectively imitates the body movements with contact-free 3D reactions in real time, free from irritating time lags. The wall of spheres was created as part of the “3D-LivingLab“ project.

Workflows are greatly simplified

The system is made up of several modules: 3D sensor, 3D data processing and image fusion as well as the actuator system itself comprising 150 individual actuators. “The wall of spheres is not only a great toy, it also represents cutting-edge technology. Real-time 3D capture and interpretation of multiple gestures without tracking sensors can radically simplify workflows – from production scenarios to health and safety,” says Dr. Peter Kühmstedt, scientist and group leader at Fraunhofer IOF.

The demonstrator system responds to the behavior of people, captures complex movements such as gestures and physical actions, and gives real-time feedback through a technical actuator system that converts electrical signals into movement on the wall of spheres. It is the person’s posture that controls the actuators. Specially developed algorithms enable human 3D movements to trigger control of the actuators, thus causing the spheres to move. “We are demonstrating very rapid measurement technology – the data is captured by a new generation of 3D sensors –, very rapid low-latency processing – the data is interpreted and converted immediately – and very rapid reactions in real time. According to the calculation results, the wall of spheres immediately mirrors the movement of the person in front of it,” says the researcher. In production environments, for example, the technology could be used to monitor a worker who is interacting with a robot and handing it parts. It could also be transferred to other application fields, such as health and safety, where it can make processes safer and more efficient. Other conceivable applications for the 3D sensor technology and the interaction components are in assembly assistance and quality control systems. Moreover, they also qualify for the monitoring of biometric access points.

The researchers are presenting their high-speed 3D technology which facilitates symbiosis between humans and machines, at the Hannover Messe. It will be on display at the Fraunhofer booth (Hall 2, Booth C22) from April 1 to 5, 2019.