Researchers are developing a method for robots to gain a sense of touch by ‘listening’ to vibrations, enabling them to recognize materials, comprehend shapes, and identify objects similarly to how humans use their hands. Humans naturally use acoustic vibrations from objects—like shaking a cup to estimate how much soda remains or tapping an armrest to check if it’s wood. This instinctive ability is something that scientists are close to imparting to robots, enhancing their expanding sensory capabilities.
Imagine you’re in a dark movie theater, wondering how much soda is left in your giant cup. Instead of removing the lid to look, you give the cup a shake and listen to the ice rattling, helping you gauge whether you’ll need a refill.
After setting the cup down, you might ponder whether the armrest is genuine wood. A few taps revealing a hollow sound lead you to conclude it’s plastic.
This intuitive interpretation of our environment through sounds generated by objects is something we do automatically. Researchers are now on the brink of embedding this skill into robots, which would significantly enhance their sensory perceptions.
Presented at the upcoming Conference on Robot Learning (CoRL 2024) from November 6-9 in Munich, Germany, new research from Duke University introduces a system called SonicSense, allowing robots to engage with their environment in ways previously achievable only by humans.
“Currently, robots predominantly rely on vision to understand their surroundings,” explained Jiaxun Liu, the lead author of the study and a first-year Ph.D. student under Boyuan Chen, a professor at Duke specializing in mechanical engineering and materials science. “We aimed to develop a solution that could handle the complexity and variety of everyday objects, giving robots a more enhanced ability to ‘feel’ and grasp the environment.”
SonicSense consists of a robotic hand with four fingers, each fitted with a contact microphone at the fingertip. These sensors pick up and capture vibrations created while the robot taps, holds, or shakes an object. Since the microphones touch the object, they can filter out background noise.
Through the recorded interactions and signals received, SonicSense can extract frequency characteristics and, using prior knowledge alongside recent AI advancements, determine the material of the object and its 3D shape. If the system encounters an unfamiliar object, it might require up to 20 interactions to identify it, but it can successfully recognize familiar objects with just four.
“SonicSense provides a novel way for robots to experience sound and touch, similar to humans, transforming how current robots perceive and interact with objects,” stated Chen, who is also involved with electrical and computer engineering and computer science departments. “While vision plays a vital role, sound can reveal additional information that may be overlooked by sight.”
In their paper and demonstrations, Chen and his team illustrate various capabilities enabled by SonicSense. For instance, when shaking a box containing dice, the system can not only count the dice inside but also identify their shape. Similarly, it can determine the volume of liquid in a bottle by shaking it, and by tapping the exterior of an object, it can create a 3D map of the object’s shape and identify its material.
While SonicSense is not the first system of its kind, it surpasses earlier attempts by utilizing four fingers instead of one, employing touch-sensitive microphones that can disregard ambient sounds, and leveraging advanced AI methodologies. This design enables recognition of objects made from multiple materials with complex shapes, reflective surfaces, and materials that are difficult for vision-based systems to analyze.
“Most datasets are compiled in controlled labs or through human observation, but we needed our robot to independently interact with objects in an unpredictable lab environment,” Liu remarked. “Replicating that complexity in simulations is challenging. Addressing the disparity between controlled environments and real-world data is crucial, and SonicSense eliminates that gap by allowing robots to engage directly with the diverse and chaotic aspects of the real world.”
These features make SonicSense a strong foundation for training robots to perceive objects in dynamic, unstructured settings. Additionally, the system is cost-effective; by using common contact microphones typical in music recording, 3D printing, and other widely available components, the construction cost is kept at just over $200.
Looking ahead, the team is focused on improving the system’s capacity to manage multiple objects simultaneously. By incorporating object-tracking algorithms, the robots will be able to navigate dynamic and cluttered spaces, moving closer to mimicking human-like adaptability for real-world tasks.
Another significant advancement lies in the design of the robotic hand itself. “This is merely the beginning. In the future, we foresee SonicSense being integrated into more sophisticated robotic hands with enhanced manipulation skills, enabling robots to perform tasks that demand delicate touch,” said Chen. “We are eager to explore future development of this technology to incorporate additional sensory modalities, such as pressure and temperature, for even more intricate interactions.”
This project received support from the Army Research Laboratory STRONG program (W911NF2320182, W911NF2220113) and DARPA’s FoundSci program (HR00112490372) as well as TIAMAT (HR00112490419).
CITATION: “SonicSense: Object Perception from In-Hand Acoustic Vibration,” Jiaxun Liu, Boyuan Chen. Conference on Robot Learning, 2024. An ArXiv version is available at: 2406.17932v2 and on the General Robotics Laboratory website.
