The praying mantis stands out among insects due to its compound eyes and the unique ability to see in three dimensions. Engineers are now trying to imitate this visual system to enhance machine vision.
Self-driving cars sometimes fail to navigate properly because their vision systems struggle to interpret stationary or slower-moving objects in 3D. This limitation is similar to insects with monocular vision, whose compound eyes excel in tracking motion and offering a broad field of view but have limited depth perception.
But the praying mantis is different.
The eyes of a praying mantis have a field of view that overlaps, giving it binocular vision and true depth perception in three dimensions.
Researchers at the University of Virginia School of Engineering and Applied Science have harnessed this unique feature, integrating advanced optoelectrical engineering and innovative “edge” computing—which processes data close to where it is collected—to create artificial compound eyes. This technology aims to tackle existing challenges that hinder machines from accurately interpreting real-world visual data, such as accuracy problems, lag in data processing, and the high computational demand.
“Our investigation of how mantis eyes function led us to realize that a biomimetic system would require the development of new technologies to emulate their biological strengths,” stated Byungjoon Bae, a Ph.D. candidate in the Charles L. Brown Department of Electrical and Computer Engineering.
About These Imitative Eyes
The research team designed “eyes” that replicate the mantis’s natural design by using microlenses combined with multiple photodiodes that generate an electrical signal when they encounter light. They created these components with flexible semiconductor materials to replicate the mantis eyes’ curved shapes and faceted arrangements.
“Creating the sensor in a hemispherical shape while ensuring it performs effectively is a cutting-edge accomplishment, ensuring a wide field of view and enhanced depth perception,” Bae explained.
“This system provides accurate spatial awareness in real time, crucial for technologies that need to engage with changing environments.”
Possible applications include low-energy vehicles and drones, autonomous cars, robotic manufacturing, surveillance, security solutions, and smart home technology.
Bae, under the guidance of Kyusang Lee, an associate professor with expertise in materials science and engineering, is the lead author of the recent publication in Science Robotics.
One of the team’s significant outcomes in their prototype was a potential power consumption reduction greater than 400 times when compared to conventional visual systems.
The Advantages of Edge Computing
Instead of relying on cloud computing, Lee’s system efficiently processes visual data in real time, significantly reducing the time and resource expenditure associated with data transfer and external computations, leading to lower energy consumption.
“The innovative breakthrough of this research is found in the integration of flexible semiconductor materials, devices that maintain precise angles, an in-sensor memory element, and distinct post-processing algorithms,” Bae noted.
The sensor array continuously tracks the changes in its surroundings, recognizing which pixels shift and transforming this data into smaller datasets for processing.
This method reflects the way insects interpret their environment through visual signals, distinguishing dynamic pixels to comprehend motion and spatial information. For instance, like many insects—and humans—the mantis leverages the principle of motion parallax, where closer objects seem to move faster than those that are further away. While one eye can create this effect, motion parallax alone doesn’t provide accurate depth perception.
What makes mantis eyes extraordinary is their use of stereopsis—the ability to see with both eyes to gauge depth—alongside their hemispherical compound eye structure and motion parallax to navigate their world.
“The effective combination of these advanced materials and algorithms enables real-time, efficient, and precise perception of 3D space,” emphasized Lee, an emerging researcher in thin-film semiconductors and smart sensors.
“Our team’s research offers significant insights that could encourage engineers and scientists by presenting an innovative biomimetic approach to solving complex visual processing issues,” he added.
This research was funded by the National Science Foundation and the U.S. Air Force Office of Scientific Research.
