Assessing the Influence of Agricultural Research Investments on Biodiversity and Land Management

New, groundbreaking research shows how, at a local scale, agricultural research and development led to improved crop varieties that resulted in global benefits to the environment and food system sustainability. New, groundbreaking research shows how, at a local scale, agricultural research and development led to improved crop varieties that resulted in global benefits to the
HomeTechnologyRevolutionizing Development: A Streamlined Approach to Efficient Simulations and AI Models

Revolutionizing Development: A Streamlined Approach to Efficient Simulations and AI Models

Researchers have developed an automated solution to help programmers enhance the performance of their deep learning algorithms by effectively utilizing two forms of redundancy in intricate data structures: sparsity and symmetry.

Neural network AI models, commonly utilized in fields like medical imaging and speech recognition, process highly complex data structures which demand significant computational resources. This intensive requirement contributes to the high energy consumption of deep-learning models.

To boost the efficiency of AI models, researchers from MIT introduced an automated system that empowers developers of deep learning algorithms to simultaneously utilize both types of data redundancy. This approach significantly lessens the computational load, bandwidth usage, and memory requirements for machine learning tasks.

Current methods for refining algorithms can be burdensome and often restrict developers to leveraging either sparsity or symmetry, which are two distinct redundancies present in deep learning data structures.

By allowing developers to create an algorithm from the ground up that harnesses both redundancies together, the MIT team’s method increased computation speeds by almost 30 times in some tests.

This system features a straightforward programming language, which makes it accessible for optimizing machine-learning algorithms across various applications. It could also assist scientists who may not be specialists in deep learning but wish to enhance the efficiency of the AI algorithms they employ for data processing. Moreover, the system holds potential in the realm of scientific computing.

“Capturing these data redundancies has historically required substantial implementation effort. Now, a scientist can simply specify what they want to compute in a more abstract manner, without needing to detail how to achieve it,” explains Willow Ahrens, an MIT postdoctoral researcher and co-author of a paper on the system, which will be presented at the International Symposium on Code Generation and Optimization.

She is joined by lead author Radha Patel ’23, SM ’24, and senior author Saman Amarasinghe, a professor in the Department of Electrical Engineering and Computer Science (EECS) and a principal researcher at the Computer Science and Artificial Intelligence Laboratory (CSAIL).

Reducing Computational Load

In machine learning, data is frequently represented and manipulated through multidimensional arrays called tensors. A tensor is akin to a matrix, which is a rectangular arrangement of numbers on two axes (rows and columns). However, unlike a two-dimensional matrix, a tensor can encompass multiple dimensions, making it more intricate to manipulate.

Deep learning models perform operations on tensors using repeated matrix multiplication and addition—a process used by neural networks to discern complex patterns in data. The extensive number of calculations required for these multidimensional data structures necessitates vast computational power and energy.

However, due to the arrangement of data within tensors, engineers can frequently expedite a neural network’s operations by eliminating unnecessary computations.

For example, if a tensor represents user reviews from an e-commerce platform, since not all users review every product, many entries in that tensor are likely to be zero. This kind of redundancy is termed sparsity, and models can conserve time and computational resources by exclusively processing non-zero values.

Additionally, if a tensor is symmetric, meaning the values in the upper half mirror those in the lower half, the model only needs to process one half, thus reducing computation. This redundancy is referred to as symmetry.

“However, trying to optimize for both redundancies can complicate the situation,” Ahrens points out.

To streamline this, Ahrens and her team developed a new compiler— a software program that converts complex code into a simpler format for machine processing. Their compiler, named SySTeC, can enhance computations by automatically exploiting both sparsity and symmetry in tensors.

The development of SySTeC began with identifying three principal optimizations achievable through symmetry.

First, if the output tensor from the algorithm is symmetric, it only needs to compute one half. Second, if the input tensor is symmetric, the algorithm only has to read one half. Finally, if the intermediate results of tensor operations are also symmetric, the algorithm can skip redundant calculations.

Simultaneous Optimizations

When using SySTeC, a developer submits their program, and the system automatically refines their code to enhance it based on all three symmetry types. The subsequent phase of SySTeC applies further transformations to ensure only non-zero values are stored, optimizing the program for sparsity.

Ultimately, SySTeC produces ready-to-use code.

“This approach allows us to reap the rewards of both optimizations. Interestingly, as the dimensions of your tensor increase, the potential for computational savings also grows,” Ahrens notes.

The researchers achieved speed increases of nearly 30 times using the code generated automatically by SySTeC.

Given its automated nature, this system could be particularly beneficial for scientists looking to process data using a newly developed algorithm.

In the future, the researchers hope to incorporate SySTeC into existing sparse tensor compiler systems to create a more seamless user experience. They also aim to use it for optimizing more complex programs.

This research is partially funded by organizations such as Intel, the National Science Foundation, the Defense Advanced Research Projects Agency, and the Department of Energy.