Developed at the Massachusetts Institute of Technology (MIT), DenseAV is capable of learning and understanding language by watching videos of people speaking. This innovative algorithm has potential applications in multimedia search, language learning, and robotics.

This new algorithm could potentially decode communication between animals.

Mark Hamilton, a PhD student in electrical engineering and computer science, leads this project alongside his colleagues at MIT’s Computer Science and Artificial Intelligence Laboratory. They aim to leverage machines to decode animal communication, starting with human language acquisition.

The inspiration for this algorithm came from a film. In one scene, a penguin falls and groans as it tries to get up. Hamilton observed that this groan seemed to imply a word, leading to the idea that sound and video could be used together to teach an algorithm a language.

This idea led to the development of DenseAV, a model designed to learn language by predicting visual content from sound. For example, hearing the phrase “bake the cake at 350” would lead the model to expect visuals of a cake or an oven.

To enable DenseAV to match sound and visual content across millions of videos, it needed to learn the context in which people were speaking. After training DenseAV on this matching task, the research team examined which pixels the model focused on while processing sounds.

When the word “dog” was spoken, the algorithm searched for dog images in the video stream, indicating it understood the meaning of the word. Similarly, when it heard a dog bark, it looked for dogs in the video.