Successfully trained an AI model to learn language in the same way that a child learns language

There have been many studies on how young children associate newly learned words with specific objects and concepts, but there have been limitations to generalizing them. Researchers at New York University conducted an unprecedented study in which they used a head-mounted camera to record footage that could recreate a child's first-person experience and train an AI based on that footage.

Grounded language acquisition through the eyes and ears of a single child | Science

The research was conducted by Vong Wai-keen , Weng Wentao, Emin Orhan, and Brenden M. Lake of the New York University Center for Data Science.

The team fitted Lake's daughter, Luna, with a head-mounted camera and recorded what she saw and heard from the age of six months until she turned two years old.

The video included Luna playing, eating, listening to picture books being read to her, and other activities across the developmental spectrum, and included approximately 250,000 words, many of which were repeated.

Can AI Learn Language the Way Babies Do? - YouTube


The team trained the AI once a week using more than 60 hours of recorded video, with images extracted from the footage and transcribed audio fed into separate modules, which were then combined in a contrast-based learning process.

For example, when a parent is speaking to a child, some of the words being spoken may refer to something the child is looking at. By combining these clues, the child learns that 'this word refers to this object or concept.'

After training the 'Child's View for Contrastive Learning (CVCL) model,' Vuong and his colleagues tested the target words by presenting four different images to test which word they referred to, in the same way that infants' word learning is measured. The CVCL model showed that it had learned many words and concepts that children encounter in their daily lives.

'Our findings suggest that recent advances in algorithms, combined with a child's natural experiences, may reshape our understanding of early language and concept acquisition,' said Vuong, lead author of the paper.

'Using AI models to study the language-learning problems children face allows us to address a classic debate: What do children need to learn a language? It appears that learning alone can do more than we previously thought,' said Lake, an associate professor in the Center for Data Science and Department of Psychology and senior author of the paper.