Project Vaani steps up inclusivity with dataset for atypical speech in three languages

While speech technology for Indian languages has moved beyond the infancy stage, it still trails English and struggles with regional dialects and cultural contexts. The challenge becomes even more acute when it comes to atypical speech (speech that deviates from standard speech patterns) in Indian languages, as most Automatic Speech Recognition (ASR) systems are designed for standard speech.

Researchers at the Indian Institute of Science (IISc), Bengaluru, have been trying to address this by building a dataset of atypical speech in Indic Languages.

The Vaani Atypical Speech Corpus, a pilot initiative under Project Vaani by IISc and ARTPARK, has so far collected approximately 10 hours of atypical speech from around 40 speakers across three languages and aims to expand the corpus in the coming days.

Missing data

Atypical speech refers to speech patterns that deviate from conventional speech. It may result from neurological, cognitive, or motor conditions. Efforts are now under way across the globe to train speech recognition software on atypical speech, making ASR systems more inclusive. “For Indian languages, there is no such data,” says Prasanta Kumar Ghosh of the Department of Electrical Engineering at IISc, who leads Project Vaani.

Project Vaani aims to capture India’s diverse languages, dialects, and speech patterns and build a corpus of 150,000 hours of natural speech data from around 1 million people across nearly 800 Indian districts. It was while working on this project that the team began exploring technology for atypical speech.

Addressing a gap

“What about people who find it difficult to speak naturally? How do we build technology for them? Why not create technology that is inclusive, instead of delivering it only to those who can speak naturally?” Mr. Ghosh explains.

With this in mind, the team set out to build a dataset of atypical speech in Indian languages. The pilot project was carried out in collaboration with Google’s Project Euphonia. However, Mr. Ghosh notes that this is a long-term initiative and hopes to expand the currently small dataset in the future, enabling further research and the development of speech technologies.

“The target corpus size is still under discussion. We are open to funding from more partners. We are also open to collaboration for both data collection and funding,” Mr. Ghosh says.

Meanwhile, the team has completed two phases of the larger Project Vaani, covering 165 Indian districts, 156,534 speakers, and 105 languages. Around 200 hours of speech data have been recorded from each district, resulting in more than 31,000 hours of speech data, which is open-sourced. Around 10-15 districts have been covered in Karnataka, capturing 18 languages including Kannada, Tulu, Beary Bashe and Marathi.

The data is collected through an ‘image-prompted recording’, where the speaker is asked to describe an image shown to them. Generic images as well as images specific to the locality are shown to the speakers. The team has prepared around 1200 generic images and 1500-2000 district-specific images.  

“This way we get multi-modal information, that is image and audio data, both in local dialect,” explains Mr. Ghosh. 

Each speaker spends around 20 seconds speaking about one image. The number of images per speaker is limited to 60. This is to ensure variety in images and speakers, explains Mr Ghosh. The IISc team, through its data collection partners, also tries to ensure balance in terms of gender, age, socio-economic status and education. 

Around 10% of the collected data has been transcribed so far, amounting to more than 2000 hours of transcription.