Discover data.
Connect the knowledge.

Explore language datasets, their collections, and the projects behind them.

109 public projects2 collections59 languages

Knowledge Atlas

Choose a collection to explore

Explore the public catalogue

All public Showcase projects, including those outside collections.

transcriptionsPartial public sample

BoLI Bagri Translation

### The Dataset The current data preview of Bagri is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This dataset was collected as part of the field visit to Kharagpur in 2022 for a field methods course as part of its M.A. Linguistics and M.Sc. in Computational Linguistics programs at Dr. Bhimaro Ambedkar University, Agra. This data was contributed to the project as part of the project's long-terms goals of digitising and publishing the datasets collected during different courses, research projects and other primary legacy materials. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 100 sentences in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Bagri Bagri belongs to the Indo-Aryan branch of the Indo-Iranian subgroup within the Indo-European language family. Structurally and historically, it is grouped with Rajasthani varieties while exhibiting strong transitional characteristics toward Punjabi. ### Speakers According to reports based on the [2011 Census of India](https://censusindia.gov.in/), Bagri has an estimated speaker population of approximately 1.9 to 2 million people, though exact counts carry uncertainty due to census classification practices that often group speakers under broader categories such as Hindi or Rajasthani. ### Distribution in India Bagri is spoken across a contiguous belt in northwestern India. Its principal geographic distribution includes: * **Rajasthan:** Northern districts such as Sri Ganganagar and Hanumangarh. * **Haryana:** Western districts including Sirsa and Fatehabad. * **Punjab:** Southwestern districts including Fazilka and Firozpur. ### Major grammatical features * **Phonology:** Possesses a contrasting set of dental and retroflex consonants (including /ɳ/ and /ɭ/). It exhibits pitch-accent or tonal characteristics similar to Punjabi, often arising from historical voiced aspirates. * **Morphology:** Uses a postpositional case system built upon inflected oblique stems. It marks masculine/feminine gender and singular/plural number across nouns, adjectives, and verbs. * **Syntax:** Features a canonical Subject-Object-Verb (SOV) word order and displays split ergativity, where agents of transitive verbs receive explicit ergative marking in perfective aspects. ### Further reading * Search or view the [Bagri language article on Wikipedia](https://en.wikipedia.org/wiki/Special:Search?search=Bagri%20language). * Refer to the [Glottolog 4.8 entry for Bagri](https://glottolog.org/resource/languoid/id/bagr1243). ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Bagri

View project ↗

transcriptionsPartial public sample

BoLI Mewati Translation

### The Dataset The current data preview of Mewati is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This dataset was collected as part of the field visit to Kharagpur in 2022 for a field methods course as part of its M.A. Linguistics and M.Sc. in Computational Linguistics programs at Dr. Bhimaro Ambedkar University, Agra. This data was contributed to the project as part of the project's long-terms goals of digitising and publishing the datasets collected during different courses, research projects and other primary legacy materials. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 100 sentences in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Mewati Mewati is a Western Indo-Aryan language, belonging to the Rajasthani group. ### Speakers According to the [2011 Census of India](https://censusindia.gov.in/), Mewati (often categorized under the broader umbrella of Hindi/Rajasthani varieties in official census returns) has an estimated 3 million to 3.6 million speakers. ### Distribution in India Mewati is primarily spoken in the Mewat cultural region of northern India, spanning: * **Haryana**: Primarily in Nuh district (formerly Mewat) and surrounding areas. * **Rajasthan**: Mainly in Alwar and Bharatpur districts. * **Uttar Pradesh**: Selected western border areas, particularly in Mathura district. * **Delhi**: Spoken by migrant communities in the National Capital Region. ### Major grammatical features * **Phonology**: Mewati features a standard Indo-Aryan phonemic system including contrastive retroflex stops (/ʈ/, /ɖ/), aspirated consonants across places of articulation, contrastive vowel length, and phonemic vowel nasalization. * **Morphology**: Nominal morphology relies on a binary gender system (masculine/feminine) and two grammatical numbers (singular/plural). Nouns undergo direct and oblique case distinctions, with overt postpositions indicating semantic roles. Verb paradigms inflect for aspect, tense, mood, and person-number agreement. * **Syntax**: Demonstrates head-final, Subject-Object-Verb (SOV) basic word order. Alignment shows split ergativity, where subjects of transitive verbs in perfective aspectual contexts take ergative postpositional marking. ### Further reading * Search for articles on [Mewati language on Wikipedia](https://en.wikipedia.org/wiki/Special:Search?search=Mewati%20language). * View language information on [Glottolog (mewa1250)](https://glottolog.org/resource/languoid/id/mewa12 ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Mewati

View project ↗

transcriptionsPartial public sample

BoLI Dhundhari Translation

### The Dataset The current data preview of Dhundhari is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This dataset was collected as part of the field visit to Kharagpur in 2022 for a field methods course as part of its M.A. Linguistics and M.Sc. in Computational Linguistics programs at Dr. Bhimaro Ambedkar University, Agra. This data was contributed to the project as part of the project's long-terms goals of digitising and publishing the datasets collected during different courses, research projects and other primary legacy materials. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 100 sentences in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Dhundhari Dhundhari is an Indo-Aryan language of Western/Central Zone, specifically belonging to the Rajasthani group.. ### Speakers According to the 2011 Census of India, Dhundhari (classified as a mother tongue under Rajasthani) is spoken by approximately 1.5 million people ([Census of India 2011 - Language Statement](https://censusindia.gov.in/nada/index.php/catalog/42561)). ### Distribution in India Spoken predominantly in the Dhundhar region of Rajasthan, encompassing the districts of Jaipur, Dausa, Tonk, Sawai Madhopur, and adjacent areas in central-eastern Rajasthan. ### Major grammatical features * **Phonology:** Maintains a typical Indo-Aryan consonant inventory with a four-way contrast in stops (voiced/voiceless, aspirated/unaspirated), retroflex consonants, and contrastive vowel nasalization. * **Morphology:** Nouns inflect for gender (masculine, feminine), number (singular, plural), and case (direct, oblique) primarily expressed through postpositions; verbal morphology marks tense, aspect, mood, and person-number agreement. * **Syntax:** Exhibits dominant Subject-Object-Verb (SOV) word order, postpositional phrase structure, and a split-ergative alignment system in perfective aspectual contexts. ### Further reading * [Dhundhari language on Wikipedia](https://en.wikipedia.org/wiki/Special:Search?search=Dhundhari%20language) ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Dhundhari

View project ↗

transcriptionsPartial public sample

BoLI Telugu Translation

### The Dataset The current data preview of Telugu is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This dataset was collected as part of the field visit to Kharagpur in 2022 for a field methods course as part of its M.A. Linguistics and M.Sc. in Computational Linguistics programs at Dr. Bhimaro Ambedkar University, Agra. This data was contributed to the project as part of the project's long-terms goals of digitising and publishing the datasets collected during different courses, research projects and other primary legacy materials. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 100 sentences in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Telugu Telugu is a major Dravidian language belonging to the South-Central Dravidian subgroup (ISO 639-3: `tel`, Glottocode: [`telu1262`](https://glottolog.org/resource/languoid/id/telu1262)). ### Speakers According to the 2011 Census of India, Telugu has [81,127,740 native speakers](https://censusindia.gov.in/census.website/node/281), making it one of the most widely spoken Dravidian languages. ### Distribution in India Telugu is predominantly spoken in the southern states of Andhra Pradesh and Telangana, where it holds official status. Significant populations of Telugu speakers are also found in neighboring states, including Tamil Nadu, Karnataka, Maharashtra, Odisha, and the union territory of Puducherry. ### Major Grammatical Features * **Phonology**: Telugu phonology features a contrast between short and long vowels, retroflex consonants (/ʈ/, /ɖ/, /ɳ/, /ɭ/), and a historical vowel harmony system. Most native words end in a vowel (typically /u/), a phonetic feature that historically earned it the description "Italian of the East." * **Morphology**: It is a highly agglutinative, suffixing language. Nouns inflect for number (singular and plural) and case (nominative, accusative, dative, genitive, locative, etc.) using postpositions. Verbs inflect for tense (past, present, future), aspect, mood, and show agreement with the subject in person, number, and gender. * **Syntax**: The basic word order is Subject-Object-Verb (SOV). Modifiers, such as adjectives and genitives, typically precede the head noun, and the language exclusively uses postpositions. ### Further Reading * [Wikipedia Article on Telugu](https://en.wikipedia.org/wiki/Telugu_language) * [Glottolog Page for Telugu](https://glottolog.org/resource/languoid/id/telu1262) ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Telugu

View project ↗

transcriptionsPartial public sample

BoLI Hmar PlantLife Narration

### The Dataset The current data preview of Hmar is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 10 narrations related to plant life in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Hmar Hmar belongs to the Kuki-Chin branch of the Tibeto-Burman language family. ### Speakers According to the 2011 Census of India, Hmar has approximately 98,394 native speakers ([2011 Census of India Language Data](https://censusindia.gov.in/nada/index.php/catalog/42561)). Exact global counts remain uncertain due to cross-border migration and local dialectal overlap. ### Distribution in India The language is primarily spoken across Northeast India, with concentrated populations in Mizoram, southern Manipur (particularly in Pherzawl and Churachandpur districts), Assam (specifically Dima Hasao and Cachar districts), Meghalaya, and Tripura. ### Major grammatical features * **Phonology:** Hmar is a tonal language utilizing distinct pitch patterns (typically high, low, and falling tones) to distinguish lexical meaning. It features a syllable-structure characteristic of Kuki-Chin languages, including initial consonant clusters, aspirated stops, and final glottal stops. * **Morphology:** The morphology is largely agglutinative with strong isolating tendencies. A prominent feature is verb stem alternation (Stem I vs. Stem II), where verb roots change form depending on syntactic context, clause type (main vs. subordinate), aspect, or nominalization. Pronominal prefixes mark agreement on verbs and possession on nouns. * **Syntax:** It exhibits a basic Subject-Object-Verb (SOV) word order and employs postpositional markers. Syntactically, Hmar displays ergative-absolutive alignment, where transitive subjects receive explicit ergative case marking while intransitive subjects and direct objects remain unmarked in the absolutive case. ### Further reading * [Hmar language on Wikipedia](https://en.wikipedia.org/wiki/Hmar_language) (Fallback search: [Search Wikipedia for Hmar language](https://en.wikipedia.org/wiki/Special:Search?search=Hmar%20language)) * [Glottolog 5.0 Entry: Hmar (hmar1241)](https://glottolog.org/resource/languoid/id/hmar1241) * [SIL ISO 639-3 Documentation: hmr](https://iso639-3.sil.org/code/hmr) ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Hmar

View project ↗

transcriptionsPartial public sample

BoLI Hmar Transport Narration

### The Dataset The current data preview of Hmar is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 10 narrations related to transport in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Hmar Hmar belongs to the Kuki-Chin branch of the Tibeto-Burman language family. ### Speakers According to the 2011 Census of India, Hmar has approximately 98,394 native speakers ([2011 Census of India Language Data](https://censusindia.gov.in/nada/index.php/catalog/42561)). Exact global counts remain uncertain due to cross-border migration and local dialectal overlap. ### Distribution in India The language is primarily spoken across Northeast India, with concentrated populations in Mizoram, southern Manipur (particularly in Pherzawl and Churachandpur districts), Assam (specifically Dima Hasao and Cachar districts), Meghalaya, and Tripura. ### Major grammatical features * **Phonology:** Hmar is a tonal language utilizing distinct pitch patterns (typically high, low, and falling tones) to distinguish lexical meaning. It features a syllable-structure characteristic of Kuki-Chin languages, including initial consonant clusters, aspirated stops, and final glottal stops. * **Morphology:** The morphology is largely agglutinative with strong isolating tendencies. A prominent feature is verb stem alternation (Stem I vs. Stem II), where verb roots change form depending on syntactic context, clause type (main vs. subordinate), aspect, or nominalization. Pronominal prefixes mark agreement on verbs and possession on nouns. * **Syntax:** It exhibits a basic Subject-Object-Verb (SOV) word order and employs postpositional markers. Syntactically, Hmar displays ergative-absolutive alignment, where transitive subjects receive explicit ergative case marking while intransitive subjects and direct objects remain unmarked in the absolutive case. ### Further reading * [Hmar language on Wikipedia](https://en.wikipedia.org/wiki/Hmar_language) (Fallback search: [Search Wikipedia for Hmar language](https://en.wikipedia.org/wiki/Special:Search?search=Hmar%20language)) * [Glottolog 5.0 Entry: Hmar (hmar1241)](https://glottolog.org/resource/languoid/id/hmar1241) * [SIL ISO 639-3 Documentation: hmr](https://iso639-3.sil.org/code/hmr) ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Hmar

View project ↗

transcriptionsPartial public sample

BoLI Hmar Narration

### The Dataset The current data preview of Hmar is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 10 narrations in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Hmar Hmar belongs to the Kuki-Chin branch of the Tibeto-Burman language family. ### Speakers According to the 2011 Census of India, Hmar has approximately 98,394 native speakers ([2011 Census of India Language Data](https://censusindia.gov.in/nada/index.php/catalog/42561)). Exact global counts remain uncertain due to cross-border migration and local dialectal overlap. ### Distribution in India The language is primarily spoken across Northeast India, with concentrated populations in Mizoram, southern Manipur (particularly in Pherzawl and Churachandpur districts), Assam (specifically Dima Hasao and Cachar districts), Meghalaya, and Tripura. ### Major grammatical features * **Phonology:** Hmar is a tonal language utilizing distinct pitch patterns (typically high, low, and falling tones) to distinguish lexical meaning. It features a syllable-structure characteristic of Kuki-Chin languages, including initial consonant clusters, aspirated stops, and final glottal stops. * **Morphology:** The morphology is largely agglutinative with strong isolating tendencies. A prominent feature is verb stem alternation (Stem I vs. Stem II), where verb roots change form depending on syntactic context, clause type (main vs. subordinate), aspect, or nominalization. Pronominal prefixes mark agreement on verbs and possession on nouns. * **Syntax:** It exhibits a basic Subject-Object-Verb (SOV) word order and employs postpositional markers. Syntactically, Hmar displays ergative-absolutive alignment, where transitive subjects receive explicit ergative case marking while intransitive subjects and direct objects remain unmarked in the absolutive case. ### Further reading * [Hmar language on Wikipedia](https://en.wikipedia.org/wiki/Hmar_language) (Fallback search: [Search Wikipedia for Hmar language](https://en.wikipedia.org/wiki/Special:Search?search=Hmar%20language)) * [Glottolog 5.0 Entry: Hmar (hmar1241)](https://glottolog.org/resource/languoid/id/hmar1241) * [SIL ISO 639-3 Documentation: hmr](https://iso639-3.sil.org/code/hmr) ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Hmar

View project ↗

transcriptionsPartial public sample

BoLI Santhali Fieldwork

### The Dataset The current data preview of Santhali is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This dataset was collected as part of the field visit to Kharagpur in 2022 for a field methods course as part of its M.A. Linguistics and M.Sc. in Computational Linguistics programs at Dr. Bhimaro Ambedkar University, Agra. This data was contributed to the project as part of the project's long-terms goals of digitising and publishing the datasets collected during different courses, research projects and other primary legacy materials. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 100 sentences in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Santhali Santhali (ISO 639-3: `sat`, Glottocode: `sant1410`) belongs to the **Austro-Asiatic** language family, specifically classified within the North Munda subgroup of the Munda branch. ### Speakers According to the [2011 Census of India](https://censusindia.gov.in/nada/index.php/catalog/42517), there are approximately 7,368,192 speakers of Santhali in India. ### Distribution in India Santhali is primarily spoken in the eastern states of India. Its major concentrations are in: * **Jharkhand** (predominantly in the Santhal Pargana division) * **West Bengal** * **Odisha** * **Bihar** Minority speaker communities are also found in Assam and Tripura. ### Major Grammatical Features * **Phonology:** Santhali features a complex phonological inventory including a contrast between voiced, voiceless, aspirated, and unaspirated stops, alongside a distinctive series of retroflex consonants. A defining characteristic of its sound system is the presence of "checked" or glottalized, unreleased stops ($[p']$, $[t']$, $[c']$, $[k']$) in syllable-final positions. Vowel harmony is also widely observed. * **Morphology:** The language is highly agglutinative. Nouns are inflected for three grammatical numbers (singular, dual, and plural) and are categorized by animacy (animate vs. inanimate) rather than grammatical gender. Pronouns feature a distinction between inclusive and exclusive first-person plural ("we including you" vs. "we excluding you"). * **Syntax:** Santhali generally exhibits a Subject-Object-Verb (SOV) word order. Its verbal morphology is exceptionally complex; verbs are inflected to agree with both the subject and the object (direct and indirect) through a highly intricate system of pronominal clitics, tense, aspect, and mood markers. ### Further Reading * [Wikipedia: Santali language](https://en.wikipedia.org/wiki/Santali_language) (Alternative search: [Santhali language on Wikipedia](https://en.wikipedia.org/wiki/Special:Search?search=Santhali%20language)) ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Santali

View project ↗

transcriptionsPartial public sample

BoLI Mara Translation

### The Dataset The current data preview of Mara is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 100 sentences in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Mara Mara belongs to the Kuki-Chin branch of the Tibeto-Burman language family. According to the [2011 Census of India](https://censusindia.gov.in/), Mara (often recorded under Lakher or Mara categories) has approximately 42,000 speakers in India, with additional speaker populations in neighboring Myanmar bringing total estimate projections higher. ### Distribution in India The language is primarily spoken in the state of Mizoram in Northeast India, predominantly within the Mara Autonomous District Council (Siaha district). ### Major grammatical features * **Phonology:** Monosyllabic and sesquisyllabic root structure with a contrastive tone system, contrastive aspiration in stops, and glottal final consonants. * **Morphology:** Highly agglutinative structure featuring verbal stem alternation (Stem I and Stem II variations characteristic of Kuki-Chin languages), bound pronominal agreement prefixes, and nominalizing affixes. * **Syntax:** Canonical Subject-Object-Verb (SOV) basic word order with split-ergative or ergative-absolutive alignment in case marking and agreement structures. ### Further reading * Search or view details on the [Mara language Wikipedia page](https://en.wikipedia.org/wiki/Special:Search?search=Mara%20language). * Consult Glottolog metadata and dialect documentation under Glottocode `mara1382` (ISO 639-3: `mrh`). ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Mara

View project ↗

transcriptionsWhole project public

BoLI Irula Translation

### The Dataset The current data preview of Irula is being released as part of the **[Project BoLI](https://boli.unreal-tece.co.in)**. This preview is a reflection of the full dataset and consists of the following - 1. Speech Recordings of 100 sentences in the language. 2. Transcriptions in IPA and native script(s). 3. Translations in English (which also act as prompts for the translation sentences). 4. Detailed speaker metadata, including their demographic, educational and linguistic profile. 5. Prompt in English and Hindi. The full dataset contains the following - 1. Translations of a minimum of 1000 carefully selected sentences. These sentences are selected to represent diverse morphosyntactic categories generally found in Indian languages such as demonstratives, classifiers, TAM morphology, and different sentence structures such as transitives and ditransitives. These sets of sentences are based on the standardised questionnaires built by Linguists for writing the sketch grammar of any human language, thereby, representing almost full range of morphosyntactic properties exhibited in the language. Unlike other benchmarks which focus largely on lexical level evaluation (aka domains), this is the first benchmark dataset that evaluates model's performance on a range of morphosyntactic structures, even the most uncommon ones. 2. More than 600 narrative speeches collected across 8 domains. These are recorded by at least 2 speakers. Narrations, along with their prompts, can be used to evaluate AI models on a range of prompt-based tasks. 3. Along with transcriptions in IPA and other scripts and translation, all data is interlinearly glossed at morphemic level. This gives a word-by-word meaning and morphosyntactic information, thereby, enabling evaluation of models on their deep grammatical knowledge, ability for cross-linguistic comparison and generalisation and reasoning capacity and skills in language-related puzzles. This allows for evaluating the model's capacity on reasoning tasks beyond mathematical reasoning tasks as well as their capacity to arrive at typological generalisations. 4. In addition to the data and prompt itself, as mentioned earlier, we are also making available the detailed speaker metadata and prompt-level metadata viz domain, elicitation method, multilingual prompt, target grammatical category (for translation sentences), etc. 5. In accordance with our data governance policy, all contributors to the dataset, including those recording the dataset and those transcribing it, are named and listed as data contributors. ### Dataset Preparation The speech included as part of this dataset was recorded by native speakers using a data collection application on a mobile device (Karya or in-house app, Atekho), by translating sentences from English or Hindi to the target language. The recorded speech was validated and then transcribed manually by trained linguists, working with the native speakers, following the [guidelines for the project](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) using MATra Lab, a part of the [LiFE Suite Ecosystem](https://life.unreal-tece.co.in), developed by [Unreal Tece LLP](https://unreal-tece.co.in). [Project BoLI Guidelines](https://docs.google.com/document/d/1h-EcxxaNiGzdWiwc1-V_S-Xt6n4MwE6F/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### About Irula Irula belongs to the Dravidian, Southern Dravidian, Tamil-Kannada sub-branch (ISO 639-3: `iru`, Glottocode: `irul1243`). ### Speakers According to the [2011 Census of India](https://censusindia.gov.in/nada/index.php/catalog/42561), the Irula speech community is estimated at approximately 200,000 native speakers. ### Distribution in India Irula is primarily spoken in Southern India. Its primary geographic concentration includes the Western Ghats region of Tamil Nadu (specifically the Nilgiris, Coimbatore, and Erode districts, as well as northern districts like Tiruvallur and Kanchipuram) and adjacent areas in Kerala (primarily the Attappadi valley in Palakkad district), extending into border areas of Karnataka. ### Major grammatical features * **Phonology:** Contains a typical Southern Dravidian consonant system with a full series of retroflex stops, nasals, and liquids, as well as distinct centralized short and long vowels (such as /ɨ/ and /ɵ/). * **Morphology:** Strictly agglutinative. Substantives inflect for case (including dative, locative, sociative, and ablative) using suffixes, while verbs take suffixal markers for tense, aspect, mood, and person-number-gender (PNG) agreement. * **Syntax:** Characterized by a canonical Subject-Object-Verb (SOV) default word order, head-final noun phrases, use of postpositions, and non-finite clause subordination (such as verbal participles). ### Further reading * [Irula language on Wikipedia](https://en.wikipedia.org/wiki/Irula_language) * [Glottolog Language Details: Irula of the Nilgiri](https://glottolog.org/resource/languoid/id/irul1243) * [ISO 639-3 Code Aspect: `iru`](https://iso639-3.sil.org/code/iru) ### About Project BoLI [Project BoLI](https://boli.unreal-tece.co.in) is the flagship project of [UnReaL-TecE LLP](https://unreal-tece.co.in), which aims to build high-quality datasets for benchmarking and evaluating different kinds of AI tasks, including speech-to-text, machine translation, grammatical analysis and reasoning tasks and prompt-based evaluation of LLMs. While the project aims to build these datasets for every Indian language and variety, the primary focus is on over 1300 underserved languages and both first and second language varieties of major, scheduled languages. The project's uniqueness is not just limited to the kind of benchmarking tasks it supports (including proposing some novel tasks) and also the kind of languages and communities it supports but also in its contextualisation and implementation of a unique data governance model (not yet implemented anywhere across the globe), which mandates that all datasets released by the project and their derivatives (including the models) are co-owned by the community members and all contributors of the project, and any permission to use the dataset is not a transfer of ownership but a revocable license to use it. The conditions under which the license could be revoked are clearly mentioned as part of the BoLI License. The complete details of the project, languages and communities supported till now, the quantum of data available till now, its data governance model and other relevant documents are all publicly accessible at the [project website](https://boli.unreal-tece.co.in). ### Ethical Considerations, Consent, IPR and Attribution Project BoLI and this repository represents our commitment to not only fair remuneration to the speakers of the language but also to co-ownership and equal IPR to all the contributors who have built the dataset. We believe this is the first step to move away from the extractive data collection and use practices and ensure fairness in our treatment of the community members. As such, we have listed all speakers and transcribers as Contributors to the dataset (we insist that they are co-owners of the dataset, even though the HuggingFace platform does not provide us an explicit way of stating that) and they are further recognised as Speakers and Annotators of the dataset. This dataset is only licensed to other researchers for use in their research projects. More details about licensing and commercial use conditions are given in the License and Commercial Use sections. [Project BoLI - Data Governance Policy](https://docs.google.com/document/d/1DBGZVFJ4RG57j4rFphKIrvgbpfpj8HqZ/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [BoLI Ethics Principle & Pledge](https://docs.google.com/document/d/1hMQzwsOJF4QDfk_DOSIGLOF3GApLQOZA/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Digital Consent Form](https://docs.google.com/document/d/1BTQUrgVTS-sVszUUDOkbnyUNusAHVYm4/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - Field Recording of Oral Consent](https://docs.google.com/document/d/1xoRxXcNES2oQqBajY3Z7HxMwpt0oOyDK/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) [Project BoLI - TnC](https://docs.google.com/document/d/1UuiIAHWXXKF202MYqLrtOy1ftUk__POW/edit?usp=sharing&ouid=106672251267017181111&rtpof=true&sd=true) ### Dataset Access Full dataset for the language can be browsed and queried on our [app](https://life.unreal-tece.co.in). The results of the evaluation of different models will also be made available on the same link. If you would like to access the full dataset for your own use or would like to work with us in collecting more data for any language or variety, or collaborate with us in this initiative in any other way, please get in touch with us.

BoLI · Irula of the Nilgiri

View project ↗

Discover data.Connect the knowledge.

Discover data.
Connect the knowledge.