https://github.com/KurdishBLARK/InterdialectCorpus

A parallel corpus of Sorani, Kurmanji and English
https://github.com/KurdishBLARK/InterdialectCorpus

corpus kurdish kurdish-language-processing machine-translation natural-language-processing parallel-corpus

Last synced: 28 days ago
JSON representation

A parallel corpus of Sorani, Kurmanji and English

• Host: GitHub
• URL: https://github.com/KurdishBLARK/InterdialectCorpus
• Owner: KurdishBLARK
• License: other
• Created: 2020-08-09T00:30:35.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2020-10-06T01:17:58.000Z (about 4 years ago)
• Last Synced: 2024-01-28T23:08:49.251Z (11 months ago)
• Topics: corpus, kurdish, kurdish-language-processing, machine-translation, natural-language-processing, parallel-corpus
• Homepage: https://kurdishblark.github.io/
• Size: 21.7 MB
• Stars: 5
• Watchers: 4
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-kurdish - Ahmadi et al's corpus - Kurmanji-Sorani aligned texts (Development / Resources)

README

        
# Kurdish Parallel Corpus

## A parallel corpus of Kurdish (Sorani and Kurmanji) and English

This repository contains a parallel corpus of Sorani (`ckb`) and Kurmanji (`kmr`) dialects of Kurdish along with English (`eng`). The development of the corpus is described in [our paper](https://arxiv.org/abs/2010.01554). Our approach is consisted of retrieving potentially-alignable news articles from multilingual websites, semi-automatically align sentences across dialects and languages based on lexical similarity and transliteration of scripts and, manually annotate correct translation pairs. For further information, please see the **[annotation guidelines](https://github.com/KurdishBLARK/InterdialectCorpus/tree/master/X_Guidelines)**.

Our parallel corpus contains three manually-aligned corpus in Sorani-Kurmanji, Sorani-English and Kurmanji-English in various formats, namely [Translation Memory eXchange](https://en.wikipedia.org/wiki/Translation_Memory_eXchange) file format (`.tmx`), parallel annotated text useful for [ParaConc](https://paraconc.com/) and raw parallel texts (`.txt`). In the latter, each line corresponds to the same line in the other aligned file.

This corpus contains **12,327** translation pairs in the two major dialects of Kurdish, Sorani and Kurmanji. We also provide **1,797** and **650** translation pairs in English-Kurmanji and English-Sorani. 

## Download

You can clone this repository or download individual directories as follows:

- **[Sorani-English](https://github.com/KurdishBLARK/InterdialectCorpus/tree/master/CKB-ENG)**

- **[Kurmanji-English](https://github.com/KurdishBLARK/InterdialectCorpus/tree/master/KMR-ENG)**

- **[Sorani-Kurmanji](https://github.com/KurdishBLARK/InterdialectCorpus/tree/master/CKB-KMR)**

In addition, the statistical models reported as the baseline in our paper are provided in the [Moses-results](https://github.com/KurdishBLARK/InterdialectCorpus/tree/master/Moses-results) directory.

## Cite this paper

If you use any part of the data, please consider citing **[this paper](https://arxiv.org/abs/2010.01554)** as follows:

	@misc{ahmadi2020leveraging,

	      title={Leveraging Multilingual News Websites for Building a Kurdish Parallel Corpus}, 

	      author={Sina Ahmadi and Hossein Hassani and Daban Q. Jaff},

	      year={2020},

	      eprint={2010.01554},

	      archivePrefix={arXiv},

	      primaryClass={cs.CL}

	}

The link to the published version of the paper will be also provided later.

## License

This corpus is available under the [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://github.com/KurdishBLARK/InterdialectCorpus/blob/master/LICENSE) license. For further information for commercial use, please get in touch with the [contributors](https://github.com/KurdishBLARK/InterdialectCorpus/graphs/contributors).