{"id":15520837,"url":"https://github.com/proycon/colibrita","last_synced_at":"2025-07-21T09:33:35.686Z","repository":{"id":66967652,"uuid":"10638926","full_name":"proycon/colibrita","owner":"proycon","description":"Colibrita is a proof-of-concept translation assistance system, translating L1 fragments in an L2 context, using machine learning and statistical machine translation techniques","archived":false,"fork":false,"pushed_at":"2018-05-23T19:35:12.000Z","size":332,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-13T23:51:04.149Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proycon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":"codemeta.json","zenodo":null}},"created_at":"2013-06-12T08:23:21.000Z","updated_at":"2018-05-23T19:35:14.000Z","dependencies_parsed_at":"2023-05-16T01:00:32.135Z","dependency_job_id":null,"html_url":"https://github.com/proycon/colibrita","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/proycon/colibrita","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibrita","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibrita/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibrita/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibrita/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proycon","download_url":"https://codeload.github.com/proycon/colibrita/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcolibrita/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266276170,"owners_count":23903983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T10:29:38.753Z","updated_at":"2025-07-21T09:33:35.667Z","avatar_url":"https://github.com/proycon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Colibrita: Translation Assistance System\n============================================\n\nColibrita is a proof-of-concept translation assistance system that can\ntranslate L1 fragments in an L2 context. \n\nThe system is designed prior to a new task (presented at SemEval 2014) concerning the translation of L1\nfragments, i.e words or phrases, in an L2 context. This type of translation can\nbe applied in writing assistance systems for language learners in which users\nwrite in their target language, but are allowed to occasionally back off to\ntheir native L1 when they are uncertain of the proper word or expression in L2.\nThese L1 fragments are subsequently translated, along with the L2 context, into\nL2 fragments.\n\nColibrita was developed to test whether L2 context information aids in translation of L1 fragments. The results are accepted for publication in ACL 2014, in the paper: *Maarten van Gompel, Antal van den Bosch*. **Translation Assistance by Translation of L1 Fragments in an L2 Context**. Proceedings of ACL 2014 Conference (to appear still)\n\n\nInstallation\n===============\n\nColibrita is written in Python 3. It is a complex system involving quite a\nnumber of dependencies.\n\nFirst make sure you have a modern linux distribution with the necessary\nprerequisites: python3, python3-dev, python3-setuptools, python3-lxml, cython3, gcc,\ng++,  autoconf,  automake, autoconf-archive, libtool , libboost-dev, libboost-python\n\nIf you intend to build your own training models, then you will also require the\nfollowing two dependencies:\n\n * **Moses** - https://github.com/moses-smt/mosesdecoder\n * **GIZA++** - http://code.google.com/p/giza-pp/ \n\nOther unix systems including FreeBSD and Mac OS X will most likely work \ntoo, but especially for that latter considerable extra effort may be required\nin installing things. The instructions here have been tailored for\nDebian/Ubuntu-based Linux distributions.\n\nIn addition to the above dependencies, Colibrita depends on pynlpl, colibri-core, Timbl and python-timbl.\n\nInstall PyNLPl from the Python Package Index (or alternatively from\nhttps://github.com/proycon/pynlpl):\n\n    $ sudo easy_install3 pynlpl\n\nDownload colibri-core from https://github.com/proycon/colibri-core and install as follows:\n\n    $ bash bootstrap\n    $ ./configure \n    $ make\n    $ sudo make install\n    $ sudo python3 ./setup.py install\n\nInstall Timbl, it may be in your package manager if you use Debian/Ubuntu:\n\n    $ sudo apt-get install timbl\n\nOtherwise obtain it from http://ilk.uvt.nl/timbl and compile manually:\n\n    $ ./configure\n    $ make\n    $ make install\n\nInstall Python-Timbl from the Python Package Index (or alternatively from\nhttps://github.com/proycon/python-timbl):\n\n    $ sudo easy_install3 python-timbl\n\nThen install colibrita from https://github.com/proycon/colibrita:\n\n     $ sudo python3 ./setup.py install\n\n**Note:** If you want to reproduce the results of our ACL paper, then make sure to do\n``git checkout v0.2.1`` in the Colibrita repository prior to installation. Colibrita may have advanced\nsince then.\n\nLast, if you want to evaluate according to well-known MT metrics such as BLEU,\nMETEOR, NIST, TER, WER, and PER; you should download and unpack\nhttp://lst.science.ru.nl/~proycon/mtevalscripts.tar.gz\n\n\nUsage\n===========\n\nThe following tools are available:\n\n * ``colibrita`` - This is the main system, it is used for training and\n   testing.\n  \n * ``colibrita-evaluate`` - Tool for evaluation of system output. Point\n   --mtevaldir to the directory where you unpacked mtevalscripts.tar.gz if you\n   want common MT metrics in your report.\n\n * ``colibrita-setgen`` - Tool for generating training \u0026 test sets from\n   parallel corpus data, GIZA++ Word Alignments and a Moses Phrasetable\n    \n * ``colibrita-datastats`` - Reports some statistics on a dataset (train or test, XML)\n\n * ``colibrita-manualsetbuild`` - Small interactive console-based script for creating\n   datasets manually\n\n\nAll tools take ``-h`` for help on usage options.\n\nSet generation\n--------------\n\nBuilding a model starts with generating a training set from a parallel corpus.\nEnsure you have two plain-text files, one in the source language, one in the\ntarget language, with one sentence per line where the line numbers across the\ntwo files are indicative of sentences that are translations of eachother. In\nthis documentation we will use two files from our ACL 2014 experiments,\nobtainable from http://lst.science.ru.nl/~proycon/colibrita-acl2014-data.zip :\n\n    * europarl200k-train.nl.txt\n    * europarl200k-train.en.txt\n\nGiven this input data, you can use Colibrita's *setgen* tool: \n\n     $ colibrita-setgen --train --mosesdir=/path/to/mosesdecoder -S nl -T en \\\n     -s europarl200k-train.nl.txt -t europarl200k-train.en.txt --bindir=/usr/local/bin \\\n     -o europarl200k\n\nThis tool will invoke Moses (which will in turn invoke GIZA++) and the\nColibri-Core patternmodeller. It builds word alignments, a phrase-translation\ntable and pattern models, and eventually produces an XML file. This process may\ntake a very long time and demands conseridable memory. The output prefix\n ``-o`` will be used in many of the output files. The parameters\n``--joinedprobabilitythreshold`` and ``--divergencefrombestthreshold`` can be\nused to prevent weaker alignments and alternatives from making it into the set,\nand correspond to the parameters λ1 and λ2 in our ACL 2014 paper.\n\nA test set can be generated in the same fashion:\n\n     $ colibrita-setgen --test --mosesdir=/path/to/mosesdecoder -S nl -T en \\\n     -s europarl200k-test.nl.txt -t europarl200k-test.en.txt --bindir=/usr/local/bin \\\n     -o europarl200k\n\n\nTraining\n--------------\n\nThe next step is feature extraction and classifier training:\n\n    $ colibrita --train -f europarl200k.train.xml -l 1 -r 1 \\\n    -o exp-l1r1 --Tclones 4 --trainfortest europarl200k-test.xml\n\n\nThe output will consist of a whole bunch of classifiers (ibase files) in the\ndirectory specified with ``-o``.\n\nSome notes about this example:\n\n * ``-f`` specifies the training set, generated by ``colibrita-setgen`` in the\n   previous step.\n * ``-l 1`` sets a left context size of one\n * ``-r 1`` sets a right context size of one\n * ``-o`` specified a new output prefix, used in generated files and a\n   directory will be generated with this name containing all classifiers and\n   intermediate files\n * ``--Tclones 4`` runs Timbl on four cores\n * ``--trainfortest`` generates only those classifiers that will be used in\n   testing, saving time and resources. But this implies the model will have to\n   be retrained if other test data is offered, and can ever be used in a\n   live setting.\n\nTesting\n---------\n\nTesting follows a very similar syntax:\n\n    $ colibrita --test -f europarl200k.test.xml -l 1 -r 1 \\\n    -o exp-l1r1 -T train-europarl200k/model/phrase-table.gz\n\nThis will generate a file exp-l1r1-output.xml that contains the system output\n\nSome notes:\n\n * ``-T`` passes the original phrase table which will be used as a fallback option\n * ``-o`` the same output prefix used in the training step, is used as input as\n   well and assumes a directory by this name exists\n\nEvaluation\n--------------\n\nSystem output can subsequently be evaluated against the test set using\n``colibrita-evaluate``:\n\n    $ colibrita-evaluate --mtevaldir /path/to/mtevalscripts \\\n    --ref europarl200k.test.xml --out exp-l1r1-output.xml\n\nA summary of all Scores will be written in ``exp-l1r1-output.summary.score`` .\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fcolibrita","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproycon%2Fcolibrita","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fcolibrita/lists"}