{"id":15520814,"url":"https://github.com/proycon/semeval2014task5","last_synced_at":"2025-04-05T04:26:13.746Z","repository":{"id":15789349,"uuid":"18528579","full_name":"proycon/semeval2014task5","owner":"proycon","description":"This is the official repository for SemEval 2014 Task 5: L2 Translation Assistant. It contains the gold standard learner corpus, evaluation results and the Python program library needed for the task. It does not contain a full translation assistance system.","archived":false,"fork":false,"pushed_at":"2015-04-16T09:28:21.000Z","size":8938,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-04T11:07:14.299Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proycon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-07T18:39:35.000Z","updated_at":"2020-01-14T03:48:53.000Z","dependencies_parsed_at":"2022-07-16T07:30:29.408Z","dependency_job_id":null,"html_url":"https://github.com/proycon/semeval2014task5","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fsemeval2014task5","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fsemeval2014task5/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fsemeval2014task5/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fsemeval2014task5/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proycon","download_url":"https://codeload.github.com/proycon/semeval2014task5/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247289261,"owners_count":20914460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T10:29:27.232Z","updated_at":"2025-04-05T04:26:13.715Z","avatar_url":"https://github.com/proycon.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"SemEval 2014 - Task 5 - L2 Writing Assistant\n============================================\n    \n*Maarten van Gompel, Iris Hendrickx, Antal van den Bosch, Radboud University Nijmegen*; \n*Els Lefever, Véronique Hoste, Ghent University*\n\nhttp://alt.qcri.org/semeval2014/task5/\n\n\nIntroduction\n----------------------\n\nThis SemEval task was organised for SemEval 2014 and offer a new cross-lingual\nand application-oriented task for SemEval that finds itself in the area where\ntechniques from Word Sense Disambiguation and Machine Translation meet.\n\nThe task concerns the translation of L1 fragments, i.e words or phrases, in an\nL2 context. This type of translation can be applied in writing assistance\nsystems for language learners in which users write in their target language,\nbut are allowed to occasionally back off to their native L1 when they are\nuncertain of the proper word or expression in L2. These L1 fragments are\nsubsequently translated, along with the L2 context, into L2 fragments.\n\nThus, participants are asked to build a translation/writing assistance system\nthat translates specifically marked L1 fragments, in an L2 context, to their\nproper L2 translation.\n\nThe task find itself on the boundary of Cross-Lingual Word Sense Disambiguation\nand Machine Translation. Full-on machine translation typically concerns the\ntranslation of whole sentences or texts from L1 to L2. This task, in contrast,\nfocuses on smaller fragments, side-tracking the problem of full word\nreordering.\n\nIn this task we focus on the following language combinations of L1 and L2\npairs: English-German, English-Spanish, French-English, and Dutch-English. Task\nparticipants may participate for all language pairs or any subset thereof.\n\nTask Description\n----------------------\n\nIn this task we ask the participants to build a translation assistance system\nrather than a full machine translation system. The general form of such a\ntranslation assistance system allows a translator or L2-language student to\nwrite in L2, but allowing him or her to back off to L1 where he is uncertain of\nthe correct lexical or grammatical form in L2. The L1 expression, a word or\nphrase, is translated by the system to L2, given the L2 context already\npresent, including right-side context if available. The aim here, as in all\ntranslation, is to carry the semantics of the L1 fragment over to L2 and find\nthe most suitable L2 expression given the already present L2 context.\n\nThe task essentially addresses a core problem of WSD, with cross-lingual\ncontext, and a sub-problem of Phrase-based Machine Translation; that of finding\nthe most suitable translation of a word or phrase.  In MT this would be\nmodelled by the translation model. Our task does not address the full\ncomplexity of sentential translation, thus evading problems associated with\nreordering and syntax. Instead it emphasizes the local semantic aspect of\nphrasal or word translation in context. The user group we have in mind is that\nof intermediate and advanced language learners, whom you generally want to\nencourage to use their target language as much as possible, but may often feel\nthe need to fall back to their native language.\n\nCurrently, language learners are forced to fall back to a bilingual dictionary\nwhen in doubt. These do not take the L2 context into account and are generally\nmore constrained to single words or short expressions. The proposed\napplication automatically generates context-dependent translation suggestions\nas writing progresses. The task tests how effective participating systems\naccomplish this.\n\nThe following example sentence pairs illustrate the idea:\n\n * Input (L1=English,L2=Spanish): “Hoy vamos a the **swimming pool**”.\n * Desired output: “Hoy vamos a **la piscina**”\n * Input (L1=English, L2=German): “Das wetter ist wirklich **abominable**”.\n * Desired output: “Das wetter ist wirklich **ekelhaft**”\n * Input (L1=French,L2=English): “I return home **parce'qu je suis fatigué**”\n * Desired output: “I return home **because I am tired**”.\n * Input (L1=Dutch, L2=English): “Workers are facing a massive **aanval op** their employment and social right.”\n * Desired output: “Workers are facing a massive **attack on** their employment and social rights”\n\nThe L2 writing assistant task can be related to two tasks that were offered in\nprevious years of SemEval: Lexical Substitution (Mihalcea et al, 2010) and\nCross-lingual Word Sense Disambiguation (Lefever and Hoste, 2010, 2013). When\ncomparing this task to the Cross-Lingual Word-Sense Disambiguation task,\nnotable differences are the fact that this task concerns not just words, but\nalso phrases. Another essential difference is the nature of the context; the\ncontext is in L2 instead of L1.\n\nEvaluation\n---------------\n\nSeveral metrics are available for automatic evaluation. First, we measure the\nabsolute accuracy $$a = c/n$$, where c is the number of fragment translations from\nthe system output that precisely match the corresponding fragments in the\nreference translation, and n is the total number of translatable fragments,\nincluding those for which no translation was found. We also introduce a\nword-based accuracy, which unlike the absolute accuracy, still gives some\ncredits to mismatches that show partial overlap with the reference\ntranslation. The system with the highest word-based accuracy wins the\ncompetition.\n\nA recall metric simply measures the number of fragments for which the system\ngenerated a translation, as a proportion of the total number of fragments. As\nno selection is made in L1 words or phrases that may appear in a an L2 context,\nand due to the way evaluation is conducted, it is important that participating\nsystems produce output for as many  possible words and phrases as possible, and\nthus achieve a high recall.\n\nIn addition to these task-specific metrics, standard MT metrics such as BLEU,\nNIST, METEOR and error rates such as WER, PER and TER, are included in the\nevaluation script as well. Scores such as BLEU will generally be high (\u003e 0.95),\nas a large portion of the sentence is already translated and only a specific\nfragment remains to be evaluated. Nevertheless, these generic metrics are\nproven to follow the same trend as the more task-specific evaluation metrics.\n\nIt regularly occurs that multiple translations are possible. In the creation of\nthe test set will take this into account by explicitly encoding valid\nalternatives. A match with an alternative counts as a valid match. Likewise, a\ntranslation assistance system may output multiple alternatives as well. We\ntherefore allow two different types of runs, following the example of the\nCross-Lingual Lexical Substitution and Cross-Lingual Word Sense Disambiguation\ntasks:\n\n * **Best** - The system must output its best guess;\n * **Out of Five** - The system may output up to five alternatives.\n\nUp to three runs may be submitted per language-pair and evaluation type\n(totalling 24 runs in total if you participate for all language pairs and all\nevaluation types)\n\nAn evaluation script that implements all these measures will be made available\nto the participants. This same script will be used to compute the final\nevaluation of this task.\n\nData and Tools\n------------------------\n\n We provide material for the following L1 and L2 pairs:\n\n * English-German\n * English-Spanish\n * French-English\n * Dutch-English\n\nBoth trial and test data will be offered in a clear and simple XML format. The\ntest data will be delivered in tokenised format. This tokenisation is done\nusing ucto. System output is expected to adhere to this same XML format so it\ncan be automatically evaluated. Output should not be detokenised, it should\nhowever respect case as evaluation will be case-sensitive. We do not provide\nany training data for this task. Participants are fee to use any suitable\ntraining material such as parallel corpora, wordnets or bilingual lexica.\n\nParticipants are encouraged to participate in as many of the four language\npairs as possible, but may also choose any subset.\n\nThe task organizers have compiled a test data set for the selected language pairs of\nalmost 500 sentences each. In the selection of test data we aim for realism, by\nselecting words and phrases that may prove challenging for language learners.\nTo achieve this, we gather language learning exercises with gaps and\ncloze-tests, as well as learner corpora with annotated errors to act as the\nbasis for our test set. When L2 sentences with such marked error fragments are\ngathered, or gaps specifically designed to test a specific aspect of language\nability, we manually translate these fragments into L1, effectively forming a\nsentence pair for the test set. Note that the test sentences will not contain\nother L2 learner errors, we only use the errors of the L2 learners to get more\nnatural places to insert the L1 phrases. \n\nWe also provide trial data for the selected language pairs consisting of 500\nsentences as well. This trial data is semi-automatically generated using a\nparallel corpus, namely the Europarl corpus (Koehn, 2005). We performed a\nmanual selection to get sentences that contain translations of appropriate\nwords or phrases that mimick the L2 writing assistant task as naturally as\npossible. It has to be noted that this trial set is less optimised for realism\nthan the test data. Nevertheless, it suffices to measure relative system\nperformance, and it is a sufficiently large set.\n\nThe trial data, test data and tools are included in this github repository.\nInstallation of the tools will be explained one of the next chapters.\n \nLearner Corpus\n=========================================\n\nWe have compiled a learner corpus for each of the four\nlanguage pairs. It is contained in this repository:\n\nSee http://github.com/proycon/semeval2014task5/corpus/ \n\nResults\n=========================================\n\nSix teams from all over the world participated in this task with their\ntranslation assistance system. The system output of all participants, as well\nas the evaluation is contained in this repository. Participants output and the\nevaluation output is found in the ``evaluation/`` directory.\n\n * Evaluation results (static):\n   [evaluation/semeval_results.html](http://htmlpreview.github.io/?https://github.com/proycon/semeval2014task5/blob/master/evaluation/semeval-results.html)\n * Evaluation results (IPython 3 Notebook): [evaluation/semeval_results.ipynb](http://raw.github.com/proycon/semeval2014task5/raw/master/evaluation/semeval_results.ipynb)\n\n\nThe evaluation has been implemented in an IPython 3 Notebook, you can run it to\nreproduce the evaluation. You may also add additional system output to the\n``evaluation/`` directory, it will be automatically included in the evaluation\nso you can measure against the SemEval participants. To run the IPython\nNotebook:\n\n * Make sure you have Python 3,  IPython3, numpy, matplotlib,  and libsemeval2014task5 installed (the latter is explained in the next chapter)\n * Clone this git repository\n * ipython3 notebook evaluation/semeval_results.ipynb\n * Point your browser to http://localhost:8080\n * Adapt the DIR variable in the notebook to reflect your own installation\n\n\nTools \u0026 Python Library - Version 2.0\n=========================================\n\n\nInstallation\n--------------\n\nSystem-wide installation:\n\n    $ sudo python ./setup.py install\n\nPython usually defaults to Python 2, but you may also use Python 3, this is\nrequired if you want to use the IPython Notebook with the results:\n\n    $ sudo python3 ./setup.py install\n\nIf you obtain an import error then make sure the package ``python-setuptools`` is\ninstalled in your distribution. You also need the package ``python-lxml``. \n\nIf you have no sudo-rights, then you can install the library locally\nusing the ``--prefix`` switch, make sure the target prefix is in your ``$PYTHONPATH``.\n\nTools\n--------------\n\nThe following tools are included:\n\n\n * ``semeval2014task5-setview``  - This tool simply prints a set to standard output in a terminal, it takes one XML file as argument\n\n * ``semeval2014task5-evaluate`` - This is the evaluation script for evaluating your system's output against the trial/test set.\n\n\nEvaluation\n--------------\n\nThe evaluation script should be run as follows:\n\n    $ semeval2014task5-evaluate --ref trialdata.xml --out systemout.xml\n\nIf you want additional Machine Translation evaluation metrics (BLEU, METEOR,\nNIST,TER, WER, PER), you need to download the MT Evaluation Scripts from \nhttp://lst-science.ru.nl/~proycon/mtevalscripts.tar.gz and explicitly pass \nthe path to where you decompressed those using ``--mtevaldir=/path/to/mtevalscripts`` .\n\nFor out-of-five mode you need to add the ``-a`` flag to take into account alternatives.\n\nFor case-insensitive scoring, add the ``-i`` flag. \n\nFormat Library\n---------------\n\nThe semeval2014task5.format module enables you to easily read and write the XML\nformat for this task.  Import the format library in your Python source as\nfollows:\n\n    import libsemeval2014task5 format as format\n\nAn example system that shows the functionality of this library has been\nimplemented in example.py, which has been heavily commented. Please consult\nthis to learn how the library works. It implements a full system that\n'translates' fragments to upper-case instead of doing actual translation from\nL1 to L2.\n\n\nXML Format Specification\n==========================================================================\n\nThe trial data and test data for this task are delivered in a straightforward\nXML format using the UTF-8 character encoding.  The output of your system has\nto adhere to this format as well, facilitating automatic evaluation.\n\nConsider the following example set with English as L1 and Spanish as L2,\ncontaining only two sentence pairs:\n\n\n    \u003csentencepairs L1=\"en\" L2=\"es\"\u003e\n        \u003cs id=\"1\"\u003e\n            \u003cinput\u003eQuiero \u003cf id=\"1\"\u003eknow whether\u003c/f\u003e \n            se puede hacer este tipo de objeción a lo que sólo es un \n            informe , no una propuesta legislativa , y si es algo que \n            puedo plantear el jueves .\u003c/input\u003e\n            \u003cref\u003eQuiero \u003cf id=\"1\"\u003esaber si\u003c/f\u003e  se\n            puede hacer este tipo de objeción a lo que sólo es un informe \n            , no una propuesta legislativa , y si es algo que puedo \n            plantear el jueves .\u003c/ref\u003e\n        \u003c/s\u003e\n        \u003cs id=\"2\"\u003e\n            \u003cinput\u003eEste reacondicionamiento , que \u003cf id=\"1\"\u003elasted\u003c/f\u003e  varios\n            meses , cortó esta importante vía de transporte entre el norte y\n            el sur de Europa .\u003c/input\u003e\n            \u003cref\u003eEste reacondicionamiento , que \u003cf id=\"1\"\u003eduró\u003c/f\u003e  varios \n            meses , cortó esta importante vía de transporte entre\n            el norte y el sur de Europa .\u003c/ref\u003e\n        \u003c/s\u003e\n    \u003c/sentencepairs\u003e\n\n\nThe format's root tag is ``\u003csentencepairs\u003e``, within it are numerous sentence pairs\nin ``\u003cs\u003e`` blocks. Each sentence pair consists of an ``\u003cinput\u003e`` sentence in L2 with a\nmarked fragment in L1 and a ``\u003cref\u003e`` (reference) sentence in L2, in which the same\nfragment is marked, this time in L2. The fragment is marked using the ``\u003cf\u003e``\nelement. Both the sentence as well as the fragment carry an ID, the fragment's\nID will always be one as we do only one fragment per sentence at a time.\n\nYour output data has to adhere to this format, but instead of ``\u003cref\u003e`` you have to\nuse ``\u003coutput\u003e``, an example:\n\n    \u003csentencepairs L1=\"en\" L2=\"es\"\u003e\n    \u003cs id=\"1\"\u003e\n        \u003cinput\u003eQuiero \u003cf id=\"1\"\u003eknow whether\u003c/f\u003e \n        se puede hacer este tipo de objeción a lo que sólo es un \n        informe , no una propuesta legislativa , y si es algo que \n        puedo plantear el jueves .\u003c/input\u003e\n        \u003coutput\u003eQuiero \u003cf id=\"1\"\u003esaber si\u003c/f\u003e  se\n        puede hacer este tipo de objeción a lo que sólo es un informe \n        , no una propuesta legislativa , y si es algo que puedo \n        plantear el jueves .\u003c/output\u003e\n    \u003c/s\u003e\n    \u003cs id=\"2\"\u003e\n        \u003cinput\u003eEste reacondicionamiento , que \u003cf id=\"1\"\u003elasted\u003c/f\u003e  varios\n        meses , cortó esta importante vía de transporte entre el norte y\n        el sur de Europa .\u003c/input\u003e\n        \u003coutput\u003eEste reacondicionamiento , que \u003cf id=\"1\"\u003eduró\u003c/f\u003e  varios \n        meses , cortó esta importante vía de transporte entre\n        el norte y el sur de Europa .\u003c/output\u003e\n    \u003c/s\u003e\n    \u003c/sentencepairs\u003e\n\nYou're not obliged to repeat the input as done here, mere output elements would be sufficient.\n\nThe out-of-five evaluation method allows you to output alternatives, this is\ndone as follows; we show an excerpt of a fragment with alternatives:\n\n    \u003cf id=\"1\"\u003eoption1\u003calt\u003eoption2\u003c/alt\u003e\u003calt\u003eoption3\u003c/alt\u003e\u003calt\u003eoption4\u003c/alt\u003e\u003calt\u003eoption5\u003c/alt\u003e\u003c/f\u003e \n\nAlways ensure that your system output is valid XML, you can test this with an XML tool such as\n``xmllint``. \n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fsemeval2014task5","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproycon%2Fsemeval2014task5","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fsemeval2014task5/lists"}