{"id":15574870,"url":"https://github.com/loomchild/maligna","last_synced_at":"2025-04-24T02:09:22.696Z","repository":{"id":52037892,"uuid":"41963025","full_name":"loomchild/maligna","owner":"loomchild","description":"Bilingual sengence aligner","archived":false,"fork":false,"pushed_at":"2023-09-30T14:48:39.000Z","size":1019,"stargazers_count":27,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-24T02:09:14.924Z","etag":null,"topics":["nlp","text-alignment","translation"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/loomchild.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2015-09-05T14:02:42.000Z","updated_at":"2023-07-04T01:06:57.000Z","dependencies_parsed_at":"2023-12-17T23:07:21.944Z","dependency_job_id":null,"html_url":"https://github.com/loomchild/maligna","commit_stats":{"total_commits":32,"total_committers":1,"mean_commits":32.0,"dds":0.0,"last_synced_commit":"dc33ef832a5286f7905cf1f70831d911d1fe8b3a"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomchild%2Fmaligna","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomchild%2Fmaligna/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomchild%2Fmaligna/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomchild%2Fmaligna/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/loomchild","download_url":"https://codeload.github.com/loomchild/maligna/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250546082,"owners_count":21448260,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nlp","text-alignment","translation"],"created_at":"2024-10-02T18:21:14.560Z","updated_at":"2025-04-24T02:09:22.675Z","avatar_url":"https://github.com/loomchild.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mALIGNa\n\n## Introduction\n\nmALIGNa is a program for aligning documents on the sentence level. \nIt contains implementations of a few groups of alignment algorithms -\nalgorithms based exclusively on the sentence length (Gale and Church, Brown), \nalgorithms based on the connections between words (Moore's algorithm), \nas well as any variation and intersection of these algorithms. \nThe aim of alignment is to obtain a bilingual corpus. \nIt can be used for creating translation memories, translation by analogy, \nmodelling probabilistic dictionaries and other applications.\nMore information about maligna can be found in an article which I co-authored \n(see Resources section).\n\n## Library\n\nmALIGNa library is available on [Maven central](http://search.maven.org/#search|ga|1|net.loomchild.maligna). \nSee [pom.xml](maligna/pom.xml) for details.\n\n## Requirements\n\nJVM 1.6 is required to run the program. \nTo build the program both JDK 1.6 and Maven are required. \nTheoretically, the program is platform independent (like Java), \nbut it was tested only on Linux and Windows.\n\n## Running\n\nAligning documents consists of several stages, which may be performed \nin different ways. Therefore, the text interface of the program is divided \ninto several independent commands. A common feature of the commands is \nthat they read input text from standard input and write the results \nto standard output, always using the native format .al. \nTherefore these programs act as filters and may be combined in a pipe using |. \nBelow is a brief description of each command, you can get more information \nabout exact parameters of individual commands using the option --help. \nAt the end of this document, there is a complete example of text \nalignment using several commands connected in a pipeline.\n\n### The 'parse' command\n\nThe parse command is used to convert an external format \nto the native format .al. It also allows you to combine several documents \ninto one, by giving a list of files as arguments. \nIt accepts input files as arguments and writes the result to standard output. \nThis command can only occur at the beginning of the pipeline.\n\n### The 'format' command\n\nThe 'format' command is used to convert the native format .al \nto an external format. It reads data from standard input and writes the result \nto the files given as arguments. This command can only occur at the end \nof the pipeline.\n\n### The 'align' command\n\nAlign command. Segments of each of the input mappings are aligned independently,\nthanks to which alignment can be performed at different levels of accuracy \n(document, paragraph, sentence), by performing consecutive alignment operations,\nand then dividing the results into smaller and smaller segments. \nFilter; may be used at any point of the pipeline.\n\n### The 'modify' command\n\nThis command performs modifications on every mapping, replacing source and \ntarget lists of segments with other lists of segments. \nBoth the amount of the segments and their contents may be changed \n(e.g. merge segments in one, split segment into more segments or \nremove unnecessary whitespace in each segment). \nFilter; may be used at any point of the pipeline.\n\n### The 'select' command\n\nBasing on certain criteria, chooses from the input list of mappings only \nsome mappings and writes them to standard output. \nEg. using this command you can choose only the most probable mappings \nor only 1 - 1 mappings. Filter; may be used at any point of the pipeline.\n\n### 3.6 The 'compare' command\n\nThis command is used for comparing two files of alignments with each other, \nprovided as arguments, returning the degree of their similarity \n(precision, recall) and the differences occurring. \nUsed for test purposes outside the pipeline.\n\n### The 'model' command\n\nThis command manipulates translation, language and length models. \nCurrently not fully implemented.\nUsed outside the pipeline.\n\n### The 'macro' command\n\nExecutes a set of predefined filtering commands, like doing a Moore alignment.\nCreated to simplify complex operations and improve the performance.\nFilter; may be used at any point of the pipeline.\n\n\n### The 'test' command\n\nRuns the program's automatic tests.\n\n### Examples\n\nBelow I have given examples of pipelines of commands that that should be used \nto align two documents in .txt files and write the results as two .txt files. \nThe output documents will contain the same number of sentences, \none per line, and sentences of corresponding numbers should be mutual \ntranslations. The commands given below will do everything required, \nhowever it is worth remembering that sometimes it's better to preserve \nthe intermediate results of the operations of each command in temporary files \ninstead of redirecting them directly to the next command for debugging purposes. \nThis example should be executed in the main directory of the project. \nReady to run example scripts can be found in the examples/scripts directory.\n\n    bin/maligna parse -c txt examples/txt/poznan-pl.txt examples/txt/poznan-de.txt | \\\n    bin/maligna modify -c split-sentence | \\\n    bin/maligna modify -c trim | \\\n    bin/maligna align -c viterbi -a poisson -n word -s iterative-band | \\\n    bin/maligna select -c one-to-one | \\\n    bin/maligna format -c txt poznan-pl-align.txt poznan-de-align.txt\n\nAnother interesting case is alignment using Moore's algorithm, \nwhich requires a properly aligned corpus to build a translation model. \nTo do this you must perform several groups of commands.\n\nSplit a text into sentences and clean them up:\n\n    bin/maligna parse -c txt examples/txt/poznan-pl.txt examples/txt/poznan-de.txt | \\\n    bin/maligna modify -c split-sentence | \\\n    bin/maligna modify -c trim \u003e \\\n    poznan-split.al\n\nAlign using sentence length-based algorithm (Brown, Gale and Church) \nand select most probable alignments:\n\n    cat poznan-split.al | \\\n    bin/maligna align -c viterbi -a poisson -n word -s iterative-band | \\\n    bin/maligna select -c one-to-one | \\\n    bin/maligna select -c fraction -f 0.85 \u003e \\\n    poznan-align-length.al\n\nFinally align using Moore's algorithm:\n\n    cat poznan-split.al | \\\n    bin/maligna align -c viterbi -a translation -n word -s iterative-band -t poznan-align-length.al \u003e \\\n    poznan-align.al\n\n## Formats\n\nMost commands expect input and output in the native format .al. \nTo use a different format, you must parse the input using the parse command. \nTo get the result in another format, you need to format it using \nthe format command.\n\n### The .al format\n\nThis is the native format for alignment files. \nContains all necessary information for each mapping: \nlists of source and target segments and mapping score (-log(probability)).\nIf mapping score is equal to \"-INF\" then all commands ignore this mapping,\nno operation is performed on it. This feature can be used to mark manual, \nhuman-aligned fragments to be preserved in the result.  \n\n### The .tmx format\n\nThe standard format for translation memories, supported by many tools. \nBoth an input and output format. For the full specification see\n\u003ca href=\"http://www.lisa.org/standards/tmx/tmx.html\"\u003ehttp://www.lisa.org/standards/tmx/tmx.html\u003c/a\u003e.\n\n### The .txt format\n\nPlain text, in the form of two files in two languages. \nOn input the whole file is treated as one segment. \nOn output, successive lines of files correspond to mappings and are \nmutual translations (which implies that numbers of lines are equal).\n\n### The presentation format\n\nAn output format which presents an alignment in a human readable manner.\n\n## Algorithms\n\nIn the program a few alignment algorithms were implemented, and thanks to the \nflexibility of the code it is easy to add new algorithms, \nmodify the existing ones and join their results together.\n\n### Gale and Church algorithm\n\nThis is a fast bilingual text alignment algorithm. The algorithm counts the \nprobability of each possible mapping, which depends on ratio of lengths of \nsentences in each language. Next it finds alignment for the whole text with \nmaximum probability (calculated as a product of individual mapping probabilities). \nDetails can be found in the Gale and Church article, referenced in resources.\n\n### Moore's algorithm\n\nThis is a modern algorithm based not only on the length of sentences but also on \ntheir contents. \nThe first phase of operation of this algorithm is alignment based on length \n(for example using Gale and Church algorithm described above). \nNext, from this alignment only 1 - 1 mappings alignments are selected, \nand from them only the most probable ones (e.g. 80% probability) are selected. \nIn this manner a relatively well aligned corpus is yielded. \nLater a translation model (IBM Model 1) and unigram language models are built \nbased on this corpus. \nIn the final alignment, probability of the translation of the source sentence to \nthe target sentence is taken into account (calculated on the basis of the \ntranslation model and the language models). \nDetails can be found in the Moore article, referenced in the resources.\n\n## Resources\n\n1. A new tool for the bilingual text aligning at the sentence level, \n   Krzysztof Jassem, Jarek Lipski, Proceedings of Intelligent Information Systems Conference 2008, Zakopane, Poland,\n   http://iis.ipipan.waw.pl/2008/proceedings/iis08-27.pdf\n2. A Program for Aligning Sentences in Bilingual Corpora, William A. Gale, Kenneth Ward Church\n3. Fast and Accurate Sentence Alignment of Bilingual Corpora, Robert C. Moore\n\n## Authors\n\n* Jarek Lipski - creation of the project, design and programming\n* Jimmy O'Regan - translation of readme file to English\n\n## Thanks\n\nI wrote this program during my studies in Machine Translation, \nand later I expanded it as practical part of a master's thesis.\nCurrently it is being developed as an open-source project and used by\nvarious other projects. Happy aligning!\n\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;-- Jarek Lipski\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floomchild%2Fmaligna","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Floomchild%2Fmaligna","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floomchild%2Fmaligna/lists"}