Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/LowResourceLanguages/champollion

Import of https://sourceforge.net/projects/champollion
https://github.com/LowResourceLanguages/champollion

Last synced: 29 days ago
JSON representation

Import of https://sourceforge.net/projects/champollion

Awesome Lists containing this project

README 

        
                     Champollion Tool Kit V1.2



About CTK

----------



Champollion Tool Kit (CTK) is a tool kit aiming to provide

ready-to-use parallel text sentence alignment tools for as many

language pairs as possible.



Built around the LDC champollion sentence aligner kernel, the tool kit

provides essential components required for accurate sentence

alignment, including sentence breakers, stemmers, pre-processing

scripts, dictionaries (if possible), post-processing scripts etc.



Currently, CTK includes tools to align English text with Arabic, Chinese,

 and Hindi translations. It can be easily expanded to other language pairs.



CTK welcomes contributions from other researchers.



CTK is written in perl.



Installation

------------



After unpack the CTK distribution, you need to set the enviorment

variable CTK to the directory where the package is unpacked, which is

this directory if you haven't done anything funny. And that's it.



To test the installation, try run the following command:



./test_installtion



It will tell you either the installation is good, or bad and in which

case minimum diagnosis will be given.



Please note, the first time you run champollion (or test_installation

which runs champollion internally), the program needs to build certain

databases, which can take up to five minutes.



Input and Output

----------------



The input files for both sides should be one segment (sentence) per

line.



The output (alignment file) looks like the following:



omitted <=> 1

omitted <=> 2

omitted <=> 3

1 <=> 4

2 <=> 5

3 <=> 6

4,5 <=> 7

6 <=> 8

7 <=> 9

8 <=> 10

9 <=> omitted



Every alignment is in the format of:



        language1 sentence ids <=> language2 sentence ids



where each language1/language2 sentence ids may contain up to four sentence ids

delimited by commas, it also can be "omitted" indicating no translation

was found. The sentence ids start at 1.



Languages

---------



CTK v1.2 supports three language pairs:



	English Chinese(GB)

	English Chinese(UTF8)

	English Arabic (UTF8)

	English Hindi (UTF8)



IMPORTANT: Because we don't have IPRs to distribute the dictionaries

we're using internally, the dictionaries included in this package

are rather small: English Chinese dictionary (about 5K headwords) 

and English Arabic dictionary (about 4K headwords). Our experiment 

shows that bigger dictionary usually leads better performance, which 

means that you may want to use your own dictionary, if it has better

coverage than the one we provide.



Command Line

------------



Command line to run English Chinese sentence aligner:



your_CTK_path/bin/champollion.EC_GB   



or 



your_CTK_path/bin/champollion.EC_utf8   



Command line to run English Arabic sentence aligner:



your_CTK_path/bin/champollion.EA   



Command line to run English Hindi sentence aligner:



your_CTK_path/bin/champollion.EH   



In addition, there is champollion.generic which can align unknown

language pairs. To run it:



your_CTK_path/bin/champollion.generic    



For languages not included in the package, it's strongly recommended

that you write a tokenizer following existing examples, and a stemmer

if possible. Even an imperfect stemmer will make a big difference.



Evaluation Corpus

-----------------



To facilitate the development of better sentence aligners, this

package also includes the manually aligned English Chinese data as an

evaluation corpus. The evaluation corpus is in 'eval' directory.



The data were selected from three sources: UN, Sinorama, and Hong Kong Hansards.



The Chinese files are:



198706005.c.txt         921008fc.txt            UN19990209_010.c.txt

200110006.c.txt         930422fc.txt

890621fc.txt            UN19930101_020.c.txt



The English files are:



198706005.e.txt         921008fe.txt            UN19990209_010.e.txt

200110006.e.txt         930422fe.txt

890621fe.txt            UN19930101_020.e.txt



The gold alignment files are:



198706005.gold.align            930422f.gold.align

200110006.gold.align            UN19930101_020.gold.align

890621f.gold.align              UN19990209_010.gold.align

921008f.gold.align



COPYRIGHT

---------



This software are protected by Common Public License, see LICENSE for

detail.



Contributions, Questions, Bug report, etc.

------------------------------------------



Please contact Xiaoyi Ma at [email protected].



Xiaoyi Ma 6/20/2011

Linguistic Data Consortium

[email protected]