https://github.com/mohsenim/persianp
A Processing Toolbox for Persian Texts
https://github.com/mohsenim/persianp
chunker lemmatizer nlp persian postagger text-analysis tokenizer
Last synced: about 1 year ago
JSON representation
A Processing Toolbox for Persian Texts
- Host: GitHub
- URL: https://github.com/mohsenim/persianp
- Owner: mohsenim
- Created: 2018-12-16T19:29:48.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2018-12-19T13:26:48.000Z (over 7 years ago)
- Last Synced: 2025-02-06T14:25:27.114Z (over 1 year ago)
- Topics: chunker, lemmatizer, nlp, persian, postagger, text-analysis, tokenizer
- Language: Java
- Homepage:
- Size: 61.3 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Persianp Processing Toolbox
Persianp is a text processing tool developed in Java to preprocessing Persian texts. The toolbox accomplishes following tasks:
* Character-level normalization
* Tokenization
* Lemmatization
* POS tagging
* Stopword detection
* Noun phrase chunking
### Using Persianp from the command line
Be sure folder `res` is next to the `jar` file.
```bash
$ java -cp persianp-toolbox-1.0.jar com.persianp.nlp.process.Process -input inputfile.txt -output outputfile.txt -task (tokenize|tag|lemmatize|taglemmatize) [-nostopword] [-prop propertyFile.properties]
```
At the moment NP chunking is not supported from the comand line.
### Using the Persianp API
Add the API to libraries of your program. The following example shows how to use the toolbox.
```
public class TestPersianp {
public static void main(String[] args) {
TestPersianp testPersianp = new TestPersianp();
testPersianp.process();
}
private void process() {
try {
Properties properties = new Properties();
properties.load(this.getClass().getClassLoader().getResourceAsStream("persianp.properties"));
Process process = new Process(properties);
InputStream in = this.getClass().getClassLoader().getResourceAsStream("testText.txt");
BufferedReader br = new BufferedReader(new InputStreamReader(in, "UTF-8"));
String line;
while ((line = br.readLine()) != null) {
process.process(line);
System.out.println(process.getText());
// process.getTokens();
// process.getTokensText();
// process.getTags();
// process.getChunkTag();
// process.getLemmas();
// process.getNonStopwordTokens();
int sentenceSize = process.getSentencesSize();
for (int j = 0; j < sentenceSize; ++j) {
// List tokensText = process.getTokensTextInSentence(j);
// List tags = process.getTagsInSentence(j);
// List lemmas = process.getLemmasInSentence(j);
List tokens = process.getTokensInSentence(j);
for (int k = 0; k < tokens.size(); ++k) {
System.out.println(tokens.get(k).getText() + "\t\t\t" + tokens.get(k).getLemma() + "\t\t\t" + tokens.get(k).getTag());
}
}
}
in.close();
br.close();
} catch (Exception e){
e.printStackTrace();
}
}
}
```
### More Information / Citing This Toolbox
Please cite the paper below if you use the Persianp toolbox in your research. It also provides more information about the toolbox.
> Mahdi Mohseni, Javad Ghofrani, and Heshaam Faili.
> "Persianp: A Persian Text Processing Toolbox".
> International Conference on Intelligent Text Processing and Computational Linguistics
CICLing 2016: Computational Linguistics and Intelligent Text Processing, pp 75-87.
Bibtex citation:
```
@InProceedings{Persianp2016,
author="Mohseni, Mahdi
and Ghofrani, Javad
and Faili, Heshaam",
title="Persianp: A Persian Text Processing Toolbox",
booktitle="Computational Linguistics and Intelligent Text Processing",
year="2018",
publisher="Springer International Publishing",
pages="75--87",
isbn="978-3-319-75477-2"
}
```