{"id":23080589,"url":"https://github.com/chen0040/java-data-text","last_synced_at":"2025-06-28T02:38:30.066Z","repository":{"id":57719649,"uuid":"91335500","full_name":"chen0040/java-data-text","owner":"chen0040","description":"Package provides java implementation of various text preprocessing methods such as tokenizers, vocabulary, text filter, stemmer, and so on","archived":false,"fork":false,"pushed_at":"2017-05-18T04:27:18.000Z","size":1204,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-12T00:44:26.133Z","etag":null,"topics":["porter-stemmer-algorithm","stop-words","word-filter"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chen0040.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-15T12:31:20.000Z","updated_at":"2024-09-09T19:23:34.000Z","dependencies_parsed_at":"2022-09-02T11:01:49.050Z","dependency_job_id":null,"html_url":"https://github.com/chen0040/java-data-text","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chen0040/java-data-text","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-data-text","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-data-text/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-data-text/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-data-text/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chen0040","download_url":"https://codeload.github.com/chen0040/java-data-text/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fjava-data-text/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262364878,"owners_count":23299661,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["porter-stemmer-algorithm","stop-words","word-filter"],"created_at":"2024-12-16T13:15:13.042Z","updated_at":"2025-06-28T02:38:30.043Z","avatar_url":"https://github.com/chen0040.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# java-data-text\n\nPackage provides java implementation of various text preprocessing methods such as tokenizers, vocabulary, text filter, stemmer, and so on\n\n[![Build Status](https://travis-ci.org/chen0040/java-data-text.svg?branch=master)](https://travis-ci.org/chen0040/java-data-text) [![Coverage Status](https://coveralls.io/repos/github/chen0040/java-data-text/badge.svg?branch=master)](https://coveralls.io/github/chen0040/java-data-text?branch=master) [![Documentation Status](https://readthedocs.org/projects/java-data-text/badge/?version=latest)](http://java-data-text.readthedocs.io/en/latest/?badge=latest)\n  \n\n# Install\n\nAdd the following dependency to your POM file:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.chen0040\u003c/groupId\u003e\n  \u003cartifactId\u003ejava-data-text\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.3\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n# Features\n\n* Porter Stemmer\n* Punctuation Filter\n* Stop Word Removal\n\n    * Xml Tag Removal\n    * Ip Address Removal\n    * Number Removal\n\n* English Tokenizer\n\n\n# Usage\n\nTo use any text filter, just create a new text filter and then calls its filter(...) method.\n\n### Porter Stemmer\n\n```java\nimport com.github.chen0040.data.text.TextFilter;\nimport com.github.chen0040.data.text.PorterStemmer;\n\nTextFilter stemmer = new PorterStemmer();\nList\u003cString\u003e words = Arrays.asList(\n        \"caresses\",\n        \"ponies\",\n        \"ties\",\n        \"caress\",\n        \"cats\",\n        \"feed\",\n        \"agreed\",\n        \"disabled\",\n        \"matting\",\n        \"mating\",\n        \"meeting\",\n        \"milling\",\n        \"messing\",\n        \"meetings\"\n);\n\nList\u003cString\u003e result = stemmer.filter(words);\nfor (int i = 0; i \u003c words.size(); ++i)\n{\n    System.out.println(String.format(\"%s -\u003e %s\", words.get(i), result.get(i)));\n}\n```\n\n### StopWord Removal\n\n```java\nimport com.github.chen0040.data.text.TextFilter;\nimport com.github.chen0040.data.text.StopWordRemoval;\n\nStopWordRemoval filter = new StopWordRemoval();\n\nfilter.setRemoveNumbers(false);\nfilter.setRemoveIpAddress(false);\nfilter.setRemoveXmlTag(false);\n\nInputStream inputStream = FileUtils.getResource(\"documents.txt\");\nBufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));\nString content = reader.lines().collect(Collectors.joining(\"\\n\"));\nreader.close();\n\nList\u003cString\u003e before = BasicTokenizer.doTokenize(content);\nList\u003cString\u003e after = filter.filter(before);\n```\n\n### Punctuation Filtering\n\n```java\nimport com.github.chen0040.data.text.TextFilter;\nimport com.github.chen0040.data.text.PunctuationFilter;\n\nTextFilter filter = new PunctuationFilter();\n\nInputStream inputStream = FileUtils.getResource(\"documents.txt\");\nBufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));\nString content = reader.lines().collect(Collectors.joining(\"\\n\"));\nreader.close();\n\nList\u003cString\u003e before = BasicTokenizer.doTokenize(content);\nList\u003cString\u003e after = filter.filter(before);\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchen0040%2Fjava-data-text","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchen0040%2Fjava-data-text","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchen0040%2Fjava-data-text/lists"}