{"id":19001305,"url":"https://github.com/oneflow-inc/text","last_synced_at":"2026-02-12T08:33:15.025Z","repository":{"id":42051034,"uuid":"419944591","full_name":"Oneflow-Inc/text","owner":"Oneflow-Inc","description":"Data loaders and abstractions for text and NLP","archived":false,"fork":false,"pushed_at":"2022-11-29T13:07:34.000Z","size":114,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-22T17:50:39.844Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Oneflow-Inc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-22T02:45:49.000Z","updated_at":"2022-04-20T06:09:21.000Z","dependencies_parsed_at":"2022-08-12T03:30:34.246Z","dependency_job_id":null,"html_url":"https://github.com/Oneflow-Inc/text","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Oneflow-Inc/text","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ftext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ftext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ftext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ftext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Oneflow-Inc","download_url":"https://codeload.github.com/Oneflow-Inc/text/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oneflow-Inc%2Ftext/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29361819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T01:03:07.613Z","status":"online","status_checked_at":"2026-02-12T02:00:06.911Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T18:10:42.392Z","updated_at":"2026-02-12T08:33:15.010Z","avatar_url":"https://github.com/Oneflow-Inc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# text\nModels, Datasets, Metrics and Utils for NLP.\n\n\n## Installation\n...\n\n\n## Usage\n\u003cdetails\u003e\n\u003csummary\u003e \u003cb\u003e Models \u003c/b\u003e \u003c/summary\u003e\n\n- **Supported model and model type**\n    \n    **bert** : {\"bert-base-cased\", \"bert-base-uncased\", \"bert-large-cased\",\"bert-large-uncased\", \"bert-base-chinese\"}\n    \n    **elmo** : {\"elmo-simplified-chinese\", \"elmo-traditional-chinese\", \"elmo-english\"}\n    \n- **Load the pretrained model**\n    \n```python\n# Load the pretrained model.\nfrom flowtext.models import bert\nbert, tokenizer, bert_config = bert(pretrained=True, model_type=bert-base-uncased', checkpoint_path=None)\n\n# In addition, you can also load normal models.\nfrom flowtext.models import BertConfig, BertModel\nconfig = BertConfig()\nbert = BertModel(config)\n```\n    \n    \n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e \u003cb\u003e Datasets \u003c/b\u003e \u003c/summary\u003e\n\n- **The dataset module currently contains:**\n\n    Language modeling:   [WikiText2, WikiText103, PennTreebank]\n    \n    Machine translation:   [IWSLT2016, IWSLT2017, Multi30k]\n    \n    Sequence tagging(e.g. POS/NER):    [UDPOS, CoNLL2000Chunking] \n    \n    Question answering:   [SQuAD1, SQuAD2]\n    \n    Text classification:   [AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDB]\n\n  \n- **Load NLP related datasets, and build dataloader**\n```python\nfrom flowtext.datasets import AG_NEWS\ntrain_iter = AG_NEWS(split='train')\nnext(train_iter)\n# Or iterate with for loop\nfor (label, line) in train_iter:\n    print(label, line)\n# Or send to DataLoader\nfrom oneflow.utils.data import DataLoader\ntrain_iter = AG_NEWS(split='train')\ndataloader = DataLoader(train_iter, batch_size=8, shuffle=False)\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e \u003cb\u003e Metrics \u003c/b\u003e \u003c/summary\u003e\n\n- **The metrics currently contains:**\n    \n    Bleu_score\n    \n    Ngram_counter\n\n- **NLP related evaluation metrics**\n```python\n\u003e\u003e\u003e from flowtext.data.metrics import bleu_score\n\u003e\u003e\u003e candidate_corpus = [['This', 'is', 'a', 'oneflow', 'bleu','test'], ['Another', 'Sentence']]\n\u003e\u003e\u003e references_corpus = [[['This', 'is', 'a', 'oneflow', 'bleu','test'], ['Completely', 'Different']], [['No', 'Match']]]\n\u003e\u003e\u003e bleu_score(candidate_corpus, references_corpus)\n0.889139711856842\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e \u003cb\u003e Utils \u003c/b\u003e \u003c/summary\u003e\n\n- **Load tokenizer**\n```python\n\u003e\u003e\u003e from flowtext.data import get_tokenizer\n# The parameter ‘tokenizer’ can support spacy, moses, toktok, revtok, subword, jieba.\n\u003e\u003e\u003e tokenizer = get_tokenizer(tokenizer=\"basic_english\", language=\"en\")\n\u003e\u003e\u003e tokens = tokenizer(\"Today is a good day!\")\n\u003e\u003e\u003e tokens\n['today', 'is', 'a', 'good', 'day', '!']\n```\n\n\u003c/details\u003e\n\n## Disclaimer on Datasets\n\nThe datasets in flowtext.datasets is a utility library that downloads and prepares public datasets. We are not responsible for hosting and distributing these data sets, nor do we guarantee their quality and fairness, nor do we claim to have the license of the data set. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license. \n\nIf you are the dataset owner and want to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please contact us through GitHub questions.\n\n## License\n\nOneFlow has a BSD-style license, as found in the [LICENSE](https://github.com/Oneflow-Inc/text/blob/main/LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneflow-inc%2Ftext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foneflow-inc%2Ftext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foneflow-inc%2Ftext/lists"}