{"id":17132076,"url":"https://github.com/dayyass/language-modeling","last_synced_at":"2025-04-13T07:55:29.060Z","repository":{"id":55091908,"uuid":"326240233","full_name":"dayyass/language-modeling","owner":"dayyass","description":"Pipeline for training Language Models using PyTorch.","archived":false,"fork":false,"pushed_at":"2022-05-24T20:34:15.000Z","size":70,"stargazers_count":12,"open_issues_count":11,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T07:55:24.361Z","etag":null,"topics":["decoding","deep-learning","gpt-2","language-modeling","lstm","natural-language-processing","ngrams","nlp","python","pytorch","rnn","sampling","text-generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dayyass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-02T17:55:26.000Z","updated_at":"2025-03-24T22:44:00.000Z","dependencies_parsed_at":"2022-08-14T11:40:50.397Z","dependency_job_id":null,"html_url":"https://github.com/dayyass/language-modeling","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flanguage-modeling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flanguage-modeling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flanguage-modeling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flanguage-modeling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dayyass","download_url":"https://codeload.github.com/dayyass/language-modeling/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248681491,"owners_count":21144700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["decoding","deep-learning","gpt-2","language-modeling","lstm","natural-language-processing","ngrams","nlp","python","pytorch","rnn","sampling","text-generation"],"created_at":"2024-10-14T19:25:57.763Z","updated_at":"2025-04-13T07:55:29.037Z","avatar_url":"https://github.com/dayyass.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### About\nPipeline for training Language Models using PyTorch.\u003cbr/\u003e\nInspired by Yandex Data School [NLP Course](https://github.com/yandexdataschool/nlp_course) ([week03](https://github.com/yandexdataschool/nlp_course/tree/2020/week03_lm): Language Modeling)\u003cbr/\u003e\n\n### Usage\nFirst, install dependencies:\n```\n# clone repo\ngit clone https://github.com/dayyass/language_modeling.git\n\n# install dependencies\ncd language_modeling\npip install -r requirements.txt\n```\n\n### Data Format\nPrepared text file with space separated words on each line.\u003cbr/\u003e\nMore about it [here](data/README.md).\n\n### Statistical Language Modeling\n#### Training\nScript for training statistical language models:\n```\npython statistical_lm/train.py --path_to_data \"data/arxiv_train.txt\" --n 3 --path_to_save \"models/3_gram_language_model.pkl\"\n```\nRequired arguments:\n- **--path_to_data** - path to train data\n- **--n** - n-gram order\n\nOptional arguments:\n- **--smoothing** - smoothing method (available: None, \"add-k\") (default: *None*)\n- **--delta** - smoothing additive parameter (only for add-k smoothing) (default: *1.0*)\n- **--path_to_save** - path to save model (default: *\"models/language_model.pkl\"*)\n- **--verbose** - verbose (default: *True*)\n\n#### Validation\nScript for validation statistical language models using perplexity:\n```\npython statistical_lm/validate.py --path_to_data \"data/arxiv_test.txt\" --path_to_model \"models/3_gram_language_model.pkl\"\n```\nRequired arguments:\n- **--path_to_data** - path to validation data\n- **--path_to_model** - path to language model\n\nOptional arguments:\n- **--verbose** - verbose (default: *True*)\n\n#### Inference\nScript for generation new sequences using statistical language models:\n```\npython statistical_lm/inference.py --path_to_model \"models/3_gram_language_model.pkl\" --prefix \"artificial\" --temperature 0.5\n```\nRequired arguments:\n- **--path_to_model** - path to language model\n\nOptional arguments:\n- **--prefix** - prefix before sequence generation (default: *\"\"*)\n- **--strategy** - decoding strategy (available: \"sampling\", \"top-k-uniform\", \"top-k\", \"top-p-uniform\", \"top-p\" and \"beam search\") (default: *\"sampling\"*)\n- **--temperature** - sampling temperature, if temperature == 0.0, always takes most likely token - greedy decoding (only for \"sampling\" decoding strategy) (default: *0.0*)\n- **--k** - top-k parameter (only for \"top-k-uniform\" and \"top-k\" decoding strategy) (default: *10*)\n- **--p** - top-p parameter (only for \"top-p-uniform\" and \"top-p\" decoding strategy) (default: *0.9*)\n- **--max_length** - max number of generated words (default: *100*)\n- **--seed** - random seed (default: *42*)\n\nCommand output with 3-gram language model trained on [*arxiv.txt*](data/README.md) with prefix \"*artificial*\" and temperature 0.5:\n```\nartificial neural network ( cnn ) architectures on h2o platform for real - world applications . \u003cEOS\u003e\n```\n\n### RNN Language Modeling\n#### Training\nScript for training RNN language models:\n```\npython rnn_lm/train.py --path_to_data \"data/arxiv_train.txt\" --path_to_save_folder \"models/rnn_language_model\" --n_epoch 5 --max_length 512 --batch_size 128 --embedding_dim 64 --rnn_hidden_size 256\n```\nRequired arguments:\n- **--path_to_data** - path to train data\n- **--n_epoch** - number of epochs\n- **--batch_size** - dataloader batch_size\n- **--embedding_dim** - embedding dimension\n- **--rnn_hidden_size** - LSTM hidden size\n\nOptional arguments:\n- **--path_to_save_folder** - path to save folder (default: *\"models/rnn_language_model\"*)\n- **--max_length** - max sentence length (chars) (default: *None*)\n- **--shuffle** - dataloader shuffle (default: *True*)\n- **--rnn_num_layers** - number of LSTM layers (default: *1*)\n- **--rnn_dropout** - LSTM dropout (default: *0.0*)\n- **--train_eval_freq** - evaluation frequency (number of batches) (default: *50*)\n- **--clip_grad_norm** - max_norm parameter in clip_grad_norm (default: *1.0*)\n- **--seed** - random seed (default: *42*)\n- **--device** - torch device (available: \"cpu\", \"cuda\") (default: *\"cuda\"*)\n- **--verbose** - verbose (default: *True*)\n\n#### Validation\nScript for validation RNN language models using perplexity:\n```\npython rnn_lm/validate.py --path_to_data \"data/arxiv_test.txt\" --path_to_model_folder \"models/rnn_language_model\" --max_length 512\n```\nRequired arguments:\n- **--path_to_data** - path to validation data\n- **--path_to_model** - path to language model\n\nOptional arguments:\n- **--max_length** - max sentence length (chars) (default: *None*)\n- **--seed** - random seed (default: *42*)\n- **--device** - torch device (available: \"cpu\", \"cuda\") (default: *\"cuda\"*)\n- **--verbose** - verbose (default: *True*)\n\n#### Inference\nScript for generation new sequences using RNN language models:\n```\npython rnn_lm/inference.py --path_to_model_folder \"models/rnn_language_model\" --prefix \"artificial\" --temperature 0.5\n```\nRequired arguments:\n- **--path_to_model_folder** - path to language model folder\n\nOptional arguments:\n- **--prefix** - prefix before sequence generation (default: *\"\"*)\n- **--temperature** - sampling temperature, if temperature == 0.0, always takes most likely token - greedy decoding (default: *0.0*)\n- **--max_length** - max number of generated tokens (chars) (default: *100*)\n- **--seed** - random seed (default: *42*)\n- **--device** - torch device (available: \"cpu\", \"cuda\") (default: *\"cuda\"*)\n\nCommand output with RNN language model trained on [*arxiv.txt*](data/README.md) with prefix \"*artificial*\" and temperature 0.5:\n```\nartificial visual information of the number , using an intervidence for detection for order to the recognition\n```\n\n### Models\nList of implemented models:\n- [x] [N-gram Language Model](https://github.com/dayyass/language_modeling/blob/b962edac04dfe10a3f87dfa16d4d37508af6d5de/model.py#L57)\n- [x] [RNN Language Model](https://github.com/dayyass/language_modeling/blob/407d02b79d6d7fd614dc7c5fd235ad269cddcb2d/rnn_lm/model.py#L6) (char-based)\n- [ ] GPT Language Model\n\n### Decoding Strategy\n- [x] greedy\n- [x] temperature sampling\n- [x] top-k-uniform\n- [x] top-k\n- [x] top-p-uniform\n- [x] top-p\n- [ ] beam search\n\n### Smoothing (only for N-gram Language Models)\n- [x] no smoothing\n- [x] add-k / Laplace smoothing\n- [ ] interpolation smoothing\n- [ ] back-off / Katz smoothing\n- [ ] Kneser-Ney smoothing\n\n### Models Comparison\nGeneration comparison available [here](https://github.com/dayyass/language-modeling/wiki/Generation-Comparison).\n\n#### Statistical Language Modeling\n| perplexity (train / test) | none             | add-k / Laplace    | interpolation | back-off / Katz | Kneser-Ney |\n| ------------------------- | ---------------- | -------------------| ------------- | --------------- | ---------- |\n| **1-gram**                | 881.27 / 1832.23 | 882.63 / 1838.22   | -             | -               | -          |\n| **2-gram**                | 95.32 / 8.57e+7  | 1106.79 / 1292.02  | -             | -               | -          |\n| **3-gram**                | 12.78 / 6.2e+22  | 7032.91 / 10499.24 | -             | -               | -          |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdayyass%2Flanguage-modeling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdayyass%2Flanguage-modeling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdayyass%2Flanguage-modeling/lists"}