{"id":17132045,"url":"https://github.com/dayyass/latent-semantic-analysis","last_synced_at":"2025-04-13T07:55:31.214Z","repository":{"id":62575202,"uuid":"410522697","full_name":"dayyass/latent-semantic-analysis","owner":"dayyass","description":"Pipeline for training LSA models using Scikit-Learn.","archived":false,"fork":false,"pushed_at":"2021-10-12T05:47:03.000Z","size":35,"stargazers_count":23,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T07:55:25.869Z","etag":null,"topics":["data-science","hacktoberfest","latent-semantic-analysis","lsa","machine-learning","natural-language-processing","nlp","pipeline","python","topic-modeling"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/latent-semantic-analysis/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dayyass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-26T10:40:10.000Z","updated_at":"2025-03-24T22:43:48.000Z","dependencies_parsed_at":"2022-11-03T18:54:35.081Z","dependency_job_id":null,"html_url":"https://github.com/dayyass/latent-semantic-analysis","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flatent-semantic-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flatent-semantic-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flatent-semantic-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dayyass%2Flatent-semantic-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dayyass","download_url":"https://codeload.github.com/dayyass/latent-semantic-analysis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248681491,"owners_count":21144700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","hacktoberfest","latent-semantic-analysis","lsa","machine-learning","natural-language-processing","nlp","pipeline","python","topic-modeling"],"created_at":"2024-10-14T19:25:50.924Z","updated_at":"2025-04-13T07:55:31.189Z","avatar_url":"https://github.com/dayyass.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![tests](https://github.com/dayyass/latent-semantic-analysis/actions/workflows/tests.yml/badge.svg)](https://github.com/dayyass/latent-semantic-analysis/actions/workflows/tests.yml)\n[![linter](https://github.com/dayyass/latent-semantic-analysis/actions/workflows/linter.yml/badge.svg)](https://github.com/dayyass/latent-semantic-analysis/actions/workflows/linter.yml)\n[![codecov](https://codecov.io/gh/dayyass/latent-semantic-analysis/branch/main/graph/badge.svg?token=Y39Q5786DL)](https://codecov.io/gh/dayyass/latent-semantic-analysis)\n\n[![python 3.6](https://img.shields.io/badge/python-3.6-blue.svg)](https://github.com/dayyass/latent-semantic-analysis#requirements)\n[![release (latest by date)](https://img.shields.io/github/v/release/dayyass/latent-semantic-analysis)](https://github.com/dayyass/latent-semantic-analysis/releases/latest)\n[![license](https://img.shields.io/github/license/dayyass/latent-semantic-analysis?color=blue)](https://github.com/dayyass/latent-semantic-analysis/blob/main/LICENSE)\n\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-black)](https://github.com/dayyass/latent-semantic-analysis/blob/main/.pre-commit-config.yaml)\n[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n[![pypi version](https://img.shields.io/pypi/v/latent-semantic-analysis)](https://pypi.org/project/latent-semantic-analysis)\n[![pypi downloads](https://img.shields.io/pypi/dm/latent-semantic-analysis)](https://pypi.org/project/latent-semantic-analysis)\n\n### Latent Semantic Analysis\nPipeline for training **LSA** models using Scikit-Learn.\n\n### Usage\nInstead of writing custom code for latent semantic analysis, you just need:\n1. install pipeline:\n```shell script\npip install latent-semantic-analysis\n```\n2. run pipeline:\n- either in **terminal**:\n```shell script\nlsa-train --path_to_config config.yaml\n```\n- or in **python**:\n```python3\nimport latent_semantic_analysis\n\nlatent_semantic_analysis.train(path_to_config=\"config.yaml\")\n```\n\n**NOTE**: more about config file [here](https://github.com/dayyass/latent-semantic-analysis/tree/main#config).\n\nNo data preparation is needed, only a **csv** file with raw text column (with arbitrary name).\n\n#### Config\nThe user interface consists of only one files:\n- [**config.yaml**](https://github.com/dayyass/latent-semantic-analysis/blob/main/config.yaml) - general configuration with sklearn **TF-IDF** and **SVD** parameters\n\nChange **config.yaml** to create the desired configuration and train LSA model with the following command:\n- **terminal**:\n```shell script\nlsa-train --path_to_config config.yaml\n```\n- **python**:\n```python3\nimport latent_semantic_analysis\n\nlatent_semantic_analysis.train(path_to_config=\"config.yaml\")\n```\n\nDefault **config.yaml**:\n```yaml\nseed: 42\npath_to_save_folder: models\n\n# data\ndata:\n  data_path: data/data.csv\n  sep: ','\n  text_column: text\n\n# tf-idf\ntf-idf:\n  lowercase: true\n  ngram_range: (1, 1)\n  max_df: 1.0\n  min_df: 1\n\n# svd\nsvd:\n  n_components: 10\n  algorithm: arpack\n```\n\n**NOTE**: `tf-idf` and `svd` are sklearn [**TfidfVectorizer**](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html?highlight=tfidf#sklearn.feature_extraction.text.TfidfVectorizer) and [**TruncatedSVD**](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html) parameters correspondingly, so you can parameterize instances of these classes however you want.\n\n#### Output\nAfter training the model, the pipeline will return the following files:\n- `model.joblib` - sklearn pipeline with LSA (TF-IDF and SVD steps)\n- `config.yaml` - config that was used to train the model\n- `logging.txt` - logging file\n- `doc2topic.json` - document embeddings\n- `term2topic.json` - term embeddings\n\n### Requirements\nPython \u003e= 3.6\n\n### Citation\nIf you use **latent-semantic-analysis** in a scientific publication, we would appreciate references to the following BibTex entry:\n```bibtex\n@misc{dayyass2021lsa,\n    author       = {El-Ayyass, Dani},\n    title        = {Pipeline for training LSA models},\n    howpublished = {\\url{https://github.com/dayyass/latent-semantic-analysis}},\n    year         = {2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdayyass%2Flatent-semantic-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdayyass%2Flatent-semantic-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdayyass%2Flatent-semantic-analysis/lists"}