{"id":19832034,"url":"https://github.com/ail-project/topic-extractor","last_synced_at":"2026-03-05T13:31:47.379Z","repository":{"id":224714341,"uuid":"722010115","full_name":"ail-project/topic-extractor","owner":"ail-project","description":"A python library to extract topic from text","archived":false,"fork":false,"pushed_at":"2024-02-27T10:42:26.000Z","size":90,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-02-28T21:28:46.991Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ail-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-22T08:47:36.000Z","updated_at":"2024-02-27T10:41:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"17be5437-86d6-452e-b789-860b4ddc13ba","html_url":"https://github.com/ail-project/topic-extractor","commit_stats":null,"previous_names":["ail-project/topic-extractor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ail-project/topic-extractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Ftopic-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Ftopic-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Ftopic-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Ftopic-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ail-project","download_url":"https://codeload.github.com/ail-project/topic-extractor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2Ftopic-extractor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30127801,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T12:40:50.676Z","status":"ssl_error","status_checked_at":"2026-03-05T12:39:32.209Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T11:36:05.440Z","updated_at":"2026-03-05T13:31:47.307Z","avatar_url":"https://github.com/ail-project.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Topic-based Extraction\nThis repo contains:\n\n1. Methods for loading documents in various formats such as txt, csv, json, jsonl, and pdf.\n2. Advanced preprocessing methods including handling numbers, word contractions, lowercasing, stopwords removal, lemmatization, etc.\n3. Three methods for using Latent Dirichlet Allocation (LDA): BERTopic, a simple LDA with optimal number of topics found using coherence scores, and an LDA model build with GridSearch to automatically determine the best LDA model to use (recommanded).\n4. Methods for visualizing the extracted topics (pyLDAVis, WorldCloud \u0026 BERTopic that got its own data viz)\n\n\n## What is LDA: Latent Dirichlet Allocation\n\nLatent Dirichlet Allocation (LDA) is a statistical model used in natural language processing (NLP) for topic modeling.\nIt assumes that documents are made up of topics, and each topic is a distribution over words. \nLDA aims to uncover these topics from a collection of documents by analyzing the co-occurrence patterns of words. \n\nIt allows for automatic discovery of topics in a set of documents, making it a valuable tool for organizing and understanding large text corpora.\nLatent Dirichlet Allocation is not a single-iteration algorithm. In the first iteration, the algorithm randomly assigns words to topics. It then goes through each word of each document and applies probability calculation formulas.\n\nThe process is then repeated through various iterations until the algorithm generates a set of topics and we obtain a satisfactory model.\n\n## LDA Models\n\nIn order to compute LDA model using either coherence scores comparison either Gridsearch, I arbitrary chose to start with at least 3 topics and test all value until 30 with a step of 2.\nIt can of course be change within the code (check for method parameters), specially if you want a faster execution.\n\nIf no method argument is given while lauching this code, it will use GridSearchCV method.\n\n\n## Installation \u0026 Usage\n\nNOTE: Python 3.9 or higher is required.\n\n```python\n\n# create a venv\npython -m venv lda\n\n# install requirements\npip install -r requirements.txt\n\n# usage\npython lda.py \u003cfile_name\u003e --method \u003cmethod_name\u003e\n\n # method_name must be in [\"gridsearch\", \"bertopic\", \"coherence\"] \n```\n\nAfter this, a graphic will be saved as lda.html and cluster and key-word can be visualized in browser.\n![PyLDAVis](static/pyldavis.png)\n\n## Usage examples\n```\npython lda.py dissertation_en.pdf --method coherence\n\n```\n\n## Potential improvements\n\n1. Add some english stopwords (will increase accuracy)\n2. Multilingual support\n3. Make it faster\n4. Save the built model in order to be used again","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Ftopic-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fail-project%2Ftopic-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Ftopic-extractor/lists"}