{"id":20150376,"url":"https://github.com/valentinarho/lda-rest","last_synced_at":"2025-04-09T20:30:23.419Z","repository":{"id":77359144,"uuid":"98435174","full_name":"valentinarho/lda-rest","owner":"valentinarho","description":"REST web service to compute and query Latent Dirichlet Allocation models","archived":false,"fork":false,"pushed_at":"2017-11-03T12:09:40.000Z","size":2604,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-23T22:24:12.971Z","etag":null,"topics":["docker","latent-dirichlet-allocation","lda","lda-model","mongo-database"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/valentinarho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-26T14:57:40.000Z","updated_at":"2025-02-06T04:47:22.000Z","dependencies_parsed_at":"2023-04-29T11:46:46.964Z","dependency_job_id":null,"html_url":"https://github.com/valentinarho/lda-rest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valentinarho%2Flda-rest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valentinarho%2Flda-rest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valentinarho%2Flda-rest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valentinarho%2Flda-rest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/valentinarho","download_url":"https://codeload.github.com/valentinarho/lda-rest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248107039,"owners_count":21048844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","latent-dirichlet-allocation","lda","lda-model","mongo-database"],"created_at":"2024-11-13T22:51:48.566Z","updated_at":"2025-04-09T20:30:23.377Z","avatar_url":"https://github.com/valentinarho.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Latent Dirichlet Allocation REST Web Service\n\nThis library provides a Python REST Web Service to access a simple pipeline to create and query LDA Models. Created models information are stored in a Mongo database and models file are stored on a shared folder in the host filesystem.\n\n## Project architecture\n\nThe system is composed by two Docker containers: \n\n* Python container, contains the API library code. This container is mapped on `localhost` on port `5000`.\n* Mongodb container, contains the database server. The container is available with the `db` hostname on port `27017`. \n\n### Run the containers and the application\n\nDownload the code from the repository, then **edit the docker-compose.yml file** to update all the mount source directories that will be shared between the containers and the host. \n\nThen, run:  \n\t\n\tdocker-compose build\n\tdocker-compose up\n\tdocker-compose exec web python db/load_fake_data.py\n\n### Python container composition\n\nThe main container is composed by\n\n* a Flask application that exposes some REST routes (see below)\n* a library with algorithms to compute and query the LDA model \n* some support tools to load initial data into the associated database\n \n### Available APIs\n\nFollowing a table describing the available APIs: \n\n| Endpoint | Http request | Description | Parameters | \n| --- | --- | --- | --- |\n| models/ | GET | Lists all models | - |\n| models/ | PUT | Creates a new model w.r.t. the provided parameters | * `model_id`: str, the id of the model to be created; * `number_of_topics`: int, the number of topics to extract; * `language`: 'en', the language of the documents; * `use_lemmer`: bool, true to perform lemmatisation, false to perform stemming; * `min_df`: int, the minimum number of documents that should contain a term to consider it; * `max_df`: float, the maximum percentage of documents that should contain a term to consider it as valid; * `chunksize`: int, the size of a chunk in LDA; * `num_passes`: int, the minimum number of passes through the dataset during learning with LDA;  * `waiting_seconds`: int, the number of seconds to wait before starting the learning; * `data_filename`: str, the filename in the 'data' folder that contains data, the file should contain a json dump of documents each one with `doc_id` and `doc_content` keys; * `data`: json dictionary, a dictionary of documents, containing document_id as key and document_content as value; * `assign_topics`: bool, true to assign topics to the newly created model and to save on db, false to ignore assignments for the learning documents; |  \n| models/`\u003cmodel-id\u003e` | GET | Shows detailed information about model with id `\u003cmodel-id\u003e` | - | \n| models/`\u003cmodel-id\u003e`/documents/ | GET | Lists all documents assigned to the model with id `\u003cmodel-id\u003e` | - |\n| models/`\u003cmodel-id\u003e`/ | DELETE | Delete the model with the specified id, stops the computation if scheduled or performed | - |\n| models/`\u003cmodel-id\u003e`/documents/`\u003cdoc-id\u003e` | GET | Shows detailed information about document with id `\u003cdoc-id\u003e` in model `\u003cmodel-id\u003e`| * `threshold`: float, the minimum probability that a topic should have to be returned as associated to the document.| \n| models/`\u003cmodel-id\u003e`/neighbors/ | GET | Computes and shows documents similar to the specified text.| * `text`: str, the text to categorize; * `limit`: int, the maximum number of similar documents to extract. |\n| models/`\u003cmodel-id\u003e`/documents/`\u003cdoc-id\u003e`/neighbors/ | GET | Computes and shows documents similar to the document identified with `\u003cdoc-id\u003e`.| * `limit`: int, the maximum number of similar documents to extract. |\n| models/`\u003cmodel-id\u003e`/topics/ | GET | Lists all topics related to the model with id `\u003cmodel-id\u003e` or extracts topics from a text if `text` is specified. | Only for extract topics from a text: * `text`, str, the text to compute topics for; * `threshold`, float, the min weight of a topic to be retrieved. |\n| models/`\u003cmodel-id\u003e`/topics/ | SEARCH | Computes and returns all topics assigned to the text. | * `text`, str, the text to compute topics for; * `threshold`, float, the min weight of a topic to be retrieved. |\n| models/`\u003cmodel-id\u003e`/topics/`\u003ctopic-id\u003e` | GET | Shows detailed information about topic with id `\u003ctopic-id\u003e` in model `\u003cmodel-id\u003e`| * `threshold`: float, the minimum probability that a topic should have to be returned as associated to the document.| \n| models/`\u003cmodel-id\u003e`/topics/`\u003ctopic-id\u003e`/documents | GET | Shows all documents associated to the topic with id `\u003ctopic-id\u003e` in model `\u003cmodel-id\u003e`| * `threshold`: float, the minimum probability of the topic that the document should have to be returned as associated to the topic.| \n| models/`\u003cmodel-id\u003e`/topics/`\u003ctopic-id\u003e`/documents | PUT | Compute topics associated to the provided document (single if `doc_id` and `doc_content` are set, multiple if `documents` is set) in model `\u003cmodel-id\u003e`| * `documents`: json dictionary, optional, keys are document ids and values are document contents; * `doc_id`, string, optional, the document id (in single case); * `doc_content`, string, optional, the document content; * `save_on_db`, bool, default True, true to save documents and topic assignments on db, False to return and forget.| \n| models/`\u003cmodel-id\u003e`/topics/`\u003ctopic-id\u003e` | PATCH | Update optional information of the topic with id `\u003ctopic-id\u003e` in model `\u003cmodel-id\u003e`| * `label`: str, optional, the topic label. * `description`: str, optional, the optional topic description. | \n \n\n\n### Load sample data\n\nTo invoke the modules that loads fake data into the database run, from the machine that is running Docker, the following command: \n\n    docker-compose exec web python db/load_fake_data.py\n\n\n### Database \n\nTo connect directly to the mongodb instance:\n\n* connect to the machine that hosts docker with ssh (optional: only if it is not the current machine)\n* the db is available on host 'db' port 27017\n\n#### Models\n\nWhen asking for model's detailed information, the required model can be in one of the following statuses: \n\n* `scheduled`, the model computation will start after the specified waiting period\n* `computing`, the model computation has been started and is currently running\n* `completed`, the model computation is finished and the model is stable\n* `killed`, the model computation has been interrupted by an error \n\n#### Languages\nThe language can be specified during in model creation message. Each model can handle only one language, chosen from: \n\n* `en` for english documents\n* `it` for italian documents, stopwords are available in `/app/resources` folder and lemmatisation is performed with `MorphIt`\n\n#### Documents\n\nDuring model computation it is possible to load documents in two ways: \n\n* load from file: provide the `data_filename` field in the request. The file should be a json file and should be contained in the data folder. The json should be a list of dictionaries, each dictionary represent a document and contains the keys `doc_id` and `doc_content`. For example: \n\n\t\t[\n\t\t\t{'doc_id': 'doc_1', 'doc_content': 'doc content 1'}, \n\t\t\t{'doc_id': 'doc_2', 'doc_content': 'doc content 2'}, \n\t\t\t{'doc_id': 'doc_3', 'doc_content': 'doc content 3'}\n\t\t]\n\n* load directly: provide the documents in the `data` field. This field should contain a dictionary of key:values where keys are document ids and values are document contents. \n\n\t\t{\n\t\t\t'doc_1': 'doc content 1', \n\t\t\t'doc_2': 'doc content 2', \n\t\t\t'doc_3': 'doc content 3'\n\t\t}\n\n#### Useful commands \n\nTo build all containers \n\n\tdocker-compose build\n\nTo run all containers \n\n\tdocker-compose up\n\nTo exec a command within a running container, e.g. load fake data into the mongo database\n\n    docker-compose exec web COMANDO ARGS\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvalentinarho%2Flda-rest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvalentinarho%2Flda-rest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvalentinarho%2Flda-rest/lists"}