{"id":19533193,"url":"https://github.com/klaragtknst/text_topic","last_synced_at":"2025-02-26T03:41:46.999Z","repository":{"id":261753910,"uuid":"874153609","full_name":"KlaraGtknst/text_topic","owner":"KlaraGtknst","description":"This repository implements a pipeline to store various data of files from a large unstructured dataset. These fields are used for topic modeling (wordclouds, based on low-dimensional versions of embedding vectors, Named Entity Clustering and document-topic incidences). The information is aggregated and visualised using FCA.","archived":false,"fork":false,"pushed_at":"2025-02-14T15:06:21.000Z","size":4447,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-26T03:41:42.784Z","etag":null,"topics":["documents","elasticsearch","embeddings","fca","ner","ner-clustering","sentence-transformers","text-data","top2vec","topic-aggregation","topics-modeling","visualisation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KlaraGtknst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-17T10:46:54.000Z","updated_at":"2025-02-14T15:06:50.000Z","dependencies_parsed_at":"2024-12-09T16:26:32.996Z","dependency_job_id":"d09132f6-ffb5-41a0-99a0-6240b1134a48","html_url":"https://github.com/KlaraGtknst/text_topic","commit_stats":null,"previous_names":["klaragtknst/text_topic"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KlaraGtknst%2Ftext_topic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KlaraGtknst%2Ftext_topic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KlaraGtknst%2Ftext_topic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KlaraGtknst%2Ftext_topic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KlaraGtknst","download_url":"https://codeload.github.com/KlaraGtknst/text_topic/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240788776,"owners_count":19857695,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["documents","elasticsearch","embeddings","fca","ner","ner-clustering","sentence-transformers","text-data","top2vec","topic-aggregation","topics-modeling","visualisation"],"created_at":"2024-11-11T02:05:55.338Z","updated_at":"2025-02-26T03:41:46.314Z","avatar_url":"https://github.com/KlaraGtknst.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# text_topic\nThis repository implements a pipeline to store various data of files from a large unstructured dataset. \nThese fields are used for topic modeling (wordclouds, based on low-dimensional versions of embedding vectors, Named Entity Clustering and document-topic incidences). \nThe information is aggregated and visualised using FCA. \n\n## Docker Container\nTo build the docker container, run the following command in the root directory of the project on server:\n```bash\n docker compose up -d\n```\nHence, the elastic search database will be available at `http://localhost:9200`.\n**-d flag** stands for detached mode. \nWhen you include this flag, \nDocker Compose runs the containers in the background and detaches the process from your terminal.\n\n## Elasticsearch Database\nBefore using the database on the watzmann server, you need to create a tunnel to the server\n(i.e. run it locally on your machine).:\n```bash\nssh -L 9200:localhost:9200 watzmann\n```\nThis is ssh port forwarding: \nport forwarding or port mapping is an application of network address translation (NAT) that redirects a communication \nrequest from one address and port number combination to another. (https://en.wikipedia.org/wiki/Port_forwarding, 22.12.2024)\n\n- first 9200: This is the port on your local machine (your client machine) where the SSH tunnel will listen for connections.\n- localhost: This is the destination host from the perspective of the SSH server (in this case, the watzmann server).\n- second 9200: This is the port on the destination host (from the server's perspective) to which traffic should be forwarded. \n- watzmann: This is the remote host (the SSH server) you are connecting to.\n\nWhen you run this command:\n1. You establish an SSH connection to the remote host watzmann.\n2. A tunnel is created between your local machine and watzmann.\n3. Any connection made to localhost:9200 on your local machine will be securely forwarded to localhost:9200 on the watzmann machine.\n\n## Start the Pipeline of filling the index\nWe have chosen to split the pipeline into two parts, because the text related fields are computed on a different server \nwhich has more graphical computation power.\nYou can see the workflow in the following image:\n\n![es_workflow.svg](doc/es_workflow.svg)\n\nYou can run the following file to initialize the index (i.e. run on the watzmann server):\n```bash\npython3 main.py\n```\nHence, the index is created, but no documents are indexed yet.\nTo index the documents and insert their metadata, run the following command (i.e. run on the watzmann server):\n```bash\npython3 insert_metadata.py\n```\n![text_related_workflow.svg](doc/text_related_workflow.svg)\n\nTo insert the text related fields of the documents, run the following command (i.e. run on the pumbaa server):\n```bash \npython3 insert_text_related_fields.py\n```\nAs you can see in the image, the text related fields compromise of three fields.\nThe first field is the text of a file either obtained directly, via a PdfReader or via an ImageCaptioner.\nThe second field is the embeddings of the text, which are computed by the `sentence-transformers` library \n([SBERT](https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-12-v3)).\nThe third field is a nested structure containing Named Entities of the text, \nobtained using the small english pipeline `en_core_web_sm` of the [spaCy](https://spacy.io/models) library.\n\n\n## Obtain incidences\nWith reference to [\"The Geometric Structure of Topic Models\", Johannes Hirth and Tom Hanika (2024)](https://arxiv.org/abs/2403.03607),\nwe obtain the document-topic and topic-word incidences.\n\n![doc_topic_inc_fca_hirth_hanika.svg](doc/doc_topic_inc_fca_hirth_hanika.svg)\n\nYou can run the following command to obtain the incidences (i.e. run on the watzmann server):\n```bash\npython3 create_fca_incidences.py\n```\n\n## Incidences to Context\nOnce the incidences are produced, you want to convert them to the FIMI format:\n```bash\nphyton3 run_topic_fca.py\n```\nThis call will also create document-topic incidences per directory which will be used later to compute a directory-topic context.\n\nThe FIMI files can be used to compute their intents via PCBO (FCALGS).\nThis algorithm is implemented in the `fcalgs` package.\nYou need to install the package first (i.e. run on the watzmann server):\n```bash\nwget https://sourceforge.net/projects/fcalgs/files/pcbo/amai/pcbo-amai.zip\nunzip pcbo-amai.zip\ncd pcbo-amai\nmake\n``` \n(website worked on 09.01.2025)\n\nAfter that, you can run the following command to compute the intents (in the `pcbo-amai` directory):\n```bash\n./pcbo -P4 /file/to/fimi/file.fimi /name/of/output/file.fimi\n```\n# Topic Modeling Strategies\n\n## Wordclouds \u0026 2D scatter of documents coloured by their directory\nTo run both strategies above (and more, you might have to comment functions you don't need),\nrun the following command:\n```bash\npython3 visualizations.py\n```\n![wordcloud_Military.svg](doc/wordcloud_Military.svg)\n![TSNE_scatter_documents_dir_2d_01_22_25.png](doc/TSNE_scatter_documents_dir_2d_01_22_25.png)\n## Named Entity Clustering\nSimilar to [\"Clustering Prominent Named Entities in Topic-Specific Text Corpora\", A. Alsudais and H. Tchalian (2019)](https://arxiv.org/pdf/1807.10800),\nwe cluster named entities of different categories across the text extracted from the files of the dataset.\nTo cluster the named entities, you can run the following command (i.e. run on the watzmann server):\n```bash\npython3 run_named_entity_clustering.py\n```\nThe workflow is displayed in the following image:\n![text_related_workflow.svg](doc/NE_Clustering.svg)\nThe results vary in quality strongly depending on the NER and text quality.\nAn example of the clustering (of dataset [EYNTKE](https://archive.org/details/ETYNTKE)) is shown in the following image.\nDifferent language families are well separated, but the topological structure forms no clear clusters.\n![named_entity_clusters_LANGUAGE_PCA_01_22_25.svg](doc/named_entity_clusters_LANGUAGE_PCA_01_22_25.svg)\n\n## Supplementary Information\nEnsure that the transformers library version is == 4.48.0.\nAs of 16.01.2025, using transformers == 4.45.0 will result in an error (on the server), since the processor used for the \nimage captioner could not be loaded.\n\nMake sure that the pretrained models are downloaded:\n```python -m spacy download en_core_web_sm```\n\n## License\nMIT License\n\nCopyright (c) 2024 - 2025 KlaraGtknst\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklaragtknst%2Ftext_topic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fklaragtknst%2Ftext_topic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklaragtknst%2Ftext_topic/lists"}