{"id":24830135,"url":"https://github.com/quving/newsminer","last_synced_at":"2025-06-26T01:03:12.678Z","repository":{"id":43382609,"uuid":"246095847","full_name":"Quving/newsminer","owner":"Quving","description":"A service that analysis news articles stored in https://newsbox.quving.com.","archived":false,"fork":false,"pushed_at":"2022-12-08T03:48:37.000Z","size":66,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T02:43:10.841Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Quving.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-09T17:11:00.000Z","updated_at":"2020-05-05T02:12:44.000Z","dependencies_parsed_at":"2023-01-24T16:00:32.013Z","dependency_job_id":null,"html_url":"https://github.com/Quving/newsminer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Quving/newsminer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quving%2Fnewsminer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quving%2Fnewsminer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quving%2Fnewsminer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quving%2Fnewsminer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Quving","download_url":"https://codeload.github.com/Quving/newsminer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quving%2Fnewsminer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261978909,"owners_count":23239417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-30T23:49:06.322Z","updated_at":"2025-06-26T01:03:12.650Z","avatar_url":"https://github.com/Quving.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Newsminer\n\n## Description\nAs the name suggests, this repository contains scripts to analyze news articles. \nFor the first, only German texts will be analyzed. These texts will be provided by the [Newsbox-Api](https://newsbox.quving.com), which will aggregate new news at regular intervals.\nGoal is to obtain a LDA-Model that is capable of clustering the news.\n\n## Result and Demo\nThe visualization of the LDA can be found on https://newsmap.quving.com.\n\n## Setup\n\n### Installation\nInstall the required dependencies by the following steps.\n1. ```virtualenv -p $(which python3.6) venv```\n2. ```source venv/bin/acticate```\n3. ```pip install -r requirements.txt```\n\n\n### Authentication\nIn order to retrieve articles stored in the Newsbox-API, credentials are required.\nYou can obtain them by mailing me (vinh-ngu@hotmail.com).\n\n```\ncurl -X POST  \\ \n    -H \"accept: application/json\" \\\n    -H \"Content-Type: application/json\" \\ \n    -d \"{ \"username\\\": \"string\", \"password\\\": \"string\"}\" \\\n    https://newsbox.quving.com/auth/token/\n```\n\nExample Response:\n```bash\n{\n  \"refresh\": \"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI0NiJ10.eyJ0b2tlbl90eXBlIjoicmVmcmVzaCIsI2J...\",\n  \"access\": \"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI2NiJ10.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZX...\"\n}\n```\n\nNow set environment-variable to make it usable in the script.\n```\nexport NEWSMINER_AUTH_TOKEN='eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI2NiJ10.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZX...'\n```\n## Documentation\n### Train german POS-Tagger\nThe German POS Tagger is required for the lemmatization and stemming. That is a necessary step for the data preprocessing\nin order to train a LDA-Model.\n\n- To do that, you need to download the trainingset first. I recommend to use \n'[tiger_release_aug07.corrected.16012013.conll09](https://www.ims.uni-stuttgart.de/documents/ressourcen/korpora/tiger-corpus/download/start.html)'\n that is provided by the University of Stuttgart.\n- Once downloaded, create a directory 'data' and move the downloaded file to that directory.\n- Finally execute ``` python train_tagger_de.py ``` It will store the tagger in pickle-format to \n```[repo]/artifacts/tagger/```\n\n \n### Stemming/Lemmatization\nExample snippet.\n```\nfrom lemmatizer import Lemmatizer\n\nif __name__ == '__main__':\n    lemmatizer = Lemmatizer()\n    text = \"Heute war ein wirklich langer Tag gewesen. Der Hund isst sein Leckerli.\"\n    text = lemmatizer.lemmatize_text(text=text)\n    print(text)\n```\n\nOutput:\n\n```heut sein ein wirklich lang tag sein der hund essen sein leckerli```\n\n### Train LDA\n- In order to train a LDA, the POS-Tagger is required (stored locally in ```.../artififacts/tagger/...```)\n- Also, you need to have access to the Newsbox-API (see 'Authentication' section above.)\n- Execute ```python train_lda.py```. It can take some minutes depending on if you're using cached files or not.\n- If it's done, the browser should open up automatically.\n\n\n## Approach\n- If the tagger tag a word as noun, the original word will be taken instead of its stem.\n- Only nouns will be respected for the lda-model.\n\n### References\n- https://datascience.blog.wzb.eu/2016/07/13/accurate-part-of-speech-tagging-of-german-texts-with-nltk/ \n- https://towardsdatascience.com/topic-modeling-and-latent-dirichlet-allocation-in-python-9bf156893c24\n\n\n## Troubleshooting\n### Mysql - Problems (Mac)\n```\n...\n\n  copying MySQLdb/constants/FIELD_TYPE.py -\u003e build/lib.macosx-10.9-x86_64-3.6/MySQLdb/constants\n  copying MySQLdb/constants/FLAG.py -\u003e build/lib.macosx-10.9-x86_64-3.6/MySQLdb/constants\n  running build_ext\n  building 'MySQLdb._mysql' extension\n  creating build/temp.macosx-10.9-x86_64-3.6\n  creating build/temp.macosx-10.9-x86_64-3.6/MySQLdb\n  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -Dversion_info=(1,4,6,'final',0) -D__version__=1.4.6 -I/usr/local/Cellar/mysql/8.0.19/include/mysql -I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c MySQLdb/_mysql.c -o build/temp.macosx-10.9-x86_64-3.6/MySQLdb/_mysql.o\n  gcc -bundle -undefined dynamic_lookup -arch x86_64 -g build/temp.macosx-10.9-x86_64-3.6/MySQLdb/_mysql.o -L/usr/local/Cellar/mysql/8.0.19/lib -lmysqlclient -lssl -lcrypto -o build/lib.macosx-10.9-x86_64-3.6/MySQLdb/_mysql.cpython-36m-darwin.so\n  ld: library not found for -lssl\n  clang: error: linker command failed with exit code 1 (use -v to see invocation)\n  error: command 'gcc' failed with exit status 1\n  ----------------------------------------\n  ERROR: Failed building wheel for mysqlclient\n```\nSome pip dependencies require the mysql_config. Thus, following fix can be applied:\n\n```\nbrew install mysql-client\nbrew install openssl\nexport LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquving%2Fnewsminer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquving%2Fnewsminer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquving%2Fnewsminer/lists"}