{"id":15014184,"url":"https://github.com/d-one/nlpeasy","last_synced_at":"2025-04-12T06:05:12.185Z","repository":{"id":57446329,"uuid":"214123717","full_name":"d-one/NLPeasy","owner":"d-one","description":"Easy Peasy Language Squeezy","archived":false,"fork":false,"pushed_at":"2023-05-03T11:37:49.000Z","size":532,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T06:04:30.504Z","etag":null,"topics":["datascience","elasticsearch","kibana","nlp","spacy"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/d-one.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-10-10T08:06:28.000Z","updated_at":"2024-05-30T11:14:08.000Z","dependencies_parsed_at":"2023-12-07T04:42:33.783Z","dependency_job_id":null,"html_url":"https://github.com/d-one/NLPeasy","commit_stats":{"total_commits":55,"total_committers":2,"mean_commits":27.5,"dds":"0.036363636363636376","last_synced_commit":"90b314aa0ccfcd3d775e1ce9680e4500eac807da"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-one%2FNLPeasy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-one%2FNLPeasy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-one%2FNLPeasy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/d-one%2FNLPeasy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/d-one","download_url":"https://codeload.github.com/d-one/NLPeasy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248525144,"owners_count":21118618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datascience","elasticsearch","kibana","nlp","spacy"],"created_at":"2024-09-24T19:45:18.087Z","updated_at":"2025-04-12T06:05:12.155Z","avatar_url":"https://github.com/d-one.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Travis Build Status](\u003chttps://img.shields.io/travis/d-one/nlpeasy/master.svg?style=flat-square\u0026logo=travis-ci\u0026logoColor=white\u0026label=build\u003e)](https://travis-ci.org/d-one/nlpeasy)\n[![pypi Version](https://img.shields.io/pypi/v/nlpeasy.svg?style=flat-square\u0026logo=pypi\u0026logoColor=white)](https://pypi.org/project/nlpeasy/)\n\nNLPeasy\n=======\n\nBuild NLP pipelines the easy way\n\n\u003e **Disclaimer:** This is in Alpha stage, lot of things can go wrong.\n\u003e It could possibly change your Elasticsearch Data the API is not fixed yet\n\u003e and even the name NLPeasy might change.\n\n* Free software: Apache Software License 2.0\n\n\nUsage\n-----\n\nFor this example to completely work you need to have Python at least in Version 3.6 installed.\nAlso you need to have install and start either\n\n- **Docker** \u003chttps://www.docker.com/get-started\u003e, direct download links for\n    [Mac (DMG)](https://download.docker.com/mac/stable/Docker.dmg) and\n    [Windows (exe)](https://download.docker.com/win/stable/Docker%20for%20Windows%20Installer.exe).\n- **Elasticsearch** and **Kibana**:\n    \u003chttps://www.elastic.co/downloads/\u003e or\n    \u003chttps://www.elastic.co/downloads/elasticsearch-oss\u003e (pure Apache licensed version)\n\nThen on the terminal issue:\n```bash\npython -m venv venv\nsource venv/bin/activate\npip install nlpeasy scikit-learn\npython -m spacy download en_core_web_md\n```\nThe package `scikit-learn` is just used in this example to get the newsgroups data and preprocess it.\nThe last command downloads a spacy model for the english language -\nfor the following you need to have at least it's `md` (=medium) version which has wordvectors.\n\n```python\nimport pandas as pd\nimport nlpeasy as ne\nfrom sklearn.datasets import fetch_20newsgroups\n\n# connect to running elastic or else start an Open Source stack on your docker\nelk = ne.connect_elastic(docker_prefix='nlp', elk_version='7.10.2', mount_volume_prefix=None)\n# If it is started on docker it will on the first time pull the images (1.3GB)!\n# Setting mountVolumePrefix=\"./elastic-data/\" would keep the data of elastic in your\n# filesystems and then the data survives container restarts\n\n# read data as Pandas data frame\nnews_raw = fetch_20newsgroups(remove=('headers', 'footers', 'quotes'))\nnews_groups = [news_raw['target_names'][i] for i in news_raw['target']]\nnews = pd.DataFrame({'newsgroup': news_groups, 'message': news_raw['data']})\n\n# setup NLPeasy pipeline with name for the elastic index and set the text column\npipeline = ne.Pipeline(index='news', text_cols=['message'], tag_cols=['newsgroup'], elk=elk)\n\npipeline += ne.VaderSentiment('message', 'sentiment')\npipeline += ne.SpacyEnrichment(nlp='en_core_web_md', cols=['message'], vec=True)\n\n# do the pipeline - just for first 100, the whole thing would take 10 minutes\nnews_enriched = pipeline.process(news.head(10000000), write_elastic=True)\n\n# Create Kibana Dashboard of all the columns\npipeline.create_kibana_dashboard()\n\n# open Kibana in webbrowser\nelk.show_kibana()\n```\n\nLet's have some fun outside of Elastic/Kibana - but this needs `pip install matplotlib`\n```python\nimport numpy as np\nfrom scipy.cluster.hierarchy import dendrogram, linkage\nimport matplotlib.pyplot as plt\ngrouped = news_enriched.loc[~news_enriched.message_vec.isna()].groupby('newsgroup')\ngroup_vec = grouped.apply(lambda z: np.stack(z.message_vec.values).mean(axis=0))\nclust = linkage(np.stack(group_vec), 'ward')\n# calculate full dendrogram\nplt.figure(figsize=(10, 10))\nplt.title('Hierarchical Clustering Dendrogram Newsgroups')\nplt.xlabel('sample index')\nplt.ylabel('distance')\ndendrogram(\n    clust,\n    leaf_rotation=0.,  # rotates the x axis labels\n    leaf_font_size=8.,  # font size for the x axis labels\n    labels=group_vec.index,\n    orientation='left'\n)\nplt.show()\n```\n\nInstallation\n------------\n\nPrerequisites:\n\n- Python 3 (we use Python 3.7)\n- Elastic: Several possibilities\n    - Have Docker installed - needs to have the docker package installed (see below).\n    - Install and start Elasticsearch and Kibana:\n        \u003chttps://www.elastic.co/downloads/\u003e or\n        \u003chttps://www.elastic.co/downloads/elasticsearch-oss\u003e (pure Apache licensed version)\n    - Use any running Elasticsearch and Kibana (on premise or cloud)...\n- Pretrained Models: See below for Spacy Language Models and WordVectors\n\nIt is recommended to use a virtual environment:\n```bash\ncd $PROJECT_DIR\npython -m venv venv\nsource venv/bin/activate\n```\nThe source statement has to be repeated whenever you open a new terminal.\n\nThen install\n```bash\npip install nlpeasy\n```\nOr the development version from GitHub:\n```bash\npip install --upgrade git+https://github.com/d-one/nlpeasy\n```\n\nIf you want to use spaCy language models download them (90-200 MB), e.g.\n```bash\npython -m spacy download en_core_web_md\n# and/or\npython -m spacy download de_core_news_md\n```\nIf you want to use pretrained [FastText-Wordvectors](https://fasttext.cc/docs/en/pretrained-vectors.html) (each ~7GB):\n```bash\ncurl -O https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.en.zip\ncurl -O https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.de.zip\n```\n\nIf you want to use Jupyter, install it to the virtual environment:\n```bash\npip install jupyterlab\n```\n\n### Development\nTo install this module in Dev-mode, i.e. change files and reload module:\n```bash\ngit clone https://github.com/d-one/nlpeasy\ncd nlpeasy\n```\n\nIt is recommended to use a virtual environment:\n```bash\npython -m venv venv\nsource venv/bin/activate\n```\n\nInstall the version in edit mode:\n```bash\npip install -e .\n```\n\nIn Jupyter you can have reloaded code when you change the files as in:\n```python\n%load_ext autoreload\n%autoreload 2\n```\n\nFeatures\n--------\n\n* Pandas based pipeline\n* Support for any extensions - now includes some for Regex, spaCy, VaderSentiment\n* Write results to ElasticSearch\n* Automatic Kibana dashboard generation\n* Have Elastic started in Docker if it is not installed locally or remotely\n* Apache License 2.0\n\nCredits\n-------\n\nThis package was created with [Cookiecutter](\u003chttps://github.com/audreyr/cookiecutter\u003e) and the [`audreyr/cookiecutter-pypackage`]\u003chttps://github.com/audreyr/cookiecutter-pypackage\u003e project template.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-one%2Fnlpeasy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fd-one%2Fnlpeasy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fd-one%2Fnlpeasy/lists"}