{"id":15671612,"url":"https://github.com/netzkolchose/elastipy","last_synced_at":"2025-05-06T20:28:26.863Z","repository":{"id":39634544,"uuid":"323788286","full_name":"netzkolchose/elastipy","owner":"netzkolchose","description":"python elasticsearch query module for easily accessing nested aggregations and such  ","archived":false,"fork":false,"pushed_at":"2025-03-04T11:10:40.000Z","size":7136,"stargazers_count":3,"open_issues_count":4,"forks_count":0,"subscribers_count":3,"default_branch":"development","last_synced_at":"2025-04-14T00:32:41.734Z","etag":null,"topics":["analytics","backend","console","elasticsearch","elasticsearch-queries","nested-aggregations","pandas-dataframe"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/netzkolchose.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-23T03:03:27.000Z","updated_at":"2025-03-04T11:10:36.000Z","dependencies_parsed_at":"2022-09-20T07:00:13.235Z","dependency_job_id":null,"html_url":"https://github.com/netzkolchose/elastipy","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netzkolchose%2Felastipy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netzkolchose%2Felastipy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netzkolchose%2Felastipy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netzkolchose%2Felastipy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/netzkolchose","download_url":"https://codeload.github.com/netzkolchose/elastipy/tar.gz/refs/heads/development","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252763600,"owners_count":21800533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","backend","console","elasticsearch","elasticsearch-queries","nested-aggregations","pandas-dataframe"],"created_at":"2024-10-03T15:04:09.709Z","updated_at":"2025-05-06T20:28:26.839Z","avatar_url":"https://github.com/netzkolchose.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## elastipy\n\nA python wrapper to make elasticsearch queries and aggregations more fun.\n\nTested with python 3.6 and 3.10 and elasticsearch 7 and 8.\n\n[![test](https://github.com/netzkolchose/elastipy/actions/workflows/tests.yml/badge.svg)](https://github.com/netzkolchose/elastipy/actions/workflows/tests.yml)\n[![Coverage Status](https://coveralls.io/repos/github/netzkolchose/elastipy/badge.svg?branch=development)](https://coveralls.io/github/netzkolchose/elastipy?branch=development)\n[![Documentation Status](https://readthedocs.org/projects/elastipy/badge/?version=latest)](https://elastipy.readthedocs.io/en/latest/?badge=latest)\n\n\nLearn more at [elastipy.readthedocs.io](https://elastipy.readthedocs.io/en/latest/).\n\nIn comparison to [elasticsearch-dsl](https://github.com/elastic/elasticsearch-dsl-py)\nthis library provides:\n- typing and IDE-based auto-completion for search and aggregation parameters.\n- some convenient data access to responses of nested bucketed aggregations and metrics\n  (also supporting [pandas](https://github.com/pandas-dev/pandas))\n\n\n#### contents\n\n- [installation](#installation)\n- [requirements](#requirements)\n- quickref\n    - [aggregations](#aggregations)\n    - [metrics](#nested-aggregations-and-metrics)\n    - [query](#queries)\n    - [exporting](#exporting)\n- [testing](#testing)\n- [development](#development)\n\n---\n\n### installation\n\nTo install elastipy using the elasticsearch 8+ backend:\n\n```shell script\npip install elastipy\n``` \n\nIf you target the elasticsearch 7 version, do:\n\n```shell script\npip install 'elasticsearch\u003c8'\npip install elastipy\n``` \n\n\n#### requirements\n\nOne thing is, of course, to [install elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html).\n\n- **elastipy** itself requires [elasticsearch-py](https://github.com/elastic/elasticsearch-py)\n- doc building is listed in [docs/requirements.txt](docs/requirements.txt) and mainly\nconsists of sphinx with the readthedocs theme.\n- generating the interface and running the tests and notebooks is listed in \n[requirements.txt](requirements.txt) and contains pyyaml and coverage as well as the \nusual stack of jupyter, scipy, matplotlib, ..   \n\n\n### configuration \n\nBy default an [elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html) host is expected at `localhost:9200`. There are currently two ways \nto specify a different connection.\n\n\n```python\nfrom elasticsearch import Elasticsearch\nfrom elastipy import Search\n\n# Use an explicit Elasticsearch client (or compatible class)\nclient = Elasticsearch(\n    hosts=[{\"host\": \"localhost\", \"port\": 9200}], \n    http_auth=(\"user\", \"pwd\")\n)\n\n# create a Search using the specified client\ns = Search(index=\"bla\", client=client)\n\n# can also be done later\ns = s.client(client)\n```\n\nCheck the Elasticsearch [API reference](https://elasticsearch-py.readthedocs.io/en/v7.10.1/api.html#elasticsearch) for all the parameters.\n\nWe can also set a default client at the program start:  \n\n\n```python\nfrom elastipy import connections\n\nconnections.set(\"default\", client)\n\n# .. or as parameters, they get converted to an Elasticsearch client\nconnections.set(\"default\", {\"hosts\": [{\"host\": \"localhost\", \"port\": 9200}]})\n\n# get a client\nconnections.get(\"default\")\n```\n\n\n\n\n    \u003cElasticsearch([{'host': 'localhost', 'port': 9200}])\u003e\n\n\n\nDifferent connections can be specified with the *alias* name:\n\n\n```python\nconnections.set(\"special\", {\"hosts\": [{\"host\": \"special\", \"port\": 1234}]})\n\ns = Search(client=\"special\")\ns.get_client()\n```\n\n\n\n\n    \u003cElasticsearch([{'host': 'special', 'port': 1234}])\u003e\n\n\n\n### aggregations\n\nMore details can be found in the [tutorial](https://elastipy.readthedocs.io/en/latest/tutorial.html).\n\n\n```python\n# get a search object\ns = Search(index=\"world\")\n\n# create an Aggregation class connected to the Search\nagg = s.agg_date_histogram(calendar_interval=\"1w\")\n# (for date-specific aggregations we can leave out the 'field' parameter \n#  it falls back to Search.timestamp_field which is \"timestamp\" by default)\n\n# submit the whole request\ns.execute()\n\n# access the response\n\nlist(agg.keys())\n```\n\n\n\n\n    ['1999-12-27T00:00:00.000Z',\n     '2000-01-03T00:00:00.000Z',\n     '2000-01-10T00:00:00.000Z',\n     '2000-01-17T00:00:00.000Z']\n\n\n\n\n```python\nlist(agg.values())\n```\n\n\n\n\n    [21, 77, 60, 42]\n\n\n\nWithout a [metric](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics.html) these numbers are the document counts.\n\nAbove example as a one-liner:\n\n\n```python\nSearch(index=\"world\").agg_date_histogram(calendar_interval=\"1w\").execute().to_dict()\n```\n\n\n\n\n    {'1999-12-27T00:00:00.000Z': 21,\n     '2000-01-03T00:00:00.000Z': 77,\n     '2000-01-10T00:00:00.000Z': 60,\n     '2000-01-17T00:00:00.000Z': 42}\n\n\n\n### nested aggregations and metrics\n\n\n```python\ns = Search(index=\"world\")\n\n# the first parameter is the name of the aggregation \n#   (if omitted it will be \"a0\", \"a1\", aso..)  \nagg = s \\\n    .agg_terms(\"occasion\", field=\"occasion\") \\\n    .agg_rare_terms(\"rare-excuses\", field=\"excuse\", max_doc_count=2) \\\n    .metric_avg(\"avg-length\", field=\"conversation_length\") \\\n    .metric_max(\"max-length\", field=\"conversation_length\") \\\n    .execute()\n```\n\nThe `rare_terms` aggregation is nested into the `terms` aggregation and \nthe metrics are siblings nested inside `rare_terms`.\n\n`keys()`, `values()`, `items()` and `to_dict()` all operate on the current aggregation.\nFor bucket aggregations they typically show the `doc_count` value.'\n\n\n```python\nagg.to_dict()\n```\n\n\n\n\n    {('dinner', 'my mouth is too dry'): 1,\n     ('dinner', \"i can't reach the spoon\"): 2}\n\n\n\nThe `rows()`, `dict_rows()` and `dump.table()` methods operate on the whole aggregation branch:\n\n\n```python\nlist(agg.dict_rows())\n```\n\n\n\n\n    [{'occasion': 'dinner',\n      'occasion.doc_count': 200,\n      'rare-excuses': 'my mouth is too dry',\n      'rare-excuses.doc_count': 1,\n      'avg-length': 163.0,\n      'max-length': 163.0},\n     {'occasion': 'dinner',\n      'occasion.doc_count': 200,\n      'rare-excuses': \"i can't reach the spoon\",\n      'rare-excuses.doc_count': 2,\n      'avg-length': 109.5,\n      'max-length': 133.0}]\n\n\n\n\n```python\nagg.dump.table(colors=False)\n```\n\n    occasion │ occasion.doc_count │ rare-excuses            │ rare-excuses.doc_count │ avg-length   │ max-length  \n    ─────────┼────────────────────┼─────────────────────────┼────────────────────────┼──────────────┼─────────────\n    dinner   │ 200                │ my mouth is too dry     │ 1 ██████████▌          │ 163.0 ██████ │ 163.0 ██████\n    dinner   │ 200                │ i can't reach the spoon │ 2 ████████████████████ │ 109.5 ████   │ 133.0 ████▉ \n\n\n### queries\n\n\n```python\nfrom elastipy import query\n\ns = Search(index=\"prog-world\")\n\n# chaining means AND\ns = s \\\n    .term(field=\"category\", value=\"programming\") \\\n    .term(\"usage\", \"widely-used\")\n\n# also can use operators\ns = s \u0026 (\n    query.Term(\"topic\", \"yet-another-api\") \n    | query.Term(\"topic\", \"yet-another-operator-overload\")\n)\n\n# .query() replaces the current query \ns = s.query(query.MatchAll())\n\nlanguages_per_country = s.agg_terms(field=\"country\").agg_terms(field=\"language\").execute()\n\nlanguages_per_country.to_dict()\n```\n\n\n\n\n    {('IT', 'PHP'): 28,\n     ('IT', 'Python'): 24,\n     ('IT', 'C++'): 21,\n     ('ES', 'C++'): 29,\n     ('ES', 'Python'): 22,\n     ('ES', 'PHP'): 18,\n     ('US', 'PHP'): 23,\n     ('US', 'Python'): 20,\n     ('US', 'C++'): 15}\n\n\n\n### exporting\n\nThere is a small helper to export stuff to elasticsearch.\n\n\n\n\n```python\nfrom elastipy import Exporter\n\nclass MyExporter(Exporter):\n    INDEX_NAME = \"my-index\"\n    \n    # mapping can be defined here\n    # it will be sent to elasticsearch before the first document is exported \n    MAPPINGS = {\n        \"properties\": {\n            \"some_field\": {\"type\": \"text\"},\n        }       \n    }   \n\ncount, errors = MyExporter().export_list(a_lot_of_objects)\n\nprint(f\"expored {count} objects, errors: {errors}\")\n```\n\n    expored 1000 objects, errors: []\n\n\nIt uses bulk requests and is very fast, supports document transformation and\ncontrol over id and sub-index of documents.\n\n\n```python\nimport datetime\n\nclass MyExporter(Exporter):\n    INDEX_NAME = \"my-index-*\"\n    MAPPINGS = {\n        \"properties\": {\n            \"some_field\": {\"type\": \"text\"},\n            \"group\": {\"type\": \"keyword\"},\n            \"id\": {\"type\": \"keyword\"},\n            \"timestamp\": {\"type\": \"date\"},\n        }       \n    }   \n\n    # if each document has a unique id value we can use it\n    # as the elasticsearch id as well. That way we do not\n    # create documents twice when exporting them again.\n    # Their data just gets updated.\n    def get_document_id(self, es_data):\n        return es_data[\"id\"]\n    \n    # we can bucket documents into separate indices \n    def get_document_index(self, es_data):\n        return self.index_name().replace(\"*\", es_data[\"group\"])\n    \n    # here we can adjust or add some data before it gets exported.\n    # it's also possible to split the data into several documents\n    #   by yielding or returning a list\n    def transform_document(self, data):\n        data = data.copy()\n        data[\"timestamp\"] = datetime.datetime.now()\n        return data\n\nMyExporter().export_list(a_lot_of_objects)\n```\n\n\n\n\n    (1000, [])\n\n\n\nIf we are tired enough we can call:\n\n\n```python\nMyExporter().delete_index()\n```\n\n\n\n\n    True\n\n\n\nThis will actually delete all sub-indices because there's this wildcard `*` in the `INDEX_NAME`.\n\n**More examples can be found [here](examples).**\n\n\n### testing\n\nTo run the tests call:\n```shell script\npython test.py\n````\n\nTo include testing against a live elasticsearch:\n```shell script\npython test.py --live\n```\n\nTo change **localhost:9200** to something different\npass any arguments as json:\n```shell script\npython test.py --live --elasticsearch '{\"hosts\": [{\"host\": \"127.0.0.5\", \"port\": 1200}], \"http_auth\": [\"user\", \"password\"]}'\n```\n\nThe live tests will create new indices and immediately destroy them afterwards. \nThey are prefixed with **elastipy---unittest-**\n\nTo check the coverage of the tests add `-c` or `-m` flags.\n`-m` will add the missing line numbers to the summary. \n\n### development\n\nThe methods for **queries** and **aggregations** as well as the **query \nclasses** are auto-generated from [yaml files](definition). \nThey include all parameters, default values and documentation.\n\n\n#### Add a missing query or aggregation\n\n1. Create a yaml file with the name of it in one of the sub-directories \nin `definition/query` or `definition/aggregation`. \n    \n    The sub-directories in `query/` are just for tidiness and\n    follow the nesting in the sidebar of the official \n    [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html). \n    The three directories below `aggregation/` actually define the\n    aggregation type `bucket`, `metric` or `pipeline`. \n\n2. Create the python code via \n   ```shell script\n   # in project root\n   python generate_interfaces.py\n   ```\n   This will update the files:\n    - [elastipy/query/generated_classes.py](elastipy/query/generated_classes.py)\n    - [elastipy/query/generated_interface.py](elastipy/query/generated_interface.py)  \n    - [elastipy/aggregation/generated_interface.py](elastipy/aggregation/generated_interface.py)\n    \n   The sphinx documentation will collect the respective documentation \n   from these files.\n\n\n#### Update the example and tutorial notebooks\n\n1. Do some changes or add a new notebook (and keep main \n    `requirements.txt` up to date).\n\n2. Execute: \n   ```shell script\n   python run_doc_notebooks.py --execute\n   ``` \n   This will convert the notebooks to `.rst` files into the [docs/](docs/) directory.\n   \n   The [docs/quickref.ipynb](docs/quickref.ipynb) notebook will even be rendered\n   as markdown into this README.  \n    \n   The `-e`/`--execute` flag is required for proper doc building. For debugging\n   purposes it can be omitted in which case the current notebook state is\n   rendered. \n   \n3. Run\n   ```shell script\n   cd docs/\n   pip install -r requirements.txt\n   make clean \u0026\u0026 make html\n   ```\n   and inspect the results in \n   [docs/_build/html/index.html](docs/_build/html/index.html).\n\nBefore committing changes run \n```shell script\npip install pre-commit\npre-commit install\n```\nThis will install a pre-commit hook from \n[.pre-commit-config.yaml](.pre-commit-config.yaml) \nthat clears the output of all notebooks.\nSince the interesting ones are already rendered to the document pages, i just \nthink this is more tidy and releases one from cleaning up the execution state\nof notebooks by hand before committing.\n  \nGenerally, i'm stuck with *restructuredtext* for the docstrings although \nbesides the `:param:` syntax i find it simply repellent. \nIt still has the most supported toolchain it seems.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetzkolchose%2Felastipy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetzkolchose%2Felastipy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetzkolchose%2Felastipy/lists"}