{"id":13783310,"url":"https://github.com/onesuper/pandasticsearch","last_synced_at":"2025-05-11T18:31:37.093Z","repository":{"id":46248278,"uuid":"72193052","full_name":"onesuper/pandasticsearch","owner":"onesuper","description":"An Elasticsearch client exposing DataFrame API","archived":false,"fork":false,"pushed_at":"2023-04-01T09:13:53.000Z","size":197,"stargazers_count":285,"open_issues_count":9,"forks_count":45,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-11-17T19:43:06.458Z","etag":null,"topics":["dataframe-api","elasticsearch-client"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onesuper.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-10-28T09:17:57.000Z","updated_at":"2024-07-01T09:40:24.000Z","dependencies_parsed_at":"2024-01-15T09:57:47.027Z","dependency_job_id":"511164dc-ad8d-4192-9366-76894fd1f5d4","html_url":"https://github.com/onesuper/pandasticsearch","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onesuper%2Fpandasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onesuper%2Fpandasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onesuper%2Fpandasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onesuper%2Fpandasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onesuper","download_url":"https://codeload.github.com/onesuper/pandasticsearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253613316,"owners_count":21936253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataframe-api","elasticsearch-client"],"created_at":"2024-08-03T19:00:18.584Z","updated_at":"2025-05-11T18:31:36.737Z","avatar_url":"https://github.com/onesuper.png","language":"Python","funding_links":[],"categories":["Elasticsearch plugins","Python"],"sub_categories":["Integrations and SQL support"],"readme":"## Pandasticsearch\n\n[![Build Status](https://travis-ci.org/onesuper/pandasticsearch.svg?branch=master)](https://travis-ci.org/onesuper/pandasticsearch) [![PyPI](https://img.shields.io/pypi/v/pandasticsearch.svg)](https://pypi.python.org/pypi/pandasticsearch)\n\n\nPandasticsearch is an Elasticsearch client for data-analysis purpose.\nIt provides table-like access to Elasticsearch documents, similar\nto the Python Pandas library and R DataFrames.\n\nTo install:\n\n```\npip install pandasticsearch\n# if you intent to export Pandas DataFrame \npip install pandasticsearch[pandas]\n```\n\n  Elasticsearch is skilled in real-time indexing, search and data-analysis.\n  Pandasticsearch can convert the analysis results (e.g. multi-level nested aggregation)\n  into [Pandas](http://pandas.pydata.org) DataFrame objects for subsequent data analysis.\n  \n\nCheckout the API doc: [http://pandasticsearch.readthedocs.io/en/latest/](http://pandasticsearch.readthedocs.io/en/latest/).\n\n## Usage\n\n### DataFrame API\n\nA `DataFrame` object accesses Elasticsearch data with high level operations.\nIt is type-safe, easy-to-use and Pandas-flavored.\n\n```python\n# Create a DataFrame object\nfrom pandasticsearch import DataFrame\ndf = DataFrame.from_es(url='http://localhost:9200', index='people', username='abc', password='abc')\n\n# Print the schema(mapping) of the index\ndf.print_schema()\n# company\n# |-- employee\n#   |-- name: {'index': 'not_analyzed', 'type': 'string'}\n#   |-- age: {'type': 'integer'}\n#   |-- gender: {'index': 'not_analyzed', 'type': 'string'}\n\n# Inspect the columns\ndf.columns\n#['name', 'age', 'gender']\n\n# Denote a column\ndf.name\n# Column('name')\ndf['age']\n# Column('age')\n\n# Projection\ndf.filter(df.age \u003c 25).select('name', 'age').collect()\n# [Row(age=12,name='Alice'), Row(age=11,name='Bob'), Row(age=13,name='Leo')]\n\n# Print the rows into console\ndf.filter(df.age \u003c 25).select('name').show(3)\n# +------+\n# | name |\n# +------+\n# | Alice|\n# | Bob  |\n# | Leo  |\n# +------+\n\n# Convert to Pandas object for subsequent analysis\ndf[df.gender == 'male'].agg(df.age.avg).to_pandas()\n#    avg(age)\n# 0        12\n\n\n# Dump all your dataset to Pandas DataFrame in memory for subsequent analysis\ndf.to_pandas()\n# ...\n\n# Limit your data amount, if your dataset is too large\ndf.limit(1000).to_pandas()\n# ...\n\n\n# Translate the DataFrame to an ES query (dictionary)\ndf[df.gender == 'male'].agg(df.age.avg).to_dict()\n# {'query': {'filtered': {'filter': {'term': {'gender': 'male'}}}}, 'aggregations': {'avg(birthYear)':\n# {'avg': {'field': 'birthYear'}}}, 'size': 0}\n\n```\n\n\n\n### Filter\n\n```python\n# Filter by a boolean condition\ndf.filter(df.age \u003c 13).collect()\n# [Row(age=12,gender='female',name='Alice'), Row(age=11,gender='male',name='Bob')]\n\n# Filter by a set of boolean conditions (by \u0026)\ndf.filter((df.age \u003c 13) \u0026 (df.gender == 'male')).collect()\n# Row(age=11,gender='male',name='Bob')]\n\n# Filter by a set of boolean conditions (by chaining)\ndf.filter(df.age \u003c 13).filter(df.gender == 'male').collect()\n# Row(age=11,gender='male',name='Bob')]\n\n# Filter by a wildcard (sql `like`)\ndf.filter(df.name.like('A*')).collect()\n# [Row(age=12,gender='female',name='Alice')]\n\n# Filter by a regular expression (sql `rlike`)\ndf.filter(df.name.rlike('A.l.e')).collect()\n# [Row(age=12,gender='female',name='Alice')]\n\n# Filter by a prefixed string pattern\ndf.filter(df.name.startswith('Al')).collect()\n# [Row(age=12,gender='female',name='Alice')]\n\n# Filter by a script\ndf.filter('2016 - doc[\"age\"].value \u003e 1995').collect()\n# [Row(age=12,name='Alice'), Row(age=13,name='Leo')]\n```\n\n\n### Aggregation\n```python\n# Aggregation\ndf[df.gender == 'male'].agg(df.age.avg).collect()\n# [Row(avg(age)=12)]\n\n# Metric alias\ndf[df.gender == 'male'].agg(df.age.avg.alias('avg_age')).collect()\n# [Row(avg_age=12)]\n\n# Groupby only (will give the `doc_count`)\ndf.groupby('gender').collect()\n# [Row(doc_count=1), Row(doc_count=2)]\n\n# Groupby and then aggregate metric\ndf.groupby('gender').agg(df.age.max).collect()\n# [Row(doc_count=1, max(age)=12), Row(doc_count=2, max(age)=13)]\n\n# Groupby and then aggregate multiple metrics(max and value_count)\ndf.groupby('gender').agg(df.age.value_count, df.age.max,).collect()\n# [Row(value_count(age)=1, max(age)=12), Row(value_count(age)=2, max(age)=13)]\n\n# Group by a set of ranges\ndf.groupby(df.age.ranges([10,12,14])).to_pandas()\n#                   doc_count\n# range(10,12,14)\n# 10.0-12.0                 2\n# 12.0-14.0                 1\n\n# Advanced ES aggregation\ndf.groupby(df.gender).agg(df.age.stats).to_pandas()\ndf.agg(df.age.extended_stats).to_pandas()\ndf.agg(df.age.percentiles).to_pandas()\ndf.groupby(df.date.date_interval('1d')).to_pandas()\n\n# Customized aggregation terms\ndf.groupby(df.age.terms(size=5, include=[1, 2, 3]))\n```\n\n### Sort\n```python\n# Sort\ndf.sort(df.age.asc).select('name', 'age').collect()\n# [Row(age=11,name='Bob'), Row(age=12,name='Alice'), Row(age=13,name='Leo')]\n\n# Sort by a script\ndf.sort('doc[\"age\"].value * 2').collect()\n# [Row(age=11,name='Bob'), Row(age=12,name='Alice'), Row(age=13,name='Leo')]\n```\n\n## Use with Another Python Client\n\nPandasticsearch can also be used with another full featured Python client:\n\n* [elasticsearch-py](https://github.com/elastic/elasticsearch-py) (Official)\n* [Elasticsearch-SQL](https://github.com/NLPchina/elasticsearch-sql)\n* [pyelasticsearch](https://github.com/pyelasticsearch/pyelasticsearch)\n* [pyes](https://github.com/aparo/pyes)\n\n\n### Build query\n\n```Python\nfrom pandasticsearch import DataFrame\nbody = df[df['gender'] == 'male'].agg(df['age'].avg).to_dict()\n \nfrom elasticsearch import Elasticsearch\nresult_dict = es.search(index=\"recruit\", body=body)\n```\n\n### Parse result\n\n```python\nfrom elasticsearch import Elasticsearch\nes = Elasticsearch('http://localhost:9200')\nresult_dict = es.search(index=\"recruit\", body={\"query\": {\"match_all\": {}}})\n\nfrom pandasticsearch import Select\npandas_df = Select.from_dict(result_dict).to_pandas()\n```\n\n\n## Compatibility\n\nAn integer argument `compat`  needs to be passed to `from_es` to resolve compatibility issues (default 2):\n\n### 5.0\n\n```\ndf = DataFrame.from_es(url='http://localhost:9200', index='people', doc_type='mapping_name', compat=5)\n```\n\nFor ES version under 7.0, a `doc_type` must be given to specify index mappings (it is deprecated in 7.0).\n\n\n### 7.0\n\n\n```\ndf = DataFrame.from_es(url='http://localhost:9200', index='people', compat=7)\n```\n\n\n## Related Articles\n\n* [Spark and Elasticsearch for real-time data analysis](https://web.archive.org/web/20150911151523/https://spark-summit.org/2015-east/wp-content/uploads/2015/03/SSE15-35-Leau.pdf)\n\n\n## LICENSE\n \nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonesuper%2Fpandasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonesuper%2Fpandasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonesuper%2Fpandasticsearch/lists"}