{"id":13795512,"url":"https://github.com/druid-io/pydruid","last_synced_at":"2026-02-19T07:00:58.419Z","repository":{"id":10299336,"uuid":"12421065","full_name":"druid-io/pydruid","owner":"druid-io","description":"A Python connector for Druid","archived":false,"fork":false,"pushed_at":"2025-09-11T01:52:41.000Z","size":590,"stargazers_count":519,"open_issues_count":90,"forks_count":203,"subscribers_count":53,"default_branch":"master","last_synced_at":"2025-10-20T16:05:55.033Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/druid-io.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2013-08-28T00:52:10.000Z","updated_at":"2025-09-25T21:19:51.000Z","dependencies_parsed_at":"2025-09-10T07:19:58.504Z","dependency_job_id":"aa1c7211-8ae6-41d6-8f34-f0fa22f2e5eb","html_url":"https://github.com/druid-io/pydruid","commit_stats":{"total_commits":236,"total_committers":63,"mean_commits":3.746031746031746,"dds":0.7838983050847458,"last_synced_commit":"bd7b741a93c11733f928d649b9927448032e11f4"},"previous_names":[],"tags_count":37,"template":false,"template_full_name":null,"purl":"pkg:github/druid-io/pydruid","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druid-io%2Fpydruid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druid-io%2Fpydruid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druid-io%2Fpydruid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druid-io%2Fpydruid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/druid-io","download_url":"https://codeload.github.com/druid-io/pydruid/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/druid-io%2Fpydruid/sbom","scorecard":{"id":357100,"data":{"date":"2025-08-11","repo":{"name":"github.com/druid-io/pydruid","commit":"692bac55f8fae141cfcb85d3862f020dfd7aed4e"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":7,"reason":"Found 22/30 approved changesets -- score normalized to 7","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":9,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Warn: project license file does not contain an FSF or OSI license."],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 29 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"32 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-48 / GHSA-fj7x-q9j7-g6q6","Warn: Project is vulnerable to: GHSA-29gw-9793-fvw7","Warn: Project is vulnerable to: PYSEC-2022-12 / GHSA-pq7m-3gw7-gq5x","Warn: Project is vulnerable to: PYSEC-2020-92 / GHSA-hj5v-574p-mj7c","Warn: Project is vulnerable to: PYSEC-2022-42969","Warn: Project is vulnerable to: PYSEC-2021-140 / GHSA-9w8r-397f-prfh","Warn: Project is vulnerable to: PYSEC-2023-117 / GHSA-mrwq-x4v8-fh7p","Warn: Project is vulnerable to: PYSEC-2021-141 / GHSA-pq64-v7f5-gqh8","Warn: Project is vulnerable to: PYSEC-2021-142 / GHSA-8q59-q68h-6hv4","Warn: Project is vulnerable to: PYSEC-2024-187 / GHSA-rqc4-2hc7-8c8v","Warn: Project is vulnerable to: GHSA-jfmj-5v4g-7637","Warn: Project is vulnerable to: PYSEC-2022-42986 / GHSA-43fp-rhv2-5gv8","Warn: Project is vulnerable to: PYSEC-2023-135 / GHSA-xqr8-7jwr-rhp7","Warn: Project is vulnerable to: PYSEC-2024-60 / GHSA-jjg7-2v4v-x38h","Warn: Project is vulnerable to: PYSEC-2021-856 / GHSA-5545-2q6w-2gh6","Warn: Project is vulnerable to: GHSA-6p56-wp2h-9hxr","Warn: Project is vulnerable to: PYSEC-2021-857 / GHSA-f7c7-j99h-c22f","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: GHSA-753j-mpmx-qq6g","Warn: Project is vulnerable to: GHSA-7cx3-6m66-7c5m","Warn: Project is vulnerable to: GHSA-8w49-h785-mj3c","Warn: Project is vulnerable to: PYSEC-2023-75 / GHSA-hj3f-6gcp-jg8j","Warn: Project is vulnerable to: GHSA-qppv-j76h-2rpx","Warn: Project is vulnerable to: GHSA-w235-7p84-xx57","Warn: Project is vulnerable to: GHSA-34jh-p97f-mpxf","Warn: Project is vulnerable to: PYSEC-2023-212 / GHSA-g4mx-q9vg-27p4","Warn: Project is vulnerable to: GHSA-pq67-6m6q-mj2v","Warn: Project is vulnerable to: PYSEC-2021-108 / GHSA-q2q7-5pp4-w6pg","Warn: Project is vulnerable to: PYSEC-2023-192 / GHSA-v845-jxx5-vc9f"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T09:53:08.301Z","repository_id":10299336,"created_at":"2025-08-18T09:53:08.302Z","updated_at":"2025-08-18T09:53:08.302Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29605794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T06:47:36.664Z","status":"ssl_error","status_checked_at":"2026-02-19T06:45:47.551Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T23:00:58.164Z","updated_at":"2026-02-19T07:00:58.388Z","avatar_url":"https://github.com/druid-io.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# pydruid\n\npydruid exposes a simple API to create, execute, and analyze [Druid](http://druid.io/) queries. pydruid can parse query results into [Pandas](http://pandas.pydata.org/) DataFrame objects for subsequent data analysis -- this offers a tight integration between [Druid](http://druid.io/), the [SciPy](http://www.scipy.org/stackspec.html) stack (for scientific computing) and [scikit-learn](http://scikit-learn.org/stable/) (for machine learning). pydruid can export query results into TSV or JSON for further processing with your favorite tool, e.g., R, Julia, Matlab, Excel. It provides both synchronous and asynchronous clients.\n\nAdditionally, pydruid implements the [Python DB API 2.0](https://www.python.org/dev/peps/pep-0249/), a [SQLAlchemy dialect](http://docs.sqlalchemy.org/en/latest/dialects/), and a provides a command line interface to interact with Druid.\n\nTo install:\n```python\npip install pydruid\n# or, if you intend to use asynchronous client\npip install pydruid[async]\n# or, if you intend to export query results into pandas\npip install pydruid[pandas]\n# or, if you intend to do both\npip install pydruid[async, pandas]\n# or, if you want to use the SQLAlchemy engine\npip install pydruid[sqlalchemy]\n# or, if you want to use the CLI\npip install pydruid[cli]\n```\nDocumentation: https://pythonhosted.org/pydruid/.\n\n# examples\n\nThe following exampes show how to execute and analyze the results of three types of queries: timeseries, topN, and groupby. We will use these queries to ask simple questions about twitter's public data set.\n\n## timeseries\n\nWhat was the average tweet length, per day, surrounding the 2014 Sochi olympics?\n\n```python\nfrom pydruid.client import *\nfrom pylab import plt\n\nquery = PyDruid(druid_url_goes_here, 'druid/v2')\n\nts = query.timeseries(\n    datasource='twitterstream',\n    granularity='day',\n    intervals='2014-02-02/p4w',\n    aggregations={'length': doublesum('tweet_length'), 'count': doublesum('count')},\n    post_aggregations={'avg_tweet_length': (Field('length') / Field('count'))},\n    filter=Dimension('first_hashtag') == 'sochi2014'\n)\ndf = query.export_pandas()\ndf['timestamp'] = df['timestamp'].map(lambda x: x.split('T')[0])\ndf.plot(x='timestamp', y='avg_tweet_length', ylim=(80, 140), rot=20,\n        title='Sochi 2014')\nplt.ylabel('avg tweet length (chars)')\nplt.show()\n```\n\n![alt text](https://github.com/metamx/pydruid/raw/master/docs/figures/avg_tweet_length.png \"Avg. tweet length\")\n\n## topN\n\nWho were the top ten mentions (@user_name) during the 2014 Oscars?\n\n```python\ntop = query.topn(\n    datasource='twitterstream',\n    granularity='all',\n    intervals='2014-03-03/p1d',  # utc time of 2014 oscars\n    aggregations={'count': doublesum('count')},\n    dimension='user_mention_name',\n    filter=(Dimension('user_lang') == 'en') \u0026 (Dimension('first_hashtag') == 'oscars') \u0026\n           (Dimension('user_time_zone') == 'Pacific Time (US \u0026 Canada)') \u0026\n           ~(Dimension('user_mention_name') == 'No Mention'),\n    metric='count',\n    threshold=10\n)\n\ndf = query.export_pandas()\nprint df\n\n   count                 timestamp user_mention_name\n0   1303  2014-03-03T00:00:00.000Z      TheEllenShow\n1     44  2014-03-03T00:00:00.000Z        TheAcademy\n2     21  2014-03-03T00:00:00.000Z               MTV\n3     21  2014-03-03T00:00:00.000Z         peoplemag\n4     17  2014-03-03T00:00:00.000Z               THR\n5     16  2014-03-03T00:00:00.000Z      ItsQueenElsa\n6     16  2014-03-03T00:00:00.000Z           eonline\n7     15  2014-03-03T00:00:00.000Z       PerezHilton\n8     14  2014-03-03T00:00:00.000Z     realjohngreen\n9     12  2014-03-03T00:00:00.000Z       KevinSpacey\n\n```\n\n## groupby\n\nWhat does the social network of users replying to other users look like?\n\n```python\nfrom igraph import *\nfrom cairo import *\nfrom pandas import concat\n\ngroup = query.groupby(\n    datasource='twitterstream',\n    granularity='hour',\n    intervals='2013-10-04/pt12h',\n    dimensions=[\"user_name\", \"reply_to_name\"],\n    filter=(~(Dimension(\"reply_to_name\") == \"Not A Reply\")) \u0026\n           (Dimension(\"user_location\") == \"California\"),\n    aggregations={\"count\": doublesum(\"count\")}\n)\n\ndf = query.export_pandas()\n\n# map names to categorical variables with a lookup table\nnames = concat([df['user_name'], df['reply_to_name']]).unique()\nnameLookup = dict([pair[::-1] for pair in enumerate(names)])\ndf['user_name_lookup'] = df['user_name'].map(nameLookup.get)\ndf['reply_to_name_lookup'] = df['reply_to_name'].map(nameLookup.get)\n\n# create the graph with igraph\ng = Graph(len(names), directed=False)\nvertices = zip(df['user_name_lookup'], df['reply_to_name_lookup'])\ng.vs[\"name\"] = names\ng.add_edges(vertices)\nlayout = g.layout_fruchterman_reingold()\nplot(g, \"tweets.png\", layout=layout, vertex_size=2, bbox=(400, 400), margin=25, edge_width=1, vertex_color=\"blue\")\n```\n\n![alt text](https://github.com/metamx/pydruid/raw/master/docs/figures/twitter_graph.png \"Social Network\")\n\n# asynchronous client\n```pydruid.async_client.AsyncPyDruid``` implements an asynchronous client. To achieve that, it utilizes an asynchronous\nHTTP client from ```Tornado``` framework. The asynchronous client is suitable for use with async frameworks such as Tornado\nand provides much better performance at scale. It lets you serve multiple requests at the same time, without blocking on\nDruid executing your queries.\n\n## example\n```python\nfrom tornado import gen\nfrom pydruid.async_client import AsyncPyDruid\nfrom pydruid.utils.aggregators import longsum\nfrom pydruid.utils.filters import Dimension\n\nclient = AsyncPyDruid(url_to_druid_broker, 'druid/v2')\n\n@gen.coroutine\ndef your_asynchronous_method_serving_top10_mentions_for_day(day\n    top_mentions = yield client.topn(\n        datasource='twitterstream',\n        granularity='all',\n        intervals=\"%s/p1d\" % (day, ),\n        aggregations={'count': doublesum('count')},\n        dimension='user_mention_name',\n        filter=(Dimension('user_lang') == 'en') \u0026 (Dimension('first_hashtag') == 'oscars') \u0026\n               (Dimension('user_time_zone') == 'Pacific Time (US \u0026 Canada)') \u0026\n               ~(Dimension('user_mention_name') == 'No Mention'),\n        metric='count',\n        threshold=10)\n\n    # asynchronously return results\n    # can be simply ```return top_mentions``` in python 3.x\n    raise gen.Return(top_mentions)\n```\n\n\n# thetaSketches\nTheta sketch Post aggregators are built slightly differently to normal Post Aggregators, as they have different operators.\nNote: you must have the ```druid-datasketches``` extension loaded into your Druid cluster in order to use these.\nSee the [Druid datasketches](http://druid.io/docs/latest/development/extensions-core/datasketches-aggregators.html) documentation for details.\n\n```python\nfrom pydruid.client import *\nfrom pydruid.utils import aggregators\nfrom pydruid.utils import filters\nfrom pydruid.utils import postaggregator\n\nquery = PyDruid(url_to_druid_broker, 'druid/v2')\nts = query.groupby(\n    datasource='test_datasource',\n    granularity='all',\n    intervals='2016-09-01/P1M',\n    filter = ( filters.Dimension('product').in_(['product_A', 'product_B'])),\n    aggregations={\n        'product_A_users': aggregators.filtered(\n            filters.Dimension('product') == 'product_A',\n            aggregators.thetasketch('user_id')\n            ),\n        'product_B_users': aggregators.filtered(\n            filters.Dimension('product') == 'product_B',\n            aggregators.thetasketch('user_id')\n            )\n    },\n    post_aggregations={\n        'both_A_and_B': postaggregator.ThetaSketchEstimate(\n            postaggregator.ThetaSketch('product_A_users') \u0026 postaggregator.ThetaSketch('product_B_users')\n            )\n    }\n)\n```\n\n# DB API\n\n```python\nfrom pydruid.db import connect\n\nconn = connect(host='localhost', port=8082, path='/druid/v2/sql/', scheme='http')\ncurs = conn.cursor()\ncurs.execute(\"\"\"\n    SELECT place,\n           CAST(REGEXP_EXTRACT(place, '(.*),', 1) AS FLOAT) AS lat,\n           CAST(REGEXP_EXTRACT(place, ',(.*)', 1) AS FLOAT) AS lon\n      FROM places\n     LIMIT 10\n\"\"\")\nfor row in curs:\n    print(row)\n```\n\n# SQLAlchemy\n\n```python\nfrom sqlalchemy import *\nfrom sqlalchemy.engine import create_engine\nfrom sqlalchemy.schema import *\n\nengine = create_engine('druid://localhost:8082/druid/v2/sql/')  # uses HTTP by default :(\n# engine = create_engine('druid+http://localhost:8082/druid/v2/sql/')\n# engine = create_engine('druid+https://localhost:8082/druid/v2/sql/')\n\nplaces = Table('places', MetaData(bind=engine), autoload=True)\nprint(select([func.count('*')], from_obj=places).scalar())\n```\n\n\n## Column headers\n\nIn version 0.13.0 Druid SQL added support for including the column names in the\nresponse which can be requested via the \"header\" field in the request. This\nhelps to ensure that the cursor description is defined (which is a requirement\nfor SQLAlchemy query statements) regardless on whether the result set contains\nany rows. Historically this was problematic for result sets which contained no\nrows at one could not infer the expected column names.\n\nEnabling the header can be configured via the SQLAlchemy URI by using the query\nparameter, i.e.,\n\n```python\nengine = create_engine('druid://localhost:8082/druid/v2/sql?header=true')\n```\n\nNote the current default is `false` to ensure backwards compatibility but should\nbe set to `true` for Druid versions \u003e= 0.13.0.\n\n\n# Command line\n\n```bash\n$ pydruid http://localhost:8082/druid/v2/sql/\n\u003e SELECT COUNT(*) AS cnt FROM places\n  cnt\n-----\n12345\n\u003e SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES;\nTABLE_NAME\n----------\ntest_table\nCOLUMNS\nSCHEMATA\nTABLES\n\u003e BYE;\nGoodBye!\n```\n\n# Contributing\n\nContributions are welcomed of course. We like to use `black` and `flake8`.\n\n```bash\npip install -r requirements-dev.txt  # installs useful dev deps\npre-commit install  # installs useful commit hooks\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdruid-io%2Fpydruid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdruid-io%2Fpydruid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdruid-io%2Fpydruid/lists"}