{"id":27627214,"url":"https://github.com/treasure-data/td-client-python","last_synced_at":"2026-01-29T06:13:04.161Z","repository":{"id":25098053,"uuid":"28519136","full_name":"treasure-data/td-client-python","owner":"treasure-data","description":"Treasure Data API library for Python","archived":false,"fork":false,"pushed_at":"2025-12-03T02:48:59.000Z","size":925,"stargazers_count":48,"open_issues_count":0,"forks_count":23,"subscribers_count":79,"default_branch":"master","last_synced_at":"2025-12-21T07:24:53.529Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/treasure-data.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2014-12-26T19:21:47.000Z","updated_at":"2025-11-21T17:37:24.000Z","dependencies_parsed_at":"2024-01-15T03:47:42.869Z","dependency_job_id":"986a0445-20e1-4f40-a914-45bb328722d9","html_url":"https://github.com/treasure-data/td-client-python","commit_stats":null,"previous_names":[],"tags_count":49,"template":false,"template_full_name":null,"purl":"pkg:github/treasure-data/td-client-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Ftd-client-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Ftd-client-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Ftd-client-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Ftd-client-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/treasure-data","download_url":"https://codeload.github.com/treasure-data/td-client-python/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treasure-data%2Ftd-client-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28865615,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T05:56:06.453Z","status":"ssl_error","status_checked_at":"2026-01-29T05:55:57.668Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-23T13:53:40.926Z","updated_at":"2026-01-29T06:13:04.145Z","avatar_url":"https://github.com/treasure-data.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\nTreasure Data API library for Python\n====================================\n\n\n.. image:: https://github.com/treasure-data/td-client-python/workflows/Python%20testing/badge.svg\n   :target: https://github.com/treasure-data/td-client-python/actions\n   :alt: Build Status on GitHub Actions\n\n\n.. image:: https://badge.fury.io/py/td-client.svg\n   :target: http://badge.fury.io/py/td-client\n   :alt: PyPI version\n\n\nTreasure Data API library for Python\n\nRequirements\n------------\n\n``td-client`` supports the following versions of Python.\n\n\n* Python 3.10+\n* PyPy\n\nInstall\n-------\n\nYou can install the releases from `PyPI \u003chttps://pypi.python.org/\u003e`_.\n\n.. code-block:: sh\n\n   $ pip install td-client\n\nIt'd be better to install `certifi \u003chttps://pypi.python.org/pypi/certifi\u003e`_ to enable SSL certificate verification.\n\n.. code-block:: sh\n\n   $ pip install certifi\n\nExamples\n--------\n\nPlease see also the examples at `Treasure Data Documentation \u003chttps://docs.treasuredata.com/display/public/PD/Python+Client+for+REST+API\u003e`_.\n\nThe td-client documentation is hosted at https://tdclient.readthedocs.io/,\nor you can go directly to the\n`API documentation \u003chttps://tdclient.readthedocs.io/en/latest/api/index.html\u003e`_.\n\nFor information on the parameters that may be used when reading particular\ntypes of data, see `File import parameters`_.\n\n.. _`file import parameters`:\n   https://tdclient.readthedocs.io/en/latest/file_import_parameters.html\n\nListing jobs\n^^^^^^^^^^^^\n\nTreasure Data API key will be read from environment variable ``TD_API_KEY``\\ , if none is given via ``apikey=`` argument passed to ``tdclient.Client``.\n\nTreasure Data API endpoint ``https://api.treasuredata.com`` is used by default. You can override this with environment variable ``TD_API_SERVER``\\ , which in turn can be overridden via ``endpoint=`` argument passed to ``tdclient.Client``. List of available Treasure Data sites and corresponding API endpoints can be found `here \u003chttps://api-docs.treasuredata.com/en/overview/aboutendpoints/#treasure-data-api-baseurls\u003e`_.\n\n.. code-block:: python\n\n   import tdclient\n\n   with tdclient.Client() as td:\n       for job in td.jobs():\n           print(job.job_id)\n\nRunning jobs\n^^^^^^^^^^^^\n\nRunning jobs on Treasure Data.\n\n.. code-block:: python\n\n   import tdclient\n\n   with tdclient.Client() as td:\n       job = td.query(\"sample_datasets\", \"SELECT COUNT(1) FROM www_access\", type=\"hive\")\n       job.wait()\n       for row in job.result():\n           print(repr(row))\n\nRunning jobs via DBAPI2\n^^^^^^^^^^^^^^^^^^^^^^^\n\ntd-client-python implements `PEP 0249 \u003chttps://www.python.org/dev/peps/pep-0249/\u003e`_ Python Database API v2.0.\nYou can use td-client-python with external libraries which supports Database API such like `pandas \u003chttp://pandas.pydata.org/\u003e`_.\n\n.. code-block:: python\n\n   import pandas\n   import tdclient\n\n   def on_waiting(cursor):\n       print(cursor.job_status())\n\n   with tdclient.connect(db=\"sample_datasets\", type=\"presto\", wait_callback=on_waiting) as td:\n       data = pandas.read_sql(\"SELECT symbol, COUNT(1) AS c FROM nasdaq GROUP BY symbol\", td)\n       print(repr(data))\n\nWe offer another package for pandas named `pytd \u003chttps://github.com/treasure-data/pytd\u003e`_ with some advanced features.\nYou may prefer it if you need to do complicated things, such like exporting result data to Treasure Data, printing job's\nprogress during long execution, etc.\n\nImporting data\n^^^^^^^^^^^^^^\n\nImporting data into Treasure Data in streaming manner, as similar as `fluentd \u003chttp://www.fluentd.org/\u003e`_ is doing.\n\n.. code-block:: python\n\n   import sys\n   import tdclient\n\n   with tdclient.Client() as td:\n       for file_name in sys.argv[:1]:\n           td.import_file(\"mydb\", \"mytbl\", \"csv\", file_name)\n\n\n.. Warning::\n   Importing data in streaming manner requires certain amount of time to be ready to query since schema update will be\n   executed with delay.\n\nBulk import\n^^^^^^^^^^^\n\nImporting data into Treasure Data in batch manner.\n\n.. code-block:: python\n\n   import sys\n   import tdclient\n   import uuid\n   import warnings\n\n   if len(sys.argv) \u003c= 1:\n       sys.exit(0)\n\n   with tdclient.Client() as td:\n       session_name = \"session-{}\".format(uuid.uuid1())\n       bulk_import = td.create_bulk_import(session_name, \"mydb\", \"mytbl\")\n       try:\n           for file_name in sys.argv[1:]:\n               part_name = \"part-{}\".format(file_name)\n               bulk_import.upload_file(part_name, \"json\", file_name)\n           bulk_import.freeze()\n       except:\n           bulk_import.delete()\n           raise\n       bulk_import.perform(wait=True)\n       if 0 \u003c bulk_import.error_records:\n           warnings.warn(\"detected {} error records.\".format(bulk_import.error_records))\n       if 0 \u003c bulk_import.valid_records:\n           print(\"imported {} records.\".format(bulk_import.valid_records))\n       else:\n           raise(RuntimeError(\"no records have been imported: {}\".format(bulk_import.name)))\n       bulk_import.commit(wait=True)\n       bulk_import.delete()\n\n\nIf you want to import data as `msgpack \u003chttps://msgpack.org/\u003e`_ format, you can write as follows:\n\n.. code-block:: python\n\n   import io\n   import time\n   import uuid\n   import warnings\n\n   import tdclient\n\n   t1 = int(time.time())\n   l1 = [{\"a\": 1, \"b\": 2, \"time\": t1}, {\"a\": 3, \"b\": 9, \"time\": t1}]\n\n   with tdclient.Client() as td:\n       session_name = \"session-{}\".format(uuid.uuid1())\n       bulk_import = td.create_bulk_import(session_name, \"mydb\", \"mytbl\")\n       try:\n           _bytes = tdclient.util.create_msgpack(l1)\n           bulk_import.upload_file(\"part\", \"msgpack\", io.BytesIO(_bytes))\n           bulk_import.freeze()\n       except:\n           bulk_import.delete()\n           raise\n       bulk_import.perform(wait=True)\n       # same as the above example\n\n\nChanging how CSV and TSV columns are read\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe ``td-client`` package will generally make sensible choices on how to read\nthe columns in CSV and TSV data, but sometimes the user needs to override the\ndefault mechanism. This can be done using the optional `file import\nparameters`_ ``dtypes`` and ``converters``.\n\nFor instance, consider CSV data that starts with the following records::\n\n  time,col1,col2,col3\n  1575454204,a,0001,a;b;c\n  1575454204,b,0002,d;e;f\n\nIf that data is read using the defaults, it will produce values that look\nlike:\n\n.. code:: python\n\n  1575454204, \"a\", 1, \"a;b;c\"\n  1575454204, \"b\", 2, \"d;e;f\"\n\nthat is, an integer, a string, an integer and another string.\n\nIf the user wants to keep the leading zeroes in ``col2``, then they can\nspecify the column datatype as string. For instance, using\n``bulk_import.upload_file`` to read data from ``input_data``:\n\n.. code:: python\n\n    bulk_import.upload_file(\n        \"part\", \"msgpack\", input_data,\n        dtypes={\"col2\": \"str\"},\n    )\n\nwhich would produce:\n\n.. code:: python\n\n  1575454204, \"a\", \"0001\", \"a;b;c\"\n  1575454204, \"b\", \"0002\", \"d;e;f\"\n\nIf they also wanted to treat ``col3`` as a sequence of strings, separated by\nsemicolons, then they could specify a function to process ``col3``:\n\n.. code:: python\n\n    bulk_import.upload_file(\n        \"part\", \"msgpack\", input_data,\n        dtypes={\"col2\": \"str\"},\n        converters={\"col3\", lambda x: x.split(\";\")},\n    )\n\nwhich would produce:\n\n.. code:: python\n\n  1575454204, \"a\", \"0001\", [\"a\", \"b\", \"c\"]\n  1575454204, \"b\", \"0002\", [\"d\", \"e\", \"f\"]\n\nType Hints\n----------\n\ntd-client-python includes comprehensive type hints (PEP 484) for improved development experience with static type checkers like mypy and pyright. Type hints are available for all public APIs.\n\n**Features:**\n\n\n* Fully typed public API with precise type annotations\n* ``py.typed`` marker file for PEP 561 compliance\n* Type aliases in ``tdclient.types`` for common patterns\n* Support for type checking with mypy, pyright, and other tools\n\n**Example with type checking:**\n\n.. code-block:: python\n\n   import tdclient\n\n   # Type checkers will understand the types\n   with tdclient.Client(apikey=\"your_api_key\") as client:\n       # client is inferred as tdclient.Client\n       job = client.query(\"sample_db\", \"SELECT COUNT(1) FROM table\", type=\"presto\")\n       # job is inferred as tdclient.models.Job\n       job.wait()\n       for row in job.result():\n           # row is inferred as dict[str, Any]\n           print(row)\n\n**Using type aliases:**\n\n.. code-block:: python\n\n   from tdclient.types import QueryEngineType, Priority\n\n   def run_query(engine: QueryEngineType, priority: Priority) -\u003e None:\n       with tdclient.Client() as client:\n           job = client.query(\"mydb\", \"SELECT 1\", type=engine, priority=priority)\n           job.wait()\n\nDevelopment\n-----------\n\nRunning tests\n^^^^^^^^^^^^^\n\nInstall the project dependencies with `uv \u003chttps://docs.astral.sh/uv/\u003e`_ (runtime\nand test extras) and execute pytest via ``uv run``.\n\n.. code-block:: sh\n\n    $ uv sync --extra test\n    $ uv run pytest tdclient/test\n\nTo run the coverage suite locally, use:\n\n.. code-block:: sh\n\n    $ uv run coverage run --source=tdclient -m pytest tdclient/test\n    $ uv run coverage report\n\nLinting and type checking\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nInstall the development extras and invoke ``ruff`` and ``pyright`` using\n``uv run``.\n\n.. code-block:: sh\n\n    $ uv sync --dev\n    $ uv run ruff format tdclient --diff --exit-non-zero-on-fix\n    $ uv run ruff check tdclient\n    $ uv run pyright tdclient\n\nRunning tests (tox)\n^^^^^^^^^^^^^^^^^^^\n\nYou can run tests against all supported Python versions with ``tox``. I'd\nrecommend you to install `pyenv \u003chttps://github.com/yyuu/pyenv\u003e`_ to manage\nadditional interpreters.\n\n.. code-block:: sh\n\n   $ pyenv shell system\n   $ for version in $(cat .python-version); do [ -d \"$(pyenv root)/versions/${version}\" ] || pyenv install \"${version}\"; done\n   $ pyenv shell --unset\n\nInstall the development extras (which include ``tox``) with ``uv``.\n\n.. code-block:: sh\n\n    $ uv sync --dev\n\nThen, run ``tox`` via ``uv``.\n\n.. code-block:: sh\n\n    $ uv run tox\n\nRelease\n^^^^^^^\n\n1. Update version `x.x.x` in `pyproject.toml`.\n2. Create a PR with `release-x.x.x` branch. Request and merge the PR.\n3. Create and push a tag `x.x.x` on `release-x.x.x` merge commit.\n4. Create a Release on GitHub will publish new version to PyPI.\n\nManual release\n~~~~~~~~~~~~~~\n\nIf you want to release manually, you can upload by twine.\n\n.. code-block:: sh\n\n   $ python -m build\n   $ twine upload dist/*\n\nLicense\n-------\n\nApache Software License, Version 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Ftd-client-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftreasure-data%2Ftd-client-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreasure-data%2Ftd-client-python/lists"}