{"id":13534317,"url":"https://github.com/aws/aws-sdk-pandas","last_synced_at":"2026-05-08T01:05:16.070Z","repository":{"id":37445260,"uuid":"172615836","full_name":"aws/aws-sdk-pandas","owner":"aws","description":"pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).","archived":false,"fork":false,"pushed_at":"2025-05-07T10:48:05.000Z","size":16869,"stargazers_count":4016,"open_issues_count":36,"forks_count":706,"subscribers_count":58,"default_branch":"main","last_synced_at":"2025-05-12T02:43:17.910Z","etag":null,"topics":["amazon-athena","amazon-sagemaker-notebook","apache-arrow","apache-parquet","athena","aws","aws-glue","aws-lambda","data-engineering","data-science","emr","etl","glue-catalog","lambda","modin","mysql","pandas","python","ray","redshift"],"latest_commit_sha":null,"homepage":"https://aws-sdk-pandas.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-02-26T01:39:02.000Z","updated_at":"2025-05-11T20:08:45.000Z","dependencies_parsed_at":"2023-12-05T12:28:49.838Z","dependency_job_id":"1a91c9ce-6cd0-4441-bb1c-34c4647515a3","html_url":"https://github.com/aws/aws-sdk-pandas","commit_stats":{"total_commits":2320,"total_committers":168,"mean_commits":13.80952380952381,"dds":0.6086206896551725,"last_synced_commit":"3ded4c2897a8eb392bdcd300ac37b64f609645a3"},"previous_names":["awslabs/aws-data-wrangler"],"tags_count":127,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-sdk-pandas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-sdk-pandas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-sdk-pandas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-sdk-pandas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws","download_url":"https://codeload.github.com/aws/aws-sdk-pandas/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253672675,"owners_count":21945479,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-athena","amazon-sagemaker-notebook","apache-arrow","apache-parquet","athena","aws","aws-glue","aws-lambda","data-engineering","data-science","emr","etl","glue-catalog","lambda","modin","mysql","pandas","python","ray","redshift"],"created_at":"2024-08-01T07:01:30.334Z","updated_at":"2026-05-08T01:05:16.058Z","avatar_url":"https://github.com/aws.png","language":"Python","funding_links":[],"categories":["HarmonyOS","Python","Integrations","数据分析","Database Clients","科学计算和数据分析","Data Analysis"],"sub_categories":["Windows Manager","[Pandas](https://pandas.pydata.org/)","Drone Frames"],"readme":"# AWS SDK for pandas (awswrangler)\n\n*Pandas on AWS*\n\nEasy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).\n\n![AWS SDK for pandas](https://github.com/aws/aws-sdk-pandas/blob/main/docs/source/_static/logo2.png?raw=true \"AWS SDK for pandas\")\n![tracker](https://d3tiqpr4kkkomd.cloudfront.net/img/pixel.png?asset=GVOYN2BOOQ573LTVIHEW)\n\n\u003e An [AWS Professional Service](https://aws.amazon.com/professional-services/) open source initiative | aws-proserve-opensource@amazon.com\n\n[![PyPi](https://img.shields.io/pypi/v/awswrangler)](https://pypi.org/project/awswrangler/)\n[![Conda](https://img.shields.io/conda/vn/conda-forge/awswrangler)](https://anaconda.org/conda-forge/awswrangler)\n[![Python Version](https://img.shields.io/pypi/pyversions/awswrangler.svg)](https://pypi.org/project/awswrangler/)\n[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)\n![Static Checking](https://github.com/aws/aws-sdk-pandas/workflows/Static%20Checking/badge.svg?branch=main)\n[![Documentation Status](https://readthedocs.org/projects/aws-sdk-pandas/badge/?version=latest)](https://aws-sdk-pandas.readthedocs.io/?badge=latest)\n\n| Source | Downloads | Installation Command |\n|--------|-----------|----------------------|\n| **[PyPi](https://pypi.org/project/awswrangler/)**  | [![PyPI Downloads](https://img.shields.io/pypi/dm/awswrangler)](https://pypi.org/project/awswrangler/) | `pip install awswrangler` |\n| **[Conda](https://anaconda.org/conda-forge/awswrangler)** | [![Conda Downloads](https://img.shields.io/conda/dn/conda-forge/awswrangler.svg)](https://anaconda.org/conda-forge/awswrangler) | `conda install -c conda-forge awswrangler` |\n\n\u003e ⚠️ **Starting version 3.0, optional modules must be installed explicitly:**\u003cbr\u003e\n➡️`pip install 'awswrangler[redshift]'`\n\n## Table of contents\n\n- [Quick Start](#quick-start)\n- [At Scale](#at-scale)\n- [Read The Docs](#read-the-docs)\n- [Getting Help](#getting-help)\n- [Logging](#logging)\n\n## Quick Start\n\nInstallation command: `pip install awswrangler`\n\n\u003e ⚠️ **Starting version 3.0, optional modules must be installed explicitly:**\u003cbr\u003e\n➡️`pip install 'awswrangler[redshift]'`\n\n```py3\nimport awswrangler as wr\nimport pandas as pd\nfrom datetime import datetime\n\ndf = pd.DataFrame({\"id\": [1, 2], \"value\": [\"foo\", \"boo\"]})\n\n# Storing data on Data Lake\nwr.s3.to_parquet(\n    df=df,\n    path=\"s3://bucket/dataset/\",\n    dataset=True,\n    database=\"my_db\",\n    table=\"my_table\"\n)\n\n# Retrieving the data directly from Amazon S3\ndf = wr.s3.read_parquet(\"s3://bucket/dataset/\", dataset=True)\n\n# Retrieving the data from Amazon Athena\ndf = wr.athena.read_sql_query(\"SELECT * FROM my_table\", database=\"my_db\")\n\n# Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrum\ncon = wr.redshift.connect(\"my-glue-connection\")\ndf = wr.redshift.read_sql_query(\"SELECT * FROM external_schema.my_table\", con=con)\ncon.close()\n\n# Amazon Timestream Write\ndf = pd.DataFrame({\n    \"time\": [datetime.now(), datetime.now()],   \n    \"my_dimension\": [\"foo\", \"boo\"],\n    \"measure\": [1.0, 1.1],\n})\nrejected_records = wr.timestream.write(df,\n    database=\"sampleDB\",\n    table=\"sampleTable\",\n    time_col=\"time\",\n    measure_col=\"measure\",\n    dimensions_cols=[\"my_dimension\"],\n)\n\n# Amazon Timestream Query\nwr.timestream.query(\"\"\"\nSELECT time, measure_value::double, my_dimension\nFROM \"sampleDB\".\"sampleTable\" ORDER BY time DESC LIMIT 3\n\"\"\")\n\n```\n\n## At scale\nAWS SDK for pandas can also run your workflows at scale by leveraging [Modin](https://modin.readthedocs.io/en/stable/) and [Ray](https://www.ray.io/). Both projects aim to speed up data workloads by distributing processing over a cluster of workers.\n\nRead our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/scale.html) or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to learn more.\n\n## [Read The Docs](https://aws-sdk-pandas.readthedocs.io/)\n\n- [**What is AWS SDK for pandas?**](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/about.html)\n- [**Install**](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html)\n  - [PyPi (pip)](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#pypi-pip)\n  - [Conda](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#conda)\n  - [AWS Lambda Layer](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#aws-lambda-layer)\n  - [AWS Glue Python Shell Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#aws-glue-python-shell-jobs)\n  - [AWS Glue PySpark Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#aws-glue-pyspark-jobs)\n  - [Amazon SageMaker Notebook](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#amazon-sagemaker-notebook)\n  - [Amazon SageMaker Notebook Lifecycle](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#amazon-sagemaker-notebook-lifecycle)\n  - [EMR](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#emr)\n  - [From source](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/install.html#from-source)\n- [**At scale**](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/scale.html)\n  - [Getting Started](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/scale.html#getting-started)\n  - [Supported APIs](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/scale.html#supported-apis)\n  - [Resources](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/scale.html#resources)\n- [**Tutorials**](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials)\n  - [001 - Introduction](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/001%20-%20Introduction.ipynb)\n  - [002 - Sessions](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/002%20-%20Sessions.ipynb)\n  - [003 - Amazon S3](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/003%20-%20Amazon%20S3.ipynb)\n  - [004 - Parquet Datasets](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/004%20-%20Parquet%20Datasets.ipynb)\n  - [005 - Glue Catalog](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/005%20-%20Glue%20Catalog.ipynb)\n  - [006 - Amazon Athena](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/006%20-%20Amazon%20Athena.ipynb)\n  - [007 - Databases (Redshift, MySQL, PostgreSQL, SQL Server and Oracle)](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/007%20-%20Redshift%2C%20MySQL%2C%20PostgreSQL%2C%20SQL%20Server%2C%20Oracle.ipynb)\n  - [008 - Redshift - Copy \u0026 Unload.ipynb](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/008%20-%20Redshift%20-%20Copy%20%26%20Unload.ipynb)\n  - [009 - Redshift - Append, Overwrite and Upsert](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/009%20-%20Redshift%20-%20Append%2C%20Overwrite%2C%20Upsert.ipynb)\n  - [010 - Parquet Crawler](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/010%20-%20Parquet%20Crawler.ipynb)\n  - [011 - CSV Datasets](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/011%20-%20CSV%20Datasets.ipynb)\n  - [012 - CSV Crawler](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/012%20-%20CSV%20Crawler.ipynb)\n  - [013 - Merging Datasets on S3](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/013%20-%20Merging%20Datasets%20on%20S3.ipynb)\n  - [014 - Schema Evolution](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/014%20-%20Schema%20Evolution.ipynb)\n  - [015 - EMR](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/015%20-%20EMR.ipynb)\n  - [016 - EMR \u0026 Docker](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/016%20-%20EMR%20%26%20Docker.ipynb)\n  - [017 - Partition Projection](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/017%20-%20Partition%20Projection.ipynb)\n  - [018 - QuickSight](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/018%20-%20QuickSight.ipynb)\n  - [019 - Athena Cache](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/019%20-%20Athena%20Cache.ipynb)\n  - [020 - Spark Table Interoperability](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/020%20-%20Spark%20Table%20Interoperability.ipynb)\n  - [021 - Global Configurations](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/021%20-%20Global%20Configurations.ipynb)\n  - [022 - Writing Partitions Concurrently](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/022%20-%20Writing%20Partitions%20Concurrently.ipynb)\n  - [023 - Flexible Partitions Filter](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/023%20-%20Flexible%20Partitions%20Filter.ipynb)\n  - [024 - Athena Query Metadata](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/024%20-%20Athena%20Query%20Metadata.ipynb)\n  - [025 - Redshift - Loading Parquet files with Spectrum](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/025%20-%20Redshift%20-%20Loading%20Parquet%20files%20with%20Spectrum.ipynb)\n  - [026 - Amazon Timestream](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/026%20-%20Amazon%20Timestream.ipynb)\n  - [027 - Amazon Timestream 2](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/027%20-%20Amazon%20Timestream%202.ipynb)\n  - [028 - Amazon DynamoDB](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/028%20-%20DynamoDB.ipynb)\n  - [029 - S3 Select](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/029%20-%20S3%20Select.ipynb)\n  - [030 - Data Api](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/030%20-%20Data%20Api.ipynb)\n  - [031 - OpenSearch](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/031%20-%20OpenSearch.ipynb)\n  - [033 - Amazon Neptune](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/033%20-%20Amazon%20Neptune.ipynb)\n  - [034 - Distributing Calls Using Ray](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/034%20-%20Distributing%20Calls%20using%20Ray.ipynb)\n  - [035 - Distributing Calls on Ray Remote Cluster](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/035%20-%20Distributing%20Calls%20on%20Ray%20Remote%20Cluster.ipynb)\n  - [037 - Glue Data Quality](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/037%20-%20Glue%20Data%20Quality.ipynb)\n  - [038 - OpenSearch Serverless](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/038%20-%20OpenSearch%20Serverless.ipynb)\n  - [039 - Athena Iceberg](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/039%20-%20Athena%20Iceberg.ipynb)\n  - [040 - EMR Serverless](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/040%20-%20EMR%20Serverless.ipynb)\n  - [041 - Apache Spark on Amazon Athena](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/041%20-%20Apache%20Spark%20on%20Amazon%20Athena.ipynb)\n- [**API Reference**](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html)\n  - [Amazon S3](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-s3)\n  - [AWS Glue Catalog](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#aws-glue-catalog)\n  - [Amazon Athena](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-athena)\n  - [Amazon Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-redshift)\n  - [PostgreSQL](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#postgresql)\n  - [MySQL](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#mysql)\n  - [SQL Server](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#sqlserver)\n  - [Oracle](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#oracle)\n  - [Data API Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#data-api-redshift)\n  - [Data API RDS](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#data-api-rds)\n  - [OpenSearch](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#opensearch)\n  - [AWS Glue Data Quality](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#aws-glue-data-quality)\n  - [Amazon Neptune](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-neptune)\n  - [DynamoDB](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#dynamodb)\n  - [Amazon Timestream](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-timestream)\n  - [Amazon EMR](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-emr)\n  - [Amazon CloudWatch Logs](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-cloudwatch-logs)\n  - [Amazon Chime](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-chime)\n  - [Amazon QuickSight](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#amazon-quicksight)\n  - [AWS STS](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#aws-sts)\n  - [AWS Secrets Manager](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#aws-secrets-manager)\n  - [Global Configurations](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#global-configurations)\n  - [Distributed - Ray](https://aws-sdk-pandas.readthedocs.io/en/3.16.1/api.html#distributed-ray)\n- [**License**](https://github.com/aws/aws-sdk-pandas/blob/main/LICENSE.txt)\n- [**Contributing**](https://github.com/aws/aws-sdk-pandas/blob/main/CONTRIBUTING.md)\n\n## Getting Help\n\nThe best way to interact with our team is through GitHub. You can open an [issue](https://github.com/aws/aws-sdk-pandas/issues/new/choose) and choose from one of our templates for bug reports, feature requests...\nYou may also find help on these community resources:\n* The #aws-sdk-pandas Slack [channel](https://join.slack.com/t/aws-sdk-pandas/shared_invite/zt-sxdx38sl-E0coRfAds8WdpxXD2Nzfrg)\n* Ask a question on [Stack Overflow](https://stackoverflow.com/questions/tagged/awswrangler)\n  and tag it with `awswrangler`\n* [Runbook](https://github.com/aws/aws-sdk-pandas/discussions/1815) for AWS SDK for pandas with Ray\n\n## Logging\n\nEnabling internal logging examples:\n\n```py3\nimport logging\nlogging.basicConfig(level=logging.INFO, format=\"[%(name)s][%(funcName)s] %(message)s\")\nlogging.getLogger(\"awswrangler\").setLevel(logging.DEBUG)\nlogging.getLogger(\"botocore.credentials\").setLevel(logging.CRITICAL)\n```\n\nInto AWS lambda:\n\n```py3\nimport logging\nlogging.getLogger(\"awswrangler\").setLevel(logging.DEBUG)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Faws-sdk-pandas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws%2Faws-sdk-pandas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Faws-sdk-pandas/lists"}