{"id":22357221,"url":"https://github.com/mdh266/polarsduckdbplayground","last_synced_at":"2025-06-29T09:02:58.469Z","repository":{"id":178785509,"uuid":"662357619","full_name":"mdh266/PolarsDuckDBPlayGround","owner":"mdh266","description":"Playing Around With Polars On AWS","archived":false,"fork":false,"pushed_at":"2023-08-27T17:56:27.000Z","size":4581,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-29T09:02:44.141Z","etag":null,"topics":["aws-ec2","data-science","dataframe-library","docker","duckdb","pandas","polars","s3","sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mdh266.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-05T00:57:41.000Z","updated_at":"2025-04-06T17:19:36.000Z","dependencies_parsed_at":"2023-09-23T05:09:28.461Z","dependency_job_id":null,"html_url":"https://github.com/mdh266/PolarsDuckDBPlayGround","commit_stats":{"total_commits":17,"total_committers":1,"mean_commits":17.0,"dds":0.0,"last_synced_commit":"631c141646ba6b0eda049ade3107f8c6727c2a0b"},"previous_names":["mdh266/polarsplayground","mdh266/polarsduckdbplayground"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mdh266/PolarsDuckDBPlayGround","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FPolarsDuckDBPlayGround","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FPolarsDuckDBPlayGround/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FPolarsDuckDBPlayGround/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FPolarsDuckDBPlayGround/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mdh266","download_url":"https://codeload.github.com/mdh266/PolarsDuckDBPlayGround/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdh266%2FPolarsDuckDBPlayGround/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262566830,"owners_count":23329680,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-ec2","data-science","dataframe-library","docker","duckdb","pandas","polars","s3","sql"],"created_at":"2024-12-04T14:13:36.981Z","updated_at":"2025-06-29T09:02:58.443Z","avatar_url":"https://github.com/mdh266.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Polars DataFrame PlayGround\n-------------------\n\nIn this post I quickly covered what I view as the limitations in [Pandas](https://pandas.pydata.org/) library:\n\n\n1. High memory usage\n2. Limited multi-core algorithms\n3. No ability to execute SQL statements (like [SparkSQL \u0026 DataFrame](https://spark.apache.org/sql/))\n4. No query planning/lazy-execution\n5. [NULL values only exist for floats not ints](https://pandas.pydata.org/docs/user_guide/integer_na.html) (this changed in Pandas 1.0+)\n6. Using [strings is inefficient](https://pandas.pydata.org/docs/user_guide/text.html) (this too changed in Pandas 1.0+)\n    \nMany of these issues have been addressed by the [Pandas 2.0 release](https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html), but **I still feel the API is awkward!** \n\nSo in this post I go over two alternatives:\n\n* [Polars](https://www.pola.rs/) for dataframes\n* [DuckDB](https://duckdb.org/) for SQL queries\n\nI cover how to get set up in with \nJuptyer lab using [Docker](https://www.docker.com/) on [AWS](https://aws.amazon.com/) as well as some basics of [Polars](https://www.pola.rs/), [DuckDB](https://duckdb.org/) and how to use the two in combination. The benefits of Polars is that,\n\n* It allows for fast parallel querying on dataframes.\n* It uses [Apache Arrow](https://arrow.apache.org/) for backend datatypes making it efficient for memory.\n* It has both lazy and eager execution mode.\n* It allows for SQL queries direcly on dataframes.\n* Its API is similar to Spark's API and allows for highly readable queries using method chaining.\n\nDuckDB is a blazingly fast [OLAP](https://aws.amazon.com/what-is/olap/) SQL query engine. In the context of this blog post I cover how to use it to run SQL queries against Pandas/Polars dataframes and even local Parquet files!\n\n\n## Using The Notebook\n----------\nYou can install the dependencies and access the notebook using \u003ca href=\"https://www.docker.com/\"\u003eDocker\u003c/a\u003e by building the Docker image with the following:\n\n\tdocker build -t polars_nb .\n\nFollowed by running the command container:\n\n\tdocker run -ip 8888:8888 -v `pwd`:/home/jovyan -t polars_nb\n\nSee \u003ca href=\"https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html\"\u003ehere\u003c/a\u003e for more info. \n\nOtherwise without Docker, make sure to use Python 3.10 and install the libraries listed in \u003ccode\u003erequirements.txt\u003c/code\u003e.  These can be installed with the command,\n\n\tpip install -r requirements.txt","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdh266%2Fpolarsduckdbplayground","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmdh266%2Fpolarsduckdbplayground","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdh266%2Fpolarsduckdbplayground/lists"}