https://github.com/dask-contrib/dask-snowflake
Dask integration for Snowflake
https://github.com/dask-contrib/dask-snowflake
dask python snowflake
Last synced: about 1 month ago
JSON representation
Dask integration for Snowflake
- Host: GitHub
- URL: https://github.com/dask-contrib/dask-snowflake
- Owner: dask-contrib
- License: bsd-3-clause
- Created: 2021-04-08T22:35:55.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2025-08-04T08:19:19.000Z (5 months ago)
- Last Synced: 2025-08-04T10:31:39.611Z (5 months ago)
- Topics: dask, python, snowflake
- Language: Python
- Homepage:
- Size: 73.2 KB
- Stars: 30
- Watchers: 16
- Forks: 10
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Dask-Snowflake
[](https://github.com/dask-contrib/dask-snowflake/actions/workflows/tests.yml)
[](https://github.com/dask-contrib/dask-snowflake/actions/workflows/pre-commit.yml)
## Installation
`dask-snowflake` can be installed with `pip`:
```shell
pip install dask-snowflake
```
or with `conda`:
```shell
conda install -c conda-forge dask-snowflake
```
## Usage
`dask-snowflake` provides `read_snowflake` and `to_snowflake` methods
for parallel IO from Snowflake with Dask.
```python
>>> from dask_snowflake import read_snowflake
>>> example_query = '''
... SELECT *
... FROM SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.CUSTOMER;
... '''
>>> ddf = read_snowflake(
... query=example_query,
... connection_kwargs={
... "user": "...",
... "password": "...",
... "account": "...",
... },
... )
```
```python
>>> from dask_snowflake import to_snowflake
>>> to_snowflake(
... ddf,
... name="my_table",
... connection_kwargs={
... "user": "...",
... "password": "...",
... "account": "...",
... },
... )
```
See their docstrings for further API information.
## Tests
Running tests requires a Snowflake account and access to a database.
The test suite will automatically look for specific `SNOWFLAKE_*`
environment variables (listed below) that must be set.
It's recommended (though not required) to store these environment variables
in a local `.env` file in the root of the `dask-snowflake` repository.
This file will be automatically ignored by `git`, reducing the risk of accidentally
commiting it.
Here's what an example `.env` file looks like:
```env
SNOWFLAKE_USER=""
SNOWFLAKE_PASSWORD=""
SNOWFLAKE_ACCOUNT="..aws"
SNOWFLAKE_WAREHOUSE=""
SNOWFLAKE_ROLE=""
SNOWFLAKE_DATABASE=""
SNOWFLAKE_SCHEMA=""
```
You may then `source .env` or install [`pytest-dotenv`](https://github.com/quiqua/pytest-dotenv)
to automatically set these environment variables.
> **_Note:_**
> If you run the tests and get an `MemoryError` mentioning
> "write+execute memory for ffi.callback()", you probably have stale
> build of `cffi` from conda-forge. Remove it and install the version
> using `pip`:
>
> ```shell
> conda remove cffi --force
> pip install cffi
> ```
## License
[BSD-3](LICENSE)