{"id":15902795,"url":"https://github.com/gjoseph92/sneks","last_synced_at":"2025-03-20T19:31:28.134Z","repository":{"id":38445907,"uuid":"498938341","full_name":"gjoseph92/sneks","owner":"gjoseph92","description":"Launch a Dask cluster from a Poetry environment","archived":false,"fork":false,"pushed_at":"2023-03-10T16:39:11.000Z","size":677,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-07T11:43:06.020Z","etag":null,"topics":["coiled","dask","dask-distributed","poetry-python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gjoseph92.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-02T00:23:12.000Z","updated_at":"2024-04-25T05:10:01.000Z","dependencies_parsed_at":"2023-01-29T04:00:50.889Z","dependency_job_id":null,"html_url":"https://github.com/gjoseph92/sneks","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjoseph92%2Fsneks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjoseph92%2Fsneks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjoseph92%2Fsneks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjoseph92%2Fsneks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gjoseph92","download_url":"https://codeload.github.com/gjoseph92/sneks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221799214,"owners_count":16882380,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coiled","dask","dask-distributed","poetry-python"],"created_at":"2024-10-06T11:43:11.364Z","updated_at":"2024-10-28T07:21:52.137Z","avatar_url":"https://github.com/gjoseph92.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sneks\n\nGet your snakes in a row.\n\n`sneks` lets you launch a [Dask cluster in the cloud](https://coiled.io/), matched to your local software environment\\*, in a single line of code. No more dependency mismatches or Docker image building.\n\n```python\nfrom sneks import get_client\n\nclient = get_client()\n```\n\n\\*your local [Poetry](https://python-poetry.org/) or [PDM](https://pdm.fming.dev/latest/) environment. You must use poetry or PDM. Locking package managers are what sensible people use, and you are sensible.\n\n*Neat! Sneks also supports ARM clusters! Just pass ARM instances in `scheduler_instace_types=`, `worker_instace_types=` and cross your fingers that all your dependencies have cross-arch wheels!*\n\n## Installation\n\n```shell\npoetry add -G dev sneks-sync\n```\n\n## A full example:\n\n```shell\nmkdir example \u0026\u0026 cd example\npoetry init -n\npoetry add -G dev sneks-sync\npoetry add distributed==2022.5.2 dask==2022.5.2 bokeh pandas pyarrow  # and whatever else you want\n```\n```python\nfrom sneks import get_client\nimport dask.dataframe as dd\n\nclient = get_client(name=\"on-a-plane\")\nddf = dd.read_parquet(\n    \"s3://nyc-tlc/trip data/yellow_tripdata_2012-*.parquet\",\n)\nprint(ddf.groupby('passenger_count').trip_distance.mean().compute())\n```\n\nOh wait, we forgot to install a dependency!\n```shell\npoetry add foobar\n```\n\nWhen we reconnect to the cluster (using the same name), the dependencies on the cluster update automatically.\n```python\nfrom sneks import get_client\nimport dask.dataframe as dd\nimport foobar  # ah, how could we forget this critical tool\n\nclient = get_client(name=\"on-a-plane\")\nddf = dd.read_csv(\n    \"s3://nyc-tlc/csv_backup/yellow_tripdata_2012-*.csv\",\n)\nmeans = ddf.groupby('passenger_count').trip_distance.mean()\nmeans.apply(foobar.optimize).compute()\n\n```\n\n## Caveats\n\nThis is still a proof-of-concept-level package. It's been used personally quite a bit, and proven reliable, but use at your own risk.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgjoseph92%2Fsneks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgjoseph92%2Fsneks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgjoseph92%2Fsneks/lists"}