{"id":15707947,"url":"https://github.com/andersy005/dask-notebooks","last_synced_at":"2025-08-30T04:07:27.423Z","repository":{"id":96024571,"uuid":"120401727","full_name":"andersy005/dask-notebooks","owner":"andersy005","description":"Dask tutorials for Big Data Analysis and Machine Learning as Jupyter notebooks ","archived":false,"fork":false,"pushed_at":"2018-02-18T04:35:02.000Z","size":579,"stargazers_count":5,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-20T03:06:07.478Z","etag":null,"topics":["dask","data-science","distributed-computing","jupyter-notebook","parallel-computing","python"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andersy005.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-06T04:29:24.000Z","updated_at":"2023-05-25T11:06:39.000Z","dependencies_parsed_at":"2023-04-24T16:47:25.038Z","dependency_job_id":null,"html_url":"https://github.com/andersy005/dask-notebooks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/andersy005/dask-notebooks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andersy005%2Fdask-notebooks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andersy005%2Fdask-notebooks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andersy005%2Fdask-notebooks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andersy005%2Fdask-notebooks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andersy005","download_url":"https://codeload.github.com/andersy005/dask-notebooks/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andersy005%2Fdask-notebooks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272800967,"owners_count":24995185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dask","data-science","distributed-computing","jupyter-notebook","parallel-computing","python"],"created_at":"2024-10-03T20:42:25.062Z","updated_at":"2025-08-30T04:07:27.364Z","avatar_url":"https://github.com/andersy005.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://i.imgur.com/M3BRrh1.png\" align=\"right\" width=\"30%\"\u003e\n\n# Dask Python Notebooks\n\nThis is a collection of [**Jupyter**](https://jupyter.org/) notebooks intended to train the reader on different [Dask](https://dask.pydata.org/en/latest/) concepts, from basic to advanced.\n\n## Overview\n\nDask provides multicore and distributed parallel execution on larger-than-memory datasets.\n\nWe can think of Dask at a high and a low level\n\n- **High Level collections:** Dask provides high-level\n  - Arrays\n  - Bag\n  - and DataFrame\n \n collections that mimic NumPy, Lists, and Pandas but can operate in parallel on datasets that don't fit into memory. Dask's high-level collections are alternatives to NumPy and Pandas for large datasets.\n \n - **Low level schedulers:** Dask provides dynamic task schedulers that execute task graphs in parallel. These execution engines power the high-level collections mentioned above but can also power custom, user-defined workloads. These schedulers are low-latency (around 1 ms) and work hard to run computations in a small memory footprint. Dask's schedulers are an alternative to direct use of `threading` or `multiprocessing` libraries in complex cases or other task scheduling systems like `Luigi` or `IPython parallel`.\n \n \nDiferent users operate at different levels but it is useful to understand both.\n\n## Basics of Dask \n\n\nThe basics of dask can be summarized as follows:\n- process data that doesn't fit into memory by breaking it into blocks and specify task chains.\n- parallelize execution of tasks across cores and even nodes of a cluster.\n- move computation to the data rather than the other way around, to minimize communication overheads.\n\n\n## Distributed \n\nDask comes with four available schedulers:\n- `dask.threaded.get`: a scheduler backed by a thread pool\n- `dask.multiprocessing.get`: a scheduler backed by a process pool\n- `dask.get`: a synchronous scheduler, good for debugging\n- `distributed.Client.get`: a distributed scheduler for executing graphs on multiple machines.\n\n\nTo select one of these for computation, you can specify at the time of asking for a result\n\n```python\nmyvalue.compute(get=dask.async.get_sync)  # for debugging\n```\n\nor set the current default, either temporarily or globally\n\n```python\nwith dask.set_options(get=dask.multiprocessing.get):\n    # set temporarily fo this block only\n    myvalue.compute()\n\ndask.set_options(get=dask.multiprocessing.get)\n# set until further notice\n```\n\nFor single-machine use, the threaded and multiprocessing schedulers are fine choices. However, for scaling out work across a cluster, the distributed scheduler is required. Indeed, this is now generally preferred for all work, because it gives you additional monitoring information not available in the other schedulers. (Some of this monitoring is also available with an explicit progress bar and profiler, see [here](http://dask.pydata.org/en/latest/diagnostics.html).)\n\n\n## Instructions\n\nA good way of using these notebooks is by first cloning the repo, and then starting your own Jupyter notebook after installing all necessary packages. \n\n\n    git clone https://github.com/andersy005/dask-notebooks.git\n\nand then install necessary packages.\n\n### a) Install into an existing environment\n\nYou will need the following core libraries\n\n    conda install numpy pandas h5py Pillow matplotlib scipy toolz pytables snakeviz dask distributed\n\nYou may find the following libraries helpful for some notebooks\n\n    pip install graphviz cachey\n    \n### b) Create a new environment\n\nIn the repo directory\n\n    conda env create -f environment.yml \n\nand then on osx/linux\n\n    source activate dask-tutorial\n\non windows\n\n    activate dask-tutorial\n\n### c) Use Dockerfile\n\nYou can build a docker image out of the provided Dockerfile.\n\n\n\n### Graphviz on Windows\n\nWindows users can install graphviz as follows\n\n1. Install Graphviz from http://www.graphviz.org/Download_windows.php\n2. Add C:\\Program Files (x86)\\Graphviz2.38\\bin to the PATH\n\nAlternatively one can use the following conda commands (one installs graphviz and one installs python-bindings for graphviz):\n\n1. `conda install -c conda-forge graphviz`\n2. `conda install -c conda-forge python-graphviz`\n\n\n## Datasets  \n\nWe will be using datasets from the [KDD Cup 1999](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html). The results \nof this competition can be found [here](http://cseweb.ucsd.edu/~elkan/clresults.html).  \n\n\n## Notebooks  \n\nThe following notebooks can be examined individually, although there is a more\nor less linear 'story' when followed in sequence. By using the same dataset\nthey try to solve a related set of tasks with it.\n\n1. [**Dask Bag creation**](https://github.com/andersy005/dask-notebooks/blob/master/01-dask-bags/01-bag-creation.ipynb): About reading files and creating bags.\n2. [**Dask Bag basics**](https://github.com/andersy005/dask-notebooks/blob/master/01-dask-bags/02-bag-basics.ipynb): A look at `map`, `filter`, `compute`, `persist`, `flatten`\n\n## Contributing\nContributions are welcome!  For bug reports or requests please [submit an issue](https://github.com/andersy005/dask-notebooks/issues).\n\n## To Do List\nThis section is for myself, but feel free to **fork the repo** and add your contributions!\n\n- [ ] Add DockerFile\n- [ ] Add environment.yml file\n- [ ] Add Binder Support\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandersy005%2Fdask-notebooks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandersy005%2Fdask-notebooks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandersy005%2Fdask-notebooks/lists"}