{"id":16335941,"url":"https://github.com/utdemir/distributed-dataset","last_synced_at":"2025-12-11T23:13:39.825Z","repository":{"id":56843493,"uuid":"124729238","full_name":"utdemir/distributed-dataset","owner":"utdemir","description":"A distributed data processing framework in Haskell.","archived":false,"fork":false,"pushed_at":"2020-06-13T05:28:09.000Z","size":896,"stargazers_count":116,"open_issues_count":19,"forks_count":5,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-12-06T21:27:15.824Z","etag":null,"topics":["aws-lambda","data-processing","distributed","haskell","spark"],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/utdemir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-11T06:19:04.000Z","updated_at":"2024-11-13T17:08:15.000Z","dependencies_parsed_at":"2022-09-07T11:41:08.675Z","dependency_job_id":null,"html_url":"https://github.com/utdemir/distributed-dataset","commit_stats":null,"previous_names":["utdemir/distributed-fork"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utdemir%2Fdistributed-dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utdemir%2Fdistributed-dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utdemir%2Fdistributed-dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utdemir%2Fdistributed-dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/utdemir","download_url":"https://codeload.github.com/utdemir/distributed-dataset/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243819033,"owners_count":20352807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-lambda","data-processing","distributed","haskell","spark"],"created_at":"2024-10-10T23:42:46.028Z","updated_at":"2025-12-11T23:13:39.762Z","avatar_url":"https://github.com/utdemir.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# distributed-dataset\n\n[![CI Status](https://github.com/utdemir/distributed-dataset/workflows/ci/badge.svg)](https://github.com/utdemir/distributed-dataset/actions)\n\nA distributed data processing framework in pure Haskell. Inspired by [Apache Spark](https://spark.apache.org/).\n\n* **An example:** [/examples/gh/Main.hs](/examples/gh/Main.hs)\n* **API documentation:** \u003chttps://utdemir.github.io/distributed-dataset/\u003e\n* **Introduction blogpost:** \u003chttps://utdemir.com/posts/ann-distributed-dataset.html\u003e\n\n## Packages\n\n### distributed-dataset\n\nThis package provides a `Dataset` type which lets you express and execute\ntransformations on a distributed multiset. Its API is highly inspired\nby Apache Spark.\n\nIt uses pluggable `Backend`s for spawning executors and `ShuffleStore`s\nfor exchanging information. See 'distributed-dataset-aws' for an\nimplementation using AWS Lambda and S3.\n\nIt also exposes a more primitive `Control.Distributed.Fork`\nmodule which lets you run `IO` actions remotely. It\nis especially useful when your task is [embarrassingly\nparallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel).\n\n### distributed-dataset-aws\n\nThis package provides a backend for 'distributed-dataset' using AWS\nservices. Currently it supports running functions on AWS Lambda and\nusing an S3 bucket as a shuffle store.\n\n### distributed-dataset-opendatasets\n\nProvides `Dataset`'s reading from public open datasets. Currently it can fetch GitHub event data from [GH Archive](https://www.gharchive.org).\n\n## Running the example\n\n* Clone the repository.\n\n  ```sh\n  $ git clone https://github.com/utdemir/distributed-dataset\n  $ cd distributed-dataset\n  ```\n\n* Make sure that you have AWS credentials set up. The easiest way is\n  to install [AWS command line interface](https://aws.amazon.com/cli/)\n  and to run:\n\n  ```sh\n  $ aws configure\n  ```\n\n* Create an S3 bucket to put the deployment artifact in. You can use\n  the console or the CLI:\n\n  ```sh\n  $ aws s3api create-bucket --bucket my-s3-bucket\n  ```\n\n* Build an run the example:\n\n  * If you use Nix on Linux:\n\n    * (Recommended) Use my binary cache on Cachix to reduce compilation times:\n\n    ```sh\n    nix-env -i cachix # or your preferred installation method\n    cachix use utdemir\n    ```\n\n    * Then:\n\n      ```sh\n      $ nix run -f ./default.nix example-gh -c example-gh my-s3-bucket\n      ```\n\n  * If you use stack (requires Docker, works on Linux and MacOS):\n\n      ```sh\n      $ stack run --docker-mount $HOME/.aws/ --docker-env HOME=$HOME example-gh my-s3-bucket\n      ```\n\n## Stability\n\nExperimental. Expect lots of missing features, bugs,\ninstability and API changes. You will probably need to\nmodify the source if you want to do anything serious. See\n[issues](https://github.com/utdemir/distributed-dataset/issues).\n\n## Contributing\n\nI am open to contributions; any issue, PR or opinion is more than welcome.\n\n* In order to develop `distributed-dataset`, you can use;\n  * On Linux: `Nix`, `cabal-install` or `stack`.\n  * On MacOS: `stack` with `docker`.\n* Use [ormolu](https://github.com/tweag/ormolu) to format source code.\n\n### Nix\n\n* You can use [my binary cache on cachix](https://utdemir.cachix.org/)\n  so that you don't recompile half of the Hackage.\n* `nix-shell` will drop you into a shell with `ormolu`, `cabal-install` and\n  `steeloverseer` alongside with all required haskell and system dependencies.\n  You can use `cabal new-*` commands there.\n* Easiest way to get a development environment would be to run `sos` at the\n  top level directory inside of a nix-shell.\n\n### Stack\n\n* Make sure that you have `Docker` installed.\n* Use `stack` as usual, it will automatically use a Docker image\n* Run `./make.sh stack-build` before you send a PR to test different resolvers.\n\n## Related Work\n\n### Papers\n\n* [Towards Haskell in Cloud](https://www.microsoft.com/en-us/research/publication/towards-haskell-cloud/) by Jeff Epstein, Andrew P. Black, Simon L. Peyton Jones\n* [Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing](https://cs.stanford.edu/~matei/papers/2012/nsdi_spark.pdf) by Matei Zaharia, et al.\n\n### Projects\n\n* [Apache Spark](https://spark.apache.org/).\n* [Sparkle](https://github.com/tweag/sparkle): Run Haskell on top of Apache Spark.\n* [HSpark](https://github.com/yogeshsajanikar/hspark): Another attempt at porting Apache Spark to Haskell.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futdemir%2Fdistributed-dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Futdemir%2Fdistributed-dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futdemir%2Fdistributed-dataset/lists"}