{"id":15287735,"url":"https://github.com/dcfjs/dcf","last_synced_at":"2025-04-09T08:08:13.024Z","repository":{"id":48016793,"uuid":"144890394","full_name":"dcfjs/dcf","owner":"dcfjs","description":"Yet another distributed compute framework","archived":false,"fork":false,"pushed_at":"2024-12-17T08:22:17.000Z","size":151,"stargazers_count":47,"open_issues_count":5,"forks_count":5,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-02T07:11:16.938Z","etag":null,"topics":["distributed-computing","nodejs","spark"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dcfjs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-15T18:41:05.000Z","updated_at":"2024-12-17T08:22:21.000Z","dependencies_parsed_at":"2024-12-31T20:14:26.585Z","dependency_job_id":"f4eaf457-d938-4fb1-99af-9f9b19dbf1c2","html_url":"https://github.com/dcfjs/dcf","commit_stats":{"total_commits":112,"total_committers":5,"mean_commits":22.4,"dds":0.6785714285714286,"last_synced_commit":"d10dd5e77bdf2a12627565c648796a7388feecb4"},"previous_names":["tdzl2003/dcf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcfjs%2Fdcf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcfjs%2Fdcf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcfjs%2Fdcf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcfjs%2Fdcf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dcfjs","download_url":"https://codeload.github.com/dcfjs/dcf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247999859,"owners_count":21031046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","nodejs","spark"],"created_at":"2024-09-30T15:36:10.720Z","updated_at":"2025-04-09T08:08:12.997Z","avatar_url":"https://github.com/dcfjs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed Computing Framework for Node.js\n\n\u003e Early development stage: this project was still under early development, many necessery feature was not done yet, use it on your own risk.\n\n[Document](https://dcf.gitbook.io/dcf/)\n\n[API Reference](https://dcf.gitbook.io/dcf/api/)\n\nA node.js version of [Spark](https://spark.apache.org/), without hadoop or jvm.\n\nYou should read [tutorial](src/samples/tutorial-0.ts) first, then you can learn Spark but use this project instead.\n\n## Async API \u0026 deferred API\n\nAny api that requires a RDD and generate a result is async, like `count`, `take`, `max` ...\nAny api that creates a RDD is deferred API, which is not async, so you can chain them like this:\n\n```js\nawait dcc\n  .parallelize([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])\n  .map(v =\u003e v + 1)\n  .filter(v =\u003e v % 2 === 0)\n  .take(10); // take is not deferred api but async\n```\n\n## Milestones\n\n#### 0.1.x: Basic\n\n- [x] local master.\n- [x] rdd \u0026 partition creation \u0026 release.\n- [x] map \u0026 reduce\n- [x] repartition \u0026 reduceByKey\n- [x] disk storage partitions\n- [x] cache\n- [x] file loader \u0026 saver\n- [x] export module to npm\n- [x] decompresser \u0026 compresser\n- [x] use debug module for information/error\n- [x] provide a progress bar.\n- [ ] sampler\n- [x] sort\n- [ ] object hash(for key) method\n- [ ] storage MEMORY_OR_DISK, and use it in sort\n- [ ] storage MEMORY_SER，storage in memory but off v8 heap.\n- [ ] config default partition count.\n\n#### 0.2.x: Remote mode\n\n- [ ] distributed master\n- [ ] runtime sandbox\n- [ ] plugin system\n- [ ] remote dependency management\n- [ ] aliyun oss loader\n- [ ] hdfs loader\n\n## How to use\n\n#### Install from npm(shell only)\n\n```\nnpm install -g dcf\n#or\nyarn global add dcf\n```\n\nThen you can use command: `dcf-shell`\n\n#### Install from npm(as dependency)\n\n```\nnpm install --save dcf\n#or\nyarn add dcf\n```\n\nThen you can use dcf with javascript or typescript.\n\n#### Run samples \u0026 cli\n\ndownload this repo, install dependencies\n\n```bash\nnpm install\n# or\nyarn\n```\n\nRun samples:\n\n```bash\nnpm run ts-node src/samples/tutorial-0.ts\nnpm run ts-node src/samples/repartition.ts\n```\n\nRun interactive cli:\n\n```bash\nnpm start\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcfjs%2Fdcf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdcfjs%2Fdcf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcfjs%2Fdcf/lists"}