{"id":13847828,"url":"https://github.com/baidu/shuttle","last_synced_at":"2025-05-05T18:31:08.978Z","repository":{"id":66049529,"uuid":"42511335","full_name":"baidu/shuttle","owner":"baidu","description":"A fast computing framework based on Galaxy","archived":false,"fork":false,"pushed_at":"2018-07-29T22:20:07.000Z","size":1156,"stargazers_count":64,"open_issues_count":1,"forks_count":24,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-08-05T19:34:57.717Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/baidu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2015-09-15T10:29:32.000Z","updated_at":"2023-07-23T06:48:32.000Z","dependencies_parsed_at":"2024-01-15T20:56:21.926Z","dependency_job_id":"d645294e-26c6-4a9a-82b7-3ddfd3650b75","html_url":"https://github.com/baidu/shuttle","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baidu%2Fshuttle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baidu%2Fshuttle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baidu%2Fshuttle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baidu%2Fshuttle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/baidu","download_url":"https://codeload.github.com/baidu/shuttle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224459218,"owners_count":17314848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T19:00:32.876Z","updated_at":"2024-11-13T13:48:05.753Z","avatar_url":"https://github.com/baidu.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"[shuttle - Galaxy上的Map Reduce计算框架](https://github.com/baidu/shuttle)\n====\n\nCopyright 2015, Baidu, Inc.\n\n## 概述\nshuttle是一个基于[Galaxy分布式集群管理系统](https://github.com/baidu/galaxy)的Map Reduce计算框架。其底层采用分布式文件系统保存中间数据，具有较好的稳定性和速度。\n\nshuttle提供类似Hadoop的用户界面，能够方便的递交不同种类的任务及监视其状态。\n\nshuttle目前仍在不断完善和发展，致力于提供更好的用户体验、更丰富的配置与功能以及更快的运算速度。\n\n## 系统架构\nshuttle与经典的Map Reduce架构类似，其利用Galaxy进行资源管理及任务调度，并从HDFS上获取输入并将输出存储在HDFS上，中间数据保存于HDFS或NFS上。\n\nshuttle由client/sdk、Master、Minion三者组成。Master将地址写入[iNexus](https://github.com/baidu/ins)，方便client/sdk以及Minion进行寻址和通信。\n* Master负责接受并处理client/sdk发来的请求，管理元数据，利用Galaxy部署Minion并划分输入，并接受Minion的请求分配和管理运算的任务。\n* Minion从Master获取任务，进行包管理并准备好用户程序运行的环境，处理输入并递交给用户程序，并获取用户程序的输出进行处理，将用户程序的状态回报Master。\n* client/sdk和用户进行交互，获取用户任务的各项信息并提交给Master，也可以从Master处获得任务的数据反馈给用户。\n\n![架构图](https://github.com/baidu/shuttle/blob/master/doc/shuttle.png?raw=true)\n\n## 系统依赖\n* 使用[Galaxy](https://github.com/baidu/galaxy)进行资源管理与任务调度\n* 使用[iNexus](https://github.com/baidu/ins)进行寻址(同时被Galaxy依赖)\n* 使用[sofa-pbrpc](https://github.com/baidu/sofa-pbrpc)进行通信\n* 使用分布式文件系统(HDFS/NFS)进行输入输出及中间数据的存储\n\n## 系统构建\n** 正在构建 **\n\n目前shuttle暂无Makefile进行构建。我们会尽快完善。\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaidu%2Fshuttle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbaidu%2Fshuttle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaidu%2Fshuttle/lists"}