{"id":15096955,"url":"https://github.com/apache/incubator-retired-amaterasu","last_synced_at":"2025-10-08T02:30:42.387Z","repository":{"id":65979428,"uuid":"105068223","full_name":"apache/incubator-retired-amaterasu","owner":"apache","description":"Apache Amaterasu","archived":true,"fork":false,"pushed_at":"2019-10-18T10:24:37.000Z","size":4663,"stargazers_count":56,"open_issues_count":4,"forks_count":24,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-10-06T00:36:43.681Z","etag":null,"topics":["amaterasu"],"latest_commit_sha":null,"homepage":"https://amaterasu.apache.org/","language":"Kotlin","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-27T20:52:34.000Z","updated_at":"2024-02-09T21:44:37.000Z","dependencies_parsed_at":"2023-02-19T19:00:28.129Z","dependency_job_id":null,"html_url":"https://github.com/apache/incubator-retired-amaterasu","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/apache/incubator-retired-amaterasu","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-retired-amaterasu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-retired-amaterasu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-retired-amaterasu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-retired-amaterasu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/incubator-retired-amaterasu/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-retired-amaterasu/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278877493,"owners_count":26061456,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amaterasu"],"created_at":"2024-09-25T16:02:50.973Z","updated_at":"2025-10-08T02:30:41.951Z","avatar_url":"https://github.com/apache.png","language":"Kotlin","funding_links":[],"categories":["大数据"],"sub_categories":["微服务框架"],"readme":"\u003c!--\n  ~ Licensed to the Apache Software Foundation (ASF) under one or more\n  ~ contributor license agreements.  See the NOTICE file distributed with\n  ~ this work for additional information regarding copyright ownership.\n  ~ The ASF licenses this file to You under the Apache License, Version 2.0\n  ~ (the \"License\"); you may not use this file except in compliance with\n  ~ the License.  You may obtain a copy of the License at\n  ~\n  ~      http://www.apache.org/licenses/LICENSE-2.0\n  ~\n  ~ Unless required by applicable law or agreed to in writing, software\n  ~ distributed under the License is distributed on an \"AS IS\" BASIS,\n  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n  ~ See the License for the specific language governing permissions and\n  ~ limitations under the License.\n  --\u003e\n# Apache Amaterasu \n\n[![Build Status](https://travis-ci.org/apache/incubator-amaterasu.svg?branch=master)](https://travis-ci.org/apache/incubator-amaterasu)\n[![License](http://img.shields.io/:license-Apache%202-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)\n\n                                               /\\\n                                              /  \\ /\\\n                                             / /\\ /  \\\n        _                 _                 / /  / /\\ \\   \n       /_\\   _ __   __ _ | |_  ___  _ _  __(_( _(_(_ )_) \n      / _ \\ | '  \\ / _` ||  _|/ -_)| '_|/ _` |(_-\u003c| || |\n     /_/ \\_\\|_|_|_|\\__,_| \\__|\\___||_|  \\__,_|/__/ \\_,_|\n                                                        \n\nApache Amaterasu is an open-source, deployment tool for data pipelines. Amaterasu allows developers to write and easily deploy data pipelines, and clusters manage their configuration and dependencies.\n\n## Download\n\nFor this preview version, we have packaged amaterasu nicely for you to just [download](https://s3-ap-southeast-2.amazonaws.com/amaterasu/amaterasu.tgz) and extract.\nOnce you do that, you are just a couple of easy steps away from running your first job.\n\n## Creating a dev/test Mesos cluster\n\nWe have also created a Mesos cluster you can use to test Amaterasu or use for development purposes.\nFor more details, visit the [amaterasu-vagrant](https://github.com/shintoio/amaterasu-vagrant) repo\n\n## Configuration\n\nConfiguring amaterasu is very simple. Before running amaterasu, open the `amaterasu.properties` file in the top-level amaterasu directory, and verify the following properties:\n\n| property   | Description                | Default value  |\n| ---------- | -------------------------- | -------------- |\n| zk         | The ZooKeeper connection\u003cbr\u003e string to be used by\u003cbr\u003e amaterasu | 192.168.33.11  |\n| master     | The clusters' Mesos master | 192.168.33.11  |\n| user       | The user that will be used\u003cbr\u003e to run amaterasu | root           |\n\n## Running a Job\n\nTo run an amaterasu job, run the following command in the top-level amaterasu directory:\n\n```\nama-start.sh --repo=\"https://github.com/shintoio/amaterasu-job-sample.git\" --branch=\"master\" --env=\"test\" --report=\"code\" \n```\n\nWe recommend you either fork or clone the job sample repo and use that as a starting point for creating your first job.\n\n# Apache Amaterasu Developers Information \n\n## Building Apache Amaterasu\n\n### Instructions to download gradle wrapper when building from source distribution\n\nAs part of the ASFs policy, when building Apache Amaterasu from a source distribution and not from git, the gradle-wrapper.jar needs to be downloaded.\nThe following command, will download the gradle wrapper from Amaterasu git repository and puts under incubator-amaterasu/gradle/wrapper\n\nwget --no-check-certificate -P incubator-amaterasu/gradle/wrapper https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar\n(or)\ncurl --insecure -L https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar \u003e incubator-amaterasu/gradle/wrapper/gradle-wrapper.jar\n\n### Building Amaterasu with Gradle\n\nto build the amaterasu home dir (for dev purposes) run:\n```\n./gradlew buildHomeDir test\n```\n\nto create a distributable jar (clean creates the home dir first) run:\n```\n./gradlew buildDistribution test\n```\n\n## Architecture\n\nAmaterasu is an Apache Mesos framework with two levels of schedulers:\n\n* The ClusterScheduler manages the execution of all the jobs\n* The JobScheduler manages the flow of a job\n\nThe main clases in Amateraso are listed bellow:\n\n    +-------------------------+   +------------------------+\n    | ClusterScheduler        |   | Kami                   |\n    |                         |--\u003e|                        |\n    | Manage jobs:            |   | Manages the jobs queue |\n    | Queue new jobs          |   | and Amaterasu cluster  |\n    | Reload interrupted jobs |   +------------------------+\n    | Monitor cluster state   |\n    +-------------------------+\n                |\n                |     +------------------------+\n                |     | JobExecutor            |\n                |     |                        |\n                +----\u003e| Runs the Job Scheduler |\n                      | Communicates with the  |\n                      | ClusterScheduler       |\n                      +------------------------+\n                                 |\n                                 |\n                      +------------------------+      +---------------------------+                      \n                      | JobScheduler           |      | JobParser                 |\n                      |                        |      |                           |\n                      | Manages the execution  |-----\u003e| Parses the kami.yaml file |\n                      | of the job, by getting |      | and create a JobManager   |\n                      | the  execution flow    |      +---------------------------+\n                      | fron the JobManager    |                    |\n                      | and comunicating with  |      +---------------------------+\n                      | Mesos                  |      | JobManager                |                      \n                      +------------------------+      |                           |\n                                 |                    | Manages the jobs workflow |\n                                 |                    | independently of mesos    |\n                      +------------------------+      +---------------------------+\n                      | ActionExecutor         |\n                      |                        |\n                      | Executes ActionRunners |\n                      | and manages state for  |\n                      | the executor           |\n                      +------------------------+\n\n                      \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-retired-amaterasu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fincubator-retired-amaterasu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-retired-amaterasu/lists"}