https://github.com/apache/incubator-retired-amaterasu
Apache Amaterasu
https://github.com/apache/incubator-retired-amaterasu
amaterasu
Last synced: 9 months ago
JSON representation
Apache Amaterasu
- Host: GitHub
- URL: https://github.com/apache/incubator-retired-amaterasu
- Owner: apache
- License: apache-2.0
- Archived: true
- Created: 2017-09-27T20:52:34.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2019-10-18T10:24:37.000Z (over 6 years ago)
- Last Synced: 2025-10-06T00:36:43.681Z (9 months ago)
- Topics: amaterasu
- Language: Kotlin
- Homepage: https://amaterasu.apache.org/
- Size: 4.45 MB
- Stars: 56
- Watchers: 12
- Forks: 24
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-java - Apache Amaterasu
README
# Apache Amaterasu
[](https://travis-ci.org/apache/incubator-amaterasu)
[](http://www.apache.org/licenses/LICENSE-2.0.txt)
/\
/ \ /\
/ /\ / \
_ _ / / / /\ \
/_\ _ __ __ _ | |_ ___ _ _ __(_( _(_(_ )_)
/ _ \ | ' \ / _` || _|/ -_)| '_|/ _` |(_-<| || |
/_/ \_\|_|_|_|\__,_| \__|\___||_| \__,_|/__/ \_,_|
Apache Amaterasu is an open-source, deployment tool for data pipelines. Amaterasu allows developers to write and easily deploy data pipelines, and clusters manage their configuration and dependencies.
## Download
For this preview version, we have packaged amaterasu nicely for you to just [download](https://s3-ap-southeast-2.amazonaws.com/amaterasu/amaterasu.tgz) and extract.
Once you do that, you are just a couple of easy steps away from running your first job.
## Creating a dev/test Mesos cluster
We have also created a Mesos cluster you can use to test Amaterasu or use for development purposes.
For more details, visit the [amaterasu-vagrant](https://github.com/shintoio/amaterasu-vagrant) repo
## Configuration
Configuring amaterasu is very simple. Before running amaterasu, open the `amaterasu.properties` file in the top-level amaterasu directory, and verify the following properties:
| property | Description | Default value |
| ---------- | -------------------------- | -------------- |
| zk | The ZooKeeper connection
string to be used by
amaterasu | 192.168.33.11 |
| master | The clusters' Mesos master | 192.168.33.11 |
| user | The user that will be used
to run amaterasu | root |
## Running a Job
To run an amaterasu job, run the following command in the top-level amaterasu directory:
```
ama-start.sh --repo="https://github.com/shintoio/amaterasu-job-sample.git" --branch="master" --env="test" --report="code"
```
We recommend you either fork or clone the job sample repo and use that as a starting point for creating your first job.
# Apache Amaterasu Developers Information
## Building Apache Amaterasu
### Instructions to download gradle wrapper when building from source distribution
As part of the ASFs policy, when building Apache Amaterasu from a source distribution and not from git, the gradle-wrapper.jar needs to be downloaded.
The following command, will download the gradle wrapper from Amaterasu git repository and puts under incubator-amaterasu/gradle/wrapper
wget --no-check-certificate -P incubator-amaterasu/gradle/wrapper https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar
(or)
curl --insecure -L https://github.com/apache/incubator-amaterasu/raw/version-0.2.0-incubating-rc4/gradle/wrapper/gradle-wrapper.jar > incubator-amaterasu/gradle/wrapper/gradle-wrapper.jar
### Building Amaterasu with Gradle
to build the amaterasu home dir (for dev purposes) run:
```
./gradlew buildHomeDir test
```
to create a distributable jar (clean creates the home dir first) run:
```
./gradlew buildDistribution test
```
## Architecture
Amaterasu is an Apache Mesos framework with two levels of schedulers:
* The ClusterScheduler manages the execution of all the jobs
* The JobScheduler manages the flow of a job
The main clases in Amateraso are listed bellow:
+-------------------------+ +------------------------+
| ClusterScheduler | | Kami |
| |-->| |
| Manage jobs: | | Manages the jobs queue |
| Queue new jobs | | and Amaterasu cluster |
| Reload interrupted jobs | +------------------------+
| Monitor cluster state |
+-------------------------+
|
| +------------------------+
| | JobExecutor |
| | |
+---->| Runs the Job Scheduler |
| Communicates with the |
| ClusterScheduler |
+------------------------+
|
|
+------------------------+ +---------------------------+
| JobScheduler | | JobParser |
| | | |
| Manages the execution |----->| Parses the kami.yaml file |
| of the job, by getting | | and create a JobManager |
| the execution flow | +---------------------------+
| fron the JobManager | |
| and comunicating with | +---------------------------+
| Mesos | | JobManager |
+------------------------+ | |
| | Manages the jobs workflow |
| | independently of mesos |
+------------------------+ +---------------------------+
| ActionExecutor |
| |
| Executes ActionRunners |
| and manages state for |
| the executor |
+------------------------+