{"id":13393679,"url":"https://github.com/gaia-pipeline/gaia","last_synced_at":"2025-05-14T14:10:24.314Z","repository":{"id":27931774,"uuid":"115614847","full_name":"gaia-pipeline/gaia","owner":"gaia-pipeline","description":"Build powerful pipelines in any programming language.","archived":false,"fork":false,"pushed_at":"2023-08-22T08:30:14.000Z","size":14793,"stargazers_count":5214,"open_issues_count":55,"forks_count":243,"subscribers_count":105,"default_branch":"master","last_synced_at":"2025-04-11T18:23:49.725Z","etag":null,"topics":["automation","build","continuous-delivery","continuous-integration","continuous-testing","cplusplus","deployment","devops","devops-tools","distributed-systems","gitops","go","java","kubernetes","pipeline","python","ruby"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gaia-pipeline.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"security/README.md","support":null,"governance":null}},"created_at":"2017-12-28T11:01:31.000Z","updated_at":"2025-04-11T10:47:35.000Z","dependencies_parsed_at":"2023-02-18T00:16:03.781Z","dependency_job_id":"39ef1c0a-d8b6-4f7b-aef7-622a40bf49fa","html_url":"https://github.com/gaia-pipeline/gaia","commit_stats":{"total_commits":305,"total_committers":17,"mean_commits":"17.941176470588236","dds":0.4426229508196722,"last_synced_commit":"79cb63136ac0a70a13fee5054f4dd8c71001a35a"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaia-pipeline%2Fgaia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaia-pipeline%2Fgaia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaia-pipeline%2Fgaia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gaia-pipeline%2Fgaia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gaia-pipeline","download_url":"https://codeload.github.com/gaia-pipeline/gaia/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254160558,"owners_count":22024571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","build","continuous-delivery","continuous-integration","continuous-testing","cplusplus","deployment","devops","devops-tools","distributed-systems","gitops","go","java","kubernetes","pipeline","python","ruby"],"created_at":"2024-07-30T17:00:58.541Z","updated_at":"2025-05-14T14:10:19.300Z","avatar_url":"https://github.com/gaia-pipeline.png","language":"Go","funding_links":[],"categories":["Go","Software Packages","DevOps","Go Tools","后端开发框架及项目","软件包","automation","go","Go 工具","Repositories","DevOps Tools"],"sub_categories":["DevOps Tools","Data Management","管理面板","DevOps 工具","devops 工具","代码分析","DevOps工具"],"readme":".. raw:: html\n\n    \u003cimg src=\"https://gist.githubusercontent.com/michelvocks/ef3894f63c3bb004bca1a2fd5f7eb644/raw/40c5799d74a6f28af1874e726083a50a3ebd877d/gaia-logo-text.png\" width=\"650px\"\u003e\n\n|build-status| |go-report| |go-doc| |apache2| |chat| |codecov|\n\nGaia is an open source automation platform which makes it easy and fun to build powerful pipelines in any programming language. Based on `HashiCorp's go-plugin`_ and `gRPC`_, gaia is efficient, fast, lightweight, and developer friendly.\n\nDevelop powerful `pipelines \u003cWhat is a pipeline?_\u003e`_ with the help of `SDKs \u003cWhy do I need an SDK?_\u003e`_ and simply check-in your code into a git repository. Gaia automatically clones your code repository, compiles your code to a binary, and executes it on-demand. All results are streamed back and formatted as a user-friendly graphical output.\n\nCheck out `gaia-pipeline.io`_ to learn more.\n\nMotivation\n==========\n\n.. begin-motivation\n\n*Automation Engineer*, *DevOps Engineer*, *SRE*, *Cloud Engineer*,\n*Platform Engineer* - they all have one in common:\nThe majority of tech people are not motivated to take up this work and they are hard to recruit.\n\nOne of the main reasons for this is the abstraction and poor execution of many automation tools. They come with their own configuration (`YAML`_ syntax) specification or limit the user to one specific programming language. Testing is nearly impossible because most automation tools lack the ability to mock services and subsystems. Even tiny things, for example parsing a JSON file, are sometimes really painful because external, outdated libraries were used and not included in the standard framework.\n\nWe believe it's time to remove all those abstractions and come back to our roots. Are you tired of writing endless lines of YAML-code? Are you sick of spending days forced to write in a language that does not suit you and is not fun at all? Do you enjoy programming in a language you like? Then Gaia is for you.\n\nHow does it work?\n=================\n\n.. begin-architecture\n\nGaia is based on `HashiCorp's go-plugin`_. It's a `plugin system`_ that uses `gRPC`_ to communicate over `HTTP/2`_. Initially, HashiCorp developed this tool for `Packer`_ but now it's heavily used by `Terraform`_, `Nomad`_, and `Vault`_ too.\n\nPlugins, also called `pipelines \u003cWhat is a pipeline?_\u003e`_, are applications which can be written in any programming language, as long as `gRPC`_ is supported. All functions, also called `jobs \u003cWhat is a job?\u003e`_, are exposed to Gaia and can form up a dependency graph that describes the order of execution.\n\nPipelines can be compiled locally or simply over the integrated build system. Gaia clones the git repository and automatically builds the included pipeline. If a change (`git push`_) happened, Gaia will automatically rebuild the pipeline for you*.\n\nAfter a pipeline has been started, all log output is returned back to Gaia and displayed in a detailed overview with their final result status.\n\nGaia uses `boltDB` for storage. This makes the installation process super easy. No external database is currently required.\n\n\\* *This requires polling or webhook to be activated.*\n\nScreenshots\n===========\n\n.. begin-screenshots\n\n|sh-login|\n|sh-overview|\n|sh-create-pipeline|\n|sh-pipeline-detailed|\n|sh-pipeline-logs|\n|sh-vault|\n|sh-settings|\n\nGetting Started\n===============\n\n.. begin-getting-started\n\nInstallation\n------------\n\nThe installation of gaia is simple and often takes a few minutes.\n\nUsing docker\n~~~~~~~~~~~~\n\nThe following command starts gaia as a daemon process and mounts all data to the current folder. Afterwards, gaia will be available on the host system on port 8080. Use the standard user **admin** and password **admin** as initial login. It is recommended to change the password afterwards.\n\n.. code:: sh\n\n    docker run -d -p 8080:8080 -v $PWD:/data gaiapipeline/gaia:latest\n\nThis uses the image with the *latest* tag which includes all required libraries and compilers for all supported languages. If you prefer a smaller image suited for your preferred language, have a look at the `available docker image tags`_.\n\nManually\n~~~~~~~~\n\nIt is possible to install Gaia directly on the host system.\nThis can be achieved by downloading the binary from the `releases page`_.\n\nGaia will automatically detect the folder of the binary and will place all data next to it. You can change the data directory with the startup parameter *-home-path* if you want.\n\nUsing helm\n~~~~~~~~~~\n\nIf you haven't got an ingress controller pod yet, make sure that you have `kube-dns` or `coredns` enabled, run this command to set it up.\n\n.. code:: sh\n\n    make kube-ingress\n\nTo init helm:\n\n.. code:: sh\n\n    helm init\n\nTo deploy gaia:\n\n.. code:: sh\n\n    make deploy-kube\n\nExample Pipelines\n-----\n\nGo\n~~~\n\n.. code:: go\n\n    package main\n\n    import (\n        \"log\"\n\n\tsdk \"github.com/gaia-pipeline/gosdk\"\n    )\n\n    // This is one job. Add more if you want.\n    func DoSomethingAwesome(args sdk.Arguments) error {\n        log.Println(\"This output will be streamed back to gaia and will be displayed in the pipeline logs.\")\n\n\t// An error occurred? Return it back so gaia knows that this job failed.\n\treturn nil\n    }\n\n    func main() {\n        jobs := sdk.Jobs{\n            sdk.Job{\n                Handler:     DoSomethingAwesome,\n\t        Title:       \"DoSomethingAwesome\",\n\t\tDescription: \"This job does something awesome.\",\n\t    },\n\t}\n\n\t// Serve\n\tif err := sdk.Serve(jobs); err != nil {\n\t    panic(err)\n\t}\n}\n\nPython\n~~~~~~~\n\n.. code:: python\n\n    from gaiasdk import sdk\n    import logging\n\n    def MyAwesomeJob(args):\n        logging.info(\"This output will be streamed back to gaia and will be displayed in the pipeline logs.\")\n        # Just raise an exception to tell Gaia if a job failed.\n        # raise Exception(\"Oh no, this job failed!\")\n\n    def main():\n        logging.basicConfig(level=logging.INFO)\n        myjob = sdk.Job(\"MyAwesomeJob\", \"Do something awesome\", MyAwesomeJob)\n        sdk.serve([myjob])\n\nJava\n~~~~\n\n.. code:: java\n\n    package io.gaiapipeline;\n\n    import io.gaiapipeline.javasdk.*;\n\n    import java.util.ArrayList;\n    import java.util.Arrays;\n    import java.util.logging.Logger;\n\n    public class Pipeline\n    {\n        private static final Logger LOGGER = Logger.getLogger(Pipeline.class.getName());\n\n        private static Handler MyAwesomeJob = (gaiaArgs) -\u003e {\n            LOGGER.info(\"This output will be streamed back to gaia and will be displayed in the pipeline logs.\");\n\t    // Just raise an exception to tell Gaia if a job failed.\n            // throw new IllegalArgumentException(\"Oh no, this job failed!\");\n        };\n\n        public static void main( String[] args )\n        {\n            PipelineJob myjob = new PipelineJob();\n            myjob.setTitle(\"MyAwesomeJob\");\n            myjob.setDescription(\"Do something awesome.\");\n            myjob.setHandler(MyAwesomeJob);\n\n            Javasdk sdk = new Javasdk();\n            try {\n                sdk.Serve(new ArrayList\u003c\u003e(Arrays.asList(myjob)));\n            } catch (Exception ex) {\n                ex.printStackTrace();\n            }\n        }\n    }\n\nC++\n~~~~\n\n.. code:: cpp\n\n   #include \"cppsdk/sdk.h\"\n   #include \u003clist\u003e\n   #include \u003ciostream\u003e\n\n   void DoSomethingAwesome(std::list\u003cgaia::argument\u003e args) throw(std::string) {\n      std::cerr \u003c\u003c \"This output will be streamed back to gaia and will be displayed in the pipeline logs.\" \u003c\u003c std::endl;\n\n      // An error occurred? Return it back so gaia knows that this job failed.\n      // throw \"Uhh something badly happened!\"\n   }\n\n   int main() {\n      std::list\u003cgaia::job\u003e jobs;\n      gaia::job awesomejob;\n      awesomejob.handler = \u0026DoSomethingAwesome;\n      awesomejob.title = \"DoSomethingAwesome\";\n      awesomejob.description = \"This job does something awesome.\";\n      jobs.push_back(awesomejob);\n\n      try {\n         gaia::Serve(jobs);\n      } catch (string e) {\n         std::cerr \u003c\u003c \"Error: \" \u003c\u003c e \u003c\u003c std::endl;\n      }\n   }\n\nRuby\n~~~~\n\n.. code:: ruby\n\n   require 'rubysdk'\n\n   class Main\n       AwesomeJob = lambda do |args|\n           STDERR.puts \"This output will be streamed back to gaia and will be displayed in the pipeline logs.\"\n\n           # An error occurred? Raise an exception and gaia will fail the pipeline.\n           # raise \"Oh gosh! Something went wrong!\"\n       end\n\n       def self.main\n           awesomejob = Interface::Job.new(title: \"Awesome Job\",\n                                           handler: AwesomeJob,\n                                           desc: \"This job does something awesome.\")\n\n           begin\n               RubySDK.Serve([awesomejob])\n           rescue =\u003e e\n               puts \"Error occured: #{e}\"\n               exit(false)\n           end\n       end\n   end\n\nNode.JS\n~~~~\n\n.. code:: javascript\n\n   const nodesdk = require('@gaia-pipeline/nodesdk');\n\n   function DoSomethingAwesome(args) {\n       console.error('This output will be streamed back to gaia and will be displayed in the pipeline logs.');\n\n       // An error occurred? Throw it back so gaia knows that this job failed.\n       // throw new Error('My error message');\n   }\n\n   // Serve\n   try {\n       nodesdk.Serve([{\n           handler: DoSomethingAwesome,\n           title: 'DoSomethingAwesome',\n           description: 'This job does something awesome.'\n       }]);\n   } catch (err) {\n       console.error(err);\n   }\n\nPipelines are defined by jobs and a function usually represents a job. You can define as many jobs in your pipeline as you want.\n\nEvery function accepts arguments. Those arguments can be requested from the pipeline itself and the values are passed back in from the UI.\n\nSome pipeline jobs need a specific order of execution. `DependsOn` allows you to declare dependencies for every job.\n\nYou can find real examples and more information on `how to develop a pipeline`_ in the docs.\n\nSecurity\n========\n\nSee the Documentation located here: `security-docs`_.\n\nDocumentation and more\n======================\n\nPlease find the docs at https://docs.gaia-pipeline.io. We also have a tutorials section over there with examples and real use-case scenarios. For example, `Kubernetes deployment with vault integration`_.\n\nQuestions and Answers (Q\u0026A)\n---------------------------\n\nWhat problem solves **Gaia**?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nLiterally every tool that was designed for automation, continuous integration (CI), and continuous deployment (CD) like Spinnaker, Jenkins, Gitlab CI/CD, TravisCI, CircleCI, Codeship, Bamboo and many more, introduced their own configuration format. Some of them don't even support *configuration/automation as code*. This works well for simple tasks like running a ``go install`` or ``mvn clean install`` but in the real world there is more to do.\n\nGaia is the first platform that does not limit the user and provides full support for almost all common programming languages without losing the features offered by todays CI/CD tools.\n\nWhat is a **pipeline**?\n~~~~~~~~~~~~~~~~~~~~~~~\nA pipeline is a real application with at least one function (we call it a Job). Every programming language can be used as long as gRPC is supported. We offer SDKs to support the development.\n\nWhat is a **job**?\n~~~~~~~~~~~~~~~~~~\nA job is a function, usually globally exposed to Gaia. Dependent on the dependency graph, Gaia will execute this function in a specific order.\n\nWhy do I need an **SDK**?\n~~~~~~~~~~~~~~~~~~~~~~~~~~\nThe SDK implements the Gaia plugin gRPC interface and offers helper functions like serving the gRPC-Server. This helps you to focus on the real problem instead of doing the boring stuff.\n\nWhich programming languages are supported?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nWe currently fully support Go, Java, Python, C++, Ruby and Node.JS.\n\nWhen do you support programming language **XYZ**?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nWe are working hard to support as much programming languages as possible but our resources are limited and we are also mostly no experts in all programming languages. If you are willing to contribute, feel free to open an issue and start working.\n\nRoadmap\n=======\n\nGaia is currently available as beta version.\n\nFeel free to open a new GitHub issue to request a new feature.\n\nContributing\n============\n\nGaia can only evolve and become a great product with the help of contributors. If you like to contribute, please have a look at our `issues section`_. We do our best to mark issues for new contributors with the label *good first issue*.\n\nIf you think you found a good first issue, please consider this list as a short guide:\n\n* If the issue is clear and you have no questions, please leave a short comment that you started working on this. The issue will be usually blocked for two weeks for you to solve it.\n* If something is not clear or you are unsure what to do, please leave a comment so we can add more detailed description.\n* Make sure your development environment is configured and set up. You need `Go installed`_ on your machine and also `nodeJS`_ for the frontend. Clone this repository and run the **make** command inside the cloned folder. This will start the backend. To start the frontend you have to open a new terminal window and go into the frontend folder. There you run **npm install** and then **npm run serve**. This should automatically open a new browser window.\n* Before you start your work, you should fork this repository and push changes to your fork. Afterwards, send a merge request back to upstream.\n\nContact\n=======\n\nIf you have any questions feel free to contact us on `slack`_.\n\n.. _`HashiCorp's go-plugin`: https://github.com/hashicorp/go-plugin\n.. _`gRPC`: https://grpc.io/\n.. _`Do not use it for mission critical jobs yet!`: https://tenor.com/view/enter-at-your-own-risk-gif-8912210\n.. _`YAML`: https://en.wikipedia.org/wiki/YAML\n.. _`releases page`: https://github.com/gaia-pipeline/gaia/releases\n.. _`Packer`: https://www.packer.io/\n.. _`Terraform`: https://www.terraform.io/\n.. _`Nomad`: https://www.nomadproject.io/\n.. _`Vault`: https://www.vaultproject.io/\n.. _`boltDB`: https://github.com/coreos/bbolt\n.. _`Unix nice level`: https://en.wikipedia.org/wiki/Nice_(Unix)\n.. _`issues section`: https://github.com/gaia-pipeline/gaia/issues\n.. _`Go installed`: https://golang.org/doc/install\n.. _`nodeJS`: https://nodejs.org/\n.. _`go-example repo`: https://github.com/gaia-pipeline/go-example\n.. _`slack`: https://slack.gaia-pipeline.io/\n.. _`Kubernetes deployment with vault integration`: https://docs.gaia-pipeline.io/tutorials/kube-vault-deploy/\n.. _`git push`: https://git-scm.com/docs/git-push\n.. _`HTTP/2`: https://http2.github.io/\n.. _`security-docs`: https://github.com/gaia-pipeline/gaia/blob/master/security/README.md\n.. _`plugin system`: https://en.wikipedia.org/wiki/Plug-in_(computing)\n.. _`available docker image tags`: https://hub.docker.com/r/gaiapipeline/gaia/tags/\n.. _`how to develop a pipeline`: https://docs.gaia-pipeline.io/develop-pipelines/\n.. _`gaia-pipeline.io`: https://gaia-pipeline.io/\n\n.. |build-status| image:: https://circleci.com/gh/gaia-pipeline/gaia/tree/master.svg?style=shield\u0026circle-token=c0e15edfb08f8076076cbbb55558af6cfecb89b8\n    :alt: Build Status\n    :scale: 100%\n    :target: https://circleci.com/gh/gaia-pipeline/gaia/tree/master\n\n.. |go-report| image:: https://goreportcard.com/badge/github.com/gaia-pipeline/gaia\n    :alt: Go Report Card\n    :target: https://goreportcard.com/report/github.com/gaia-pipeline/gaia\n\n.. |go-doc| image:: https://godoc.org/github.com/gaia-pipeline/gaia?status.svg\n    :alt: GoDoc\n    :target: https://godoc.org/github.com/gaia-pipeline/gaia\n\n.. |apache2| image:: https://img.shields.io/badge/license-Apache-blue.svg\n    :alt: Apache licensed\n    :target: https://github.com/gaia-pipeline/gaia/blob/master/LICENSE\n\n.. |chat| image:: https://gaia-slack-invite.herokuapp.com/badge.svg\n    :alt: Slack\n    :target: https://gaia-slack-invite.herokuapp.com/\n\n.. |codecov| image:: https://codecov.io/gh/gaia-pipeline/gaia/branch/master/graph/badge.svg\n    :target: https://codecov.io/gh/gaia-pipeline/gaia\n\n.. |sh-login| image:: https://github.com/gaia-pipeline/gaia/blob/master/screenshots/login.png\n    :alt: gaia login screenshot\n    :width: 650px\n\n.. |sh-overview| image:: https://github.com/gaia-pipeline/gaia/blob/master/screenshots/overview.png\n    :alt: gaia overview screenshot\n    :width: 650px\n\n.. |sh-create-pipeline| image:: https://github.com/gaia-pipeline/gaia/blob/master/screenshots/create-pipeline.png\n    :alt: gaia create pipeline screenshot\n    :width: 650px\n\n.. |sh-vault| image:: https://github.com/gaia-pipeline/gaia/blob/master/screenshots/vault.png\n    :alt: gaia Vault screenshot\n    :width: 650px\n\n.. |sh-pipeline-detailed| image:: https://github.com/gaia-pipeline/gaia/blob/master/screenshots/detail-pipeline.png\n    :alt: gaia pipeline detailed screenshot\n    :width: 650px\n\n.. |sh-pipeline-logs| image:: https://github.com/gaia-pipeline/gaia/blob/master/screenshots/logs-pipeline.png\n    :alt: gaia pipeline logs screenshot\n    :width: 650px\n\n.. |sh-settings| image:: https://github.com/gaia-pipeline/gaia/blob/master/screenshots/settings.png\n    :alt: gaia settings screenshot\n    :width: 650px\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaia-pipeline%2Fgaia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgaia-pipeline%2Fgaia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgaia-pipeline%2Fgaia/lists"}