{"id":13514081,"url":"https://github.com/raystack/optimus","last_synced_at":"2025-05-16T07:05:54.626Z","repository":{"id":36982054,"uuid":"350431276","full_name":"raystack/optimus","owner":"raystack","description":"Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.","archived":false,"fork":false,"pushed_at":"2024-06-08T13:29:09.000Z","size":12829,"stargazers_count":748,"open_issues_count":53,"forks_count":155,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-08T16:09:10.160Z","etag":null,"topics":["airflow","analytics","analytics-engineering","automation","bigquery","business-intelligence","data-modelling","data-pipelines","data-transformation","data-warehouse","dataops","elt","etl","golang","workflows"],"latest_commit_sha":null,"homepage":"https://raystack.github.io/optimus","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raystack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-22T17:28:04.000Z","updated_at":"2025-04-05T13:17:39.000Z","dependencies_parsed_at":"2024-10-13T01:40:35.919Z","dependency_job_id":"76733d16-30d8-4f4d-85ce-c0d61fa1c20d","html_url":"https://github.com/raystack/optimus","commit_stats":{"total_commits":461,"total_committers":26,"mean_commits":17.73076923076923,"dds":0.8134490238611713,"last_synced_commit":"5874457c7bbe9ed08f41c8300e984ac7fc28f78a"},"previous_names":["raystack/optimus","odpf/optimus"],"tags_count":72,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Foptimus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Foptimus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Foptimus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Foptimus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raystack","download_url":"https://codeload.github.com/raystack/optimus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254485062,"owners_count":22078767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","analytics","analytics-engineering","automation","bigquery","business-intelligence","data-modelling","data-pipelines","data-transformation","data-warehouse","dataops","elt","etl","golang","workflows"],"created_at":"2024-08-01T05:00:45.840Z","updated_at":"2025-05-16T07:05:49.616Z","avatar_url":"https://github.com/raystack.png","language":"Go","readme":"# Optimus\n\n[![verify workflow](https://github.com/raystack/optimus/actions/workflows/verify.yml/badge.svg)](verification)\n[![publish latest workflow](https://github.com/raystack/optimus/actions/workflows/publish-latest.yml/badge.svg)](build)\n[![Coverage Status](https://coveralls.io/repos/github/raystack/optimus/badge.svg?branch=main)](https://coveralls.io/github/raystack/optimus?branch=main)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg?logo=apache)](LICENSE)\n[![Version](https://img.shields.io/github/v/release/raystack/optimus?logo=semantic-release)](Version)\n\nOptimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management. It enables data analysts and engineers to transform their data by writing simple SQL queries and YAML configuration while Optimus handles dependency management, scheduling and all other aspects of running transformation jobs at scale.\n\n\u003cp align=\"center\" style=\"margin-top:30px\"\u003e\u003cimg src=\"./docs/static/img/optimus.svg\" /\u003e\u003c/p\u003e\n\n## Key Features\n\nDiscover why users choose Optimus as their main data transformation tool.\n\n- **Warehouse management:** Optimus allows you to create and manage your data warehouse tables and views through YAML based configuration.\n- **Scheduling:** Optimus provides an easy way to schedule your SQL transformation through a YAML based configuration.\n- **Automatic dependency resolution:** Optimus parses your data transformation queries and builds a dependency graphs automaticaly instead of users defining their source and taget dependencies in DAGs.\n- **Dry runs:** Before SQL query is scheduled for transformation, during deployment query will be dry-run to make sure it passes basic sanity checks.\n- **Powerful templating:** Optimus provides query compile time templating with variables, loop, if statements, macros, etc for allowing users to write complex tranformation logic.\n- **Cross tenant dependency:** Optimus is a multi-tenant service, if there are two tenants registered, serviceA and serviceB then service B can write queries referencing serviceA as source and Optimus will handle this dependency as well.\n- **Hooks:** Optimus provides hooks for post tranformation logic. e,g. You can sink BigQuery tables to Kafka.\n- **Extensibility:** Optimus support Python transformation and allows for writing custom plugins.\n- **Workflows:** Optimus provides industry proven workflows using git based specification management and REST/GRPC based specification management for data warehouse management.\n\n## Usage\n\nOptimus has two components, Optimus service that is the core orchestrator installed on server side, and a CLI binary used to interact with this service. You can install Optimus CLI using homebrew on macOS:\n\n```shell\n$ brew install raystack/tap/optimus\n$ optimus --help\n\nOptimus is an easy-to-use, reliable, and performant workflow orchestrator for\ndata transformation, data modeling, pipelines, and data quality management.\n\nUsage:\n  optimus [command]\n\nAvailable Commands:\n  backup      Backup a resource and its downstream\n  completion  Generate the autocompletion script for the specified shell\n  extension   Operate with extension\n  help        Help about any command\n  init        Interactively initialize Optimus client config\n  job         Interact with schedulable Job\n  migration   Command to do migration activity\n  namespace   Commands that will let the user to operate on namespace\n  playground  Play around with some Optimus features\n  plugin      Manage plugins\n  project     Commands that will let the user to operate on project\n  resource    Interact with data resource\n  secret      Manage secrets to be used in jobs\n  scheduler   Scheduled/run job related functions\n  serve       Starts optimus service\n  version     Print the client version information\n\nFlags:\n  -h, --help       help for optimus\n      --no-color   Disable colored output\n\nUse \"optimus [command] --help\" for more information about a command.\n```\n\n## Documentation\n\nExplore the following resources to get started with Optimus:\n\n- [Guides](https://raystack.github.io/optimus/docs/guides/create-job/) provides guidance on using Optimus.\n- [Concepts](https://raystack.github.io/optimus/docs/concepts/overview/) describes all important Optimus concepts.\n- [Reference](https://raystack.github.io/optimus/docs/reference/api/) contains details about configurations, metrics and other aspects of Optimus.\n- [Contribute](https://raystack.github.io/optimus/docs/contribute/contributing/) contains resources for anyone who wants to contribute to Optimus.\n\n## Running locally\n\nOptimus requires the following dependencies:\n\n- Golang (version 1.16 or above)\n- Git\n\nRun the following commands to compile `optimus` from source\n\n```shell\n$ git clone git@github.com:raystack/optimus.git\n$ cd optimus\n$ make\n```\n\nUse the following command to run\n\n```shell\n$ ./optimus version\n```\n\nOptimus service can be started with\n\n```shell\n$ ./optimus serve\n```\n\n`serve` command has few required configurations that needs to be set for it to start. Read more about it in [getting started](https://raystack.github.io/optimus/docs/getting-started/configuration).\n\n## Compatibility\n\nOptimus is currently undergoing heavy development with frequent, breaking API changes. Current major version is zero (v0.x.x) to accommodate rapid development and fast iteration while getting early feedback from users (feedback on APIs are appreciated). The public API could change without a major version update before v1.0.0 release.\n\n## Contribute\n\nDevelopment of Optimus happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving Optimus.\n\nRead our [contributing guide](https://raystack.github.io/optimus/docs/contribute/contributing) to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Optimus.\n\nTo help you get your feet wet and get you familiar with our contribution process, we have a list of [good first issues](https://github.com/raystack/optimus/labels/good%20first%20issue) that contain bugs which have a relatively limited scope. This is a great place to get started.\n\n## License\n\nOptimus is [Apache 2.0](LICENSE) licensed.\n","funding_links":[],"categories":["Go","golang"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraystack%2Foptimus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraystack%2Foptimus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraystack%2Foptimus/lists"}