{"id":13425738,"url":"https://github.com/spotify/styx","last_synced_at":"2025-03-15T20:31:10.595Z","repository":{"id":13564510,"uuid":"74431644","full_name":"spotify/styx","owner":"spotify","description":"\"The path to execution\", Styx is a service that schedules batch data processing jobs in Docker containers on Kubernetes.","archived":true,"fork":false,"pushed_at":"2023-07-12T07:56:05.000Z","size":9731,"stargazers_count":267,"open_issues_count":24,"forks_count":50,"subscribers_count":71,"default_branch":"master","last_synced_at":"2025-03-09T22:34:32.387Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spotify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-22T03:44:55.000Z","updated_at":"2024-08-08T14:10:46.000Z","dependencies_parsed_at":"2024-05-01T20:44:45.476Z","dependency_job_id":"766a5a4e-bf38-4c39-97e6-a2539a8c9140","html_url":"https://github.com/spotify/styx","commit_stats":null,"previous_names":[],"tags_count":448,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fstyx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fstyx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fstyx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fstyx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spotify","download_url":"https://codeload.github.com/spotify/styx/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243790948,"owners_count":20348378,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T00:01:17.691Z","updated_at":"2025-03-15T20:31:08.959Z","avatar_url":"https://github.com/spotify.png","language":"Java","funding_links":[],"categories":["Java","容器化工具"],"sub_categories":[],"readme":"# This repo was archived! \nWe decided to discontinue the Styx oss repo.\n\n# Styx\n\n[![CircleCI](https://circleci.com/gh/spotify/styx/tree/master.svg?style=shield)](https://circleci.com/gh/spotify/styx)\n[![Coverage Status](https://codecov.io/gh/spotify/styx/branch/master/graph/badge.svg)](https://codecov.io/gh/spotify/styx)\n[![License](https://img.shields.io/github/license/spotify/styx.svg)](LICENSE)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.spotify/styx/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.spotify/styx)\n\nA batch job scheduler for Kubernetes\n\n## Description\n\nStyx is a service that is used to trigger periodic invocations of Docker containers. The information\nneeded to schedule such invocations, is read from a set of files on disk or an external service\nproviding such information. The service takes responsibility for triggering and possibly also\nre-triggering invocations until a successful exit status has been emitted or some other limit has\nbeen reached. Styx is built using the [Apollo] framework and uses [Kubernetes] for container\norchestration.\n\nStyx can optionally provide some dynamic arguments to container executions that indicates which time\nperiod a particular invocation belongs to. For example an hourly job for the first hour of\n2016-01-01 might have the dynamic argument `2016-01-01T00` appended to the container invocation.\n\nThe envisioned main use case for Styx is to execute data processing job, possibly long running\nprocesses that transform data periodically. Its initial use case is to run workflows of jobs\norchestrated using [Luigi], but it does not have any intrinsic ties to Luigi. Styx can just as well\nexecute a container with some simple bash scripts.\n\nStyx was built to function smoothly on Google Cloud Platform, thus it makes use of Google products\nsuch as Google Cloud Datastore, Google Cloud Bigtable and Google Container Engine. However, the\nintegrations with these products are all done through clear interfaces and other backends can easily\nbe added.\n\n## Key concepts\n\nThe key concept that Styx concerns itself with is Workflows. A Workflow is either enabled or\ndisabled and has a Schedule. A Schedule specifies how often a Workflow should be triggered, which\nDocker image to run and which arguments to pass to it on each execution. Each time a Workflow is\ntriggered, a Workflow Instance is created. The Workflow instance is tracked as 'active' until at\nleast one execution of the Docker image returns with a 0 exit code. Styx keeps track of Workflow\nInstance executions and provides information about them via the API.\n\n## Development status\n\nStyx is actively being developed and deployed internally at Spotify where it is being used to run\nmore than 10000 production workflows. Because of how we build and integrate infrastructure components at\nSpotify, this repository does not contain a GUI at the time of writing, while we do have one\ninternally. The goal is to break out more of these components into open source projects that\ncomplement each other.\n\n### More docs\n\n* [Styx design]\n* [External services]\n* [API Specification](doc/api.apib) - [HTML version](https://spotify.github.io/styx/api.html)\n\n## Usage\n\n### Setup\n\nA fully functional Service can be found in [styx-standalone-service](./styx-standalone-service).\nThis packaging contains both the API and Scheduler service in one artifact. This is how you build\nand run it.\n\nThe following configuration keys in\n[`styx-standalone.conf`](./styx-standalone-service/src/main/resources/styx-standalone.conf) have\nto be specified for the service to work:\n\n```yaml\n# Google Container Engine (GKE) cluster\nstyx.gke.default.project-id = \"\"\nstyx.gke.default.cluster-zone = \"\"\nstyx.gke.default.cluster-id = \"\"\nstyx.gke.default.namespace = \"\"\n\n# Google Cloud Bigtable instance\nstyx.bigtable.project-id = \"\"\nstyx.bigtable.instance-id = \"\"\n\n# Google Cloud Datastore config\nstyx.datastore.project-id = \"\"\nstyx.datastore.namespace = \"\"\n```\n\nBuild the project:\n\n```bash\n\u003e mvn package\n```\n\nRun the service:\n\n```bash\n\u003e java -jar styx-standalone-service/target/styx-standalone-service.jar\n```\n\n### Workflow configuration\n\nRefer to [API Specification](https://spotify.github.io/styx/api.html#workflows-workflows-post) for how to deploy a workflow.\n\n```\nid: my-workflow\ndocker_image: my-workflow:0.1\ndocker_args: ['./run.sh', '{}']\nschedule: hourly\noffset: PT1H\nservice_account: my-service-account@my-project.iam.gserviceaccount.com\nrunning_timeout: PT2H\nretry_condition: \"(#tries \u003c 2 \u0026\u0026 #triggerType == 'backfill') || (#triggerType != 'backfill')\"\n```\n\n#### `id` **[string]**\nA unique identifier for the workflow (lower-case-hyphenated). This identifier is used to refer to\n the workflow through the API.\n\n#### `docker_image` **[string]**:\nThe Docker image that should be executed.\n\n#### `docker_args` **[string]**\nThe list of arguments passed to the Docker container.\n\nThis list should only contain strings. Any occurrences of the `{}` placeholder argument will be\nreplaced with the current partition date or datehour. Note that it must be quoted in the yaml file\nin order not to be interpreted as an object.\n\nExample arguments for the supported schedule values:\n```\n- hourly - 2016-04-01T14, 2016-04-01T15, ... (UTC hours)\n- daily  - 2016-04-01,    2016-04-02,    ...\n- weekly - 2016-04-04,    2016-04-11,    ... (Mondays)\n```\n\n#### `schedule` **[string]**\nHow often the workflow should be triggered and what the `{}` placeholder will be replaced with in\n`docker_args`.\n\nSupports [cron] syntax, along with a set of human readable aliases:\n```\n@hourly,   hourly   = 0 * * * *\n@daily,    daily    = 0 0 * * *\n@weekly,   weekly   = 0 0 * * MON\n@monthly,  monthly  = 0 0 1 * *\n@yearly,   yearly   = 0 0 1 1 *\n@annually, annually = 0 0 1 1 *\n```\n\n#### `offset` **[string]**\nAn [ISO 8601 Duration] specification for offsetting the cron schedule.\n\nThis is useful for when setting up a schedule that needs to be offset in time relative to the\nschedule timestamps. For instance, an hourly schedule that needs to process a bucket of data\nfor each hour will not be able to run until at the end of that hour. We can then use an offset\nvalue of `PT1H`. The injected placeholder would reflect a logical time of the schedule\n(00, 01, 02, ...) one hour earlier than the actual run time (01, 02, 03, ...). This is specially\nuseful for irregular schedules.\n\nIn fact, it is so common that we need to use a \"last hour\" parameter in jobs that we've set the\ndefault offset for all the well known (aliased) schedules to +1 period. E.g for an `@hourly`\nschedule, the default offset is `PT1H`, and for a `@daily` schedule the offset is `P1D`\n\nExample: a job needs to run daily at 2 AM but the partition argument needs to be midnight\n\n```yaml\nschedule: '@daily'\noffset: P1DT2H\n```\n\nAt 2017-06-30T02 the execution for 2017-06-29 will be triggered.\n\n#### `service_account` **[email address]**\nThe [Service Account] email address belonging to a project in [Google Cloud Platform].\n\nIf the workflow intends to use keys of a [Service Account],\nStyx will create both JSON and p12 keys for the specified `service_account`, rotate keys on daily basis,\nand garbage collect unused keys older than 48h.\n\nStyx stores the created keys in [Kubernetes Secrets] and mounts them under `/etc/styx-wf-sa-keys/`\nin the container.\n\nStyx injects an environment variable to the container named as `GOOGLE_APPLICATION_CREDENTIALS` pointing\nto the JSON key file.\n\nIn order for Styx to be able to create/delete keys for the `service_account` of a workflow,\nthe [Service Account] that Styx itself runs as should be granted `Service Account Key Admin`\nrole for the `service_account` of the workflow.\n\nIf authorization is enabled for the service, the `service_account` will be used to authorize\ndeployments and actions (create/modify/delete, trigger a new instance, retry/halt an\nexisting instance and create backfill) on the workflow. To authorize an account, grant it the\n[configured role]) for the [Service Account] of the workflow.\n\nFor information on how to grant an account a role in a [Service Account], follow this\nguide: [Granting Roles to Service Accounts].\n\n#### `env` **[dictionary]**\n\nCustom environment variables to be injected into running container.\n\n#### `running_timeout` **[string]**\nAn [ISO 8601 Duration] specification for timing out container execution. The default is configurable in styx conf file through `styx.stale-state-ttls.running`.\nIf not set it defaults to `styx.stale-state-ttls.default`.\nThe upper boundary of the running_timeout is configurable through `styx.max-running-timeout`.\nIf not set it defaults to `styx.stale-state-ttls.running`, which defaults to `styx.stale-state-ttls.default` as stated above.\n\n#### `retry_condition` **[string]**\nA [SpEL](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#expressions) boolean expression.\nIf the expression evaluates to `false`, Styx will stop retrying and halt the workflow instance immediately. This\nconfiguration has no impact on possible max number of tries, meaning it can only be used to halt workflow instance\nearlier.\n\nThe following variables will be injected by Styx so that they can be used in the expression:\n* \\#exitCode: the exit code from the last execution\n* \\#tries: total number of tries, which equals to `1` when the first time a workflow instance gets executed and `2` when\n  the first retry is issued\n* \\#consecutiveFailures: total number of consecutive failures that is not `missing dependency`\n* \\#triggerType: `natural`, `backfill` or `ad-hoc`\n\n### Triggering and executions\n\nEach time a Workflow Schedule is triggered, Styx will treat that trigger as a first class entity.\nEach Trigger will have at least one Execution which can potentially take a long time to execute. If\nanother Trigger happens during this time, both triggers will be active, each with one running\ncontainer. Because Styx treats each Trigger individually, it can ensure that each one of them\ncomplete successfully.\n\nStyx does not assume anything about what is executed in the container, it only cares about the exit\ncode. Any execution returning a non-zero exit code will either cause a re-try to be scheduled, or be\ninterpreted as a permanent failure of the workflow instance. For detailed description of exit codes,\nplease refer to **Workflow state graph** section in [Styx design].\n\n### Injected environment variables\n\nFor each execution, Styx will inject a set of environment variables into the Docker container.\n\n| Variable Name | Description |\n|---|---|\n| `STYX_COMPONENT_ID` | The component id of the workflow. This will be the filename of the file which defines the workflow schedule. |\n| `STYX_WORKFLOW_ID` | The workflow id of the workflow. This is the `id` field specified in the workflow schedule. |\n| `STYX_PARAMETER` | The parameter argument. See section about `docker_args` above. |\n| `STYX_SERVICE_ACCOUNT` | The service account. |\n| `STYX_COMMIT_SHA` | The commit-sha of the workflow. |\n| `STYX_DOCKER_ARGS` | The arguments passed to the container. |\n| `STYX_DOCKER_IMAGE` | The docker image. |\n| `STYX_TRIGGER_ID` | The ID of the trigger. |\n| `STYX_TRIGGER_TYPE` | The type of the trigger. Possible values are: `natural`, `adhoc`, `backfill` and `unknown` |\n| `STYX_EXECUTION_ID` | A unique identifier for the execution. This is the execution id used to identify execution attempts of a trigger. |\n| `STYX_LOGGING`  | Container logging format, `text` or `structured`. |\n| `STYX_ENVIRONMENT`  | Styx environment, `staging`, `production`, `testing`, etc. Should be defined in [`styx-standalone.conf`](./styx-standalone-service/src/main/resources/styx-standalone.conf). |\n| `STYX_EXECUTION_COUNTER` | **to be implemented** - A counter indicating which execution this is. Goes from 0..N per trigger. |\n\n\n### High availability\n\nSince version 2.0, Styx supports full HA (High Availability) where both [styx-api-service] and [styx-scheduler-service] can be set up\nto have multiple instances.\n\n### Authorization\n\nEnabling authorization means that any workflow with a configured `service_account` will only allow authorized\nusers to deploy and manage it.\n\nYou can enable authorization in your configuration, either for [all workflows] or a [subset of workflows].\nYou will also need to provide the [name of the role][configured role] to use for determining if an account is an authorized\nuser of the `service_account` or not. [Read more about how to authorize accounts for a service account here](#service_account-email-address).\n\n## Development\n\n### Backwards Compatibility \u0026 API Stability\n\nMost core features and Styx APIs are considered to be stable and changes to them should be backwards compatible. Styx users should normally not be affected by minor/patch releases. \n\nStable features and APIs:\n\n* Styx CLI\n* Styx REST API\n* Styx Client API\n* Styx Workflows\n\nAll other APIs are considered unstable and can change at any time. E.g. the `StyxScheduler` class offers some plugin APIs for customizing functionality but they might change at any time. This should only affect engineers that link against and customize the Styx services.\n\n\n### Code Coverage\n\nAn aggregate code coverage report for the entire project is created by the `styx-report` submodule.\n\n```bash\n\u003e mvn clean verify\n\u003e open styx-report/target/site/jacoco-aggregate/index.html\n```\n\nCircleCI builds submit code coverage reports to [codecov.io]. In addition, the aggregate\nJaCoCo report can be viewed under the Artifacts tab in the CircleCI build view.\n\n---\n\nThis project adheres to the [Open Code of Conduct][code-of-conduct]. By participating, you are\nexpected to honor this code.\n\n[Styx design]: doc/design-overview.md\n[External services]: doc/external-services.md\n[Kubernetes]: http://kubernetes.io/\n[Apollo]: https://spotify.github.io/apollo/\n[Luigi]: https://github.com/spotify/luigi\n[code-of-conduct]: https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md\n[cron]: https://en.wikipedia.org/wiki/Cron\n[ISO 8601 Duration]: https://en.wikipedia.org/wiki/ISO_8601#Durations\n[Kubernetes Secrets]: https://kubernetes.io/docs/concepts/configuration/secret/\n[Service Account]: https://cloud.google.com/compute/docs/access/service-accounts\n[Google Cloud Platform]: https://cloud.google.com/\n[Granting Roles to Service Accounts]: https://cloud.google.com/iam/docs/granting-roles-to-service-accounts\n[codecov.io]: https://codecov.io/gh/spotify/styx\n[styx-api-service]: https://github.com/spotify/styx/tree/master/styx-api-service\n[styx-scheduler-service]: https://github.com/spotify/styx/tree/master/styx-scheduler-service\n[configured role]: ./styx-standalone-service/src/main/resources/styx-standalone.conf#L66\n[all workflows]: ./styx-standalone-service/src/main/resources/styx-standalone.conf#L73\n[subset of workflows]: ./styx-standalone-service/src/main/resources/styx-standalone.conf#L77\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fstyx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspotify%2Fstyx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fstyx/lists"}