{"id":13586495,"url":"https://github.com/fastlane-queue/fastlane","last_synced_at":"2025-04-07T15:32:06.132Z","repository":{"id":33637247,"uuid":"143786613","full_name":"fastlane-queue/fastlane","owner":"fastlane-queue","description":"Fastlane is a redis and docker based queueing service.","archived":true,"fork":false,"pushed_at":"2022-12-08T01:32:15.000Z","size":927,"stargazers_count":48,"open_issues_count":16,"forks_count":14,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-04-14T22:37:13.797Z","etag":null,"topics":["docker","queue","task-manager","worker"],"latest_commit_sha":null,"homepage":"https://fastlane.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fastlane-queue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-06T21:44:15.000Z","updated_at":"2024-03-12T14:11:11.000Z","dependencies_parsed_at":"2023-01-15T01:46:17.168Z","dependency_job_id":null,"html_url":"https://github.com/fastlane-queue/fastlane","commit_stats":null,"previous_names":[],"tags_count":66,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastlane-queue%2Ffastlane","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastlane-queue%2Ffastlane/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastlane-queue%2Ffastlane/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fastlane-queue%2Ffastlane/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fastlane-queue","download_url":"https://codeload.github.com/fastlane-queue/fastlane/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247679640,"owners_count":20978091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","queue","task-manager","worker"],"created_at":"2024-08-01T15:05:36.533Z","updated_at":"2025-04-07T15:32:05.630Z","avatar_url":"https://github.com/fastlane-queue.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"![fastlane](fastlane-logo.svg)\n\n[![github repo](https://img.shields.io/badge/github-repo-blue.svg)](https://github.com/fastlane-queue/fastlane) [![Build Status](https://travis-ci.org/fastlane-queue/fastlane.svg?branch=master)](https://travis-ci.org/fastlane-queue/fastlane) [![Codacy Badge](https://api.codacy.com/project/badge/Coverage/f1325bf0a8e8419692487d3b5d2fa36b)](https://www.codacy.com/app/heynemann/fastlane?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=fastlane-queue/fastlane\u0026amp;utm_campaign=Badge_Coverage) [![Docs](https://readthedocs.org/projects/fastlane/badge/?version=latest)](https://fastlane.readthedocs.io/en/latest/?badge=latest) [![Codacy Badge](https://api.codacy.com/project/badge/Grade/f1325bf0a8e8419692487d3b5d2fa36b)](https://www.codacy.com/app/heynemann/fastlane?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=fastlane-queue/fastlane\u0026amp;utm_campaign=Badge_Grade) [![BCH compliance](https://bettercodehub.com/edge/badge/fastlane-queue/fastlane?branch=master)](https://bettercodehub.com/results/fastlane-queue/fastlane) [![Issues](https://img.shields.io/github/issues/fastlane-queue/fastlane.svg)](https://github.com/fastlane-queue/fastlane/issues)\n\n[![demo](https://asciinema.org/a/219455.svg)](https://asciinema.org/a/219455)\n\n## [fastlane](https://github.com/fastlane-queue/fastlane) service\n\n[fastlane](https://github.com/fastlane-queue/fastlane) is a [redis](https://redis.io/)-based queueing service that outsmarts everyone else by using containers.\n\nMore seriously, though, [fastlane](https://github.com/fastlane-queue/fastlane) allows you to easily implement new workers in the form of containers.\n\nInstead of the tedious, repetitive work of yesteryear where you had to implement a worker in language X or Y, you just spin a new container with all the dependencies you require already previously installed, and instruct [fastlane](https://github.com/fastlane-queue/fastlane) to run a command in that container. Bang! Instant Super-Powered Workers!\n\n## Features\n\n-   [x]  [Ad-Hoc execution of jobs (run job right now)](tests/func/test_adhoc.py#L11)\n-   [x]  [Scheduled execution of jobs (run job next sunday at 6am, or run in 10 minutes from now)](tests/func/test_scheduled.py#L13);\n-   [x]  [Crontab execution of jobs (run job at `*/10 * * * *` - every ten minutes)](tests/func/test_cron.py#L11);\n-   [x]  [Allows job details to be updated](tests/func/test_update_details.py#L13);\n-   [x]  [API to retrieve tasks, `/tasks`](tests/func/test_get_tasks.py#L11);\n-   [x]  API to retrieve task details, `/tasks/my-task` (`\u003ctaskUrl\u003e`);\n-   [x]  API to retrieve job details, `\u003ctaskUrl\u003e/jobs/\u003cjobId\u003e` (`\u003cjobUrl\u003e`);\n-   [x]  API to stop running job (`\u003cjobUrl\u003e/stop`);\n-   [x]  API to retry job (`\u003cjobUrl\u003e/retry`);\n-   [x]  API to get logs(`\u003cjobUrl\u003e/logs`), stdout (`\u003cjobUrl\u003e/stdout`) and stderr (`\u003cjobUrl\u003e/stderr`) for last execution in jobs;\n-   [x]  Job log output streaming using WebSockets (`ws://\u003cjobUrl\u003e/ws`) and `\u003cjobUrl\u003e/stream`;\n-   [x]  API to retrieve execution details, `\u003cjobUrl\u003e/executions/\u003cexecutionId\u003e` (`\u003cexecutionUrl\u003e`);\n-   [x]  API to stop execution, `\u003cexecutionUrl\u003e/stop`;\n-   [x]  API to get logs(`\u003cexecutionUrl\u003e/stdout`), stdout (`\u003cexecutionUrl\u003e/stdout`) and stderr (`\u003cexecutionUrl\u003e/stderr`) for execution;\n-   [x]  Job execution log output streaming using WebSockets (`ws://\u003cexecutionUrl\u003e/ws`) and `\u003cexecutionUrl\u003e/stream`;\n-   [x]  Additional Job Metadata (useful for webhooks);\n-   [x]  Configurable retries per job;\n-   [x]  Configurable exponential back-off for retries and failures in monitoring of jobs;\n-   [x]  Configurable hard timeout for each execution;\n-   [x]  E-mail subscription to tasks;\n-   [x]  Healthcheck of the MongoDB and Redis Connections;\n-   [ ]  Web hooks on job start;\n-   [x]  Web hooks on job completion;\n-   [x]  Redact any env that contains blacklisted keywords;\n-   [ ]  Exponential back-off parameters per job;\n-   [x]  Self-healing handling of interrupted jobs;\n-   [ ]  Self-healing handling of unscheduled periodical jobs;\n-   [x]  Workers should handle SIGTERM and exit gracefully;\n-   [x]  [Docker](https://docs.docker.com/) Container Runner (with [docker](https://docs.docker.com/) host pool);\n-   [x]  [Docker](https://docs.docker.com/) Pool per task name (Regular Expressions);\n-   [x]  Rename [docker](https://docs.docker.com/) containers after processing their details;\n-   [x]  Command to prune processed containers;\n-   [x]  Routes to remove/put back [docker](https://docs.docker.com/) host in job balancing;\n-   [ ]  [Docker](https://docs.docker.com/) SSL connections;\n-   [ ]  Circuit breaking when [docker](https://docs.docker.com/) host is unavailable;\n-   [x]  Container Environment Variables per Job;\n-   [x]  Configurable global limit for number of running jobs per task name (Regular Expressions);\n-   [ ]  Limit of concurrent job executions per task;\n-   [ ]  Kubernetes Container Runner;\n-   [x]  [MongoDB](https://www.mongodb.com/) Task and Job Storage;\n-   [x]  Structured Logging;\n-   [x]  Monitoring of job completion;\n-   [x]  Job Expiration;\n-   [x]  Status Page with details on the farm status (executors, scheduled tasks and queue sizes);\n-   [x]  Error handling mechanism (Sentry built-in, extensible)\n-   [ ]  Per-job Error handling mechanism (Sentry built-in, extensible)\n-   [ ]  Usage metrics (extensible);\n-   [x]  Support [Redis](https://redis.io/) and [Redis](https://redis.io/) Sentinel;\n-   [ ]  Support [Redis](https://redis.io/) Cluster;\n-   [ ]  Comprehensive test coverage;\n-   [x]  CORS headers in every API request (configurable);\n-   [x]  Basic Auth support;\n-   [x]  gzip all JSON responses for the API (for requests that accept gzip);\n-   [x]  Store IP address of enqueued job for auditing (`X-Real-IP`, then `X-Forwarded-For`, then `request.addr`);\n-   [ ]  Admin to inspect tasks and jobs.\n\n## Getting started\n\nGetting [fastlane](https://github.com/fastlane-queue/fastlane) up and running is very simple if you have both [docker](https://docs.docker.com/) and [docker-compose](https://docs.docker.com/compose/) installed.\n\nWe'll use a sample docker compose that gets all our requirements up (ps: this [docker-compose](https://docs.docker.com/compose/) file runs [Docker In Docker](https://hub.docker.com/_/docker/) and requires privileged mode to run):\n\n```bash\n$ curl https://raw.githubusercontent.com/fastlane-queue/fastlane/master/docker-compose-sample.yml | docker-compose -f - up -d\n\nStarting fastlane...\nCreating fastlane_mongo_1       ... done\nCreating fastlane_docker-host_1 ... done\nCreating fastlane_redis_1       ... done\nCreating fastlane_fastlane_1    ... done\nfastlane started successfully.\n```\n\nAfter that you can start using [fastlane](https://github.com/fastlane-queue/fastlane). For more details on getting started, read the [following page](https://fastlane.readthedocs.io/en/latest/getting-started/).\n\n## Documentation\n\nRead more about fastlane at [read the docs](https://fastlane.readthedocs.io/en/latest/).\n\n## Contributing\n\nLogo was created by [Artur Sousa](https://github.com/arturfsousa).\n\n## License\n\nFastlane is licensed under the [MIT license](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffastlane-queue%2Ffastlane","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffastlane-queue%2Ffastlane","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffastlane-queue%2Ffastlane/lists"}