{"id":13505410,"url":"https://github.com/facebookarchive/bistro","last_synced_at":"2025-09-27T10:30:51.484Z","repository":{"id":13610013,"uuid":"16303023","full_name":"facebookarchive/bistro","owner":"facebookarchive","description":"Bistro is a flexible distributed scheduler, a high-performance framework supporting multiple paradigms while retaining ease of configuration, management, and monitoring.","archived":true,"fork":false,"pushed_at":"2023-03-21T04:04:09.000Z","size":11254,"stargazers_count":1029,"open_issues_count":6,"forks_count":144,"subscribers_count":70,"default_branch":"main","last_synced_at":"2025-09-24T21:39:38.822Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://bistro.io","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookarchive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-01-28T05:26:00.000Z","updated_at":"2025-05-16T12:48:13.000Z","dependencies_parsed_at":"2024-05-15T23:03:20.919Z","dependency_job_id":"b5bbd093-48d7-4c04-b2ec-ec7148962aa3","html_url":"https://github.com/facebookarchive/bistro","commit_stats":null,"previous_names":["facebook/bistro"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/facebookarchive/bistro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookarchive%2Fbistro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookarchive%2Fbistro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookarchive%2Fbistro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookarchive%2Fbistro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookarchive","download_url":"https://codeload.github.com/facebookarchive/bistro/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookarchive%2Fbistro/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276888844,"owners_count":25722716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-25T02:00:09.612Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T00:01:05.869Z","updated_at":"2025-09-27T10:30:50.480Z","avatar_url":"https://github.com/facebookarchive.png","language":"C++","funding_links":[],"categories":["\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++","C++"],"sub_categories":[],"readme":"# Bistro: A fast, flexible toolkit for scheduling and running distributed tasks\n\n[![Build Status](https://travis-ci.org/facebook/bistro.svg?branch=master)](https://travis-ci.org/facebook/bistro)\n\nThis README is a very abbreviated introduction to Bistro. Visit\nhttp://facebook.github.io/bistro for a more structured introduction, and for the docs.\n\nBistro is a toolkit for making distributed computation systems. It can\nschedule and run distributed tasks, including data-parallel jobs.  It\nenforces resource constraints for worker hosts and data-access bottlenecks.\nIt supports remote worker pools, low-latency batch scheduling, dynamic\nshards, and a variety of other possibilities.  It has command-line and web\nUIs.\n\nSome of the diverse problems that Bistro solved at Facebook:\n - Safely run map-only ETL tasks against live production databases (MySQL,\n   HBase, Postgres).\n - Provide a resource-aware job queue for batch CPU/GPU compute jobs.\n - Replace Hadoop for a periodic online data compression task on HBase,\n   improving time-to-completion and reliability by over 10x.\n\nYou can run Bistro \"out of the box\" to suit a variety of different\napplications, but even so, it is a tool for engineers.  You should be able\nto get started just by reading the documentation, but when in doubt, look at\nthe code --- it was written to be read.\n\nSome applications of Bistro may involve writing small plugins to make it fit\nyour needs.  The code is built to be extensible.  Ask for tips, and we'll do\nour best to [help](https://www.facebook.com/groups/bistro.scheduler).  In\nreturn, we hope that you will send a pull request to allow us to share your\nwork with the community.\n\n## Early release\n\nAlthough Bistro has been in production at Facebook for over 3 years, the\npresent public release is partial, including just the server components.\n\n## Install the dependencies and build\n\nBistro needs a 64-bit Linux, Folly, FBThrift, Proxygen, boost, and\nlibsqlite3.  You need 2-3GB of RAM to build, as well as GCC 4.9 or above.\n\n`build/README.md` documents the usage of Docker-based scripts that build\nBistro on Ubuntu 14.04, 16.04, and Debian 8.6.  You should be able to follow\nvery similar steps on most modern Linux distributions.\n\nIf you run into dependency problems, look at `bistro/cmake/setup.cmake` for\na full list of Bistro's external dependencies (direct and indirect).  We\ngratefully accept patches that improve Bistro's builds, or add support for\nvarious flavors of Linux and Mac OS.\n\nThe binaries will be in `bistro/cmake/{Debug,Release}`.  Available build\ntargets are explained here:\n   http://cmake.org/Wiki/CMake_Useful_Variables#Compilers_and_Tools\nYou can start Bistro's unit tests by running `ctest` in those directories.\n\n## Your first Bistro run\n\nThis is just one simple demo, but Bistro is a very flexible tool. Refer to\nhttp://facebook.github.io/bistro/ for more in-depth information.\n\nWe are going to start a single Bistro scheduler talking to one 'remote'\nworker.\n\nAside: The scheduler tracks jobs, and data shards on which to execute them.\nIt also makes sure only to start new tasks when the required resources are\navailable.  The remote worker is a module for executing centrally scheduled\nwork on many machines.  The UI can aggregate many schedulers at once, so\nusing remote workers is optional --- a share-nothing, many-scheduler system\nis sometimes preferable.\n\nLet's make a task to execute:\n\n```\ncat \u003c\u003cEOF \u003e ~/demo_bistro_task.sh\n#!/bin/bash\necho \"I got these arguments: \\$@\"\necho \"stderr is also logged\" 1\u003e\u00262\necho \"done\" \u003e \"\\$2\"  # Report the task status to Bistro via a named pipe\nEOF\nchmod u+x ~/demo_bistro_task.sh\n```\n\nOpen two terminals, one for the scheduler, and one for the worker.\n\n```\n# In both terminals\ncd bistro/bistro\n# Start the scheduler in one terminal\n./cmake/Debug/server/bistro_scheduler \\\n  --server_port=6789 --http_server_port=6790 \\\n  --config_file=scripts/test_configs/simple --clean_statuses \\\n  --CAUTION_startup_wait_for_workers=1 --instance_node_name=scheduler\n# Start the worker in another\nmkdir /tmp/bistro_worker\n./cmake/Debug/worker/bistro_worker --server_port=27182 --scheduler_host=:: \\\n  --scheduler_port=6789 --worker_command=\"$HOME/demo_bistro_task.sh\" \\\n  --data_dir=/tmp/bistro_worker\n```\n\nYou should be seeing some lively log activity on both terminals. In several\nseconds, the worker-scheduler negotiation should complete, and you should\nsee messages like \"Task ...  quit with status\" and \"Got status\".\n\nSince we passed `--clean_statuses`, the scheduler will not persist any task\ncompletions that happened during this run.  The worker, on the other hand,\nwill keep a record of the task logs in `/tmp/bistro_worker/task_logs.sql3`.\n\nIf you want task completions to persist across runs, tell Bistro where to\nput the SQLite database, via `--data_dir=/tmp/bistro_scheduler` and\n`--status_table=task_statuses`\n\n```\nmkdir /tmp/bistro_scheduler\n./cmake/Debug/server/bistro_scheduler \\\n  --server_port=6789 --http_server_port=6790 \\\n  --config_file=scripts/test_configs/simple \\\n  --data_dir=/tmp/bistro_scheduler --status_table=task_statuses \\\n  --CAUTION_startup_wait_for_workers=1 --instance_node_name=scheduler\n```\n\nYou can query the running scheduler via its REST API:\n\n```\ncurl -d '{\"a\":{\"handler\":\"jobs\"},\"b\":{\"handler\":\"running_tasks\"}}' :::6790\ncurl -d '{\"my subquery\":{\"handler\":\"task_logs\",\"log_type\":\"stdout\"}}' :::6790\n```\n\n**Pro-tip:** For ease of reading, pipe the output through either `jq` or\n`json_pp` (from a Perl package). For longer outputs, try `| jq -C . | less -R`.\n\nYou should also take a look at the scheduler configuration to see how its\njobs, nodes, and resources were specified.\n\n```\nless scripts/test_configs/simple\n```\n\nFor debugging, we typically invoke the binaries like this:\n\n```\ngdb cmake/Debug/worker/bistro_worker -ex \"r ...\" 2\u003e\u00261 | tee WORKER.txt\n```\n\nWhen configuring a real deployment, be sure to carefully review the `--help`\nof the scheduler \u0026 worker binaries, as well as the documentation on\nhttp://facebook.github.io/bistro.  And don't hesitate to ask for help in the group:\nhttps://www.facebook.com/groups/bistro.scheduler\n\n## License\n\nSee [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookarchive%2Fbistro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookarchive%2Fbistro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookarchive%2Fbistro/lists"}