{"id":21293126,"url":"https://github.com/criteo/mesos-command-modules","last_synced_at":"2025-07-11T16:32:20.345Z","repository":{"id":54373751,"uuid":"127880807","full_name":"criteo/mesos-command-modules","owner":"criteo","description":"Mesos modules running external commands","archived":false,"fork":false,"pushed_at":"2021-03-02T09:44:19.000Z","size":2475,"stargazers_count":3,"open_issues_count":3,"forks_count":5,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-04-13T05:52:23.778Z","etag":null,"topics":["isolator","mesos-modules"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/criteo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-03T09:03:55.000Z","updated_at":"2021-02-22T16:09:31.000Z","dependencies_parsed_at":"2022-08-13T13:41:01.182Z","dependency_job_id":null,"html_url":"https://github.com/criteo/mesos-command-modules","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fmesos-command-modules","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fmesos-command-modules/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fmesos-command-modules/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Fmesos-command-modules/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/criteo","download_url":"https://codeload.github.com/criteo/mesos-command-modules/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225741166,"owners_count":17516895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["isolator","mesos-modules"],"created_at":"2024-11-21T13:53:29.417Z","updated_at":"2024-11-21T13:53:30.036Z","avatar_url":"https://github.com/criteo.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mesos Command Modules\n\n[![Build Status](https://api.travis-ci.org/criteo/mesos-command-modules.svg?branch=master)](https://travis-ci.org/criteo/mesos-command-modules)\n\nThis repository is a collection of mesos modules\ndelegating the events to external commands.\n\nHere is the list of supported modules:\n- Hook (all methods)\n- Isolator (prepare, isolate, cleanup, watch, usage methods)\n\n## Production readyness\n\nThis project has been running in production on our infrastructure for since June 2018. It is called on every container start/stop and to monitor metrics.\nIt represent around 4000 calls / second.\n\n## Getting Started\n\nFollow the [build instructions](#build-instructions) to build the modules for\nyour architecture and your own version of Mesos.\n\nCreate or edit the file containing the modules configuration for your Mesos\ncluster and add the following libraries with whatever commands you want\nto run for each type of events. All commands are optional so they can be\nremoved from this configuration.\n\nFor each module definition, `module_name` key is mandatory.\n\nNote: the `debug` parameter is optional and can be set to true if you want to\nenable the logging of all inputs received and all outputs produced by the\ncommands.\n\n\n```\n{\n  \"libraries\": [\n    {\n      \"file\": \"/tmp/mesos/modules/libmesos_command_modules.so\",\n      \"modules\": [\n        {\n          \"name\": \"com_criteo_mesos_CommandHook\",\n          \"parameters\": [\n            {\n              \"key\": \"hook_slave_run_task_label_decorator_command\",\n              \"value\": \"/opt/mesos/modules/slaveRunTaskLabelDecorator.sh\"\n            },\n            {\n              \"key\": \"hook_slave_executor_environment_decorator_command\",\n              \"value\": \"/opt/mesos/modules/slaveExecutorEnvironmentDecorator.sh\"\n            },\n            {\n              \"key\": \"hook_slave_remove_executor_hook_command\",\n              \"value\": \"/opt/mesos/modules/slaveRemoveExecutorHook.sh\"\n            },\n            {\n              \"key\": \"module_name\",\n              \"value\": \"example_hook\"\n            },\n            {\n              \"key\": \"debug\",\n              \"value\": \"false\"\n            }\n          ]\n        },\n        {\n          \"name\": \"com_criteo_mesos_CommandIsolator\",\n          \"parameters\": [\n            {\n              \"key\": \"isolator_prepare_command\",\n              \"value\": \"/opt/mesos/modules/prepare.sh\"\n            },\n            {\n              \"key\": \"isolator_prepare_timeout\",\n              \"value\": \"10\"\n            },\n            {\n              \"key\": \"isolator_cleanup_command\",\n              \"value\": \"/opt/mesos/modules/cleanup.sh\"\n            },\n            {\n              \"key\": \"isolator_watch_command\",\n              \"value\": \"/opt/mesos/modules/watch.sh\"\n            },\n            {\n              \"key\": \"isolator_watch_frequence\",\n              \"value\": \"10\"\n            },\n            {\n              \"key\": \"module_name\",\n              \"value\": \"example_isolator\"\n            },\n            {\n              \"key\": \"debug\",\n              \"value\": \"false\"\n            }\n          ]\n        }\n      ]\n    }\n  ]\n}\n```\n\nMake sure this file will be taken into account by adding the\n[--modules](http://mesos.apache.org/documentation/latest/configuration/master-and-agent/)\nflag to you mesos agents.\n\nAdd the hook to the Mesos agent command with the\n`--hooks com_criteo_mesos_CommandHook` flag and/or add the isolator with\nthe `--isolation com_criteo_mesos_CommandIsolator` flag.\n\nThen, restart the Mesos slaves to complete the installation and have a\nlook at the logs to confirm that your scripts are called when some of your\nconfigured events are triggered.\n\nNote: com_criteo_mesos_CommandIsolator2, com_criteo_mesos_CommandIsolator3, ... are also defined to allow to have several distinct isolators.\n\n## Build Instructions\n\n### With docker (recommended)\n\nThis will build an image that will compile mesos-command-modules.\nThis image has an already built mesos source tree in it so no need to recompile mesos which could take a lot of time otherwise.\n\n```shell\n    $ docker build . -t mesos-command-modules\n    $ docker run -v \"$(pwd):/src/mesos-command-modules\" mesos-command-modules\n```\n\n### With mesos source tree already built\n\n```shell\n    $ export MESOS_ROOT_DIR=[ directory where Mesos was cloned, e.g. ~/repos/mesos ]\n    $ export MESOS_BUILD_DIR=[ directory where Mesos was BUILT, e.g. ~/repos/mesos/build ]\n    $ mkdir build\n    $ cd build\n    $ cmake ..\n    $ make\n    $ make test\n```\n\nPlease note that you must run **clang-format** before commiting your change,\notherwise the Travis job will fail. To apply clang-format, type:\n\n```shell\nmake clang-format\n```\n\n## Test on a slave\n\n```shell\n    $ ./scripts/build_and_upload_module.sh user@hostname\n```\n\n## Deploy a test environment\n\nFollow [examples/README.md](./examples/README.md).\n\n## Design choices\n\n### Forking a process for each event\n\nWe prefered forking in order to ensure the statelessness of the commands.\nThe downside of this choice is that forking a process might be slow but\nwe do not expect to have billions of calls on each agent anyway.\n\nWarning: the usage method of Isolator can actually be called very often (on every call for /monitor/statistics endpoint is called which call usage for every container each time). It make a lot of call.\n\n### Using temporary files as inputs and outputs buffers\n\nWe implemented passing inputs and retrieving outputs from the external\ncommands with temporary files for simplicity. Indeed, a lot of programming\nlanguages can easily read the content of a file while reading a pipe is not\nso trivial (see the tests).\n\n## TODO\n\n* Add tests to check the behavior of the CommandRunner when temporary files are\n  manually deleted when a script is running.\n* Automate integration tests using docker-compose in examples/ to execute it in\n  Travis and ensure non-regression.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcriteo%2Fmesos-command-modules","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcriteo%2Fmesos-command-modules","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcriteo%2Fmesos-command-modules/lists"}