{"id":21293053,"url":"https://github.com/criteo/fromconfig-yarn","last_synced_at":"2026-05-22T05:10:50.035Z","repository":{"id":52367327,"uuid":"359805191","full_name":"criteo/fromconfig-yarn","owner":"criteo","description":"A fromconfig Launcher for Yarn","archived":false,"fork":false,"pushed_at":"2022-11-25T13:47:00.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-13T17:05:06.129Z","etag":null,"topics":["config-as-code","fromconfig","python3","yarn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/criteo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-20T12:20:38.000Z","updated_at":"2022-11-24T15:34:32.000Z","dependencies_parsed_at":"2023-01-22T19:16:20.279Z","dependency_job_id":null,"html_url":"https://github.com/criteo/fromconfig-yarn","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Ffromconfig-yarn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Ffromconfig-yarn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Ffromconfig-yarn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/criteo%2Ffromconfig-yarn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/criteo","download_url":"https://codeload.github.com/criteo/fromconfig-yarn/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243762232,"owners_count":20343972,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["config-as-code","fromconfig","python3","yarn"],"created_at":"2024-11-21T13:53:06.614Z","updated_at":"2026-05-22T05:10:44.984Z","avatar_url":"https://github.com/criteo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FromConfig Yarn\n[![pypi](https://img.shields.io/pypi/v/fromconfig-yarn.svg)](https://pypi.python.org/pypi/fromconfig-yarn)\n[![ci](https://github.com/criteo/fromconfig-yarn/workflows/Continuous%20integration/badge.svg)](https://github.com/criteo/fromconfig-yarn/actions?query=workflow%3A%22Continuous+integration%22)\n\nA [fromconfig](https://github.com/criteo/fromconfig) `Launcher` for yarn execution.\n\n\u003c!-- MarkdownTOC --\u003e\n\n- [Install](#install)\n- [Quickstart](#quickstart)\n- [Usage Reference](#usage-reference)\n  - [Options](#options)\n\n\u003c!-- /MarkdownTOC --\u003e\n\n\n\u003ca id=\"install\"\u003e\u003c/a\u003e\n## Install\n\n```bash\npip install fromconfig_yarn\n```\n\n\u003ca id=\"quickstart\"\u003e\u003c/a\u003e\n## Quickstart\n\nOnce installed, the launcher is available with the name `yarn`.\n\nGiven the following module\n\n```python\nclass Model:\n    def __init__(self, learning_rate: float):\n        self.learning_rate = learning_rate\n\n    def train(self):\n        print(f\"Training model with learning_rate {self.learning_rate}\")\n```\n\n\nand config files\n\n```yaml\n# config.yaml\nmodel:\n  _attr_: foo.Model\n  learning_rate: \"${params.learning_rate}\"\n\n# params.yaml\nparams:\n  learning_rate: 0.001\n\n# launcher.yaml\nyarn:\n  name: test-fromconfig\n\nlogging:\n  level: 20\n\nlauncher:\n  run: yarn\n```\n\nRun (assuming you are in a Hadoop environment)\n\n```bash\nfromconfig config.yaml params.yaml launcher.yaml - model - train\n```\n\nWhich prints\n\n```\nINFO:fromconfig.launcher.logger:- yarn.name: test-fromconfig\nINFO:fromconfig.launcher.logger:- logging.level: 20\nINFO:fromconfig.launcher.logger:- params.learning_rate: 0.001\nINFO:fromconfig.launcher.logger:- model._attr_: foo.Model\nINFO:fromconfig.launcher.logger:- model.learning_rate: 0.001\nINFO skein.Driver: Driver started, listening on 12345\nINFO:fromconfig_yarn.launcher:Uploading pex to viewfs://root/user/path/to/pex\nINFO:cluster_pack.filesystem:Resolved base filesystem: \u003cclass 'pyarrow.hdfs.HadoopFileSystem'\u003e\nINFO:cluster_pack.uploader:Zipping and uploading your env to viewfs://root/user/path/to/pex\nINFO skein.Driver: Uploading application resources to viewfs://root/user/...\nINFO skein.Driver: Submitting application...\nINFO impl.YarnClientImpl: Submitted application application_12345\nINFO:fromconfig_yarn.launcher:TRACKING_URL: http://12.34.56/application_12345\n```\n\nYou can also monkeypatch the relevant functions to \"fake\" the Hadoop environment with\n\n```bash\npython monkeypatch_fromconfig.py config.yaml params.yaml launcher.yaml - model - train\n```\n\nThis example can be found in [`docs/examples/quickstart`](docs/examples/quickstart).\n\n\n\u003ca id=\"usage-reference\"\u003e\u003c/a\u003e\n## Usage Reference\n\n\u003ca id=\"options\"\u003e\u003c/a\u003e\n### Options\n\nTo configure Yarn, add a `yarn` entry to your config.\n\nYou can set the following parameters.\n\n- `env_vars`: A list of environment variables to forward to the container(s)\n- `hadoop_file_systems`: The list of available filesystems\n- `ignored_packages`: The list of packages not to include in the environment\n- `jvm_memory_in_gb`: The JVM memory (default, `8`)\n- `memory`: The executor's memory (default, `32 GiB`)\n- `num_cores`: The executor's number of cores (default, `8`)\n- `package_path`: The HDFS location where to save the environment\n- `zip_file`: The path to an existing `pex` file, either local or on HDFS\n- `name`: The application name\n- `queue`: The yarn queue to submit the application to\n- `node_label`: The label of the hadoop node to be scheduled\n- `pre_script_hook`: A script to be executed before python is invoked\n- `extra_env_vars`: A mapping of extra environment variables to forward to the container(s)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcriteo%2Ffromconfig-yarn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcriteo%2Ffromconfig-yarn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcriteo%2Ffromconfig-yarn/lists"}