{"id":19707027,"url":"https://github.com/geyang/jaynes","last_synced_at":"2025-04-29T16:33:46.874Z","repository":{"id":39601327,"uuid":"442348917","full_name":"geyang/jaynes","owner":"geyang","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-21T05:02:03.000Z","size":1641,"stargazers_count":8,"open_issues_count":4,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-21T05:42:22.632Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geyang.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-28T04:31:58.000Z","updated_at":"2025-04-21T05:02:07.000Z","dependencies_parsed_at":"2025-04-22T02:17:38.782Z","dependency_job_id":null,"html_url":"https://github.com/geyang/jaynes","commit_stats":null,"previous_names":[],"tags_count":101,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geyang%2Fjaynes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geyang%2Fjaynes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geyang%2Fjaynes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geyang%2Fjaynes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geyang","download_url":"https://codeload.github.com/geyang/jaynes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251540571,"owners_count":21605931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T21:37:44.020Z","updated_at":"2025-04-29T16:33:41.851Z","avatar_url":"https://github.com/geyang.png","language":"Python","readme":"Jaynes, A Utility for training ML models on AWS, GCE, SLURM, with or without docker \n====================================================================================\n\n|Downloads|\n\nOverview\n--------\n\nThe reality of ML training in universities is that we use what ever\nhardware we are given (for free). This means that we might have a few\nbeefy GPU machines, an HPC cluster, plus some GCE/AWS credits that we\nget through grants. `Jaynes \u003chttps://github.com/episodeyang/jaynes\u003e`__\nis a well-designed python package that makes running across these\ninhomogenous hardward resources a pleasure.\n\n**install** (requires unix operating system.)\n\n.. code-block:: bash\n\n   pip install jaynes\n\nThe best way to get started with jaynes is to take a look at one of the\nexample projects in the\n`[geyang/jaynes-starter-kit] \u003chttps://github.com/geyang/jaynes-starter-kit\u003e`__.\nFor a rough idea, here is how to use jaynes to launch a training\nfunction:\n\nTo run **locally**:\n\n.. code-block:: python\n\n   import jaynes\n\n   def training(arg_1, key_arg=None, key_arg_2=None):\n       print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')\n\n   jaynes.config(mode=\"local\", arg_1=10, key_arg=0.3)\n   jaynes.run(training)\n   jaynes.listen()\n\nWe recommend setting up a ``main`` training function with the following\nsinature:\n\n.. code-block:: python\n\n   from params_proto import ParamsProto\n\n   class Args(ParamsProto):\n       seed = 100\n       lr = 3e-4\n       # ...\n       \n   def main(**deps):\n       from ml_logger import logger\n       \n       Args._update(deps)\n       logger.log_params(Args=vars(Args))\n       \n       # ... your main training steps\n\nThis way you can call the main fn directly for local debugging, but\nlaunch it as an entry point at scale.\n\nSetup\n-----\n\nJaynes has gone through a large number of iterations. This version\nincorporates best practices we learned from other open-source\ncommunities. You can specify a ``jaynes.yml`` config file (`copy one\nfrom our sample project to get started! \u003cexample_projects\u003e`__) for the\ntype of hosts (ssh/docker/singularity) and launchers\n(ssh/ec2/gce/slurm), so that none of those settings need to appear in\nyour ML python script. When called from python, Jaynes automatically\ntraverses the file tree to find the root of the project, the same way as\ngit.\n\nFor example, to run your code-block on a remote computer via ssh:\n\n.. code-block:: yaml\n\n   # your_project/jaynes.yml\n   version: 0\n   verbose: true\n   run:  # this is specific to each launch, and is dynamically overwritten in-memory\n     mounts:\n       - !mounts.S3Code\n         s3_prefix: s3://ge-bair/jaynes-debug\n         local_path: .\n         host_path: /home/ubuntu/\n         container_path: /Users/geyang/learning-to-learn\n         pypath: true\n         excludes: \"--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info'   --exclude='*.pkl'\"\n         compress: true\n     runner:\n       !runners.Docker\n       name:   # not implemented yet\n       image: \"episodeyang/super-expert\"\n       startup: \"yes | pip install jaynes ml-logger -q\"\n       work_directory: \"{mounts[0].container_path}\"\n       ipc: host\n     host:\n       envs: \"LANG=utf-8\"\n       pre_launch: \"pip install jaynes ml-logger -q\"\n     launch:\n       type: ssh\n       ip: \u003cyour ip address\u003e\n       username: ubuntu\n       pem: ~/.ssh/your_rsa_key\n\nIn python (your code-block):\n\n.. code-block:: python\n\n   # your_project/launch.py\n   import jaynes\n\n   def training(arg_1, key_arg=None):\n       print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')\n\n   jaynes.run(training)\n\nUsing Modes\n-----------\n\nA lot of times you want to setup a different run **modes** so it is easy\nto switch between them during development.\n\n.. code-block:: yaml\n\n   # your_project/jaynes.yml\n   version: 0\n   mounts: # mount configurations Available keys: NOW, UUID,\n     - !mounts.S3Code \u0026code-block_mount\n       s3_prefix: s3://ge-bair/jaynes-debug\n       local_path: .\n       host_path: /home/ubuntu/jaynes-mounts/{NOW:%Y-%m-%d}/{NOW:%H%M%S.%f}\n       # container_path: /Users/geyang/learning-to-learn\n       pypath: true\n       excludes: \"--exclude='*__pycache__' --exclude='*.git' --exclude='*.idea' --exclude='*.egg-info' --exclude='*.pkl'\"\n       compress: true\n   hosts:\n     hodor: \u0026hodor\n       ip: \u003cyour ip address\u003e\n       username: ubuntu\n       pem: ~/.ssh/incrementium-berkeley\n   runners:\n     - !runners.Docker \u0026ssh_docker\n       name: \"some-job\"  # only for docker\n       image: \"episodeyang/super-expert\"\n       startup: yes | pip install jaynes ml-logger -q\n       envs: \"LANG=utf-8\"\n       pypath: \"{mounts[0].container_path}\"\n       launch_directory: \"{mounts[0].container_path}\"\n       ipc: host\n       use_gpu: false\n   modes: # todo: add support to modes.\n     hodor:\n       mounts:\n         - *code-block_mount\n       runner: *ssh_docker\n       launch:\n         type: ssh\n         \u003c\u003c: *hodor\n\nnow run in python\n\n.. code-block:: python\n\n   # your_project/launch.py\n   import jaynes\n\n   def training(arg_1, key_arg=None):\n       print(f'training is running! (arg_1={arg_1}, key_arg={key_arg})')\n\n   jaynes.config(mode=\"hodor\")\n   jaynes.run(training)\n\nToDos\n-----\n\n-  ☐ more documentation\n-  ☐ singularity support\n-  ☐ GCE support\n-  ☐ support using non-s3 code-block repo.\n\nDone\n~~~~\n\n-  ☒ get the initial template to work\n\nInstallation\n------------\n\n.. code-block:: bash\n\n   pip install jaynes\n\nUsage (**Show me the Mo-NAY!! :moneybag::money_with_wings:**)\n-------------------------------------------------------------\n\nCheck out the `test_projects \u003cexample_projects\u003e`__ folder for projects\nthat you can run.\n\nTo Develop\n----------\n\n.. code-block:: bash\n\n   git clone https://github.com/episodeyang/jaynes.git\n   cd jaynes\n   make dev\n\nTo test, run\n\n.. code-block:: bash\n\n   make test\n\nThis ``make dev`` command should build the wheel and install it in your\ncurrent python environment. Take a look at the\n`https://github.com/episodeyang/jaynes/blob/master/Makefile \u003chttps://github.com/episodeyang/jaynes/blob/master/Makefile\u003e`__ for details.\n\n**To publish**, first update the version number, then do:\n\n.. code-block:: bash\n\n   make publish\n\nAcknowledgements\n----------------\n\nThis code-block is inspired by @justinfu’s\n`doodad \u003chttps://github.com/justinjfu/doodad\u003e`__, which is in turn built\non top of Peter Chen’s script.\n\nThis code-block is written from scratch to allow a more permissible\nopen-source license (BSD). Go bears :bear: !!\n\n.. |Downloads| image:: http://pepy.tech/badge/jaynes\n   :target: http://pepy.tech/project/jaynes\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeyang%2Fjaynes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeyang%2Fjaynes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeyang%2Fjaynes/lists"}