{"id":19066144,"url":"https://github.com/epfml/epfml-utils","last_synced_at":"2025-02-22T03:16:19.917Z","repository":{"id":109046686,"uuid":"596466060","full_name":"epfml/epfml-utils","owner":"epfml","description":"Tools for experimentation and using run:ai. The aim is for these to be small self-contained utilities that are used by multiple people.","archived":false,"fork":false,"pushed_at":"2023-03-16T15:31:11.000Z","size":51,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-02-02T18:48:36.885Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epfml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-02T08:43:25.000Z","updated_at":"2023-02-07T13:26:14.000Z","dependencies_parsed_at":"2023-04-05T23:33:34.883Z","dependency_job_id":null,"html_url":"https://github.com/epfml/epfml-utils","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fepfml-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fepfml-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fepfml-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2Fepfml-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epfml","download_url":"https://codeload.github.com/epfml/epfml-utils/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240118419,"owners_count":19750491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T00:54:38.619Z","updated_at":"2025-02-22T03:16:19.912Z","avatar_url":"https://github.com/epfml.png","language":"Python","readme":"# EPFML Utilities\n\nInternal tools for the MLO lab of EPFL.\n\n## Installation\n\n```shell\n❯ pip install epfml-utils\n```\n\nAdd environment variables to your `~/.bashrc` or equivalent file:\n\n```bash\nexport EPFML_STORE_S3_ACCESS_KEY=\"\"\nexport EPFML_STORE_S3_SECRET_KEY=\"\"\nexport EPFML_STORE_S3_BUCKET=\"\"\nexport EPFML_LDAP=\"\"\n# (Get those values from a friend.)\nexport EPFML_STORE_S3_ENDPOINT=\"https://s3.epfl.ch\"\n```\n\nand make sure they are loaded:\n```bash\nsource ~/.bashrc\necho $EPFML_LDAP  # Check if this prints your username.\n```\n\n\n## Key-value store\n\nThis key-value store can help to transfer information between machines.\nDo not expect this to be fast or high-volume.\n__Don't__ use this 100's of times in a training script.\n\n### Command-line usage\n\nOn one machine:\n```shell\n❯ epfml store set my_name \"Bob\"\n```\nOn any other machine:\n```shell\n❯ epfml store get my_name\nBob\n```\n\n### Python usage\n\n```python\nimport torch\nimport epfml.store\n\nepfml.store.set(\"my_data\", {\"name\": \"Bob\", \"lab\": \"MLO\"})\nepfml.store.set(\"tensor\", torch.zeros(4))\n```\n\n```python\nprint(epfml.store.get(\"tensor\"))\nepfml.store.unset(\"tensor\")\nprint(epfml.store.pop(\"my_data\"))  # get and delete\n```\n\n\n## Transporting code between machines\n\n### Packing\n\nUpload a copy of the current working directory:\n\n```shell\n❯ epfml bundle pack\n📦 Packaged and shipped.\n⬇️ Unpack with `epfml bundle unpack mlotools_20230202_a205e830 -o .`.\n```\n\nTo exclude (large / non-code) files from the package, add a config file to the directory\n\n```shell\n❯ epfml bundle init\n📦 Default config file written to `/Users/vogels/epfl/mlotools/.epfml.bundle.toml`.\n```\n\nand customize it to your needs.\n\n### Unpacking\n\nYou can download the code into a directory:\n\n```shell\n❯ epfml bundle unpack mlotools_20230202_a205e830 -o some_directory\n```\n\nOr you can run a training script, or any other shell command, in a temporary check-out of the package:\n\n```shell\n❯ epfml bundle exec mlotools_20230202_a205e830 -- du -sh\n🏃 Running inside a tmp clone of package `mlotools_20230202_a205e830`.\n160K    .\n```\n\n## Contributing\n\nThis repository is meant to be a collection of independent tools that each serve a simple well-defined purpose.\n\nIf you want to improve any of the tools in this repo, or contribute new tools. Take the following steps:\n\n1. Clone this repository.\n2. Initialize pre-commit: `cd epfml-utils \u0026\u0026 pre-commit install` (`pip install pre-commit`). This runs a couple of checks before you commit to keep this repo clean.\n3. Ask someone else to scrutinize your contributions. They can help you to improve the API and catch bugs. You can also check these very nice [API design principles](https://github.com/google/etils/blob/main/docs/api-design.md) for tips.\n\nTo release a new version on PyPi, just increase the version number in `pyproject.yoml` and commit to Github.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Fepfml-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepfml%2Fepfml-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Fepfml-utils/lists"}