{"id":19199817,"url":"https://github.com/rose-stl-lab/zihao-s-toolbox","last_synced_at":"2025-07-14T04:41:17.139Z","repository":{"id":220900193,"uuid":"752876723","full_name":"Rose-STL-Lab/Zihao-s-Toolbox","owner":"Rose-STL-Lab","description":"A toolbox of shared utilities across different projects.","archived":false,"fork":false,"pushed_at":"2024-09-08T16:54:49.000Z","size":315,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-31T20:21:48.981Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rose-STL-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-05T02:27:58.000Z","updated_at":"2024-09-29T17:58:28.000Z","dependencies_parsed_at":"2024-02-19T03:32:19.114Z","dependency_job_id":"ef220a85-4079-41a9-9a03-e70d6d6a18aa","html_url":"https://github.com/Rose-STL-Lab/Zihao-s-Toolbox","commit_stats":null,"previous_names":["rose-stl-lab/zihao-s-toolbox"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FZihao-s-Toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FZihao-s-Toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FZihao-s-Toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rose-STL-Lab%2FZihao-s-Toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rose-STL-Lab","download_url":"https://codeload.github.com/Rose-STL-Lab/Zihao-s-Toolbox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253172913,"owners_count":21865587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T12:29:05.683Z","updated_at":"2025-05-09T01:25:36.225Z","avatar_url":"https://github.com/Rose-STL-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kube Toolbox\n\n## 0. Introduction\n\nThis toolbox is designed to facilitate the synchronization between a Kubernetes (Kube) cluster and local GPU machines, aiming to streamline the process of running and managing batch experiments. The main advantages of using this toolbox include:\n\n- Synchronization of environment variables (found in `.env` files) between Nautilus cluster executions and local runs, allowing for easy access to credentials via `os.environ`.\n- Simplification of the process to load environment variables, Python environments, and startup scripts.\n- Data and output folder synchronization through S3 storage.\n- Central management of all potential hyperparameters for various datasets and models.\n- Compatibility with a wide range of projects, even those already utilizing configuration files.\n- Capability to execute all combinations of experiments in parallel with a single command.\n\n### Requirements\n\nPython packages:\n\n- boto3 (required)\n- pyyaml (required)\n- loguru (optional, for better logging)\n\nOther requirements:\n\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) (required, for interacting with Kubernetes cluster)\n- [s5cmd](https://github.com/peak/s5cmd) (optional, needed to be in PATH, for faster S3 operations)\n\n### Before proceeding\n\n- Use **hyphen -** instead of **underscore _** for separators in your project name\n- The setup would be a lot easier if you sync up your github, gitlab repo name, conda environment name, and carefully follow all naming conventions. While you can customize if some of them do not match, I cannot test over all scenarios.\n\n## 1. Quick Start (for Nautilus user):\n\nImagine a scenario where you're handling \"machine learning\" workloads with two datasets you wish to use both in the cluster and locally, and you have two baseline models to evaluate.\n\n#### Step 1: Set Up Your Project Repository\n\nStart by creating a new git repository with `src` and `data` directories:\n\n```bash\nmkdir example; cd example; mkdir src; mkdir data; git init\n```\n\n**Add the toolbox repository as a submodule:**\n\n```bash\ngit submodule add https://github.com/Rose-STL-Lab/Zihao-s-Toolbox.git src/toolbox\n```\n\n**Create symbolic links for Makefile and launch.py at the root of your workspace:**\n\n```bash\nln -s src/toolbox/launch.py .\nln -s src/toolbox/Makefile .\n```\n\n#### Step 2: Prepare Baselines and Datasets\n\nGenerate two baseline files and two datasets as follows:\n\n```bash\necho \"1,2,3\" \u003e data/1.txt\necho \"4,5,6\" \u003e data/2.txt\necho \"import sys; print(sum(map(float, open(sys.argv[1]).read().split(','))) / 3)\" \u003e src/avg.py  # Compute average\necho \"import sys, statistics; print(statistics.median(map(float, open(sys.argv[1]).read().strip().split(','))))\" \u003e src/med.py  # Compute median\n```\n\nCreate a .env file with your S3 bucket details:\n\n```bash\nvim .env\n```\n\n```ini\nS3_BUCKET_NAME=example\nS3_ENDPOINT_URL=https://s3-west.nrp-nautilus.io\n```\n\nSetup your Kubernetes configuration in `config/kube.yaml` (replace `\u003cNAMESPACE\u003e` and `\u003cUSER\u003e` with your details):\n\n```bash\nmkdir -p config; vim config/kube.yaml\n```\n\n```yaml\nproject_name: example\nnamespace: \u003cNAMESPACE\u003e\nuser: \u003cUSER\u003e\nimage: gitlab-registry.nrp-nautilus.io/zihaozhou/example\n```\n\n\u003e Note: Here we created the image for you. If you want to create your own image, please refer to Section 4. After creation of your own image, you no longer need to specify the `image` field in `kube.yaml`. (unless you are collaborating with others and the repo is created under their accounts)\n\nDefine the experiment configurations in `config/launch.yaml`:\n\n```bash\nvim config/launch.yaml\n```\n\n```yaml\nproject_name: example\nmodel:\n  average:\n    # \u003cfn\u003e will be automatically replaced by hparam values\n    command: \u003e-\n      [](make download file=data/;) python src/avg.py \u003cfn\u003e\n    # # Model can also have hparam, hparam could be either list or single value\n    # hparam:\n    #   hid_dim: [256, 512]\n    #   ...\n    # # Override *non-projectwise* kube config, see Section 2\n    gpu_count: 0\n    # memory: 8  # 8GiB\n    # image: gitlab-registry.nrp-nautilus.io/prp/jupyter-stack/minimal\n    # ...\n  median:\n    command: \u003e-\n      [](make download file=data/;) python src/med.py \u003cfn\u003e\n    gpu_count: 0\ndataset:\n  data1:\n    hparam:\n      # Launch will automatically consider all combinations of hparam\n      # hparam preceded by _ will NOT appear in the job name\n      # If your hparam contain special characters, you must _ them\n      _fn: data/1.txt\n  data2:\n    hparam:\n      _fn: data/2.txt\nrun:\n  dataset: [data1]\n  model: [average]\n```\n\n\u003e #### Understanding Launch Configuration\n\u003e\n\u003e - `[](make download file=data/;)` is a syntax sugar. Sometimes, we expect slight difference between local and remote commands (like we don't need to re-download data in local runs). Here we use `[]` to indicate the local command (which is doing nothing here), and `()` to indicate the remote command.\n\u003e - Another useful syntax sugar is `##comment##`, as yaml does not support comments in multi-line strings. You can use a pair of `##` to indicate comments, and they will be removed before execution.\n\u003e - In `python src/med.py \u003cfn\u003e`, `\u003cfn\u003e` will be replaced by the values defined in `hparam` section. Hypermeters can be defined in both `model` and `dataset` sections. They are placeholders such that you don't need to copy and paste the same command with slight modifications.\n\u003e - `gpu_count` are model-wise / dataset-wise kubernetes configurations that can be overridden in `launch.yaml`. See Section 2 for all overridable fields. If you don't specify them, they will be inherited from `kube.yaml`. If you specify the same field in both `model` and `dataset`, the one in `model` will take precedence.\n\u003e - `run` section specifies the combinations of experiments to run. You can also add `hparam` to the `run` section to specify the hyperparameters you want to run. If you don't specify the `hparam`, all possible combinations of hyperparameters will be run.\n\u003e\n\u003e #### Advanced Configuration\n\u003e\n\u003e - You can specify an additional file section. (example: `file: [src/temp.py]`). Then, when you run `make pod` or `make job`, the specified files will be automatically uploaded (and overwrites the preexisting file) to the pod or job. This is particularly useful when you are debugging and don't want to make git commit. By default, `config/kube.yaml`, `config/launch.yaml`, and `.env` will be uploaded. You can specify `file: null` to disable this behavior. You can also run `make copy pod=\u003cpod_name\u003e` to upload files to a running pod.\n\u003e   - This only supports a limited number of **text files** and will fill the command section with encoding text. The advantage is that you don't need to worry about file uploads for every job or pod creation. If your file section is too long, the pod could fail due to command length limit.\n\u003e - The hparam sections can be a list of hparam dictionaries with the *same keys*. See below for an example. Why do we need this? Sometimes we don't want to run all combinations of hyperparameters, but only a subset of them. In this case, `make` will create three jobs, `train=paper`, `train=original`, and `train=scale`.\n\u003e\n\u003e ```\n\u003e hparam:\n\u003e     train:\n\u003e         paper:\n\u003e           _learning_rate: 0.000000008192\n\u003e           _lr_scheduler: linear\n\u003e           _lr_warmup_steps: 0\n\u003e         original:\n\u003e           _learning_rate: 1e-5\n\u003e           _lr_scheduler: constant\n\u003e           _lr_warmup_steps: 500\n\u003e         scale:\n\u003e           _learning_rate: 1e-5\n\u003e           _lr_scheduler: constant\n\u003e           _lr_warmup_steps: 500\n\u003e ```\n\u003e\n\u003e - Overridable kube fields can also be directly specified at the root level of `launch.yaml`. Examples are\n\u003e\n\u003e ```\n\u003e model: ...\n\u003e dataset: ...\n\u003e run: ...\n\u003e gpu_count: 1\n\u003e ```\n\u003e\n\u003e They will override the corresponding fields in `kube.yaml`.\n\n#### Running Locally\n\nExecute the following command to run experiments locally: `make local`\n\nExample output:\n\n```\nRunning {\n    \"_fn\": \"data/1.txt\"\n} ... \n\npython src/avg.py data/1.txt\n\n2.0\n```\n\nChange the run section in `config/launch.yaml` to\n\n```\nrun:\n   dataset: [data1, data2]\n   model: [average, median]\n```\n\nand execute `make local` to run all possible combinations of experiments sequentially.\n\n```\nRunning {\n    \"_fn\": \"data/1.txt\"\n} ... \n\npython src/avg.py data/1.txt\n\n2.0\nRunning {\n    \"_fn\": \"data/1.txt\"\n} ... \n\npython src/med.py data/1.txt\n\n2.0\nRunning {\n    \"_fn\": \"data/2.txt\"\n} ... \n\npython src/avg.py data/2.txt\n\n5.0\nRunning {\n    \"_fn\": \"data/2.txt\"\n} ... \n\npython src/med.py data/2.txt\n\n5.0\n```\n\n#### Uploading Data to Remote Storage\n\nIf you have an S3 bucket, update the credentials in `.env` and use the following command to upload your dataset: `make upload file=data/`\n\nExample output:\n\n```bash\nUploaded data/1.txt to data/1.txt\nUploaded data/2.txt to data/2.txt\n```\n\nIf you have installed [s5cmd](https://github.com/peak/s5cmd), `make` will automatically use it as the backend for S3 operations to improve performance. Otherwise, it will use [boto3](https://github.com/boto/boto3).\n\nFor users without an S3 bucket, request access from the Nautilus matrix chat or use buckets provided by `rosedata.ucsd.edu`. You don't need a bucket for this tutorial.\n\n### Remote Pod\n\nTo create a remote pod, run: `make pod`\n\nExample output:\n\n```\npod/\u003cYOUR-USERNAME\u003e-example-interactive-pod created\n```\n\nYou can now navigate to the shell of the pod by running: `kubectl exec -it \u003cYOUR-USERNAME\u003e-example-interactive-pod -- /bin/bash`. Now, run `make download file=data/; make local`. You should see exactly the same output as running locally.\n\n### Remote Batch Jobs\n\nTo run all possible combinations of experiments in parallel with Nautilus, run: `make job`\n\nExample output:\n\n```bash\nJob 'example-average-data1' not found. Creating the job.\njob.batch/example-average-data1 created\nJob 'example-median-data1' not found. Creating the job.\njob.batch/example-median-data1 created\nJob 'example-average-data2' not found. Creating the job.\njob.batch/example-average-data2 created\nJob 'example-median-data2' not found. Creating the job.\njob.batch/example-median-data2 created\n```\n\nAfter a while, you can run `kubectl logs -f \u003cYOUR-USERNAME\u003e-example-average-data1` to check the logs of the job. You would see\n\n```\nDownloaded data/1.txt to ./data/1.txt\nDownloaded data/2.txt to ./data/2.txt\n2.0\n```\n\nFinally, run `make delete` to cleanup all workloads.\n\n\u003e Be careful: `make delete` operates by removing all pods and jobs under your user label.\n\n---\n\n## 2. Kube Utilities Specification\n\n`config/kube.yaml`:\n\n```yaml\n##### Project-wise configuration, should be the same across all experiments\n# Will not be overwritten by the launch.yaml\nproject_name: str, required, used for k8s resource name, env name and more, no underscore_, hyphen- allowed\nuser:         str, required, k8s user name\nnamespace:    str, required, k8s namespace\n\n# If you want to use a different environment name\nconda_env_name: str, default to \u003cproject_name\u003e\n\n##### Other field, can be overwritten in launch.yaml #####\n\n# env will be overridden by `.env`, therefore never effective in `kube.yaml`\n# however, specify env in `launch.yaml` can add new env variables\nenv: \n  \u003cenv-key\u003e: \u003cenv-value\u003e\n\n## If startup_script is not explicitly specified, the script will automatically pull the latest git repo using ssh_host and ssh_port, and activate the default environment using conda_home and conda_env_name.\nstartup_script: str, default to pull the latest git repo and submodules, activate the default conda environment, switch external S3 to internal S3 endpoint\nextra_startup_script: str, default to empty, if you want to run the default script and add a few lines of additional commands\nconda_home: str, default to /opt/conda\nssh_host:   str, default to gitlab-ssh.nrp-nautilus.io \nssh_port:   int, default to 30622\n\n# Command for interactive pod\nserver_command: str, default to `sleep infinity`\n\n## For CPU and Memory, the limit will be twice the requested\ngpu_count:        int, default to 0\ncpu_count:        int, in cores, default to 5\nephemeral_storage: int, in gigabytes, default to 100\nmemory:           int, in gigabytes, default to 32\n\n## Mount PVC to path\nvolumes:\n  \u003cpvc-name\u003e: \u003cmount-path\u003e\n\n## Image pull related\nimage:              str, default to \u003cregistry_host\u003e/\u003cgitlab_user\u003e/\u003cproject_name\u003e:latest\ngitlab_user:        str, default to \u003cuser\u003e\nregistry_host:      str, default to gitlab-registry.nrp-nautilus.io\nimage_pull_secrets: str, default to \u003cproject-name\u003e-read-registry\n## Prefix of the names of your workloads\nprefix: str\n## Will tolerate no-schedule taints\ntolerations: \n  - \u003ctoleration-key\u003e\ngpu_whitelist:\n  - \u003cusable-gpu-list\u003e\nhostname_blacklist:\n  - \u003cunusable-node-hostnames-list\u003e\n## High-performance GPU specified in https://ucsd-prp.gitlab.io/userdocs/running/gpu-pods/#choosing-gpu-type. Example: \"a100\", \"rtxa6000\". Once set, gpu_whitelist and gpu_blacklist will be ignored. \nspecial_gpu: str\n```\n\n`gpu_whitelist` and `gpu_blacklist` cannot be both set. If gpu_whitelist is set, only the specified GPUs will be used. If gpu_blacklist is set, all GPUs except the specified ones will be used. The same applies to `hostname_blacklist` and `hostname_whitelist`.\n\nExample GPU list:\n\n```yaml\n  - NVIDIA-TITAN-RTX\n  - NVIDIA-RTX-A4000\n  - NVIDIA-RTX-A5000\n  - Quadro-RTX-6000\n  - NVIDIA-GeForce-RTX-3090\n  - NVIDIA-GeForce-GTX-1080-Ti\n  - NVIDIA-GeForce-GTX-2080-Ti\n  - NVIDIA-A10\n```\n\nEnsure consistency in project_name across your GitLab repository (\u003cproject_name\u003e.git), conda environment (envs/\u003cproject_name\u003e), image pull secret (\u003cproject_name\u003e-read-registry), and S3 configuration (\u003cproject_name\u003e-s3cfg). Avoid underscores in project_name (use hyphen instead).\n\nYour GitLab username would be used as user to label your kube workloads (label: \u003cuser\u003e). For registry details, refer to the GitLab container registry documentation.\n\n## 3. S3 Utilities Specification\n\n### Usage\n\nCreate a .env file in your project repository with these values:\n\n```bash\nS3_BUCKET_NAME=\u003cyour_s3_bucket_name\u003e\nAWS_ACCESS_KEY_ID=\u003cyour_access_key\u003e\nAWS_SECRET_ACCESS_KEY=\u003cyour_secret_key\u003e\nS3_ENDPOINT_URL=https://...\n```\n\nLoad environment by `export $(grep -v '^#' .env | xargs -d '\\n')` or through `make` commands.\n\nYou can perform wildcard searches, downloads, uploads, or deletions on S3 files:\n\n```bash\n❯ make interactive file='Model/*t5*wise*419*'\nLocal files matching pattern:\n\nS3 files matching pattern:\nModel/Yelp/t5_model_12weeks_wise-sky-249-northern-dawn-288_epoch_1419/config.json\n...\n\nChoose an action [delete (local), remove (S3), download, upload, exit]:\n```\n\nUse single quotes to prevent shell wildcard expansion. The S3 bucket will sync with your current directory by default, maintaining the original file structure and creating necessary directories.\n\nBeyond Make commands, you can also directly import the functions from `src/toolbox/s3utils.py` to your Python scripts.\n\n```python\nfrom toolbox.s3utils import download_s3_path\n\nfolder_path = f\"Data/{dataset}\"\ndownload_s3_path(folder_path)\n```\n\nIf your Python script is invoked via `make`, the environment variables will be automatically loaded.\n\n## 4. Example Creation of [Nautilus](https://portal.nrp-nautilus.io/) Gitlab Image\n\nThis section will guide you through the process of creating a GitLab Docker image based on your git repo using the Nautilus platform. This is useful for those looking to automate their deployment and integration workflows using GitLab's CI/CD features. The result image can integrate nicely with Kubeutils.\n\n\u003e Note: If Nautilus SSH is no longer `gitlab-ssh.nrp-nautilus.io:30622`, please modifies `SSH_CONFIG` and .`gitlab-ci.yml` correspondingly.\n\n### Prerequisites\n\nBefore you begin, make sure you have:\n\n- A Github account, where you can register at [here](https://github.com).\n- A Nautilus Gitlab account, where you can register at [here](https://gitlab.nrp-nautilus.io/).\n- Familiarity with SSH, docker container, continuous integration and deployment (CI/CD) concepts\n\n### Steps\n\n1. Create a git repo at Nautilus [Gitlab](https://gitlab.nrp-nautilus.io/) with the name `example`. Don't initialize the repository with any file. If you want to use a different name, remember to replace `example` with your repo name in the following steps. Also, make sure your name is all lowercase and without any special characters.\n2. Create a git repo with the same name at Github.\n3. Generate an SSH key pair with the name `example` on your local machine using the following command: `ssh-keygen -f example -N \"\"`.\n4. Generate an SSH key pair with the name `example-write` on your local machine using the following command: `ssh-keygen -f example-write -N \"\"`.\n\n\u003e Note: The `example` key is kept in the image for pulling the code from the private repository, while the `example-write` key is used for mirroring the code to Gitlab. Be careful — if you accidentially dropped the `example-write` key in the image and later make it public, anyone can push code to your repository.\n\n5. Add the public key `example.pub` to Gitlab Repo - Settings - **Deploy Keys**. Title: `example`, **don't** grant write permission.\n6. Add the public key `example-write.pub` to Gitlab Repo - Settings - **Deploy Keys**. Title: `example-write`, grant write permission.\n7. Add the public key `example.pub` to Github Repo - Settings - **Deploy Keys**. Title: `example`, **don't** grant write permission.\n8. Deploy tokens are used to securely download (pull) Docker images from your GitLab registry without requiring sign-in. Under Gitlab Repo - Settings - Repository - **Deploy Tokens**, create new deploy token with name `example-write-registry`. Grant both `write_registry` and `read_registry` access. Take a note of the `username` and `password` for this token for Gitlab CI.\n9. Create new deploy token with name `example-read-registry`. Grant `read_registry` access. Take a note of the `username` and `password` for this token for Kubernetes experiments.\n\n\u003e Note: The `example-write-registry` token is used for pushing the image to the registry from Github, while the `example-read-registry` token is used in the kube cluster to pull the image.\n\n9. Run the follow command to upload the read tokens to the cluster.\n\n```bash\nkubectl create secret docker-registry example-read-registry \\\n    --docker-server=gitlab-registry.nrp-nautilus.io \\\n    --docker-username=\u003cusername\u003e \\\n    --docker-password=\u003cpassword\u003e\n```\n\n10. In **Github** Repo - Settings - Secrets and variables - **Actions**, enter the following repository secrets:\n\n- SSH_CONFIG: `SG9zdCBnaXRodWIuY29tCiAgSG9zdE5hbWUgZ2l0aHViLmNvbQogIFVzZXIgZ2l0CiAgSWRlbnRpdHlGaWxlIH4vLnNzaC9pZF9yc2EKCkhvc3QgZ2l0bGFiLXNzaC5ucnAtbmF1dGlsdXMuaW8KICBIb3N0TmFtZSBnaXRsYWItc3NoLm5ycC1uYXV0aWx1cy5pbwogIFVzZXIgZ2l0CiAgUG9ydCAzMDYyMgogIElkZW50aXR5RmlsZSB+Ly5zc2gvaWRfcnNhCgo=`, which is the base64 encoding of\n\n```\nHost github.com\nHostName github.com\nUser git\nIdentityFile ~/.ssh/id_rsa\n\nHost gitlab-ssh.nrp-nautilus.io\nHostName gitlab-ssh.nrp-nautilus.io\nUser git\nPort 30622\nIdentityFile ~/.ssh/id_rsa\n```\n\n- DOCKER_PASSWORD: the write `password` from the previous step.\n- DOCKER_USERNAME: the write `username` from the previous step.\n- GIT_DEPLOY_KEY: base64 encode the **read** deploy key you created (`base64 -i example`, don't include any new lines).\n- GITLAB_DEPLOY_KEY: base64 encode the **write** deploy key you created (`base64 -i example-write`, don't include any new lines).\n- GITLAB_USERNAME: your gitlab user name, which is in the middle of your gitlab repo URL.\n\n7. Create the following files under your repo:\n\n- `environment.yml`\n\n```yaml\nname: example\nchannels:\n  - conda-forge\n  - nvidia\ndependencies:\n  - python=3.11.*\n  - pip\n  - poetry=1.*\n```\n\n- `Dockerfile`\n  - Can be copied using `cp src/toolbox/Dockerfile .`\n  - Can be copied using `mkdir -p .github/workflows; cp src/toolbox/workflows/docker.yml .github/workflows/docker.yml`\n  - Can be copied using `mkdir -p .github/workflows; cp src/toolbox/workflows/mirror.yml .github/workflows/mirror.yml`\n\nYou shall verify the environment creation on your local machine:\n\n```bash\nconda env create -n example --file environment.yml\nconda activate example\npoetry init\n## Add dependencies interactively or through poetry add\n## Examples:\npoetry source add --priority=explicit pytorch-gpu-src https://download.pytorch.org/whl/\u003ccuda_version\u003e\npoetry add --source pytorch-gpu-src torch\npoetry add numpy==1.26.2\n...\n## Run code on your local machine to make sure all required dependencies are installed.\n```\n\nThis procedure creates the lock file, `poetry.lock`. Commit it to the git repository. Push to the Github will compile the image. Any modification of the environment related files (see workflow file) will trigger the image update.\n\nYou may check out `https://github.com/ZihaoZhou/example` and `https://gitlab.nrp-nautilus.io/ZihaoZhou/example` as a reference.\n\n### (Alternative) Manual Docker Build\n\nIf you:\n\n- Don't need CI\n- Don't have full control over the repository\n- Use different branch than `main` or `master`\n- Are not allowed to use Github, or\n- Just want to build the image manually,\n\nYou can follow the following steps.\n\n1. Create a git repo at Nautilus Gitlab ... (same as above)\n2. Create a git repo at Github or any other git hosting service ...\n3. Generate a read SSH key pair ...\n4. Generate a write SSH key pair ...\n5. Add the public key to Gitlab ...\n6. Create folder `.ssh/` in your working directory, copy your SSH read private key to `.ssh/` and rename it to `id_rsa`. Create the `.ssh/config` file with the following content:\n\n```config\nHost gitlab-ssh.nrp-nautilus.io\n    HostName gitlab-ssh.nrp-nautilus.io\n    User git\n    Port 30622\n    IdentityFile ~/.ssh/id_rsa\n\nHost github.com\n    HostName github.com\n    User git\n    IdentityFile ~/.ssh/id_rsa\n...\n(or other git hosting service)\n```\n\n\u003e **Warning**: Add `/.ssh*` to your `.gitignore` to avoid uploading your secret credentials to the repository.\n\n7. Run `ssh-keyscan -p 30622 gitlab-ssh.nrp-nautilus.io \u003e\u003e .ssh/known_hosts`, `ssh-keyscan github.com \u003e\u003e .ssh/known_hosts` (or other git hosting service) to add the host key to known_hosts.\n8. Copy the Dockerfile to your local working directory, and then change PROJECT_NAME to your project name, change PROJECT_SSH_URL to your hosting service URL. You may switch to different branch by adding `--branch \u003cbranch-name\u003e` after git clone.\n9. Run the following command to build the image:\n\n```bash\ndocker build -t gitlab-registry.nrp-nautilus.io/\u003cuser-name\u003e/\u003cproject-name\u003e:\u003ccustom-tag\u003e .\ndocker login gitlab-registry.nrp-nautilus.io\n# Enter your write-registry username and password\ndocker push gitlab-registry.nrp-nautilus.io/\u003cuser-name\u003e/\u003cproject-name\u003e:\u003ccustom-tag\u003e\ndocker tag gitlab-registry.nrp-nautilus.io/\u003cuser-name\u003e/\u003cproject-name\u003e:\u003ccustom-tag\u003e gitlab-registry.nrp-nautilus.io/\u003cuser-name\u003e/\u003cproject-name\u003e:latest\ndocker push gitlab-registry.nrp-nautilus.io/\u003cuser-name\u003e/\u003cproject-name\u003e:latest\n```\n\n10. If you encountered any error, you may comment out error lines in the Dockerfile and then run\n\n```bash\ndocker run -it /bin/bash gitlab-registry.nrp-nautilus.io/\u003cuser-name\u003e/\u003cproject-name\u003e:\u003ccustom-tag\u003e /bin/bash\n```\n\nto enter the image and test the following command manually.\n\n11. After the image is successfully pushed, you can free space and delete all built images by running `docker images | grep '\u003cproject-name\u003e' | awk '{print $3}' | xargs docker rmi` and `docker rmi $(docker images -f \"dangling=true\" -q)`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frose-stl-lab%2Fzihao-s-toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frose-stl-lab%2Fzihao-s-toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frose-stl-lab%2Fzihao-s-toolbox/lists"}