{"id":21275288,"url":"https://github.com/ffreemt/colab-misc-utils","last_synced_at":"2025-07-24T03:06:05.563Z","repository":{"id":62562725,"uuid":"322488099","full_name":"ffreemt/colab-misc-utils","owner":"ffreemt","description":"Miscellaneous utils mainly intended for use in colab","archived":false,"fork":false,"pushed_at":"2021-01-03T03:06:48.000Z","size":98,"stargazers_count":1,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-06T02:05:37.119Z","etag":null,"topics":["colab","github","ssh","utils"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ffreemt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-18T04:26:51.000Z","updated_at":"2022-06-03T13:56:01.000Z","dependencies_parsed_at":"2022-11-03T15:17:57.435Z","dependency_job_id":null,"html_url":"https://github.com/ffreemt/colab-misc-utils","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ffreemt/colab-misc-utils","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffreemt%2Fcolab-misc-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffreemt%2Fcolab-misc-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffreemt%2Fcolab-misc-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffreemt%2Fcolab-misc-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ffreemt","download_url":"https://codeload.github.com/ffreemt/colab-misc-utils/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ffreemt%2Fcolab-misc-utils/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264756300,"owners_count":23659299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["colab","github","ssh","utils"],"created_at":"2024-11-21T09:29:19.354Z","updated_at":"2025-07-24T03:06:05.537Z","avatar_url":"https://github.com/ffreemt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# colab miscellaneous utils [![Codacy Badge](https://app.codacy.com/project/badge/Grade/83b7b2cb3ade4589812917f187a8abab)](https://www.codacy.com/gh/ffreemt/colab-misc-utils/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=ffreemt/colab-misc-utils\u0026amp;utm_campaign=Badge_Grade)[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)[![PyPI version](https://badge.fury.io/py/clmutils.svg)](https://badge.fury.io/py/clmutils)\nMiscellaneous utils mainly intended for use in colab\n\n## Installation\n```bash\npip install clmutils  # clm: colab-misc\n# pip install clmutils -U to update\n```\nOr clone the repo `https://github.com/ffreemt/colab-misc-utils.git` in Colab/jupyter:\n```\n!git clone https://github.com/ffreemt/colab-misc-utils.git\n%cd colab-misc-utils\n!pip install -r requirements.txt\n```\n\n## Usage:  with a setting file dotenv/`.env`\n\n1. `setup_git`\n    ```python\n    from clmutils import setup_git, Settings\n    config = Settings()\n    setup_git(\n      user_name=config.user_name,\n      user_email=config.user_email,\n      priv_key=config.gh_key\n    )\n    ```\n    Do the usual `git pull`, amend codes and `git push` stuff.\n\n    For more on `Connecting to GitHub with SSH`, refer to  https://docs.github.com/cn/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh\n\n2. `setup_ssh_tunnel`\n    * Run these lines in Colab\n    ```python\n    from clmutils import setup_ssh_tunnel, Settings\n    config = Settings()\n    setup_ssh_tunnel(\n      remote_host=config.remote_host,\n      remote_user=config.remote_user,\n      priv_key=config.cl_key,\n      remote_pubkey=config.remote_pubkey\n    )\n    ```\n\n    * Append the value of `cl_key_pub` in `dotenv/.env` (below) in remote computer's `~/.ssh/authorized_keys`.\n\n    * In the remote computer:\n    ```bash\n    ssh colab\n    ```\n\n`clmutils.Setting` will look for `dotenv` or `.env` in `/content/drive/MyDrive` if google drive is momunted; otherwise it looks for `.env` in the current dir and parents dir. (Sample .env: [`.env.sample`](https://github.com/ffreemt/colab-misc-utils/blob/master/.env.sample))\n\n`dotenv/.env` basically contains the necessary information to setup git for github or ssh tunnel:\n  * For git:\n      * gh_key: private key\n      * user_name/user_email;\n  * For reverse ssh tunnel:\n      * cl_key/cl_key_pub: private key/public key of Colab\n      * remote_pubkey: public_key of the remote computer\n      * remote_user: login name\n\n`dotenv/.env` is read by `clmutils.Setting` as follows\n```bash\nfrom clmutils import Settings\nconfig = Settings()\n# config = Setting(_env_file=file_path)\n```\n`config` will then have attributes: `config.gh_key, user_name, user_email,\n    cl_key, cl_key_pub, remote_pubkey, remote_user`.\nIf a particular attribute is not assigned a value in `dotenv/.env`, it defaults to empty string (`\"\"`).\n\n`dotenv` or `.env` has the following info and format (lines starting with a `#` are comments):\n```bash\n# for git\nuser_email = \"...@......\"\nuser_name = \"......\"\ngh_key = \"-----BEGIN EC PRIVATE KEY-----\n...............................................................9\n...............................................................J\nK9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==\n-----END EC PRIVATE KEY-----\n\"\n\n# for ssh tunnel\nremote_host = \"168.138.222.163\"\nremote_user = \"ubuntu\"\n\n# colab's private key, to be put in ~/.ssh/id_ed25519\n# or as specified in ~/.ssh/config\ncl_key = \"-----BEGIN OPENSSH PRIVATE KEY-----\n.....................................................................W\nQyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQAAAJDlRQSF5UUE\nhQAAAAtzc2gtZWQyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQ\n.....................................................................f\nDQRG5jrz/VCKKt2uu2eBAAAACWZvci1jb2xhYgECAwQ=\n-----END OPENSSH PRIVATE KEY-----\n\"\n\n# to be put in remote computer's ~/.ssh/autorized_keys\ncl_key_pub = \"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIKVka3/2JsB32jG5Zx+5KOfDQRG5jrz/VCKKt2uu2eB colab-key\"\n\n# to be put in colab's ~/.ssh/authorized_keys\nremote_pubkey = \"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE60sowZ4M0MA5nTGIH1RN54zERTuWSddFKqyeWZzaKv for-colab\"\n```\n\n## Usage: withtout dotenv/`.env`\n### Set up `github` with ssh using `clmutils.setup_git`\n\nFor manually setting up github with ssh, refer to\n[https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh)\n\nAssume you configure git as follows:\n```bash\ngit config --global user.email your-email-address\ngit config --global user.name your-github-username\n```\n\nWith `clmutils`, you'd do:\n```python\nfrom clmutils import setup_git`\n\nuser_email = \"your-email-address\"\nuser_name = \"your-github-username\"\ngh_key = \\\n\"\"\"\n-----BEGIN EC PRIVATE KEY-----\nMH.............................................................9\nAwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ\nK9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==\n-----END EC PRIVATE KEY-----\n\"\"\".strip() + \"\\n\"\n\nsetup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)\n```\nYou then upload the `public key` for `gh_key` to [https://github.com/settings/keys](https://github.com/settings/keys).\n\nRefer to Step 2 [https://support.cloudways.com/using-git-command-line-ssh/](https://support.cloudways.com/using-git-command-line-ssh/) for how to generate a private/public key pair. You can also use clmutils.gen_keypair to do that in Python.\n\n### Alternatively, set up `github` with ssh in 4 steps\n\n1. Write a private key to `~/.ssh/gh-key`\n```python\nfrom clmutils import create_file\ngh_key = \\\n\"\"\"\n-----BEGIN EC PRIVATE KEY-----\nMH.............................................................9\nAwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ\nK9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==\n-----END EC PRIVATE KEY-----\n\"\"\".strip() + \"\\n\"\n# Do not remove .strip() + \"\\n\"\n# the private key is very picky about format\n\ncreate_file(gh_key, dest=\"~/.ssh/gh-key\")\n```\n2. Set up `github.com` config for `git push`\n```python\nfrom clmutils import append_content\nconfig_github_entry = \\\n\"\"\"\nHost github.com\n   HostName github.com\n   User git\n   IdentityFile ~/.ssh/gh-key\n\"\"\"\nappend_content(config_github_entry, dest=\"~/.ssh/config\")\n```\n3. Verify that everything is OK, from a cell\n```ipynb\n!ssh -o StrictHostKeyChecking=no -T git@github.com\n```\nIf you see something similar to\n```bash\nHi your-name! You've successfully authenticated, but GitHub does not provide shell access.\n```\nyou are good to go.\n\n4. `git config --global`\nYou can now set up `git config global` from a cell, e.g.\n```ipynb\n!git config --global user.email your-email-address\n!git config --global user.name your-github-username\n# !ssh-keyscan github.com \u003e\u003e ~/.ssh/known_hosts\n```\nYou are ready to clone your own repo, run your app and generate new data, update the repo and push to `github`.\n\n## Utils planned\n\n* :white_check_mark: `setup_git` sets up `git` for `github.com`\n\n* :white_check_mark:  `reverse_ssh_tunnel`\n sets up a reverse ssh tunnel to a remote host with via autossh\n\n* Auxiliary utils\n  * :white_check_mark: `create_file`\n    creates a file with given mode, e.g. for `.ssh/id_rsa` or `IdentityFile` in `.ssh/config`\n\n  * :white_check_mark: `apppend_content`\n   appends some content to a file, e.g., for appended a public key to `.ssh/authorized_keys`\n\n  * :white_check_mark: `chmod600`\n     `chmod` of a file\n\n  * :white_check_mark: `gen_keypair` generates private/public key pair.\n\n  * :white_check_mark: `run_cmd` wraps\n      `subprocess.check_output`\n\n  * :white_check_mark: `run_cmd1` wraps\n      `subprocess.Popen`\n\n  * :white_check_mark: `check_running`  shows running processes with names containings a certain patter; based on `psutil.Proceess()'s `cmdline` and `status`\n\n## Demo: notebooks in Colab\n\n### git push from Colab: one line (`setup_git()`)\n```\n!pip install clmutils\nfrom clmutils import setup_git\ngh_key = \"\"\"...\"\"\"\nuser_name = \"...\"\nuser_email = \"...\"\nsetup_git(setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)\n```\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1u-eNhJpG64ajP-fPO3jtSQzdlAMSxtwE?usp=sharing#scrollTo=svB7ci6VzLnl)\n\n### `git push` from Colab in several steps\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1n0agOGg8rBoR0Ld3WAvh20QzXeZZ7xCk?usp=sharing) (in Chinese, shouldn't be too difficult to follow without knowing any Chinese though, just click through :smiley:)\n### Reverse ssh tunnel for ssh to Colab VM\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CIstRNIZjKhMqCch-FRyoIoiFjGAOGii?usp=sharing?usp=sharing)\nin English (I may provide a Chinese version later)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fffreemt%2Fcolab-misc-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fffreemt%2Fcolab-misc-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fffreemt%2Fcolab-misc-utils/lists"}