Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tensorchord/envd
🏕️ Reproducible development environment
https://github.com/tensorchord/envd
buildkit developer-tools development-environment docker hacktoberfest llmops mlops mlops-workflow model-serving
Last synced: about 1 month ago
JSON representation
🏕️ Reproducible development environment
- Host: GitHub
- URL: https://github.com/tensorchord/envd
- Owner: tensorchord
- License: apache-2.0
- Created: 2022-04-11T09:04:19.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-15T02:30:06.000Z (about 2 months ago)
- Last Synced: 2024-04-15T08:19:19.613Z (about 2 months ago)
- Topics: buildkit, developer-tools, development-environment, docker, hacktoberfest, llmops, mlops, mlops-workflow, model-serving
- Language: Go
- Homepage: https://envd.tensorchord.ai/
- Size: 3.2 MB
- Stars: 1,909
- Watchers: 22
- Forks: 152
- Open Issues: 135
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: CODEOWNERS
Lists
- awesome-python-data-science - envd - 🏕️ machine learning development environment for data science and AI/ML engineering teams. (Experimentation / Synthetic Data)
- awesome-production-machine-learning - envd - Machine learning development environment for data science and AI/ML engineering teams. (Model Training Orchestration)
- awesome-mlops - envd - Machine learning development environment for data science and AI/ML engineering teams. (Machine Learning Platform)
- awesome-datascience - envd
- awesome-llmops - envd - square) | (Training / IDEs and Workspaces)
- awesome-repositories - tensorchord/envd - 🏕️ Reproducible development environment (Go)
- awesome-stars - tensorchord/envd - 🏕️ Reproducible development environment (Go)
- my-awesome - tensorchord/envd - 🏕️ Reproducible development environment (Go)
- awesome-datascience - envd
- awesome-python-data-science - envd - 🏕️ machine learning development environment for data science and AI/ML engineering teams. (Experimentation / NLP)
- awesome-stars - tensorchord/envd - 🏕️ Reproducible development environment (mlops)
- awesome-stars - tensorchord/envd - `★1924` 🏕️ Reproducible development environment (Go)
- awesome-stars - tensorchord/envd - 🏕️ Reproducible development environment for AI/ML (Go)
- Awesome-LLMOps - envd - 🏕️ Reproducible development environment for AI/ML. (IDEs and Workspaces)
- awesome-stars - tensorchord/envd - 🏕️ Reproducible development environment (Go)
- awesome-stars - tensorchord/envd - 🏕️ Reproducible development environment (Go)
README
Development environment for AI/ML
## What is envd?
envd (`ɪnˈvdɪ`) is a command-line tool that helps you create the container-based development environment for AI/ML.
Creating development environments is not easy, especially with today's complex systems and dependencies. With everything from Python to CUDA, BASH scripts, and Dockerfiles constantly breaking, it can feel like a nightmare - until now!
Instantly get your environment running exactly as you need with a simple declaration of the packages you seek in build.envd and just one command: `envd up`!
## Why use `envd`?
Environments built with `envd` provide the following features out-of-the-box:
**Simple CLI and language**
`envd` enables you to quickly and seamlessly integrate powerful CLI tools into your existing Python workflow to provision your programming environment without learning a new language or DSL.
```python
def build():
install.python_packages(name = [
"numpy",
])
shell("zsh")
config.jupyter()
```
**Isolation, compatible with OCI image**
With `envd`, users can create an isolated space to train, fine-tune, or serve. By utilizing sophisticated virtualization technology as well as other features like [buildkit](https://github.com/moby/buildkit), it's an ideal solution for environment setup.
`envd` environment image is compatible with [OCI image specification](https://github.com/opencontainers/image-spec). By leveraging the power of an OCI image, you can make your environment available to anyone and everyone! Make it happen with a container registry like Harbor or Docker Hub.
**Local, and cloud**
`envd` can now be used on a hybrid platform, ranging from local machines to clusters hosted by Kubernetes. Any of these options offers an efficient and versatile way for developers to create their projects!
```sh
$ envd context use local
# Run envd environments locally
$ envd up
...
$ envd context use cluster
# Run envd environments in the cluster with the same experience
$ envd up
```
Check out the [doc](https://envd.tensorchord.ai/teams/kubernetes.html) for more details.
**Build anywhere, faster**
`envd` offers a wealth of advantages, such as remote build and software caching capabilities like pip index caches or apt cache, with the help of [buildkit](https://github.com/moby/buildkit) - all designed to make your life easier without ever having to step foot in the code itself!
Reusing previously downloaded packages from the PyPI/APT cache saves time and energy, making builds more efficient. No need to redownload what was already acquired before – a single download is enough for repeat usage!
With Dockerfile v1, users are unable to take advantage of PyPI caching for faster installation speeds - but `envd` offers this support and more!
Besides, `envd` also supports remote build, which means you can build your environment on a remote machine, such as a cloud server, and then push it to the registry. This is especially useful when you are working on a machine with limited resources, or when you expect a build machine with higher performance.
**Knowledge reuse in your team**
Forget copy-pasting Dockerfile instructions - use envd to easily build functions and reuse them by importing any Git repositories with the `include` function! Craft powerful custom solutions quickly.
```python
envdlib = include("https://github.com/tensorchord/envdlib")
def build():
base(os="ubuntu20.04", language="python")
envdlib.tensorboard(host_port=8888)
```
envdlib.tensorboard is defined in github.com/tensorchord/envdlib
```python
def tensorboard(
envd_port=6006,
envd_dir="/home/envd/logs",
host_port=0,
host_dir="/tmp",
):
"""Configure TensorBoard.
Make sure you have permission for `host_dir`
Args:
envd_port (Optional[int]): port used by envd container
envd_dir (Optional[str]): log storage mount path in the envd container
host_port (Optional[int]): port used by the host, if not specified or equals to 0,
envd will randomly choose a free port
host_dir (Optional[str]): log storage mount path in the host
"""
install.python_packages(["tensorboard"])
runtime.mount(host_path=host_dir, envd_path=envd_dir)
runtime.daemon(
commands=[
[
"tensorboard",
"--logdir",
envd_dir,
"--port",
str(envd_port),
"--host",
"0.0.0.0",
],
]
)
runtime.expose(envd_port=envd_port, host_port=host_port, service="tensorboard")
```
## Getting Started 🚀
### Requirements
- Docker (20.10.0 or above)
### Install and bootstrap `envd`
`envd` can be installed with `pip`, or you can download the binary [release](https://github.com/tensorchord/envd/releases) directly. After the installation, please run `envd bootstrap` to bootstrap.
```bash
pip install --upgrade envd
```
After the installation, please run `envd bootstrap` to bootstrap:
```bash
envd bootstrap
```
Read the [documentation](https://envd.tensorchord.ai/guide/getting-started.html#install-and-bootstrap-envd) for more alternative installation methods.
> You can add `--dockerhub-mirror` or `-m` flag when running `envd bootstrap`, to configure the mirror for docker.io registry:
>
>```bash title="Set docker mirror"
>envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn
>```
### Create an `envd` environment
Please clone the [`envd-quick-start`](https://github.com/tensorchord/envd-quick-start):
```bash
git clone https://github.com/tensorchord/envd-quick-start.git
```
The build manifest `build.envd` looks like:
```python title=build.envd
def build():
base(os="ubuntu20.04", language="python3")
# Configure the pip index if needed.
# config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
install.python_packages(name = [
"numpy",
])
shell("zsh")
```
*Note that we use Python here as an example but please check out examples for other languages such as R and Julia [here](https://github.com/tensorchord/envd/tree/main/examples).*
Then please run the command below to set up a new environment:
```bash
cd envd-quick-start && envd up
```
```bash
$ cd envd-quick-start && envd up
[+] ⌚ parse build.envd and download/cache dependencies 2.8s ✅ (finished)
=> download oh-my-zsh 2.8s
[+] 🐋 build envd environment 18.3s (25/25) ✅ (finished)
=> create apt source dir 0.0s
=> local://cache-dir 0.1s
=> => transferring cache-dir: 5.12MB 0.1s
...
=> pip install numpy 13.0s
=> copy /oh-my-zsh /home/envd/.oh-my-zsh 0.1s
=> mkfile /home/envd/install.sh 0.0s
=> install oh-my-zsh 0.1s
=> mkfile /home/envd/.zshrc 0.0s
=> install shell 0.0s
=> install PyPI packages 0.0s
=> merging all components into one 0.3s
=> => merging 0.3s
=> mkfile /home/envd/.gitconfig 0.0s
=> exporting to oci image format 2.4s
=> => exporting layers 2.0s
=> => exporting manifest sha256:7dbe9494d2a7a39af16d514b997a5a8f08b637f 0.0s
=> => exporting config sha256:1da06b907d53cf8a7312c138c3221e590dedc2717 0.0s
=> => sending tarball 0.4s
envd-quick-start via Py v3.9.13 via 🅒 envd
⬢ [envd]❯ # You are in the container-based environment!
```
### Set up Jupyter notebook
Please edit the `build.envd` to enable jupyter notebook:
```python title=build.envd
def build():
base(os="ubuntu20.04", language="python3")
# Configure the pip index if needed.
# config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
install.python_packages(name = [
"numpy",
])
shell("zsh")
config.jupyter()
```
You can get the endpoint of the running Jupyter notebook via `envd envs ls`.
```bash
$ envd up --detach
$ envd envs ls
NAME JUPYTER SSH TARGET CONTEXT IMAGE GPU CUDA CUDNN STATUS CONTAINER ID
envd-quick-start http://localhost:42779 envd-quick-start.envd /home/gaocegege/code/envd-quick-start envd-quick-start:dev false Up 54 seconds bd3f6a729e94
```
## Difference between v0 and v1
> **Note**
> To use the `v1` config file, add `# syntax=v1` to the first line of your `build.envd` file.
| Features | v0 | v1 |
| --- | --- | --- |
| is default for `envd[(a)](#v1-moby) |
> **Note**
> (a) To use the `moby` builder, you will need to create a new context with `envd context create --name moby-test --builder moby-worker --use`.
> For more information about the `moby` builder, check the [issue-1693](https://github.com/tensorchord/envd/issues/1693).
> **Important**
> For more details, check the [upgrade to v1](https://envd.tensorchord.ai/guide/v1.html) doc.
## More on documentation 📝
See [envd documentation](https://envd.tensorchord.ai/guide/getting-started.html).
## Roadmap 🗂️
Please checkout [ROADMAP](https://envd.tensorchord.ai/community/roadmap.html).
## Contribute 😊
We welcome all kinds of contributions from the open-source community, individuals, and partners.
- Join our [discord community](https://discord.gg/KqswhpVgdU)!
- To build from the source, please read our [contributing documentation](https://envd.tensorchord.ai/community/contributing.html) and [development tutorial](https://envd.tensorchord.ai/developers/development.html).
[](https://gitpod.io/#https://github.com/tensorchord/envd)
## Contributors ✨
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
Friends A.
📖 🎨
Aaron Sun
📓 💻
Aka.Fido
📦 📖 💻
Alex Xi
💻
Bingtan Lu
💻
Bingyi Sun
💻
Ce Gao
💻 📖 🎨 📆
Frost Ming
💻 📖
Guangyang Li
💻
Gui-Yue
💻
Haiker Sun
💻
Ikko Ashimine
💻
Isaac
💻
JasonZhu
💻
Jian Zeng
🎨 🤔 🔬
Jinjing Zhou
🐛 💻 🎨 📖
Jun
📦 💻
Kaiyang Chen
💻
Keming
💻 📖 🤔 🚇
Kevin Su
💻
Ling Jin
🐛 🚇
Manjusaka
💻
Nino
🎨 💻
Pengyu Wang
📖
Sepush
📖
Shao Wang
💻
Siyuan Wang
💻 🚇 🚧
Suyan
📖
To My
📖
Tumushimire Yves
💻
Wei Zhang
💻
Weixiao Huang
💻
Weizhen Wang
💻
XRW
💻
Xu Jin
💻
Xuanwo
💬 🎨 🤔 👀
Yijiang Liu
💻
Yilong Li
📖 🐛 💻
Yuan Tang
💻 🎨 📖 🤔
Yuchen Cheng
🐛 🚇 🚧 🔧
Yuedong Wu
💻
Yunchuan Zheng
💻
Zheming Li
💻
Zhenguo.Li
💻 📖
Zhenzhen Zhao
🚇 📓 💻
Zhizhen He
💻 📖
cutecutecat
💻
dqhl76
📖 💻
heyjude
💻
jimoosciuc
📓
kenwoodjw
💻
li mengyang
💻
nullday
🤔 💻
rrain7
💻
tison
💻
wangxiaolei
💻
wyq
🐛 🎨 💻
x0oo0x
💻
xiangtianyu
📖
xieydd
💻
xing0821
🤔 📓 💻
xxchan
📖
zhyon404
💻
杨成锴
💻
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
## License 📋
[Apache 2.0](./LICENSE)
