https://github.com/converged-computing/flux-burst
Remote bursting for Flux using a plugin framework (prototype in Python)
https://github.com/converged-computing/flux-burst
burst cloud flux-framework
Last synced: 5 months ago
JSON representation
Remote bursting for Flux using a plugin framework (prototype in Python)
- Host: GitHub
- URL: https://github.com/converged-computing/flux-burst
- Owner: converged-computing
- License: mit
- Created: 2023-06-23T21:17:44.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2023-11-02T18:21:14.000Z (over 2 years ago)
- Last Synced: 2025-09-10T05:36:56.266Z (9 months ago)
- Topics: burst, cloud, flux-framework
- Language: Python
- Homepage: https://converged-computing.github.io/flux-burst/
- Size: 3.63 MB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Flux Burst
[](#contributors-)
[](https://pypi.org/project/flux-burst/)
This is a Python module to coordinate Flux bursting. 🧋️
- ⭐️ [Documentation](https://converged-computing.github.io/flux-burst/) ⭐️
- 📦️ [Pypi Package](https://pypi.org/project/flux-burst/) 📦️
## Plugins
Current and desired plugins are:
- [flux-burst-local](https://github.com/converged-computing/flux-burst-local) to "burst" on a local HPC system
- [flux-burst-gke](https://github.com/converged-computing/flux-burst-gke) to burst to Google Kubernetes Engine
- [flux-burst-eks](https://github.com/converged-computing/flux-burst-eks) to burst to Amazon EKS
- [flux-burst-compute-engine](https://github.com/converged-computing/flux-burst-compute-engine) to burst to Google Cloud Compute Engine
## Questions
- How should the plugins (or client) manage checking when to create / destroy clusters?
- Can we have a better strategy for namespacing different bursts (e.g., beyond burst-0, burst-1, ..., burst-N)
- We need a reasonable default for what a plugin should do if something fails (e.g., setup/config)
- How should each plugin decide what size cluster to make? Right now I'm just taking the max size of the job, and we are assuming the jobs need the same node type.
- We will eventually want to use namespaces in a meaningful way (e.g., users)
- We will eventually want a specific burst for a job to be able to customize in more detail, e.g., the namespace or other attribute that comes from a jobspec (right now they are global to the plugin)
- Who controls cleanup? It can be done by the flux-burst global controller or a plugin, automated or manual, either way.
- All plugins should have support to read in YAML parameters (some spec for bursting)
- All plugins should be able to match a resource request to, for example, instance types.
- Should the plugin "local queue" (self.jobs) assume to be associated with one burst, where the size is the max job size?
- Should we derive names based on provided name + size so clusters are unique by name and size?
## 😁️ Contributors 😁️
We use the [all-contributors](https://github.com/all-contributors/all-contributors)
tool to generate a contributors graphic below.
## License
HPCIC DevTools is distributed under the terms of the MIT license.
All new contributions must be made under this license.
See [LICENSE](https://github.com/converged-computing/flux-burst/blob/main/LICENSE),
[COPYRIGHT](https://github.com/converged-computing/flux-burst/blob/main/COPYRIGHT), and
[NOTICE](https://github.com/converged-computing/flux-burst/blob/main/NOTICE) for details.
SPDX-License-Identifier: (MIT)
LLNL-CODE- 842614
