https://github.com/blackportal-ai/nebula
A Package Manager for Machine Learning Datasets and Models.
https://github.com/blackportal-ai/nebula
blackportal blackportal-ai dataset datasets machine-learning ml nebula package-manager rust
Last synced: 6 days ago
JSON representation
A Package Manager for Machine Learning Datasets and Models.
- Host: GitHub
- URL: https://github.com/blackportal-ai/nebula
- Owner: blackportal-ai
- License: bsd-3-clause
- Created: 2024-12-16T11:09:06.000Z (4 months ago)
- Default Branch: master
- Last Pushed: 2025-03-16T11:34:03.000Z (30 days ago)
- Last Synced: 2025-04-02T11:07:10.229Z (13 days ago)
- Topics: blackportal, blackportal-ai, dataset, datasets, machine-learning, ml, nebula, package-manager, rust
- Language: Rust
- Homepage: https://blackportal.ai
- Size: 129 KB
- Stars: 75
- Watchers: 4
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
- fucking-awesome-rust - blackportal-ai/nebula - A Package Manager for Machine Learning Datasets and Models.  (Libraries / Artificial Intelligence)
- awesome-rust - blackportal-ai/nebula - A Package Manager for Machine Learning Datasets and Models.  (Libraries / Artificial Intelligence)
- trackawesomelist - blackportal-ai/nebula (⭐55) - A Package Manager for Machine Learning Datasets and Models.  (Recently Updated / [Who Wants to Be a Millionare](https://www.boardgamecapital.com/who-wants-to-be-a-millionaire-rules.htm))
README
[![discord badge]](https://discord.gg/g5HtkAzRNG)
[![x handle]][x badge]
[x badge]: https://twitter.com/intent/follow?screen_name=BlackPortal_AI
[x handle]: https://img.shields.io/twitter/follow/BlackPortal_AI.svg?style=social&label=Follow
[discord badge]: https://img.shields.io/discord/1320514043424931861
A package manager for machine learning datasets and models.
## Main Goal
Design of overal architecture and documentation to start onboarding contributors.
Current state of the [Architecture](./architecture.md)
## Nebula CLI
Uses either a legecy (interactive) command-line or a ratatui frontend. Whereas the latter is
experimental and hidden behind the feature flag `tui`.
### Commands
The Nebula CLI provides a set of commands to interact with the Nebula package manager.
This is a copy from the `nebula_cli --help` output:
```shell
A package manager for machine learning datasets and models acting as client for Nebula registries.
Usage: nebula_cli [OPTIONS] [COMMAND]
Commands:
init init a virtual environment in the given folder (not yet)
status prints status information (not yet)
install Installs a package (not yet)
update Updates a specific package or all packages (not yet)
uninstall Uninstall a specific package or all packages (not yet)
search Searches packages by complex criteria (not yet)
list List packages that fit simple criteria e.g.(non)-installed,
sync Sync the local cache with the remote registry
help Print this message or the help of the given subcommand(s)
Options:
--tui use a [ratatui] based terminal user interface instead of a simple cmd-tool
-i, --interactive start the cmd-tool in interactive mode, that allows typing multiple commands
-v, --verbose use verbose output, only in non TUI mode
-t, --tick-rate Tick rate, i.e. number of ticks per second in tui [default: 4]
-f, --frame-rate Frame rate, i.e. number of frames per second in tui [default: 60]
-h, --help Print help
-V, --version Print version
help # Shows help for a specific command
```
Examples:
```shell
nebula sync # gets the newest metadata locally from the remote registry
nebula search climate_data # Search for packages related to climate data
nebula install neural_net_model_v2 --version 1.0.1 # Install a specific version of a model
nebula install climate_dataset_2023 # Install the latest version of a dataset
nebula update --all # Update all installed datasets and models
nebula uninstall outdated_model # Remove an outdated model
```
## Nebula Registry
The Nebula CLI communicates with the registry via gRPC using [Tonic](https://github.com/hyperium/tonic). The registry can be self-hosted if desired and using the CLI we can configure the registry URL.
As for now the registry supports the following endpoints:
```protobuf
service NebulaPackageQuery {
// Gets detailed information for one specific package
rpc GetPackageInfo (PackageRequest) returns (PackageInfo);
// List all packages with very simple search criteria
rpc ListPackages (ListPackagesRequest) returns (PackageList);
// Search packages applying several filters
rpc SearchPackages (SearchPackagesRequest) returns (PackageList);
}
```
For more information see the [proto file](./nebula_common/proto/nebula.proto).
The datasets and models are stored elsewhere for now and based on the URL the client is expected to send further GET requests.
## Nebula Registry Web
In the far away future we might implement a web interface for the Nebula registry.
## Contributors
The following contributors have either helped to start this project, have contributed
code, are actively maintaining it (including documentation), or in other ways
being awesome contributors to this project. **We'd like to take a moment to recognize them.**
[
](https://github.com/mjovanc)
[
](https://github.com/DarthB)
## License
The BSD 3-Clause License.