{"id":28476283,"url":"https://github.com/flintml/flintml","last_synced_at":"2026-01-18T12:36:52.580Z","repository":{"id":288844672,"uuid":"969085680","full_name":"flintml/flintml","owner":"flintml","description":"ML infrastructure for teams that just want to get sh*t done.","archived":false,"fork":false,"pushed_at":"2025-06-26T13:39:54.000Z","size":5929,"stargazers_count":78,"open_issues_count":1,"forks_count":5,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-26T14:32:36.229Z","etag":null,"topics":["data-science","deltalake","jupyter","machine-learning","mlops","polars"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flintml.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-19T10:57:15.000Z","updated_at":"2025-06-26T13:31:25.000Z","dependencies_parsed_at":"2025-06-26T14:37:27.685Z","dependency_job_id":null,"html_url":"https://github.com/flintml/flintml","commit_stats":null,"previous_names":["bosonstack/boson","flintml/flint","flintml/flintml"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/flintml/flintml","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flintml%2Fflintml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flintml%2Fflintml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flintml%2Fflintml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flintml%2Fflintml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flintml","download_url":"https://codeload.github.com/flintml/flintml/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flintml%2Fflintml/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28536002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T10:13:46.436Z","status":"ssl_error","status_checked_at":"2026-01-18T10:13:11.045Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","deltalake","jupyter","machine-learning","mlops","polars"],"created_at":"2025-06-07T15:06:32.168Z","updated_at":"2026-01-18T12:36:52.559Z","avatar_url":"https://github.com/flintml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003c!-- Logo --\u003e\n  \u003cimg width=\"60%\" src=\"docs/_assets/logo-text.png\" alt=\"FlintML Logo Text\" /\u003e\u003cbr/\u003e\n\n  \u003c!-- Badges, all inside the same HTML block --\u003e\n  \u003cimg src=\"https://img.shields.io/badge/version-v0.2.1-cf051c\" alt=\"Version 0.2.1\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/license-BSL_1.1-blue\" alt=\"License BSL 1.1\" /\u003e\n\n  \u003c/br\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n**FlintML** is the all-in-one, self-hosted ML platform for real-world teams. Simple, flexible, fast to deploy, and built for people solving actual problems—not chasing hype.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"100%\" src=\"docs/_assets/screenshot.png\" alt=\"FlintML Logo Text\" /\u003e\u003cbr/\u003e\n\u003c/div\u003e\n\n## 🚀 Quickstart\n\n```bash\ncurl -sL https://raw.githubusercontent.com/flintml/flint/main/flintml-quickstart.sh | bash\n```\n\nFlintML will become available at `localhost:8701`. The first time you execute code may take a couple of minutes while FlintML downloads the relevant worker image.\n\n**Note:** ensure Docker is installed and you are on a Linux (or WSL) machine (ARM currently not supported.)\n\n## \u003cimg src=\"docs/_assets/logo.png\" alt=\"FlintML Logo\" width=\"17\"/\u003e Why FlintML?\n\nFlintML enables teams to deliver end-to-end ML quickly and with minimal infrastructure overhead. With FlintML, all key components of the MLOps process are centralised, providing an integrated and developer-centric experience.\n\n**Core features:**\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **[Delta Lake](https://github.com/delta-io/delta) storage layer** – adds ACID guarantees and time travel to your data pipelines with scalable, versioned storage.\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **Unified data catalog** – tables, models, artefacts, and any other file types are treated as first-class data citizens.\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **Efficient data processing with [Polars](https://github.com/pola-rs/polars)** – leverage lazy execution for memory-efficient data operations.\n\n**Integrated tools:**\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **Experiment tracking with [Aim](https://github.com/aimhubio/aim)** – run experiments and compare them all in one place.\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **Familiar notebook development environment** – all functions are seamlessly integrated with Jupyter Lab.\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **Workflow orchestration via [Dagster](https://github.com/dagster-io/dagster) (WIP\\*)** – load data, retrain models, and run inference on a schedule.\n\n**Platform \u0026 Deployment:**\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **Flexible and reproducible compute** – switch between compute [drivers](docs/concepts.md#drivers) to fit your use case, or write your own.\n- \u003cimg src=\"docs/_assets/red-check.png\" width=\"16\" alt=\"✔\"\u003e **Composable, declarative infrastructure** – have full control over your Docker Compose deployment.\n\n## 🔎 Demo\n\nTo get a sense of what you can do with FlintML, check out the [Instacart Kaggle example](examples/instacart.ipynb). You can also read about FlintML [concepts](docs/concepts.md) and check out the [reference](docs/reference.md) to learn more about the platform's capabilities\n\n## ⚙️ Customising Your Deployment\n\n### Data Storage\n\nFlintML ships with its own [Storage](docs/concepts.md#flint-control-plane) service that depends on the mounts, `storage_data` and `storage_meta`. If you wish to specify custom volumes, you should create an override `docker-compose.override.yml` and compose it when spinning up flint. See the [docs](https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/).\n\n### Environment Variables\n\n#### `FLINT_PORT`\n\nDefines the port number at which the FlintML platform will be served.\n\nDefault of `8701`.\n\n#### `STORAGE_USER` and `STORAGE_PASSWORD`.\n\nThese environment variables define the login credentials that [Worker Containers](docs/concepts.md#worker-containers) and services in the [Flint Control Plane](docs/concepts.md#flint-control-plane) use to authenticate to the [Flint Catalog](docs/concepts.md#flint-catalog).\n\nDefaults are `admin` and `password` respectively.\n\n#### `DRIVER_CONFIG`\n\nThis must be the file path to the desired [Worker Configuration](#worker-configuration).\n\nNo default. See [example](src/.env.example) env file.\n\n#### `DOCKER_SOCKET`\n\nFlintML uses [Worker Containers](docs/concepts.md#worker-containers) to execute code and run jobs. These containers are orchestrated by the configured driver. If the [Local Driver](docs/concepts.md#local-driver) is selected, the Docker socket must be mounted to the Compute Manager service in the [Flint Control Plane](docs/concepts.md#flint-control-plane).\n\nSince mounting the Docker socket has security implications, the default value for this environment variable is `/dev/null`, meaning that the Docker socket IS NOT mounted.\n\nThe [quickstart](#quickstart) uses the Local Driver for simplicity and thus mounts the docker socket. See [example](src/.env.example) env file.\n\n### Worker Configuration\n\nTo see the full worker specificaton, see the [schema](src/compute-manager/worker-config-schema.json). Some example configurations are provided below.\n\n[Local Driver](docs/concepts.md#local-driver) with mounted NAS volume:\n\n```yaml\ndriver:\n  type: local\n  image: flintml/worker-base:latest\n  mounts:\n    # \u003cmount_name\u003e: \u003chost_mount_point\u003e\n    image-data: /mnt/nas/images # Will be available at `/mnt/image-data` inside worker container.\n```\n\nTo customise compute environments, you will need to use a [custom image](docs/concepts.md#custom-images).\n\n### Building Locally\n\nAll FlintML codefiles live under `src/`. You will see two sets of Compose, env and worker config files - `build` and `release-template`. `release-template` files are used to define the release tarball and thus can be ignored for local development. \n\nTo build locally, you must firstly build the base worker image so it exists in your local Docker registry:\n\n```bash\ndocker build -f ./src/worker-base/Dockerfile -t worker-base:latest ./src/\n```\n\nThen you can spin up the platform by using the `build` files:\n\n```bash\ndocker compose -f ./src/docker-compose.build.yml --env-file ./src/.env.build up\n```\n\n**Note:** If you update the dependencies of any packages in `src/common-lib`, make sure you run `./update-common-lib.sh` to update dependant Poetry lock files.\n\n## 🎯 Roadmap\n\n1. Workflows (in-progress)\n2. Data upload support (can be achieved by defining a worker mount but is inconvenient)\n2. Libcloud driver\n3. Multi-user support\n\n## 🤝 Contributing\n\nWe would be stoked for you to get involved with FlintML development! If you'd like to get more involved, please contact us at contact@flintml.com.\n\n- 💬 [Start a discussion](https://github.com/bosonstack/boson/discussions)\n- 🛠️ [Fix a bug](https://github.com/bosonstack/boson/issues/new)\n- 🧠 [Request a feature](https://github.com/bosonstack/boson/issues/new)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflintml%2Fflintml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflintml%2Fflintml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflintml%2Fflintml/lists"}