{"id":22849963,"url":"https://github.com/mozilla-ai/lumigator","last_synced_at":"2025-12-28T17:11:10.101Z","repository":{"id":255648233,"uuid":"773965129","full_name":"mozilla-ai/lumigator","owner":"mozilla-ai","description":"Source code for Mozilla.ai's Lumigator platform","archived":false,"fork":false,"pushed_at":"2025-07-14T16:48:53.000Z","size":13466,"stargazers_count":238,"open_issues_count":194,"forks_count":21,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-07-14T21:28:23.960Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://mozilla-ai.github.io/lumigator/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mozilla-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-18T17:52:00.000Z","updated_at":"2025-07-14T20:12:21.000Z","dependencies_parsed_at":"2024-09-16T09:58:44.987Z","dependency_job_id":"da28468f-1121-4aeb-91f5-65df56035df2","html_url":"https://github.com/mozilla-ai/lumigator","commit_stats":null,"previous_names":["mozilla-ai/lumigator"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/mozilla-ai/lumigator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-ai%2Flumigator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-ai%2Flumigator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-ai%2Flumigator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-ai%2Flumigator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mozilla-ai","download_url":"https://codeload.github.com/mozilla-ai/lumigator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mozilla-ai%2Flumigator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269831517,"owners_count":24482294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-13T05:01:25.645Z","updated_at":"2025-12-28T17:11:05.046Z","avatar_url":"https://github.com/mozilla-ai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Mozilla.ai Lumigator 🐊\n\n[![Lumigator pipeline](https://github.com/mozilla-ai/lumigator/actions/workflows/lumigator_pipeline.yaml/badge.svg?branch=main)](https://github.com/mozilla-ai/lumigator/actions/workflows/lumigator_pipeline.yaml)\n[![Docs](https://github.com/mozilla-ai/lumigator/actions/workflows/build_and_publish_docs.yaml/badge.svg)](https://github.com/mozilla-ai/lumigator/actions/workflows/build_and_publish_docs.yaml)\n\nLumigator is an open-source platform developed by [Mozilla.ai](https://www.mozilla.ai/) to help\nusers select the most suitable language model for their specific needs. Currently, Lumigator\nsupports the evaluation of summarization and translation tasks using sequence-to-sequence models such as those based on BART or T5 architectures, as well as causal models like GPT and Mistral (see the models [here](https://mozilla-ai.github.io/lumigator/get-started/suggested-models.html#model-types-and-parameters)). We plan to expand support to additional machine\nlearning tasks and use cases in the future.\n\nTo learn more about Lumigator's features and capabilities, see the\n[documentation](https://mozilla-ai.github.io/lumigator/), or get started with the\n[example notebook](/notebooks/walkthrough.ipynb) for a platform API walkthrough.\n\n\u003e [!NOTE]\n\u003e Lumigator is in the early stages of development. It is missing important features and\n\u003e documentation. You should expect breaking changes in the core interfaces and configuration\n\u003e structures as development continues.\n\n## Why Lumigator?\n\nAs more organizations turn to AI for solutions, they face the challenge of selecting the best model\nfrom an ever-growing list of options. The AI landscape is evolving rapidly, with [twice as many new\nmodels released in 2023 compared to the previous year](https://hai.stanford.edu/research/ai-index-report).\nHowever, in spite of existing [benchmarks and leaderboards](https://crfm.stanford.edu/helm/classic/latest/#/leaderboard) for some scenarios, one may find it challenging to compare models for their specific domain and use case.\n\n[The 2024 AI Index Report](https://arxiv.org/pdf/2405.19522)\nhighlighted that AI evaluation tools aren’t (yet) keeping up with the pace of development, making it\nharder for developers and businesses to make informed choices. Without a clear method for\ncomparing models, many teams end up using suboptimal solutions, or just choosing models based on\nhype, slowing down product progress and innovation.\n\nWith Lumigator MVP, Mozilla.ai aims to make model selection transparent, efficient, and empowering.\nLumigator provides a framework for comparing LLMs, using task-specific metrics to evaluate how well\na model fits your project’s needs. With Lumigator, we want to ensure that you’re not just picking a\nmodel—you’re picking the right model for your use case.\n\n## Get started\n\nThe simplest way to set up Lumigator is to deploy it locally using Docker Compose. To this end, you\nneed to have the following prerequisites installed on your machine:\n\n- A working installation of [Docker](https://docs.docker.com/engine/install/).\n    - On a Mac, you need Docker Desktop `4.3` or later and docker-compose `1.28` or later.\n    - On Linux, you need to follow the\n      [post-installation steps](https://docs.docker.com/engine/install/linux-postinstall/).\n- The system Python (version managers such as uv should be deactivated)\n\nYou can run and develop Lumigator locally using Docker Compose. This creates four container\nservices networked together to make up all the components of the Lumigator application:\n\n- `minio`: Local storage for datasets that mimics S3-API compatible functionality.\n- `backend`: Lumigator’s FastAPI REST API.\n- `ray`: A Ray cluster for submitting several types of jobs.\n- `frontend`: Lumigator's Web UI\n\n\u003e [!NOTE]\n\u003e Lumigator requires an SQL database to hold metadata for datasets and jobs. The local deployment\n\u003e uses SQLite for this purpose.\n\n\u003e [!WARNING]\n\u003e By default, Lumigator uses a local bucket named \"lumigator-storage\", created in minio.\n\u003e Important: we don't host or manage any public bucket with this name.\n\nTo start Lumigator locally, follow these steps:\n\n1. Clone the Lumigator repository:\n\n    ```bash\n    git clone git@github.com:mozilla-ai/lumigator.git\n    ```\n\n1. Navigate to the repository root directory:\n\n    ```bash\n    cd lumigator\n    ```\n\n1. If your system has an NVIDIA GPU, you have an additional pre-requirement: [install the NVIDIA Container Toolkit following their instructions](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). After that, open a terminal and run:\n    ```bash\n    export RAY_WORKER_GPUS=1\n    export RAY_WORKER_GPUS_FRACTION=1.0\n    export GPU_COUNT=1\n    ```\n    **Important: Continue the next steps in this same terminal.**\n\n\n1. If you intend to use Mistral API, OpenAI API or Deepseek API, you can easily configure your API keys in the Lumigator UI (under [Settings](https://mozilla-ai.github.io/lumigator/operations-guide/configuration.html#api-settings)) after you start Lumigator.\n\n1. From that same terminal, start Lumigator with:\n\n    ```bash\n    make start-lumigator\n    ```\n\nThe last command uses Docker Compose to launch all necessary containers for you.\nTo verify that Lumigator is running, open a web browser and navigate to\n[`http://localhost`](http://localhost): you should see Lumigator's UI.\n\nNow that Lumigator is running, you can start using it. The platform provides a REST API that allows\nyou to interact with the system. Run the [example notebook](/notebooks/walkthrough.ipynb) for a\nquick walkthrough.\n\n\u003e [!NOTE]\n\u003e When you run an experiment or generate ground truth in Lumigator for the first time, the required models will be downloaded. This may cause an initial delay. However, once downloaded, the models are cached locally, ensuring significantly faster subsequent runs.\n\nDespite the fact this is a local setup, it lends itself to more distributed scenarios. For instance,\none could provide different `AWS_*` environment variables to the backend container to connect to any\nprovider’s S3-compatible service, instead of minio. Similarly, one could provide a different\n`RAY_HEAD_NODE_HOST` to move compute to a remote ray cluster, and so on. Ref to the operational guides in the\n[docs](https://mozilla-ai.github.io/lumigator/) for more deployment options.\n\nIf you want to permanently set any of the environment variables above, you should add them directly to your\n[user configuration file](https://mozilla-ai.github.io/lumigator/operations-guide/configuration.html#how-should-i-set-my-own-settings) (`user.conf`) which can be created in Lumigator's 'dot' folder (e.g. `~/.lumigator/user.conf`).\n\n### Lumigator UI\nAlternatively, you can also use the UI to interact with Lumigator. Once a Lumigator session is up and running, the UI can be accessed by visiting [`http://localhost`](http://localhost). On the **Datasets** tab, first upload a csv data with columns `examples` and (optionally) `ground_truth`. Next, the dataset can be used to run an evaluation using the **Experiments** tab.\n\n### Terminate Lumigator session\nTo stop the containers you started using Docker Compose, simply run the following command:\n\n```bash\nmake stop-lumigator\n```\n\n\u003e [!NOTE]\nSince Lumigator is in active development, we always pull the latest images for the frontend and backend in our Docker Compose setup. If you are looking for a stable release, please check out the latest Git tag on our [Releases](https://github.com/mozilla-ai/lumigator/releases) page.\n**Important: You cannot simply change the images in the docker-compose file. Ensure that your working directory matches the Git tag you are using to maintain compatibility.\n\n## Kubernetes Installation\n\nYou can also deploy Lumigator on Kubernetes using our [helm](https://github.com/mozilla-ai/lumigator/tree/main/infra/helm) chart.\n\n## Documentation\n\nFor the complete Lumigator documentation, visit the\n[docs page](https://mozilla-ai.github.io/lumigator/).\n\n## Contribute\n\nFor contribution guidelines, see the [CONTRIBUTING.md](https://github.com/mozilla-ai/lumigator/blob/main/CONTRIBUTING.md) file.\n\n## Questions? Problems? Suggestions?\n\nTo report a bug or request a feature, please open a\n[GitHub issue](https://github.com/mozilla-ai/lumigator/issues/new/choose/). Be sure to check if\nsomeone else has already created an issue for the same topic.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmozilla-ai%2Flumigator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmozilla-ai%2Flumigator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmozilla-ai%2Flumigator/lists"}