{"id":14970634,"url":"https://github.com/ofnote/lightex","last_synced_at":"2025-08-23T14:06:24.984Z","repository":{"id":62576073,"uuid":"196822426","full_name":"ofnote/lightex","owner":"ofnote","description":"A Flexible, Modular ML Experiment Framework","archived":false,"fork":false,"pushed_at":"2020-01-22T12:26:54.000Z","size":193,"stargazers_count":12,"open_issues_count":9,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-07-30T18:44:52.796Z","etag":null,"topics":["data-scientist","docker","experimentation","expt-manager","job-dispatcher","kubernetes","machine-learning","modular","pytorch","storage-virtualization","tensorflow","visualizers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ofnote.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-14T10:21:51.000Z","updated_at":"2024-05-21T08:49:34.000Z","dependencies_parsed_at":"2022-11-03T17:37:03.254Z","dependency_job_id":null,"html_url":"https://github.com/ofnote/lightex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ofnote/lightex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofnote%2Flightex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofnote%2Flightex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofnote%2Flightex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofnote%2Flightex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ofnote","download_url":"https://codeload.github.com/ofnote/lightex/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofnote%2Flightex/sbom","scorecard":{"id":703114,"data":{"date":"2025-08-11","repo":{"name":"github.com/ofnote/lightex","commit":"86aa1306356d20b714f1970fddc981f668ca06e5"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.8,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Code-Review","score":0,"reason":"Found 0/24 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: backend/mlflow/Dockerfile:1: pin your Docker image by updating continuumio/miniconda3:latest to continuumio/miniconda3:latest@sha256:4a2425c3ca891633e5a27280120f3fb6d5960a0f509b7594632cdd5bb8cbaea8","Warn: containerImage not pinned by hash: examples/efficient-net/Dockerfile:1: pin your Docker image by updating pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime to pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime@sha256:0a5c38be5121a53b23a7f76431017b3a18146693790d1299f3b22e070faedfba","Warn: containerImage not pinned by hash: examples/mnist-pytorch/Dockerfile:1: pin your Docker image by updating pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime to pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime@sha256:0a5c38be5121a53b23a7f76431017b3a18146693790d1299f3b22e070faedfba","Warn: containerImage not pinned by hash: examples/sklearn/Dockerfile:1: pin your Docker image by updating continuumio/miniconda3:latest to continuumio/miniconda3:latest@sha256:4a2425c3ca891633e5a27280120f3fb6d5960a0f509b7594632cdd5bb8cbaea8","Warn: pipCommand not pinned by hash: backend/mlflow/Dockerfile:6","Warn: pipCommand not pinned by hash: examples/efficient-net/Dockerfile:3","Warn: pipCommand not pinned by hash: examples/efficient-net/Dockerfile:7","Warn: pipCommand not pinned by hash: examples/efficient-net/Dockerfile:10","Warn: pipCommand not pinned by hash: examples/efficient-net/Dockerfile:12","Warn: pipCommand not pinned by hash: examples/mnist-pytorch/Dockerfile:3","Warn: pipCommand not pinned by hash: examples/mnist-pytorch/Dockerfile:7","Warn: pipCommand not pinned by hash: examples/mnist-pytorch/Dockerfile:10","Warn: pipCommand not pinned by hash: examples/sklearn/Dockerfile:3","Warn: pipCommand not pinned by hash: examples/sklearn/Dockerfile:6","Info:   0 out of  10 pipCommand dependencies pinned","Info:   0 out of   4 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}}]},"last_synced_at":"2025-08-22T05:41:28.935Z","repository_id":62576073,"created_at":"2025-08-22T05:41:28.936Z","updated_at":"2025-08-22T05:41:28.936Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271751896,"owners_count":24814702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-scientist","docker","experimentation","expt-manager","job-dispatcher","kubernetes","machine-learning","modular","pytorch","storage-virtualization","tensorflow","visualizers"],"created_at":"2024-09-24T13:43:53.397Z","updated_at":"2025-08-23T14:06:24.953Z","avatar_url":"https://github.com/ofnote.png","language":"Python","readme":"![experimental](https://img.shields.io/badge/stability-experimental-orange.svg)\n\n\n## LightEx\n\n`LightEx` is a lightweight experiment framework to create, monitor and record your machine learning experiments. Targeted towards individual data scientists, researchers, small teams and, in general, resource-constrained experimentation. Compatible with all machine-learning frameworks.\n\n**Project Status:** Alpha\n\nUnlike most experiment frameworks, `LightEx`sports a modular, and highly configurable design:\n\n* **dispatcher**: run experiments using `process`,`docker` containers or `kubernetes` pods. Switch between modes seamlessly by minor changes to config.\n* **mulogger**: log metrics and artifacts to multiple logger backends, using an *unified* API. Supports `mlflow`, `tensorboard` and `trains` — add new loggers easily as *plugins*.\n* **namedconf**:  python `dataclass` based flexible and *unified* configuration specification for jobs, parameters and model architectures. Config instances are *named* and can be *locally modified*.\n* **qviz**: query, compare and visualize your experiment results.\n\nThe run environment and parameters for your experiments are specified using a config file `lxconfig.py` in your project directory. Modify, inherit, and create new *named* config instances, on-the-fly, as you perform a series of experiments. \n\nLearn more about the **anatomy** of a ML experimentation framework [here](docs/anatomy.md). Also, why [yet another experiment framework] (#yet-another-experiment-framework) ?\n\n#### Benefits\n\n Start with a basic `train` or `eval` project. In a few minutes,\n\n* introduce systematic logging (multiple loggers) and visualization to your project\n* go from running a single experiment to multiple parameterized experiments, e.g.,\n  * multiple training runs over a set of hyper-parameters.\n  * multiple `efficient-net` or `bert` train or eval runs.\n  * a neural architecture search over multiple architectures in parallel.\n    \n\n### Installation\n\n\u003e  pip install -U lightex\n\n### Quick Start\n\nImagine we have an existing ML project, with the following model train command:\n\n\u003e `train.py --data-dir ./data —-lr 0.1 -—hidden_dim 512` \n\nNow, let us bring in `lightex` to organize and scale experiments for the project. In the main project directory, initialize `lightex` :\n\n\u003e `lx init`                               \n\nThis creates files `lxconfig.py` and `run_expts.py`. The file `lxconfig.py` contains pre-defined `dataclass`es for specifying *named* experiment configs.\n\n* The main class `Config`, contains three fields: `er` , `hp` and `run` (see the example below), which store configurations for logging and storage **resources**, **hyper-parameters**, and **runtime**, respectively. \n* By configuring these three fields (depending on the project environment, parameters), we can *seamlessly* generate multiple experiments via different dispatchers and log the results to one or more logger backends.\n* The config classes `Resources`, `HP` and `Run` for these fields are pre-defined  in `lxconfig.py`. See [config.md](docs/config.md) for full description of the defined dataclasses.\n* `Config` also includes a `get_experiments` function, which generates a list of experiment configs to be executed by the dispatcher. .\n\n```python\n\n@dataclass\nclass Config:\n    er: Resources \t\t\t\t\t#(Logging, Storage resources)\n    hp: HP \t\t\t\t\t\t#(Hyper-parameters of model, training)\n    run: Run \t\t\t\t\t\t#(Run-time config)\n\n    def get_experiments(self): #required: generate a list of experiments to run\n        expts = [Experiment(er=self.er, hp=self.hp, run=self.run)]\n        return expts\n```\n\nTo create a `Config` instance for our existing project, we instantiate \n\n* class `HP` with parameter values (`lr` and `hidden_dim`), and \n* class `Run` 's `cmd` field with the train command **template**. The template refers to `Experiment` objects returned by `get_experiments` function (of this `Config` instance).\n  * Need not create the template manually. Use [`argparse_to_command`](lightex/namedconf/config_utils.py) in your existing code to generate it.\n* class `Resources` with default values (stores experiments in `/tmp/ltex`).\n\n```python\ncmd=\"python train.py --data-dir {{run.data_dir}} --lr {{hp.lr}} --hidden_dim {{hp.hidden_dim}}\" #template placeholders refer to fields of Experiment instance\nRu1 = Run(cmd=cmd, experiment_name=\"find_hp\")\nH2 = HP(lr=1e-2, hidden_dim=512)\nR1 = Resources()\n\nC1 = Config(er=R1, hp=H1, run=Ru1)\n```\n\nOnce the config instance `C1` is defined, run your experiments as follows:\n\n\u003e python run_expts.py -c C1\n\n\n**That's it!** Now, your experiments, logs, metrics and models are executed and recorded systematically.\n\n------\n\n\n\n### Modify Experiment Parameters, Experiment Groups\n\nModify configs from previous experiments quickly using `replace` and run new experiments. \n\n**Example**: Create a new `HP` instance and replace it in `C1` to create a new `Config`. *Recursive* replace also supported.\n\n```python\nH2 = HP(lr=1e-3, hidden_dim=1024)\nC2 = replace(C1, hp=H2) #inherit er=R1 and run=Ru1\n```\n\n\u003e python run_expts.py -c C2\n\n##### Experiment Groups\n\nWe can create multiple similar experiments programmatically and run them in parallel. To specify such **experiment groups**, specify a set of `HP`s in a `HPGroup` and update the `get_experiments` function (see [scripts/lxconfig.py](scripts/lxconfig.py) for an example). In the usual case, these experiments have different hyper-parameters but share `Resources` and `Run` instances.\n\n**Note**: Although LightEx pre-defines the dataclass hierarchy, it allows the developer plenty of flexibility in defining the individual fields of classes, in particular, the fields of the `HP` class. \n\n### Adding Logging to your Code\n\nUse the unified `MultiLogger` [API](lightex/mulogger) to log metrics and artifacts to multiple logger backends.\n\nSupported Loggers: `mlflow`, `tensorboard`,`trains`, `wandb`.\n\n```python\nfrom lightex.mulogger import MLFlowLogger, MultiLogger, PytorchTBLogger\nlogger = MultiLogger(['mlflow', 'trains'])\nlogger.start_run()\n# log to trains only\nlogger.log('trains', ltype='hpdict', value={'alpha': alpha, 'l1_ratio': l1_ratio})\n# log to mlflow only\nlogger.log('mlflow', ltype='scalardict', value={'mae': mae, 'rmse': rmse, 'r2': r2}, step=1)\n# log to all\nlogger.log('*', ltype='scalardict', value={'mae': mae, 'rmse': rmse, 'r2': r2}, step=3)\n# log scalars and tensors, if supported by the logger backend\nlogger.log('trains', ltype='1d', name='W1', value=Tensor(..), histogram=True, step=4)\nlogger.end_run()\n```\n\nOr, use one of the existing loggers' API directly.\n\n```python\nlogger = MLFlowLogger(); mlflow = logger.mlflow  # call mlflow API\nlogger = PytorchTBLogger() ; writer = logger.writer #call tensorboard's API\n# Similarly, use TrainsLogger for trains and WBLogger for wandb\n```\n\n**Note**: Except for changes in logging, no changes are required to your existing code! Setup the logger backend using scripts [here](backend/).\n\n### Switch to Docker\n\nSetting up the `lxconfig` instances pays off here! \n\nNow, add a `Dockerfile` to your project which builds the runtime environment with all the project dependencies. Update the `Build` instance inside `Resources` config. See [examples/sklearn](examples/sklearn), for example.\n\n\u003e python run_expts.py -c C2 -e docker\n\nBoth your code and data are mounted on the container (no copying involved) — minimal disruption in your dev cycle.\n\n## Advanced Features\n\nMore advanced features are in development stage.\n\n**Running Experiments on multiple nodes / servers**\n\nIf you've setup a docker `swarm` or `kubernetes` cluster, few changes to the existing config instance allow changing the underlying experiment dispatcher.\n\nWe need to virtualize code (by adding to Dockerfile) and storage.\n\nCreate a shared NFS on your nodes. Switch storage config to the NFS partition. Setup scripts will be added.\n\n**Better QViz module, Logger Plugins**\n\nImprovements to qviz module and a better plugin system for loggers being developed.\n\n**Setup Summary**\n\nIn summary, `LightEx` involves the following **one-time setup**:\n\n- config values in `lxconfig.py`\n- Setup backend logger servers (only the ones required). Instructions [here](backend/). (Optional)\n- Update logging calls in your code to call `mulogger` API. (Optional)\n- Dockerfile for your project, if you want to use containers for dispatch. (Optional)\n\nWhile `LightEx` is quick to start with, it is advisable to spend some time understanding the [config schema](llightex/config_base.py).\n\n\n\n### Dependencies\n\nPython \u003e 3.6 (require `dataclasses`, included during install). Other dependencies defined in [setup.py](setup.py).\n\n\n\n### Design Challenges\n\n- The configuration system allows you to abstract away the locations of the **code** (project directory), **data** directory, and the **output** directory, via the `cmd` field of `Run`. This enables running your code as-is across all dispatcher environments (process, docker, kubernetes). \n- A significant part of experiment manager design is about setting up and propagating a giant web of configuration variables. \n  - No optimal choice here: `json`,`yaml`,`jsonnet`— all formats have issues. \n  - Using `dataclass`es, we can write complex config specs, with built-in inheritance and ability to do *local updates*. Tiny bit of a learning curve here, bound to python, but we gain a lot of flexibility.\n- A unified `mulogger` API to abstract away the API of multiple logging backends.\n- Designing multiple dispatchers with similar API, enabling containers and varying storage options.\n- Read more on challenges [here](docs/anatomy.md).\n\n### References\n\n- ML Experiment Frameworks: [kubeflow](https://github.com/kubeflow/kubeflow), [mlflow](https://www.mlflow.org/docs/latest/index.html), [polyaxon](https://polyaxon.com/), ...\n- Loggers: [sacred](https://sacred.readthedocs.io/en/latest/index.html), [trains](https://github.com/allegroai/trains), [wandb](https://www.wandb.com/),  [Trixi](https://github.com/MIC-DKFZ/trixi), [ml_logger](https://github.com/episodeyang/ml_logger)\n- Motivating Dataclasses [intro](https://blog.jetbrains.com/pycharm/2018/04/python-37-introducing-data-class/), [how-different](https://stackoverflow.com/questions/47955263/what-are-data-classes-and-how-are-they-different-from-common-classes)\n- Flexible configuration\n  - in modeling: allennlp, gin, jiant.\n  - in orchestration: [ksonnet](https://github.com/ksonnet), kubernetes-operator \n- On the pains of ML experimentation\n  - an article from [wandb](https://www.wandb.com/articles/iteratively-fine-tuning-neural-networks-with-weights-biases) \n\nMost current (2019 Q3) tools focus on the *logger* component and provide selective `qviz` components. `kubeflow` and `polyaxon` are tied to the (k8s) *dispatcher*. Every tool has its own version of config management — mostly *yaml* based, where config types are absent or have a non-nested config class. Config-specific languages have been also proposed (ksonnet, sonnet, gin).\n\n\n\n### Yet Another Experiment Framework\n\nSystematic experimentation tools are essential for a data scientist. Unfortunately, many existing tools (`kubeflow`, `mlflow`, `polyaxon`) are too monolithic, kubernetes-first, cloud-first, target very diverse audiences and hence spread too thin, and yet lack important dev-friendly features. `sacred` 's design is' tightly coupled and requires several `sacred`-specific changes to your main code. Other tools cater only to a specific task , e.g., `tensorboard` only handles log recording and visualization. Also, contrasting different experiment frameworks is hard: there is no standardized expt-management architecture for machine learning and most open-source frameworks are undergoing a process of adhoc requirements discovery. \n\n### Author\n\nNishant Sinha, [OffNote Labs](http://offnote.co) (nishant@offnote.co, @[medium](https://medium.com/@ekshakhs), @[twitter](https://twitter.com/ekshakhs))\n\n### Contributors\n\nAkarsh E S, [github](https://github.com/AkarshES)\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofnote%2Flightex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fofnote%2Flightex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofnote%2Flightex/lists"}