Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dagworks-inc/hamilton
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
https://github.com/dagworks-inc/hamilton
dag data-analysis data-engineering data-science dataframe etl etl-framework etl-pipeline feature-engineering featurization hacktoberfest lineage llmops machine-learning mlops numpy orchestration pandas python software-engineering
Last synced: 28 days ago
JSON representation
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
- Host: GitHub
- URL: https://github.com/dagworks-inc/hamilton
- Owner: DAGWorks-Inc
- License: bsd-3-clause-clear
- Created: 2023-02-23T17:16:48.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-04-23T00:34:57.000Z (7 months ago)
- Last Synced: 2024-04-23T05:14:42.664Z (7 months ago)
- Topics: dag, data-analysis, data-engineering, data-science, dataframe, etl, etl-framework, etl-pipeline, feature-engineering, featurization, hacktoberfest, lineage, llmops, machine-learning, mlops, numpy, orchestration, pandas, python, software-engineering
- Language: Jupyter Notebook
- Homepage: https://hamilton.dagworks.io/en/latest/
- Size: 49.9 MB
- Stars: 1,312
- Watchers: 12
- Forks: 80
- Open Issues: 129
-
Metadata Files:
- Readme: README-DOCS.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-systematic-trading - Hamilton - commit/dagworks-inc/hamilton/main) ![GitHub Repo stars](https://img.shields.io/github/stars/dagworks-inc/hamilton?style=social)| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc. (Basic Components / Computation)
- awesome-llmops - Hamilton - inc/hamilton.svg?style=flat-square) | (Large Scale Deployment / Workflow)
- awesome-production-machine-learning - Hamilton - inc/hamilton.svg?style=social) - Hamilton is a micro-orchestration framework for defining dataflows. Runs anywhere python runs (e.g. jupyter, fastAPI, spark, ray, dask). Brings software engineering best practices without you knowing it. Use it to define feature engineering transforms, end-to-end model pipelines, and LLM workflows. It complements macro-orchestration systems (e.g. kedro, luigi, airflow, dbt, etc.) as it replaces the code within those macro tasks. Comes with a self-hostable UI that captures lineage & provenance, execution telemetry & data summaries, and builds a self-populating catalog; usable in development as well as production. (Data Pipeline)
- awesome-data-engineering - Hamilton - Hamilton is a lightweight library to define data transformations as a directed-acyclic graph (DAG). If you like dbt for SQL transforms, you will like Hamilton for Python processing. (Workflow)
- awesome-ai-repositories - hamilton - inc/hamilton><img src="https://img.shields.io/github/stars/dagworks-inc/hamilton?style=social" width=100/></a> | (RAG Framework)
README
# Documentation
Instructions for managing documentation on read the docs.
# Build locally
To build locally, you need to run the following -- make sure you're in the root of the repo:
```bash
pip install .[docs]
```
and then one of the following to build and view the documents:
```bash
sphinx-build -b dirhtml -W -E -T -a docs /tmp/mydocs
python -m http.server --directory /tmp/mydocs
```
or for auto rebuilding do:
```bash
sphinx-autobuild -b dirhtml -W -E -T --watch hamilton/ -a docs /tmp/mydocs
```
Then it'll be running on port 8000.Note: readthedocs builds will fail if there are ANY WARNINGs in the build.
So make sure to check the build log for any warnings, and fix them, else you'll waste time debugging readthedocs
build failures.# reST vs myST
We use both! The general breakdown of when to use which is:
1. For documentation that we want to be viewable in github, use myST.
2. Otherwise default to using reST.