https://github.com/DAGWorks-Inc/hamilton
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
https://github.com/DAGWorks-Inc/hamilton
dag data-analysis data-engineering data-science dataframe etl etl-framework etl-pipeline feature-engineering hacktoberfest lineage llmops machine-learning mlops orchestration pandas python rag software-engineering
Last synced: 27 days ago
JSON representation
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
- Host: GitHub
- URL: https://github.com/DAGWorks-Inc/hamilton
- Owner: DAGWorks-Inc
- License: bsd-3-clause-clear
- Created: 2023-02-23T17:16:48.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-11-27T05:12:15.000Z (5 months ago)
- Last Synced: 2024-12-02T10:14:07.398Z (5 months ago)
- Topics: dag, data-analysis, data-engineering, data-science, dataframe, etl, etl-framework, etl-pipeline, feature-engineering, hacktoberfest, lineage, llmops, machine-learning, mlops, orchestration, pandas, python, rag, software-engineering
- Language: Jupyter Notebook
- Homepage: https://hamilton.dagworks.io/en/latest/
- Size: 75.5 MB
- Stars: 1,885
- Watchers: 17
- Forks: 125
- Open Issues: 127
-
Metadata Files:
- Readme: README-DOCS.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# Documentation
Instructions for managing documentation on read the docs.
# Build locally
To build locally, you need to run the following -- make sure you're in the root of the repo:
```bash
pip install .[docs]
```
and then one of the following to build and view the documents:
```bash
sphinx-build -b dirhtml -W -E -T -a docs /tmp/mydocs
python -m http.server --directory /tmp/mydocs
```
or for auto rebuilding do:
```bash
sphinx-autobuild -b dirhtml -W -E -T --watch hamilton/ -a docs /tmp/mydocs
```
Then it'll be running on port 8000.
Note: readthedocs builds will fail if there are ANY WARNINGs in the build.
So make sure to check the build log for any warnings, and fix them, else you'll waste time debugging readthedocs
build failures.
# SimplePDF
To create a PDF, you can run the following:
```bash
sphinx-build -b simplepdf -W -E -T -a docs /tmp/mydocs
# or if you want to auto-rebuild:
sphinx-autobuild -b simplepdf -W -E -T --watch hamilton/ -a docs /tmp/mydocs
```
The PDF will be in `/tmp/mydocs` in a few minutes.
# reST vs myST
We use both! The general breakdown of when to use which is:
1. For documentation that we want to be viewable in github, use myST.
2. Otherwise default to using reST.