Projects in Awesome Lists tagged with data-processing-pipelines

https://github.com/NVIDIA/NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs

data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication

Last synced: 29 Jul 2025

https://github.com/NVIDIA-NeMo/Curator

Scalable data pre processing and curation toolkit for LLMs

data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication

Last synced: 20 Jul 2025

https://github.com/edrewitz/wxdata

A Python package of end-to-end weather data clients & raw data clients with VPN/PROXY support, data processors that decode variable keys from GRIB format into a plain-language format & various tools for assisting Python automated workflows, querying meteorological datasets and filling gaps in meteorological data.

automation data data-clients data-engineering data-engineering-pipeline data-processing data-processing-pipelines data-science meteorology meteorology-library python weather-data