Projects in Awesome Lists tagged with large-scale-data-processing

A curated list of projects in awesome lists tagged with large-scale-data-processing .

• Sort

  • Recently synced
  • Stars

https://github.com/nvidia/nemo-curator

Scalable data pre processing and curation toolkit for LLMs

data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication

Last synced: 13 Apr 2025

https://github.com/NVIDIA/NeMo-Curator

Scalable data pre processing and curation toolkit for LLMs

data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication

Last synced: 27 Nov 2024

https://github.com/IBM/data-prep-kit

Open source project for data preparation of LLM application builders

code-quality data data-prep data-preparation data-preprocessing data-preprocessing-pipelines datacuration datarecipes deduplication finetuning large-language-models large-scale-data-processing llm llmapps malware python ray spark

Last synced: 11 Jan 2025