https://github.com/uber/petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
https://github.com/uber/petastorm

deep-learning machine-learning parquet parquet-files pyarrow pyspark pytorch sysml tensorflow

Last synced: about 2 months ago
JSON representation

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

• Host: GitHub
• URL: https://github.com/uber/petastorm
• Owner: uber
• License: apache-2.0
• Created: 2018-06-15T23:15:29.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2023-12-02T05:11:31.000Z (10 months ago)
• Last Synced: 2024-06-30T20:36:44.201Z (3 months ago)
• Topics: deep-learning, machine-learning, parquet, parquet-files, pyarrow, pyspark, pytorch, sysml, tensorflow
• Language: Python
• Homepage:
• Size: 2.69 MB
• Stars: 1,763
• Watchers: 41
• Forks: 281
• Open Issues: 177
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE

Awesome Lists containing this project

• awesome-list - Petastorm - Enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. (Deep Learning Framework / Deployment & Distribution)
• awesome-mlops - Petastorm - Enables single machine or distributed training and evaluation of deep learning models. (Optimization Tools)
• awesome-python-machine-learning-resources - GitHub - 49% open · ⏱️ 24.08.2022): (分布式机器学习)