https://github.com/stitchfix/splits
A Python library for dealing with splittable files
https://github.com/stitchfix/splits
Last synced: 9 months ago
JSON representation
A Python library for dealing with splittable files
- Host: GitHub
- URL: https://github.com/stitchfix/splits
- Owner: stitchfix
- License: mit
- Created: 2014-08-05T23:39:23.000Z (almost 12 years ago)
- Default Branch: master
- Last Pushed: 2019-12-10T18:57:12.000Z (over 6 years ago)
- Last Synced: 2025-04-16T21:25:24.919Z (over 1 year ago)
- Language: Python
- Size: 39.1 KB
- Stars: 42
- Watchers: 158
- Forks: 8
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Splits
Splits is a library for reading and writing files in splittable chunks.
It works on any file-like object.
There is built in support for writing and reading split files from S3.
It also has built in support for gzip.
## Installation
```
$ pip install splits
```
## Usage
```python
from splits import SplitWriter, SplitReader
from splits.s3 import S3File, GzipS3File
if __name__ == '__main__':
with SplitWriter('s3://test-bucket/test-multifile',
suffix='.txt', lines_per_file=100,
fileClass=GzipS3File) as w:
w.writelines([str(x) for x in range(0, 1000)])
with SplitReader('s3://test-bucket/test-multifile',
fileClass=GzipS3File) as r:
for line in r:
print line
```
## Tests
```
$ pip install tox
```
To run the tests in both Python2 and Python3 run,
```
$ tox
```