https://github.com/src-d/sparkpickle
Pure Python implementation of reading SequenceFile-s with pickles written by Spark's saveAsPickleFile()
https://github.com/src-d/sparkpickle
pickle python spark
Last synced: about 2 months ago
JSON representation
Pure Python implementation of reading SequenceFile-s with pickles written by Spark's saveAsPickleFile()
- Host: GitHub
- URL: https://github.com/src-d/sparkpickle
- Owner: src-d
- License: apache-2.0
- Created: 2016-11-07T17:32:19.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-06-08T09:31:07.000Z (about 8 years ago)
- Last Synced: 2025-05-05T05:05:11.209Z (about 2 months ago)
- Topics: pickle, python, spark
- Language: Python
- Homepage:
- Size: 26.4 KB
- Stars: 24
- Watchers: 8
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://travis-ci.org/src-d/sparkpickle) [](https://pypi.python.org/pypi/sparkpickle)
SparkPickle
===========
Pure Python implementation of reading SequenceFile-s with pickles written by
Spark's [saveAsPickleFile()](http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.saveAsPickleFile).
This is needed if you store the results from Spark in the efficient binary pickle
format and want to load them locally on your computer, without any Spark installation,
given only the actual files.
[Article about creating this project.](https://blog.sourced.tech/post/reading_pyspark_pickles_locally)
Installation
------------
```
pip install sparkpickle
```
Supports Python 2.7 and 3.x.
Usage
-----
View the contents of the file via command line:
```
python -m sparkpickle /path/to/file
```
Code:
```python
import sparkpickle
for obj in sparkpickle.load_gen("/path/to/file"):
print(obj)
```
API
---
There are 3 functions: `load()`, `loads()` and `load_gen()`. The first two
are similar to those found in "pickle" package, whereas the last one is the
generator which yields deserialized objects and thus provides the minimal
memory footprint.
License
-------
Apache 2.0.