Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/enigma-io/s3plz
A polite, minimal interface for sending python objects to and from Amazon S3.
https://github.com/enigma-io/s3plz
Last synced: about 1 month ago
JSON representation
A polite, minimal interface for sending python objects to and from Amazon S3.
- Host: GitHub
- URL: https://github.com/enigma-io/s3plz
- Owner: enigma-io
- Created: 2014-12-18T00:48:07.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2016-02-22T16:36:48.000Z (almost 9 years ago)
- Last Synced: 2024-10-08T08:48:32.850Z (2 months ago)
- Language: Python
- Homepage:
- Size: 61.5 KB
- Stars: 57
- Watchers: 14
- Forks: 8
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- starred-awesome - s3plz - A polite, minimal interface for sending python objects to and from Amazon S3. (Python)
README
# s3plz
A polite, minimal interface for sending python objects
to and from Amazon S3.
## Installation
```
pip install s3plz
```
## Tests
To run tests, you must first set these
environmental variabels:
```
export AWS_ACCESS_KEY_ID='fdsaf'
export AWS_SECRET_ACCESS_KEY='fdsaf'
export S3PLZ_TEST_BUCKET='s3://my-cool-bucket'
```
and then run:
```
nosetests
```
## Bacic Usage
```python
import s3plz
# Return an `s3plz.S3` object with
# methods for sending objects
# to and from Amazon S3.
plz = s3plz.connect('s3://asteroid',
key='navigate',
secret='shield',
serializer="json.gz",
public = False
)
# You can also set `AWS_ACCESS_KEY_ID` and
# `AWS_SECRET_ACCESS_KEY` as environmental variables
# instead of passing `key` and `secret` to `s3plz.connect`
# Serialize an object, format its
# filepath, put it on s3, and return
# the formatted filepath (with an absolute s3path)
# for your records
obj1 = {"key": "value"}
filepath = 'test/{key}.json.gz'
fp = plz.put(obj1, filepath, **obj1)
print fp
# >>> 's3://asteroid/test/value.json.gz'
# you can now fetch this object with its filepath
obj2 = plz.get(fp)
assert(obj1 == obj2)
```
## Customization.
## Filepaths
`s3plz` will attempt to format your filepath
for you given arbitrary `**kwargs` passed to
any method. You also have access to UTC
time via the "@" operator.
These include:
- **@second**: `56`
- **@minute**: `54`
- **@hour**: `23`
- **@day**: `29`
- **@month**: `01`
- **@year**: `2014`
- **@timestamp**: `1234567`
- **@date_path**: `2014/01/14`
- **@date_slug**: `2014-01-14`
- **@datetime_slug**: `2013-12-12-06-08-52`
- **@uid**: `dasfas-23r32-sad-3sadf-sdf`
For instance,
``` python
import s3plz
obj = {"key": "value"}
filepath = 'test/{key}/{@date_path}/{@uid}.json.gz'
plz = s3plz.connect('s3://my-bucket', serializer="json.gz")
fp = plz.put(obj, filepath, **obj)
print fp
# >>> 's3://my-bucket/test/value/2014/08/25/3225-sdsa-35235-asdfas-235.json.gz'
```
### Serialization
By default, `s3plz` will send strings to/from S3. You can also serialize / deserialize objects to / from `json.gz`, `json`, `gz`, `zip`, or `pickle` (set with `serializer` via `s3plz.connect`). These can also be changed on-the-fly by passing `serializer` as a `kwarg` into `get`, `put`, `stream` or `upsert`.
For example,
```python
import s3plz
s3 = s3plz.connect('s3://my-bucket')
obj1 = {"foo":"bar"}
fp = s3.put(obj1, "test/{foo}.json.gz", serializer="json.gz", **obj1)
obj2 = s3.get(fp, serializer="json.gz")
assert(obj1 == obj2)
string1 = "hello world"
fp = s3.put(string1, "test/string.zip", serializer="zip")
string2 = s3.get(fp, serializer="zip")
assert(string1 == string2)
```
However, you can also inherit from the core `s3plz.S3` class and overwrite the `serialize` and `deserialize` methods:
```python
from s3plz import S3
class SqlAlchemyToS3(S3):
def serialize(self, obj):
return "Do something here."
def deserialize(self, string):
return "Undo it."
s3 = SqlAlchemyToS3('s3://bucket')
print s3.get('s3://bucket/file.mycoolformat')
# >>> `A SqLAlchemy Model`
```
## API
Full API Methods
- `S3.put(data, filepath, headers, serializer, **kw)`
* desciption: Upload a file if it doesnt already exist,
otherwise return False
* params:
- `data`: Object to upload
- `filepath`: filepath format string
- `headers`: http headers to set on the object.
- `serializer`: serializer to use for the object.
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: s3uri / False
- `S3.upsert(data, filepath, headers, serializer, **kw)`
* desciption: Upload a file if it doesnt already exist, otherwise return False
* params:
- `data`: Object to upload
- `filepath`: filepath format string
- `headers`: http headers to set on the object.
- `serializer`: serializer to use for the object.
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: s3uri / False
- `S3.get(filepath, headers, serializer, **kw)`
* desciption: Download a file from s3. If it doesn't exist return None.
* params:
- `filepath`: filepath format string
- `headers`: http headers to set on the object.
- `serializer`: serializer to use for the object.
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: deserialized contents
- `S3.get_meta(filepath,**kw)`
* desciption: Get a dictionary of metadata fields for a filepath. If it doesn't exist, return {}
* params:
- `filepath`: filepath format string
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: dict of metadata.
- `S3.get_age(filepath, **kw)`
* desciption: Get the age of a a filepath. If it doesn't exist, return {}
* params:
- `filepath`: filepath format string
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: datetime.timedelta
- `S3.exists(filepath, **kw)`
* desciption: Check if a file exists on s3.
* params:
- `filepath`: filepath format string
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: s3_uri / False
- `S3.ls(filepath, **kw)`
* desciption: Return a generator of filepaths under a directory.
* params:
- `filepath`: filepath format string
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: generator of s3_uri's
- `S3.stream(filepath, headers, serializer, **kw)`
* desciption: Return a generator of tuples of (s3_uri, contents) under a directory.
* params:
- `filepath`: filepath format string
- `headers`: http headers to set on the object.
- `serializer`: serializer to use for the object.
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.
* returns: generator of tuples of (s3_uri, contents) from s3.
- `S3.delete(filepath, **kw)`
* desciption: Delete a filepath from s3.
* params:
- `filepath`: filepath format string
- `**kw`: arbitrary kwargs to pass to the `filepath` format string.