Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/zacharyvoase/django-castor

A content-addressable storage backend for Django.
https://github.com/zacharyvoase/django-castor

Last synced: 3 months ago
JSON representation

A content-addressable storage backend for Django.

Awesome Lists containing this project

README 

        
# `django-castor`



`django-castor` is a re-usable app for Django which provides a

**content-addressable storage backend**. The main class,

`djcastor.storage.CAStorage`, is a type of `FileSystemStorage` which saves files

under their SHA-1 digest.



*   No matter how many times the same file is uploaded, it will only ever be

    stored once, thus eliminating redundancy.



*   Filenames are pseudorandom and made up only of hexadecimal characters, so

    you don’t have to worry about filename collisions or sanitization.



*   `django-castor` shards files in the uploads directory based on their

    digests; this prevents filesystem issues when too many files are in a single

    directory.



For more information on the CAS concept, see the [wikipedia page][].



  [wikipedia page]: http://en.wikipedia.org/wiki/Content-addressable_storage



## Installation



    pip install django-castor  # or

    easy_install django-castor



## Usage



Basic usage is as follows:



    from django.db import models

    from djcastor import CAStorage



    class MyModel(models.Model):

        ...

        uploaded_file = models.FileField(storage=CAStorage(),

                                         upload_to='uploads')



At the moment, Django requires a non-empty value for the `upload_to` parameter.

Note that `CAStorage` will **not** use this value; if you need to customize the

destination for uploaded files, use the `location` parameter (see below).



For extended usage, there are several options you can pass to the `CAStorage`

constructor. The first two are inherited from the built-in `FileSystemStorage`:



*   `location`: The absolute path to the directory that will hold uploaded

    files. If omitted, this will be set to the value of the `MEDIA_ROOT`

    setting.



*   `base_url`: The URL that serves the files stored at this location. If

    omitted, this will be set to the value of the `MEDIA_URL` setting.



`CAStorage` also adds two custom options:



*   `keep_extension` (default `True`): Preserve the extension on uploaded files.

    This allows the webserver to guess their `Content-Type`.



*   `sharding` (default `(2, 2)`): The width and depth to use when sharding

    digests, expressed as a two-tuple. The following examples show how these

    parameters affect the sharding:



        >>> digest = '1f09d30c707d53f3d16c530dd73d70a6ce7596a9'



        >>> print shard(digest, width=2, depth=2)

        1f/09/1f09d30c707d53f3d16c530dd73d70a6ce7596a9



        >>> print shard(digest, width=2, depth=3)

        1f/09/d3/1f09d30c707d53f3d16c530dd73d70a6ce7596a9



        >>> print shard(digest, width=3, depth=2)

        1f0/9d3/1f09d30c707d53f3d16c530dd73d70a6ce7596a9



## Caveats



The first small caveat is that content-addressable storage is not suited to

rapidly-changing content. If your website modifies the contents of file fields

on a regular basis, it might be a better idea to use a UUID-based storage

backend for those fields.



The second, more important caveat with this approach is that if the parent model

of a file is deleted, the file will remain on disk. Because individual files may

be referred to by more than one model, and `django-castor` has no awareness of

these references, it leaves file deletion up to the developer.



For the most part, you can get away without deleting uploads. In fact,

content-addressable storage is often used for long-term archival systems, where

files are immutable and must be kept for future auditing (usually for compliance

with government regulations).



If disk space is at a premium and you need to delete uploads, there are two

approaches you might want to take:



*   Garbage collection: write a script that walks through the list of uploaded

    files and checks references to each one. If no references are found, delete

    the file.



*   Reference counting: denormalize the `FileField` into a separate model, and

    keep a count of all the models pointing to it. Once this count reaches zero,

    delete the file from the filesystem.



## (Un)license



This is free and unencumbered software released into the public domain.



Anyone is free to copy, modify, publish, use, compile, sell, or distribute this

software, either in source code form or as a compiled binary, for any purpose,

commercial or non-commercial, and by any means.



In jurisdictions that recognize copyright laws, the author or authors of this

software dedicate any and all copyright interest in the software to the public

domain. We make this dedication for the benefit of the public at large and to

the detriment of our heirs and successors. We intend this dedication to be an

overt act of relinquishment in perpetuity of all present and future rights to

this software under copyright law.



THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS

FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE

LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF

CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE

SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.



For more information, please refer to