https://github.com/isagalaev/django_dust

Distributed Upload STorage for Django. A file backend that mirrors all incoming media files to several servers
https://github.com/isagalaev/django_dust

Last synced: 6 days ago
JSON representation

Distributed Upload STorage for Django. A file backend that mirrors all incoming media files to several servers

• Host: GitHub
• URL: https://github.com/isagalaev/django_dust
• Owner: isagalaev
• License: bsd-3-clause
• Created: 2010-01-20T17:51:49.000Z (almost 15 years ago)
• Default Branch: master
• Last Pushed: 2012-01-31T20:59:03.000Z (almost 13 years ago)
• Last Synced: 2024-08-05T19:37:14.168Z (3 months ago)
• Language: Python
• Homepage:
• Size: 176 KB
• Stars: 65
• Watchers: 6
• Forks: 15
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
## SUMMARY

File backend django_dust (Distributed Upload STorage) can store files coming

into a Django site on several servers simultaneously. This works for files

uploaded by users from admin or custom Django forms and also files created

in application code provided it uses standrad [Storage API][1].

The backend is needed for sites working in multi-server environment in order

to accept uploaded files and have them available on all media servers

immediately for subsequent web requests that could be routed to any machine.

The backend can also work in a completely remote mode when files are stored

on different servers than those that handle web requests.

Django_dust nicely handles situation when some of media servers are unavailable

by keeping a queue of failed operations to repeat them afterwards.

## INSTALLATION AND SETUP

1.  Download and install python package in usual way. Django_dust requires

    [httplib2][2] library version at least 0.4.

2.  Put django_dust into INSTALLED_APPS:

        INSTALLED_APPS = (

            ...

            'django_dust',

        )

3.  Create the retry queue table with `syncdb`.

4.  Setup a scheduled task to run a command for processing the queue:

        ./manage.py process_failed_media

5.  Set django_dust as a default file backend:

        DEFAULT_FILE_STORAGE = 'django_dust.storage.DistributedStorage'

    Strictly speaking this is not obligatory. If needed the backend can be used

    for only particular file model fields or manually anywhere in code.

6.  Define backend settings:

        # List of file storage hosts

        DUST_HOSTS = [

            'hostname1',

            'hostanem2',

            # ...

        ]

    Other optional settings:

        # Retry storage backend setting -- import path for storage module.

        # Default is DB backend. See django_dust/backends/base for class

        # and method reference

        DUST_RETRY_STORAGE_BACKEND = 'django_dust.backends.db'

        # Whether to use a local file system. If files are stored on those same

        # servers that handle web requests then setting this flag to True

        # allows django_dust to access files locally saving time and bandwidth

        # on existence checking operations. Defaults to False.

        DUST_USE_LOCAL_FS = True

        # Timeout in seconds for accessing hosts over network. Defaults to 2.

        DUST_TIMEOUT = 2

    When using local file system make sure that `MEDIA_ROOT` points to the same

    place where a web server looks for files under `MEDIA_URL`. This is usually

    already the case.

## SETTING UP HTTP SERVERS

The backend uses HTTP to transfer files to media servers. For this to work

those servers should support HTTP methods PUT and DELETE. Despite of them

being part of HTTP itself they are traditionally implemented by external

modules supporting WebDAV. Hence make sure that  web servers have this module

enabled. For security reasons it's better to enable it only for internal IPs

to prevent external users from being able to write and delete server files.

An example of lighttpd config:

    server.modules += (

      "mod_webdav",

    )

    $HTTP["remoteip"] ~= "^192\.168\.0\.[0-9]+$" {

      "webdav.activate = "enable"

    }

Note that URLs for PUTting and DELETEing files are built using `MEDIA_URL` and

are essentially the same URLs that browsers use to access them.

[1]: http://docs.djangoproject.com/en/dev/ref/files/storage/

[2]: http://code.google.com/p/httplib2/