https://github.com/mozboz/transformapetl

https://github.com/mozboz/transformapetl

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/mozboz/transformapetl
• Owner: mozboz
• Created: 2016-09-05T18:33:11.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2016-12-01T16:01:36.000Z (about 8 years ago)
• Last Synced: 2024-08-02T12:47:52.693Z (4 months ago)
• Language: Python
• Size: 43 KB
• Stars: 1
• Watchers: 2
• Forks: 4
• Open Issues: 3
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-starred - mozboz/transformapetl - (others)

README

        
# Transformap ETL

## Setup dev:

    make installpostgres

    make virtualenv

    make setupdb

    source .env/bin/activate

    make runserver

Refer to Makefile for further details; it should serve as a guide to how to

manage the local Python development environment.

## Postgresql JSONB field support in Django

The JSON field support appears to be quite good, django's queryset API supports

several operations for querying data inside JSONB fields.

See: https://docs.djangoproject.com/en/1.10/ref/contrib/postgres/fields/#containment-and-key-operations

## Misc

ETL business logic: places/management/commands

REST API configuration: places/api.py

## Deployment

For development, testing or demo purposes, use the Django development server:

python manage.py runserver 0.0.0.0:9000

Navigate to http://localhost:9000 ...

- /admin for the Admin area

- /places-api for the REST API interface

- /map-instance/MAP-INSTANCE-ID for the GEOJSON API

For production use, deploy Django using WSGI. See:

https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/

## Usage

To run a job, create a YAML job file in /jobs, then:

    python manage.py transformap -i jobs/jobfile.yaml

This fetch and save the partner's data to the database.