{"id":18496415,"url":"https://github.com/oddbird/mlt","last_synced_at":"2025-07-25T13:13:03.930Z","repository":{"id":1484764,"uuid":"1731854","full_name":"oddbird/mlt","owner":"oddbird","description":null,"archived":false,"fork":false,"pushed_at":"2011-12-12T21:25:02.000Z","size":23607,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-07-15T08:55:26.169Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oddbird.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-05-11T06:27:45.000Z","updated_at":"2018-11-27T16:52:57.000Z","dependencies_parsed_at":"2022-07-29T18:08:52.959Z","dependency_job_id":null,"html_url":"https://github.com/oddbird/mlt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oddbird/mlt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oddbird%2Fmlt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oddbird%2Fmlt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oddbird%2Fmlt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oddbird%2Fmlt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oddbird","download_url":"https://codeload.github.com/oddbird/mlt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oddbird%2Fmlt/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266789035,"owners_count":23984219,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T13:29:20.095Z","updated_at":"2025-07-25T13:13:03.905Z","avatar_url":"https://github.com/oddbird.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Master Lookup Table\n===================\n\nDevelopment\n-----------\n\nIf you want to run this project in a `virtualenv`_ to isolate it from other\nPython projects on your system, create a virtualenv and activate it.  Then\nrun ``bin/install-reqs`` to install the dependencies for this project into\nyour Python environment. Python 2.7 is required.\n\nYou'll need a PostGIS-enabled PostgreSQL 9.0 database, with the ``citext``\ncontrib module loaded into it, for this project. See the `GeoDjango\ninstallation documentation`_ for more details on setting up PostGIS and a\nPostGIS template database. To enable the ``citext`` module, connect to the\n``template_postgis`` database as the Postgres superuser and run ``\\i\n/usr/share/postgresql/9.0/contrib/citext.sql`` (this is the location of the\n``citext.sql`` file on Ubuntu; will vary depending on your Postgres\ninstall). Once you have a PostGIS template database with ``citext`` enabled,\ncreate your database for MLT with a command like ``createdb -T template_postgis\nmlt``.\n\nYou'll probably need to create an ``mlt/settings/local.py`` file with some\ndetails of your local configuration, including, most likely, your database name\nand user (unless they are both named \"mlt\", the default).  See\n``mlt/settings/local.sample.py`` for a sample that can be copied to\n``mlt/settings/local.py`` and modified.\n\nOnce this configuration is done, you should be able to run ``./manage.py\nsyncdb --migrate``, then ``./manage.py runserver`` and access the MLT in\nyour browser at ``http://localhost:8000``.\n\n.. _virtualenv: http://www.virtualenv.org\n.. _GeoDjango installation documentation: http://docs.djangoproject.com/en/1.3/ref/contrib/gis/install/\n\nTo install the necessary Ruby Gems for Compass/Sass development, run\n``bin/install-gems requirements/gems.txt``.  Update\n``requirements/gems.txt`` if newer gems should be used.\n\n\nImporting initial addresses\n---------------------------\n\nThe MLT includes a user interface for importing batches of addresses from a CSV\nfile, but this interface can't be used for initially populating the MLT, as the\nnumber of addresses in the initial import is likely too large to complete\nwithin a reasonable time for a web request. In order to manually populate\ninitial addresses, run ``python manage.py shell`` and then the following\ncommands in the shell.\n\nThis assumes a four-column CSV file, with a header row, with columns for\n``pl``, ``street``, ``city``, ``state``. A user is necessary so that the\naddresses appear in the changelog as \"created\"; swap out the username below for\nan actual user that you've already created::\n\n    \u003e\u003e\u003e from django.contrib.auth.models import User\n    \u003e\u003e\u003e user = User.objects.get(username=\"example\")\n    \u003e\u003e\u003e from mlt.map.importer import CSVAddressImporter\n    \u003e\u003e\u003e importer = CSVAddressImporter(\n    ...     user, fieldnames=[\"pl\", \"street\", \"city\", \"state\"], header=True)\n    \u003e\u003e\u003e importer.process_file(\"/path/to/addresses.csv\")\n\nNote that this will take quite a while depending on the number of addresses;\npossibly as long as several hours.\n\n\nDeployment\n----------\n\nIn addition to the above configuration, in any production deployment this\nentire app should be served exclusively over HTTPS (since almost all use of the\nsite is authenticated, and serving authenticated pages over HTTP invites\nsession hijacking attacks). Ideally, the non-HTTP URLs should redirect to the\nHTTPS version. The ``SESSION_COOKIE_SECURE`` setting should be set to ``True``\nwhen the app is served over HTTPS.\n\nThere is an alternate settings file available, ``mlt/settings/prod.py`` which\npre-sets some settings that are generally appropriate for a production\ndeployment. Set the ``DJANGO_SETTINGS_MODULE`` environment variable to\n``mlt.settings.prod`` in order to use this settings file (overrides in\n``mlt/settings/local.py`` will still be respected).\n\nThe production settings also require a Celery daemon process, ``celeryd``, to\nbe running, in order to handle some tasks asynchronously and speed up the user\nexperience (if ``celeryd`` is not running, parcel-loading will not work, and\nbulk address changes will not be recorded in the change history). For quick\ntesting and debugging, you can simply run ``python manage.py celeryd -l info``\nin a terminal. For real deployment, you'll want to daemonize it and run it in\nthe background; see the documentation on `running celeryd as a daemon`_. You\nmay also want to read the `Celery monitoring and management guide`_ for more\nin-depth information about monitoring and managing your Celery instance.\n\nThis app also uses the new `staticfiles contrib app`_ in Django 1.3 for\ncollecting static assets from reusable components into a single directory\nfor production serving.  Under \"runserver\" in development this is handled\nautomatically.  In production, run ``./manage.py collectstatic`` to collect\nall static assets into the ``collected-assets`` directory (or whatever\n``STATIC_ROOT`` is set to in ``settings_local.py``), and make those\ncollected assets available by HTTP at the ``STATIC_URL`` setting.\n\n.. _running celeryd as a daemon: http://celery.readthedocs.org/en/latest/cookbook/daemonizing.html\n.. _Celery monitoring and management guide: http://celery.readthedocs.org/en/latest/userguide/monitoring.html\n.. _staticfiles contrib app: http://docs.djangoproject.com/en/1.3/howto/static-files/\n\n\nJSON API\n--------\n\nA simple JSON API is available for querying addresses and batches. The API is\navailable at the root URL of ``/api/v1/``.\n\n\nAPI Keys\n~~~~~~~~\n\nEvery API call must include the HTTP header ``X-Api-Key``, whose value must be\na valid API key. API keys can be created via the MLT admin interface. An API\ncall without a valid API key will return an HTTP 403 response, with the\nfollowing body::\n\n    {\"success\": False, \"error\": \"Invalid API key.\"}\n\n\nResource URLs\n~~~~~~~~~~~~~\n\nThe response to a query to the top-level API URL (``/api/v1/``) will have a\n``resource_urls`` key listing the resource URLs available via the\nAPI. Currently these are::\n\n    {\n        \"resource_urls\":\n            {\n            \"addresses\": \"/api/v1/addresses/\",\n            \"batches\": \"/api/v1/batches/\",\n            },\n        \"success\": True,\n        }\n\n\nResult formats\n~~~~~~~~~~~~~~\n\nThe body of every API response includes a top-level boolean ``success``\nkey. The value of this key will be true if the request completed successfully;\nfalse if an error occurred. In the latter case the body will also include an\n``error`` key describing the nature of the error.\n\nList resource responses will also include a ``total`` key, giving the total\nnumber of resources matching the given filters (even though not all might be\ndisplayed due to paging).\n\n\nSorting\n~~~~~~~\n\nEvery resource list URL can accept one or more sort fields via the ``sort`` key\nin the URL querystring. Any field of the returned data for that resource type\ncan be sorted on; valid fields are listed in the reference for that resource\ntype. Prepend a ``-`` to the field name to sort descending rather than\nascending on that field. An example multi-field sorted query URL::\n\n    /api/v1/batches/?sort=city\u0026sort=-street\n\n\nPaging\n~~~~~~\n\nAll list resources are paged by default, with a default page size of 20\nitems. Paging is controlled by offset/limit via ``start`` and ``num`` keys in\nthe URL querystring, rather than by page number. Results will begin with the\n``start``-th item, and ``num`` items will be returned. For example, the\nfollowing query will return 10 addresses, beginning with the 11th address (in\nother words, the second page of size-10 pages)::\n\n    /api/v1/addresses/?start=11\u0026num=10\n\n\nFiltering\n~~~~~~~~~\n\nList resources can be filtered by the value of fields on the resource (see\nbelow for full list of fields for each resource type). Filters are provided in\nthe URL querystring::\n\n    /api/v1/batches/?tag=foo\n\nTimestamp fields can be filtered on using \"[date]\" or \"[date1] to [date2]\",\ne.g.::\n\n    /api/v1/batches/?timestamp=11/5/2011+to+11/10/2011\n\n\nAddresses\n~~~~~~~~~\n\nEach address result includes the following fields::\n\n    id\n    street\n    city\n    state\n    street_number\n    street_prefix\n    street_name\n    street_type\n    street_suffix\n    notes\n    multi_units\n    complex_name\n    pl\n    mapped_by\n    mapped_timestamp\n    needs_review\n    batches\n\nThe ``mapped_by`` field should be sorted/filtered as ``mapped_by__username``,\ne.g.::\n\n    /api/v1/addresses/?mapped_by__username=blametern\n\nThe ``batches`` field contains a list of batches the address was imported as\npart of; each batch will have ``user``, ``timestamp``, and ``tag``\nkeys. Addresses can be filtered by batch using ``batches__tag``, e.g.::\n\n    /api/v1/addresses/?batches__tag=foo\n\nAddresses can be sorted by latest batch timestamp using\n``latest_batch_timestamp``::\n\n    /api/v1/addresses/?sort=latest_batch_timestamp\n\n\nBatches\n~~~~~~~\n\nEach batch includes the following fields::\n\n    timestamp\n    tag\n    user\n    addresses_url\n\nThe ``addresses_url`` field is the API URL to get a list of all addresses in\nthis batch.\n\nThe ``user`` field should be sorted or filtered as ``user__username``.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foddbird%2Fmlt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foddbird%2Fmlt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foddbird%2Fmlt/lists"}