Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/kanedata/find-that-postcode

A server for getting information about postcodes, with an elasticsearch back end
https://github.com/kanedata/find-that-postcode

Last synced: 5 months ago
JSON representation

A server for getting information about postcodes, with an elasticsearch back end

Awesome Lists containing this project

README 

          
# Find that Postcode



This project creates an elasticsearch index based on the UK postcode file, and

runs a webserver on top of it for making queries. It's like a more lightweight

and less sophisticated version of [MapIt](https://mapit.mysociety.org.uk/).



## Setup



### 1. Setup elasticsearch



[Follow the instructions](https://www.elastic.co/downloads/elasticsearch) to

download and install elasticsearch.



Run the server. At the moment the scripts only work on a default server of

`localhost:9200` but a future version will have configurable host and port.



### 2. Install python dependencies



You'll need to install the python `elasticsearch` and `flask` libraries, either

directly through `pip` or by running a virtual environment and running:



```bash

pip install -r requirements.txt

```



The code is written in python 3 and hasn't been tested in python 2.



### 3. Point flask to the app



Flask needs to know which app it's running. The easiest way to do this is to create

a file called `.env` in the project directory, and add the following contents:



```bash

FLASK_APP=findthatpostcode

FLASK_ENV=development



# S3 credentials for boundaries

S3_REGION=XXXXXXXX

S3_ENDPOINT=XXXXXXXX

S3_ACCESS_ID=XXXXXXXX

S3_SECRET_KEY=XXXXXXXX

S3_BUCKET=XXXXXXXX

```



### 4. Create elasticsearch indexes



Run `flask init-db` to create the needed index and mappings

before data import.



### 5. Import postcodes



Run the following to import the data and save postcodes to the

elasticsearch index:



```bash

flask import nspl

```



This will then run the import process, fetching the latest version of the

right file. It takes a while to run as there are over

2.5 million postcodes. The data will be around 1.3 GB in size on the disk.



### 6. Import area codes



Run the following to import the code history database and register of geographic codes.



```bash

flask import rgc

flask import chd

flask import msoanames # imports the names for MSOAs from House of Commons Library

```



The URL of the files used can be customised with the `--url` parameter. Unfortunately the

ONS geoportal doesn't provide a persistent URL to the latest data.



### 6. Import boundaries (optional)



Boundaries are uploaded as individual area files to S3 storage.



Boundary files can be found on the [ONS Geoportal](http://geoportal.statistics.gov.uk/datasets?q=Latest_Boundaries&sort_by=name&sort_order=asc).

Generally the "Generalised Clipped" versions should be used to minimise the file

size. Open each boundary file link and find the "API" link on the right hand

side, and copy the `GeoJSON` link, or download the file.



These files are the latest available at April 2017:



- Countries: 

- Westminster Parliamentary Constituencies: 

- Counties and unitary authorities: 

- Local Authority Districts: 

- Regions: 

- CCGs: 

- European electoral regions: 

- Local Enterprise Partnerships: 

- NHS Commissioning Regions: 

- National Parks: 

- Police Force areas: 

- Travel to Work Areas: 

- Major Towns and Cities: 

- Combined Authorities: 



These files are large:



- Parishes (11,000):  --code-field=par22cd

- Wards (8,900):  - Didn't work

- LSOAs (35,000):  - Didn't work

- MSOAs (7,200):  - Didn't work

- Built-up Areas (5,800): 

- Built-up Area Sub-divisions (1,800): 



Import the boundary files by running:



```bash

flask import boundaries "https://opendata.arcgis.com/datasets/094f326b0b1247e3bcf1eb7236c24679_0.geojson"

```



You can add more than one URL to each import script.



### 7. Import placenames (optional)



A further related dataset is placenames. The [ONS has a list of these](http://geoportal.statistics.gov.uk/datasets/a6c138d17ac54532b0ca8ee693922f10_0)

which can be imported using the `import placenames` command. An entry for each

placename is added to the `geo_placenames` elasticsearch index.



```bash

flask import placenames

```



The `--url` parameter can be used to customise the URL used.



### 7. Import statistics (optional)



Statistics can be added to areas, using ONS data. The available statistics are

added to LSOAs, but could also be added to other areas.



```bash

flask import imd2025

flask import imd2019

flask import imd2015

```



The `--url` parameter can be used to customise the URL used to get the data.



### Run tests



```bash

python -m pytest tests

```



## Using the data



### Run the server



The project comes with a simple server (using the [flask](https://flask.palletsprojects.com/) framework) allowing

you to look at postcodes. The server returns either html pages (using `.html`)

or json data by default.



Run the server by:



```bash

flask run

```



By default the server is available at .



#### Server endpoints



The server has a number of possible uses:



- `/postcodes/SW1A+1AA.html` gives information about a particular postcode.

- `/areas/E09000033.html` gives information about an area, including example postcodes.

- `/areas/search.html?q=Winchester` finds any areas containing a search query.

- `/areatypes/laua.html` gives information about a type of area, including lists of

  example codes.

- `/areatypes.html` lists all the possible area types.

- `/points/53.490911,-2.095804.html` gives details of the postcode closest to the

  latitude, longitude point. If it's more than 10km from the nearest postcode it's

  assumed to be outside the UK.



### Elasticsearch REST api



The data is also now available in an elasticsearch index to be used in other local

applications using the elasticsearch REST api.



#### Find details on a postcode



```bash

curl "http://localhost:9200/geo_postcode/_doc/SW1A+1AA?pretty"

```



```json

{

  "_index": "postcode",

  "_type": "postcode",

  "_id": "SW1A 1AA",

  "_version": 1,

  "found": true,

  "_source": {

    "bua11": "E34004707",

    "oac11": "2C3",

    "park": "E99999999",

    "osnrth1m": 179645,

    "buasd11": "E35000546",

    "lsoa11": "E01004736",

    "lsoa21": "E01004736",

    "pcon": "E14000639",

    "pct": "E16000057",

    "nuts": "E05000644",

    "pcds": "SW1A 1AA",

    "ccg": "E38000031",

    "osgrdind": 1,

    "eer": "E15000007",

    "hlthau": "E18000007",

    "imd": 16419,

    "ward": "E05000644",

    "wz11": "E33031119",

    "ctry": "E92000001",

    "oseast1m": 529090,

    "pcd2": "SW1A 1AA",

    "laua": "E09000033",

    "rgn": "E12000007",

    "location": {

      "lon": -0.141588,

      "lat": 51.501009

    },

    "lat": 51.501009,

    "usertype": 1,

    "cty": "E99999999",

    "ttwa": "E30000234",

    "lep1": "E37000023",

    "pcd": "SW1A1AA",

    "teclec": "E24000014",

    "dointr": "1980-01-01T00:00:00",

    "oa11": "E00023938",

    "oa21": "E00023938",

    "long": -0.141588,

    "pfa": "E23000001",

    "ru11ind": "A1",

    "ruc21": "UN1",

    "hro": "E19000003",

    "msoa11": "E02000977",

    "msoa21": "E02000977",

    "lep2": null,

    "doterm": null

  }

}

```



#### Find details on an area



```bash

curl "http://localhost:9200/geo_area/_doc/E00046056?pretty"

```



```json

{

  "_index": "postcode",

  "_type": "code",

  "_id": "E00046056",

  "_version": 2,

  "found": true,

  "_source": {

    "code": "E00046056",

    "name": "",

    "name_welsh": null,

    "statutory_instrument_id": "1111/1001",

    "statutory_instrument_title": "GSS re-coding strategy",

    "date_start": "2009-01-01T00:00:00",

    "date_end": null,

    "parent": "E01009081",

    "entity": "E00",

    "owner": "ONS",

    "active": true,

    "areaehect": 3.75,

    "areachect": 3.75,

    "areaihect": 0,

    "arealhect": 3.75,

    "sort_order": "E00046056",

    "predecessor": ["00CNFN0006"],

    "successor": [],

    "equivalents": {

      "ons": "00CNFN0006"

    }

  }

}

```



## Todo / future features



- Find areas containing a point



## Dokku setup



```bash

# create app

dokku apps:create find-that-postcode



# add permanent data storage

dokku storage:mount find-that-postcode /var/lib/dokku/data/storage/find-that-postcode:/data



# enable domain

dokku domains:enable find-that-postcode

dokku domains:add find-that-postcode findthatpostcode.uk



# elasticsearch

sudo dokku plugin:install https://github.com/dokku/dokku-elasticsearch.git elasticsearch

dokku elasticsearch:create find-that-postcode-es

dokku elasticsearch:link find-that-postcode-es find-that-postcode



# SSL

sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

dokku config:set --no-restart find-that-postcode DOKKU_LETSENCRYPT_EMAIL=your@email.tld

dokku letsencrypt find-that-postcode

dokku letsencrypt:cron-job --add

```



### 2. Add as a git remote and push



On local machine:



```bash

git remote add dokku dokku@SERVER_HOST:find-that-postcode

git push dokku main

```



### 3. Setup and run import



On Dokku server run:



```bash

# setup and run import

dokku config:set find-that-postcode FLASK_APP=findthatpostcode

dokku config:set find-that-postcode S3_REGION=XXXXXXXX

dokku config:set find-that-postcode S3_ENDPOINT=XXXXXXXX

dokku config:set find-that-postcode S3_ACCESS_ID=XXXXXXXX

dokku config:set find-that-postcode S3_SECRET_KEY=XXXXXXXX

dokku config:set find-that-postcode S3_BUCKET=XXXXXXXX

dokku run find-that-postcode flask init-db

dokku run find-that-postcode flask import nspl --year=2011

dokku run find-that-postcode flask import nspl --year=2021

dokku run find-that-postcode flask import rgc

dokku run find-that-postcode flask import chd

dokku run find-that-postcode flask import msoanames

dokku run find-that-postcode flask import imd2025

dokku run find-that-postcode flask import imd2019

dokku run find-that-postcode flask import imd2015

dokku run find-that-postcode flask import placenames



# import boundaries

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/7be6a3c1be3b4385951224d2f522470a_0.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/094f326b0b1247e3bcf1eb7236c24679_0.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/0de4288db3774cb78e45b8b74e9eab31_0.geojson

dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Local_Authority_Districts_May_2022_UK_BGC_V3/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/284d82f437554938b0d0fbb3c6522007_0.geojson

dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Clinical_Commissioning_Groups_April_2021_EN_BGC/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/20595dbf22534e20944c9cee42c665b3_0.geojson

dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/LEP_MAY_2021_EN_BGC_V2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/edcbf58c70004d0f8d44501d07c38fe9_0.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/f41bd8ff39ce4a2393c2f454006ea60a_0.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/282af275c1a24c2ea64ff9e05bdd7d7d_0.geojson

dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Travel_to_Work_Areas_December_2011_UK_BGC_v2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson

dokku run find-that-postcode flask import boundaries https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/TCITY_2015_EW_BGG_V2/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/c6bd4568af5947519cf266b80a94de2e_0.geojson



# large boundary files

dokku run find-that-postcode flask import boundaries --code-field=par18cd https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Parishes_May_2022_EW_BGC/FeatureServer/0/query?outFields=*&where=1%3D1&f=geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/d2dce556b4604be49382d363a7cade72_0.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/e993add3f1944437bc91ec7c76100c63_0.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/29fdaa2efced40378ce8173b411aeb0e_2.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/f6684981be23404e83321077306fa837_0.geojson

dokku run find-that-postcode flask import boundaries https://opendata.arcgis.com/datasets/1f021bb824ee4820b353b4b58fab6df5_0.geojson

```