{"id":16821224,"url":"https://github.com/andrewharvey/asgs2pgsql","last_synced_at":"2025-10-11T21:38:36.732Z","repository":{"id":2944166,"uuid":"3957306","full_name":"andrewharvey/asgs2pgsql","owner":"andrewharvey","description":"Scripts to load the ABS ASGS into PostgreSQL (with PostGIS geometries)","archived":false,"fork":false,"pushed_at":"2020-09-15T12:13:14.000Z","size":158,"stargazers_count":11,"open_issues_count":4,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-11T03:08:55.858Z","etag":null,"topics":["abs-asgs","census-data"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andrewharvey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-04-07T11:22:13.000Z","updated_at":"2020-09-15T12:13:17.000Z","dependencies_parsed_at":"2022-08-31T04:20:23.385Z","dependency_job_id":null,"html_url":"https://github.com/andrewharvey/asgs2pgsql","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/andrewharvey/asgs2pgsql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewharvey%2Fasgs2pgsql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewharvey%2Fasgs2pgsql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewharvey%2Fasgs2pgsql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewharvey%2Fasgs2pgsql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andrewharvey","download_url":"https://codeload.github.com/andrewharvey/asgs2pgsql/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andrewharvey%2Fasgs2pgsql/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279008836,"owners_count":26084517,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abs-asgs","census-data"],"created_at":"2024-10-13T10:59:29.572Z","updated_at":"2025-10-11T21:38:36.715Z","avatar_url":"https://github.com/andrewharvey.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# asgs2pgsql\nasgs2pgsql provides the Australian Bureau of Statistics (ABS) [Australian Statistical Geography Standard](http://www.abs.gov.au/websitedbs/D3310114.nsf/home/Australian+Statistical+Geography+Standard+(ASGS)) (ASGS) in a PostgreSQL database using the PostGIS extension to store the geospatial data.\n\nScripts to generate GeoJSON and MBTiles outputs from the PostGIS data are included too.\n\nYou may either use these scripts to [build your own database from source data](#build-from-source), or `pg_restore` a [pre-built database dump](#loading-an-existing-db-dump).\n\nThe ASGS is published as a combination of non-spatial CSV files and spatial SHAPE files. Both of these files are used in this loader and combined together to form this (unofficial) ASGS PostgreSQL schema and flat GeoJSON/MBTiles.\n\nYou may be interested in the corresponding [abs2pgsql loader scripts](https://github.com/andrewharvey/abs2pgsql) which will load the ABS 2011 Census of Population and Housing into PostgreSQL, making use of this ASGS schema to provide the geographic standard to those statistics.\n\n## About the ASGS\nThe ASGS consists of around 22 individual structures which are classified into ABS Structures and Non-ABS Structures [as shown in this diagram](http://www.abs.gov.au/websitedbs/D3310114.nsf/4a256353001af3ed4b2562bb00121564/c453c497aadde71cca2576d300026a38/$FILE/ASGS%202011%20Structure%20and%20Summary.pdf).\n\nThe ASGS is broken down into Volumes 1-5.\n\n* Volume 1 - 1270.0.55.001 consists of the Main Structure and Greater Capital City Statistical Areas ([documentation](http://www.abs.gov.au/ausstats/subscriber.nsf/log?openagent\u00261270055001_july%202011.pdf\u00261270.0.55.001\u0026Publication\u0026D3DC26F35A8AF579CA257801000DCD7D\u0026\u0026July%202011\u002623.12.2010\u0026Latest))\n* Volume 2 - 1270.0.55.002 consists of the Indigenous Structure ([documentation](http://www.abs.gov.au/AUSSTATS/subscriber.nsf/log?openagent\u00261270055002_july%202011.pdf\u00261270.0.55.002\u0026Publication\u0026FE2D2D707996F20ACA25791000152669\u0026\u0026July%202011\u002620.09.2011\u0026Previous))\n* Volume 3 - 1270.0.55.003 consists of the Non-ABS Structures ([documentation](http://www.abs.gov.au/AUSSTATS/subscriber.nsf/log?openagent\u00261270055003_oct%202011.pdf\u00261270.0.55.003\u0026Publication\u0026469CDA45CE2B94CCCA257937000D966F\u0026\u0026July%202011\u002631.10.2011\u0026Previous))\n* Volume 4 - 1270.0.55.004 consists of the Significant Urban Areas and Section of State/Urban Centres and Localities Structure (SUA, SOS/UCL) ([documentation](http://www.abs.gov.au/AUSSTATS/subscriber.nsf/log?openagent\u00261270055004_july%202011.pdf\u00261270.0.55.004\u0026Publication\u00261080B7CB374FC771CA257A980013D404\u0026\u0026July%202011\u002616.10.2012\u0026Latest))\n* Volume 5 - 1270.0.55.005 consists of the Remoteness Structure ([documentation](http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1270.0.55.005July%202011?OpenDocument))\n* Correspondences - 1270.0.55.006 provides correspondences between the older ASGC and the newer ASGS [documentation](http://www.abs.gov.au/AUSSTATS/abs@.nsf/productsbyCatalogue/A08009E6A2BA5ABBCA257A2900197A49?OpenDocument))\n\nThe ABS Structures (vol 1, 2, 4, 5) are usually updated for each Census and the Non-ABS Structures (vol 3) are updated annually as required (so not all volume 3 structures are updated each year)\n\n| Volume | Structure     | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 |\n|--------|---------------|------|------|------|------|------|------|------|\n| 1      | Main, GCCSA   | :+1: |      |      |      |      | :+1: |      |\n| 2      | Indigenous    | :+1: |      |      |      |      | :+1: |      |\n| 3      | LGA\\*         | :+1: | :+1: | :+1: | :+1: | :+1: | :+1: | :+1: |\n| 3      | SED\\*         | :+1: |      |      |      |      | :+1: | :+1: |\n| 3      | CED\\*         | :+1: |      | :+1: |      |      | :+1: | :+1: |\n| 3      | POA           | :+1: |      |      |      |      | :+1: |      |\n| 3      | SSC\\*         | :+1: |      |      |      |      | :+1: |      |\n| 3      | ADD           | :+1: |      |      |      |      | :+1: |      |\n| 3      | NRMR          | :+1: |      |      |      |      | :+1: |      |\n| 3      | TR            | :+1: | :+1: | :+1: | :+1: | :+1: | :+1: |      |\n| 4      | SUA, UCL, SOS | :+1: |      |      |      |      | :+1: |      |\n| 5      | Remoteness    | :+1: |      |      |      |      | :+1: |      |\n\n\\*Keep in mind the Volume 3 stuctures are approximations indended for statistical purposes, not as a substitute for administrative boundaries. The asterix (\\*) indicates avaliable via the [PSMA Administrative Boundaries](https://data.gov.au/dataset/psma-administrative-boundaries) and corresponding processing scripts at https://github.com/andrewharvey/psma-admin-bdys-data.\n\n## Schema\n### Codes\nPrimary key codes for the ASGS structures are generally made unique by concatenating the code with the code of the parent structure which it was built from. For example the S/T structure is built up from the SA4 structure. That is S/T's are built up from one or more SA4's.\n\nThis means that the unique code for SA4's is only unique within its S/T, so to obtain a globally unique code for that SA4 you need to prepend the S/T code.\n\nSo for example SA1's are unique with respect to their 11 digit code. However that 11 digit code is made up of S/T . SA4 . SA4 . SA3 . SA2 . SA1.\n\n## Copyright\nThe ABS ASGS data is Copyright (c) Commonwealth of Australia and as per http://www.abs.gov.au/websitedbs/D3310114.nsf/Home/©+Copyright?opendocument it is released under the [Creative Commons Attribution 2.5 Australia license](http://creativecommons.org/licenses/by/2.5/au/).\n\nAll the files within this asgs2pgsql repository are released under the [CC0](http://creativecommons.org/publicdomain/zero/1.0/) license by Andrew Harvey \u003candrew.harvey4@gmail.com\u003e. Although not required, I would prefer you give attribution and release derived works or modifications under the same CC0 license.\n\n    To the extent possible under law, the person who associated CC0\n    with this work has waived all copyright and related or neighboring\n    rights to this work.\n    http://creativecommons.org/publicdomain/zero/1.0/\n\n## Build from Source\nRunning these scripts is akin to building software from source. If you just want a copy of the database without needing to build it from source skip to the [last section of this README](#loading-an-existing-db-dump).\n\n### Requirements\nDebian dependencies: `make, gdal-bin (\u003e= 1.7.0), libdbd-pg-perl, (postgis \u003e= 2.1), libtext-csv-perl, libtext-csv-xs-perl, unzip, wget`\n\nThe scripts assume you have a PostgreSQL database up and running. We leave authentication to this database your responsibility through the [PostgreSQL environment variables](http://www.postgresql.org/docs/current/static/libpq-envars.html) (see #setting-up-the-database-environment).\n\nThe simplest authentication setup is to allow local unauthenticated access to your database. You can do this by adding the following example line to `/etc/postgresql/*/main/pg_hba.conf` (replacing `database` and `user` as you choose).\n\n    local   database   user      trust\n\nThis allows the `database` database to be accessed by the `user` user without authentication via local socket connections.\n\n### Setting up the database environment\nYou need to set up and export some PG environment variables otherwise the PostgreSQL defaults will be used. For example,\n\n    export PGHOST=localhost # not necessary if localhost\n    export PGDATABASE=database\n    export PGUSER=user\n\nRefer to the [PostgreSQL documentation](http://www.postgresql.org/docs/current/static/libpq-envars.html) for details on the environment variables which you can set.\n\nIf you don't already have a database and database user set up, then on Debian you could:\n\n    sudo su - postgres\n    createuser --no-createdb --no-createrole --superuser user\n    createdb --owner=user database\n    exit\n\n### Stage 1: Downloading the source ASGS data\nThe download parameters are hard configured within `01-download-asgs.sh`. To avoid downloading more than you need you can comment out or delete files you don't need. the bottom part of `01-download-asgs.sh` was created from `00-make-download-code.sh`.\n\n    make download\n\nThis should download and unzip the ASGS Volume 1-6 files for years 2011 and onwards.\n\n### Stage 2: Loading the ASGS data into the database schema\nThis stage assumes you have the 02-ASGS-UNZIP directory from stage 1. With this just run,\n\n    make\n\n#### Coordinate Systems\nWhen you load geographic data into PostgreSQL using the PostGIS extension you must define the coordinate system of that data. The coordinate system you should use depends on what you are most likely to use the data for.\n\n- default -\u003e unprojected lat longs in the GDA94 datum\n- web mapping -\u003e [EPSG:3857](http://wiki.openstreetmap.org/wiki/EPSG:3857)\n- analysis and calculations based on area of the regions -\u003e GDA94 / Australian Albers coordinate system ([EPSG:3577](http://spatialreference.org/ref/epsg/3577/))\n\nTo switch the coordinate system see the configuration section at the top of the `Makefile`.\n\n### Stage 3: Materialised Pyramids of Generalised Geometries (Optional)\nYou can optionally produce materialised pyramid tables of the generalised geometries,\n\n    make generalization_pyramid\n\n## PG Dump\n### Producing a new db dump\nOnce everything has been loaded into PostgreSQL using these scripts you can\ncreate a PostgreSQL dump file using,\n\n    pg_dump --format plain --schema \"asgs_2011\" --no-owner | xz \u003e asgs_2011.sql.xz\n    pg_dump --format plain --schema \"asgs_2015\" --no-owner | xz \u003e asgs_2015.sql.xz\n    pg_dump --format plain --schema \"asgs_2016\" --no-owner | xz \u003e asgs_2016.sql.xz\n\n### Loading an existing db dump\nI host a copy of these files at http://tianjara.net/data/asgs2pgsql/. After\nsetting your PG\\* environment variables and creating a database (with the\nPostGIS extensions loaded) as described [above](#setting-up-the-database-environment),\nyou can load the database dumps using,\n\n    xzcat asgs_2011.sql.xz | psql -f -\n    xzcat asgs_2015.sql.xz | psql -f -\n    xzcat asgs_2016.sql.xz | psql -f -\n\n## Known Issues\n* You may run into the issue [described here](http://postgis.org/documentation/manual-2.0/ch07.html#id575278).\n\n## How to use the resulting database\nSome examples of scripts you can run to pull intelligence from the resulting\nasgs2pgsql schema are included in the examples directory.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewharvey%2Fasgs2pgsql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandrewharvey%2Fasgs2pgsql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandrewharvey%2Fasgs2pgsql/lists"}