{"id":19300935,"url":"https://github.com/opendrr/opendrr-api","last_synced_at":"2025-04-22T10:32:19.154Z","repository":{"id":36991968,"uuid":"240357559","full_name":"OpenDRR/opendrr-api","owner":"OpenDRR","description":"REST API for OpenDRR data / API REST pour les données OpenDRR","archived":false,"fork":false,"pushed_at":"2024-06-09T23:25:00.000Z","size":10402,"stargazers_count":4,"open_issues_count":75,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-01T22:23:25.000Z","etag":null,"topics":["government-of-canada","natural-resources-canada"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDRR.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-02-13T20:47:00.000Z","updated_at":"2021-12-20T19:46:52.000Z","dependencies_parsed_at":"2024-04-02T05:22:46.840Z","dependency_job_id":"c6e2d92a-25e8-4662-aca6-01f7c95d73bb","html_url":"https://github.com/OpenDRR/opendrr-api","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDRR%2Fopendrr-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDRR%2Fopendrr-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDRR%2Fopendrr-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDRR%2Fopendrr-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDRR","download_url":"https://codeload.github.com/OpenDRR/opendrr-api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250221374,"owners_count":21394699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["government-of-canada","natural-resources-canada"],"created_at":"2024-11-09T23:16:11.462Z","updated_at":"2025-04-22T10:32:18.705Z","avatar_url":"https://github.com/OpenDRR.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# opendrr-api\n\nREST API for OpenDRR data\n\n[![GitHub Super-Linter](https://github.com/OpenDRR/opendrr-api/workflows/Lint%20Code%20Base/badge.svg)](https://github.com/marketplace/actions/super-linter)\n![GitHub](https://img.shields.io/github/license/OpenDRR/opendrr-api)\n\u003cimg src=\"https://github.com/OpenDRR/documentation/blob/master/models/opendrr-stack.png\" width=\"600\"\u003e\n\nopendrr-api is a repository used in conjunction with the [model-factory](https://github.com/opendrr/model-factory/) repository.\nIt contains Python, Bash, and Dockerfile scripts to successfully implement a REST API for OpenDRR data which includes a PostGIS database, Elasticsearch, Kibana, and pygeoapi.\nThe model-factory repository contains the necessary scripts to transform the OpenDRR source data into risk profile indicators that go into the PostGIS database.\n\n- **postgis/**\n  - Scripts to setup the postGIS database.\n- **pygeoapi/**\n  - Scripts to setup pygeoapi.\n- **python/**\n  - Scripts to process opendrr source data, model-factory scripts, and load elasticsearch indexes.\n- docker-compose-run.yml, docker-compose.yml, opendrr-config_template.yml\n  - docker compose and opendrr config files\n- requirements.txt\n  - list of modules and versions required to be installed.  `$ pip install -r requirements.txt`\n\nRefer to the [releases section](https://github.com/OpenDRR/opendrr-api/releases) for latest version changes.\n\n## How to build your own stack - Setup in your local environment\n\n### 1. Prerequisites\n\n- [Docker engine](https://docs.docker.com/get-docker/) installed and running\n- Download or clone this repository to your local development environment\n\n### 2. Edit the Docker environment settings\n\nMake a copy of the `sample.env` file and rename it to `.env`. Make changes if required otherwise leave the default settings.\n\nThe settings below can be found in the .env file and can be adjusted to **'true'** or **'false'** depending on your preference.\nFor example, if you want to load the PSRA data into their own PostGIS database, Elasticsearch, and Kibana, you can set processPSRA and loadPsraModels to 'true' and have all other options set to 'false'.\nSpecifying the features that are only required can save you time.\n\nProcessing the Earthquake Scenarios (DSRA) / Probabilistic Earthquake Risk (PSRA) source data:\n\n    processDSRA=true\n    processPSRA=true\n\nLoading the indexes into Elasticsearch and Kibana:\n\n    loadDsraScenario=true\n    loadPsraModels=true\n    loadHazardThreat=false\n    loadPhysicalExposure=true\n    loadRiskDynamics=true\n    loadSocialFabric=true\n\n### 3. Edit the Python container configuration\n\nMake a copy of `python/sample_config.ini` and rename it `config.ini`. Open this file in an editor, add the required [github_token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and set the remaining parameters as follows:\n\n    [auth]\n    # GitHub Token for Private Repo Accesss\n    github_token = ghp_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n\n    [rds]\n    # PostGIS Connection Details\n    postgres_host = db-opendrr\n    postgres_port = 5432\n    postgres_un = postgres\n    postgres_pw = password\n    postgres_db = opendrr\n    postgres_address = db-opendrr:5432/opendrr\n\n    [es]\n    # Elasticsearch Connection Details\n    es_un = elastic\n    es_pw = changeme\n    es_endpoint = elasticsearch-opendrr:9200\n    kibana_endpoint = localhost:5601\n    # version of indices to be configured (i.e. v1.4.1)\n    index_version = v1.4.4\n\n### 4. Run docker-compose\n\n    docker-compose up --build\n\n\u003e NOTE: you will see errors thrown by the opendrr-api_pygeoapi-opendrr_1 container as the stack builds. These can be ignored.\n\nOnce the stack is built the user will need to verify that everything is working.\n\n\u003e NOTE: you can stop the stack whenever you like with `Ctrl-C` or `docker-compose stop`. See below on how you can bring the stack back up without re-building.\n\n### 5. Verify that everything is working\n\nCheck Elasticsearch to ensure that the indexes were created:\n\n\u003chttp://localhost:9200/_cat/indices?v\u0026pretty\u003e\n\nYou should see something similar to:\n\n    health status index ...\n    green  open   afm7p2_lrdmf_scenario_shakemap_intensity_building\n    green  open   .apm-custom-link\n    green  open   afm7p2_lrdmf_damage_state_building\n    green  open   .kibana_task_manager_1\n    green  open   afm7p2_lrdmf_social_disruption_building\n    green  open   .apm-agent-configuration\n    green  open   afm7p2_lrdmf_recovery_time_building\n    green  open   afm7p2_lrdmf_scenario_shakemap_intensity_building\n    green  open   afm7p2_lrdmf_casualties_building\n    green  open   .kibana_1\n\nYou can explore the indexes in Elasticsearch using Kibana\n\n\u003chttp://localhost:5601\u003e\n\nCheck pygeoapi to make sure the collections can be accessed\n\n\u003chttp://localhost:5001/collections\u003e\n\nFeature collections can be accessed as follows or by clicking on the links provided on the collections page\n\n\u003chttp://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building/items?f=json\u0026limit=1\u003e\n\nYou should see something similar to:\n\n    {\n        \"type\": \"FeatureCollection\",\n        \"features\": [\n            {\n            \"type\": \"Feature\",\n            \"geometry\": {\n                \"type\": \"Point\",\n                \"coordinates\": [\n                -117.58484079,\n                49.58943143\n                ]\n            },\n            \"properties\": {\n                \"AssetID\": \"59002092-RES3A-W1-PC\",\n                \"Sauid\": \"59002092\",\n                \"sL_Asset_b0\": 27.602575,\n                \"sL_Bldg_b0\": 27.602575,\n                \"sLr_Bldg_b0\": 1,\n                \"sLr2_BCR_b0\": 0.000819,\n                \"SLr2_RoI\": 0.001359,\n                \"sL_Str_b0\": 27.602575,\n                \"sLsd_Str_b0\": 23.4638,\n                \"sL_NStr_b0\": 0,\n                \"sLsd_NStr_b0\": 0,\n                \"sL_Cont_b0\": 0,\n                \"sLsd_Cont_b0\": 0,\n                \"sL_Asset_r2\": 0.311704,\n                \"sL_Bldg_r2\": 0.311704,\n                \"sLr_Bldg_r2\": 1,\n                \"sL_Str_r2\": 0.311704,\n                \"sLsd_Str_r2\": 0.264966,\n                \"sL_NStr_r2\": 0,\n                \"sLsd_NStr_r2\": 0,\n                \"sL_Cont_r2\": 0,\n                \"sLsd_Cont_r2\": 0,\n                \"geom_point\": \"0101000020E6100000AD9A10086E655DC0D18A357D72CB4840\"\n            },\n            \"id\": \"59002092\"\n            }\n        ],\n        \"numberMatched\": 173630,\n        \"numberReturned\": 1,\n        \"links\": [\n            {\n            \"type\": \"application/geo+json\",\n            \"rel\": \"self\",\n            \"title\": \"This document as GeoJSON\",\n            \"href\": \"http://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building/items?f=json\u0026amp;limit=1\"\n            },\n            {\n            \"rel\": \"alternate\",\n            \"type\": \"application/ld+json\",\n            \"title\": \"This document as RDF (JSON-LD)\",\n            \"href\": \"http://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building/items?f=jsonld\u0026amp;limit=1\"\n            },\n            {\n            \"type\": \"text/html\",\n            \"rel\": \"alternate\",\n            \"title\": \"This document as HTML\",\n            \"href\": \"http://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building/items?f=html\u0026amp;limit=1\"\n            },\n            {\n            \"type\": \"application/geo+json\",\n            \"rel\": \"next\",\n            \"title\": \"items (next)\",\n            \"href\": \"http://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building/items?startindex=1\u0026amp;limit=1\"\n            },\n            {\n            \"type\": \"application/json\",\n            \"title\": \"Economic loss buildings\",\n            \"rel\": \"collection\",\n            \"href\": \"http://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building\"\n            }\n        ],\n        \"timeStamp\": \"2020-08-18T22:46:10.513010Z\"\n        }\n\n## Interacting with the endpoints\n\n### Querying pygeoapi\n\nRefer to the pygeoapi documentation for general guidance:\n\n\u003c!-- textlint-disable --\u003e\n\u003chttp://localhost:5001/openapi?f=html\u003e\n\u003c!-- textlint-enable --\u003e`\n\n\u003e NOTE: querying is currently limited to spatial extent and exact value queries. For more complex querying use Elasticsearch (see below).\n\n#### To filter on a specfic attribute\n\n\u003chttp://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building/items?sH_Mag=7.2\u003e\n\n#### To filter using a bounding box\n\n\u003chttp://localhost:5001/collections/afm7p2_lrdmf_scenario_shakemap_intensity_building/items?bbox=-119,48.8,-118.9,49.8\u0026f=json\u003e\n\n### Querying Elasticsearch\n\n#### Range query\n\n\u003chttp://localhost:9200/afm7p2_lrdmf_scenario_shakemap_intensity_building/_search?q=properties.sH_PGA:[0.047580+TO+0.047584]\u003e\n\nOR using curl:\n\n    curl -XGET \"http://localhost:9200/afm7p2_lrdmf_scenario_shakemap_intensity_building/_search\" -H 'Content-Type:\n    application/json' -d'\n    {\n        \"query\": {\n            \"range\": {\n                \"properties.sH_PGA\": {\n                    \"gte\": 0.047580,\n                    \"lte\": 0.047584\n                }\n            }\n        }\n    }'\n\n#### Specific value\n\n\u003chttp://localhost:9200/afm7p2_lrdmf_scenario_shakemap_intensity_building/_search?q=properties.sH_PGA:0.047584\u003e\n\nOR using curl:\n\n    curl -XGET \"http://localhost:9200/afm7p2_lrdmf_scenario_shakemap_intensity_building/_search\" -H 'Content-Type:\n    application/json' -d'\n    {\n        \"query\": {\n            \"match\": {\n                \"properties.sH_PGA\" : 0.047584\n            }\n        }\n    }'\n\n#### Bounding box query\n\n    curl -XGET \"http://localhost:9200/afm7p2_lrdmf_scenario_shakemap_intensity_building/_search\" -H 'Content-Type:\n    application/json' -d'\n    {\n        \"query\": {\n            \"bool\": {\n                \"filter\": [\n                    {\n                        \"geo_shape\": {\n                            \"geometry\": {\n                                \"shape\": {\n                                    \"type\": \"envelope\",\n                                    \"coordinates\": [ [ -118.7, 50 ], [ -118.4, 49.9 ] ]\n                                },\n                                \"relation\": \"intersects\"\n                            }\n                        }\n                    }\n                ]\n            }\n        }\n    }'\n\n#### Nearest query\n\n    curl -XGET \"http://localhost:9200/nhsl_hazard_threat_all_indicators_s/_search\" -H 'Content-Type:\n    application/json' -d'\n    {\n      \"query\": {\n        \"geo_shape\": {\n          \"geometry\": {\n            \"shape\": {\n              \"type\": \"circle\",\n              \"radius\": \"20km\",\n              \"coordinates\": [ -118, 49 ]\n            }\n          }\n        }\n      }\n    }'\n\n## Interacting with the spatial database\n\nThe spatial database is implemented using PostGIS. You can connect to PostGIS using [pgAdmin](https://www.pgadmin.org/) with the connection parameters in your `.env` file. For example:\n\n    POSTGRES_USER: postgres\n    POSTGRES_PASSWORD: password\n    POSTGRES_PORT: 5432\n    DB_NAME: opendrr\n\n### Adding datasets to QGIS\n\nYou have two options:\n\n#### Connect to PostGIS\n\n1. Add a \"New Connection\" by right clicking on the \"PostGIS\" data type in the browser\n2. Enter a name for your connection (i.e. \"OpenDRR\")\n3. Add the credentials as per your `.env` file (see above)\n4. Click the \"OK\" button\n\n#### Connect to OGC OpenAPI - Features\n\n1. Add a \"New Connection\" by right clicking on the \"WFS / OGC API -Features\" data type in the browser\n2. Enter a name for your connection (i.e. \"OpenDRR\")\n3. Enter `http://localhost:5001` in the URL field\n4. Select \"OGC API - Features\" in the \"Version\" dropdown\n4. Click the \"OK\" button\n\n## Start/Stop the stack\n\nOnce the stack is built you only need to re-build when there is new data. The `docker-compose-run.yml` script is an override that you can use to run the built stack - it doesn't create the python container that pulls the latest code and data from GitHub to populate the stack.\n\nTo start the stack:\n\n    docker-compose -f docker-compose-run.yml start\n\nTo stop the stack:\n\n    docker-compose -f docker-compose-run.yml stop\n\n## Updating or rebuilding the stack\n\nTake the stack down and remove the volumes:\n\n    docker-compose down -v\n\nRebuild the stack:\n\n    docker-compose up --build\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrr%2Fopendrr-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrr%2Fopendrr-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrr%2Fopendrr-api/lists"}