{"id":43352210,"url":"https://github.com/dimagi/connect-gis","last_synced_at":"2026-02-02T02:32:44.069Z","repository":{"id":298077712,"uuid":"997462611","full_name":"dimagi/connect-gis","owner":"dimagi","description":"A simple tool for visualizing cases and generating reports","archived":false,"fork":false,"pushed_at":"2025-08-14T09:34:19.000Z","size":14972,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-20T17:50:51.258Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dimagi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-06T15:09:21.000Z","updated_at":"2025-08-14T09:34:22.000Z","dependencies_parsed_at":"2025-06-09T09:51:01.389Z","dependency_job_id":null,"html_url":"https://github.com/dimagi/connect-gis","commit_stats":null,"previous_names":["dimagi/connect-gis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dimagi/connect-gis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fconnect-gis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fconnect-gis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fconnect-gis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fconnect-gis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dimagi","download_url":"https://codeload.github.com/dimagi/connect-gis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dimagi%2Fconnect-gis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29001654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-02T01:32:03.847Z","status":"online","status_checked_at":"2026-02-02T02:00:07.448Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-02T02:32:43.373Z","updated_at":"2026-02-02T02:32:44.064Z","avatar_url":"https://github.com/dimagi.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Building Density and Clustering API\n\nThis is a Flask-based web application that provides APIs for fetching building data, performing clustering (K-means and grid-based), and generating reports based on geospatial data. The application integrates with Google Earth Engine (GEE) and a PostgreSQL database with PostGIS extension to process building data within specified polygons or around pin locations. It supports clustering buildings using balanced K-means or grid-based clustering and generates CSV reports summarizing ward-level visit data.\n\n## Features\n- **Building Data Retrieval**: Fetches building data within a polygon from either Google Earth Engine or a PostgreSQL database, with optional filters for minimum area and confidence.\n- **Clustering**:\n  - **K-means Clustering**: Performs balanced K-means clustering based on a specified number of clusters or buildings per cluster.\n  - **Grid-based Clustering**: Generates a grid over a polygon, assigns buildings to grid cells, and clusters grids to balance building counts.\n- **Reporting**: Generates CSV reports summarizing ward-level visit data, with options to include building-to-visit distance metrics.\n- **Geospatial Support**: Uses PostGIS for spatial queries and GeoPandas for handling geospatial data.\n- **CORS Support**: Allows cross-origin requests for frontend integration.\n\n## Prerequisites\n- **Python**: Version 3.8 or higher.\n- **PostgreSQL**: With PostGIS extension enabled for spatial queries.\n- **Google Earth Engine Account**: For accessing building data via GEE.\n- **Dependencies**: Listed in `requirements.txt`.\n\n## Installation\n\n1. **Clone the Repository**\n   ```bash\n   git clone https://github.com/Thushar12E45/dimagi-map-project.git\n   cd dimagi-map-project\n   ```\n\n2. **Set Up a Virtual Environment**\n   ```bash\n   python -m venv venv\n   source venv/bin/activate  # On Windows: venv\\Scripts\\activate\n   ```\n\n3. **Install Dependencies**\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. **Set Up Environment Variables**\n   Create a `.env` file in the project root with the following variables:\n   ```\n   GEE_CREDS=\u003cyour-google-earth-engine-credentials-json\u003e\n   GEE_PROJECT_NAME=\u003cyour-google-earth-engine-project-name\u003e\n   DB_USER=\u003cyour-postgres-username\u003e\n   DB_PASSWORD=\u003cyour-postgres-password\u003e\n   DB_HOST=\u003cyour-postgres-host\u003e\n   DB_PORT=\u003cyour-postgres-port\u003e\n   DB_NAME=\u003cyour-postgres-database-name\u003e\n   HOST_URL=\u003cyour-host-url\u003e  # Optional, defaults to https://connectgis.dimagi.com\n   ```\n   - `GEE_CREDS`: JSON string of Google Earth Engine service account credentials.\n   - Database credentials for PostgreSQL connection.\n\n\n5. **Database Setup**\n   - Ensure the PostgreSQL database is running and has the PostGIS extension enabled.\n   - The `buildings` table should contain building data with spatial geometry.\n\n\n6. **Run the Application**\n   ```bash\n   python app.py\n   ```\n   The app runs on `http://0.0.0.0:5000` in debug mode by default.\n\n# Google Earth Engine (GEE) Setup Guide\n\n## Prerequisites\n- Google Cloud account\n- Earth Engine access (sign up at [earthengine.google.com](https://earthengine.google.com/))\n\n## Setup Steps\n\n### 1. Create a Google Cloud Project\n1. Go to [Google Cloud Console](https://console.cloud.google.com/)\n2. Click \"Create Project\"\n3. Enter project name and details\n4. Click \"Create\"\n\n### 2. Register Project for Earth Engine\n1. After project creation, register it for:\n   - **Commercial use** (if applicable), or\n   - **Non-commercial use** (for research/academic purposes)\n2. Wait for approval (typically 1-2 business days)\n\n### 3. Enable Earth Engine API\n1. Search / Navigate to: **APIs \u0026 Services** → **Library**\n2. Search for \"Google Earth Engine API\"\n3. Click \"Enable\"\n\n### 4. Local Development Setup\n```bash\n# Install Earth Engine Python API\npip install earthengine-api\n\n# Authenticate (will open browser)\nearthengine authenticate\n```\n\n### 4. Configuration\nAdd your GEE project name to the .env file\n```dotenv\nGEE_PROJECT_NAME=\u003cyour-google-earth-engine-project-name\u003e\n```\n\n## Important Notes\n- **Local Development**: Authentication via **earthengine authenticate** is sufficient for local use (no credentials needed in **.env**)\n- **Production / Other Environments**: You must add the appropriate GEE credentials to your .env file:\n```dotenv\nGEE_CREDS=\u003cyour-service-account-credentials-json\u003e\n```\n\n# Building Data Insertion Guide\n\n## Overview\nThe building data insertion system downloads building footprint data from [Overture Maps](https://overturemaps.org/) and loads it into a PostgreSQL database for use by the clustering application. This is a one-time setup process required before using the main application.\n\n## What it Does\n- Downloads building footprint data for an entire country from Overture Maps\n- Processes the data in manageable tiles to handle large datasets\n- Calculates building metrics (area, centroids, confidence scores)\n- Stores data in PostGIS-enabled PostgreSQL database\n- Creates spatial indexes for optimal query performance\n- Supports parallel processing for faster data loading\n\n\n## Configuration for Different Countries\n\n### 1. Update Bounding Box Coordinates\nEdit `SQL_SCRIPTS/db_insertion_buildings.py` to set the target country's bounding box:\n\n```python\n# Example for Kenya\nleft, bottom = 33.9098, -4.6796  # SW corner\nright, top = 41.9058, 5.5059     # NE corner\n```\n\n### 2. Update Configuration Variables\n```python\n# Change base filename to match your country\nbase_filename = \"kenya_buildings\"\n```\n\n## Running the Data Insertion\n\n### 1. Single Instance (Slower)\n```bash\ncd SQL_SCRIPTS\npython db_insertion_buildings.py --instance-id 0 --total-instances 1\n```\n\n### 2. Parallel Processing (Recommended)\nRun multiple instances simultaneously for faster processing:\n\n```bash\n# Example with 4 parallel instances\npython db_insertion_buildings.py --instance-id 0 --total-instances 4 \u0026\npython db_insertion_buildings.py --instance-id 1 --total-instances 4 \u0026\npython db_insertion_buildings.py --instance-id 2 --total-instances 4 \u0026\npython db_insertion_buildings.py --instance-id 3 --total-instances 4\n```\n\n## Recovery from Failures\nThe script supports resuming from interruptions:\n1. Failed tiles are logged in `failed_tiles.csv`\n2. Restart the script with same parameters - it will skip completed tiles\n3. For persistent failures, check the error messages in the failed tiles log\n\n\n## API Endpoints\n\n### 1. Home (`/`)\n- **Method**: GET\n- **Description**: Renders the `index.html` template with the configured `HOST_URL`.\n- **Response**: HTML page.\n\n### 2. Get Building Density (`/get_building_density`)\n- **Method**: POST\n- **Description**: Fetches building data within a polygon or around a pin and performs clustering.\n- **Request Body**:\n  ```json\n  {\n    \"clusteringType\": \"kMeans|balancedKMeans|bottomUp\",\n    \"noOfClusters\": \u003cint\u003e,  // Number of clusters (default: 3)\n    \"noOfBuildings\": \u003cint\u003e,  // Target buildings per cluster (default: 250)\n    \"buildingsAreaInMeters\": \u003cfloat\u003e,  // Minimum building area (default: 0)\n    \"buildingsConfidence\": \u003cint\u003e,  // Minimum confidence (0-100, default: 0)\n    \"thresholdVal\": \u003cint\u003e,  // Tolerance percentage (default: 10)\n    \"fetchClusters\": \u003cboolean\u003e,  // Whether to perform clustering (default: false)\n    \"dbType\": \"GEE|DB\",  // Data source (Google Earth Engine or Database)\n    \"polygon\": [[lng, lat], ...],  // Polygon coordinates (for kMeans/balancedKMeans)\n    \"pin\": [lng, lat]  // Pin coordinates (for bottomUp)\n  }\n  ```\n- **Response**: JSON with building count, GeoJSON features, and optional cluster data.\n- **Example Response**:\n  ```json\n  {\n    \"building_count\": 100,\n    \"buildings\": {\"type\": \"FeatureCollection\", \"features\": [...]},\n    \"clusters\": [{\"coordinates\": [lng, lat], \"cluster\": \u003cint\u003e, \"numOfBuildings\": \u003cint\u003e}, ...]\n  }\n  ```\n\n### 3. Get Building Density V2 (`/get_building_density_v2`)\n- **Method**: POST\n- **Description**: Fetches buildings from the database, generates a grid, assigns buildings to grid cells, and performs grid-based clustering.\n- **Request Body**:\n  ```json\n  {\n    \"polygon\": [[lng, lat], ...],  // Polygon coordinates\n    \"noOfClusters\": \u003cint\u003e,  // Number of clusters (default: 3)\n    \"thresholdVal\": \u003cint\u003e,  // Tolerance percentage (default: 10)\n    \"gridLength\": \u003cint\u003e,  // Grid size in meters (default: 50)\n    \"buildingsAreaInMeters\": \u003cfloat\u003e,  // Minimum building area (default: 0)\n    \"buildingsConfidence\": \u003cint\u003e  // Minimum confidence (0-100, default: 0)\n  }\n  ```\n- **Response**: JSON with building count, GeoJSON features, grid GeoJSON, and cluster data.\n- **Example Response**:\n  ```json\n  {\n    \"building_count\": 100,\n    \"buildings\": {\"type\": \"FeatureCollection\", \"features\": [...]},\n    \"grids\": {\"type\": \"FeatureCollection\", \"features\": [...]},\n    \"clusters\": [{\"coordinates\": [lng, lat], \"cluster\": \u003cint\u003e, \"grid_index\": \u003cint\u003e}, ...]\n  }\n  ```\n\n### 4. Generate Report (`/generate_report`)\n- **Method**: POST\n- **Description**: Generates a CSV report summarizing ward-level visit data.\n- **Request Body**:\n  ```json\n  {\n    \"data\": [{\"latitude\": \u003cfloat\u003e, \"longitude\": \u003cfloat\u003e, \"flw_id\": \u003cint\u003e}, ...],\n    \"fetchVisitToBuildingsVal\": \u003cboolean\u003e  // Include building-to-visit distance metrics (default: true)\n  }\n  ```\n- **Response**: CSV file (`Ward_summary_report.csv`) with ward visit summary data.\n- **Example CSV Headers** (with `fetchVisitToBuildingsVal=true`):\n  ```csv\n  state_name,lga_name,ward_name,population,total.visits,total.buildings,num.phc.serve.ward,median.visit.to.phc,max.visit.to.phc,median.building.to.phc,max.buildings.to.phc,unique.flws,coverage,percent.building.100.plus.to.visit,percent.building.200.plus.to.visit,percent.building.500.plus.to.visit,percent.building.10000.plus.to.visit\n  ```\n\n## Project Structure\n```\n├── app.py              # Main Flask application\n├── .env                # Environment variables (not tracked)\n├── requirements.txt    # Python dependencies\n├── templates/\n│   └── index.html      # Frontend template\n```\n\n## Notes\n- **Google Earth Engine**: Ensure valid GEE credentials are provided in `.env` for the `/get_building_density` endpoint with `dbType=GEE`.\n- **Performance**: For large polygons, reduce the number of buildings or grid cells to avoid GEE’s 5000-element limit.\n- **Security**: Sanitize inputs to prevent SQL injection (handled via parameterized queries in the code).\n- **CORS**: Configured to allow all origins (`*`). Adjust in production for security.\n\n## Troubleshooting\n- **GEE Initialization Error**: Verify `GEE_CREDS` in `.env` and ensure the credentials file is correctly formatted.\n- **Database Connection Error**: Check PostgreSQL credentials and ensure the database is accessible.\n- **No Buildings Found**: Ensure the polygon or pin coordinates are valid and contain buildings in the database or GEE dataset.\n- **Clustering Issues**: Adjust `thresholdVal` or reduce `noOfClusters`/`noOfBuildings` if clustering fails due to insufficient data.\n\n\n## Deployment\nThe [Connect GIS app](https://connectgis.dimagi.com/) is hosted on AWS by running a dockerized version of the app on the EC2 instance. The database is hosted as an AWS RDS service. \n\n### Deploying changes\nIn order to deploy new change you need access to the production server on AWS (the details and credentials is in 1Password - search for \"ConnectGIS\").\n\nDeploying new changes requires the following steps:\n\n```bash\n# Navigate to folder\ncd projects/dimagi-map-project/\n\n# Pull latest changes\ngit pull\n\n# Build the docker image\ndocker build -t map-clustering .\n\n# Stop the existing running container\ndocker stop map-clustering\n\n# Remove the named instance\ndocker container remove map-clustering\n\n# Run the new image with the port binding\ndocker run -d -p 5010:5000 --name map-clustering map-clustering\n```\n\n### Updating environment variables\nThe production environment file also lives in 1Password, but you'll also need to update the `.env` on the server (`~/projects/connect-gis/.env`). ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimagi%2Fconnect-gis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdimagi%2Fconnect-gis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdimagi%2Fconnect-gis/lists"}