{"id":13583436,"url":"https://github.com/unep-grid/mapx","last_synced_at":"2025-04-09T05:08:45.037Z","repository":{"id":39386819,"uuid":"91441949","full_name":"unep-grid/mapx","owner":"unep-grid","description":"MapX ","archived":false,"fork":false,"pushed_at":"2024-10-29T17:26:22.000Z","size":113144,"stargazers_count":86,"open_issues_count":112,"forks_count":23,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-10-29T20:52:14.532Z","etag":null,"topics":["dashboard","docker","geoserver","mapbox-gl-js","maps","nodejs","postgres","r","shiny-apps","spatial-data","storymaps","vector","webpack","wms"],"latest_commit_sha":null,"homepage":"https://app.mapx.org","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/unep-grid.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-16T09:43:40.000Z","updated_at":"2024-10-21T14:03:47.000Z","dependencies_parsed_at":"2024-03-18T10:27:31.572Z","dependency_job_id":"4e21b460-4f26-4172-a182-a9264882e5de","html_url":"https://github.com/unep-grid/mapx","commit_stats":{"total_commits":3032,"total_committers":13,"mean_commits":"233.23076923076923","dds":0.04782321899736153,"last_synced_commit":"528a130e698f0d5f3f298ef8af0b89b418520cd3"},"previous_names":["unep-grid/map-x-mgl"],"tags_count":536,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unep-grid%2Fmapx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unep-grid%2Fmapx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unep-grid%2Fmapx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unep-grid%2Fmapx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/unep-grid","download_url":"https://codeload.github.com/unep-grid/mapx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247980837,"owners_count":21027808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dashboard","docker","geoserver","mapbox-gl-js","maps","nodejs","postgres","r","shiny-apps","spatial-data","storymaps","vector","webpack","wms"],"created_at":"2024-08-01T15:03:28.678Z","updated_at":"2025-04-09T05:08:45.012Z","avatar_url":"https://github.com/unep-grid.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# [MapX](https://app.mapx.org/)\n\n![MapX CI/CD](https://github.com/unep-grid/mapx/actions/workflows/build.yml/badge.svg?branch=staging)\n\n[MapX](https://app.mapx.org/) is an online platform for managing geospatial data on natural resources, developed by [UNEP/GRID-Geneva](https://unepgrid.ch/en) - a data centre resulting from the partnership between\n[UN Environment Programme](https://www.unep.org/), the Swiss [Federal Office for the Environment](https://www.bafu.admin.ch/) and the [University of Geneva](https://unige.ch/).\n\nField applications of MapX are varied and include chemical management, disaster risk reduction, biodiversity planning, land-use planning, extractive industry, renewable energy and environmental security. \n\nMapX targets a wide community of users that are primarily UN Environment Programme and partners, the Secretariats of Multilateral Environmental Agreements (MEAs) and other UN agencies mandated to collect and use geospatial data in environmental decision-making. Civil society groups, non-governmental organizations, academia and citizens complement this set of users. \n\nMapX was designed in 2014 and since then continuously improved with wide international stakeholder consultations. \n\nMapX is fully integrated into the World Environment Situation Room, which is the UNEP data and knowledge platform.\n\n![mapx preview](app/src/png/mapx-preview.png \"MapX\")\n\n\n## Development \n\nDevelopment servers are launched from within Docker containers, to match as closely as possible the environment found in production. Some commands, tools and config are still currently needed on your local computer.\n\n### Requirement\n\n__Mendatory__ \n\n- `docker` v20.10+\n\n__Optional__  \n\n- `node` v16.0+\n- `g++`\n- `npm`\n- `yq`\n- `git`\n\n\n### Hosts\n\nSome browsers require to modify your hosts file to link custom MapX local \"subdomains\". It could be as simple as adding those lines to `/etc/hosts/` and restarting your browser, if needed: \n\n```sh\n127.0.0.1 app.mapx.localhost \n127.0.0.1 api.mapx.localhost\n127.0.0.1 search.mapx.localhost\n127.0.0.1 wsecho.mapx.localhost\n127.0.0.1 probe.mapx.localhost\n127.0.0.1 apidev.mapx.localhost\n127.0.0.1 dev.mapx.localhost\n127.0.0.1 geoserver.mapx.localhost\n```\n\n### Docker\n\nThe included `docker-compose.yml` allows to setup a development environment.\n\nTrigger the following script which init some required directories and copy the default environment variable to `./mapx.dev.env` (if missing):\n\n```sh\n./mapx.dev.init.sh\n```\n\nFinally, launch the mapx stack:\n\n```sh\n# Pull the latest builds\ndocker compose pull\n\n# Launch postgres first : in case of first launch, some tables and roles must be created\ndocker compose up pg\n\n# Launch other services\ndocker compose up\n```\n\nThe application should be available at \u003chttp://app.mapx.localhost:8880/\u003e (curl -H Host:app.mapx.localhost \u003chttp://127.0.0.1:8880/).\u003e\n\nAn admin user is available as `admin@localhost` which can be used to login; get the password by browsing the web mail at \u003chttp://mail.mapx.localhost:8880/.\u003e\n\n\n### Build Process\n\nThe build process has been automated using GitHub Actions and follows three main workflows:\n\n1. **Push-based Builds** (Staging)\n   - Triggered by commits to the `staging` branch\n   - Runs full CI process (lint, test, build)\n   - Builds and pushes Docker images with version tags\n   - Uses GitHub Actions cache for faster builds\n   - Images tagged with both version and 'latest'\n\n2. **Manual Builds** (workflow_dispatch)\n   - Can be triggered manually through GitHub Actions\n   - Identical process to push-based builds\n   - Useful for testing or rebuilding without code changes\n\n3. **Scheduled Builds** (Weekly)\n   - Runs automatically every Monday at 7 AM UTC\n   - Performs full build process without pushing images\n   - Validates build integrity with fresh dependencies\n   - Creates GitHub issues for any failures\n   - Ensures long-term reliability\n\nTo create properly formatted commit messages, use:\n```sh\nnpm run commit\n```\n\nThis ensures commits follow the [Conventional Commits](https://www.conventionalcommits.org/) specification, which helps in automatic versioning and changelog generation.\n\nThe build process uses GitHub Actions cache (type=gha) for efficient builds while ensuring weekly scheduled builds can catch any potential issues with dependencies or base images.\n\n### Version Management\n\nMapX uses [release-it](https://github.com/release-it/release-it) to manage versions across the entire project. This automatically:\n- Runs tests to verify the build is stable\n- Updates version numbers in all package.json files and version.txt\n- Creates proper git commits and tags\n- (Optional) Pushes changes to GitHub\n\n#### Creating a New Version\n\nTo create a new version, use one of the following npm scripts:\n\n```bash\n# Regular release (e.g., 1.13.15)\nnpm run release\n\n# Alpha release (e.g., 1.13.15-alpha.0)\nnpm run release:alpha\n\n# Beta release (e.g., 1.13.15-beta.0)\nnpm run release:beta\n```\n\n**IMPORTANT:** Do not manually edit version numbers or run update scripts directly. The version update scripts are automatically invoked by the release-it process.\n\n#### After Version Update\n\nAfter creating a new version:\n\n1. Push the changes and tags to GitHub:\n   ```bash\n   git push origin staging --tags\n   ```\n\n2. This will trigger the CI/CD pipeline which will:\n   - Build the Docker images\n   - Tag them with the new version number\n   - Push them to Docker Hub (for production deployment)\n\n### Docker Build Process\n\nMapX uses different Docker build approaches for development and production:\n\n#### Local Development\n\nLocal builds use standard `docker build` for simplicity and efficiency. The image has to be built locally, as buildx failed to build multi-platform api/app images reliably on x86. \n\n```bash\n# Build both app and api images with local tag\nnpm run build:local\n\n# Or build them individually\nnpm run build:app_local\nnpm run build:api_local\n\n# You can also build directly from the service directories\ncd app\n./build.sh\n\ncd api\n./build.sh\n\n# Or specify a specific version tag if needed\ncd app\n./build.sh 1.13.14\n\n# Build geoserver image with multi-platform support\ncd geoserver\n./build.sh \u003cversion\u003e [platforms]  # Default platforms: linux/amd64,linux/arm64\n\n# Build search image with multi-platform support\ncd meili\n./build.sh \u003cversion\u003e [platforms]  # Default platforms: linux/amd64,linux/arm64\n```\n\nThe docker-compose.yml file is configured to use the `:local` tag for app and api services, which allows for efficient local development on any platform (ARM or x86) while maintaining compatibility with our CI/CD pipeline.\n\nThis approach solves platform mismatch issues when developing on ARM machines (like M1/M2 Macs) while the remote images are built for x86 only.\n\n#### Production Builds\n\nProduction builds are handled by GitHub Actions using `docker buildx`:\n- Automated builds triggered by:\n  - Push to staging branch\n  - Manual workflow dispatch\n  - Weekly scheduled builds (Mondays 7 AM UTC)\n- GitHub Actions cache for efficient builds\n- Automated testing and validation\n\n##### Manual Builds\n\nTo trigger a manual build:\n1. Go to GitHub Actions\n2. Select the appropriate workflow:\n   - \"MapX CI/CD\" for app/api\n   - \"Build Support Services\" for geoserver/search\n3. Click \"Run workflow\"\n4. Select options and trigger build\n\n\n#### Known issues\n\nPostgis: `OperationalError: could not access file \"$libdir/postgis-X.X` _Solution:_ run `docker compose exec pg update-postgis.sh`\n\n\n### Development session for the `app` service\n\nInstall all modules listed as dependencies in `package.json` for the `app` service, the `sdk` and the websocket handler `ws_handler` :\n\n```sh\nnpm run init\n```\nStart a development session for the `app` service:\n\n- Automatically build all client side dependencies, included dictionaries and translation ( which needs some config, see below )\n\n```sh\n$ cd ./app\n$ npm run dev\n```\n\n- Launch the server from within the running `app_dev` container. In another terminal window, launch the dev server :\n\n```sh\ndocker compose exec app_dev R\n\u003e source('run.R') \n\n# OR, as a single line for a non-interactive session:\ndocker compose exec app_dev Rscript --vanilla run.R\n```\n\nThen, an instance of mapx should be available at \u003chttp://dev.mapx.localhost:8880/\u003e for which the source code from `./app/` is mounted as `/app/` in the container.\n\n__Note for auto-translation__:\nAutomatic translation requires a valid Google cloud config file, which path should be refered in the host – not inside the docker container – as an environment variable named `GOOGLE_APPLICATION_CREDENTIALS`, accessible from your local node. You can test this with :\n\n```sh\n$ node -e 'console.log(process.env.GOOGLE_APPLICATION_CREDENTIALS)'\n# Should return, as an example :\n# \u003e /home/\u003cyour name\u003e/.google_cloud_\u003cyour service\u003e.json\n```\n\n### Development session for the `api` service\n\nSetup the environmental variables for the `api` service in `mapx.dev.env` as follows:\n\n```sh\nAPI_HOST=api\nAPI_PORT=3030\nAPI_PORT_DEV=3333\nAPI_PORT_PUBLIC=8880\nAPI_HOST_PUBLIC=api.mapx.localhost\nAPI_HOST_PUBLIC_DEV=apidev.mapx.localhost\n```\n\nStart the `Express.js` development server:\n\n```sh\n$ docker compose up -d\n$ docker compose exec api_dev node inspect index.js port=3333\ndebug\u003e c\n```\n\nThe instance now should use the api service at \u003chttp://apidev.mapx.localhost:8880/\u003e for which the source from `./api/` is mounted as `/api/` in the container.\n\nIf you want to use the prod version of the `api_dev` service, setup the environmental variables in `mapx.dev.env`as follows:\n\n```sh\nAPI_HOST=api\nAPI_PORT=3030\nAPI_PORT_DEV=3030\nAPI_PORT_PUBLIC=8880\nAPI_HOST_PUBLIC=api.mapx.localhost\nAPI_HOST_PUBLIC_DEV=api.mapx.localhost\n```\n\n### Development session for the `routines` service\n\n```sh\ndocker compose exec routines node inspect routines.js\ndebug\u003e c\n```\n\n\n\n### `app` end-to-end tests \n\nMapx use a custom end-to-end testing tool, which features the mapx's `sdk`. The testing coverage is partial, but should cover the largest part of all MapX features, while also tesing the `sdk`, as a all tests are written using common `sdk` async methods.  \n\n\n```sh\ncd app \u0026\u0026 npm run test:sdk\n```\n\n#### `api` tests\n\nRun tests within the development container (currently disabled):\n\n```sh\ndocker compose exec api sh\nnpm run test\n```\n\n# Countries boundaries layer\n\nA sample dataset of countries boundaries (  polygons ) is included in this code repo, and will add a table named `mx_countries` in the database. The main purpose of this layer is croping dataset during exportation. \n\n## Citation\n\nAdministrative boundaries generalized by UNEP/GRID-Geneva (2019) based on the Global Administrative Unit Layers (GAUL) dataset (G2015_2014), implemented by FAO within the CountrySTAT and Agricultural Market Information System (AMIS) projects (2015).\n\n## Generalization \u003e= mapx 1.8.26 \n\nStarting with version 1.8.26, a lightweight version of `mx_countries` is included.\n\nIt was generated with [mapshaper](https://mapshaper.org/) using the following parameters:\n\n- import:\n  - detect line intersections = true\n  - snap vertices = true\n- simplification:\n  - prevent shape removal = true\n  - method: Visvalingam / weighted area\n  - percentage of removable points to retain: 3%\n\nOnce the simplification was done, the data was repaired in `mapshaper` and then in `QGIS 3.18` using the `Fix geometries` tool. All geometries are valid according to GEOS rules.\n\n\n## Generalization \u003c mapx 1.8.26 \n\nThe generalization was made using Feature Manipulation Engine (FME) with the following settings:\n\n- Algorithm: Douglas (generalize)\n- Generalization Tolerance: 0.02\n- Preserve Shared Boundaries: Yes\n- Shared Boundaries Tolerance: None\n- Preserve Path Segment: No\n\nGeometries obtained from FME have been repaired in PostGIS using ST_MakeValid() function.\n\n\n## Restrictions\n\nYou are free to modify and/or adapt any Data provided for your own use, reproduction as well as unlimited use within your organization. The Data is licensed and distributed by UNEP/GRID-Geneva. Redistribution to a Third party or reseller is formerly prohibited at any stage whatsoever by UNEP/GRID-Geneva.\n\n## Disclaimer\n\nDue to the generalization process, the administrative boundaries of the countries have been modified. Therefore, this dataset can only be used for unofficial cartographic purposes for global mapping using a scale not higher than 1:25 million. It should not be used in any way as a reference for national boundaries. Territorial information from this dataset do not imply the expression of any opinion whatsoever on the part of the UNEP/GRID-Geneva concerning the legal status of any country, territory, city or area, or of its authorities, or concerning the delimitation of its frontiers or boundaries. The Data is being delivered to you \"AS IS\" and UNEP/GRID-Geneva makes no warranty as to its use or performance.\n\nUNEP/GRID-Geneva cannot be held responsible for a misuse of this file and its consequences.\n\n# PostgreSQL passwords update\n\nProcedure to follow if PostgreSQL passwords need to be updated for security reason (or any other reasons).\n\n1. Launch MapX stack with Docker Compose:\n\n    ```sh\n    docker compose up\n    ```\n\n2. Once your stack is up, update PostgreSQL passwords in the environment file:\n\n    - `POSTGRES_PASSWORD`\n    - `POSTGRES_USER_WRITE_PASSWORD`\n    - `POSTGRES_USER_READ_PASSWORD`\n    - `POSTGRES_USER_CUSTOM_PASSWORD`\n\n3. Connect to PostgreSQL using psql:\n\n    ```sh\n    docker compose exec pg psql -U {POSTGRES_USER}\n    ```\n\n4. Queries to run in psql to update the passwords. Be careful to respect the order in which the queries are run.\n\n    ```sql\n    ALTER ROLE {POSTGRES_USER_READ} WITH PASSWORD '{POSTGRES_USER_READ_PASSWORD}';\n    ALTER ROLE {POSTGRES_USER_WRITE} WITH PASSWORD '{POSTGRES_USER_WRITE_PASSWORD}';\n    ALTER ROLE {POSTGRES_USER_CUSTOM} WITH PASSWORD '{POSTGRES_USER_CUSTOM_PASSWORD}';\n    ALTER ROLE {POSTGRES_USER} WITH PASSWORD '{POSTGRES_PASSWORD}';\n    \\q\n    ```\n\n5. Force Compose to stop and recreate all containers to avoid any problems related to passwords update:\n\n    ```sh\n    docker compose up -d --force-recreate\n    ```\n\n\n\n\u0026copy; 2014-present unepgrid.ch\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funep-grid%2Fmapx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funep-grid%2Fmapx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funep-grid%2Fmapx/lists"}