{"id":20710990,"url":"https://github.com/entur/asag","last_synced_at":"2026-03-05T02:01:19.350Z","repository":{"id":46642885,"uuid":"118426899","full_name":"entur/asag","owner":"entur","description":"Update mapbox tilset with netex stop place data by converting stop place netex data to geojson","archived":false,"fork":false,"pushed_at":"2026-02-23T05:08:09.000Z","size":278,"stargazers_count":2,"open_issues_count":61,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2026-02-23T13:32:24.090Z","etag":null,"topics":["carbon","geojson","java-11","mapbox-tilset","neon","netex","radon","ror","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/entur.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-01-22T08:16:22.000Z","updated_at":"2026-02-23T05:03:18.000Z","dependencies_parsed_at":"2023-01-22T13:30:34.017Z","dependency_job_id":"5e0a1ee0-e3ea-450e-9972-d9028635fcfc","html_url":"https://github.com/entur/asag","commit_stats":null,"previous_names":[],"tags_count":89,"template":false,"template_full_name":null,"purl":"pkg:github/entur/asag","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fasag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fasag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fasag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fasag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/entur","download_url":"https://codeload.github.com/entur/asag/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fasag/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30106124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T01:39:18.192Z","status":"online","status_checked_at":"2026-03-05T02:00:06.710Z","response_time":93,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["carbon","geojson","java-11","mapbox-tilset","neon","netex","radon","ror","spring-boot"],"created_at":"2024-11-17T02:13:43.162Z","updated_at":"2026-03-05T02:01:19.344Z","avatar_url":"https://github.com/entur.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# asag ![Build](https://github.com/entur/asag/actions/workflows/push.yml/badge.svg)\n\n**asag** is a Kubernetes CronJob that keeps a [Mapbox](https://mapbox.com) tileset up to date with NeTEx stop-place data exported from Entur's Tiamat stop registry.\n\nMaintained by **team-ror** (`team.rutedata@entur.org`).\n\n---\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Architecture](#architecture)\n- [Data Flow](#data-flow)\n- [Components](#components)\n- [Configuration](#configuration)\n- [Docker](#docker)\n- [Helm / Kubernetes](#helm--kubernetes)\n- [CI/CD](#cicd)\n- [Development](#development)\n- [Testing](#testing)\n- [Dependencies](#dependencies)\n\n---\n\n## Overview\n\nOnce a day (06:30 UTC) the job:\n\n1. Downloads `tiamat_export_geocoder_latest.zip` from a GCS bucket\n2. Parses the NeTEx XML inside it, filtering out expired entities\n3. Maps `StopPlace`, `Quay`, `Parking` and `TariffZone` entities to a GeoJSON `FeatureCollection`\n4. Uploads the resulting `.geojson` file to Mapbox via their Uploads API (using temporary AWS S3 credentials)\n5. Polls Mapbox until the tileset processing is complete (or times out)\n6. Posts a status summary to Slack\n\nThe application has **no HTTP server** — it is a pure batch ETL job (`WebApplicationType.NONE`).\n\n---\n\n## Architecture\n\n```\n┌─────────────────────┐\n│  Google Cloud       │\n│  Storage (GCS)      │  tiamat_export_geocoder_latest.zip\n└────────┬────────────┘\n         │ download\n         ▼\n┌─────────────────────┐\n│  BlobStoreService   │  unzip\n└────────┬────────────┘\n         │\n         ▼\n┌─────────────────────────────────────────────────────────┐\n│  DeliveryPublicationStreamToGeoJson                     │\n│                                                         │\n│  NeTEx XML ──JAXB──► filter by validity period          │\n│                       │                                 │\n│                       ├─► StopPlaceToGeoJsonFeatureMapper│\n│                       ├─► QuayToGeoJsonFeatureMapper    │\n│                       ├─► ParkingToGeoJsonFeatureMapper │\n│                       └─► TariffZoneToGeoJsonFeatureMapper│\n└────────┬────────────────────────────────────────────────┘\n         │ GeoJSON FeatureCollection\n         ▼\n┌─────────────────────┐\n│  Mapbox API         │  GET /uploads/v1/{user}/credentials\n│  (credentials)      │  → temporary AWS S3 credentials\n└────────┬────────────┘\n         │\n         ▼\n┌─────────────────────┐\n│  AWS S3             │  PUT .geojson (via temp credentials)\n└────────┬────────────┘\n         │\n         ▼\n┌─────────────────────┐\n│  Mapbox Uploads API │  POST /uploads/v1/{user}\n│  + status polling   │  GET  /uploads/v1/{user}/{id}\n│  (max 20 retries,   │  (20 s delay between retries)\n│   20 s interval)    │\n└────────┬────────────┘\n         │\n         ▼\n┌─────────────────────┐\n│  Slack Webhook      │  started / success / error / timeout\n└─────────────────────┘\n```\n\n---\n\n## Data Flow\n\n| Step | Route ID | Description |\n|------|----------|-------------|\n| 1 | `mapbox-download-latest-tiamat-export-to-folder` | Download zip from GCS |\n| 2 | `mapbox-unzip-tiamat-export` | Unzip NeTEx archive to local dir |\n| 3 | `mapbox-find-first-xml-file-recursive` | Locate the NeTEx XML file |\n| 4 | `mapbox-transform-from-tiamat` | NeTEx XML → GeoJSON |\n| 5 | `mapbox-retrieve-aws-credentials` | Fetch temporary S3 credentials from Mapbox |\n| 6 | `upload-mapbox-data-aws` | Upload GeoJSON to S3 |\n| 7 | `initiate-mapbox-upload` | POST to Mapbox Uploads API |\n| 8 | `mapbox-poll-retry-upload-status` | Poll until complete/error/timeout |\n\nAll routes are wired together in `MapBoxUpdateRouteBuilder.java` using Apache Camel DSL.\n\n---\n\n## Components\n\n### `mapbox/`\n\n| Class | Responsibility |\n|-------|---------------|\n| `MapBoxUpdateRouteBuilder` | Master Camel route orchestrator |\n| `DeliveryPublicationStreamToGeoJson` | Streaming NeTEx XML → GeoJSON transformer |\n| `AwsS3Uploader` | Upload file to AWS S3 via temporary Mapbox credentials |\n| `ValidityFilter` | Exclude NeTEx entities past their validity period |\n| `StopPlaceToGeoJsonFeatureMapper` | Map `StopPlace` → GeoJSON Feature |\n| `QuayToGeoJsonFeatureMapper` | Map `Quay` → GeoJSON Feature |\n| `ParkingToGeoJsonFeatureMapper` | Map `Parking` → GeoJSON Feature |\n| `TariffZoneToGeoJsonFeatureMapper` | Map `TariffZone` → GeoJSON Feature |\n| `ZoneToGeoJsonFeatureMapper` | Shared base mapper for zone-type entities |\n| `MapperHelper` | Reflection/enum utilities for property mapping |\n| `KeyValuesHelper` | Extract NeTEx `keyList` values |\n\n### `service/`\n\n| Class | Responsibility |\n|-------|---------------|\n| `BlobStoreService` | Read files from Google Cloud Storage |\n| `UploadStatusHubotReporter` | Post Slack notifications via webhook |\n\n### `netex/`\n\n| Class | Responsibility |\n|-------|---------------|\n| `PublicationDeliveryHelper` | JAXB utilities for parsing NeTEx XML |\n\n---\n\n## Configuration\n\nAll configuration is injected via environment variables (Kubernetes ConfigMap / Secrets). There is no `application.properties` file.\n\n### Required variables\n\n| Variable | Spring property | Description |\n|----------|----------------|-------------|\n| `BLOBSTORE_GCS_CONTAINER_NAME` | `blobstore.gcs.container.name` | GCS bucket containing the NeTEx export |\n| `BLOBSTORE_GCS_PROJECT_ID` | `blobstore.gcs.project.id` | GCP project ID |\n| `MAPBOX_ACCESS_TOKEN` | `mapbox.access.token` | Mapbox API access token (secret) |\n| `MAPBOX_USER` | `mapbox.user` | Mapbox account username |\n| `MAPBOX_TILESET_FILE_NAME` | `mapbox.tileset.file.name` | Tileset identifier / dataset name |\n| `HELPER_SLACK_ENDPOINT` | *(via entur helpers slack)* | Slack webhook URL (secret) |\n\n### Optional variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `BLOBSTORE_GCS_CREDENTIAL_PATH` | *(workload identity)* | Path to GCP service account JSON |\n| `MAPBOX_DOWNLOAD_DIRECTORY` | `files/mapbox` | Local working directory for temp files |\n| `TIAMAT_EXPORT_BLOBSTORE_SUBDIRECTORY` | `tiamat/geocoder` | Sub-path within the GCS bucket |\n| `JAVA_OPTIONS` | `-server -Xmx1500m -Dfile.encoding=UTF-8` | JVM flags |\n| `TZ` | `Europe/Oslo` | Container timezone |\n\n### Production values (from Helm)\n\n```\nBLOBSTORE_GCS_CONTAINER_NAME = ror-kakka-production\nBLOBSTORE_GCS_PROJECT_ID     = ent-kakka-prd\nMAPBOX_TILESET_FILE_NAME     = neon-1287\nMAPBOX_DOWNLOAD_DIRECTORY    = files/tmp/mapbox\n```\n\n---\n\n## Docker\n\n**Base image:** `adoptopenjdk/openjdk11:alpine-jre`\n\n```dockerfile\nFROM adoptopenjdk/openjdk11:alpine-jre\nWORKDIR /deployments\nCOPY target/asag-*-SNAPSHOT.jar asag.jar\nRUN addgroup appuser \u0026\u0026 adduser --disabled-password appuser --ingroup appuser\nRUN chown -R appuser:appuser /deployments\nUSER appuser\nCMD java $JAVA_OPTIONS -jar asag.jar\n```\n\nKey points:\n- Minimal Alpine-based JRE image\n- Non-root `appuser` for security\n- No `EXPOSE` — batch job, no inbound ports\n- JVM tuned via `$JAVA_OPTIONS` at runtime\n\n### Build \u0026 run locally\n\n```bash\n# Build the JAR\nmvn package -DskipTests\n\n# Build the image\ndocker build -t asag:local .\n\n# Run (minimal example)\ndocker run --rm \\\n  -e BLOBSTORE_GCS_CONTAINER_NAME=my-bucket \\\n  -e BLOBSTORE_GCS_PROJECT_ID=my-project \\\n  -e BLOBSTORE_GCS_CREDENTIAL_PATH=/creds/sa.json \\\n  -e MAPBOX_ACCESS_TOKEN=pk.xxx \\\n  -e MAPBOX_USER=myuser \\\n  -e MAPBOX_TILESET_FILE_NAME=my-tileset \\\n  -e HELPER_SLACK_ENDPOINT=https://hooks.slack.com/... \\\n  -v /path/to/sa.json:/creds/sa.json:ro \\\n  asag:local\n```\n\n### Copy generated GeoJSON from a running pod\n\nWhile the job is running (look for `Upload: {\"tileset\":...}` in the logs):\n\n```bash\nkubectl cp \u003cpod-name\u003e:/deployments/files/mapbox/\u003ctileset\u003e.geojson .\n```\n\n---\n\n## Helm / Kubernetes\n\nChart location: `helm/asag/`\n\n### Chart summary\n\n| Field | Value |\n|-------|-------|\n| Chart name | `asag` |\n| Chart version | `0.1.0` |\n| App version | `0.1.99.2` |\n| Dependency | `entur/common:1.21.1` |\n\nThe chart uses Entur's internal `common` Helm library which renders CronJob, ConfigMap, ExternalSecret, PodDisruptionBudget, and VerticalPodAutoscaler manifests from a shared template.\n\n### Kubernetes resource summary (production)\n\n| Resource | Kind | Notes |\n|----------|------|-------|\n| `asag` | `CronJob` | Schedule `30 06 * ? *`, `concurrencyPolicy: Forbid` |\n| `asag` | `ConfigMap` | Runtime env vars |\n| `asag-mapbox` | `ExternalSecret` | `MAPBOX_ACCESS_TOKEN`, `MAPBOX_USER` |\n| `asag-slack` | `ExternalSecret` | `HELPER_SLACK_ENDPOINT` |\n| `asag` | `PodDisruptionBudget` | `minAvailable: 0%` |\n| `asag` | `VerticalPodAutoscaler` | `updateMode: Off` (advisory) |\n\n### Resource requests / limits\n\n| | CPU | Memory |\n|-|-----|--------|\n| Request | 0.1 | 1000 Mi |\n| Limit | 1 | 3000 Mi |\n\n### Security context\n\n- `runAsNonRoot: true`\n- `runAsUser: 1000`\n- `allowPrivilegeEscalation: false`\n- `seccompProfile: RuntimeDefault`\n\n### Useful Helm commands\n\n```bash\n# Lint the chart\nhelm lint helm/asag\n\n# Template (dry-run)\nhelm template asag helm/asag -f helm/asag/values.yaml\n\n# Update chart dependencies\nhelm dependency update helm/asag\n```\n\n---\n\n## CI/CD\n\n### `push.yml` — Build, test and publish\n\n**Trigger:** push or PR to `master`\n\n| Job | Condition | What it does |\n|-----|-----------|-------------|\n| `maven-verify` | always | Checkout, Java 11 (Liberica), `mvn verify`, upload JAR artifact |\n| `docker-build` | push to master, entur org | Build Docker image from JAR artifact |\n| `docker-push` | after docker-build | Push image to registry |\n\nRequires secrets: `JFROG_USER`, `JFROG_PASS` (Entur Artifactory for internal Maven dependencies).\n\n### `codeql.yml` — Security scanning\n\n**Trigger:** push/PR to master + weekly (`0 3 * * MON`)\n\nUses `entur/gha-security/.github/workflows/code-scan.yml@v2` for Java 11 CodeQL analysis.\n\n---\n\n## Development\n\n### Prerequisites\n\n- Java 11\n- Maven 3.6+\n- Access to Entur's Artifactory (for internal Maven packages like `gcp-storage` and `netex-java-model`)\n\n### Build\n\n```bash\nmvn verify -s .github/workflows/settings.xml\n```\n\nThe `-s` flag points to a settings file that configures Entur's internal Maven repository. For local builds without Artifactory access, you may need to install internal packages manually.\n\n### Maven settings\n\nThe Maven settings file at `.github/workflows/settings.xml` is downloaded from Entur's GitHub org during CI. It configures the Artifactory repositories for:\n- `org.entur.ror.helpers:gcp-storage`\n- `org.entur.ror.helpers:slack`\n- `org.entur:netex-java-model`\n\n---\n\n## Testing\n\n**Framework:** JUnit 4 + AssertJ + WireMock (via `spring-cloud-contract-wiremock`)\n\n```bash\nmvn test\n```\n\n### Test classes\n\n| Test | Type | Description |\n|------|------|-------------|\n| `MapBoxUpdateRouteBuilderTest` | Integration | Full Camel route: success, error, and timeout scenarios |\n| `DeliveryPublicationStreamToGeoJsonTest` | Unit | NeTEx XML → GeoJSON conversion |\n| `StopPlaceToGeoJsonFeatureMapperTest` | Unit | Stop place mapping, adjacent sites, parent/child refs |\n| `QuayToGeoJsonFeatureMapperTest` | Unit | Quay public code mapping |\n| `ParkingToGeoJsonFeatureMapperTest` | Unit | Parking capacity, vehicle types, covered status |\n| `ZoneToGeoJsonFeatureMapperTest` | Unit | Point and polygon geometry from NeTEx centroid/polygon |\n\n### Test infrastructure\n\n- **`TestConfig`** — `@Profile(\"test\")` bean that replaces `BlobStoreService` with a Mockito mock (no GCS needed)\n- **WireMock** — stubs Mapbox credentials, upload initiation, status polling, and Slack webhook\n- **Test resources** — `publication-delivery.xml`, `adjacent_sites_netex.xml`, `stops.zip`\n\n### Integration test scenarios\n\n| Scenario | Expected outcome |\n|----------|-----------------|\n| Mapbox upload succeeds | Route property `finished` |\n| Mapbox returns error status | Route property `error` |\n| Mapbox keeps returning incomplete | Route property `timeout` after 20 retries |\n\n---\n\n## Dependencies\n\n### Runtime (key libraries)\n\n| Library | Version | Purpose |\n|---------|---------|---------|\n| Spring Boot | 2.1.1.RELEASE | Application framework |\n| Apache Camel | 2.22.3 | Integration routing |\n| `netex-java-model` (Entur) | 1.0.13 | NeTEx JAXB model |\n| `geojson-jackson` | 1.8.1 | GeoJSON serialisation |\n| AWS Java SDK S3 | 1.11.502 | S3 upload (AWS SDK v1) |\n| `gcp-storage` (Entur helpers) | 1.86 | GCS client |\n| `slack` (Entur helpers) | 1.86 | Slack notifications |\n| Apache HttpComponents | 4.5.2 | HTTP client |\n| Logstash Logback Encoder | 5.3 | Structured JSON logging |\n| Guava | 26.0-jre | Utilities |\n| commons-compress | 1.18 | ZIP handling |\n| JAXB API / Runtime | 2.3.0 / 2.3.0.1 | XML binding (Java 11 compat) |\n\n### Test\n\n| Library | Version |\n|---------|---------|\n| JUnit | 4.12 |\n| AssertJ | 3.3.0 |\n| spring-cloud-contract-wiremock | 1.2.2.RELEASE |\n| camel-test-spring | 2.22.3 |\n\n\u003e **Note for refactoring / dependency updates:**\n\u003e All major dependencies are significantly out of date. Key upgrade targets:\n\u003e - Spring Boot `2.1.1` → `3.x` (requires Java 17+, Jakarta EE namespace migration)\n\u003e - Apache Camel `2.22.3` → `4.x` (major API changes; route DSL largely compatible)\n\u003e - AWS SDK `1.11.502` → AWS SDK v2 (`2.x`)\n\u003e - JUnit `4.12` → JUnit 5 (Jupiter)\n\u003e - Base Docker image `adoptopenjdk/openjdk11` → `eclipse-temurin:21-jre-alpine` (AdoptOpenJDK is archived)\n\u003e - `hibernate-validator` `5.2.3.Final` → `8.x` (aligns with Spring Boot 3)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fentur%2Fasag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fentur%2Fasag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fentur%2Fasag/lists"}