{"id":16110373,"url":"https://github.com/02jandal/osm-bjk","last_synced_at":"2025-03-16T08:32:39.588Z","repository":{"id":192895372,"uuid":"687668304","full_name":"02JanDal/osm-bjk","owner":"02JanDal","description":"BästaJävlaKartan is a tool that compares OpenStreetMap data against various open data sources from Sweden, and reports and deviations it finds.","archived":false,"fork":false,"pushed_at":"2024-04-18T18:58:51.000Z","size":398,"stargazers_count":3,"open_issues_count":26,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-27T05:56:10.307Z","etag":null,"topics":["open-data","openstreetmap","sweden"],"latest_commit_sha":null,"homepage":"https://osm-bjk.jandal.se/","language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/02JanDal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-09-05T18:51:18.000Z","updated_at":"2024-08-19T08:38:51.000Z","dependencies_parsed_at":"2024-04-18T19:56:17.215Z","dependency_job_id":"f4d62cb4-1d41-4e75-8893-995fd625b225","html_url":"https://github.com/02JanDal/osm-bjk","commit_stats":null,"previous_names":["02jandal/osm-bjk"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/02JanDal%2Fosm-bjk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/02JanDal%2Fosm-bjk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/02JanDal%2Fosm-bjk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/02JanDal%2Fosm-bjk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/02JanDal","download_url":"https://codeload.github.com/02JanDal/osm-bjk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243807358,"owners_count":20350989,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["open-data","openstreetmap","sweden"],"created_at":"2024-10-09T19:37:22.483Z","updated_at":"2025-03-16T08:32:38.911Z","avatar_url":"https://github.com/02JanDal.png","language":"PLpgSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OSM BästaJävlaKartan\n\n![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/02JanDal/osm-bjk/frontend.yml)![GitHub License](https://img.shields.io/github/license/02JanDal/osm-bjk)\n\nBästaJävlaKartan (or BJK, or OSM-BJK, for short, Swedish for best fucking map) is a tool that compares OpenStreetMap data against various open data sources from Sweden, and reports and deviations it finds.\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://wiki.openstreetmap.org/wiki/B%C3%A4staJ%C3%A4vlaKartan\"\u003eDocumentation\u003c/a\u003e - \u003ca href=\"https://osm-bjk.jandal.se/\"\u003eLive site\u003c/a\u003e\n\u003c/p\u003e\n\n## Screenshots\n\n![App Screenshot](https://wiki.openstreetmap.org/w/images/8/89/BastaJavlaKartan_first_page.png)\n\n## Architecture\n\n```mermaid\nflowchart TD\n    Frontend --\u003e|HTTP| nginx-ingress\n    subgraph Kubernetes\n        nginx-ingress --\u003e|HTTP| PostgREST\n        PostgREST --\u003e|SQL| PostgreSQL\n        nginx-ingress --\u003e|HTTP| pg_tileserv\n        pg_tileserv --\u003e|SQL| PostgreSQL\n        Airflow --\u003e|SQL| PostgreSQL\n        pgAdmin --\u003e|SQL| PostgreSQL\n    end\n\n    style Airflow stroke-dasharray: 5 5\n    style pgAdmin stroke-dasharray: 5 5\n```\n\nBJK consist of four main components:\n\n- A database (PostgreSQL with PostGIS)\n- An Apache Airflow instance with several DAGs for fetching data and calculating deviations\n- Two database-to-API-proxies (one for REST and one for vector tiles)\n- A web frontend\n\nThe database contains one schema with OSM data, using a relatively \"raw\" table structure (comparable to that of the OSM main database but without history) which is synced using replication in one of the Airflow DAGs, currently once every 10 minutes. That DAG also calculates PostGIS geometries which are also stored in the database in the same tables.\n\nA second schema in the database contains data from various upstream sources, which are also synced using Airflow DAGs.\n\nA third schema contains the deviations and other metadata that is made available over the API.\n\nA relatively large amount of the business logic is implemented in SQL. For example, deviations are calculated purely using SQL right now, the DAGs to recalculate them just call into SQL.\n\n## Installation \u0026 Contributing\n\nSetting up the full BJK from scratch is a significant undertaking, and the initial loading of data can take several days depending on your hardware specs. However, it is still possible to contribute to some parts of it with just a limited setup, especially the frontend.\n\n### Frontend (changing how the deviations are presented to the user)\n\nThis one is easy, and you're even provided with a [devcontainer](https://containers.dev/) for convenience. However, the manual steps are as follows:\n\n1. Prerequisites: Node.js with NPM\n2. Clone this repo\n3. Go into the [`frontend`](https://github.com/02JanDal/osm-bjk/tree/main/frontend) folder\n4. Run `npm install`\n5. Run `npm run dev`\n6. Visit [`http://localhost:5137/`](http://localhost:5137/) in your browser\n\n### Database (changing how deviations are calculated, adding deviation calculation for a new dataset)\n\nA little harder, depending on what you want to do. Get started by:\n\n1. Prerequisites: Docker, Python\n2. Clone this repo\n3. Go into the [`database`](https://github.com/02JanDal/osm-bjk/tree/main/database) folder\n4. Run `pip install -r requirements.txt`\n5. Run `python db.py watch`\n\nYou can now edit any of the files in the [`database/migrations`](https://github.com/02JanDal/osm-bjk/tree/main/database/migrations) or [`database/tests`](https://github.com/02JanDal/osm-bjk/tree/main/database/tests) folder, and they will automatically be applied and tested in an ephemeral Docker container.\n\nFor just adjusting small parts this might be enough, however, if you want to add deviation calculation for an entirely new dataset or a similar change you'll need the data. If the data is already being fetched (see [Datasets](https://osm-bjk.jandal.se/datasets)) you can ask in an issue for a database extract to work with, otherwise, you'll first have to add the fetching code (or ask for it to be added in an issue), see below.\n\n### DAGs (changing how data is fetched, adding a new dataset)\n\nThis is the most involved, and this section is likely to be incomplete (please file an issue if you find something that's incorrectly described here!).\nYou essentially have two options; you can either install a Kubernetes environment (I recommend using `kind` or `microk8s`) or run Airflow \"locally\".\n\nOnce you have Airflow running, you can start editing the DAGs in [`dags`](https://github.com/02JanDal/osm-bjk/tree/main/dags). See the existing dags for how this should be done.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F02jandal%2Fosm-bjk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F02jandal%2Fosm-bjk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F02jandal%2Fosm-bjk/lists"}