{"id":48392771,"url":"https://github.com/entur/sobek","last_synced_at":"2026-04-06T00:31:40.565Z","repository":{"id":324682466,"uuid":"1094866417","full_name":"entur/sobek","owner":"entur","description":"Module also known as the backend for Vehicle Register (\"nasjonalt materiellregister - nmr\")","archived":false,"fork":false,"pushed_at":"2026-03-27T11:23:56.000Z","size":1097,"stargazers_count":1,"open_issues_count":22,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-27T19:58:11.718Z","etag":null,"topics":["ror"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"eupl-1.2","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/entur.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-12T09:25:19.000Z","updated_at":"2026-03-19T09:52:30.000Z","dependencies_parsed_at":"2026-02-26T09:13:23.549Z","dependency_job_id":null,"html_url":"https://github.com/entur/sobek","commit_stats":null,"previous_names":["entur/sobek"],"tags_count":78,"template":false,"template_full_name":null,"purl":"pkg:github/entur/sobek","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fsobek","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fsobek/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fsobek/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fsobek/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/entur","download_url":"https://codeload.github.com/entur/sobek/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/entur%2Fsobek/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31455368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T21:22:52.476Z","status":"ssl_error","status_checked_at":"2026-04-05T21:22:51.943Z","response_time":75,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ror"],"created_at":"2026-04-06T00:31:39.974Z","updated_at":"2026-04-06T00:31:40.556Z","avatar_url":"https://github.com/entur.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sobek ![Build](https://github.com/entur/sobek/actions/workflows/entur-push.yml/badge.svg)\nModule also known as the backend for Vehicle Register (\"nasjonalt materiellregister - nmr\")\n\nIt is used nationally in Norway, and other places.\nSobek is created with technologies like Spring Boot, Hibernate, Postgis, Jersey and Jackson. It is based on the same technology stack as Tiamat, see https://github.com/entur/tiamat\n\n## Core functionality\n### NeTEx imports\n* Supports different pre steps and merging options for vehicle data (vehicle, vehicle type, deck plan, vehicle model), handling bad data quality.\n* Assigns unique IDs to vehicles (if desired).\n* Validates incoming data against the XML schema.\n\n### NeTEx exports\nSupports exporting vehicles and other entities to the http://netex-cen.eu/ format.\nThere are many options for exports:\n* Asynchronous exports to google cloud storage. Asynchronous exports handles large amount of data, even if exporting thousands of vehicles.\n* Synchronous exports directly returned\n* Several export parameters and filtering (ex: query or administrative polygons filtering)\n* Exports can be validated against the NeTEx schema, ensuring quality.\n\n### GraphQL API\nSobek provides a rich GraphQL API for vehicles, vehicle types, deck plans and so on, support the same parameters as the NeTEx export API.\nIt also supports mutations. So you can update or create entities.\nThere are also graphql processes (named functions) which allows functionality like merging vehicle types and deck plans\n\n### A ReactJS Frontend\nA frontend for Sobek is available. It's name is Hathor.\nSee https://github.com/entur/hathor\n\n### Mapping of IDs\nAfter import vehicles and assigning new IDs to vehicles, sobek keeps olds IDs in a mapping table.\nThe mapping table between old and new IDs is available through the GraphQL API and a REST endpoint.\n\n### Versioning\nVehicles and other entities are versioned. This means that you have full version history of data and what person that made those changes.\nSobek also includes a diff tool. This is used to compare and show the difference between two versions of a vehicle (or other entity).\n\n\n## Build\n\n```shell\nmvn clean install\n```\n\nYou need the directory `/deployments/data` with rights for the user who\nperforms the build.\n\n## Integration tests\nSobek uses testcontainers to run integration tests against a real database.  To run Testcontainers-based tests, you need a Docker-API compatible container runtime\nfor more detail see https://www.testcontainers.org/supported_docker_environment/\n\n(default profiles are set in application.properties)\n\n## Running the service\n\nThere are several options for running the service depending on what you need.\n\n - [Run locally for development](#run-locally-for-development) is for people intending to maintain, modify and improve \n   sobek's source code\n - [Run sobek with Docker compose](#run-sobek-with-docker-compose) if you just need to get the service running\n - [Run with external properties file and PostgreSQL](#run-with-external-properties-file-and-postgresql) for low \n   level debugging\n\n\u003e **Note!** Each of these configurations use unique port numbers and such, be sure to read the provided documentation \n\u003e and configuration files for more details.\n\n## Run locally for development\n\nLocal development is a combination of using Docker Compose based configuration for starting up the supporting \nservices and running Spring Boot with at least `local` profile enabled.\n\nWhen running,\n\n - sobek will be available at `http://localhost:37999`\n - PostGIS will be available at `localhost:37433`\n\n### 1. Start Local Environment through Docker Compose\n\nSobek has [docker-compose.yml](./docker-compose.yml) which contains all necessary dependent services for running sobek in\nvarious configurations. It is assumed this environment is always running when the service is being run locally\n(see below).\n\n\u003e **Note!** This uses the compose version included with modern versions of Docker, not the separately installable\n\u003e `docker-compose` command.\n\nAll Docker Compose commands run in relation to the `docker-compose.yml` file located in the same directory in which the\ncommand is executed.\n\n```shell\n# run with defaults - use ^C to shutdown containers\ndocker compose up\n# run with additional profiles, e.g. with LocalStack based AWS simulator\ndocker compose --profile aws up\n# run in background\ndocker compose up -d # or --detach\n# shutdown containers\ndocker compose down\n# shutdown containers included in specific profile\ndocker compose --profile aws down\n```\n\n#### Supported Docker Compose profiles\n\nDocker Compose has its own profiles which start up additional supporting services to e.g. make specific feature \ndevelopment easier. You may include any number of additional profiles when working with Docker Compose by listing \nthem in the commands with the `--profile {profile name}` argument. Multiple profiles are activated by providing the \nsame attribute multiple times, for example starting Compose environment with profiles a and b would be\n```shell\ndocker compose --profile a --profile b up\n```\n\nThe provided profiles for Sobek development are\n\n\n| profile | description                                                                                       |\n|:--------|---------------------------------------------------------------------------------------------------|\n| `aws`   | Starts up [LocalStack](https://www.localstack.cloud/) meant for developing AWS specific features. |\n\n\nSee [Docker Compose reference](https://docs.docker.com/compose/reference/) for more details.\n\nSee [Supported Docker Compose Profiles](#supported-docker-compose-profiles) for more information on provided profiles.\n\n### 2. Run the Service\n\n#### Available Spring Boot Profiles\n\n\u003e **Note!** You must choose at least one of the options from each category below!\n\n\u003e **Note!** `local` profile must always be included!\n\n##### Storage\n\n| profile                | description                                                                                                                                                     |\n|:-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `gcs-blobstore`        | GCP GCS implementation of sobek's blob storage                                                                                                                 |\n| `local-blobstore`      | Use local directory as backing storage location.                                                                                                                |\n| `rutebanken-blobstore` | Use [`rutebanken-helpers/storage`][rutebanken-storage] based implementation for storage. Must be combined with one of the supported extra profiles (see below). |\n\n[rutebanken-storage]: https://github.com/entur/rutebanken-helpers/tree/master/storage\n\n###### Supported `rutebanken-blobstore` extra profiles\n\nIf this profile is chosen, an additional implementation must be chosen to activate the underlying actual implementation.\nSupported extra profiles are\n\n| extra profile          | description                              |\n|:-----------------------|------------------------------------------|\n| `local-disk-blobstore` | Similar to `local-blobstore`.            |\n| `in-memory-blobstore`  | Entirely in-memory based implementation. |\n\n\n**Example: Activating `in-memory-blobstore` for local development**\n```properties\nspring.profiles.active=local,rutebanken-blobstore,in-memory-blobstore,local-changelog\n```\n\nSee the [`RutebankenBlobStoreServiceConfiguration`](sobek-app/src/main/java/org/rutebanken/sobek/config/RutebankenBlobStoreConfiguration.java)\nclass for configuration keys and additional information.\n\n##### Changelog\n\n| profile           | description                                                        |\n|:------------------|--------------------------------------------------------------------|\n| `local-changelog` | Simple local implementation which logs the sent events to `stdout` |\n| `activemq`        | JMS based ActiveMQ implementation.                                 |\n| `google-pubsub`   | GCP PubSub implementation for publishing sobek entity changes.    |\n\n#### Supported Docker Compose Profiles\n\nSobek's [`docker-compose.yml`](./docker-compose.yml) comes with built-in profiles for various use cases. The profiles \nare mostly optional, default profile contains all mandatory configuration while the named profiles add features on \ntop of that. You can always activate zero or more profiles at the same time, e.g.\n\n```shell\ndocker compose --profile first --profile second up\n# or\nCOMPOSE_PROFILES=first,second docker compose up\n```\n\n### Default profile (no activation key)\n\nStarts up PostGIS server with settings matching the ones in [`application-local.properties`](./src/main/resources/application-local.properties).\n\n\n#### Run It!\n\n**IntelliJ**: Right-click on `SobekApplication.java` and choose Run (or Cmd+Shift+F10). Open Run -\u003e Edit \nconfigurations, choose the correct configuration (Spring Boot -\u003e App), and add a comma separated list of desired \nprofiles (e.g. `local,local-blobstore,activemq`) to Active profiles. Save the configuration.\n\n**Command line**:\n```bash\nmvn clean install -DskipTests \u0026\u0026 mvn -pl sobek-app spring-boot:run\n```\nNote: This is a multi-module project. The `install` step is needed to build dependency modules\n(`sobek-model`, `sobek-service`, etc.) before running the app module. After the initial build,\nyou only need to re-run `install` when you change code outside `sobek-app`.\n\n## Run Sobek with Docker compose\nTo run Sobek with Docker compose, you need to have a docker-compose.yml file. In docker-compose folder you will find a compose.yml file.:\n\n```shell\ndocker compose up\n```\n\nThis will start Sobek with PostgreSQL. and you can access Sobek on http://localhost:1888 and the database on http://localhost:5433 \nand graphiql on http://localhost:8780/services/vehicles/graphql , At start up sobek copy empty schema to the database. Spring properties are set in application.properties.\nSecurity is disabled in this setup.\n\n## Run with external properties file and PostgreSQL\nTo run with PostgreSQL you need an external `application.properties`. Below is an example of `application.properties`:\n\n```properties\nspring.jpa.database=POSTGRESQL\nspring.datasource.platform=postgres\nspring.jpa.properties.hibernate.hbm2ddl.import_files_sql_extractor=org.hibernate.tool.hbm2ddl.MultipleLinesSqlCommandExtractor\nspring.jpa.hibernate.hbm2ddl.import_files_sql_extractor=org.hibernate.tool.hbm2ddl.MultipleLinesSqlCommandExtractor\nspring.jpa.hibernate.ddl-auto=none\n\nspring.http.gzip.enabled=true\n\n#spring.jpa.properties.hibernate.format_sql=true\n\nspring.jpa.properties.hibernate.order_updates=true\nspring.jpa.properties.hibernate.batch_versioned_data=true\n\nspring.flyway.enabled=true\nspring.flyway.table=schema_version\n\nserver.compression.mime-types=application/json,application/xml,text/html,text/xml,text/plain\n\nspring.jpa.hibernate.id.new_generator_mappings=true\nspring.jpa.hibernate.use-new-id-generator-mappings=true\n\nnetex.import.enabled.types=MERGE,INITIAL,ID_MATCH,MATCH\n\nmanagement.endpoints.web.exposure.include=info,env,metrics\nmanagement.endpoints.prometheus.enabled=true\nmanagement.metrics.endpoint.export.prometheus.enabled=true\n\nspring.jpa.properties.hibernate.jdbc.batch_size=20\nspring.jpa.properties.hibernate.default_batch_fetch_size=16\nspring.jpa.properties.hibernate.generate_statistics=false\n\nchangelog.publish.enabled=false\n\njettyMaxThreads=10\njettyMinThreads=1\n\nspring.datasource.hikari.maximumPoolSize=40\nspring.datasource.hikari.leakDetectionThreshold=30000\n\nsobek.locals.language.default=eng\n\ndebug=true\n\n# Disable feature detection by this undocumented parameter. Check the org.hibernate.engine.jdbc.internal.JdbcServiceImpl.configure method for more details.\nspring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults = false\n\n# Because detection is disabled you have to set correct dialect by hand.\nspring.jpa.database-platform=org.hibernate.dialect.PostgreSQL9Dialect\n\nspring.jpa.properties.hibernate.dialect=org.hibernate.spatial.dialect.postgis.PostgisDialect\n\nspring.database.driverClassName=org.postgresql.Driver\nspring.datasource.url=jdbc:postgresql://localhost:5436/sobek\nspring.datasource.username=sobek\nspring.datasource.password=sobek\n\n#OAuth2 Resource Server\nspring.security.oauth2.resourceserver.jwt.issuer-uri=https:http://localhost:8082/realms/entur\nsobek.oauth2.resourceserver.auth0.entur.internal.jwt.audience=hathor\n\nspring.cloud.gcp.pubsub.enabled=false\n\naspect.enabled=true\n\n\nserver.port=1888\n\nblobstore.gcs.blob.path=exports\nblobstore.gcs.bucket.name=sobek-test\nblobstore.gcs.project.id=carbon-1287\n\nsecurity.basic.enabled=false\nmanagement.security.enabled=false\nauthorization.enabled = true\nrutebanken.kubernetes.enabled=false\n\nasync.export.path=/tmp\n\npublicationDeliveryUnmarshaller.validateAgainstSchema=false\npublicationDeliveryStreamingOutput.validateAgainstSchema=false\nnetex.validPrefix=NSR\nnetex.profile.version=2.0:NO-NeTEx:2.0\nblobstore.local.folder=/tmp/local-gcs-storage/sobek/export\nspring.profiles.active=local-blobstore,activemq\n```\n\nTo start Sobek with this configuration, specify **spring.config.location**:\n\n`java -Dspring.config.location=/path/to/sobek.properties -Denv=dev -jar sobek.jar`\n\n## Database\n\n### HikariCP\nSobek is using HikariCP. Most properties should be be possible to be specified in in application.properties, like `spring.datasource.initializationFailFast=false`. More information here. https://github.com/brettwooldridge/HikariCP/wiki/Configuration\nSee also http://stackoverflow.com/a/26514779\n\n## ID Generation\n### Background\nFor objects with codespace other than NMR, Sobek expects that the ID is unique and respects IDs generated elsewhere.\nFor objects with codespace NMR, Sobek generates IDs itself.\nFor objects without ID, Sobek generates IDs itself and puts them in the NMR codespace.\n\n## Keycloak/Auth0\nBoth Sobek and Hathor are set up to be used with Keycloak or Auth0.\nA detailed guide on how to setup Keycloak can be found [here](./Keycloak_Setup_Guide.md).\n\n## Validation for incoming and outgoing NeTEx publication delivery\n\nIt is possible to configure if sobek should validate incoming and outgoing NeTEx xml when unmarshalling or marshalling publication deliveries.\nDefault values are true. Can be deactivated with setting properties to false.\n```properties\npublicationDeliveryStreamingOutput.validateAgainstSchema=false\npublicationDeliveryUnmarshaller.validateAgainstSchema=true\n```\n\n## Synchronous NeTEx export with query params\nIt is possible to export vehicles and related data directly to NeTEx format. This is the endpoint:\nhttps://api.dev.entur.io/vehicles/v1/netex\n\n### Limit size of results\n```http request\nGET https://api.dev.entur.io/vehicles/v1/netex?size=1000\n```\n\n### Page\n```http request\nGET https://api.dev.entur.io/vehicles/v1/netex?page=1\n```\n\n### All Versions\n```allVersions```. Acceptable values are true or false. If set to true, all versions of matching vehicles will be returned.\nIf set to false, the highest version by number will be returned for matching vehicles. This parameter is not enabled when using the version valitity parmaeter.\n\n## Async NeTEx export from Sobek\nAsynchronous export uploads exported data to google cloud storage. When initiated, you will get a job ID back.\nWhen the job is finished, you can download the exported data.\n\n*Most of the parameters from synchronous export works with asynchronous export as well!*\n\n### Start async export:\n```\ncurl https://api.dev.entur.io/vehicles/v1/netex/export/initiate\n```\nPro tip: Pipe the output from curl to xmllint to format the output:\n```\ncurl https://api.dev.entur.io/vehicles/v1/netex/export/initiate | xmllint --format -\n```\n\n### Check job status:\n```\ncurl https://api.dev.entur.io/vehicles/v1/netex/export\n```\n\n### When job is done. Download it:\n```\ncurl https://api.dev.entur.io/vehicles/v1/netex/export/130116/content | zcat | xmllint --format - \u003e export.xml\n```\n\n## Truncate data in sobek database\nClean existing data in postgresql (streamline if frequently used):\n```\nTRUNCATE vehicle CASCADE;\nTRUNCATE vehicle_type CASCADE;\nTRUNCATE deck_plan CASCADE;\n```\n\n## Import data into Sobek\n\nIf you are running this from `spring:run`, then you need to make sure that you have enough memory available for the java process (in case of large data sets).\nAnother issue is thread stack size, which might need to be increased when coping with really large NeTEx imports.\nExample:\n```\nexport MAVEN_OPTS='-Xms256m -Xmx1712m -Xss256m -XX:NewSize=64m -XX:MaxNewSize=128m -Dfile.encoding=UTF-8'\n```\n\n### Import NeTEx file without *NMR* IDs\nThis NeTEx file should not contain NSR ID. (The NSR prefix is configurable in the class ValidPrefixList)\n* Sobek will match existing vehicles, vehicle types, vehicle models and deck plans based on their original ID.\n\nSobek will return the modified NeTEx structure with it's own NSR IDs. Original IDs will be present in key value list on each object.\n\n```shell\ncurl -XPOST -H\"Content-Type: application/xml\" -d@my-nice-netex-file.xml http://localhost:1997/services/vehicles/netex\n```\n\n### Importing with importType=INITIAL\n\nWhen importing with _importType=INITIAL_, a parallel stream will be created, spawning the original process. During import, user authorizations is checked, thus accessing SecurityContextHolder.\nBy default, SecurityContextHolder use DEFAULT\\_LOCAL\\_STRATEGY. When using INITIAL importType, you should tell Spring to use MODE\\_INHERITABLETHREADLOCAL for SecurityContextHolder, allowing Spring to duplicate Security Context in spawned threads.\nThis can be done setting env variable :\n```shell\n-Dspring.security.strategy=MODE_INHERITABLETHREADLOCAL\n```\n\nIf not, the application may complain about user not being authenticated if Spring tries to check authorization in a spawned process\n\n## GraphQL\nGraphQL endpoint is available on\n```\nhttps://api.dev.entur.io/vehicles/v1/graphql\n```\n\nTip: GraphiQL UI available on https://api.dev.entur.io/graphql-explorer/vehicles using *GraphiQL*:\nhttps://github.com/graphql/graphiql\n(Use e.g. `Modify Headers` for Chrome to add bearer-token for mutations)\n\n## Flyway\nTo create the database for sobek, download and use the flyway command line tool:\nhttps://flywaydb.org/documentation/commandline/\n\n### Migrations\nMigrations are executed when sobek is started.\n\n### Schema changes\nCreate a new file according to the flyway documentation in the folder `resources/db/migrations`.\nCommit the migration together with code changes that requires this schema change. Follow the naming convention.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fentur%2Fsobek","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fentur%2Fsobek","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fentur%2Fsobek/lists"}