{"id":23122696,"url":"https://github.com/folio-org/mod-remote-sync","last_synced_at":"2025-08-17T02:31:06.922Z","repository":{"id":38003029,"uuid":"364282677","full_name":"folio-org/mod-remote-sync","owner":"folio-org","description":"Pluggable data synchronisation with arbitrary sources - initially focussed on LASER resources into Subscriptions and Agreements. Funded by European Regional Development Fund (EFRE)","archived":false,"fork":false,"pushed_at":"2024-10-28T20:26:01.000Z","size":747,"stargazers_count":1,"open_issues_count":13,"forks_count":3,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-10-28T21:27:05.114Z","etag":null,"topics":["connectivity","folio-lsp"],"latest_commit_sha":null,"homepage":"","language":"Groovy","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/folio-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-04T14:25:45.000Z","updated_at":"2022-02-18T14:12:49.000Z","dependencies_parsed_at":"2023-10-04T02:25:03.131Z","dependency_job_id":"286de6a1-aedb-42c7-b34f-9f0ebc4bcd14","html_url":"https://github.com/folio-org/mod-remote-sync","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-remote-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-remote-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-remote-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/folio-org%2Fmod-remote-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/folio-org","download_url":"https://codeload.github.com/folio-org/mod-remote-sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230080758,"owners_count":18169617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["connectivity","folio-lsp"],"created_at":"2024-12-17T07:29:51.852Z","updated_at":"2024-12-17T07:29:52.663Z","avatar_url":"https://github.com/folio-org.png","language":"Groovy","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mod-remote-sync\n\n![Development 2021-2022 funded by European Regional Development Fund (EFRE)](https://github.com/folio-org/mod-remote-sync/raw/master/assets/EFRE_2015_quer_RGB.jpg)\n\nMod-remote-sync is a generic FOLIO module for system librarians which helps reconcile and keep in sync a remote set of resources with a local FOLIO system. \n\nThe module is generic in the sense that without configuration it has no utility. System librarians must develop and upload \"Agent\" modules for fetching data from remote systems and processing that data. mod-remote-sync provides services to these agents such as scheduling, error reporting, and feedback case management which makes the reconcilliation process interactive.\n\nFor example, a systems librarian wishes to keep a copy of their LASER licenses in FOLIO and to keep that subset of licenses up to date. For historical reasons, licenses may already exist in FOLIO and whenever FOLIO encounters a new license, the systems librarian wishes to have a choice to create a new FOLIO license to track the LASER license, or to map to an existing FOLIO license. mod-remote-sync provides a generic \"Resource-Mapping-Feedback\" process whereby an agent can request that the user provide such mappings and halts processing until the condition is satisfied.\n\n# IMPORTANT NOTE\n\nThis module now uses code signing by default. This means that in order to upload a configuration with executable code the code needs to be signed by an approved\ncertificate. This is to prevent arbitrary / malicious code execution. \n\nTHis is an EXPLORATORY MODULE : \n\nBackend Module to provide data synchronisation with arbitrary sources - initially focussed on LASER resources into Subscriptions and Agreements for Leipzig but with wider scope if we get it right.\n\n\nHow does mod-remote-sync work?\n\nThe \u003c\u003ccontroller\u003e\u003e class mod_remote_sync.SettingController exposes a \"worker\" endpoint that is called either\nby a timer or manually invoked to trigger an invocation of the mod-remote-sync main control loop.\n\nThe main control loop triggers the \u003c\u003cservice\u003e\u003e ExtractService via it's start() method using a \u003c\u003cpromise\u003e\u003e\n\nThe main control loop started by mod_remote_sync.ExtractService performs the following 3 tasks\n\n  1. runSourceTasks - Fetch records from remote sources using groovy scripts which implement the mod_remote_sync.source.RemoteSyncActivity interface\n  2. runExtractTasks - Each set of source records is examined for new records and new transformation tasks created for those records\n  3. runTransformationTasks - Try to complete each of the transformation tasks\n\nWithin step 3 above we may discover that we need to ask questions of the human operator. Transformation tasks in step three can register feedbackItems which\ngenerate UI tasks for an operator to complete. Having registered their decision, subsequent loops should be able to complete record import.\n\n## DEVOPS\n\n### Container Params\n\nThis is a spring boot (grails 4) application which uses springs default environment overide conventions. As such, a line in the\nservice/grails-app/conf/application.yaml file like\n\n     url: \"jdbc:postgresql://${db.host:localhost}:${db.port:5432}/${db.database:olfdev}\"\n\nCan access environment variables provided through (for example) a conainer ENV setting. In the example above, the env DB_PORT would be mapped to db.port.\nOther conventions apply - see here: https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config\n\nThe following container params can be set\n\n* REQUIRE_SIGNED_CODE - This is a special remote-sync setting that requires uploaded modules to be digitally signed \"ON\" | \"OFF\" (Default is OFF)\n\nStandard env overrides:\n\n* DB_DATABASE\n* DB_HOST\n* DB_USERNAME\n* DB_PASWORD\n* DB_MAXPOOLSIZE\n* DB_PORT\n* EVENTS_PUBLISHER_BOOTSTRAP_SERVERS\n* EVENTS_CONSUMER_BOOTSTRAP_SERVERS\n* EVENTS_PUBLISHER_ZK_CONNECT\n* EVENTS_CONSUMER_ZK_CONNECT\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffolio-org%2Fmod-remote-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffolio-org%2Fmod-remote-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffolio-org%2Fmod-remote-sync/lists"}