{"id":28202719,"url":"https://github.com/connexta/replication","last_synced_at":"2026-03-10T11:31:27.742Z","repository":{"id":34196454,"uuid":"166127250","full_name":"connexta/replication","owner":"connexta","description":"Replication Application","archived":false,"fork":false,"pushed_at":"2025-10-24T11:24:14.000Z","size":2450,"stargazers_count":1,"open_issues_count":194,"forks_count":15,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-10-24T13:22:21.365Z","etag":null,"topics":["ion"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/connexta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-01-16T23:25:32.000Z","updated_at":"2023-07-31T19:13:36.000Z","dependencies_parsed_at":"2023-01-15T05:13:13.287Z","dependency_job_id":"1bab022c-7341-4206-9199-72faafddfa0a","html_url":"https://github.com/connexta/replication","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/connexta/replication","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/connexta%2Freplication","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/connexta%2Freplication/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/connexta%2Freplication/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/connexta%2Freplication/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/connexta","download_url":"https://codeload.github.com/connexta/replication/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/connexta%2Freplication/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30332264,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ion"],"created_at":"2025-05-17T00:17:47.657Z","updated_at":"2026-03-10T11:31:27.734Z","avatar_url":"https://github.com/connexta.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Replication [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=replication\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=replication) [![Known Vulnerabilities](https://snyk.io/test/github/connexta/replication/badge.svg)](https://snyk.io/test/github/connexta/replication) [![CircleCI](https://circleci.com/gh/connexta/replication.svg?style=svg)](https://circleci.com/gh/connexta/replication)\n\n## Overview\nReplication is the process of creating a copy of a subset of data and storing it on another DDF or ION based System. Data can be pulled from a remote DDF and saved to another DDF or ION system. Metacards produced by replication are marked with a \"Replication Origins\" attribute and a tag of \"replicated\". Replication will automatically start transferring data once an admin creates a replication configuration.\n\n\n## Known Issues, Limitations, and Assumptions\nReplication is still at an early stage in its lifecycle, so there are a few details that the user should be aware of.\n\n#### Fanout Proxies\nReplicating from a DDF system that is configured as a Fanout Proxy will result in the replication of records from sources configured in that system.\n\nReplicating to a DDF system that is configured as a Fanout Proxy will result in the replication of records only to the fanout and not its sources.\n\n#### Connected Sources\nReplicating from a DDF sytem that is configured with Connected Sources will result in the replication of records from the Connected Sources in addition to any local records.\n\n#### Derived Resources\nDerived resources, from products such as NITFs, will not be replicated.\n\n## Docker Compose Deployment\n\n#### Prerequisites\nReplication is deployed as a docker stack in a docker swarm. So, before deploying replication,\nyou need a running docker instance with an initialized swarm. Once you have a swarm running you can \nconfigure it for replication with the following steps.\n\n###### Configuration\nThe configuration that replication uses needs to be populated in docker config\n\n|Config Name | Description|\n|------------|------------|\n|replication-spring-config| The spring-boot application.yml for replication. Example below.|\n\n\nExample replication-spring-config\n```yaml\nlogging:\n  level: \n  #You can adjust the log levels of a package or class in this section \n    root: INFO\n    org.apache.cxf.interceptor.LoggingOutInterceptor: WARN\n    org.apache.cxf.interceptor.LoggingInInterceptor: WARN\n    javax.xml.soap: ERROR\nspring:\n  data:\n    solr:\n    #This is the URL the replication service will use to communicate with solr.\n    #As long as you use the docker compose file you won't need to change this.\n      host: http://replication-solr:8983/solr \n  profiles.active: Classic\nreplication:\n  #This is the number of seconds between each replication, lower it if you're going to be testing.\n  period: 300\n  #Timeouts for calls to sites\n  connectionTimeout: 30\n  receiveTimeout: 60\n  #The ID of the local site. All replications will go to/from this site. Direction will be determined \n  #by the type and kind of site being replicated with. This field needs to be set, and a site with\n  #this ID needs to be saved before any replication will take place. \n  localSite: some-unique-id-1234\n  #The remote sites to handle replication for, remove this to handle replication for all sites.\n  sites:\n  - site1\n  - site2\n  \n# Exposes metrics\nmanagement:\n  endpoint:\n    metrics:\n      enabled: true\n    prometheus:\n      enabled: true\n  endpoints:\n    web:\n      exposure:\n        include: 'prometheus,metrics,health,info'\n  metrics:\n    export:\n      prometheus:\n        enabled: true\n```\n\nTo create a docker config use the `config create` command, which uses this syntax:\n`docker config create \u003cCONFIG_NAME\u003e \u003cFILE_DIRECTORY\u003e`\n\nExample:\n`docker config create replication-spring-config replication/configs/application.yml`\n\n###### Profiles\n\nReplication can be run with one of two profiles. You can specify which profile to use in the 'spring.profiles.active'\nproperty as demonstrated in the example above. \"Classic\" will use the classic monolithic implementation.\n\"Ion\" will use the new scalable, cloud oriented implementation.\n\n###### Metrics \n\nReplication supports reporting metrics through Micrometer. Prometheus is used as the metrics collection platform. The replication-spring-config provides example configuration for exposing metrics from within the application.\n\n###### Grafana\n\nA Grafana dashboard `grafana-dashboard.json` is provided, which can be imported into Grafana.\n\n###### Secrets\nReplication requires certs and ssl configurations in order to talk with remote DDF based systems. This information is stored in docker secrets.\n\n|Secret Name | Description|\n|------------|------------|\n|replication-truststore|A truststore to use for TLS|\n|replication-keystore|A keystore for this system to use TLS|\n|replication-ssl|SSL properties for TLS including passwords for the truststore and keystore|\n\nExample replication-ssl\n```properties\njavax.net.ssl.trustStorePassword=changeit\njavax.net.ssl.trustStoreType=jks\njavax.net.ssl.keyStorePassword=changeit\njavax.net.ssl.keyStoreType=jks\njavax.net.ssl.certAlias=localhost\n```\nOnly the properties that differ from the defaults above need to be specified in replication-ssl\n\nTo add a docker secret use the `secret create` command, which uses this syntax:\n`docker secret create \u003cSECRET_NAME\u003e \u003cFILE_DIRECTORY\u003e`\n\nExample:\n`docker secret create replication-truststore replication/secrets/truststore.jks`\n\n#### Running\nRunning the stack will start up a solr service and the replication service.\n\n```\ndocker stack deploy -c docker-compose.yml repsync\n```\n\n#### Adding Replication Configuration\nReplication can be configured by using the Solr rest endpoint.\n```\ncurl -H \"Content-Type: application/json\" \\\n-d @/path/to/json/config/file.json \\\nhttp://localhost:8983/solr/\u003ctarget-core\u003e/update?commitWithin=1000\n``` \n\n#### Types and Kinds of sites\nSites can be of different types and kinds. The type and kind of the remote site (the one that's not \nthe local site) will determine whether the replication is a push, pull, harvest, or both push and pull.\n\n|Site Type | Site Kind | Replication Direction |\n|------------|------------|------------|\n|DDF | TACTICAL | BIDIRECTIONAL |\n|DDF | REGIONAL | HARVEST |\n|ION | TACTICAL | BIDIRECTIONAL |\n|ION | REGIONAL | PUSH |\n\n###### Directions\nHere's how the various replication directions are defined:\nPUSH - Send information to the remote site to be stored.\nPULL - Retrieve information from the remote site and store it locally.\nBIDIRECTIONAL - Perform both a push and a pull.\nHARVEST - Similar to a pull but harvesting will ignore updates and deletes on replicated information.  \n\n###### Adding Site Example\nCreate a json file with site descriptions like the example below and you can use the following curl \ncommand to save those sites for replication to use.\nExample sites.json\n```json\n  [\n      {\n        \"version\": 1,\n        \"id\": \"some-unique-id-1234\",\n        \"name\": \"RepSync-Node1\",\n        \"description\": \"Replication Site 1\",\n        \"url\": \"https://host1:8993/services/\",\n        \"type\": \"DDF\",\n        \"kind\": \"TACTICAL\",\n        \"polling_period\": 600000,\n        \"parallelism_factor\": 1\n       },\n      {\n        \"version\":1,\n        \"id\": \"another-unique-id-5678\",\n        \"name\": \"RepSync-Node2\",\n        \"description\": \"Replication Site 2\",\n        \"url\": \"https://host2:8993/services\",\n        \"type\": \"DDF\",\n        \"kind\": \"TACTICAL\",\n        \"polling_period\": 600000,\n        \"parallelism_factor\": 1\n      }\n   ]\n```\n```\ncurl -H \"Content-Type: application/json\" \\\n-d @sites.json \\\nhttp://localhost:8983/solr/replication_site/update?commitWithin=1000\n```\n###### Adding Replication Filters Example\nCreate a json file with filter descriptions like the example below and you can use the following curl \ncommand to save those filters for replication to use.\nExample filters.json\n```json\n   [\n      {\n        \"name\":\"pdf-harvest\",\n        \"site_id\":\"remote-site-id\",\n        \"filter\":\"\\\"media.type\\\" like 'application/pdf'\",\n        \"suspended\":false,\n        \"priority\": 0,\n        \"id\":\"unique-filter-id-98765\",\n        \"version\":1\n      }\n    ]\n```\n```\ncurl -H \"Content-Type: application/json\" \\\n-d @filters.json \\\nhttp://localhost:8983/solr/replication_filter/update?commitWithin=1000\n```\n\n#### Removing Replication Configuration\nExample removing all sites with the name 'Test'\n```\ncurl -X POST \\\n  'http://localhost:8983/solr/replication_site/update?commit=true' \\\n  -H 'Content-Type: application/xml' \\\n  -d '\u003cdelete\u003e\u003cquery\u003ename_txt:Test\u003c/query\u003e\u003c/delete\u003e'\n```\nExample removing all sites\n```\ncurl -X POST \\\n  'http://localhost:8983/solr/replication_site/update?commit=true' \\\n  -H 'Content-Type: application/xml' \\\n  -d '\u003cdelete\u003e\u003cquery\u003e*:*\u003c/query\u003e\u003c/delete\u003e'\n```\n\n#### Try it out\nYou can try replication for yourself using the steps below as a high level overview. Details on how \nto complete steps related to setting up replication can be found above, starting with \n\"Docker Compose Deployment\".\n1. Make sure docker is up and running. Start up a docker swarm if you haven't already.\n3. Create docker config\n4. Create docker secrets\n5. Deploy stack\n6. Create two sites, Both as REGIONAL DDFs, with URLs pointing to running DDF instances.\n7. The remote site (The site that isn't the local site) will be your source of data. Upload test data to the source site if it has none.\n8. Create filter. The \"site_id\" should match the ID of remote site. The filter can be changed to something like \"\\\"title\\\" like 'test'\" or \n\t\"\\\"title\\\" like '*'\" to replicate everything.\n9. execute `docker service logs -f repsync_ion-replication` to view the logs and wait for replication to occur. Once you start seeing logs check the\n\tlocal site to see the data start coming in.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconnexta%2Freplication","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconnexta%2Freplication","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconnexta%2Freplication/lists"}