{"id":19326911,"url":"https://github.com/panosoft/slate-replicator","last_synced_at":"2025-02-24T06:22:10.776Z","repository":{"id":57135042,"uuid":"57162272","full_name":"panosoft/slate-replicator","owner":"panosoft","description":"A Postgresql Event Source Table Replicator","archived":false,"fork":false,"pushed_at":"2017-09-05T17:13:44.000Z","size":71,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-23T04:27:29.003Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/panosoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-04-26T21:13:40.000Z","updated_at":"2017-03-30T07:39:23.000Z","dependencies_parsed_at":"2022-09-04T10:00:35.123Z","dependency_job_id":null,"html_url":"https://github.com/panosoft/slate-replicator","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-replicator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-replicator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-replicator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-replicator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/panosoft","download_url":"https://codeload.github.com/panosoft/slate-replicator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240427718,"owners_count":19799548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T02:15:19.861Z","updated_at":"2025-02-24T06:22:10.743Z","avatar_url":"https://github.com/panosoft.png","language":"JavaScript","readme":"# DEPRECATED - Please see [elm-slate/replicator](https://github.com/elm-slate/replicator)\n\n\n# slate-replicator\nPostgresql Event Source Table Replicator For slate\n\nThe purpose of the slate-replicator is to copy a table of immutable events from a Postgresql Event Source database to an events table in one or more Destination databases.\n\nReplication is triggered by events being added to the Event Source database events table.\n\nslate-replicator requires node version 6 or greater.\n\n# Installation\n\u003e npm install -g @panosoft/slate-replicator\n\n# Usage\n\n#### Start slate-replicator\n\n    slate-replicator [options]\n\n  Options:\n\n    -h, --help                 output usage information\n    -c, --config-filename \u003cs\u003e  configuration file name\n    --dry-run                  if specified, display run parameters and end program without starting replicator\n\n#### Stop slate-replicator\n\n    ^C to stop replicator\n\nThe replicator copies in a transactional manner so it can be stopped at any time\n\n## Configuration file\n\n### Sample configuration file\n\n```javascript\n  var config = {\n    // optional parameter.  maximum events written to the replicationDestinations events tables per database operation.  minimum value:  10000.\n    maxEventsPerWrite: 10000,\n    // optional parameter.  database connection timeout in milliseconds.  default value:  15000.\n    connectTimeout: 15000,\n    // connection parameters to immutable event source database events table\n    eventSource: {\n      host: 'localhost',\n      databaseName: 'sourceDb',\n      // optional parameter.  connection attempt will fail if missing and needed by database.\n      user: 'user1',\n      // optional parameter.  connection attempt will fail if missing and needed by database.\n      password: 'pass1'\n    },\n    // array of connection parameters to one or more replication databases events tables.  one replication destination is required.\n    replicationDestinations: [\n      {\n        host: 'localhost',\n        databaseName: 'replicationDb1',\n        // optional parameter.  connection attempt will fail if missing and needed by database.\n        user: 'user1',\n        // optional parameter.  connection attempt will fail if missing and needed by database.\n        password: 'pass1'\n      },\n      {\n          host: 'localhost',\n          databaseName: 'replicationDb2',\n          // optional parameter.  connection attempt will fail if missing and needed by database.\n          user: 'user1',\n          // optional parameter.  connection attempt will fail if missing and needed by database.\n          password: 'pass1'\n    ]\n  };\n  module.exports = config;\n  ```\n#### maxEventsPerWrite\n\u003e An optional parameter that specifies the maximum number of events that can be written to the events table in each `replicationDestination` database per write operation.  Default and minimum value is `10000`.\n\n#### connectTimeout\n\u003e An optional parameter that specifies the maximum number of milliseconds to wait to connect to a database before throwing an Error.  Default value is `15000` milliseconds.\n\n\n#### eventSource\n  \u003e Parameters used to connect to the Event Source database\n\n| Field         | Required | Description                \n| ------------- |:--------:| :---------------------------------------\n| host          | Yes      | database server name\n| databaseName  | Yes      | database name containing the events table         \n| user          | No       | database user name.  connection attempt will fail if missing and required by database.\n| password      | No       | database user password.  connection attempt will fail if missing and required by database.\n\n#### replicationDestinations\n\u003e An array of one or more objects containing parameters identical to `eventSource` parameters.  These parameters are used to connect to `replicationDestination` databases which must contain a table called `events`.  The `replicationDestination` databases are where the events from the `eventSource` database `events` table are copied.\n\n# Database Schema\nAll databases used in the replication process must have an `events` table that has the same schema.\n\nThe `eventSource` database has more functionality than the `replicationDestination` databases.\n\nThe `events` table in the `eventSource` database has a trigger and trigger function that support the notification process mentioned in the [`Run`](#Run) section.\n\nAlso the `eventSource` database has an `id` table that is used to provide values for the `id` column in the `events` table.\n\nThere is also an `insert_events` function in the `eventSource` database that is used to insert events into the `events` table in a transactional manner.\n\nFor further database schema information, please refer to [`slate-init-db`](https://github.com/panosoft/slate-init-db).\n\n# Operations\n### Start up validations\n- Configuration parameters are validated\n- The locations for the `eventSource events` table and `replicationDestination events` tables are validated, e.g. the source and destinations are unique\n- If the `events` table in any `replicationDestination` database is non-empty, then the first event in that table must be equal the first event in the `eventSource` database\n- Row counts and maximum id column values in the `eventSource` and `replicationDestination events` tables are checked for consistency\n- If the `slate-replicator` is started in `--dry-run` mode then it will validate and display configuration parameters without running the replicator\n- All start-up information and any configuration errors are logged\n\n### Run\nThe `slate-replicator` listens for notifications generated by an `events` table trigger in the `eventSource` database using the `Postgresql LISTEN` command. This triggers the replication process. Notifications are generated by the `Postgresql pg_notify` function when events are inserted into the `events` table in the `eventSource` database.\n\nEach time the replication process is triggered, it will replicate all events that are missing from the `replicationDestination` database.\n\n### Shutdown\nThe `slate-replicator` can be shutdown using the `Cntrl-C` keystroke.\n\n### Exceptions\nThe `slate-replicator` process ends when any exception occurs.  Since the replication process is transactional, the `slate-replicator` can be restarted at any time.  If an exception stops the `slate-replicator`, some other process needs to start it again.\n\n### Logging\nThe `slate-replicator` uses the `bunyan` logging library to log all exceptions and informational messages. Logging is done to the console as the replicator was designed to be run in a Docker container.\n\n### Test Tools\nThere are two Testing tools, `loadPersonData.js` and `eventsDiff.js`, provided in the test directory to aid testing the `slate-replicator`:\n\n#### loadPersonData.js\nThis program can be used to test `slate-replicator` by loading the `events` table in the `eventSource` database with realistic looking event data supplied by using the [`faker`](https://www.npmjs.com/package/faker) library.\n\nIt is started by running `slate-loadPersonData [options]`.\n\nIt also creates validation data in each event that can be optionally checked by the eventsDiff.js program to provide a more thorough validation of `slate-replicator` processing.\n\n#### eventsDiff.js\nThis program validates the data in each `events` table being compared, and then compares data in the `events` table in the `eventSource` database with data in the `events` table in the `replicationDestination` database as specified in the configuration file.\n\nIt is started by running `slate-eventsDiff [options]`.\n\nThis program can be run such that all `events` data row differences can be reported or the program can stop after detecting a configurable number of `events` data row differences.\n\nThe `loadPersonData` and `eventsDiff` programs can be used to test `slate-replicator` in the following manner:\n- Run the `loadPersonData` program to create data in the `events` table in the `eventSource` database.  Multiple `loadPersonData` programs can be run at the same time for a more robust test.\n- Run the `slate-replicator` to replicate the test data to an `events` table in a `replicationDestination` database, and stop the `slate-replicator` using `Cntrl-C` when replication is complete.  The `slate-replicator` program can be started before, during, or after the `loadPersonData` program(s) are running.\n- Configure and run the `eventsDiff` program to validate and compare data in the two `events` tables processed by the replicator program.  If the replicator ran properly, then there should be no validation errors or event differences reported by the `eventsDiff` program.\n\n#### loadPerfTestData.js\nThere is an additional Testing tool, `loadPerfTestData.js`, provided in the test directory that was written to create data in the `events` table in order to test database and index performance.\n\nIt is started by running `slate-loadPerfTestData [options]`.\n\nFurther information regarding how to use the test tools can be found by running the test tool using `--help` for `[options]` or by reading the comments in the test programs.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpanosoft%2Fslate-replicator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpanosoft%2Fslate-replicator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpanosoft%2Fslate-replicator/lists"}