{"id":19326881,"url":"https://github.com/panosoft/slate-init-db","last_synced_at":"2025-02-24T06:22:10.305Z","repository":{"id":57135048,"uuid":"62421306","full_name":"panosoft/slate-init-db","owner":"panosoft","description":"Creates and initializes a Postgresql database for use by Slate applications","archived":false,"fork":false,"pushed_at":"2017-09-05T17:22:26.000Z","size":40,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-02-23T04:27:29.078Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/panosoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-07-01T21:44:16.000Z","updated_at":"2016-07-01T21:52:48.000Z","dependencies_parsed_at":"2022-09-04T07:30:43.112Z","dependency_job_id":null,"html_url":"https://github.com/panosoft/slate-init-db","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-init-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-init-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-init-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/panosoft%2Fslate-init-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/panosoft","download_url":"https://codeload.github.com/panosoft/slate-init-db/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240427706,"owners_count":19799544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T02:15:13.137Z","updated_at":"2025-02-24T06:22:10.260Z","avatar_url":"https://github.com/panosoft.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEPRECATED - Please see [elm-slate/init-slate-db](https://github.com/elm-slate/init-slate-db)\n\n\n\n# slate-init-db\nCreates and initializes a Postgresql database for use by Slate applications.\n\nThe purpose of slate-init-db is to create and initialize a new `Postgresql` database to contain either a `source` or `destination` events table (see [`Database Initialization`](#database-initialization)).\n\nslate-init-db requires the `Postgresql contrib` package be installed on the `source` database server when initializing a `source` database so that the `dblink` extension can be installed (see [`Source Database Disaster Recovery`](#source-database-disaster-recovery)).\n\n# Installation\n\u003e npm install -g @panosoft/slate-init-db\n\n# Usage\n\n#### Run slate-init-db\n\n    slate-init-db [options]\n\n    Options:\n\n      -h, --help                               output usage information\n      --host \u003cname\u003e                            database server name\n      --user \u003cname\u003e                            database user name.  must have database creation privileges.  if not specified, prompt for user name.\n      --password \u003cpassword\u003e                    database password.  if not specified, prompt for password.\n      --connect-timeout \u003cmillisecs\u003e            database connection timeout.  if not specified, defaults to 15000 millisecs.\n      -n, --new-database \u003cname\u003e                name of database to create\n      -t, --table-type \u003csource | destination\u003e  type of events table to create in new database:  must be \"source\"  or \"destination\"\n      --dry-run                                if specified, display run parameters and end program without performing database initialization\n\n# Operations\n### Start up validations\n- Run options are validated\n- Database to be created must NOT exist and its name must be a valid `Postgresql` identifier\n- If `slate-init-db` is started in `--dry-run` mode then it will validate and display run options without performing database initialization\n- All start up information and any options errors are logged\n\n### Error Recovery\n- All operational errors will be logged\n- If errors are reported when running `slate-init-db` then the new database was not initialized properly and MUST be deleted manually before re-running\n\n# Database Initialization\nInitialization differs depending on the `table-type`.\n\nWhen creating a `source` database, the following source-only database objects are created: an `id` table, an `insert_events` function, a `restore_events` function, an `events` table NOTIFY trigger and its trigger function, and an `events` table COMMAND CHECK trigger and it trigger function.\n\nThe `id` table in a `source` database is used to assign ids to the rows as they are inserted into the `events` table by the `insert_events` function. The ids start at 1 and are guaranteed to be consecutive.\n\nThe `id` and `ts` column values in a `source` events table row are generated by the `insert_events` function. This means that the singleton `source` database is the master clock for all events.\n\nThe `restore_events` function is used to perform `source` database disaster recovery (see [`Source Database Disaster Recovery`](#source-database-disaster-recovery)).\n\n## Database Initialization Details\n\n### Source and Destination\n\n```sql\n--create events table\n\nCREATE TABLE events\n(\n  id bigint NOT NULL,\n  ts timestamp with time zone NOT NULL,\n  event jsonb NOT NULL,\n  CONSTRAINT events_pkey PRIMARY KEY (id)\n)\nWITH (\n  OIDS=FALSE\n);\n\n--create events table indexes\n\nCREATE INDEX events_event_entityname on events ((event #\u003e\u003e '{entityName}'));\n\nCREATE INDEX events_event_operation on events ((event #\u003e\u003e '{operation}'));\n\nCREATE INDEX events_event_propertyname on events ((event #\u003e\u003e '{propertyName}'));\n\nCREATE INDEX events_event_entityid on events ((event #\u003e\u003e '{entityId}'));\n\nCREATE INDEX events_ts on events (ts);\n```\n\n### Source ONLY\n\n#### Events Table NOTIFY trigger and trigger function (the trigger and trigger function do not exist in a destination database)\n\n```sql\n--create NOTIFY trigger function\n\nCREATE FUNCTION notify_event_insert() RETURNS trigger AS $$\nDECLARE\nBEGIN\n  PERFORM pg_notify('eventsinsert', json_build_object('table', TG_TABLE_NAME, 'id', NEW.id, 'event', NEW.event )::text);\n  RETURN new;\nEND;\n$$ LANGUAGE plpgsql;\n\n--create NOTIFY trigger\n\nCREATE TRIGGER notify_insert AFTER INSERT ON events\nFOR EACH ROW EXECUTE PROCEDURE notify_event_insert();\n```\n\n#### Events Table COMMAND CHECK trigger and trigger function (the trigger and trigger function do not exist in a destination database)\n\n```sql\n--create COMMAND CHECK trigger function\n\nCREATE FUNCTION filter_sql_command()\n\tRETURNS TRIGGER as $$\nBEGIN\n\tRAISE EXCEPTION 'cannot perform SQL command on events table:  %', TG_OP;\nEND;\n$$ LANGUAGE plpgsql;\n\n--create COMMAND CHECK trigger\n\nCREATE TRIGGER check_sql_command BEFORE UPDATE OR DELETE OR TRUNCATE ON events\nFOR EACH STATEMENT EXECUTE PROCEDURE filter_sql_command();\n```\n\n#### ID Table Initialization (this table does not exist in a destination database)\n\n```sql\nCREATE TABLE id (\n  id bigint NOT NULL,\n  CONSTRAINT id_pkey PRIMARY KEY (id))\nWITH (\n  OIDS=FALSE\n);\n```\n#### insert_events function (this function does not exist in a destination database)\n\n```sql\nCREATE FUNCTION insert_events(insertValues text)\n\tRETURNS integer AS $$\nDECLARE\n\tts timestamp with time zone;\n\tstartId bigint;\n\tnextStartId bigint;\n\tidxs bigint[];\n\tids bigint[];\n\tlastIdx integer;\n\tidx bigint;\n\trowsInserted bigint;\n\tcountRows bigint;\n\tmaxIndex bigint;\n\ttsMatches text[];\nBEGIN\n\tLOCK TABLE ID IN ACCESS EXCLUSIVE MODE;\n\t-- get timestamp to insert in each ts column\n\tts := transaction_timestamp();\n\t-- insertValues is a string value that represents the column values to insert for one or more rows for one INSERT statement.\n\t-- insertValues is formatted to follow the VALUES keyword of the statement \"INSERT INTO events (id, ts, event) VALUES \"\n\t-- (e.g. '($1[1], $2, '\u003cjson string for event\u003e'), ($1[2], $2, '\u003cjson string for event\u003e')'.\n\t-- The id column for each row to insert is formatted as a substitution parameter, $1[x], where x is an index value.\n\t-- index values start at 1 for the first row to insert and must be a consecutive positive integer for each additional row.\n\t-- The ts (event timestamp) column is generated by this function and is represented  by the parameter $2.\n\n\tSELECT ARRAY(SELECT unnest(regexp_matches(insertValues, '\\$1\\[([0-9]+)\\]', 'g'))) into idxs;\n\tSELECT ARRAY(SELECT unnest(regexp_matches(insertValues, '(\\$2),', 'g'))) into tsMatches;\n\tcountRows := 0;\n\tlastIdx := 0;\n\t-- find the row count, maximum index for a row, and check that the indices are consecutive positive integers.\n\tFOREACH idx IN ARRAY idxs\n\tLOOP\n\t\tcountRows := countRows + 1;\n\t\tIF lastIdx = 0 THEN\n\t\t\tlastIdx := idx;\n\t\tELSE\n\t\t\tIF idx = lastIdx + 1 THEN\n\t\t\t\tlastIdx := idx;\n\t\t\t\tmaxIndex := idx;\n\t\t\tELSE\n\t\t\t\tRAISE EXCEPTION 'Parameter index is not consecutive at ------\u003e %,  previous index ------\u003e %', idx, lastIdx\n\t\t\t\t\tUSING HINT = 'Parameter for id column of the form \"$1[x]\" where x is the 1-based index for the row to be inserted is not greater than the previous row''s index';\n\t\t\tEND IF;\n\t\tEND IF;\n\tEND LOOP;\n\tIF countRows \u003c 1 THEN\n\t\tRAISE EXCEPTION 'No inserted rows found with id substitution parameters' USING HINT = 'id column substitution parameter value for row to be inserted must be of the form \"$1[x]\" where x is the 1-indexed based index of the row';\n\tEND IF;\n\tIF countRows != maxIndex THEN\n\t\tRAISE EXCEPTION 'Number of rows to be inserted (%) does not match the highest row index (%)', countRows, maxIndex USING HINT = 'The highest id column parameter substitution parameter index value does not match the number of rows to be inserted';\n\tEND IF;\n\tIF countRows != coalesce(array_length(tsMatches, 1), 0) THEN\n\t\tRAISE EXCEPTION 'Number of rows to be inserted (%) does not match number of rows with a ts substitution parameter (%)', countRows, coalesce(array_length(tsMatches, 1), 0) USING HINT = 'ts column parameter substitution value for row to be inserted must be \"$2\"';\n\tEND IF;\n\n\t-- update id table to point to the next starting id value to use\n\tUPDATE id SET id = id + countRows RETURNING id INTO nextStartId;\n\t-- start id for first insert statement\n\tstartId := nextStartId - countRows;\n\t-- get ids to use for each inserted row's id column\n\tSELECT into ids ARRAY(SELECT generate_series(startId, nextStartId - 1));\n\t-- RAISE NOTICE 'ids ----\u003e %', ids;\n\tEXECUTE 'INSERT INTO events (id, ts, event) VALUES ' || insertValues USING ids, ts;\n\tGET DIAGNOSTICS rowsInserted = ROW_COUNT;\n\tRETURN rowsInserted;\nEND;\n$$ LANGUAGE plpgsql;\n```\n#### restore_events function (this function does not exist in a destination database)\n\n```sql\nCREATE FUNCTION restore_events(fromHost text, fromDatabase text, fromDatabaseUser text, fromDatabasePassword text, OUT rows_restored bigint, OUT next_insert_id bigint)\n\tAS $$\nDECLARE\n\tgetEventsStmt CONSTANT text = 'SELECT id, ts, event FROM events ORDER BY id';\n\tgetCountMaxIdStmt CONSTANT text = 'SELECT MAX(id) AS maxid, count(*) AS count FROM events';\n\tconnectionInfo text;\n\tsourceEventsCount bigint;\n\tsourceEventsMaxId bigint;\n\tsourceNextIdValue bigint;\n\tfromEventsCount bigint;\n\tfromEventsMaxId bigint;\nBEGIN\n\tconnectionInfo := 'host=' || fromHost || ' dbname=' || fromDatabase || ' user=' || fromDatabaseUser || ' password=' || fromDatabasePassword;\n\t-- get row count from Source events table.  must be 0 after being created by slate-init-db.\n\tSELECT count(*) from events into sourceEventsCount;\n\tIF sourceEventsCount != 0 THEN\n\t\tRAISE EXCEPTION 'Source events table row count (%) is not 0', sourceEventsCount USING HINT = 'The Source events database must be initialized with slate-init-db';\n\tEND IF;\n\t-- get next event id from id table.  must be 1 after being created by slate-init-db.\n\tSELECT id from id into sourceNextIdValue;\n\tIF sourceNextIdValue != 1 THEN\n\t\tRAISE EXCEPTION 'Source id table id value (%) is not 1', sourceNextIdValue USING HINT = 'The Source events database must be initialized with slate-init-db';\n\tEND IF;\n\t-- get maximum event id and row count from events table in remote database being used to restore the Source events table.\n\t-- maximum event id must be 1 or greater and equal to the row count.\n\tSELECT fe.maxid, fe.count FROM dblink(connectionInfo, getCountMaxIdStmt) AS fe(maxid bigint, count bigint) INTO fromEventsMaxId, fromEventsCount;\n\tIF fromEventsMaxId IS NULL OR fromEventsMaxId \u003c 1 THEN\n\t\tRAISE EXCEPTION 'The from events table maximum id value (%) is not 1 or greater', fromEventsMaxId USING HINT = 'The events table used to restore the Source events table must have a maximum id value of 1 or greater';\n\tEND IF;\n\tIF fromEventsCount != fromEventsMaxId THEN\n\t\tRAISE EXCEPTION 'The from events table row count (%) is not equal to the from events table maximum id (%)', fromEventsCount, fromEventsMaxId USING HINT = 'The events table used to restore the Source events table is not valid';\n\tEND IF;\n\t-- copy the events in order by id from the remote events table to the Source events table.\n\tINSERT INTO events (id, ts, event)\n\t\tSELECT fe.id, fe.ts, fe.event\n\t\t\tFROM dblink(connectionInfo, getEventsStmt)\n\t\t\tAS fe(id bigint, ts timestamp with time zone, event jsonb);\n\tGET DIAGNOSTICS rows_restored = ROW_COUNT;\n\tSELECT MAX(id), count(*) FROM events INTO sourceEventsMaxId, sourceEventsCount;\n\tIF sourceEventsCount != sourceEventsMaxId THEN\n\t\tRAISE EXCEPTION 'The Source events table row count (%) is not equal to the Source events table maximum id (%)', sourceEventsCount, sourceEventsMaxId USING HINT = 'The restored Source events table is not valid';\n\tEND IF;\n\tIF sourceEventsCount != rows_restored THEN\n\t\tRAISE EXCEPTION 'The Source events table row count (%) is not equal to the Source events table rows restored (%)', sourceEventsCount, rows_restored USING HINT = 'The Source events table restore had a program logic error';\n\tEND IF;\n\t-- update the id value in the Source id table to the maximum id value + 1 from the remote events table used to restore the Source events table.\n\tUPDATE id SET id = sourceEventsMaxId + 1 RETURNING id INTO next_insert_id;\n\t-- the count of events copied to the Source events table and the next events table id value to be used for the next event inserted into the Source events table are returned.\nEND;\n$$ LANGUAGE plpgsql;\n```\n\n# Source Database Disaster Recovery\n\nIf a `source` database disaster occurs, the `source` database tables can be restored using the `restore_events` function residing in the `source` database.\n\nTo run the `restore_events` function, the `Postgresql contrib` package must be installed on the database server where the `source` database resides.\n\nThe `dblink` extension from the `Postgresql contrib` package must be installed into the `source` database.  This extension is installed by `slate-init-db` when a `source` database is initialized.\n\nAn online backup database is required to perform the disaster recovery procedure.\n\nOne or more online backup databases can be created by using the `slate-replicator`.  For further information, please see refer to [`slate-replicator`](https://github.com/panosoft/slate-replicator).\n\n## Source Database Recovery Steps\n- Stop any programs using the `source` database and drop any connections to it\n- Stop any programs modifying the backup database\n- Delete the `source` database being recovered (if it still exists and is corrupt)\n- Create and initialize a new `source` database using `slate-init-db`\n- Run the `restore_events` function with parameters pointing to the online backup database while connected to the newly initialized `source` database\n- Optionally use the eventsDiff program in the `slate-replicator` project to compare the restored `source` database events table with the backup database events table\n\n### Running the restore_events Function\n\nIn order to recover the `source` database from the backup database, run the SQL statement below while connected to the `source` database substituting the appropriate parameters for the backup database.\n\n```sql\n--restore the source database from a backup database on host 'backupDatabaseHostName' using database 'backupDatabaseName' accessed with database user 'backupDatabaseUser' and password 'backupDatabasePassword'\n\nSELECT restore_events(fromHost := 'backupDatabaseHostName', fromDatabase := 'backupDatabaseName', fromDatabaseUser := 'backupDatabaseUser', fromDatabasePassword := 'backupDatabasePassword');\n```\n\nIf there are any errors running the `restore_events` events function, then the cause of the errors must be corrected and the Recovery Steps need to be performed again **from the beginning**.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpanosoft%2Fslate-init-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpanosoft%2Fslate-init-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpanosoft%2Fslate-init-db/lists"}