{"id":23067061,"url":"https://github.com/dvsa/cvs-tsk-update-store","last_synced_at":"2025-04-03T09:14:18.129Z","repository":{"id":37438722,"uuid":"338074536","full_name":"dvsa/cvs-tsk-update-store","owner":"dvsa","description":"Lambda to (Async) push data changes from DynamoDB stores into a relational DB model.","archived":false,"fork":false,"pushed_at":"2024-04-15T03:51:39.000Z","size":3924,"stargazers_count":1,"open_issues_count":27,"forks_count":1,"subscribers_count":8,"default_branch":"develop","last_synced_at":"2024-04-15T04:43:07.592Z","etag":null,"topics":["app-dev","cvs-project"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dvsa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-02-11T15:49:50.000Z","updated_at":"2024-04-16T09:49:57.327Z","dependencies_parsed_at":"2024-01-15T16:48:29.065Z","dependency_job_id":"682fe9f7-325c-46f6-b4e4-0be5be4ba79e","html_url":"https://github.com/dvsa/cvs-tsk-update-store","commit_stats":null,"previous_names":[],"tags_count":1968,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvsa%2Fcvs-tsk-update-store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvsa%2Fcvs-tsk-update-store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvsa%2Fcvs-tsk-update-store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dvsa%2Fcvs-tsk-update-store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dvsa","download_url":"https://codeload.github.com/dvsa/cvs-tsk-update-store/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246970323,"owners_count":20862509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["app-dev","cvs-project"],"created_at":"2024-12-16T05:16:02.243Z","updated_at":"2025-04-03T09:14:18.109Z","avatar_url":"https://github.com/dvsa.png","language":"TypeScript","readme":"# cvs-tsk-update-store\n\n## Overview\nLambda handler and business logic for the CVS task \"Update Store\":\n\n1. Receives DynamoDB stream events, containing NoSQL document snapshots (\"**images**\")\n2. Converts these images to a TypeScript model, then to an SQL-friendly tuple.\n3. Upserts (inserts or updates) tuples into RDS [Aurora][aurora], a relational SQL database.\n\nSee full conversion procedure in sections below.\n\nThis Lambda currently supports the following conversions:\n\n| Source (DynamoDB)   | Destination (Aurora) |\n|---------------------|----------------------|\n| `Technical_Records` | `technical_record`   |\n| `Test_Results`      | `test_result`        |\n\nSee full list of affected Aurora tables in sections below.\n\n### Relation to other services\n`cvs-tsk-update-store` was built as part of the **VOTT** project. See:\n* [Project home](https://wiki.dvsacloud.uk/display/HVT/Vehicle+Operator+Test+Transparency)\n* [Architecture diagram](https://wiki.dvsacloud.uk/pages/viewpage.action?spaceKey=HVT\u0026title=VOTT+Architecture)\n* The following diagram:\n\n![Summary Diagram](summary-diagram.png)\n\n### Cloning\nThis project makes use of [Git submodules][git-submodules] to import [cvs-nop][cvs-nop]. Instead of the regular `git clone`, use:\n\n```shell\ngit clone git@github.com:dvsa/cvs-tsk-update-store.git --recurse-submodules\n```\n\nOr, if already cloned:\n\n```shell\ngit submodule update --init --recursive\n```\n\nBy default, the submodule will be in a detached `HEAD` state. If you need a specific branch on `cvs-nop`, you will need to:\n\n```shell\ncd cvs-nop\ngit checkout feature/CVSB-XXXXX\ngit pull\n```\n\n### Debugging\n`cvs-tsk-update-store` is **quiet by default**. It will log almost nothing to CloudWatch unless an error occurs.\n\nTo change this, set the environment variable `DEBUG` to any non-null value, e.g. `DEBUG=1`.\n\nThis will cause numerous debug logs to fire. Messages will include event information, SQL, template variables, procedure entries and exits and more.\n\n### Tests\n```shell\nnpm run test\n```\nRuns all unit tests\n\n```shell\nnpm run test-all\n```\nRuns all the tests (Unit and Integration)\n\n### Integration tests\nIntegration tests depend on:\n* a local Docker installation, with `docker` on the system's path\n* a local Liquibase installation, with `liquibase` or `liquibase.bat` (on Windows) on the system's path\n\nThey work by:\n1. Creating a MySQL container\n2. Spawning a Liquibase executable\n3. Applying a Liquibase changelog to the running container\n4. Overriding connection pool configuration to connect to container\n\nRun them using:\n```shell\nnpm run test-local-i\n```\n\nOn Windows, you will need to manually run:\n* `SET USE_CONTAINERIZED_DATABASE=1`\n* Confirm: `echo %USE_CONTAINERIZED_DATABASE%` should return `1`before the above, or run these tests with an IntelliJ configuration which sets the right environment variable for you.\n\nOn Tanio Artino's advice, there is a distinction between running integration tests locally, and running them on Jenkins.\n* `USE_CONTAINERIZED_DATABASE=0` will attempt to talk to `127.0.0.1:3306`. This only works if you manually start a DB or, in the case of Jenkins, if there is a pre-existing DB running on port `3306`.\n* `USE_CONTAINERIZED_DATABASE=1` will spin up a TC container as described above.\n\nTo run the \"Jenkins\" version, e.g. `USE_CONTAINERIZED_DATABASE=0`, use:\n```shell\nnpm run test-i\n```\n\n## Full conversion procedure\nRough ordering - may not follow code exactly.\n\n1. DynamoDB table emits event on `INSERT`, `MODIFY` or `REMOVE`\n2. Event validation: is this really a DynamoDB stream event?\n3. Derive correct SQL operation `INSERT` (supported), `UPDATE` (supported) or `DELETE` (not implemented)\n4. Parse DynamoDB document snapshot into usable image object\n5. Obtain source table name from event source ARN\n6. Get conversion procedure by table name\n7. Start conversion procedure given image, table name and SQL operation\n8. Parse root image: map all applicable fields to equivalent TypeScript object hierarchy\n9. Generate `INSERT INTO (...) ON DUPLICATE KEY UPDATE (...)` for each destination table\n10. Transactionally execute all SQL statements, rolling back on error\n11. Return upsertion result containing all primary and foreign keys\n\n## Technical Record conversion\nFor the full field-to-column mapping, see `tech-record-document-conversion.ts`.\n\n* Source (DynamoDB) table: `Technical_Records`\n* Destination (Aurora) tables, in upsertion order:\n  * `vehicle`\n  * `make_model`\n  * `vehicle_class`\n  * `vehicle_subclass`\n  * `identity`\n  * `contact_details`\n  * `technical_record`\n  * `psv_brakes`\n  * `axle_spacing`\n  * `microfilm`\n  * `plate`\n  * `axle`\n\n## Test Result conversion\nFor the full field-to-column mapping, see `test-result-record-conversion.ts`.\n\n* Source (DynamoDB) table: `Test_Results`\n* Destination (Aurora) tables, in upsertion order:\n  * `vehicle`\n  * `test_station`\n  * `tester`\n  * `vehicle_class`\n  * `preparer`\n  * `identity`\n  * `fuel_emission`\n  * `test_type`\n  * `test_result`\n  * `defect`\n  * `location`\n  * `test_defect`\n  * `custom_defect`\n\n[aurora]: https://aws.amazon.com/rds/aurora\n[git-submodules]: https://git-scm.com/book/en/v2/Git-Tools-Submodules\n[cvs-nop]: https://github.com/dvsa/cvs-nop\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvsa%2Fcvs-tsk-update-store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdvsa%2Fcvs-tsk-update-store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdvsa%2Fcvs-tsk-update-store/lists"}