Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/hmcts/civil-ccd-migration-tool
https://github.com/hmcts/civil-ccd-migration-tool
hmcts-ccd jenkins-cft jenkins-cft-a-c
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/hmcts/civil-ccd-migration-tool
- Owner: hmcts
- License: mit
- Created: 2022-07-21T15:53:58.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-11-08T05:43:38.000Z (about 1 month ago)
- Last Synced: 2024-11-08T06:28:56.243Z (about 1 month ago)
- Topics: hmcts-ccd, jenkins-cft, jenkins-cft-a-c
- Language: Java
- Homepage:
- Size: 158 KB
- Stars: 0
- Watchers: 21
- Forks: 1
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Civil CCD Data Migration Tool
## Basic overview
It works by accessing the ccd-data-store-api as the system user, grabbing and filtering all cases, and then migrating the filtered cases.
To perform the migration there needs to be an event defined in the consuming case type that is defined with the ID `migrateCase`, this needs to be defined in CCD Definition.
This event `migrateCase` is then triggered by the `CaseMigrationProcessor`and as it is a CCD event it can have the standard CCD hooks,
i.e. `about-to-start`, `about-to-submit`, `submitted`.
Civil makes use of the `about-to-submit` hook to then perform the main part of migration.
### Features
- Ability to migrate single case by providing the CCD case Id i.e. `migration.caseIds=1234`
- Ability to migrate list of cases by providing comma separated list of case Ids i.e. `migration.caseIds=1234,56789`
- Ability to `dryRun` which should not invoke the update case, but still give us the count that will be migrated with actual run
- Ability to stop when reached to maximum no of cases to process. this can can be configured before each run using `migration.maxCasesToProcess`
- Performance improvements to fetch cases page by page bases and then process/migrate cases in parallel. Have ability to configure the number of threads `numThreads`
- Performance improvements to use elastic search while fetching the cases.
- Ability to turn on/off elastic search as we do't have ES for all of our lower environments. i.e. `migration.esEnabled=true`
- Further more ability to configure the query size for ES. i.e. `migration.esQuerySize=500`
- We can configure what event we need to fire, by default it uses `migrateCase` CCD event.
### Configuration properties
```yaml
migration:
id: ${MIGRATION_ID}
caseIds: ${MIGRATION_CASE_IDS:}
caseType: ${MIGRATION_CASE_TYPE}
eventId: ${MIGRATION_CASE_EVENT_ID:migrateCase}
eventDescription: ${MIGRATION_CASE_EVENT_DESC:Migrate Case}
eventSummary: ${MIGRATION_CASE_EVENT_SUMMARY:Migrate Case}
dryRun: ${MIGRATION_DRY_RUN:false}
maxCasesToProcess: ${MIGRATION_CASE_LIMIT:50000}
numThreads: ${MIGRATION_NO_OF_THREADS:20}
esEnabled: ${MIGRATION_ES_ENABLED:false}
esQuerySize: ${MIGRATION_ES_QUERY_SIZE:500}
```
### Processor framework
The migration framework for data migrations within CCD that runs the following process:
## Build
To build the project run
```shell
./gradlew clean build
```
this will generate a jar in the `build/libs` directory which can then be used when running the migration.
## Running
To run the jar you will need to do the following
```shell
java -jar \
-Dspring.application.name="civil-ccd-case-migration-tool" \
-Didam.api.url="https://idam-api.aat.platform.hmcts.net" \
-Didam.client.id="[CLIENT ID]" \
-Didam.client.secret="[VALUE IN VAULT]" \
-Didam.client.redirect_uri="https://civil-service-aat.service.core-compute-aat.internal/oauth2/callback" \
-Dcore_case_data.api.url="http://ccd-data-store-api-aat.service.core-compute-aat.internal" \
-Didam.s2s-auth.url="http://rpe-service-auth-provider-aat.service.core-compute-aat.internal" \
-Didam.s2s-auth.microservice="civil-service" \
-Didam.s2s-auth.totp_secret="[VALUE IN VAULT]" \
-Didam.username="[email protected]" \
-Didam.password="[VALUE IN VAULT]" \
-Dmigration.id="My Migration" \
-Dmigration.jurisdiction="Civil" \
-Dmigration.caseType="[CASE TYPE]" \
-Dmigration.esEnabled="[true/false]" \
-Dmigration.dryRun="[true/false]" \
-Dlogging.level.root="ERROR" \
-Dlogging.level.uk.gov.hmcts.reform="INFO" \
PATH/TO/MIGRATION.jar
```
where
- `idam.client.secret`
- `idam.s2s-auth.totp_secret`
- `migration.idam.password`
can all be found in the relevant vault.
Note that the parameters given are using AAT environment as an example.
NOTE: `migration.esEnabled` can be set to false for the environments where we don't have elastic search available. i.e. `local` or `preview`.
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.