https://github.com/ansforge/psc-rass-loader
module de chargement des modifications quotidiennes du RASS dans la base de données Sec-PSC
https://github.com/ansforge/psc-rass-loader
sec-psc
Last synced: 12 months ago
JSON representation
module de chargement des modifications quotidiennes du RASS dans la base de données Sec-PSC
- Host: GitHub
- URL: https://github.com/ansforge/psc-rass-loader
- Owner: ansforge
- Created: 2023-09-12T08:02:42.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-11-26T10:44:20.000Z (over 1 year ago)
- Last Synced: 2025-01-11T09:50:03.977Z (over 1 year ago)
- Topics: sec-psc
- Language: Java
- Homepage:
- Size: 52.5 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# pscload-v2
pscload V2
Component to load RASS data in psc db
## Developement
### Tool versions
Please note that `psc-rass-loader` depends on deprecated JDK APIs, and the build
breaks from JDK 21 on.
This can be addressed by :
* Installing a JDK-11 or 17 version (this can be done locally instead of system-wide)
* Setting up the JAVA_HOME variable to this JDK version.
### Distribution history
This ecosystem uses many independant components, some of which live an independant life in distinct repositories.
For each release of `psc-ps-api`, [the psc-compoents' distribution history](https://github.com/ansforge/psc-components/blob/main/DISTRIBUTION.md)
file will need to be updated with the new version, so that we can keep track of compatible component versions,
and go back to a previous working distribution if need be.
### Release procedure
Whenever a version is ready for release, run the following commands on the `main` branch (or on the maintenance branch if we're about to issue a production FIX). This should run on any shell, be it `bash`, `cmd` or if needed `gitbash`.
```bash
mvn release:prepare -DautoVersionSubmodules=true -DtagNameFormat=@{version}
git push
git push origin
```
where `` stands for the new version.
Eg to relase `1.0.1` :
```bash
mvn release:prepare -DautoVersionSubmodules=true -DtagNameFormat=@{version}
git push
git push origin 1.0.1
```
## Troubleshooting
For trouble shooting see the [DEBUG](DEBUG.md) section.
## Qualification
To help qualification, two test datasets have been added (see below).
To use extract files from these datasets, set the psload key `extract_download_url` in Vault for the instance to `https://raw.githubusercontent.com/ansforge/psc-rass-loader/main/pscload/src/test/resources//`. For example, to use the `BaseExtract.zip` extract from dataset **qualif-extracts**, use : https://raw.githubusercontent.com/ansforge/psc-rass-loader/main/pscload/src/test/resources/qualif-extracts/BaseExtract.zip
### qualif-extracts
This datasets consists in 5 extracts + 1 toggle file (see the [qualif-extracts directory](pscload/src/test/resources/qualif-extracts))
around 100,000 data lines :
* **BaseExtract.zip** : full 100,000 dataset to use as reference.
* **Extract_20.000_changes.zip** : BaseExtract.zip with 20.000 changed identities. This will trigger 20.000 updates (and switching back to `BaseExtract.zip` will trigger 20000 more updates). 20.000 deletes are 20% updated lines and willtrigger an alert.
* **Extract_20.000_deletes.zip** : BaseExtract.zip with 20.000 less identities. This will trigger 20.000 deletes (and siwtching back to `BaseExtract.zip` will trigger 20000 creates). 20.000 deletes are 20% deleted lines and will trigger an alert (but 20000 creates won't)
* **Extract_400_changes.zip** : this will trigger 400 updates (no alert as this is less than 5% change)
* **Extract_400_deletes.zip** : this will trigger 400 deletes (no alert as this is less than 5% change)
* **toggle-1.csv**: this is a toggle file to be uploaded to add or remove toggles. Of course, subsets can be used too.
### CT-AMAR
This dataset consists in 3 extracts and 2 CSV files, in order (see the [CT-AMAR directory](pscload/src/test/resources/CT-AMAR)):
1. **0-RAZ.zip** : first extract, used to wipe the database.
1. **1-CREATES.zip** : second extract, used to create reference identities.
1. **2_Correspondance_ADELI_RPPS_creates.csv** : this file defines identity toggles (should be loaded through the add toggle action in the admin portal).
1. **3-UPDATES.zip** : this triggers updates on some identities
1. **4_Correspondance_ADELI_RPPS_modifications.csv** : this file defines updates in toggles, to be loaded throug the add toggle action in the admin portal.