Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/lblod/cevi-lpdc-migratie

Repository for the migration of CEVI LPDC instances to ABB LPDC
https://github.com/lblod/cevi-lpdc-migratie

Last synced: 22 days ago
JSON representation

Repository for the migration of CEVI LPDC instances to ABB LPDC

Awesome Lists containing this project

README 

        
# cevi-lpdc-migratie



Repository for the one-time migration of CEVI LPDC instances to ABB LPDC.



> **Important!!** \

> Rerun migration for each environment again by changing sparqlClientUrl, because concepts are different for each environment 



### preparation before use



**Note**: in the next usage explanation `LPDC_CEVI.xml` is used as example, please add your own bestuurseenheid relevant files.



- Place your cevi xml in folder `\src\resources\LPDC_CEVI.xml`

  - By default all in folder `\src\resource\*` but `LPDC_CEVI.xml` are git ignored.

- Ensure you have created a `\src\migrate-.ts`

  - One for each bestuurseenheid that needs migrating is needed.

  - It contains 

    - The xml file to process

    - the bestuurseenheid uuid

    - the lokaal bestuur nuts lau code url

    - the language chosen by the bestuurseenheid: `Language.INFORMAL` or `Language.FORMAL`

    - the sparqlclienturl for the environment where you will upload the results. This is internally use to look up the correct concept id for a given product number. You might need to tunnel locally. And of course, the environment needs to be running.

  - as an example: `\src\migrate-cevi.ts` 

```typescript

import {runProcess} from "./migration-process";

import {Language} from "./language";



async function main() {



  const xmlFileName = 'src/resources/LPDC_CEVI.xml';

  const bestuurseenheidUuid = '6025a5d1ca2262a784f002edd8ce9ca9073ae3d5ebc6b6b5531f05a29e9250af';

  const lokaalBestuurNutsLauUrl = 'http://data.europa.eu/nuts/code/BE23646024';

  const language = Language.INFORMAL;

  const sparqlClientUrl = `http://localhost:8896/sparql`;



  await runProcess(xmlFileName, bestuurseenheidUuid, lokaalBestuurNutsLauUrl, language, sparqlClientUrl);

}



main();

```

- Ensure you created in `package.json` a script to run the `\src\migrate-.ts` file

  - as an example:  `"migrate-cevi": "ts-node src/migrate-cevi.ts"`

- As a pre running step : ensure that the product ids of following archived concepts are replaced. You have to manually replace in the xml. Is not programmed.

  - 21629 -> 5006

  - 21628 -> 5002

  - 21625 -> 5008

  - 21624 -> 5005

  - 21620  -> 5003

  - 21619 -> 5004

  - 734 -> 5001

  - 21591 -> 5007



### running the migration script: creating the ttl and sparql files



- The following will only create ttl and sparql files. It will not upload these files to the server. Allowing manual verification first.

- It will also not verify if any of these results are already uploaded to the environment. xml is just transformed in ttl and sparql.

- Run the migrate script on command line: `npm run migrate-cevi`

- Following timestamped files are created on each run under `\src\migration-results`:

  - ![lpdc-cevi-migration-results.png](images%2Flpdc-cevi-migration-results.png)

  - All timestamps are [Coordinated Universal Time](https://en.wikipedia.org/wiki/Coordinated_Universal_Time).

  - `LPDC_CEVI-2023-10-19T06:43:16.960Z.xml` : xml file is copied and versioned.

  - `LPDC_CEVI-2023-10-19T06:43:16.960Z-imported-cevi-instances.csv`: an overview of cevi instance id, title, whether import was successful, if not, augmented with reason 

  - `LPDC_CEVI-2023-10-19T06:43:16.960Z-mapping-failed-fields-log.csv`: each cevi instance id and non-mappable fields the reason

  - `LPDC_CEVI-2023-10-19T06:43:16.960Z-migration.log`: contains messages, warnings, errors

  - `LPDC_CEVI-2023-10-19T06:43:16.960Z-migration.ttl`: ttl to upload, in quad format

  - `LPDC_CEVI-2023-10-19T06:43:16.960Z-update-concept-display-configuration.sparql`: sparql query to update the concept display configurations of linked concepts



### uploading the ttl and sparql files to the environment



- Upload the ttl file in the virtuoso conductor:

  - Linked Data > Quad Store Upload

  - Choose ttl file

  - No need to fill in the Named Graph IRI*

  - Upload

  - Should only take a few seconds

- Execute the sparql file using the sparql endpoint.

- Run and execute the ['adres id migration script'](https://github.com/lblod/app-lpdc-digitaal-loket/tree/development/migration-scripts/adressen) to fill up the address id's for the migrated addresses.