https://github.com/eea/eionet.xmlconv
Eionet XML Conversion and QA
https://github.com/eea/eionet.xmlconv
Last synced: about 1 year ago
JSON representation
Eionet XML Conversion and QA
- Host: GitHub
- URL: https://github.com/eea/eionet.xmlconv
- Owner: eea
- License: other
- Created: 2014-10-16T20:03:34.000Z (over 11 years ago)
- Default Branch: ConverterWorkerIntegrationMain
- Last Pushed: 2025-03-10T12:43:04.000Z (over 1 year ago)
- Last Synced: 2025-03-25T04:41:51.442Z (about 1 year ago)
- Language: Java
- Size: 75.3 MB
- Stars: 5
- Watchers: 56
- Forks: 3
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
## QA and Converters (XMLCONV) software
### Prerequisites
* Java 11
* Maven 3.3.9
* Tomcat 9.x
* MySQL 8.x
* Docker 1.12 or higher
## Installation Guide
Create local.properties file by copying the contents from default.properties as an example.
Edit the values of system paths, database url and other properties to what is relevant.
"app.home" property represents the system folder which is the root for subfolders where all content files will be stored.
Tomcat user should have write permissions on "app.home" subdirectories.
### Development Phase - Building the .war
The default profile is using the docker maven plugin to setup a mysql database for use with the integration tests phase.
Ideally, the mysql database should be using tmpfs filesystem, in order for the integration tests to run faster.
To create a .war file for deployment with tomcat, you can run any of the following commands:
To run unit and integration tests before building, run:
$ mvn clean install -Denv=jenkins
To skip the integration tests (not recommended), you can add -Dmaven.test.skip=true e.g
$ mvn clean install -Dmaven.test.skip=true
### Runtime configuration
The Xmlconv application needs a number of configuration options, such as database connection information, FME credentials, BaseX server config etc.
Every configuration option can be provided either in the properties file that the app is built with, or with an environmental variable at runtime.
The environmental configuration of a property is used if found.
This way, whatever the local.properties the app was built with, it can run at any host through environment configuration, for example by changing the database config and app.home.
A helpful configuration key named initial.admin can be set to bootstrap in the ACLs the initial administrators at runtime.
In case you want to provide environmentally the key/value pairs you have to set CATALINA_OPTS in Tomcat with the key/value pairs you want to set:
e.g. CATALINA_OPTS=-Dapp.home=/opt/xmlconv -Dapp.host=converstest.eionet.europa.eu -Dconfig.db.jdbcurl=jdbc:mysql://mysql:3306/xmlconv -Dconfig.db.driver=com.mysql.cj.jdbc.Driver -Dconfig.db.user=root -Dconfig.db.password=xxxxx -Dbasexserver.host=basex -Dbasexserver.port=1984 -Dbasexserver.user=admin -Dbasexserver.password=admin
### Docker configuration
After having built the WAR file with maven, it can be directly used in docker containers thanks to the environmental configuration. The Dockerfile can be used to build a ready-to-deploy image of xmlconv :
$ docker build -t eeacms/xmlconv:latest .
There is a script docker.hub.sh that builds the WAR and pushes to the Docker Hub, one to the latest tag, and one with a timestamp, for versioning of images.
### Rancher deployments
An example docker-compose for usage on Rancher deployments can be found on docker/xmlconv along with an example environment file.
**Important Note:**
When deploying to rancher platform, any environment variables provide to tomcat container through the *CATALINA_OPTS* variable, should not include new lines for
tomcat versions 9.0.24 and above.
### Parameters configuration for mock application
For test purposes a mechanism has been added to mimic long running jobs. This mechanism uses maven class shadowing to override 2 key classes of
basex library (the library used to run xquery scripts), providing timeouts and frequent control checks in order to halt a long running job. When
below parameters are set in a testing environment all CR calls will be directed to a mock cr application which has been created for this purpose
and what really does is returning a file (like the sparql results that CR would sent) but very slowly, in order to mimic a very busy CR with a
very bad Sparql query. In a production environment redirection to mock application should be disabled and calls to CR will proceed normally.
#### parameters config for test environment
* config.cr.host=https://cr.eionet.europa.eu
* config.cr.mockCrUrl=http://mockxquerydelay.ewxdevel1dub.eionet.europa.eu
* config.enableXqueryCrCallsInterception=true
#### parameters config for production
* config.cr.host=
* config.cr.mockCrUrl=
* config.enableXqueryCrCallsInterception=false
#### Setup a dockerized rabbitmq instance locally:
docker run -d --hostname my-rabbit -p 0.0.0.0:15672:15672 -p 5672:5672 --name some-rabbit -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password rabbitmq:3-management
#### Property for enabling scheduled task that orchestrates creation/deletion of jobExecutor containers
* env.enable.jobExecutor.rancher.scheduled.task=false
It should be set to false locally so as not to run.
#### Rabbitmq rancher stack configuration
To properly configure rabbitmq stack, a file rabbitmq.conf needs to be created in the directory /etc/rabbitmq/ of the rabbitmq stack.
The file should contain the line "consumer_timeout = 16200000", in order to increase the default timeout value that a rabbitmq consumer
should ack its delivery before the channel is closed.
#### Rabbitmq for CDR-Converters commmunication
We need to have a service alias in our stack that is linked to the rabbitmq stack.
We also need the following properties to be set up:
- env.cdr.rabbitmq.host
- env.cdr.rabbitmq.port
- env.cdr.rabbitmq.username
- env.cdr.rabbitmq.password
#### Converters rancher stack configuration
When converters is deployed for the first time, the properties maxLightJobExecutorContainersAllowed and maxHeavyJobExecutorContainersAllowed are added in database
table "PROPERTIES". These properties should then be configured by an admin user from UI tab "Admin tools" -> "View and Edit Properties" according to needs.
#### log4j2 RollingFile appender configuration
Add the following environment variables that will be used in log4j2.xml:
- A variable logFilePath with value a path to a folder where log files will be stored.
- A variable with name queryLogRetainAll and value=false if log files should be deleted, true otherwise.
- A variable queryLogRetentionDays with value the duration in days (e.g P20D) for which log files will be retained. Log files older than the specified duration will be deleted.