Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/SoftwareAG/webmethods-suite-logfile-archiver

Archive and (after a customizable retention period) delete log files from webMethods Suite products
https://github.com/SoftwareAG/webmethods-suite-logfile-archiver

ant operations webmethods

Last synced: about 2 months ago
JSON representation

Archive and (after a customizable retention period) delete log files from webMethods Suite products

Awesome Lists containing this project

README 

        
# webMethods Suite Log File Archiver



The job of the Log File Archiver is to archive and (after a customizable

retention period) delete log files from webMethods Suite products.

This prevents file systems from filling up and helps to cater for compliance

requirements.



Log File Archiver has been implemented as an Ant script, because that 

makes it very easy to work with files. Also, changes can be done quickly 

and easily, when a certain environment needs different handling.



To make installation and use easier for standard environments, it

comes with convenience scripts (batch and shell) to make its use

as easy as possible.



Currently supported webMethods components are

- Integration Server and Microservices Runtime

- Universal Messaging

- MWS

- Optimize

  - Analytic Engine

  - Web Service Data Collector

- Command Central

  - CCE

  - SPM



If you need support for other components, please create a GitHub issue

or simply implement it yourself and feed back your improvement as a 

pull request.



## How It Works



The overall approach is that the script loops over all components

and does the following things:



- Change into the log file directory of the component.

- Identify all files that are to be processed on that day.

  Those are effectively the files with a date-time-stamp, which points

  to a day in the past. In other words, all files that have

  accumulated since the last run of Log File Archiver will be processed.

  - On the very first run all files that have been created, perhaps over

    months or even years, will be processed. This can be a lot of files

    and a significant amount of space.

  - When run on a daily basis, usually via something like a Cron job,

    this means that files for only the last day will be processed.

  - As an exception to the last point, there are cases when the logging

    sub-system still holds a lock on the file from a past day. This

    typically happens when nothing was written into a specific log file

    over the entire day, which can prevent the roll-over from

    happening. With such a lock in place, there is no way for Log File

    Archiver, or any other tool, to safely move the file. It will

    therefore be skipped and tried the next time. While unlikely,

    it is possible this happens multiple days in a row.

  - It is important to note that this selection happens based on the

    file name and NOT the age of the file (as per the file system).

    This is different from deleting the archived individual files

    (see below), which happens based on the actual age of files.

- The first action with the selected files will be to put them into

  a ZIP file for long-term retention. This ZIP file will be created

  in `$WEBMETHODS_HOME/archive/logs/`. The name fill follow the pattern

  `_yyyy-MM-dd_HH-mm-ss-SSS.zip`. Example for

  Integration Server: `is_2023-11-22_15-53-02-114.zip`.

  The ZIP file generated by the first run can be rather big, if

  the environment has already been running for a while (see above).

- The second action will be to move all selected files into the respective

  archive log directory. The default location for this is 

  `$WEBMETHODS_HOME/archive/logs//`.

- As a third step all individual log files in the per-compnent

  sub-directories (as opposed to the ZIP files in the main

  log archive directory) that have "expired" will be deleted.

  Expired means files older than a certain number of days, based

  on the age of the file in the file system and not based on

  their names. 

- The fourth and last step is deleting those long-term ZIP files that

  have expired. Their default retention is now 10 years.

  - Please verify

    the legal requirements for how long you must store these files. It is

    also common that after the legally imposed retention period you are

    actually required to delete them. Again, this needs to be handled by

    someone who is qualified to give legal advice within your jurisdiction.

  - Special care is needed on systems where the Log File Archiver was first

    run some time after the system went online. Here you can end up with the

    situation that a ZIP file that as a whole must be kept, contains files

    that must(!) be deleted. Example: Assume your system was running for

    2 years prior to the installation of Log File Archiver. In that case,

    the first ZIP file that is 9 years old will most likley contain files that

    are actually 11 years old. So you may be required to delete those

    individual files from the ZIP file.



## Installation



### Scripted Installation



Currently for Windows and Linux systems only, there are

batch/script files ([install.bat](install.bat) and [install.sh](install.sh)) 

to perform the installation.



- If you used the default location (`c:\SoftwareAG` or `/opt/softwareag`) 

or have set the environment variable `WEBMETHODS_HOME` to point to your 

installation you simply run it

  ```bash

  install.bat

  ```

  or

  ```bash

  install.sh

  ```



- Otherwise you must specify the location of the webMethods installation

  as a parameter

  ```bash

  install.bat WEBMETHODS_INSTALLATION

  ```

  or

  ```bash

  install.sh WEBMETHODS_INSTALLATION

  ```



The installation will place Log File Archiver into

`$WEBMETHODS_HOME/tool/operations/logArchiver`. It is not possible to

specify a different location using these installation scripts.

If you need a different place, you

must perform the installation manually as described in the next section.



### Manual Installation



Alternatively, you can perform the installation manually. In this case 

you must ensure that Ant and Java are installed and set up correctly:



- Install a JDK and Ant on your machine. The details depend on your

  environment and are not in scope here. You will, however, find

  plenty of information on that subject on the Internet.

- Configure the following environment variables

  - `WEBMETHODS_HOME` pointing to the location directory of the 

    webMethods suite

  - `ANT_HOME` pointing to your Ant installation location. The easiest 

    way will be to use the installation that usually comes with the

    webMethods Suite.

  - `JAVA_HOME` pointing to a JDK. The easiest way will be to use the 

    installation that usually comes with the webMethods Suite.

- Ensure that `$ANT_HOME/bin` and `$JAVA_HOME/bin` are part of the 

  search path (`PATH` variable)



You can now perform the installation.



  - To install into the default location (`$WEBMETHODS_HOME/tools/operations`):

    ```bash

    ant install

    ```

  - To install into a ustom installation directory:

    ```bash

    ant -Dinstall.dir= install

    ```



### Create Archive for Transfer to Target System



As a last option, you can also build a ZIP file for later manual 

installation. This will usually be your preferred way, if one or 

both of the two following scenario are relevant:



- You populate your servers or build container images with data

  from a fixed location. Those locations are typically secured

  and provide functionality like check-sums, auditing for file

  changes, etc.

- You want to use dedicated releases that were created from 

  the source code at a certain point in time. Also, such releases

  are typically subject to an elaborate and audited release

  process.



As a prerequisite, similar to the manual installation, you also

need to set up Ant and Java (environment variables and path)

like explained in the section for manual installation.



Then you can simply run Ant without any additional parameters:

 

 ```bash

ant

```



The ZIP archive will be created into a new folder (`./dist`) that

will be created automatically. This folder is excluded from

source control by Git.



Depending on the details of how things are handled, you may have to

make the `run_log-archive.sh` script executable on the

target environment. If you are unsure about the exact permissions

to set, it is recommend to err on the side of caution and use

`chmod 700 run_log-archive.sh`. That ensures that only the owner

of the file can directly execute it.



## Usage



Similar to the installation, there are also batch/script files

to invoke the actual log file archiver.



The script files' (`run_log-archive.bat` and `run_log-archive.sh`)

logic is such that on a normal system no further setup work is required.

But on systems where a more elaborate configuration exists, those

specifics are taken into account. Therefore the following environment

variables are used (if they are all defined, the archiver files can be

stored anywhere):



- `WEBMETHODS_HOME`: Installation location of the webMethods Suite for

  which the log files should be archived. If not defined, it will be

  assumed that the script is installed in

  `$WEBMETHODS_HOME/tools/operations/logArchiver` and the value derived

  from that.

- `JAVA_HOME`:

  - On Linux only and if it is not set, the script uses the contents of

  `/etc/profile.d/jdk.sh` if it exists.

  - If not defined, the JVM that comes with the webMethods Suite will

  be used (depends on name of JVM folder and may therefore not work on all

  versions of the webMethods Suite).

- `ANT_HOME`:

  - On Linux only and if it is not set, the script uses the contents of

  `/etc/profile.d/ant.sh` if it exists.

  - If not defined, the Ant that comes as part of the webMethods Suite

    will be used (in `$WEBMETHODS_HOME/common/lib/ant`).



It is recommended for most cases to set up a scheduled job (e.g. via Cron)

on the operating system to execute the script on a daily basis. Please be

aware that in this case the environment variables may differ from what is

normally available when a user is logged in. This is normal for running

Cron jobs and in now way specific to Log File Archiver



It is also an option to run the Ant script from a Continuous Integration

server (like e.g. Jenkins). In that case the control over environment

variables may be easier than via a Cron job.



On Windows systems there have been cases where, due to file locks, log

files could only be moved one or several days after that should have

happened. While this delay may be a bit annoying, it is usually not a

real problem.

  

## Parameters



The behavior is controlled by Ant properties with defaults being in the

script. Those defaults can be overridden from the command line by using

standard Ant syntax.



Example:



```bash

ant -f build_log-archive.xml -DretentionDays=12

```



The following parameters can be used



| Name | Default | Format | Description | Min. Version |

| ---  | ---     | ---    | ---         | ---          |

| `webMethods.home` | OS environment variable `$WEBMETHODS_HOME` | Valid path | Location of the webMethods installation | 1.0 |

| `createZIP` | true | true/false | Should a ZIP be created per invocation | 1.1 |

| `retentionDays` | 30 | positive number | Number after which log files (not ZIP) shall be deleted | 1.0 |

| `retentionDaysZIP` | 3660 | positive number | Number after which ZIP archives shall be deleted | 1.1 |

| `instancesNameIS` | default | text | Name of IS instance for which files shall be archived | 1.1 |

| `instanceNameUM` | umserver | text | Name of UM server instance for which file shall be archived | 1.2 |



______________________

These tools are provided as-is and without warranty or support. They do not constitute part of the Software AG product suite. Users are free to use, fork and modify them, subject to the license agreement. While Software AG welcomes contributions, we cannot guarantee to include every contribution in the master project.



Contact us at [TECHcommunity](mailto:[email protected]?subject=Github/SoftwareAG) if you have any questions.