https://github.com/pat8901/hpc-data-parser

Analyzes Grid Engine log files converting them into a format suitable for time series analysis using Grafana.
https://github.com/pat8901/hpc-data-parser

grafana large-data sqlite

Last synced: 8 months ago
JSON representation

Analyzes Grid Engine log files converting them into a format suitable for time series analysis using Grafana.

• Host: GitHub
• URL: https://github.com/pat8901/hpc-data-parser
• Owner: pat8901
• License: mit
• Created: 2024-04-19T19:05:23.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-05-08T15:39:37.000Z (almost 2 years ago)
• Last Synced: 2025-05-21T13:59:34.000Z (9 months ago)
• Topics: grafana, large-data, sqlite
• Language: Java
• Homepage:
• Size: 85 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.txt
  • License: License.txt

Awesome Lists containing this project

README

          
Grid Engine Log Analyzer

Overview:

This program is designed to analyze Grid Engine log files and convert them into a format suitable for time series analysis using Grafana. 

It consists of a Java program for log file parsing and CSV creation, followed by a bash script to convert the CSV into a SQLite database.

Features:

    Reads Grid Engine log files and extracts relevant data.

    Converts log file data into a CSV format.

    Uses a bash script to create a SQLite database from the CSV file.

    Generates binned data tables for use in time series analysis.

Installation: (This program only runs on Linux systems. Only been tested on Ubuntu)

    Ensure Java (JDK 11 or higher) is installed

    Clone the repository: git clone https://github.com/pat8901/hpc-data-parser.git

    Compile the Java program using the gradle wrapper: ./gradlew build

    Ensure SQLite is installed on your system for the bash script to work.

Usage:

    Ensure that ge-log-analyzer-1.0.0.jar, make_db.sh, and ge_log_file are all in the same directory. (This limitation will be fixed in a later version release)

    For example:

        john@john-pc:~/Desktop/grid_engine_program$ ls

        ge-log-analyzer-1.0.0.jar  make_db.sh  source_log.txt

    Run the Java program: java -jar ge-log-analyzer-1.0.0.jar

    Enter the path to the Grid Engine log file when prompted.

    The program will create a CSV file with the reformatted data. 

    Using the newly created CSV file the program will automatically call the bash script to create a SQLite database with binned data tables.

Requirements:

    Java (JDK 11 or higher)

    SQLite

    Access to a Grafana server for visualization (optional)

Example: (Bash)

$ java -jar ge-log-analyzer-1.0.0

Enter the path to the Grid Engine log file: /path/to/grid_engine.log

License:

This project is licensed under the MIT License - see the LICENSE file for details.