Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/cyse7125-su24-team10/webapp-cve-consumer
Go application that processes CVE data from Kafka streams and stores it in PostgreSQL. It features secure Kafka integration, efficient database handling, and health check endpoints. Dockerized and managed through a CI/CD pipeline with Jenkins, it ensures streamlined deployment and reliable CVE data monitoring.
https://github.com/cyse7125-su24-team10/webapp-cve-consumer
aws docker golang groovy jenkins semantic-release
Last synced: 3 days ago
JSON representation
Go application that processes CVE data from Kafka streams and stores it in PostgreSQL. It features secure Kafka integration, efficient database handling, and health check endpoints. Dockerized and managed through a CI/CD pipeline with Jenkins, it ensures streamlined deployment and reliable CVE data monitoring.
- Host: GitHub
- URL: https://github.com/cyse7125-su24-team10/webapp-cve-consumer
- Owner: cyse7125-su24-team10
- License: mit
- Created: 2024-06-24T08:33:28.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2024-09-09T21:14:42.000Z (18 days ago)
- Last Synced: 2024-09-13T07:01:12.330Z (15 days ago)
- Topics: aws, docker, golang, groovy, jenkins, semantic-release
- Language: Go
- Homepage:
- Size: 10.2 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Webapp CVE Consumer
[](https://aws.amazon.com/)
[](https://golang.org/)
[](https://www.jenkins.io/)
[](https://groovy-lang.org/)
[](https://www.docker.com/)
## Introduction
The cve-consumer application is designed to process CVE (Common Vulnerabilities and Exposures) data from a Kafka stream and insert it into a PostgreSQL database.
Key components of the application include:
1. **Kafka Integration**: Utilizes the Segmentio Kafka library to connect to Kafka brokers with SASL authentication. The application reads messages from the "cve" topic and processes them.
2. **Database Interaction**: Connects to a PostgreSQL database using Go's `database/sql` package. It inserts or updates CVE data in the `cve.CVE` table, handling conflicts by updating existing records.
3. **Health Checks**: Implements HTTP endpoints for health and liveliness checks. It verifies connectivity to Kafka brokers and the PostgreSQL database.
4. **Configuration**: Configured via environment variables loaded using the `godotenv` package, covering Kafka and PostgreSQL connection details.
5. **Dockerization**: Built using a multi-stage Dockerfile. The application is first compiled in a Golang image and then deployed in a lean scratch image.
6. **CI/CD**: Managed by Jenkins pipelines, with semantic versioning and release automation handled by Semantic Release. The CI/CD pipeline includes steps for cloning the repository, building and pushing Docker images, and validating commit messages.
This setup ensures robust data processing, streamlined deployment, and reliable monitoring of CVE data within the system.
## Prerequisites
- [Jenkins server](https://www.terraform.io/) should be up and running
- [Dockerhub account](https://hub.docker.com/) should be configured for the Jenkins job to upload docker images on and have a working personal access token
- Working personal access token for github repository
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed on your machine
- [Go](https://go.dev/doc/install) should be configured on local machine
- Configure the webhooks as mentioned in the groovy scripts
- Set up releaserc locally
## Setup Instructions
1. **Clone the Repository:**
```bash
git clone [email protected]:cyse7125-su24-team10/webapp-cve-consumer.git
cd webapp-cve-consumer
```
2. **To build and test a docker image of this application locally**
```bash
docker buildx rm newbuilderx || true
docker buildx create --use --name newbuilderx --driver docker-container
docker buildx build --file Dockerfile --platform linux/amd64,linux/arm64 -t ${imageName}:${localTag} --load .
docker buildx rm newbuilderx
```
3. **To push to a docker registry**
```bash
docker buildx rm newbuilderx || true
docker buildx create --use --name newbuilderx --driver docker-container
docker buildx build --file Dockerfile --platform linux/amd64,linux/arm64 -t ${registry}:${latestTag} --push .
docker buildx rm newbuilderx
```
4. **Any changes to the repository is configured to trigger a jenkins job to create releases and also upload an updated docker image to the configured docker registry**