{"id":26279563,"url":"https://github.com/hokushin118/cba-devops","last_synced_at":"2026-05-12T04:33:48.206Z","repository":{"id":280785695,"uuid":"943160437","full_name":"hokushin118/cba-devops","owner":"hokushin118","description":"This repository provides the infrastructure and configuration for the CBA project's cloud-native microservices.","archived":false,"fork":false,"pushed_at":"2025-03-12T18:15:07.000Z","size":68,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-12T19:25:24.153Z","etag":null,"topics":["docker","docker-compose","k8s","openshift","tekton"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hokushin118.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-05T09:08:53.000Z","updated_at":"2025-03-12T18:15:06.000Z","dependencies_parsed_at":"2025-03-05T10:23:50.376Z","dependency_job_id":"5942f69a-fcf3-4524-9c7a-ca8536774840","html_url":"https://github.com/hokushin118/cba-devops","commit_stats":null,"previous_names":["hokushin118/cba-devops"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hokushin118%2Fcba-devops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hokushin118%2Fcba-devops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hokushin118%2Fcba-devops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hokushin118%2Fcba-devops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hokushin118","download_url":"https://codeload.github.com/hokushin118/cba-devops/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243589329,"owners_count":20315471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","k8s","openshift","tekton"],"created_at":"2025-03-14T14:15:26.747Z","updated_at":"2026-05-12T04:33:48.195Z","avatar_url":"https://github.com/hokushin118.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DevOps Configuration\n\nThis repository provides the infrastructure and configuration for the CBA\nproject's cloud-native microservices.\n\n## Environment Profiles\n\nThis project utilizes distinct environment profiles to manage infrastructure\nand application configurations for different stages of development and\ndeployment.\n\n**Available Profiles:**\n\n* **development (default):**\n    * Used for local development and testing.\n    * Provides a development-friendly configuration.\n* **docker:**\n    * Used for running microservices within Docker containers.\n    * Configures the application for a containerized environment.\n* **production:**\n    * Used for deploying microservices in a production environment.\n    * Optimized for performance and security.\n\n**Profile-Specific Environment Variables:**\n\nEnvironment variables specific to each profile are defined in `.env.\u003cprofile\u003e`\nfiles located in the project's root directory.\n\n* `.env`: Default environment variables (development profile).\n* `.env.docker`: Environment variables for the Docker profile.\n* `.env.production`: Environment variables for the production profile.\n\n**Important Security Note (Production):**\n\n* The `.env.production` file may contain sensitive information (e.g., database\n  credentials, API keys). **Never commit this file to a version control\n  repository.** Use secure methods for deploying production secrets, such as\n  environment variables managed by your deployment platform or dedicated secret\n  management tools.\n\n**Setting the Active Profile:**\n\nThe active profile is determined by the `APP_SETTINGS` environment variable.\n\n* Example (setting the Docker profile):\n    ```bash\n    export APP_SETTINGS=docker\n    ```\n\n## Local Development\n\nTo run the microservices locally for development and testing, follow these\nsteps:\n\n1. **Clone the Repository:**\n    * Clone the `cba-devops` repository to your local machine.\n   ```bash\n   git clone [https://github.com/hokushin118/cba-devops.git](https://github.com/hokushin118/cba-devops.git)\n   ```\n\n2. **Navigate to the Project Directory:**\n    * Change your current directory to the cloned repository.\n   ```bash\n   cd cba-devops\n   ```\n\n3. **Start Infrastructure Services with Docker Compose:**\n    * Use Docker Compose to launch the necessary infrastructure services (e.g.,\n      databases, message queues, Keycloak).\n    * This command starts the services defined in both `docker-compose.yml`\n      and `docker-compose.\u003cprofile\u003e.yml`.\n   ```bash\n   docker compose -f docker-compose.yml -f docker-compose.\u003cprofile\u003e.yml up --build\n   ```\n    * Example (test profile):\n   ```bash\n   docker compose -f docker-compose.yml -f docker-compose.test.yml up --build\n   ```\n    * The `--build` flag will build any images that are not already built.\n\n4. **Run the Microservices:**\n    * Once the infrastructure services are running, you can start the\n      individual microservices. Refer to the specific microservice's\n      documentation for instructions on how to run it locally (e.g.,\n      using `python app.py` or similar).\n\n**Important Notes:**\n\n* Ensure Docker and Docker Compose are installed on your machine.\n* `docker-compose.\u003cprofile\u003e.yml` (e.g., `docker-compose.test.yml`) includes\n  services required for the specified profile, such as test databases or mock\n  services.\n* Refer to the individual microservice's README for specific setup instructions\n  and dependencies.\n\n**Stopping Services:**\n\n* To stop the services, press **Ctrl+C** in the terminal where they are\n  running, or execute:\n    ```bash\n    docker compose -f docker-compose.yml -f docker-compose.\u003cprofile\u003e.yml down\n    ```\n* Example (test profile):\n    ```bash\n    docker compose -f docker-compose.yml -f docker-compose.test.yml down\n    ```\n* To remove volumes as well, add the `-v` flag:\n    ```bash\n    docker compose -f docker-compose.yml -f docker-compose.test.yml down -v\n    ```\n\n**Extending Docker Compose Configuration:**\n\n* For details on extending Docker Compose configurations,\n  see: [Extend your Compose files](https://docs.docker.com/compose/how-tos/multiple-compose-files/extends)\n\n## Environment Variables\n\nEnvironment variables for each profile are configured in the `.env` files.\n\n## Deployment on Kubernetes\n\nThis guide outlines the steps to deploy microservices on\na [Kubernetes](https://kubernetes.io) cluster.\n\n**Configuration Files:**\n\n* [Kubernetes](https://kubernetes.io) configuration files are located in\n  the `k8s/` directory.\n\n**Deployment Steps:**\n\nTo deploy a microservice to [Kubernetes](https://kubernetes.io), use the\nfollowing commands:\n\n1. **Namespace**: Apply the namespace yaml first. All other resources will be\n   created within this namespace (**cba-dev**).\n\n```bash\nkubectl apply -f .k8s/namespaces/cba-dev-ns.yml\n```\n\n2. **ConfigMap**: Apply the microservice ConfigMap yaml next. The Deployment\n   depends on it.\n\n```bash\nkubectl apply -n cba-dev -f .k8s/configmaps/\u003cname of microservice\u003e-srv-cm.yml\n```\n\nFor example:\n\n```bash\nkubectl apply -n cba-dev -f .k8s/configmaps/account-srv-cm.yml\n```\n\n3. **Secret**: Apply the microservice Secret yaml. The Deployment also depends\n   on it.\n\n```bash\nkubectl apply -n cba-dev -f .k8s/secrets/\u003cname of microservice\u003e-srv-secret.yml\n```\n\nFor example:\n\n```bash\nkubectl apply -n cba-dev -f .k8s/secrets/account-srv-secret.yml\n```\n\n4. **Deployment**: Apply the microservice Deployment yaml last. It depends on\n   the Namespace, ConfigMap, and Secret.\n\n```bash\nkubectl apply -n cba-dev -f .k8s/deployments/\u003cname of microservice\u003e-srv-deployment.yml\n```\n\nFor example:\n\n```bash\nkubectl apply -n cba-dev -f .k8s/deployments/account-srv-deployment.yml\n```\n\n5. **Service**: Apply the microservice Service yaml. It depends on the Pods\n   created by the Deployment.\n\n```bash\nkubectl apply -n cba-dev -f .k8s/services/\u003cname of microservice\u003e-srv-service.yml\n```\n\nFor example:\n\n```bash\nkubectl apply -n cba-dev -f .k8s/services/account-srv-service.yml\n```\n\n6. **HPA**: Apply the microservice HPA yaml. It depends on the Deployment.\n\n```bash\nkubectl apply -n cba-dev -f .k8s/hpas/\u003cname of microservice\u003e-srv-hpa.yml\n```\n\nFor example:\n\n```bash\nkubectl apply -n cba-dev -f .k8s/hpas/account-srv-hpa.yml\n```\n\n**Verification:**\n\nCheck the pods are running with:\n\n```bash\nkubectl get pods -n cba-dev\n```\n\nCheck the Service is created with:\n\n```bash\nkubectl get services -n cba-dev\n```\n\nCheck the HPA is created with:\n\n```bash\nkubectl get hpa -n cba-dev\n```\n\n## Deployment on Red Hat OpenShift with Tekton\n\nThis guide outlines the steps to deploy your application\non [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift)\nusing\n[Tekton](https://tekton.dev) Pipelines.\n\n[Tekton](https://tekton.dev) is a cloud-native solution for building CI/CD\nsystems. It consists of [Tekton](https://tekton.dev) Pipelines, which provides\nthe building blocks, and of supporting components, such as Tekton CLI and\nTekton Catalog, that make Tekton a complete ecosystem. For more information,\nsee the [Tekton documentation](https://tekton.dev/docs/).\n\n**Prerequisites:**\n\n* **Red Hat OpenShift Cluster:** You need access to\n  a [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift)\n  cluster.\n* **Red Hat OpenShift CLI (oc):** Install the OpenShift CLI. Download it\n  from [Red Hat Downloads](https://access.redhat.com/downloads/content/290/ver=4.17/rhel---9/4.17.16/x86_64/product-software).\n    * Verify installation: `oc version`\n* **Tekton CLI (tkn):** Install the Tekton CLI. Download it\n  from [Tekton CLI Releases](https://github.com/tektoncd/cli/releases).\n    * Example (RHEL 9):\n        ```bash\n        rpm -Uvh [https://github.com/tektoncd/cli/releases/download/v0.37.1/tektoncd-cli-0.37.1_Linux-64bit.rpm](https://github.com/tektoncd/cli/releases/download/v0.37.1/tektoncd-cli-0.37.1_Linux-64bit.rpm)\n        ```\n      or\n        ```bash\n        curl -LO [https://github.com/tektoncd/cli/releases/download/v0.37.1/tkn_0.37.1_Linux_x86_64.tar.gz](https://github.com/tektoncd/cli/releases/download/v0.37.1/tkn_0.37.1_Linux_x86_64.tar.gz)\n        sudo tar xvzf tkn_0.37.1_Linux_x86_64.tar.gz -C /usr/local/bin/ tkn\n        ```\n    * Verify installation: `tkn version`\n* **Tekton Pipelines:** Tekton Pipelines must be installed on your [Red Hat\n  OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift)\n  cluster.\n\n**Configuration Files:**\n\n* [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift)\n  configuration files: `openshift/` directory.\n* Tekton custom Task files: `openshift/tekton/tasks/` directory.\n* Tekton Pipeline files: `openshift/tekton/pipelines/` directory.\n\n**Setup Steps:**\n\n1. Login to the [Red Hat OpenShift](https://www.redhat.\n   com/en/technologies/cloud-computing/openshift) cluster, using the following\n   command:\n\n```bash\noc login --token=\u003ctoken\u003e --server=https://api.,,.p1.openshiftapps.com:6443\n```\n\n2. Install [Tekton Pipeline](https://github.com/tektoncd/pipeline/releases)\n   on [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift)\n   using the following command:\n\n```bash\noc apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.68.0/release.yaml\n```\n\n3. [Download](https://github.com/tektoncd/cli/releases) and install [Tekton\n   CLI](https://tekton.dev/docs/cli) on your machine. For example, to download\n   and install **Tekton CLI** on **RHEL 9**, use the following commands:\n\n```bash\nrpm -Uvh https://github.com/tektoncd/cli/releases/download/v0.37.1/tektoncd-cli-0.37.1_Linux-64bit.rpm\n```\n\nor\n\n```bash\ncurl -LO https://github.com/tektoncd/cli/releases/download/v0.37.1/tkn_0.37.1_Linux_x86_64.tar.gz\nsudo tar xvzf tkn_0.37.1_Linux_x86_64.tar.gz -C /usr/local/bin/ tkn\n```\n\nAfter installation (using either method), verify that Tekton is installed\ncorrectly, using the following command:\n\n```bash\ntkn version\n```\n\n4. To create a workspace for pipeline\n   on [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift),\n   use the following\n   commands:\n\n```bash\noc create -f .openshift/cba-pipeline-pvc.yml\n```\n\n5. Apply the Secret yaml.\n\n```bash\noc apply -f .openshift/cba-pipeline-secret.yml\n```\n\n6. To create custom [Tekton](https://tekton.dev) tasks for pipeline\n   on [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift),\n   use the following commands:\n\n```bash\noc apply -f .openshift/tekton/tasks/run-cleanup-workspace.yml \noc apply -f .openshift/tekton/tasks/run-flake8-lint.yml \noc apply -f .openshift/tekton/tasks/run-nose-tests.yml \noc apply -f .openshift/tekton/tasks/run-trivy-scan.yml \noc apply -f .openshift/tekton/tasks/run-database-migration.yml \noc apply -f .openshift/tekton/tasks/run-revert-database-migration.yml \n```\n\nApply the run-github-clone-w-token.yml if you are using a private repository.\n\n```bash\noc apply -f .openshift/tekton/tasks/run-github-clone-w-token.yml \n```\n\nTo verify the created custom tasks, using the following command:\n\n```bash\noc get tasks\n```\n\n7. The **clone** task requires the **git-clone**, the **build** task\n   requires **buildah** and the **deploy** task requires the\n   \"\"openshift-client\"\" tasks from the **Tekton Hub**, use the following\n   commands to install them:\n\n```bash\ntkn hub install task git-clone\ntkn hub install task buildah\ntkn hub install task openshift-client\n```\n\nMake sure that the **git-clone**, **buildah** and **openshift-client** tasks\nare available in\nthe [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift)\nusing the following command:\n\n```bash\noc get clustertask\n```\n\n8. To create [Tekton](https://tekton.dev) pipelines\n   on [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift),\n   use\n   the following commands:\n\n```bash\noc apply -f .openshift/tekton/pipelines/cba-pipeline.yml\noc apply -f .openshift/tekton/pipelines/cba-db-migration-revert-pipeline.yml\n```\n\n**Running Pipelines:**\n\n1. To start CI/CD pipeline\n   on [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift),\n   use the following command:\n\n```bash\ntkn pipeline start cba-pipeline \\\n            -p repo-url=\u003cGITHUB_REPO_URL\u003e \\\n            -p branch=\u003cBRANCH\u003e \\\n            -p build-image=\u003cDOCKER_IMAGE\u003e \\\n            -p deploy-enabled=\u003cDEPLOY_ENABLED\u003e \\\n            -w name=\u003cWORKSPAVCE_NAME\u003e,claimName=\u003cPVC_CLAIM_NAME\u003e \\\n            -s pipeline \\\n            --showlog\n```\n\n- **GITHUB_REPO_URL** - URL of the GitHub repository\n- **BRANCH** - name of the branch\n- **DOCKER_IMAGE** - docker image with tag and registry, for example: `quay.\n  io/username/cba:latest`\n- **DEPLOY_ENABLED** - Enable/disable deployment step, for example: `true`\n- **WORKSPACE_NAME** - name of the workspace specified in the pipeline yaml\n  file, for example: `cba-pipeline-pvc`\n- **PVC_CLAIM_NAME** - name of the PVC claim created for pipeline, for\n  example: `cba-pipeline-pvc`\n\nFor example (Account microservice):\n\n```bash\ntkn pipeline start cba-pipeline \\\n            -p repo-url=\"https://github.com/hokushin118/account-service.git\" \\\n            -p branch=\"main\" \\\n            -p build-image=hokushin/account-service:latest \\\n            -p deploy-enabled=false \\\n            -w name=cba-pipeline-workspace,claimName=cba-pipeline-pvc \\\n            -s pipeline \\\n            --showlog\n```\n\n2. To start the database migration revert pipeline\n   on [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift),\n   use the following command:\n\n```bash\ntkn pipeline start cba-pipeline \\\n            -p repo-url=\u003cGITHUB_REPO_URL\u003e \\\n            -p branch=\u003cBRANCH\u003e \\\n            -p revision=\u003cREVISION\u003e \\\n            -w name=\u003cWORKSPAVCE_NAME\u003e,claimName=\u003cPVC_CLAIM_NAME\u003e \\\n            -s pipeline \\\n            --showlog\n```\n\n- **GITHUB_REPO_URL** - URL of the GitHub repository\n- **BRANCH** - name of the branch\n- **REVISION** - database migration revision id, for example: `1234abcd'\n- **WORKSPACE_NAME** - name of the workspace specified in the pipeline yaml\n  file, for example: `cba-pipeline-pvc`\n- **PVC_CLAIM_NAME** - name of the PVC claim created for pipeline, for\n  example: `cba-pipeline-pvc`\n\nFor example (Account microservice):\n\n```bash\ntkn pipeline start cba-pipeline \\\n            -p repo-url=\"https://github.com/hokushin118/account-service.git\" \\\n            -p branch=\"main\" \\\n            -p revision=\"1234abcd\" \\\n            -w name=cba-pipeline-workspace,claimName=cba-pipeline-pvc \\\n            -s pipeline \\\n            --showlog\n```\n\nTo make the pipeline ran successfully, run the following command:\n\n```bash\ntkn pipelinerun ls\n```\n\nYou can check the logs of the last pipeline run with:\n\n```bash\ntkn pipelinerun logs --last\n```\n\n## Keycloak Identity and Access Management (IAM)\n\nCBA microservices leverage [Keycloak](https://www.keycloak.org) for robust\nauthentication and authorization.\n\n**Authentication and Authorization:**\n\n* Microservices utilize [JWT](http://www.jwt.io) tokens for authentication and\n  authorization.\n* These tokens are issued by the Keycloak IAM server.\n* Microservices validate the received [JWT](http://www.jwt.io) tokens.\n* The [JWT](http://www.jwt.io) tokens are signed using the RSA256 algorithm.\n\n**Realm Configuration:**\n\n* The configuration for the `cba-dev` realm is stored in\n  the `.infrastructure/keycloak/cba-dev-realm.json` file.\n* This configuration is automatically imported when Keycloak is deployed using\n  Docker Compose.\n\n**Realm Users (Development/Testing):**\n\nThe following test users are available in the development/testing realm\n**cba-dev**.\n\n| **user name** | **password** | **roles**                     |\n|---------------|--------------|-------------------------------|\n| admin         | admin        | __ROLE_ADMIN__, __ROLE_USER__ |\n| test          | test         | __ROLE_USER__                 |\n\n**Important Note:** The `admin` user listed above is a test user within the\ndevelopment/testing realm **cba-dev**. It is distinct from the `admin`\nsuperuser account associated with the **master** realm.\n\n**Realm Roles (Development/Testing):**\n\n* For development and testing purposes, the following realm roles are added:\n    * `ROLE_USER`: Represents a standard user role.\n    * `ROLE_ADMIN`: Represents an administrator role with elevated privileges.\n\n**Accessing Keycloak:**\n\n* **OpenID Configuration URL:\n  ** [http://localhost:28080/realms/cba-dev/.well-known/openid-configuration](http://localhost:28080/realms/cba-dev/.well-known/openid-configuration) (\n  Provides the OpenID Connect discovery document.)\n* **Admin Console URL:** [http://localhost:28080](http://localhost:28080) (\n  Provides a web interface for Keycloak administration.)\n\n**Important Security Note:**\n\n* **Default Admin Credentials:** The default administrator credentials (\n  username: `admin`, password: `admin`) are provided for development purposes\n  only. **Do not use these credentials in a production environment.**\n* **Change Default Credentials:** Immediately change the default admin password\n  after initial setup.\n\n**Admin Console Credentials (Development Only):**\n\n| **user name** | **password** |\n|---------------|--------------|\n| admin         | admin        |\n\n## Prometheus Monitoring\n\nCBA microservices are monitored using [Prometheus](https://prometheus.io), a\npowerful open-source monitoring and alerting toolkit.\n\n**Configuration:**\n\nThe [Prometheus](https://prometheus.io) configuration is located\nin `.infrastructure/prometheus/prometheus.yml`. This file defines the metrics\nscraping targets and other settings.\n\n**Accessing Prometheus:**\n\nYou can access the [Prometheus](https://prometheus.io) web interface and\nendpoints at the following URLs:\n\n* **Prometheus Web UI:** [http://localhost:19090](http://localhost:19090) (\n  Provides a graphical interface for querying and visualizing metrics.)\n* **Metrics Endpoint:\n  ** [http://localhost:19090/metrics](http://localhost:19090/metrics) (Displays\n  the raw metrics data scraped by [Prometheus](https://prometheus.io).)\n* **Targets Endpoint:\n  ** [http://localhost:19090/targets](http://localhost:19090/targets) (Shows\n  the status of the configured scraping targets.)\n\n**Key Features:**\n\n* [Prometheus](https://prometheus.io) allows you to monitor the health and\n  performance of our microservices in real-time.\n* You can use\n  the [Prometheus Query Language (PromQL)](https://prometheus.io/docs/prometheus/latest/querying/basics/)\n  to create custom queries and dashboards.\n* Alerting rules can be configured to notify you of critical issues.\n\n**Note:** Ensure [Prometheus](https://prometheus.io) is running (e.g., via\nDocker Compose) to access these endpoints.\n\n## Grafana Dashboards\n\nCBA microservices utilize [Grafana](https://grafana.com) to create and manage\ndashboards, visualizing metrics collected\nby [Prometheus](https://prometheus.io) from our microservices.\n\n**Purpose:**\n\n[Grafana](https://grafana.com) dashboards provide real-time, visual\nrepresentations of key performance indicators (KPIs) and application health\nmetrics. They empower teams to:\n\n* **Monitor Application Performance:** Track request rates, response times,\n  error rates, and resource utilization.\n* **Identify Anomalies:** Quickly detect unexpected behavior and potential\n  issues.\n* **Analyze Trends:** Gain insights into application performance over time.\n* **Troubleshoot Issues:** Drill down into specific metrics to diagnose\n  problems effectively.\n* **Improve Visibility:** Offer a centralized view of application health for\n  development, operations, and business teams.\n\n**Configuration:**\n\n[Grafana](https://grafana.com) configurations are managed via Docker Compose\nand provisioning files, ensuring reproducible and version-controlled setups.\n\n**Accessing Grafana:**\n\nThe [Grafana](https://grafana.com) web interface is available at:\n\n* [http://localhost:3000](http://localhost:3000)\n  (Provides a graphical interface for querying and visualizing metrics.)\n* Upon the initial launch, you will be prompted to enter your credentials.\n  Use **admin** for both the username and password.\n* You will then be prompted to change your initial login credentials.\n\n**Key Configuration Aspects:**\n\n* **Data Source Provisioning:**\n    * [Prometheus](https://prometheus.io) is configured as a data source using\n      YAML provisioning files (\n      e.g., `.infrastructure/metrics/grafana/provisioning/datasources/prometheus_ds.yaml`).\n    * This ensures [Grafana](https://grafana.com) automatically connects to the\n      correct [Prometheus](https://prometheus.io) instance upon startup.\n    * The configuration specifies the Prometheus URL, access method (proxy or\n      direct), and other relevant settings.\n* **Dashboard Provisioning:**\n    * [Grafana](https://grafana.com) dashboards are defined in JSON format and\n      provisioned using YAML files (e.g., `all.yml`) located in\n      the `.infrastructure/metrics/grafana/provisioning/dashboards` directory.\n    * This enables version control and automated deployment of dashboards.\n    * Dashboards visualize key metrics, including:\n        * Request rates and response times.\n        * HTTP status code distributions.\n        * Resource utilization (CPU, memory).\n        * Custom application metrics.\n* **Dynamic Data Source Linking:**\n    * In containerized environments (Docker Compose),\n      the [Prometheus](https://prometheus.io) data source UID is dynamically\n      injected into [Grafana](https://grafana.com) dashboard configurations\n      using environment variables.\n    * This ensures seamless connection to\n      the [Prometheus](https://prometheus.io) instance without manual\n      intervention.\n* **Environment Variables:**\n    * [Grafana](https://grafana.com) admin credentials are set via environment\n      variables: `GF_ADMIN_USER` and `GF_ADMIN_PASSWORD`.\n* **Docker Compose:**\n    * [Grafana](https://grafana.com) runs as a Docker container, simplifying\n      deployment and management.\n    * Docker Compose links Grafana to the [Prometheus](https://prometheus.io)\n      service, enabling seamless communication.\n    * Volumes are used for persistent storage of Grafana data and provisioning\n      files.\n\n**Available Dashboards:**\n\n* **Account Monitoring Dashboard:**\n    * [Account](https://github.com/hokushin118/account-service) microservice\n      Grafana dashboard.\n    * Located in\n      the `.infrastructure/metrics/grafana/provisioning/dashboards/account-monitoring.json`\n      file.\n    * Focuses on real-time monitoring (3-second refresh, 5-minute time range).\n    * Covers essential metrics like request rates, error rates, response times,\n      and resource usage.\n    * Utilizes common Prometheus queries for Flask applications.\n    * Monitors performance of successful requests and errors.\n    * Uses percentiles to monitor request latency.\n    * Monitors CPU and memory usage.\n    * Displays:\n        * Rate of successful (HTTP 200) requests per second.\n        * Rate of error requests (non-200 HTTP status codes) per second.\n        * Total requests per minute, broken down by HTTP status code.\n        * Average response time (seconds) for successful requests (HTTP 200)\n          over 30 seconds.\n        * 50th percentile (median) request duration (seconds) for successful\n          requests (HTTP 200) over 30 seconds.\n        * Resident memory usage (bytes) of the Flask application process.\n        * Percentage of successful requests (HTTP 200) with response times\n          under 250 milliseconds over 30 seconds.\n        * 90th percentile request duration (seconds) for successful requests (\n          HTTP 200) over 30 seconds.\n        * CPU usage of the Flask application process.\n\n## Audit with Kafka\n\nCBA microservices utilize [Apache Kafka](https://kafka.apache.org) for robust,\nscalable, and asynchronous audit logging. This architecture facilitates\ncentralized logging, real-time monitoring, and efficient data analysis, crucial\nfor debugging, security, and compliance. This setup is optimized for\ndevelopment, testing, and local experimentation.\n\n**Key Features:**\n\n* **Asynchronous Logging:** [Apache Kafka](https://kafka.apache.org) enables\n  non-blocking audit logging, minimizing performance impact on core\n  microservices.\n* **Centralized Logging:** Consolidated audit logs\n  in [Apache Kafka](https://kafka.apache.org) simplify analysis and correlation\n  across services.\n* **Scalability:** [Apache Kafka](https://kafka.apache.org)'s distributed\n  nature allows for handling increasing audit log volumes.\n* **Real-time Monitoring:** Kafka Streams or similar tools can be used for\n  real-time analysis of audit data.\n\n**Prerequisites:**\n\n* **Docker:** Ensure Docker is installed and running on your system.\n* **Docker Compose:** Ensure Docker Compose is installed.\n* **Environment Variables:** Configure necessary environment variables via\n  a `.env` file or directly in your shell. See the\n  example `docker-compose-kafka.yml` for required variables.\n\n**Launching the Kafka Audit Environment:**\n\n1. **Start Kafka Audit Services:**\n    * Open your terminal and navigate to the directory\n      containing `docker-compose-kafka.yml`.\n    * Execute the following command to start\n      the [Apache Zookeeper](https://zookeeper.apache.org)\n      and [Apache Kafka](https://kafka.apache.org) containers:\n\n        ```bash\n        docker compose -f docker-compose-kafka.yml up -d\n        ```\n\n        * `-f docker-compose-kafka.yml`: Specifies the Docker Compose file to\n          use.\n        * `up`: Starts the containers.\n        * `-d`: Runs the containers in detached mode (background).\n\n2. **Verify Service Status:**\n    * Confirm the services are running correctly:\n\n        ```bash\n        docker ps\n        ```\n\n        * This command lists running Docker containers. You should see the\n          Zookeeper and Kafka containers.\n\n3. **View Container Logs:**\n    * To view the logs of a specific container (e.g., Kafka):\n\n        ```bash\n        docker logs kafka-1\n        ```\n\n        * Replace `kafka-1` with the actual container name.\n\n**Accessing Kafka:**\n\n* **From within the Kafka container:** Use `localhost:9093`\n  for `kafka-console-consumer.sh` and related commands.\n* **From other Docker containers on the same network:** Use `kafka-1:9092` (or\n  the appropriate internal port).\n\n**Interacting with Kafka:**\n\n* **Using `kafka-console-consumer.sh` (within the Kafka container):**\n\n    ```bash\n    docker exec -it kafka-1 /opt/bitnami/kafka/bin/kafka-console-consumer.sh \\\n      --bootstrap-server localhost:9093 --topic audit-events --from-beginning\n    ```\n\n**Important Notes:**\n\n* **Environment Variables:** Use a `.env` file for managing environment\n  variables.\n* **Production:** This setup is for development. For production environments,\n  consider a multi-node Kafka cluster and proper security configurations.\n\n# MongoDB for Local Development (Image Metadata Storage)\n\nCBA microservices utilize [MongoDB](https://www.mongodb.com) for storing image\nmetadata within the CBA microservice\narchitecture. [MongoDB](https://www.mongodb.com) provides a\nflexible and scalable NoSQL database solution, well-suited for potentially\nevolving metadata structures. This document outlines how to set up MongoDB\nlocally for development and testing using Docker Compose.\n\n**Key Features of this Setup:**\n\n* **Flexible Document Model:** MongoDB's BSON document format is ideal for\n  storing varied and potentially nested image metadata without requiring a\n  rigid upfront schema.\n* **Persistent Storage:** Uses a Docker named volume (mongodb-data) to persist\n  your metadata even when the container is stopped and restarted.\n* **Simplified Local Setup:** Provides a\n  single-node [MongoDB](https://www.mongodb.com) instance suitable\n  for local development, testing, and experimentation via Docker Compose.\n* **Configurable:** Leverages environment variables (via a .env file) for\n  database initialization details (user, password, database name).\n\n**Prerequisites:**\n\n* **Docker:** Ensure Docker is installed and running on your system.\n* **Docker Compose:** Ensure Docker Compose is installed.\n* **Environment Variables:** Configure necessary environment variables via\n  a `.env` file or directly in your shell. See the\n  example `docker-compose-mongo.yml` for required variables.\n\n**Launching the MongoDB Environment:**\n\n1. **Start MongoDB Services:**\n    * Open your terminal and navigate to the directory\n      containing `docker-compose-mongo.yml`.\n    * Execute the following command to start\n      the MongoDB container:\n\n        ```bash\n        docker compose -f docker-compose-mongo.yml up -d\n        ```\n\n        * `-f docker-compose-mongo.yml`: Specifies the Docker Compose file to\n          use.\n        * `up`: Starts the containers.\n        * `-d`: Runs the containers in detached mode (background).\n\n2. **Verify Service Status:**\n    * Confirm the services are running correctly:\n\n        ```bash\n        docker compose -f docker-compose-mongo.yml ps\n        ```\n\n        * This command lists running Docker containers. You should\n          see the MongoDB container.\n\n3. **View Container Logs:**\n    * To view the logs of a specific container (e.g., MongoDB):\n\n        ```bash\n        docker compose -f docker-compose-mongo.yml logs mongo\n        ```\n\n**Connecting to MongoDB:**\n\n* **From Your Microservice (Running in Docker on the same network):**\n\n  The microservice container (defined in a separate docker-compose.yml or\n  started manually on the same network) should connect using the service name\n  defined in docker-compose-mongo.yml (which is mongo by default) and the\n  internal MongoDB port (27017).\n\n  Example Connection String (adjust user/pass/db):\n\n  ```\n  mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@mongo:\n  27017/${MONGO_INITDB_DATABASE}?authSource=admin\n  ```\n\n* **From Your Host Machine (e.g., MongoDB Compass, Studio 3T, local script):**\n\n  Connect using localhost (or 127.0.0.1) and the mapped host port specified in\n  the ports section of docker-compose-mongo.yml (e.g., 27017 if you used 27017:\n  27017).\n\n  Use the same root username and password from your .env file.\n  Authentication Database should typically be admin.\n\n  Example Connection String:\n\n  ```\n  mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@localhost:\n  27017/${MONGO_INITDB_DATABASE}?authSource=admin\n   ```\n\n  **Interacting with MongoDB:**\n\n  You can directly interact with MongoDB inside the running container using\n  the MongoDB Shell (mongosh):\n\n    ```bash\n    docker compose -f docker-compose-mongo.yml exec mongo mongosh \\\n      --username ${MONGO_INITDB_ROOT_USERNAME} \\\n      --password ${MONGO_INITDB_ROOT_PASSWORD} \\\n      --authenticationDatabase admin \\\n      ${MONGO_INITDB_DATABASE}\n    ```\n\n  This opens an interactive shell connected to your specified database. You can\n  run commands like db.collectionName.find(), show collections, etc.\n\n* **Important Notes:**\n\n* **Development Only:** This single-node setup is intended for local\n  development and testing. It lacks the high availability and fault tolerance\n  of a production MongoDB replica set or Atlas cluster.\n* **Security:** The root user/password provides full admin access. Be mindful\n  of the security implications, especially if mapping the port publicly (which\n  is discouraged). Do not use default or weak passwords. Ensure your\n  production .env file is ignored by Git.\n* **Data Persistence:** Data is stored in the mongodb-data Docker volume on\n  your host machine. To completely reset the database, you need to stop the\n  container (docker compose -f ... down), remove the volume (docker volume rm \u003c\n  volume_name\u003e), and then start it again (docker compose -f ... up -d).\n\n# MinIO for Local Object Storage (S3 Compatible)\n\nCBA microservices utilize [MinIO](https://min.io) to provide\nan [S3-compatible](https://en.wikipedia.org/wiki/Amazon_S3)\nobject storage service locally via Docker Compose. This setup simulates\nmultiple drives using Docker volumes, allowing you to test applications that\ninteract with S3 APIs and potentially observe [MinIO](https://min.io)'s erasure\ncoding behavior in a development environment.\n\n**Key Features of this Setup:**\n\n* **S3 Compatibility:** Provides a standard S3 API\n  endpoint (http://localhost:9000) for CBA microservices or tools.\n* **Web Console:** Includes the [MinIO](https://min.io) Console web\n  UI (http://localhost:9001) for easy bucket and object management.\n* **Local Erasure Coding Simulation:** Uses multiple Docker volumes (/data1 to\n  /data4) to mimic the multi-drive setup required for [MinIO](https://min.io)'s\n  default erasure coding, enabling testing of this feature locally.\n* **Persistent Storage::** Uses Docker named volumes (minio-data1, etc.) to\n  persist your buckets and objects even when the container is stopped and\n  restarted.\n* **Configurable:** Leverages environment variables (via a .env file) for root\n  credentials.\n\n**Prerequisites:**\n\n* **Docker:** Ensure Docker is installed and running on your system.\n* **Docker Compose:** Ensure Docker Compose is installed.\n* **Environment Variables:** A .env file is required in the same directory as\n  `docker-compose-minio.yml` (or your main compose file if combined).\n\n**Launching the MongoDB Environment:**\n\n1. **Start [MinIO](https://min.io) Service:**\n    * Open your terminal and navigate to the directory\n      containing `docker-compose-minio.yml`.\n    * Execute the following command to start\n      the [MinIO](https://min.io) container:\n\n        ```bash\n        docker compose -f docker-compose-minio.yml up -d\n        ```\n\n        * `-f docker-compose-minio.yml`: Specifies the Docker Compose file to\n          use.\n        * `up`: Starts the containers.\n        * `-d`: Runs the containers in detached mode (background).\n\n2. **Verify Service Status:**\n    * Confirm the services are running correctly:\n\n        ```bash\n        docker compose -f docker-compose-minio.yml ps\n        ```\n\n        * This command lists running Docker containers. You should\n          see the [MinIO](https://min.io) container.\n\n3. **View Container Logs:**\n    * To view the logs of a specific container (e.g., [MinIO](https://min.io)):\n\n        ```bash\n        docker compose -f docker-compose-minio.yml logs minio\n        ```\n\n**Accessing to [MinIO](https://min.io):**\n\n* **MinIO Console (Web UI):**\n\n    * Open your web browser and navigate to: http://localhost:9001.\n    * Log in using the **MINIO_ROOT_USER** and **MINIO_ROOT_PASSWORD** from\n      your .env file.\n\n* **S3 API Endpoint:**\n\n    * Configure your S3 clients (AWS CLI, SDKs like boto3, mc) to use:\n\n    - Endpoint URL: http://localhost:9000.\n    - Access Key ID: The value of **MINIO_ROOT_USER** from .env.\n    - Secret Access Key: The value of **MINIO_ROOT_PASSWORD** from .env.\n    - Region: Often optional for [MinIO](https://min.io).\n\n* **From Other Docker Containers (on the same network):**\n\n    * Use the service name (minio by default) and the internal API port (9000).\n    * Example Endpoint URL: http://minio:9000.\n\n* **Example AWS CLI Configuration:**\n\n    ```\n    aws configure --profile minio-local\n    AWS Access Key ID [None]: admin # MINIO_ROOT_USER\n    AWS Secret Access Key [None]: minio12345 # MINIO_ROOT_PASSWORD\n    Default region name [None]: # Leave blank\n    Default output format [None]: json\n    \n    # List buckets\n    aws --profile minio-local --endpoint-url http://localhost:9000 s3 ls\n    \n    # Create a bucket\n    aws --profile minio-local --endpoint-url http://localhost:9000 s3 mb s3://my-test-bucket\n    \n    # Upload a file\n    aws --profile minio-local --endpoint-url http://localhost:9000 s3 cp ./local-file.txt s3://my-test-bucket/\n    ```\n\n* **Important Notes:**\n\n* **Development Only:** This single-node setup is not suitable for production\n  High Availability. Production requires a distributed setup across multiple\n  servers.\n* **Security:** Use strong, unique credentials in your .env file and ensure\n  it's not committed to version control.\n* **Data Persistence:** Data is stored in the minio-dataX Docker volumes. To\n  completely reset MinIO storage, stop the container (docker compose -f ...\n  down), remove the volumes (docker volume rm minio-data1 minio-data2\n  minio-data3 minio-data4), and start again (docker compose -f ... up -d).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhokushin118%2Fcba-devops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhokushin118%2Fcba-devops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhokushin118%2Fcba-devops/lists"}