{"id":30736179,"url":"https://github.com/backbase/audit-export-job","last_synced_at":"2025-09-03T20:54:52.913Z","repository":{"id":287316157,"uuid":"964327441","full_name":"Backbase/audit-export-job","owner":"Backbase","description":"Audit export job to upload data from audit tables into private content services","archived":false,"fork":false,"pushed_at":"2025-04-15T13:45:57.000Z","size":471,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-08-29T11:56:41.121Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Backbase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":"audit-export-job/CHANGELOG.md","citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-11T03:29:53.000Z","updated_at":"2025-04-15T13:46:00.000Z","dependencies_parsed_at":"2025-04-11T04:39:47.643Z","dependency_job_id":null,"html_url":"https://github.com/Backbase/audit-export-job","commit_stats":null,"previous_names":["backbase/audit-export-job"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Backbase/audit-export-job","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Backbase%2Faudit-export-job","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Backbase%2Faudit-export-job/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Backbase%2Faudit-export-job/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Backbase%2Faudit-export-job/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Backbase","download_url":"https://codeload.github.com/Backbase/audit-export-job/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Backbase%2Faudit-export-job/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273509770,"owners_count":25118448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-03T20:54:50.352Z","updated_at":"2025-09-03T20:54:52.897Z","avatar_url":"https://github.com/Backbase.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Audit-export-job\n\n\n## Problem statement\nDue to high audit data volumes in PROD, the audit-services is slowing down which also causes impact to the Message Queue (Azure Service Bus, AWS SQS, Kafka, ActiveMQ..)\n### Requirements\n\n**One time tasks**\n\nTo enhance audit-service performance, we need to achieve legacy audit data (before a specific date) in Audit database by uploading into Azure Blob\nManually cleanup legacy audit data (before a specific date) from Audit database\n\n**Scheduled tasks**\n\nSchedule a daily task to export and upload legacy audit data  onto Blob  (older than X days)\nUpon successful export task, run a task to cleanup legacy audit data (older than X days)\n\n## Overview design\n![image](https://github.com/user-attachments/assets/200af864-8d67-4798-b455-dffa8731c18c)\n\n\n### Sequence diagram \n![image](https://github.com/user-attachments/assets/44c58160-515d-40d6-8df9-4aba0d50dae8)\n\n\n- Trigger audit job batch export based on a cron scheduler (configurable)\n- Check and create audit repository \u003caudit-csv-repository\u003e if not exists\n- Make sure \u003caudit-csv-repository\u003e has PRIVATE access, only authenticated and authorized users can access this data\n- Read user audit messages from Audit Database (Read Only Replica) by date range (archiveStartDate-archiveEndDate)\n- Process and **compress** audit data to save upload storage and improve upload speed\n- Upload to **Azure Blob Storage** via **content-services**.\n- Store one audit file per user with the following output file format: \u003cbr\u003e\n File Path: \"\\\u003carchiveStartDate\\\u003e_\\\u003carchiveEndDate\\\u003e_\\\u003cuuid\\\u003e_audit-export.csv\".\n- Upon successful upload, access to file content via an API (protected by access control): \u003cbr\u003e\n  **Parameters:**\n    - _decompress_ (false/true) return file content in compressed text or decompress it to csv format\n    - _repositoryId_: storage repository id\n    - _contentstream-id_: uploaded document id\n\n    \u003e GET {{gateway}}/api/audit-export-job/client-api/v2/contentstream-id/{{repositoryId}}/{{contentstream-id}}?decompress=true \n  \n    **Authorization**\n\n      {\n        \"functionId\": \"1013\",\n        \"functionCode\": \"audit\",\n        \"resource\": \"Audit\",\n        \"name\": \"Audit\",\n        \"privileges\": [\n            {\n                \"privilege\": \"view\",\n                \"limits\": []\n            }\n        ]\n      }\n    \n\n- Check data integrity of the uploaded file by comparing its content with database data (optional, enabled by default)\n- Upon successful job run, if the job is daily job, cleanup audit data from main audit db replica (optional, disabled by default)\n\n### Deployment configuration\n\n\u003cdetails\u003e\n\u003csummary\u003eAudit export job\u003c/summary\u003e\n\n````shell\n  audit-export-job:\n    chart: generic-integration\n    repoURL: \"{{ .sharedACR }}\"\n    version: \"{{ .genericIntegrationVersion }}\"\n    additionalClasses:\n      - common-configs\n    tags:\n      - custom\n    values:\n      replicaCount: 1\n      fullnameOverride: audit-export-job\n      image:\n        registry: project\n        repository: audit-export-job\n        tag: *auditExportVersion\n      resources:\n        requests:\n          cpu: 3000m\n          memory: 3Gi\n        limits:\n          cpu: 7000m\n          memory: 8Gi\n      mqbroker:\n        enabled: false\n      database:\n        enabled: true\n      env:\n        spring.autoconfigure.exclude: \"org.springframework.cloud.netflix.eureka.loadbalancer.LoadBalancerEurekaAutoConfiguration\"\n        server.port: \"8080\"\n        SPRING_LIQUIBASE_ENABLED: \"true\"\n        spring.profiles.active: \"json-logging\"\n        backbase.stream.cxp.contentServiceUrl: \"http://contentservices-export:8080\"\n        backbase.stream.cxp.serviceId: \"contentservices-export\"\n        spring.datasource.hikari.maximum-pool-size: \"100\"\n        spring.datasource.hikari.minimum-idle: \"20\"\n        spring.datasource.hikari.connection-timeout: \"45000\"\n        spring.datasource.hikari.idle-timeout: \"600000\"\n        spring.datasource.hikari.max-lifetime: \"1800000\"\n        spring.datasource.hikari.leak-detection-threshold: \"120000\"\n        spring.jpa.properties.hibernate.default_batch_fetch_size: \"1000\"\n        backbase.audit.batch.export.enabled: \"true\"\n        backbase.audit.batch.export.cronExpression: \"0 10 14 16 * ?\"\n        backbase.audit.batch.export.chunkSize: \"500000\"\n        backbase.audit.batch.export.batchWriteSize: \"25000\"\n        backbase.audit.batch.export.daily: \"false\"\n        backbase.audit.batch.export.params.requestId: \"1\"\n        #backbase.audit.batch.export.params.retentionDays: \"30\"\n        backbase.audit.batch.export.params.archiveStartDate: \"2025-01-01\"\n        backbase.audit.batch.export.params.archiveEndDate: \"2025-02-01\"\n        backbase.audit.batch.cleanup.enabled: \"false\"\n        backbase.audit.batch.cleanup.chunkSize: \"100\"\n        backbase.audit.batch.cleanup.batchWriteSize: \"100\"\n        backbase.audit.batch.csvRemoval.enabled: \"false\"\n        backbase.audit.batch.csvRemoval.params.requestId: \"1\"\n        backbase.audit.batch.csvRemoval.params.exportJobId: \"28\"\n        AUDIT_DB_HOST:\n          value:\n            configMapKeyRef:\n              key: mssql-endpoint\n              name: database\n              optional: false\n        AUDIT_DB_PORT:\n          value:\n            configMapKeyRef:\n              key: mssql-port\n              name: database\n              optional: false\n        AUDIT_DB_SERVERNAME:\n          value:\n            configMapKeyRef:\n              key: mssql-server-name\n              name: database\n              optional: false\n        AUDIT_DB_SID: \"audit\"\n        AUDIT_DB_PASSWORD:\n          value:\n            secretKeyRef:\n              key: db_password\n              name: db-audit\n        AUDIT_DB_USERNAME:\n          value:\n            secretKeyRef:\n              key: db_username\n              name: db-audit\n        spring.datasource.read-replica.username: \"$(AUDIT_DB_USERNAME)\"\n        spring.datasource.read-replica.password: \"$(AUDIT_DB_PASSWORD)\"\n        spring.datasource.read-replica.driverClassName: \"com.microsoft.sqlserver.jdbc.SQLServerDriver\"\n        spring.datasource.read-replica.url: \"jdbc:sqlserver://$(AUDIT_DB_HOST):$(AUDIT_DB_PORT);database=$(AUDIT_DB_SID);user=$(AUDIT_DB_USERNAME)@$(AUDIT_DB_SERVERNAME);password=$(AUDIT_DB_PASSWORD);encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;applicationIntent=readonly;\"\n        spring.datasource.auditdb.username: \"$(AUDIT_DB_USERNAME)\"\n        spring.datasource.auditdb.password: \"$(AUDIT_DB_PASSWORD)\"\n        spring.datasource.auditdb.driverClassName: \"com.microsoft.sqlserver.jdbc.SQLServerDriver\"\n        spring.datasource.auditdb.url: \"jdbc:sqlserver://$(AUDIT_DB_HOST):$(AUDIT_DB_PORT);database=$(AUDIT_DB_SID);user=$(AUDIT_DB_USERNAME)@$(AUDIT_DB_SERVERNAME);password=$(AUDIT_DB_PASSWORD);encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;\"\n\n````\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eContent service (cloned)\u003c/summary\u003e\n\n````shell\n  contentservices-export:\n    chart: generic-integration\n    repoURL: \"{{ .sharedACR }}\"\n    version: \"{{ .genericIntegrationVersion }}\"\n    additionalClasses:\n      - common-configs\n      - high-service-resource\n    tags:\n      - custom\n    values:\n      fullnameOverride: contentservices-export\n      replicaCount: 5\n      mqbroker:\n        enabled: true\n      database:\n        enabled: true\n      image:\n        tag: *backbaseBomVersion\n        repository: contentservices\n        registry: shared\n      env:\n        LOGGING_LEVEL_COM_BACKBASE: \"DEBUG\"\n        LOGGING_LEVEL_COM: \"DEBUG\"\n        \"spring.cloud.loadbalancer.ribbon.enabled\": \"false\"\n        \"spring.autoconfigure.exclude\": org.springframework.cloud.netflix.eureka.loadbalancer.LoadBalancerEurekaAutoConfiguration\n        \"SPRING_LIQUIBASE_ENABLED\": \"true\"\n        \"spring.profiles.active\": \"json-logging\"\n        \"contentservices.storage.connectors.azureblob.service\": \"azure-blobstore-connector\"\n        \"contentservices.storage.defaultImplementation\": \"azureblob\"\n        \"contentservices.storage.signedUrl.enabled\": \"true\"\n        \"CONTENTSERVICES_USE_AZURE_BLOB_STORAGE_CONNECTOR\": \"true\"\n        \"spring.cloud.azure.eventhubs.kafka.enabled\": \"false\"\n        \"contentservices.whitelist.enabled\": \"true\"\n        \"contentservices.whitelist.allowedContentTypes\": \"image/jpeg,image/pjpeg,image/gif,image/png,image/svg+xml,application/pdf,application/json,application/zip,application/rtf,text/plain,application/octet-stream,text/html,text/csv,text/x-handlebars-template,message/rfc822,application/x-x509-key\"\n        \"contentservices.antivirus.service\": \"clamav-antivirus-connector\"\n        \"contentservices.antivirus.check-on-save.enabled\": \"false\"\n        \"contentservices.antivirus.check-on-query.enabled\": \"false\"\n        \"spring.servlet.multipart.max-file-size\": \"15MB\"\n        \"SIG_SECRET_KEY\":\n          value:\n            secretKeyRef:\n              name: jwt\n              key: jwt-internal-secretkey\n              optional: false\n````\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbackbase%2Faudit-export-job","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbackbase%2Faudit-export-job","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbackbase%2Faudit-export-job/lists"}