{"id":16082152,"url":"https://github.com/varunon9/audit-logging-framework","last_synced_at":"2025-03-18T06:30:24.000Z","repository":{"id":93827031,"uuid":"149894932","full_name":"varunon9/audit-logging-framework","owner":"varunon9","description":"An audit logging framework for Relational databases.","archived":false,"fork":false,"pushed_at":"2018-09-29T18:08:58.000Z","size":36,"stargazers_count":8,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T07:49:27.567Z","etag":null,"topics":["audit-logging","audit-logging-framework","kafka","mongodb","mongoose","mysql","nodejs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/varunon9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-22T16:28:20.000Z","updated_at":"2023-12-27T20:29:00.000Z","dependencies_parsed_at":"2023-06-15T06:30:50.204Z","dependency_job_id":null,"html_url":"https://github.com/varunon9/audit-logging-framework","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunon9%2Faudit-logging-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunon9%2Faudit-logging-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunon9%2Faudit-logging-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunon9%2Faudit-logging-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/varunon9","download_url":"https://codeload.github.com/varunon9/audit-logging-framework/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243906211,"owners_count":20366989,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audit-logging","audit-logging-framework","kafka","mongodb","mongoose","mysql","nodejs"],"created_at":"2024-10-09T11:26:02.471Z","updated_at":"2025-03-18T06:30:23.994Z","avatar_url":"https://github.com/varunon9.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## audit-logging-framework\n\nAn audit logging framework for Relational databases.\n\n### Prerequisite\n\nYou must have following softwares already setup-\n\n1. kafka (Pub-Sub model messaging system, a Big Data tool) \n2. zookeeper (It is already shipped with kafka. Kafka uses ZooKeeper to manage the cluster)\n3. mongodb (To store audit logs data)\n4. node and npm (development and build environment)\n\n\n### Working\n\nThis app has been developed to work with Relational databases. Its working can be explained in following steps-\n\n\u003cdl\u003e\n  \u003cdt\u003eTo create an audit-log: you hit an API http://localhost:5000/auditLog\u003c/dt\u003e\n  \u003cdd\u003e\n  method: POST\n\n  request:\n\n    {\n      \"tableId\": 1,\n      \"rowId\": 1,\n      \"oldData\": {\n        \"id\": 1,\n        \"name\": \"varun\",\n        \"age\": 22,\n        \"profession\": \"engineer\"\n      },\n      \"newData\": {\n        \"id\": 1,\n        \"name\": \"varun kumar\",\n        \"profession\": \"software developer\"\n      },\n      \"ipAddress\": \"localhost\",\n      \"user\": {\n        \"email\": \"varunon9@gmail.com\"\n      },\n      \"miscellaneous\": {\n        \"platform\": \"web\",\n        \"os\": \"linux\"\n      }\n    }\n\n  Here **oldData** and **newData** are complete data of a row in table. **user** is user details and **miscellaneous** is any other data that you want to store in logs.\n  **tableId** and **rowId** are unique identifiers for auditLogs. You can assign all your tables a unique id. \n  This data is pushed to kafka for processing. Processing is basically calculating diff of **oldData** and **newData** and creating a new payload. This new payload is then stored to mongodb. Stored data would look like-\n\n    {\n      \"_id\" : ObjectId(\"5baf76b53b471c41857a1a0d\"),\n      \"rowId\" : 1,\n      \"tableId\" : 1,\n      \"updatedAt\" : ISODate(\"2018-09-29T12:57:25.540Z\"),\n      \"__v\" : 0,\n      \"createdAt\" : ISODate(\"2018-09-29T12:57:25.540Z\"),\n      \"changeHistory\" : [\n        {\n          \"date\" : ISODate(\"2018-09-29T12:54:20.137Z\"),\n          \"_id\" : ObjectId(\"5baf76b5e8480034fa2197ce\"),\n          \"ipAddress\" : \"localhost\",\n          \"user\" : {\n            \"email\" : \"varunon9@gmail.com\"\n          },\n          \"miscellaneous\" : {\n            \"platform\" : \"web\",\n            \"os\" : \"linux\"\n          },\n          \"log\" : {\n            \"name\" : \"varun kumar\",\n            \"profession\" : \"software developer\"\n          }\n        }\n      ]\n    }\n\n  If you hit `http://localhost:5000/auditLog` again with same tableId and rowId, logs would be pushed to *changeHistory*. When you create a new record in table (in your relational database), you pass **oldData** as `{}` (empty object). In that case all of **newData** would be pushed to *changeHistory*.\n  \u003c/dd\u003e\n\n  \u003cdt\u003eTo retrieve audit-log: you hit same API http://localhost:5000/auditLog\u003c/dt\u003e\n  \u003cdd\u003e\n  method: GET\n\n  queryParams: tableId, rowId, count\n\n  example-\n\n    GET http://localhost:5000/auditLog/?tableId=1\u0026rowId=1\u0026count=5\n\n  In response you would get *changeHistory* of given **tableId** and **rowId** with no of results = *count*.\n\n  The data would be sorted according to data i.e. latest 5 results\n  \u003c/dd\u003e\n\n\u003c/dl\u003e\n\n### How to install\n\n1. Clone repo `git clone https://github.com/varunon9/audit-logging-framework.git`\n2. Go inside repo `cd audit-logging-framework`\n3. Edit `config/index.js` for connection details.\n4. Go to kafka installation directory `cd /opt/kafka_2.11-1.0.0`\n5. First start zookeeper `sudo bin/zookeeper-server-start.sh config/zookeeper.properties`\n6. Start kafka `sudo bin/kafka-server-start.sh config/server.properties`\n7. You will have to create kafka-topic (name mentioned in `config/index.js`)\n8. create topic named audit-logging `bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic audit-logging`\n9. Now come back to project installation directory and start app `npm start`\n10. Log data from your service class i.e. when data is created/updated in table. POST `localhost:5000/auditLog/`\n11. Retrieve data when showing audit-logs. GET `localhost:5000/auditLog`\n\n### Blog\n\nhttps://medium.com/@varunon9/an-audit-logging-framework-for-relational-databases-f902550cd849\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarunon9%2Faudit-logging-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvarunon9%2Faudit-logging-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarunon9%2Faudit-logging-framework/lists"}