{"id":20252639,"url":"https://github.com/streamdal/log-processor","last_synced_at":"2025-03-03T16:43:05.854Z","repository":{"id":222564673,"uuid":"733647853","full_name":"streamdal/log-processor","owner":"streamdal","description":"Process logs based on Streamdal pipelines","archived":false,"fork":false,"pushed_at":"2024-04-27T22:33:14.000Z","size":5409,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-14T03:10:54.232Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamdal.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-19T20:01:38.000Z","updated_at":"2024-10-06T07:56:52.000Z","dependencies_parsed_at":"2024-02-15T00:26:01.475Z","dependency_job_id":"423923e1-dcc1-4d3d-a6fc-40f138e72edc","html_url":"https://github.com/streamdal/log-processor","commit_stats":null,"previous_names":["streamdal/log-processor"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Flog-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Flog-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Flog-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Flog-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamdal","download_url":"https://codeload.github.com/streamdal/log-processor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241703642,"owners_count":20006228,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T10:17:43.168Z","updated_at":"2025-03-03T16:43:05.829Z","avatar_url":"https://github.com/streamdal.png","language":"Go","readme":"\u003cdiv align=\"center\"\u003e\n   \n\u003cimg src=\"./assets/streamdal-logo-dark.png#gh-dark-mode-only\"\u003e\u003cimg src=\"./assets/streamdal-logo-light.png#gh-light-mode-only\"\u003e  \n\n[![GitHub](https://img.shields.io/github/license/streamdal/streamdal)](https://github.com/streamdal/streamdal)\n[![Discord](https://img.shields.io/badge/Community-Discord-4c57e8.svg)](https://discord.gg/streamdal)\n\n\u003c/div\u003e  \n\n## Real-Time Log Processing with Streamdal and Go\n\n### Overview\n\nLeverage the power of Streamdal to supercharger your existing logging solution.\n\n### Features\n\n**Real-Time Log Processing:** Fast processing of incoming logs with minimal latency.\n\n**PII Redaction:** Automated redaction of sensitive information from logs.\n\n**Centralized Rule Management:** Streamdal provides a UI for central rule management. These rules are then pulled in from the log-processing agents across your organization.\n\n**Scalable Architecture:** Designed to handle high volumes of log data efficiently.Blazing fast pipelines, all function built using WASM, and pushed down to local log-processors to distribute load to the edge and allow for realtime processing.  \n\n**Go Processing Agent:** Custom-built agent for processing logs.\n\n### Getting Started\n\nThe steps below will deploy the Streamdal stack along with a logstash environment to demonstrate how the solution works\n\n1. Deploy Streamdals Server see https://github.com/streamdal/streamdal/tree/main/install/docker\n1. Clone this repo ```git clone git@github.com:streamdal/log-processor.git``\n1. Bring up development environment ```cd log-processor/;docker-compose up -d```\n1. View Streamdal UI http://127.0.0.1:8080/\n1. Execute log generator to send sample data ```python3 log-generator.py```\n1. Create a pipeline\n![Create Pipeline](assets/create-pipeline.png)\n1. Attach Pipeline to the Data Stream\n![Attach Pipeline](assets/attach-pipeline.png)\n1. Use Streamdal's tail to confirm IPs are masked\n![Tail](assets/tail.png)\n1. You should see the masked IP\n![Live Tail](assets/live-tail.png)\n1. Goto kibana http://127.0.0.1:5601/\n1. Create a kibana pattern\n![Index Pattern](assets/index-pattern.png)\n1. Confirm IP data is masked in Kibana\n![Kibana](assets/Kibana.png)\n\n\n### Deploying in prod / existing logstash\n\n1. Deploy Streamdal to kubernetes using the helm https://github.com/streamdal/streamdal/tree/main/install/helm\n1. Deploy log-processor to all your logstash agents using docker or via the binary streamdal/log-processor\n1. Update logstash agent to send the json logs you want to process to the log-processor \n\n```\ninput {\n  tcp {\n    port =\u003e 5044\n    codec =\u003e json_lines\n  }\n}\n\noutput {\n  stdout { codec =\u003e rubydebug}\n  tcp {\n    host =\u003e \"go-app\" \n    port =\u003e 6000\n    codec =\u003e json_lines\n  }\n}\n```\n4. Add a section to recieve the processed data from the Streamdal log-processor and output to final destination\n\n```\ninput {\n  tcp {\n    port =\u003e 7002\n    codec =\u003e json_lines\n  }\n}\n\noutput {\n  elasticsearch {\n    hosts =\u003e [\"elasticsearch:9200\"] # Assumes Elasticsearch service is named 'elasticsearch' in docker-compose\n    index =\u003e \"processed-logs-%{+YYYY.MM.dd}\" # Customize the index name as needed\n  }\n}\n```\n5. Access the Streamdal Console you deployed earlier to apply whatever pipelines / rules needed.\n\n\n# Community\n\nWe're building Streamdal in the open and we'd love for you to join us!\n\nJoin our [Discord](https://discord.gg/streamdal)!","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Flog-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamdal%2Flog-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Flog-processor/lists"}