{"id":19557203,"url":"https://github.com/gingray/log-analysis","last_synced_at":"2026-04-17T15:33:23.693Z","repository":{"id":151847153,"uuid":"186414469","full_name":"gingray/log-analysis","owner":"gingray","description":"Suite to analyse logs from your app on local machine use Kibana and ElasticSearch","archived":false,"fork":false,"pushed_at":"2019-05-13T13:23:58.000Z","size":1,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-10T15:45:07.422Z","etag":null,"topics":["docker","docker-compose","elasticsearch","json","kibana","logs"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gingray.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-13T12:22:47.000Z","updated_at":"2019-05-13T13:23:59.000Z","dependencies_parsed_at":"2023-05-12T07:45:11.044Z","dependency_job_id":null,"html_url":"https://github.com/gingray/log-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gingray/log-analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Flog-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Flog-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Flog-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Flog-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gingray","download_url":"https://codeload.github.com/gingray/log-analysis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Flog-analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31934348,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T12:37:54.787Z","status":"ssl_error","status_checked_at":"2026-04-17T12:37:25.095Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","elasticsearch","json","kibana","logs"],"created_at":"2024-11-11T04:40:46.640Z","updated_at":"2026-04-17T15:33:23.676Z","avatar_url":"https://github.com/gingray.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Analyse Logs\nI'll desctibe my workflow for logs analitics its not perfect but in most cases it can do the job. Basically its for internal use.\n## Prerequisites\nI assume that you logs store in papertrail but it can be anywhere\n\n1. fetch logs from parpertrail example:\n   ```\n    papertrail  --min-time '2019-05-13 00:00:00' --max-time '2019-05-13 23:59:59'  evt_type \u003e 2019-05-13_events.log\n   ```\n   you need to have papertrail and setup it https://github.com/papertrail/papertrail-cli\n2. logs on your machine you need to do docker-compose up -d (with kibanan and elasticsearch) \n3. Use logcli (https://github.com/gingray/logcli) to extract json object form you logs.\n   ```bash\n   logcli extract_json --filenames=example1.log example2.log\n   ```\n   assume that you have log lines looks like this:\n   ```text\n   May 10 01:00:02 0f04a46f7fd2 production-evt: I, [2019-05-10T08:00:02.090892 #1]  INFO -- : {\"evt_type\":\"EvtClass\",\"status\":\"info\",\"time\":\"2019-05-10T08:00:02+0000\",\"payload\":{\"time_elapsed_human\":\"00:00:00.323\",\"time_elapsed\":0.323117733001709},\"trace_id\":\"9d35ca3a-b52f-462d-b5f1-f55ccc781e5e\"}\n   ```\n   its not a valid json object but logcli able to extract valid from each line after this command you'll get\n   ```json\n   {\"evt_type\":\"EvtClass\",\"status\":\"info\",\"time\":\"2019-05-10T08:00:02+0000\",\"payload\":{\"time_elapsed_human\":\"00:00:00.323\",\"time_elapsed\":0.323117733001709},\"trace_id\":\"9d35ca3a-b52f-462d-b5f1-f55ccc781e5e\"}\n   ```\n   on each line\n4. than push these logs to ealsicsearch\n   ```bash\n   logcli elasticsearch --elasticsearch-url=http://localhost:9201 --filenames=example1.json example2.json\n   ```\n5. Enjoy\n\nNOTE:\nbefore push data to elasticsearch make jusre that eac line is valid json object. Data pushed in ES by batches 100 per request its not optimize in terms of backpressure to a ES but its was able to process ~400mb logs in a seconds but objects that was used it was relatively small.\n\nWhy ES on 2001 port?\nBecause I've ES already on port 9200 for my dev env thats why to avoid conflicts its on 9201\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgingray%2Flog-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgingray%2Flog-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgingray%2Flog-analysis/lists"}