{"id":16101538,"url":"https://github.com/mendhak/aws-elb-logster","last_synced_at":"2025-04-06T00:27:52.701Z","repository":{"id":15294221,"uuid":"18023891","full_name":"mendhak/aws-elb-logster","owner":"mendhak","description":"Logster module for AWS Elastic Load Balancer access logs","archived":false,"fork":false,"pushed_at":"2014-03-24T15:05:19.000Z","size":496,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-12T06:38:50.353Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mendhak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-03-23T01:20:04.000Z","updated_at":"2023-09-08T16:46:26.000Z","dependencies_parsed_at":"2022-07-31T04:08:51.688Z","dependency_job_id":null,"html_url":"https://github.com/mendhak/aws-elb-logster","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mendhak%2Faws-elb-logster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mendhak%2Faws-elb-logster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mendhak%2Faws-elb-logster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mendhak%2Faws-elb-logster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mendhak","download_url":"https://codeload.github.com/mendhak/aws-elb-logster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247419638,"owners_count":20936009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-09T18:50:13.359Z","updated_at":"2025-04-06T00:27:52.682Z","avatar_url":"https://github.com/mendhak.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"aws-elb-logster\n===============\n\nLogster module for AWS ELB logs. Extremely alpha, under development.\n\n[Logster](https://github.com/etsy/logster) is a utility for reading log files and generating metrics in Graphite or Ganglia or Amazon CloudWatch. It comes with parsers for common log types such as Apache and Squid.  \n\n\n###AWS ELB Logs\n\nAmazon have introduced [access logs for Elastic Load Balancers](http://aws.typepad.com/aws/2014/03/access-logs-for-elastic-load-balancers.html). The log files are stored in an S3 bucket and their format is described in their documentation  [here](http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/access-log-collection.html). \n\n\n###Parser - ELBLogster.py\n\nThe `ELBLogster.py` parser will accept a line from an ELB access log and gather the following:\n\n* HTTP status codes (200s, 300s, etc) - `http.200`, `http.206`, `http.304` ...\n* Bytes Sent, Bytes Received - `bytes.sent`, `bytes.received`\n* Country code (from IP address) - `country.IE`, `country.SG` ...\n \n\n###IP to country code - geoip.py\n\n`geoip.py` is a simple [wrapper](http://blog.brush.co.nz/2009/07/geoip/) around `GeoIP.dat` which in turn is a free downloadable database from [MaxMind](http://dev.maxmind.com/geoip/legacy/geolite/).  During a logster run, `ELBLogster.py` will download `GeoIP.dat` if it is stale, so that `geoip.py` can perform an IP to Country Code lookup. \n\n### Output\n\nCurrent output when using `--output=graphite`, but this is subject to change.  \n\n    graphiteserver:2003 eu-west-1.bytes.sent 633779898 1395667198\n    graphiteserver:2003 eu-west-1.bytes.received 15582365 1395667198\n    graphiteserver:2003 eu-west-1.country.KW 41 1395667198\n    graphiteserver:2003 eu-west-1.country.SN 5 1395667198\n    graphiteserver:2003 eu-west-1.country.SC 2 1395667198\n    graphiteserver:2003 eu-west-1.country.SA 7 1395667198\n    graphiteserver:2003 eu-west-1.country.MZ 6 1395667198\n    graphiteserver:2003 eu-west-1.country.SG 2282 1395667198\n    graphiteserver:2003 eu-west-1.country.SE 28 1395667198\n    graphiteserver:2003 eu-west-1.country.AT 279 1395667198\n    graphiteserver:2003 eu-west-1.http.200 31081 1395667198\n    graphiteserver:2003 eu-west-1.http.206 27 1395667198\n    graphiteserver:2003 eu-west-1.http.304 6975 1395667198\n    graphiteserver:2003 eu-west-1.http.500 407 1395667198\n    graphiteserver:2003 eu-west-1.http.302 503 1395667198\n    graphiteserver:2003 eu-west-1.http.404 482 1395667198\n\n### Usage\n\nAs with any other logster parser, place `ELBLogster.py` and `geoip.py` in your Python path, then run it\n\n    sudo logster --metric-prefix=eu-west-1 --dry-run --output=graphite --graphite-host=graphiteserver:2003 ELBLogster ~/your-elb-log/2014-03-14.log \n\nI like to 'cheat' and place the files at the site packages for logster directly, for example `/usr/local/lib/python2.7/dist-packages/logster-0.0.1-py2.7.egg/logster/parsers/`\n\nYou may need to `pip install pytz requests` \n\nRemember that the AWS ELB logs are not contiguous - a new one is created every 5 or 60 minutes, so you must download and concatenate them yourself, then feed it to logster as shown above.  A simple example is shown in [this gist](https://gist.github.com/mendhak/9717352).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmendhak%2Faws-elb-logster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmendhak%2Faws-elb-logster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmendhak%2Faws-elb-logster/lists"}