{"id":13990400,"url":"https://github.com/dCache/logstash4dcache","last_synced_at":"2025-07-22T12:32:23.845Z","repository":{"id":16452133,"uuid":"19203995","full_name":"dCache/logstash4dcache","owner":"dCache","description":"logstash parser and config for integration with dCache","archived":false,"fork":false,"pushed_at":"2016-05-28T18:37:52.000Z","size":225,"stargazers_count":5,"open_issues_count":2,"forks_count":2,"subscribers_count":25,"default_branch":"master","last_synced_at":"2024-08-09T13:17:18.456Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dCache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-27T12:48:48.000Z","updated_at":"2020-02-10T14:05:33.000Z","dependencies_parsed_at":"2022-09-18T23:11:30.161Z","dependency_job_id":null,"html_url":"https://github.com/dCache/logstash4dcache","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dCache%2Flogstash4dcache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dCache%2Flogstash4dcache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dCache%2Flogstash4dcache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dCache%2Flogstash4dcache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dCache","download_url":"https://codeload.github.com/dCache/logstash4dcache/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227098781,"owners_count":17730631,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-09T13:02:40.197Z","updated_at":"2024-11-29T10:30:26.767Z","avatar_url":"https://github.com/dCache.png","language":null,"readme":"logstash4dcache\n===============\n\nLogstash + elasticsearch + kibana are modern tools to visualize logs and time-stamped data and can be used to monitor and visualize dCache's billing information.\n\n![dCache billing demo dashboard][1]\n\n*The following setup is tested with Scientific Linux 6.5 with apache2*\n------------------------------------------------------------------------\n\nInstallation\n==============\n\n##On the node with billing files\n```\n# /etc/yum.repos.d/logstash.repo\n[logstash-1.4]\nname=logstash repository for 1.4.x packages\nbaseurl=http://packages.elasticsearch.org/logstash/1.4/centos\ngpgcheck=1\ngpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch\nenabled=1\n```\n\n```\n# yum install logstash logstash-contrib\n```\n###Configure logstash\nCreate a file ***/etc/logstash/conf.d/dcache-billing.conf:***\n```\ninput {\n  file {\n    path =\u003e \"/var/lib/dcache/billing/**/billing-20*\"\n    sincedb_path =\u003e \"/var/tmp/sincedb-dcache\"\n    # uncomment next line if you want to import existing data\n    # start_position =\u003e beginning\n    # uncomment next line to import files with logstash 2.2.x\n    # ignore_older =\u003e 0\n    type =\u003e \"dcache-billing\"\n  }\n}\n\nfilter {\n\n  if \"RemoveFiles=\" in [message] {\n  # Because RemoveFiles= is the only(source needed) non-conforming event.\n    grok {\n      patterns_dir =\u003e \"/etc/logstash/patterns\"\n      match =\u003e [ \"message\", \"%{REMOVE_ON_POOL}\" ]\n      named_captures_only =\u003e true\n      tag_on_failure =\u003e [ \"_parse_dcache_failure10\" ]\n    } # End of grok\n    mutate {\n      split =\u003e [ \"pnfsids\", \",\" ]\n      add_tag =\u003e [ \"dcache_billing_removed\" ]\n    } # End of Mutate to make a real list of the entries in pnfsids\n\n  } else {\n\n    grok {\n      patterns_dir =\u003e \"/etc/logstash/patterns\"\n      match =\u003e [ \"message\", \"%{TRANSFER_CLASSIC}\" ]\n      match =\u003e [ \"message\", \"%{STORE_CLASSIC}\" ]\n      match =\u003e [ \"message\", \"%{RESTORE_CLASSIC}\" ]\n      match =\u003e [ \"message\", \"%{REQUEST_CLASSIC}\" ]\n      match =\u003e [ \"message\", \"%{REQUEST_DCAP}\" ]\n      match =\u003e [ \"message\", \"%{REMOVE_CLASSIC}\" ]\n      match =\u003e [ \"message\", \"%{REMOVE_SRM}\" ]\n      named_captures_only =\u003e true\n      remove_field =\u003e [ \"message\" ]\n      tag_on_failure =\u003e [ \"_parse_dcache_failure00\" ]\n    }\n\n  } # End of if else\n\n\n  date {\n    match =\u003e [ \"billing_time\", \"MM.dd HH:mm:ss\" ]\n    timezone =\u003e \"CET\"\n    remove_field =\u003e [ \"billing_time\" ]\n  }\n\n  alter {\n    condrewrite =\u003e [\n      \"is_write\", \"true\", \"write\",\n      \"is_write\", \"false\", \"read\"\n    ]\n  }\n}\n\noutput {\n  elasticsearch {\n    host =\u003e \"monitoring-host-name\"\n    index =\u003e \"dcache-billing-%{+YYYY.MM.dd}\"\n    template_name =\u003e \"billing\"\n    protocol =\u003e \"http\"\n  }\n}\n```\n\n###Add parser for dCache's billing files\n```\n# mkdir /etc/logstash/patterns\n# wget -O /etc/logstash/patterns/dcache-billing  https://raw.githubusercontent.com/dCache/logstash4dcache/master/patterns/dcache-billing\n```\n##On the node which will run as monitoring system\n```\n# /etc/yum.repos.d/elasticsearch.repo\n[elasticsearch-1.2]\nname=Elasticsearch repository for 1.2.x packages\nbaseurl=http://packages.elasticsearch.org/elasticsearch/1.2/centos\ngpgcheck=1\ngpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch\nenabled=1\n```\n\n```\n# yum install yum-conf-epel httpd elasticsearch redis\n```\n\nDownload and configure Kibana\n```\n# wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.1.tar.gz\n# tar -xvzf kibana-3.0.1.tar.gz\n# mv kibana-3.0.1 /var/www/html/kibana3\n```\n\nIn the file ***/var/www/html/kibana3/config.js*** update line with 'elasticsearch:' (line 32?):\n```\n   elasticsearch: \"http://\u003cyour-host-name\u003e:9200\",\n```\n\nAdd apache config file to access kibana ***/etc/httpd/conf.d/kibana.conf***:\n```\n  ProxyPass /elasticsearch http://localhost:9200\n  ProxyPassReverse /elasticsearch http://localhost:9200\n  Alias /kibana /var/www/html/kibana3\n\n  \u003cLocation /kibana\u003e\n    Order deny,allow\n    Deny from all\n    Allow from \u003cyour-host-or-network\u003e\n  \u003c/Location\u003e\n```\n\nConfigure redis, in ***/etc/redis.conf***\n```\nbind \u003cyour-ip-here\u003e\n```\nNotice, that bind ip **MUST** match one used in logstash config file in *elasticsearch* section.\n\nUpdate ***/etc/sysconfig/elasticsearch*** to adjust memory available for elasticsearch. Do not use more then 1/2 of total memory.\n```\nES_HEAP_SIZE=4g\n```\n\n\nTweak elasticsearch ***for single node*** performance. Update ***/etc/elasticsearch/elasticsearch.yml***\n\n```\nindex.number_of_shards: 1\nindex.number_of_replicas: 0\n```\n###Ready to go!\n\nOn monitoring host:\n```\n# service redis start\n# service elasticsearch start\n# service httpd start\n```\n\nWe need to update tamplate for newly crated indexes (databases)\n```\n# curl -XPUT 'http://localhost:9200/_template/billing' -d '\n  {\n    \"order\":0,\n    \"template\":\"dcache-billing-*\",\n    \"settings\":{\n      \"index.refresh_interval\":\"5s\"\n    },\n    \"mappings\":{\n      \"_default_\":{\n        \"dynamic_templates\":[\n          {\n            \"string_fields\":{\n              \"mapping\":{\n                \"index\":\"analyzed\",\n                \"omit_norms\":true,\n                \"type\":\"string\",\n                \"fields\":{\n                  \"raw\":{\n                    \"index\":\"not_analyzed\",\n                    \"ignore_above\":256,\n                    \"type\":\"string\"\n                  }\n                }\n              },\n              \"match_mapping_type\":\"string\",\n              \"match\":\"*\"\n            }\n          }\n        ],\n        \"properties\":{\n          \"geoip\":{\n            \"dynamic\":true,\n            \"path\":\"full\",\n            \"properties\":{\n              \"location\":{\n                \"type\":\"geo_point\"\n              }\n            },\n            \"type\":\"object\"\n          },\n          \"@version\":{\n            \"index\":\"not_analyzed\",\n            \"type\":\"string\"\n          }\n        },\n        \"_all\":{\n          \"enabled\":true\n        }\n      }\n    }\n  }\n'\n```\n\nOn billing host:\n```\n# service logstash start\n```\n\nThis configuration will crate a new index (database) for each day. The database for May 1 2014 can be dropped by:\n```\ncurl -XDELETE 'http://localhost:9200/dcache-billing-2014.05.01'\n```\n\nCreating a dashboard\n====================\n\nPoint your browser to (http://your-monitoring-host/kibana/index.html#/dashboard/file/default.json)\nIn the right upper corner, you can see the load icon ( in for of a folder ). Click on it -\u003e Advanced -\u003e Gist and paste this url [dCache-billing-demo](https://gist.github.com/kofemann/d9d0a9c489ccd3f79563)\n\nYou should get the same dashboard shown above.\n\n***We will be happy to see your own dashboards!***\n##Happy dCache monitoring!\n\n  [1]: https://raw.githubusercontent.com/kofemann/logstash4dcache/master/demo/dCache-billing-demo.png\n","funding_links":[],"categories":["Others"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FdCache%2Flogstash4dcache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FdCache%2Flogstash4dcache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FdCache%2Flogstash4dcache/lists"}