{"id":21739375,"url":"https://github.com/grrlopes/produ_logs","last_synced_at":"2026-04-12T09:38:21.865Z","repository":{"id":151413500,"uuid":"99517369","full_name":"grrlopes/Produ_logs","owner":"grrlopes","description":null,"archived":false,"fork":false,"pushed_at":"2017-08-07T03:37:33.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-21T00:44:30.385Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grrlopes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-06T22:42:34.000Z","updated_at":"2017-08-06T22:42:34.000Z","dependencies_parsed_at":"2023-07-04T04:51:18.364Z","dependency_job_id":null,"html_url":"https://github.com/grrlopes/Produ_logs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/grrlopes/Produ_logs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grrlopes%2FProdu_logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grrlopes%2FProdu_logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grrlopes%2FProdu_logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grrlopes%2FProdu_logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grrlopes","download_url":"https://codeload.github.com/grrlopes/Produ_logs/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grrlopes%2FProdu_logs/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266963370,"owners_count":24013037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T06:08:39.946Z","updated_at":"2026-04-12T09:38:16.821Z","avatar_url":"https://github.com/grrlopes.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Produ_logs(título tempo)\n\nO objetivo dessa documentação é descrever todas as tecnologias que foram utilizadas para criar o ambiente de coleta, provisionamento dos logs.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://blog.netapsys.fr/wp-content/uploads/2016/05/ELASTIC_LOGSTASH_KIBANA.jpg\"\u003e\n\u003c/p\u003e\n--\n\n* [Objetivo do projeto](#objetivo-do-projeto)\n* [Tecnologia usada](#tecnologia-usada)\n* [Download Instalação](#download-instalação)\n* [Configuração do Elasticsearch](#configuração-do-elasticsearch)\n* [Configuração do Kibana](#configuração-do-Kibana)\n* [Configuração do Mongodb](#Configuração-do-mongodb)\n* [Configuração do logstash](#Configuração-do-logstash)\n* [Modelo conector](#modelo-conector)\n* [Topologia](#Topologia)\n\n## Objetivo do projeto\n\nDesenvolver um ambiente que capture, filtre e que persista as informação de logs gerada pelo lançador rundeck / ansible.\n\n## tecnologia usada\n\n* **Elasticsearch** --\u003e Baseado no Apache Lucene, um servidor de busca e indexação textual, o objetivo do Elasticsearch \n    é fornecer um método de se catalogar e efetuar buscas em grandes massas de informação por meio de interfaces REST \n    que recebem/provêm informações em formato JSON.\n\n* **Logstash** --\u003e Criado pela Elastic, o conceito do Logstash é fornecer pipelines de dados, através do qual podemos \n    suprir as informações contidas nos arquivos de logs das nossas aplicações – além de outras fontes – para diversos \n    destinos, como uma instância de Elasticsearch, um bucket S3 na Amazon, um banco de dados MongoDB, entre outros\n\n* **Mongodb** --\u003e O MongoDB é um document database(banco de dados de documentos), mas não são os documentos da \n    família Microsoft, mas documentos com informações no formato JSON. A ideia é o documento representar toda a \n    informação necessária, sem a restrição dos bancos relacionais\n\n* **Grafana** --\u003e Ferramentas web para criação e exibição de gráficos.\n\n* **Kibana** --\u003e O Kibana foi desenvolvido pela Elastic com o intuito de fornecer uma interface rica que permita \n    consultas analíticas e/ou a construção de dashboards, com base nas informações contidas dentro de um Elasticsearch.\n\n* **\n\n## Download / Instalação\n\n* **Elasticsearch** \n    \n    1. https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz\n    2. tar -zxvf elasticsearch-5.5.1.tar.gz\n\n* **Logstash**\n\n    1. https://artifacts.elastic.co/downloads/logstash/logstash-5.5.1.tar.gz\n    2. tar -zxvf logstash-5.5.1.tar.gz\n\n* **Mongodb**\n\n    1. https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-amazon-3.4.6.tar.gz\n    2. tar -zxvf mongodb-linux-x86_64-amazon-3.4.6.tar.gz\n\n* **Grafana**\n\n    1. https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.4.2.linux-x64.tar.gz \n    2. tar -zxvf grafana-4.4.2.linux-x64.tar.gz \n\n* **Kibana**\n\n    1. https://artifacts.elastic.co/downloads/kibana/kibana-5.5.1-linux-x86_64.tar.gz\n    2. tar -zxvf kibana-5.5.1-linux-x86_64.tar.gz\n\n* **\n\n## Configuração do mongodb\n\n* **Usuário mongod**\n\n  1. useradd -d /usr/local/mongodb -g mongodb -r mongodb\n  2. chown mongodb.mongodb mongodb/ -R\n\n* **Execução do mongod**\n\n  1. bin/mongod -dbpath data/db/ --directoryperdb \u003e /tmp/mongodb.log 2\u003c\u00261 \u0026\n  \n## Configuração do logstash\n\n* **Execução do logstash**\n  \n  1. bin/logstash -f \"etc/pipeline-rundeck.conf\"\n\n* **Pipeline de configuração**\n\n```\ninput {\n\ttcp {\n\t\tport =\u003e 6511\n\t\ttype =\u003e 'rundeck'\n\t}\n}\n\nfilter {\n\tjson {\n\t\tsource =\u003e message\n\t}\n\truby {\n\tcode =\u003e \"\n\t\thash = event.to_hash\n\t\thash.each do |k,v|\n\t\tif v == nil\n\t\t\tevent.remove(k)\n\t\tend\n\t\tend\n\t\"\n\t}\n\tmutate {\n\t\tconvert =\u003e { \"message\" =\u003e \"string\" }\n\t\tremove_field =\u003e [\n\t\t\t\"line\", \"event.stepctx\"\n\t\t]\n\t}\n\tif[message] =~ /nil/ {\n\t\tmutate {\n\t\t\tremove_field =\u003e [\"execution.id\",\"execution.serverUrl\", \n\t\t\t\"execution.group\", \"@timestamp\", \"port\", \"execution.executionType\", \n\t\t\t\"execution.username\", \"execution.serverUUID\", \"@version\", \"host\", \n\t\t\t\"execution.url\",\"totallines\", \"execution.retryAttempt\", \"execution.wasRetry\", \n\t\t\t\"execution.loglevel\",\"execution.name\", \"line\", \"datetime\", \"event.step\", \"event.node\", \n\t\t\t\"event.user\", \"execution.user.name\", \"eventType\", \"message\", \"execution.project\", \n\t\t\t\"event.stepctx\", \"execution.execid\", \"loglevel\"]\n\t\t}\n\t}\n}\n\noutput {\n\tif[message] != \"PLAY RECAP **************************\n  *******************************************\" and \n\t[message] != \"TASK [apache : Configuring Apache files] **********************\n  *****************\" and \n\t[message] != \"TASK [Gathering Facts] ***************************\n  ******************************\" and \n\t[message] != \"PLAY [all] ********************************\n  *************************************\" and \n\t[message] != \"\" {\n\t\tstdout  { codec =\u003e rubydebug }\n\t\tmongodb {\n\t\t\turi =\u003e \"mongodb://localhost\"\n\t\t\tdatabase =\u003e \"rundecklog\"\n\t\t\tcollection =\u003e \"ansible\"\n\t\t}\n\t\telasticsearch {\n\t\t\tmanage_template =\u003e true\n\t\t\thosts =\u003e [\"localhost:9200\"]\n\t\t\tindex =\u003e \"rundecklog-%{+YYYY.MM.dd}\"\n\t\t}\n\t}\n}\n```\n\n## Modelo conector\n\n* **mongodb doc sucesso**\n\n```\n{\n        \"_id\" : ObjectId(\"59848657b959ec530700001b\"),\n        \"execution.id\" : \"7ac894bd-3629-49e1-94cb-4dd449f7f050\",\n        \"execution.name\" : \"Ansible Playbook\",\n        \"type\" : \"rundeck\",\n        \"datetime\" : NumberLong(\"1501857383983\"),\n        \"execution.serverUrl\" : \"http://172.20.150.95/\",\n        \"execution.group\" : null,\n        \"@version\" : \"1\",\n        \"host\" : \"172.20.150.95\",\n        \"event.step\" : \"1\",\n        \"execution.wasRetry\" : \"false\",\n        \"event.node\" : \"ansible-core\",\n        \"event.user\" : \"root\",\n        \"execution.user.name\" : \"admin\",\n        \"eventType\" : \"log\",\n        \"message\" : \"node-03: ok=2  changed=0  unreachable=0  failed=0\",\n        \"execution.project\" : \"ansible-lab\",\n        \"execution.execid\" : \"6049\",\n        \"@timestamp\" : \"\\\"2017-08-04T14:36:07.680Z\\\"\",\n        \"port\" : 43880,\n        \"execution.executionType\" : \"scheduled\",\n        \"execution.username\" : \"admin\",\n        \"loglevel\" : \"NORMAL\",\n        \"execution.serverUUID\" : null,\n        \"execution.url\" : \"http://172.20.150.95/project/ansible-lab/execution/follow/6049\",\n        \"execution.retryAttempt\" : \"0\",\n        \"execution.loglevel\" : \"INFO\"\n}\n\n        message: [node1, node2, node3]\n```\n\n* **mongodb doc falha**\n\n```\n{\n        \"_id\" : ObjectId(\"5984f9ccb959ecd3a60000ae\"),\n        \"execution.id\" : \"7ac894bd-3629-49e1-94cb-4dd449f7f050\",\n        \"execution.name\" : \"Ansible Playbook\",\n        \"execution.user.name\" : \"admin\",\n        \"eventType\" : \"log\",\n        \"message\" : \"Execution failed: 9005 in project ansible-lab: [Workflow result: , \n            step failures: {1=Dispatch failed on 1 nodes: [ansible-core: NonZeroResultCode: Remote command failed with \n            exit status 1]}, Node failures: {ansible-core=[NonZeroResultCode: Remote command failed with exit status 1]}, \n            status: failed]\",\n        \"type\" : \"rundeck\",\n        \"execution.project\" : \"ansible-lab\",\n        \"execution.execid\" : \"9005\",\n        \"datetime\" : NumberLong(\"1501886940930\"),\n        \"@timestamp\" : \"\\\"2017-08-04T22:48:44.905Z\\\"\",\n        \"execution.serverUrl\" : \"http://172.20.150.95/\",\n        \"port\" : 50460,\n        \"execution.executionType\" : \"scheduled\",\n        \"execution.username\" : \"admin\",\n        \"loglevel\" : \"ERROR\",\n        \"@version\" : \"1\",\n        \"host\" : \"172.20.150.95\",\n        \"execution.url\" : \"http://172.20.150.95/project/ansible-lab/execution/follow/9005\",\n        \"execution.retryAttempt\" : \"0\",\n        \"execution.wasRetry\" : \"false\",\n        \"execution.loglevel\" : \"INFO\"\n}\n```\n\n## Topologia\n\n![Produb](/home/gabriel/workspace_produban/documentacao_foguete/logo.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrrlopes%2Fprodu_logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrrlopes%2Fprodu_logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrrlopes%2Fprodu_logs/lists"}