{"id":20111324,"url":"https://github.com/openstack/ci-log-processing","last_synced_at":"2026-05-31T21:31:52.032Z","repository":{"id":38621019,"uuid":"422417040","full_name":"openstack/ci-log-processing","owner":"openstack","description":"OpenStack CI log processing tool. Mirror of code maintained at opendev.org.","archived":false,"fork":false,"pushed_at":"2025-10-14T14:22:26.000Z","size":384,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-10-14T16:03:12.394Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://opendev.org/openstack/ci-log-processing","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openstack.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-29T02:24:58.000Z","updated_at":"2025-10-14T14:22:31.000Z","dependencies_parsed_at":"2024-01-19T10:05:06.433Z","dependency_job_id":"f7f70d64-e57f-4cb3-811b-7b2a71995aba","html_url":"https://github.com/openstack/ci-log-processing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openstack/ci-log-processing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fci-log-processing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fci-log-processing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fci-log-processing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fci-log-processing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openstack","download_url":"https://codeload.github.com/openstack/ci-log-processing/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openstack%2Fci-log-processing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33750474,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T18:15:33.887Z","updated_at":"2026-05-31T21:31:52.015Z","avatar_url":"https://github.com/openstack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"OpenStack CI log processing\n===========================\n\nThe goal of this repository is to provide and check\nfunctionality of new log processing system base on\nzuul log scraper tool.\n\nLog Scraper\n-----------\n\nThe Log Scraper tool is responsible for periodical\ncheck by using Zuul CI API if there are new builds available\nand if there are some, it would push the informations to\nthe log processing system.\n\nLog Sender\n----------\n\nThe Zuul Log Sender tool is responsible for periodical check\ndirectory, if there are some files that should be send to the\nElasticsearch service.\nNOTE: build directories that does not provide files `buildinfo`\nand `inventory.yaml` file are skipped.\n\nOpensearch Dashboards Backup\n----------------------------\n\nThe OpenSearch Dashboards Backup script helps to create OpenSearch\nDashboards objects versioning.\n\nAvailable workflows\n-------------------\n\nThe Openstack CI Log Processing project is provides configuration\nfor sending logs from Zuul CI to the Opensearch host.\n\nLogscraper, logsender\n\nThis workflow removes bottlenecks by removing: log gearman client,\nlog gearman worker and logstash service. Logs are downloaded when\navailable to the disk, then parsed by logsender and send directly to the\nOpensearch host.\n\nWith this solution, log workflow looks like:\n\n.. code-block:: shell\n\n   +----------------+  1. Get last builds info      +-----------------+\n   |                | ----------------------------\u003e |                 |\n   |   Logscraper   |                               |  Zuul API       |\n   |                | \u003c---------------------------- |                 |\n   +----------------+    2. Fetch data              +-----------------+\n            |\n            +------------------------------------------------+\n                                                             |\n                                     3. Download logs;       |\n                                     include inventory.yaml  |\n                                     and build info          |\n                                                             |\n                                                             V\n                   +----------------+               +----------------+\n                   |                |               |                |\n                   |   Logsender    | \u003c------------ |  Download dir  |\n                   |                |               |                |\n                   +----------------+               +----------------+\n                           |\n    4. Parse log files;    |\n    add required fields;   |\n    send to Opensearch     |\n                           |\n                           v\n                  +-----------------+\n                  |                 |\n                  |   Opensearch    |\n                  |                 |\n                  +-----------------+\n\nYou can deploy your log workflow by using example Ansible playbook that\nyou can find in `ansible/playbooks/check-services.yml` in this project.\n\nTesting\n-------\n\nThe part of OpenStack CI log processing runs a complete testing and\ncontinuous-integration environment, powered by `Zuul\n\u003chttps://zuul-ci.org/\u003e`__.\n\nAny changes to logscraper script or tests will trigger jobs to\nthoroughly test those changes.\n\nTo run a single test: *tox -epy38 \u003ctest_name\u003e*\n\n\nBenchmarking\n------------\n\nThe large Zuul CI deployments requires many CI log processing resources.\nIn that case, we can do a benchmark with two log processing deployments.\nAll tests will do same:\n\n  - send 100 log builds to Elasticsearch that is running on same host\n  - logscraper will be using 4 workers\n  - VM will have 8 vcpus, 16 GB of RAM\n\nTesting workflows:\n\n* loggearman and logstash\n\nThis workflow will spawn 3 additional loggearman workers because it this\nservice is a bottleneck in that log ci workflow.\n\nYou can do it with command:\n\n.. code-block:: shell\n\n   for i in {1..3}; do \\\n     podman run --network host -d --name loggearman-worker-$i \\\n      --volume /etc/loggearman:/etc/loggearman:z \\\n      --volume /var/log/loggearman:/var/log/loggearman:z \\\n      quay.io/software-factory/loggearman:latest \\\n      log-gearman-worker -c /etc/loggearman/worker.yml --foreground  -d /var/log/loggearman/worker.log\n\nTo remove:\n\n.. code-block:: shell\n\n   for i in {1..3}; do \\\n     podman stop loggearman-worker-$i ; podman rm loggearman-worker-$i\n\n\nOn the end, basic calucations:\n\n.. code-block:: python\n\n   import datetime\n   start = datetime.datetime.fromisoformat(\"2022-02-28 16:44:59\")\n   stop = datetime.datetime.fromisoformat(\"2022-02-28 16:46:01\")\n   print((stop-start).total_seconds())\n\n\nTime spend to run logscraper and wait for finish all loggearman workers took: 62 seconds and\nit takes 680MB of RAM.\n\n\n* logsender workflow\n\nThis workflow will only use logsender tool and it will push the logs\ndirectly to the Elasticsearch service. Same as in previous test,\nit will be executed on 4 processes.\n\nTo download logs:\n\n.. code-block:: shell\n\n   logscraper \\\n    --zuul-api-url https://zuul.opendev.org/api/tenant/openstack \\\n    --checkpoint-file /tmp/results-checkpoint \\\n    --worker 8 \\\n    --max-skipped 100 \\\n    --download True \\\n    --directory /tmp/logscraper\n\nThis operation took: 30 seconds and it uses 130 MB of RAM.\n\n.. code-block:: shell\n\n   logsender --username admin --password mypassword --host localhost --port 9200 --insecure --workers 4\n\n\nTime spend to run logscraper and wait for finish all loggearman workers took: 35 second and\nit takes 520 MB of RAM.\n\nConclusion:\n\nThe logsender way seems to use less memory (on Opendev deployment, logstash\nservice is on different host, but 4096 MB of RAM is not enough) and it is faster,\nbut the logscraper and logsender process was executed one by one - on the\nbeginning logscraper download logs, then logsender send them to\nElasticsearch.\n\nContinuous Deployment\n---------------------\nOnce changes are reviewed and committed, they will be applied\nautomatically to the production hosts.\n\nContributing\n============\nContributions are welcome!\n\nCurrently only unit tests are available. In the future,\nfunctional tests would be added.\n\nDocumentation\n=============\nThe latest documentation is available at\nhttp://docs.openstack.org/infra/ci-log-processing\n\nThat documentation is generated from this repository. You can generate\nit yourself with ``tox -e docs``.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenstack%2Fci-log-processing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenstack%2Fci-log-processing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenstack%2Fci-log-processing/lists"}