{"id":19736952,"url":"https://github.com/travis-ci/travis-logs","last_synced_at":"2025-04-06T11:09:14.203Z","repository":{"id":4502804,"uuid":"5642248","full_name":"travis-ci/travis-logs","owner":"travis-ci","description":"Processes log updates from the job runner (worker), and streams them to the web client, aggregates them, and archives to S3.","archived":false,"fork":false,"pushed_at":"2025-01-21T13:07:18.000Z","size":1567,"stargazers_count":55,"open_issues_count":25,"forks_count":29,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-03-30T10:07:12.942Z","etag":null,"topics":["log","logger","logging","logs"],"latest_commit_sha":null,"homepage":"https://travis-ci.com","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/travis-ci.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-09-01T18:25:56.000Z","updated_at":"2025-01-21T13:06:40.000Z","dependencies_parsed_at":"2024-11-29T12:26:31.587Z","dependency_job_id":"6a3ac38f-6f71-4e6d-ab1e-1b6c1123c68d","html_url":"https://github.com/travis-ci/travis-logs","commit_stats":null,"previous_names":[],"tags_count":145,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Ftravis-logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Ftravis-logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Ftravis-logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travis-ci%2Ftravis-logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/travis-ci","download_url":"https://codeload.github.com/travis-ci/travis-logs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247471521,"owners_count":20944158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["log","logger","logging","logs"],"created_at":"2024-11-12T01:09:00.917Z","updated_at":"2025-04-06T11:09:14.187Z","avatar_url":"https://github.com/travis-ci.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Travis Logs\n\n[![Build Status](https://travis-ci.org/travis-ci/travis-logs.svg?branch=master)](https://travis-ci.org/travis-ci/travis-logs)\n\nTravis Logs processes log updates which are streamed from [Travis\nWorker](https://github.com/travis-ci/worker) instances via\n[RabbitMQ](http://www.rabbitmq.com/). The log parts are streamed via\n[Pusher](http://pusher.com/) to the web client ([Travis\nWeb](http://github.com/travis-ci/travis-web)) and added to the database.\n\nOnce all log parts have been received, and a timeout has passed (10 seconds\ndefault), the log parts are aggregated into one final log.\n\nTravis Logs archives logs to S3 and the database records are purged once it is\nverified that the logs are archived correctly.\n\n## Local Development\n\nWhen developing locally, one may want to set certain config params via env vars,\nsuch as a `DATABASE_URL` that points to a valid PostgreSQL server.  See the\n`.example.env` file for examples.\n\n## Process types\n\nSome of the process types listed in [`./Procfile`](./Procfile) depend on other\nprocess types, while others are independent:\n\n### `drain` and `drain_sharded` process\n\nThe `drain` process is responsible for consuming log parts messages via AMQP and\nbatching them together as enqueued jobs in the `log_parts` sidekiq queue.\n`drain_sharded` is the same, yet connects differently to AMQP.\n\n### `web` process\n\nThe `web` process runs a Sinatra web app that exposes APIs to handle\ninteractions with other Travis applications and the external Pusher service.\n\n### `worker_critical` process\n\nThe `worker_critical` process is responsible for handling jobs from the\nfollowing sidekiq queues:\n\n#### `logs.pusher_forwarding` sidekiq queue\n\nThe jobs in the `logs.pusher_forwarding` queue forward each log part\nindividually to Pusher.\n\n### `worker_high` process\n\nThe `worker_high` process is responsible for handling jobs from the following\nsidekiq queues:\n\n#### `log_parts` sidekiq queue\n\nThe jobs in the `log_parts` sidekiq queue write batches of log parts records to\nthe `log_parts` table.\n\n#### `aggregate` sidekiq queue\n\nThe jobs in the `aggregate` sidekiq queue combine all `log_parts` records for a\ngiven log id into a single content blob that is set on the corresponding `logs`\nrecord and then deletes the `log_parts` records.\n\n### `worker_low` process\n\nThe `worker_low` process is responsible for handling jobs from the following\nsidekiq queues:\n\n#### `archive` sidekiq queue\n\nJobs in the `archive` sidekiq queue move the content of each fully aggregated\nlog record from the database to S3.  Once archiving is complete, a job is sent\nfor consumption in the `purge` sidekiq queue.\n\n#### `purge` sidekiq queue\n\nJobs in the `purge` sidekiq queue set the log record content to NULL after\nverifying that the archived (S3) content fully matches the log record content.\nIf there is a mismatch, the log id is sent to the `archive` sidekiq queue for\nre-archiving.\n\n### `aggregate_sweeper` process\n\nThe `aggregate_sweeper` process is an optional process that periodically queries\nthe `log_parts` table for records that may have been missed by the event-based\naggregation process that flows through the `aggregate` sidekiq queue.\n\n## Database specifics\n\n### Schema management\n\nThe schema and migrations for travis-logs are managed with\n[sqitch](http://sqitch.org/).  All of the deploy, verify, and revert scripts may\nbe found in the `./db/` directory.\n\nTo install sqitch locally, you can run:\n\n```\n$ script/install-sqitch\n```\n\nTo run sqitch, you can run:\n\n```\n$ script/sqitch-heroku DATABASE_URL travis-logs-staging status\n```\n\nFor more information on how to use sqitch and how to add migrations, you can\ntake a look at the [sqitch tutorial](https://metacpan.org/pod/sqitchtutorial).\n\n### Data lifecycle\n\nThe process types above use PostgreSQL for various operations, with a structure\nof two tables: `logs` and `log_parts`.  Normal operations may be generalized as\na progression from writing to `log_parts`, to combining those records into\n`logs`, and then moving the content to S3.\n\nFor this reason, the `log_parts` table at any one time is mostly empty space,\nwith the size reported by PostgreSQL being significantly larger than what is\nreally there.  To a lesser degree, the `logs` table is also mostly empty,\nalthough the live record count will continue to grow over the lifetime of a\ndeployment as metadata is retained after the content has been moved to S3.\n\n### Partitioned `log_parts`\n\nIn order to address the empty space growth caused by the high record churn of\n`log_parts`, the deployments of travis-logs used for hosted Travis CI use the\n[pg_partman](https://github.com/keithf4/pg_partman) extension to drop daily\npartitions that are 2 days old.\n\nThe partitions are maintained by running the `partman.run_maintenance` query,\ntriggered via a daily Heroku scheduled job.  Because the `log_parts` table is\nbeing accessed constantly in production, and various operations within\n`partman.run_maintenance` require a PostgreSQL lock type of\n`AccessExclusiveLock` of the `log_parts` table, the implementation of the\nmaintenance operation includes a redis-based switch that prevents access to the\n`log_parts` table via other processes.\n\nDuring the maintenance operation, sidekiq workers will sleep and retry, then\nresume upon maintenance completion.  Any requests to `web` dynos during\nmaintenance that require access to the `log_parts` table will return `503`.\nThis is certainly not ideal, and more changes may be considered to further\nreduce production impact in the future.  In practice, the complete maintenance\noperation lasts about 1 minute.\n\n## License \u0026 copyright information\n\nSee LICENSE file.\n\nCopyright (c) 2018 Travis CI GmbH\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravis-ci%2Ftravis-logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftravis-ci%2Ftravis-logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravis-ci%2Ftravis-logs/lists"}