{"id":18868248,"url":"https://github.com/letsencrypt/ctile","last_synced_at":"2025-04-14T14:31:45.701Z","repository":{"id":187388300,"uuid":"676696986","full_name":"letsencrypt/ctile","owner":"letsencrypt","description":null,"archived":false,"fork":false,"pushed_at":"2024-02-27T22:36:16.000Z","size":55,"stargazers_count":8,"open_issues_count":1,"forks_count":4,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-28T03:51:12.704Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/letsencrypt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-09T19:52:21.000Z","updated_at":"2024-12-17T11:46:21.000Z","dependencies_parsed_at":"2023-08-10T06:24:36.394Z","dependency_job_id":"b5eae310-a0a3-4976-a378-6632b48cc700","html_url":"https://github.com/letsencrypt/ctile","commit_stats":null,"previous_names":["jsha/ctile"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/letsencrypt%2Fctile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/letsencrypt%2Fctile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/letsencrypt%2Fctile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/letsencrypt%2Fctile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/letsencrypt","download_url":"https://codeload.github.com/letsencrypt/ctile/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248897209,"owners_count":21179557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T05:13:15.269Z","updated_at":"2025-04-14T14:31:45.385Z","avatar_url":"https://github.com/letsencrypt.png","language":"Go","readme":"# CTile\n\nThis is a caching proxy for the get-entries endpoint of a CT log, which uses S3\nas its backing store. It uses the concept of \"tiles\" of entries, where each tile\nis a fixed size N (e.g. 256 log entries) and the sequence of tiles starts at 0.\nRegardless of what `start` and `end` parameters CTile receives for a request, it\nwill transform those into a tile-sized request to its backend, by rounding down\n`start` to the nearest multiple of N and requesting exactly N items from the\nbackend. If the request is successful, CTile checks that the response contains\nexactly N items, re-encodes as gzipped CBOR, and stores the result in S3. It then\nreturns modified JSON to the user, removing items from the head and tail to ensure\nthat the first entry actually corresponds to the first entry requested by the user\nand that the response includes at most as many entries as requested.\n\nWhen looking up entries in the cache, CTile also rounds `start` down to the\nnearest multiple of N, and requests a single tile from the S3 backend. The CT\nprotocol allows the server to return fewer results than requested, so CTile does\nnot attempt to request multiple tiles to fulfil a large request. If a request's\n`start` parameter is one less than the end of a tile, CTile will respond with a\nsingle entry. This is similar to how Trillian's [align_getentries\nflag](https://github.com/google/certificate-transparency-go/blob/6e118585d9d9757b739353829becec378f47e10b/trillian/ctfe/handlers.go#L50)\nworks, and is in fact compatible with that flag, so long as CTile's tile size is\nless than or equal to Trillian's max_get_entries flag.\n\nWhen a user requests a range of get-entries near the end of the log, CTile\nusually won't be able to get a full tile's worth of entries from the backend,\nbecause the requisite number of entries haven't been sequenced yet. In this\ncase, CTile does not write anything to the S3 backend and simply passes\nthrough the entries returned from the server (after appropriate tweaks to match\nthe start and end parameters from the user request).\n\n# How To\n\nYou must have an S3 bucket set up, and AWS credentials for a role that has read\nand write access to that S3 bucket. CTile uses the AWS Go SDK with the default\ncredential provider, and so will [pull credential\ninformation](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials)\nfrom environment variables, an AWS config file, or ambient credentials for an\nEC2 instance. You'll need to manually specify the AWS region for your S3 bucket\nby setting the environment variable AWS_REGION.\n\nYou must also know the maximum get-entries size for the log you are mirroring.\nIf you operate the log, you will know this from your own configs. Otherwise, you\ncan figure it out by making a get-entries request with `end` much larger than\n`start`, and counting the entries. You should set CTile's tile-size to exactly\nequal this maximum. It's possible to set a tile-size lower, but only if the log\nis not using `align_getentries`.\n\nExample invocation:\n\n```\nexport AWS_REGION=us-west-2\ngo run . -log-url https://oak.ct.letsencrypt.org/2023 \\\n    -tile-size 256 -s3-bucket some-bucket -full-request-timeout 30s -s3-prefix oak2023\n```\n\n```\ncurl 'localhost:8080/ct/v1/get-entries?start=0\u0026end=999999999' -i  | less\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fletsencrypt%2Fctile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fletsencrypt%2Fctile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fletsencrypt%2Fctile/lists"}