{"id":28512224,"url":"https://github.com/barre/compact_log","last_synced_at":"2026-03-02T21:37:14.003Z","repository":{"id":296902760,"uuid":"994833903","full_name":"Barre/compact_log","owner":"Barre","description":"A dual-API Certificate Transparency (CT) log implementation. CompactLog serves the same Merkle tree through both the RFC 6962 Certificate Transparency API and the Static CT API","archived":false,"fork":false,"pushed_at":"2025-06-25T08:22:26.000Z","size":3789,"stargazers_count":43,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-25T08:34:28.798Z","etag":null,"topics":["certificate","certificate-transparency","certificate-transparency-logs","certificates","rfc6962","tlog","x509"],"latest_commit_sha":null,"homepage":"https://www.merklemap.com","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Barre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-02T14:51:11.000Z","updated_at":"2025-06-25T08:22:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"60d9af54-da52-459a-8f3b-5238841bb476","html_url":"https://github.com/Barre/compact_log","commit_stats":null,"previous_names":["barre/compact_log"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Barre/compact_log","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Barre%2Fcompact_log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Barre%2Fcompact_log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Barre%2Fcompact_log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Barre%2Fcompact_log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Barre","download_url":"https://codeload.github.com/Barre/compact_log/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Barre%2Fcompact_log/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263398571,"owners_count":23460546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["certificate","certificate-transparency","certificate-transparency-logs","certificates","rfc6962","tlog","x509"],"created_at":"2025-06-09T00:37:12.294Z","updated_at":"2026-03-02T21:37:13.941Z","avatar_url":"https://github.com/Barre.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/readme_illustration.png\" alt=\"README Illustration\" style=\"border-radius: 10px;\" width=\"600\" /\u003e\n\u003c/p\u003e\n\n**⚠️ This is a work in progress. While the CT functionality works, this should not be yet used in production.**\n\n# CompactLog\n\nA dual-API Certificate Transparency (CT) log implementation. CompactLog serves the same Merkle tree through both the RFC 6962 Certificate Transparency API and the Static CT API (C2SP draft), built on SlateDB to explore how LSM-tree storage can address traditional CT log scalability challenges.\n\n## Live Instance\n\nCompactLog is running in production with live endpoints available:\n\n- **RFC 6962 API**: https://compact-log.ct.merklemap.com/ct/v1/get-sth\n- **Static CT API**: https://compact-log.ct.merklemap.com/checkpoint\n\n### Live Monitoring\n\nView real-time performance metrics and system health:\n\n**[Live Monitoring Dashboard](https://dashboards.merklemap.com/public-dashboards/0ba4b40dda0449d7ab29ac8d6be90dd9?from=now-1h\u0026to=now\u0026timezone=browser)**\n\n## Performance Benchmarks\n\nCompactLog delivers exceptional performance compared to newer Static CT API-only implementations, while uniquely providing both RFC 6962 and Static CT APIs from the same server. Recent benchmarks demonstrate significant advantages over dedicated static-only CT servers:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/latency_percentiles_1000.png\" alt=\"Latency Percentiles at 1000 Connections\" width=\"48%\" /\u003e\n  \u003cimg src=\"assets/average_latency.png\" alt=\"Average Latency Scaling\" width=\"48%\" /\u003e\n\u003c/p\u003e\n\n**Key Performance Highlights:**\n- **2.5-4x lower latency** compared to Sunlight (static-only)\n- **4-5x lower latency** compared to Cloudflare's Azul (static-only)\n- **Linear scalability** with connection count, maintaining consistent performance under load\n- All while serving both RFC 6962 and Static CT APIs simultaneously\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/throughput_comparison.png\" alt=\"Throughput Comparison\" width=\"48%\" /\u003e\n  \u003cimg src=\"assets/error_rates.png\" alt=\"Error Rates\" width=\"48%\" /\u003e\n\u003c/p\u003e\n\n**Reliability \u0026 Throughput:**\n- **Zero errors** across all connection levels (50, 1000, 3000 concurrent connections)\n- **17x higher throughput** than Cloudflare Azul at 1000 connections\n- **1.5x higher throughput** than Sunlight at 3000 connections\n\nThese benchmarks demonstrate that CompactLog's dual-API architecture doesn't compromise performance. By leveraging efficient batching and LSM-tree storage, it delivers better performance than specialized static-only implementations while providing the flexibility of both APIs.\n\n## Overview\n\nThis implementation provides a complete Certificate Transparency log that:\n\n- Accepts X.509 certificate chains and pre-certificates\n- Issues Signed Certificate Timestamps (SCTs)\n- Maintains a cryptographically verifiable Merkle tree\n- Provides inclusion and consistency proofs via RFC 6962 API\n- Serves the same tree data through Static CT API (C2SP draft)\n- Stores data in cloud object storage (S3, Azure Blob) or local filesystem\n\n## Performance Metrics\n\nCompactLog achieves exceptional performance through its batching strategy and efficient storage design:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/write_throughput.png\" alt=\"Write Throughput\" width=\"100%\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eSustained write throughput of 3-4K requests/second across different certificate chain types\u003c/em\u003e\n\u003c/p\u003e\n\n### Key Performance Characteristics\n\n- **Zero Merge Delay**: Certificates are immediately incorporated into the Merkle tree\n- **High Throughput**: Sustained 3-4K certificate submissions per second\n- **Low Latency**: P95 latencies under 500ms, P99 under 1 second\n- **Efficient Deduplication**: ~67% certificate chain deduplication rate\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/system_metrics.png\" alt=\"System Metrics\" width=\"100%\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eReal-time system metrics showing tree growth, SCT issuance success rate, and zero storage queue backlog\u003c/em\u003e\n\u003c/p\u003e\n\n## Storage Architecture\n\n### Core Design Decisions\n\nCompactLog makes three fundamental design choices that differentiate it from other CT implementations:\n\n1. **LSM-tree storage via SlateDB** instead of relational databases or custom storage engines.\n2. **STH-boundary versioning** - only persisting tree state at published checkpoints.\n3. **Synchronous tree updates** - achieving a Maximum Merge Delay (MMD) of 0 seconds.\n\n### How MMD is Eliminated Entirely\n\nMany CT log implementations have a Merge Delay (MD) of minutes to hours, where submitted certificates aren't yet included in the Merkle tree. This exists because:\n\n- Many implementations issue SCTs immediately, then incorporate certificates later via background processes.\n- Some implementations have expensive tree update operations.\n- Consistency requires coordinating distributed components.\n\nCompactLog eliminates a MD by reversing this order - certificates are incorporated **before** SCTs are issued:\n\n```\nSubmission 1 ─┐\nSubmission 2 ─┼─ Wait up to 500ms ─→ Batch tree update ─→ All SCTs returned\nSubmission 3 ─┘                             └── Certificates already incorporated\n```\n\nThe 500ms delay is submission latency, not a merge delay. Once SCTs are issued, certificates are already in the tree.\n\n### Batching Strategy in Action\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/write_requests_detailed.png\" alt=\"Batching Strategy\" width=\"100%\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eWrite request patterns showing effective batching - spikes at the beginning represent batch formation\u003c/em\u003e\n\u003c/p\u003e\n\nThe batching system:\n\n- Collects submissions for up to 500ms (configurable) to form a batch\n- Updates the Merkle tree once for the entire batch\n- Returns SCTs only after certificates are incorporated in the tree\n- No background processing - certificates are immediately available for proofs\n\n### Request Latency Profile\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/request_latency.png\" alt=\"Request Latency\" width=\"100%\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eP95 and P99 latencies showing consistent sub-second response times across all operation types\u003c/em\u003e\n\u003c/p\u003e\n\nThe latency profile demonstrates:\n\n- **P95 add-chain**: 471ms (includes batching delay)\n- **P95 add-pre-chain**: 513ms\n- **P95 get operations**: 52-68ms\n- **P99 latencies**: Generally under 1 second\n\n### Traditional vs CompactLog Timing\n\n**Traditional CT implementations:**\n\n```\nSubmit cert → Issue SCT immediately → [MMD period] → Incorporate in tree\n```\n\n**CompactLog:**\n\n```\nSubmit cert → [Batch delay ≤500ms] → Incorporate in tree → Issue SCT\n```\n\nResult: Traditional logs have MMD measured in minutes/hours; CompactLog has an MMD of 0 seconds.\n\n### STH-Boundary Versioning\n\nCompactLog versions nodes only at STH publication boundaries:\n\n- Update nodes in-memory during batch operations  \n- Store O(log n) versioned nodes only at STH publication\n- With STHs every k certificates: reduces versioned storage from O(n log n) to O(n log n / k)\n\n**Example**: Publishing STHs every 1000 certificates reduces versioned storage overhead by 1000x.\n\n### Storage Schema\n\n```\n# Core data\nleaf:{index} → certificate/precert data\nvnode:{node}@{version} → node hash (version = STH boundary)\nnver:{node} → latest version of node\n\n# Operational state\nmeta → current tree size\ncommitted_size → last STH boundary\nhash:{leaf_hash} → tree index\ncert_sct:{cert_hash} → SCT data\n\n# Certificate storage (deduplication)\ncert:{cert_hash} → certificate binary data\nentry:{index} → deduplicated log entry\n```\n\n### Certificate Chain Deduplication\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/deduplication_rate.png\" alt=\"Certificate Deduplication\" width=\"100%\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eConsistent ~67% deduplication rate for certificate chains, significantly reducing storage requirements\u003c/em\u003e\n\u003c/p\u003e\n\nCompactLog stores certificate chains using content-addressable storage:\n\n1. **Entry structure**: Each log entry stores SHA-256 hashes of certificates rather than the certificates themselves\n2. **Certificate store**: Certificates are stored separately under `cert:{hash}` keys\n3. **Deduplication**: Multiple entries referencing the same certificate (e.g., intermediate CA certs) share the same stored copy\n4. **Reconstruction**: The API reconstructs full certificate chains by resolving hash references during retrieval\n\nThe `DeduplicatedLogEntry` structure contains:\n\n- Certificate hash (32 bytes)\n- Chain certificate hashes (array of 32-byte hashes)\n- Original metadata (timestamp, index, entry type)\n\n### Consistency Model\n\nEvery operation maintains strict consistency:\n\n- Reads see the latest committed STH state\n- Writes are serialized through asynchronous locking (synchronous from client perspective)\n- Proofs only available at STH boundaries (ensuring stable references)\n- No eventual consistency - all operations are immediately visible\n\n## Architectural Approach\n\n### The Static CT API Design Challenge\n\nThe Static CT API was designed to serve immutable tiles from simple storage (like CDNs), but implementations still face fundamental challenges:\n\n1. **Sequencing**: Certificates must be assigned sequential indices\n2. **Deduplication**: Preventing duplicate entries requires state\n3. **Coordination**: Multiple writers need consistency guarantees\n4. **Atomicity**: Tiles must reflect a consistent tree state\n\nMany implementations require or heavily depend on external databases for:\n- Deduplication tracking and caching\n- Write coordination between multiple processes\n- Sequencing and state management\n- Staging and crash recovery mechanisms\n\n### CompactLog's Approach\n\nCompactLog uses a single LSM-tree database (SlateDB) that stores its data directly on cloud object storage (S3, Azure Blob) or POSIX filesystem. The same database handles:\n\n- Certificate sequencing through atomic batch operations\n- Deduplication via content-addressable storage keys\n- Tree versioning at STH boundaries\n- On-demand tile generation from stored nodes\n\nWhile CompactLog generates tiles dynamically rather than storing pre-computed files, its underlying storage consists entirely of immutable objects in cloud storage or append-only files on disk. All coordination, deduplication, and sequencing happen through this single storage layer without requiring additional databases or complex write pipelines.\n\nThis design trades direct CDN serving of pre-generated tiles for a much simpler write path and operational model. The LSM-tree structure naturally fits CT's workload pattern - primarily appending new certificates while efficiently serving historical tree states at any STH boundary - all while leveraging the same cloud object storage that would typically host static tiles.\n\n## Configuration\n\nCreate `Config.toml` or let the system generate defaults:\n\n```toml\n[server]\nbind_addr = \"0.0.0.0:8080\"\n\n[storage]\nprovider = \"local\"  # \"aws\", \"azure\", or \"local\"\n\n[storage.local]\npath = \"/tmp/ct-log-storage\"\n\n[keys]\nprivate_key_path = \"keys/private_key.pem\"\npublic_key_path = \"keys/public_key.pem\"\n```\n\nFor cloud storage, configure provider-specific credentials in the respective sections.\n\n## Running\n\n```bash\n# Start with default local configuration\ncargo run --release\n\n# With debug logging\nRUST_LOG=debug cargo run --release\n```\n\nThe system automatically generates ECDSA P-256 keys and default configuration if not present.\n\n## API Endpoints\n\nCompactLog exposes both RFC 6962 and Static CT API endpoints on the same server:\n\n### RFC 6962 API\n- `POST /ct/v1/add-chain` - Submit certificate chain\n- `POST /ct/v1/add-pre-chain` - Submit pre-certificate chain  \n- `GET /ct/v1/get-sth` - Get signed tree head\n- `GET /ct/v1/get-entries` - Get log entries\n- `GET /ct/v1/get-proof-by-hash` - Get inclusion proof by hash\n- `GET /ct/v1/get-entry-and-proof` - Get entry and inclusion proof\n- `GET /ct/v1/get-sth-consistency` - Get consistency proof\n- `GET /ct/v1/get-roots` - Get accepted root certificates\n\n### Static CT API (C2SP)\n- `GET /checkpoint` - Get current checkpoint (signed note format)\n- `GET /tile/{level}/{index}` - Get Merkle tree tile\n- `GET /tile/data/{index}` - Get data tile (gzip compressed)\n- `GET /issuer/{fingerprint}` - Get issuer certificate by SHA-256 fingerprint\n\nBoth APIs serve the exact same Merkle tree data, just in different formats suited to their respective use cases.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarre%2Fcompact_log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbarre%2Fcompact_log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarre%2Fcompact_log/lists"}