{"id":29021636,"url":"https://github.com/scriptedalchemy/dictionary-compression","last_synced_at":"2025-06-26T02:07:11.974Z","repository":{"id":301057029,"uuid":"1008024013","full_name":"ScriptedAlchemy/dictionary-compression","owner":"ScriptedAlchemy","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-25T00:12:43.000Z","size":217594,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-25T00:24:22.194Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ScriptedAlchemy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-24T22:56:10.000Z","updated_at":"2025-06-25T00:12:47.000Z","dependencies_parsed_at":"2025-06-25T00:34:59.557Z","dependency_job_id":null,"html_url":"https://github.com/ScriptedAlchemy/dictionary-compression","commit_stats":null,"previous_names":["scriptedalchemy/dictionary-compression"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ScriptedAlchemy/dictionary-compression","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScriptedAlchemy%2Fdictionary-compression","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScriptedAlchemy%2Fdictionary-compression/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScriptedAlchemy%2Fdictionary-compression/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScriptedAlchemy%2Fdictionary-compression/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ScriptedAlchemy","download_url":"https://codeload.github.com/ScriptedAlchemy/dictionary-compression/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScriptedAlchemy%2Fdictionary-compression/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261984646,"owners_count":23240304,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-26T02:07:08.640Z","updated_at":"2025-06-26T02:07:11.957Z","avatar_url":"https://github.com/ScriptedAlchemy.png","language":"TypeScript","readme":"# Dictionary Compression Toolkit\n\nA comprehensive suite of tools for experimenting with HTTP compression dictionaries, including dynamic dictionary generation, compression testing, and proxy servers.\n\n## 🌟 Featured: Compression Dictionary Testing Service\n\n### **[dictionary-proxy/compression-dictionary/](./dictionary-proxy/compression-dictionary/)**\n\nA powerful Node.js web application for testing compression dictionary effectiveness with **recent major improvements**:\n\n#### ✅ **Recent Improvements (January 2025)**\n- **🚀 60-second timeout** (vs 1-second) for proper multi-URL fetching\n- **📏 Configurable dictionary sizes** up to 1MB (removed 64KB hardcoded limit)\n- **🔧 Minimum dictionary size option** to prevent unwanted auto-reduction\n- **📊 Organized results by dictionary type** (side-by-side brotli/zstd comparisons)\n- **🧪 Updated comprehensive test suite** for new functionality\n\n#### **Features:**\n- **Dynamic Dictionary Generation** (`/generate/`) - Creates custom compression dictionaries from multiple URLs\n- **Static Dictionary Testing** (`/static/`) - Tests using previous file versions as compression dictionaries\n- **Multi-Algorithm Support** - Brotli and Zstandard compression with various levels\n- **Anonymous URL Fetching** - Custom User-Agent strings and rate limiting\n- **Comprehensive Testing** - Compares dictionary-based vs standard compression\n\n#### **Quick Start:**\n```bash\ncd dictionary-proxy/compression-dictionary\npnpm install\npnpm run dev  # Starts on http://localhost:4200\n```\n\n**Expected Results:** 15-40% compression improvement over standard algorithms\n\n---\n\n## 🛠️ Additional Tools\n\n### **[dictionary-proxy/](./dictionary-proxy/)**\nCloudflare worker for experimenting with compression dictionaries on arbitrary sites through proxying.\n\n### **[express-proxy-server/](./express-proxy-server/)**\nProduction-ready Express.js proxy server with compression middleware and dictionary support.\n\n### **[reverse-proxy-worker/](./reverse-proxy-worker/)**\nCloudflare Worker implementation for reverse proxy with compression dictionary support.\n\n### **[development-tools/](./development-tools/)**\nTesting utilities, benchmarking scripts, and Cypress end-to-end tests for compression validation.\n\n### **[demo-site/](./demo-site/)**\nReact-based demonstration site showcasing dictionary compression in action.\n\n---\n\n## 🧰 Compression Libraries\n\n### **[brotli-wasm-compress/](./brotli-wasm-compress/) \u0026 [brotli-wasm-compress-rust/](./brotli-wasm-compress-rust/)**\nWebAssembly builds of Brotli compression with dictionary support:\n- C++ implementation with Emscripten\n- Rust implementation with wasm-pack\n- Browser and Node.js compatibility\n\n### **[zstd-wasm-compress/](./zstd-wasm-compress/)**\nWebAssembly build of Zstandard compression with dictionary training capabilities.\n\n---\n\n## 📋 Prerequisites\n\n- **Node.js 20+**\n- **pnpm 8.0+**\n- **Compression Tools**: `brotli`, `zstd`, `dictionary_generator`\n- **Chrome 130+** with experimental flags (for browser testing):\n  - `chrome://flags/#enable-compression-dictionary-transport`\n  - `chrome://flags/#enable-compression-dictionary-transport-backend`\n\n## 🚀 Quick Start\n\n```bash\n# Clone the repository\ngit clone https://github.com/ScriptedAlchemy/dictionary-compression.git\ncd dictionary-compression\n\n# Install dependencies\npnpm install\n\n# Start the main compression dictionary service\ncd dictionary-proxy/compression-dictionary\npnpm run dev\n```\n\nVisit `http://localhost:4200` to access the compression dictionary testing interface.\n\n## 📖 Documentation\n\nEach tool includes comprehensive documentation:\n- **[Compression Dictionary Service](./dictionary-proxy/compression-dictionary/README.md)** - Main testing application\n- **[Dictionary Proxy](./dictionary-proxy/README.md)** - Cloudflare worker setup\n- **[Workflow Analysis](./dictionary-proxy/compression-dictionary/WORKFLOW_ANALYSIS.md)** - Detailed technical analysis\n\n## 🔧 Development\n\n```bash\n# Start main service with auto-reload\ncd dictionary-proxy/compression-dictionary\npnpm run dev\n\n# Run comprehensive tests\npnpm test\n\n# Start Express proxy server\ncd ../express-proxy-server\nnode server.js\n```\n\n## 📊 Performance\n\nDictionary compression can achieve **15-40% better compression** than standard algorithms when properly configured:\n\n- **Bootstrap CSS (232KB)**: 92.3% compression vs 90.2% with Brotli alone\n- **Bootstrap JS (80KB)**: 87.3% compression vs 73.8% with Brotli alone\n- **Optimal for**: Repeated patterns in HTML, CSS, JavaScript, and JSON\n\n## 🌐 Browser Support\n\n- **Chrome 130+**: Full compression dictionary transport support (experimental)\n- **Other browsers**: Graceful fallback to standard compression algorithms\n\n## 📚 Technical Standards\n\n- [HTTP Compression Dictionary Transport](https://datatracker.ietf.org/doc/draft-ietf-httpbis-compression-dictionary/)\n- [Brotli RFC 7932](https://tools.ietf.org/html/rfc7932)\n- [Zstandard RFC 8878](https://tools.ietf.org/html/rfc8878)\n\n## 📄 License\n\nApache 2.0 License - see [LICENSE](./LICENSE) for details.\n\n---\n\n**⭐ Star this repo if you find it useful for compression dictionary experimentation!**","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscriptedalchemy%2Fdictionary-compression","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscriptedalchemy%2Fdictionary-compression","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscriptedalchemy%2Fdictionary-compression/lists"}