https://github.com/scriptedalchemy/dictionary-compression
https://github.com/scriptedalchemy/dictionary-compression
Last synced: 8 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/scriptedalchemy/dictionary-compression
- Owner: ScriptedAlchemy
- License: apache-2.0
- Created: 2025-06-24T22:56:10.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-06-25T00:12:43.000Z (8 months ago)
- Last Synced: 2025-06-25T00:24:22.194Z (8 months ago)
- Language: TypeScript
- Size: 208 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Dictionary Compression Toolkit
A comprehensive suite of tools for experimenting with HTTP compression dictionaries, including dynamic dictionary generation, compression testing, and proxy servers.
## 🌟 Featured: Compression Dictionary Testing Service
### **[dictionary-proxy/compression-dictionary/](./dictionary-proxy/compression-dictionary/)**
A powerful Node.js web application for testing compression dictionary effectiveness with **recent major improvements**:
#### ✅ **Recent Improvements (January 2025)**
- **🚀 60-second timeout** (vs 1-second) for proper multi-URL fetching
- **📏 Configurable dictionary sizes** up to 1MB (removed 64KB hardcoded limit)
- **🔧 Minimum dictionary size option** to prevent unwanted auto-reduction
- **📊 Organized results by dictionary type** (side-by-side brotli/zstd comparisons)
- **🧪 Updated comprehensive test suite** for new functionality
#### **Features:**
- **Dynamic Dictionary Generation** (`/generate/`) - Creates custom compression dictionaries from multiple URLs
- **Static Dictionary Testing** (`/static/`) - Tests using previous file versions as compression dictionaries
- **Multi-Algorithm Support** - Brotli and Zstandard compression with various levels
- **Anonymous URL Fetching** - Custom User-Agent strings and rate limiting
- **Comprehensive Testing** - Compares dictionary-based vs standard compression
#### **Quick Start:**
```bash
cd dictionary-proxy/compression-dictionary
pnpm install
pnpm run dev # Starts on http://localhost:4200
```
**Expected Results:** 15-40% compression improvement over standard algorithms
---
## 🛠️ Additional Tools
### **[dictionary-proxy/](./dictionary-proxy/)**
Cloudflare worker for experimenting with compression dictionaries on arbitrary sites through proxying.
### **[express-proxy-server/](./express-proxy-server/)**
Production-ready Express.js proxy server with compression middleware and dictionary support.
### **[reverse-proxy-worker/](./reverse-proxy-worker/)**
Cloudflare Worker implementation for reverse proxy with compression dictionary support.
### **[development-tools/](./development-tools/)**
Testing utilities, benchmarking scripts, and Cypress end-to-end tests for compression validation.
### **[demo-site/](./demo-site/)**
React-based demonstration site showcasing dictionary compression in action.
---
## 🧰 Compression Libraries
### **[brotli-wasm-compress/](./brotli-wasm-compress/) & [brotli-wasm-compress-rust/](./brotli-wasm-compress-rust/)**
WebAssembly builds of Brotli compression with dictionary support:
- C++ implementation with Emscripten
- Rust implementation with wasm-pack
- Browser and Node.js compatibility
### **[zstd-wasm-compress/](./zstd-wasm-compress/)**
WebAssembly build of Zstandard compression with dictionary training capabilities.
---
## 📋 Prerequisites
- **Node.js 20+**
- **pnpm 8.0+**
- **Compression Tools**: `brotli`, `zstd`, `dictionary_generator`
- **Chrome 130+** with experimental flags (for browser testing):
- `chrome://flags/#enable-compression-dictionary-transport`
- `chrome://flags/#enable-compression-dictionary-transport-backend`
## 🚀 Quick Start
```bash
# Clone the repository
git clone https://github.com/ScriptedAlchemy/dictionary-compression.git
cd dictionary-compression
# Install dependencies
pnpm install
# Start the main compression dictionary service
cd dictionary-proxy/compression-dictionary
pnpm run dev
```
Visit `http://localhost:4200` to access the compression dictionary testing interface.
## 📖 Documentation
Each tool includes comprehensive documentation:
- **[Compression Dictionary Service](./dictionary-proxy/compression-dictionary/README.md)** - Main testing application
- **[Dictionary Proxy](./dictionary-proxy/README.md)** - Cloudflare worker setup
- **[Workflow Analysis](./dictionary-proxy/compression-dictionary/WORKFLOW_ANALYSIS.md)** - Detailed technical analysis
## 🔧 Development
```bash
# Start main service with auto-reload
cd dictionary-proxy/compression-dictionary
pnpm run dev
# Run comprehensive tests
pnpm test
# Start Express proxy server
cd ../express-proxy-server
node server.js
```
## 📊 Performance
Dictionary compression can achieve **15-40% better compression** than standard algorithms when properly configured:
- **Bootstrap CSS (232KB)**: 92.3% compression vs 90.2% with Brotli alone
- **Bootstrap JS (80KB)**: 87.3% compression vs 73.8% with Brotli alone
- **Optimal for**: Repeated patterns in HTML, CSS, JavaScript, and JSON
## 🌐 Browser Support
- **Chrome 130+**: Full compression dictionary transport support (experimental)
- **Other browsers**: Graceful fallback to standard compression algorithms
## 📚 Technical Standards
- [HTTP Compression Dictionary Transport](https://datatracker.ietf.org/doc/draft-ietf-httpbis-compression-dictionary/)
- [Brotli RFC 7932](https://tools.ietf.org/html/rfc7932)
- [Zstandard RFC 8878](https://tools.ietf.org/html/rfc8878)
## 📄 License
Apache 2.0 License - see [LICENSE](./LICENSE) for details.
---
**⭐ Star this repo if you find it useful for compression dictionary experimentation!**