{"id":47770779,"url":"https://github.com/compiler-explorer/ce-win-file-cache","last_synced_at":"2026-04-03T09:09:30.717Z","repository":{"id":318380787,"uuid":"1000527947","full_name":"compiler-explorer/ce-win-file-cache","owner":"compiler-explorer","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-07T19:51:59.000Z","size":5910,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-01T09:53:58.507Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/compiler-explorer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-11T23:45:02.000Z","updated_at":"2025-10-07T19:51:37.000Z","dependencies_parsed_at":"2025-10-06T22:09:41.652Z","dependency_job_id":"ce89e9ea-3d2d-451f-947a-279542cde67c","html_url":"https://github.com/compiler-explorer/ce-win-file-cache","commit_stats":null,"previous_names":["compiler-explorer/ce-win-file-cache"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/compiler-explorer/ce-win-file-cache","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-win-file-cache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-win-file-cache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-win-file-cache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-win-file-cache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/compiler-explorer","download_url":"https://codeload.github.com/compiler-explorer/ce-win-file-cache/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compiler-explorer%2Fce-win-file-cache/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31344635,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T08:03:20.796Z","status":"ssl_error","status_checked_at":"2026-04-03T08:00:37.834Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-03T09:09:28.797Z","updated_at":"2026-04-03T09:09:30.707Z","avatar_url":"https://github.com/compiler-explorer.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CeWinFileCacheFS\n\n[![Tests](https://github.com/compiler-explorer/ce-win-file-cache/actions/workflows/test.yml/badge.svg)](https://github.com/compiler-explorer/ce-win-file-cache/actions/workflows/test.yml)\n[![Code Quality](https://github.com/compiler-explorer/ce-win-file-cache/actions/workflows/code-quality.yml/badge.svg)](https://github.com/compiler-explorer/ce-win-file-cache/actions/workflows/code-quality.yml)\n\nA WinFsp-based hybrid cache filesystem for Compiler Explorer, designed to efficiently serve multiple MSVC compiler versions by caching frequently-used files locally while falling back to network shares for less-critical files.\n\n## Features\n\n- **Hybrid Caching**: Intelligent caching of compiler executables and headers with LRU eviction\n- **Async Download Manager**: Multi-threaded download system with configurable worker pools\n- **Prometheus Metrics**: Comprehensive metrics collection for monitoring cache performance\n- **Always-Cached Directory Tree**: Fast directory navigation with complete metadata caching\n- **Memory Cache Manager**: High-performance in-memory caching with configurable size limits\n- **JSON Configuration**: Flexible configuration with support for multiple compiler versions\n- **Cross-Platform Development**: Windows target with macOS development and testing support\n\n## Prerequisites\n\n### For Production (Windows)\n- Windows 10/11 or Windows Server 2016+\n- [WinFsp](https://github.com/winfsp/winfsp) installed\n- Visual Studio 2019+ or compatible C++ compiler with C++20 support\n- CMake 3.20+\n\n### For Development (macOS/Linux)\n- C++20 compatible compiler (GCC 10+, Clang 12+)\n- CMake 3.20+\n- prometheus-cpp (automatically downloaded via FetchContent)\n\n## Building\n\n### Native Windows Build (Recommended)\n\n1. Clone this repository with submodules:\n```cmd\ngit clone --recursive https://github.com/your-org/ce-win-file-cache.git\ncd ce-win-file-cache\ngit submodule update --init --recursive\n```\n\n2. Install [WinFsp](https://github.com/winfsp/winfsp/releases) if not already installed\n\n3. Run the native Windows build:\n```cmd\nbuild-msvc.bat\n```\n\nThis will:\n- Set up the MSVC environment automatically\n- Build with optimizations enabled\n- Copy required DLLs and config files to the output directory\n\n### Building from WSL with MSVC\n\nIf developing from WSL, you can cross-compile using MSVC:\n\n1. **Important**: Source must be on a Windows drive (e.g., `/mnt/d/` or `/mnt/c/`)\n2. Run the WSL build script:\n```bash\n./build-msvc.sh\n```\n\n### Alternative Build Methods\n\n#### Manual CMake Build\n```cmd\n# Set up MSVC environment first (run from Developer Command Prompt)\nmkdir build \u0026\u0026 cd build\ncmake .. -G \"Visual Studio 17 2022\" -A x64\ncmake --build . --config Release\n```\n\n#### Wine Build (Experimental)\nFor cross-platform development or CI on Linux:\n```bash\n./build-wine.sh\nwine build-wine/bin/CeWinFileCacheFS.exe --help\n```\n\n### macOS/Linux Development Build\n\nFor cross-platform development and testing:\n\n```bash\n# Build with integrated static analysis\n./build-macos.sh\n\n# Run comprehensive test suite\n./run_all_tests.sh\n\n# Available options\n./run_all_tests.sh --help     # Show usage information  \n./run_all_tests.sh --clean    # Clean build before testing\n./run_all_tests.sh --quick    # Skip CMake configuration\n```\n\nThe `build-macos.sh` script includes integrated clang-tidy static analysis (if available) to ensure code quality and catch potential issues during development.\n\nThe test runner builds and executes 12 comprehensive test programs:\n- Cache operations and performance validation\n- Async download manager with stress testing\n- Prometheus metrics collection and validation  \n- Directory tree caching and navigation\n- Configuration loading and validation\n- Edge case handling and error scenarios\n- Glob pattern matching with comprehensive unit tests\n- JSON configuration parsing and validation\n\n### Testing the Build\n\nAfter building, test the installation:\n\n**On Windows:**\n```cmd\n# Test config parsing\n.\\build-msvc\\bin\\CeWinFileCacheFS.exe --test-config\n\n# Test path resolution\n.\\build-msvc\\bin\\CeWinFileCacheFS.exe --test-paths\n\n# Test network mapping\n.\\build-msvc\\bin\\CeWinFileCacheFS.exe --test-network\n\n# Run all tests\n.\\build-msvc\\bin\\CeWinFileCacheFS.exe --test\n```\n\n**On macOS/Linux:**\n```bash\n# Run comprehensive test suite\n./run_all_tests.sh\n```\n\n## Configuration\n\nCreate a `compilers.json` file (see included example):\n\n```json\n{\n  \"global\": {\n    \"total_cache_size_mb\": 8192,\n    \"eviction_policy\": \"lru\",\n    \"cache_directory\": \"D:\\\\CompilerCache\",\n    \"download_threads\": 6\n  },\n  \"metrics\": {\n    \"enabled\": true,\n    \"bind_address\": \"127.0.0.1\",\n    \"port\": 8080,\n    \"endpoint_path\": \"/metrics\"\n  },\n  \"compilers\": {\n    \"msvc-14.40\": {\n      \"network_path\": \"\\\\\\\\127.0.0.1\\\\efs\\\\compilers\\\\msvc\\\\14.40.33807-14.40.33811.0\",\n      \"cache_size_mb\": 2048,\n      \"cache_always\": [\n        \"bin/Hostx64/x64/*.exe\",\n        \"bin/Hostx64/x64/*.dll\",\n        \"include/**/*.h\",\n        \"lib/x64/*.lib\"\n      ],\n      \"prefetch_patterns\": [\n        \"include/**/*.h\",\n        \"include/**/*.hpp\"\n      ]\n    },\n    \"windows-kits-10\": {\n      \"network_path\": \"\\\\\\\\127.0.0.1\\\\efs\\\\compilers\\\\windows-kits-10\",\n      \"cache_size_mb\": 1024,\n      \"cache_always\": [\n        \"Include/**/*.h\",\n        \"Lib/**/*.lib\",\n        \"bin/**/*.exe\"\n      ],\n      \"prefetch_patterns\": [\n        \"Include/**/*.h\"\n      ]\n    },\n    \"ninja\": {\n      \"network_path\": \"\\\\\\\\127.0.0.1\\\\efs\\\\compilers\\\\ninja\",\n      \"cache_size_mb\": 64,\n      \"cache_always\": [\n        \"*.exe\"\n      ],\n      \"prefetch_patterns\": []\n    }\n  }\n}\n```\n\n## Quick Start\n\nFor detailed installation and configuration instructions, see the **[Installation \u0026 Usage Guide](docs/INSTALLATION_AND_USAGE.md)**.\n\n### Basic Usage\n\n1. **Install WinFsp** from [GitHub Releases](https://github.com/winfsp/winfsp/releases)\n2. **Configure** your compilers in `compilers.json` (see example below)\n3. **Mount** the filesystem:\n   ```cmd\n   CeWinFileCacheFS.exe --config compilers.json --mount M:\n   ```\n4. **Access** your compilers through the cached mount point (e.g., `M:\\msvc-14.40\\bin\\cl.exe`)\n\n### Example: Network Share to Cached Path\n\n| Network Location | Cached Path |\n|-----------------|-------------|\n| `Z:\\compilers\\msvc\\14.40\\bin\\cl.exe` | `M:\\msvc-14.40\\bin\\cl.exe` |\n| `\\\\server\\tools\\ninja.exe` | `M:\\tools\\ninja.exe` |\n\nFiles accessed through the cached paths are automatically cached for faster subsequent access.\n\n### Command Line Options\n\n#### Runtime Options\n- `-c, --config FILE`: Configuration file (default: compilers.json)\n- `-m, --mount POINT`: Mount point (default: M:)\n- `-u, --volume-prefix`: Volume prefix for UNC paths\n- `-d, --debug [LEVEL]`: Enable debug logging\n- `-h, --help`: Show help message\n\n#### Testing Options (No WinFsp required)\n- `-t, --test`: Run all test modes without mounting\n- `--test-config`: Test configuration file parsing only\n- `--test-paths`: Test virtual path to network path resolution\n- `--test-network`: Test network path mapping validation\n\n### Example Usage\n\n```cmd\n# Test configuration before mounting\nCeWinFileCacheFS.exe --test\n\n# Mount to drive M: with default config\nCeWinFileCacheFS.exe\n\n# Mount to specific directory with custom config\nCeWinFileCacheFS.exe --config my-compilers.json --mount C:\\\\compilers\n\n# Enable debug logging\nCeWinFileCacheFS.exe --debug --mount M:\n\n# Test specific functionality\nCeWinFileCacheFS.exe --test-paths --config compilers.json\n```\n\n## Architecture\n\n### File States\n\n- **VIRTUAL**: File exists in metadata only\n- **CACHED**: File is stored locally\n- **PLACEHOLDER**: Metadata exists, content fetched on demand\n- **FETCHING**: Currently downloading from network\n- **NETWORK_ONLY**: Always fetch from network\n\n### Cache Policies\n\n- **ALWAYS_CACHE**: Permanently cache these files\n- **ON_DEMAND**: Cache after first access\n- **NEVER_CACHE**: Always fetch from network\n\n### Directory Structure\n\n```\n/\n├── msvc-14.40/\n│   ├── bin/Hostx64/x64/    # Executables (always cached)\n│   ├── include/            # Headers (on-demand caching)\n│   └── lib/x64/           # Libraries (on-demand caching)\n├── windows-kits-10/\n│   ├── Include/           # Windows SDK headers\n│   ├── Lib/              # Windows SDK libraries\n│   └── bin/              # SDK tools\n└── ninja/\n    └── ninja.exe         # Build system executable\n```\n\n## Performance Considerations\n\n- Executables (`*.exe`, `*.dll`) are cached on first access\n- Headers are cached based on usage patterns\n- Cache eviction uses LRU algorithm\n- Background prefetching for common files\n- Network timeouts handled gracefully\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Mount fails**: Ensure WinFsp is installed and you have admin privileges\n2. **Network access denied**: Check credentials and network share permissions\n3. **Cache full**: Increase `total_cache_size_mb` or clear cache directory\n\n### Debug Logging\n\nEnable debug logging for troubleshooting:\n```cmd\nCeWinFileCacheFS.exe --debug 1 --mount M:\n```\n\n### Cache Management\n\nClear cache manually:\n```cmd\nrmdir /s /q C:\\CompilerCache\n```\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests if applicable\n5. Submit a pull request\n\n### Continuous Integration\n\nThe project includes comprehensive CI/CD workflows:\n\n- **Linux CI**: Runs comprehensive test suite on every push/PR with multiple Ubuntu versions and compilers\n- **Windows CI**: Full WinFsp integration testing with MSVC builds on Windows runners  \n- **Code Quality**: Static analysis, warning checks, and documentation validation\n\nAll tests run on their target platforms - Linux tests cover core logic without WinFsp dependencies, while Windows tests validate the complete WinFsp integration.\n\n## Documentation\n\nComprehensive documentation is available in the [`docs/`](docs/) directory:\n\n- **[Installation \u0026 Usage Guide](docs/INSTALLATION_AND_USAGE.md)** - Complete setup and configuration guide\n- **[Architecture Overview](docs/ARCHITECTURE.md)** - System design, components, and visual diagrams\n- **[Windows CI Pipeline](docs/WINDOWS_CI_PLAN.md)** - Complete CI/CD implementation details\n- **[Testing Guide](docs/TESTING.md)** - Test framework and validation procedures\n- **[Caching Design](docs/CACHING_DESIGN.md)** - Cache algorithms and policies\n- **[Async Download Flow](docs/ASYNC_DOWNLOAD_FLOW.md)** - Multi-threaded download system\n- **[Development Setup](docs/REMOTE_DEV_GUIDE.md)** - Remote development configuration\n- **[TODO List](docs/TODO.md)** - Implementation roadmap and pending items\n\n## Monitoring and Metrics\n\nThe system provides comprehensive Prometheus metrics for monitoring cache performance:\n\n### Available Metrics\n\n- **Cache metrics**: Hit/miss rates, cache size, eviction counts\n- **Download metrics**: Queue depth, completion rates, failure reasons, duration histograms\n- **Filesystem metrics**: Operation counts and file open duration\n- **Network metrics**: Operation success rates and latency\n\n### Accessing Metrics\n\nWhen metrics are enabled in configuration:\n```bash\n# View metrics in Prometheus format\ncurl http://127.0.0.1:8080/metrics\n```\n\nMetrics include dynamic labels for detailed analysis:\n- Cache hits/misses by operation type\n- Download failures by specific reason\n- Network operations by success/failure status\n\n## Current Implementation Status\n\n### ✅ Completed Components\n- **Memory Cache Manager**: Full LRU caching with metrics integration\n- **Async Download Manager**: Multi-threaded downloads with comprehensive testing\n- **Directory Tree Caching**: Always-cached directory structure for fast navigation\n- **Prometheus Metrics**: Complete metrics collection with dynamic labels\n- **JSON Configuration**: Full configuration parsing and validation\n- **Glob Pattern Matching**: Proper glob matching for file patterns (*, **, ?)\n- **Test Infrastructure**: Comprehensive test suite with automated runner\n- **Windows CI/CD Pipeline**: Complete MSVC build and WinFsp integration testing\n- **Cross-Platform Development**: Linux development with Windows production target\n- **Security Hardening**: Comprehensive memory safety analysis and fixes\n- **Thread Safety**: Per-node locking for concurrent directory operations\n- **Static Analysis Integration**: clang-tidy checks integrated into build process\n\n### 📝 Remaining Work\n- **WinFsp Filesystem Operations**: Complete file read/write operations\n- **Production Deployment**: Enhanced logging, error recovery, and monitoring\n- **Performance Optimization**: Profile and optimize cache algorithms\n\n## Roadmap\n\n- [ ] WinFsp filesystem driver integration\n- [ ] Write support for compiler outputs  \n- [ ] File integrity verification\n- [ ] Compression for cached files\n- [ ] GUI configuration tool\n- [ ] Production logging and monitoring\n- [ ] Docker container support","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompiler-explorer%2Fce-win-file-cache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcompiler-explorer%2Fce-win-file-cache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompiler-explorer%2Fce-win-file-cache/lists"}