{"id":45345254,"url":"https://github.com/deeprave/pipedoc","last_synced_at":"2026-02-21T11:07:53.517Z","repository":{"id":306843159,"uuid":"1025521953","full_name":"deeprave/pipedoc","owner":"deeprave","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-27T23:14:24.000Z","size":129,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-28T01:08:53.170Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deeprave.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-24T11:34:53.000Z","updated_at":"2025-07-27T23:14:27.000Z","dependencies_parsed_at":"2025-07-28T01:23:27.000Z","dependency_job_id":null,"html_url":"https://github.com/deeprave/pipedoc","commit_stats":null,"previous_names":["deeprave/pipedoc"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/deeprave/pipedoc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeprave%2Fpipedoc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeprave%2Fpipedoc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeprave%2Fpipedoc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeprave%2Fpipedoc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deeprave","download_url":"https://codeload.github.com/deeprave/pipedoc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deeprave%2Fpipedoc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29679146,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T09:33:50.764Z","status":"ssl_error","status_checked_at":"2026-02-21T09:33:19.949Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-21T11:07:52.861Z","updated_at":"2026-02-21T11:07:53.511Z","avatar_url":"https://github.com/deeprave.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pipedoc\n\nA Python utility that serves concatenated markdown files through a named pipe to multiple processes simultaneously.\n\n## Features\n\n- **SOLID Architecture**: Refactored with Single Responsibility Principle, Open/Closed Principle, and Dependency Inversion\n- **Click CLI Interface**: Modern command-line interface with extensible commands\n- **Enterprise Logging System**: Comprehensive logging with multiple handlers, formats, and configuration options\n- **Version Management**: Detailed version information with multiple output formats and development tracking\n- **Comprehensive Testing**: Full test suite with pytest integration\n- **Package Structure**: Proper Python package with entry points for easy installation\n- **Recursive File Discovery**: Finds all markdown files in directory structure\n- **Content Processing**: Concatenates files with clear separators showing file paths\n- **Named Pipe Serving**: Creates named pipes for inter-process communication\n- **Multi-Client Support**: Serves content to multiple processes simultaneously using threads\n- **Connection Queueing**: FIFO queue for handling bursts of simultaneous connections\n- **Graceful Shutdown**: Handles signals and cleans up resources properly\n\n## Installation\n\n### From Source (Development)\n\n```bash\n# Clone or navigate to the project directory\ncd pipedoc\n\n# Install in development mode with uv\nuv sync\n\n# Or install with pip\npip install -e .\n```\n\n### From Package\n\n```bash\n# Install from PyPI (when published)\npip install pipedoc\n\n# Or with uv\nuv add pipedoc\n```\n\n## Usage\n\n### Basic Usage\n\n```bash\n# Serve markdown files from a directory\npipedoc ~/path/to/my/docs\n\n# With verbose output\npipedoc ~/path/to/my/docs --verbose\n\n# Show version\npipedoc --version\n\n# Show help\npipedoc --help\n```\n\n### Advanced Commands\n\n```bash\n# Get information about markdown files without serving\npipedoc info ~/path/to/my/docs\n\n# Alternative serve command\npipedoc serve ~/path/to/my/docs\n\n# Version management\npipedoc version                    # Show version information\npipedoc version --verbose          # Detailed version with Python and platform info\npipedoc version --json             # Machine-readable JSON output\n```\n\n### Logging Options\n\n```bash\n# Verbosity levels\npipedoc ~/docs --quiet             # Warnings and errors only\npipedoc ~/docs --verbose           # Debug level (-v)\npipedoc ~/docs -vv                 # Very verbose with detailed context\n\n# Output formats\npipedoc ~/docs --json              # JSON structured logging\npipedoc ~/docs --log-format detailed  # Timestamped detailed format\n\n# File logging\npipedoc ~/docs --log-file app.log  # Log to file in addition to console\npipedoc ~/docs --log-handlers rotating --log-file logs/app.log  # Rotating files\n\n# Component filtering\npipedoc ~/docs --log-exclude WorkerPool,MetricsCollector  # Exclude noisy components\npipedoc ~/docs --log-include-only ConnectionManager       # Only specific components\n```\n\n### Environment Configuration\n\n```bash\n# Configure logging via environment variables\nexport PIPEDOC_LOG_VERBOSITY=verbose\nexport PIPEDOC_LOG_FORMAT=json\nexport PIPEDOC_LOG_FILE=logs/pipedoc.log\nexport PIPEDOC_LOG_HANDLERS=console,rotating\npipedoc ~/docs  # Uses environment configuration\n```\n\n### Reading from the Pipe\n\nOnce the server is running, you can read from the named pipe in other terminals:\n\n```bash\n# Terminal 1 (start the server)\npipedoc ~/my-documentation\n\n# Terminal 2 (read from the pipe)\ncat /tmp/pipedoc_12345  # Use the actual pipe path shown by the server\n\n# Terminal 3 (another reader)\nless /tmp/pipedoc_12345  # Each reader gets the full content independently\n\n# Terminal 4 (process with a script)\npython -c \"\nwith open('/tmp/pipedoc_12345', 'r') as f:\n    content = f.read()\n    print(f'Read {len(content)} characters')\n\"\n```\n\n## Architecture\n\nThe project follows SOLID principles with a clean component-based architecture. For detailed technical documentation, see [ARCHITECTURE.md](ARCHITECTURE.md).\n\n### Core Components\n\n- **`MarkdownFileFinder`**: Discovers and filters markdown files (SRP)\n- **`ContentProcessor`**: Reads and processes file contents (SRP)\n- **`PipeManager`**: Orchestrates specialized pipe management components (SRP)\n  - **`MetricsCollector`**: Thread-safe connection statistics, queue metrics, and monitoring\n  - **`WorkerPool`**: Enhanced ThreadPoolExecutor wrapper with capacity management\n  - **`PipeResource`**: Named pipe lifecycle management\n  - **`ConnectionManager`**: Race condition prevention with always-ready writer pattern and FIFO connection queueing\n- **`MarkdownPipeServer`**: Orchestrates all components (SRP + DIP)\n- **`CLI`**: Click-based command-line interface (SRP)\n\n### Package Structure\n\n```\nsrc/pipedoc/\n├── __init__.py              # Package exports and version management\n├── cli.py                  # Click-based CLI interface with logging options\n├── server.py               # Main server orchestration\n├── file_finder.py          # File discovery logic\n├── content_processor.py    # Content processing logic\n├── pipe_manager.py         # Enhanced pipe manager (orchestrator)\n├── metrics_collector.py    # Thread-safe connection metrics\n├── worker_pool.py          # Enhanced ThreadPoolExecutor wrapper\n├── pipe_resource.py        # Named pipe lifecycle management\n├── connection_manager.py   # Race condition prevention\n├── app_logger.py           # Application logger interface and implementations\n├── logging_config.py       # Comprehensive logging configuration system\n├── connection_events.py    # Connection lifecycle events\n├── event_handlers.py       # Event handler implementations\n└── event_system.py         # General-purpose event system\n\ntests/\n├── conftest.py                           # Shared test fixtures\n├── test_file_finder.py                   # File finder tests\n├── test_content_processor.py             # Content processor tests\n├── test_pipe_manager.py                  # Enhanced pipe manager tests\n├── test_metrics_collector.py             # Metrics component tests\n├── test_worker_pool.py                   # Worker pool component tests\n├── test_pipe_resource.py                 # Pipe resource component tests\n├── test_connection_manager.py            # Connection manager component tests\n├── test_enhanced_pipe_manager_integration.py # Integration tests\n├── test_enhanced_error_handling.py       # Error handling tests\n├── test_performance_integration.py       # Performance tests\n└── test_server.py                        # Server integration tests\n```\n\n## Development\n\n### Setup Development Environment\n\n```bash\n# Setup with uv (recommended)\nuv sync --dev\n\n# Or install in development mode\npip install -e \".[dev]\"\n```\n\n### Running Tests\n\n```bash\n# Run tests\npytest tests/\n\n# Run tests with coverage\npytest tests/ --cov --cov-report=term-missing --cov-report=html\n\n# Using justfile (recommended)\njust test           # Run tests only\njust coverage       # Run tests with coverage\njust coverage-html  # Open coverage report in browser\n```\n\n### Code Quality\n\n```bash\n# Linting and formatting with ruff\nruff check src/ tests/\nruff check src/ tests/ --fix  # Auto-fix issues\nruff format src/ tests/\nruff format --check src/ tests/  # Check formatting\n\n# Using justfile (recommended)\njust lint           # Check linting\njust lint-fix       # Fix linting issues\njust format         # Format code\njust format-check   # Check formatting\njust quality        # Run all quality checks\n```\n\n### Development Workflow\n\n```bash\n# See all available commands\njust                    # Show simple recipe list\njust help               # Show detailed help with categories and examples\n\n# Common development tasks\njust dev-install        # Install with dev dependencies\njust quality            # Run linting, formatting, and tests\njust coverage           # Generate coverage report\njust status             # Show project status\njust clean              # Clean up generated files\njust setup              # Setup development environment from scratch\n```\n\n## Supported File Extensions\n\n- `.md`\n- `.markdown`\n- `.mdown`\n- `.mkd`\n\n## How it Works\n\n1. **File Discovery**: Uses `glob` to recursively find markdown files\n2. **Content Processing**: Reads and concatenates files with clear separators\n3. **Component Architecture**: Uses specialized components for different responsibilities:\n   - **PipeResource**: Creates and manages FIFO pipes\n   - **WorkerPool**: Manages ThreadPoolExecutor with capacity control\n   - **ConnectionManager**: Prevents race conditions with always-ready writer pattern and connection queueing\n   - **MetricsCollector**: Tracks connection statistics and queue metrics thread-safely\n4. **Multi-Threading**: Enhanced thread pool serves multiple processes concurrently\n5. **Connection Queueing**: FIFO queue handles connection bursts when worker pool is at capacity\n6. **Race-Free Serving**: Always-ready writer pattern eliminates connection timing issues\n7. **Error Isolation**: Component failures don't cascade across the system\n8. **Signal Handling**: Graceful shutdown on SIGINT/SIGTERM with proper cleanup\n\n## Examples\n\n### Example Output Format\n\nWhen files are concatenated, they include clear separators:\n\n```\n============================================================\nFILE: README.md\n============================================================\n\n# My Project\n\nThis is the main readme file.\n\n============================================================\nFILE: docs/guide.md\n============================================================\n\n# User Guide\n\nDetailed usage instructions here.\n```\n\n### Integration with Other Tools\n\n```bash\n# Use with grep to search across all files\npipedoc ~/docs | grep \"TODO\"\n\n# Use with word count\npipedoc ~/docs | wc -w\n\n# Use with a custom processor\npipedoc ~/docs | python my_processor.py\n```\n\n## Requirements\n\n- Python 3.8+\n- Unix-like system (for named pipes)\n- Click 8.0+ (for CLI functionality)\n\n## Development Dependencies\n\n- pytest 8.0+ (testing)\n- pytest-cov 6.0+ (coverage reporting)\n- ruff 0.12+ (linting and formatting)\n- just (command runner, optional but recommended)\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes following SOLID principles\n4. Add comprehensive tests\n5. Ensure code quality checks pass\n6. Submit a pull request\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Documentation\n\n- **[ARCHITECTURE.md](ARCHITECTURE.md)**: Detailed technical architecture and component design\n- **[CHANGELOG.md](CHANGELOG.md)**: Release notes and version history\n- **[LOGGING.md](LOGGING.md)**: Comprehensive logging configuration guide\n- **[EVENTS.md](EVENTS.md)**: Event system documentation and lifecycle monitoring\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeprave%2Fpipedoc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeeprave%2Fpipedoc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeeprave%2Fpipedoc/lists"}