{"id":24009431,"url":"https://github.com/rteeter/logcollection","last_synced_at":"2025-07-08T19:10:07.716Z","repository":{"id":266213649,"uuid":"894746122","full_name":"rteeter/logCollection","owner":"rteeter","description":"This project is a minimal, secure, and lightweight log retrieval server for Unix-based systems. It provides a REST API and a web interface to allow users to easily access and monitor log files stored in the /var/log directory.","archived":false,"fork":false,"pushed_at":"2024-12-18T02:48:40.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-08T03:57:19.867Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rteeter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-26T23:04:34.000Z","updated_at":"2024-12-13T02:30:52.000Z","dependencies_parsed_at":"2024-12-03T06:38:27.613Z","dependency_job_id":null,"html_url":"https://github.com/rteeter/logCollection","commit_stats":null,"previous_names":["rteeter/logcollection"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rteeter%2FlogCollection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rteeter%2FlogCollection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rteeter%2FlogCollection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rteeter%2FlogCollection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rteeter","download_url":"https://codeload.github.com/rteeter/logCollection/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240675906,"owners_count":19839434,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-08T03:57:52.592Z","updated_at":"2025-02-25T13:29:48.470Z","avatar_url":"https://github.com/rteeter.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Log Retrieval Server\n\n## Overview\nA minimal, secure, and lightweight log retrieval server for Unix-based systems, providing both REST API and web interface access to log files from /var/log. Built with only Python standard library dependencies, it supports authentication, filtering, and configurable line limits.\n\n## Features\n- Minimal Python standard library dependencies\n- Secure log file access\n- Flexible log retrieval options (filtering, line limits)\n- Authentication support\n- Web UI for easy log access\n\n## Documentation\n- [Technical Implementation Guide](https://furtive-dart-4e3.notion.site/Technical-Implementation-15b4d0fe275280478618ce5221ad554a?pvs=4)\n\n## Requirements\n- Python 3.11+\n- Unix-like operating system\n- Read access to /var/log directory (may require sudo)\n\n## Development Setup\n\n1. Clone the repository:\n```bash\ngit clone git@github.com:rteeter/logCollection.git\ncd logCollection/log-retrieval-server\n```\n\n2. Create virtual environment with Python 3.11:\n```bash\npython3.11 -m venv .venv\nsource .venv/bin/activate\n```\n\n3. Verify Python version:\n```bash\npython --version  # Should show Python 3.11.x\n```\n\n4. Install test requirements:\n```bash\npip install -r requirements-test.txt\n```\n\n## Usage\n\n### Running the Server\n```bash\n# Basic usage\npython src/log_retrieval_server.py\n\n# Specify custom port\npython src/log_retrieval_server.py -p 9000\n\n# Add authentication\npython src/log_retrieval_server.py -t mysecrettoken\n```\n\n### API Endpoints\n\n#### GET /logs\nRetrieve log entries with flexible filtering\n\n**Query Parameters:**\n- `filename`: (Required) Name of the log file\n- `lines`: Number of recent log entries to retrieve (default: 1000)\n- `filter`: Keyword to filter log entries\n\n**Response Format:**\n```json\n{\n    \"filename\": \"system.log\",\n    \"total_entries\": 100,\n    \"entries\": [\n        \"2024-01-01 INFO: Log entry 1\",\n        \"2023-12-31 ERROR: Log entry 2\",\n        ...\n    ]\n}\n```\n\n**Example Requests:**\n```bash\n# Get last 100 lines from system log\ncurl \"http://localhost:8000/logs?filename=system.log\u0026lines=100\"\n\n# Filter system log for Configuration entries\ncurl \"http://localhost:8000/logs?filename=system.log\u0026filter=Configuration\"\n\n# With authentication if enabled\ncurl -H \"Authorization: Bearer mysecrettoken\" \"http://localhost:8000/logs?filename=system.log\"\n```\n\n## Web Interface\nA basic web interface is available for demonstrating the log retrieval functionality:\n\n1. Access the UI by opening `http://localhost:8000/` in your browser\n2. Enter the authentication token if required\n3. Specify the log file name (e.g., system.log)\n4. Optionally set the number of lines and filter text\n5. Click \"Retrieve Logs\" to view the results\n\nExample:\n```bash\n# Start the server with authentication\npython src/log_retrieval_server.py -t mysecrettoken\n\n# Then open in browser:\nhttp://localhost:8000/\n```\n\n## Security Considerations\n- Restricts file access to specified log directory\n- Optional token-based authentication\n- Input validation and sanitization\n- Directory traversal prevention\n\n## Testing\n\n### Running Tests\nMake sure you're in your virtual environment, then:\n```bash\npython -m unittest tests/test_log_retrieval_server.py\n```\n\nTest coverage includes:\n- Basic log file reading and filtering\n- Line limit enforcement\n- Authentication token handling\n- Security checks (directory traversal prevention)\n- Log ordering verification\n- Empty file handling\n- Error cases\n\n### Manual Testing\n1. Start the server:\n```bash\npython src/log_retrieval_server.py -t mysecrettoken\n```\n\n2. Test basic functionality:\n```bash\n# Test authentication\ncurl -H \"Authorization: Bearer mysecrettoken\" \\\n     \"http://localhost:8000/logs?filename=system.log\"\n\n# Test line limiting\ncurl \"http://localhost:8000/logs?filename=system.log\u0026lines=10\"\n\n# Test filtering\ncurl \"http://localhost:8000/logs?filename=system.log\u0026filter=ERROR\"\n```\n\n3. Test web interface:\n- Open `http://localhost:8000` in browser\n- Enter authentication token\n- Try various log files and filters\n\n## Limitations\n- Requires root/sudo for full `/var/log` access\n- No built-in rate limiting\n- Basic authentication mechanism\n\n## Potential Improvements\n- Add more robust authentication\n- Implement rate limiting\n- Support for compressed log files\n- Add request logging and monitoring\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frteeter%2Flogcollection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frteeter%2Flogcollection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frteeter%2Flogcollection/lists"}