{"id":47169014,"url":"https://github.com/shehab910/iis-logs-parser","last_synced_at":"2026-03-13T04:40:45.410Z","repository":{"id":272517527,"uuid":"916618875","full_name":"shehab910/iis-logs-parser","owner":"shehab910","description":"A high-performance Go backend service for parsing and managing IIS (Internet Information Services) W3C Extended Log Format files.","archived":false,"fork":false,"pushed_at":"2026-01-17T07:29:40.000Z","size":97,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-17T18:28:29.176Z","etag":null,"topics":["go","gola","iis","par","parsin"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shehab910.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-01-14T13:04:14.000Z","updated_at":"2026-01-17T07:31:00.000Z","dependencies_parsed_at":"2025-01-30T00:22:15.966Z","dependency_job_id":"8584b4b5-b936-4e80-af99-dbc6e332407f","html_url":"https://github.com/shehab910/iis-logs-parser","commit_stats":null,"previous_names":["shehab910/iis-logs-parser"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/shehab910/iis-logs-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shehab910%2Fiis-logs-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shehab910%2Fiis-logs-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shehab910%2Fiis-logs-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shehab910%2Fiis-logs-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shehab910","download_url":"https://codeload.github.com/shehab910/iis-logs-parser/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shehab910%2Fiis-logs-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30458026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-13T03:55:51.346Z","status":"ssl_error","status_checked_at":"2026-03-13T03:55:33.055Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","gola","iis","par","parsin"],"created_at":"2026-03-13T04:40:44.962Z","updated_at":"2026-03-13T04:40:45.371Z","avatar_url":"https://github.com/shehab910.png","language":"Go","readme":"# IIS Logs Parser\n\nA high-performance Go backend service for parsing and managing IIS (Internet Information Services) W3C Extended Log Format files. Features a REST API with JWT authentication, domain management, and optimized batch processing for large log files.\n\n## Features\n\n- Parse IIS W3C Extended Log Format files\n- High-performance batch processing using PostgreSQL `COPY` command\n- JWT-based authentication with email verification\n- Multi-tenant domain management\n- Background processing scheduler for uploaded log files\n- RESTful API built with Gin\n\n## Requirements\n\n- Go 1.23+\n- PostgreSQL 12+\n- SMTP server (for email verification)\n\n## Installation\n\n```bash\ncd iis-logs-parser\ngo mod download\ngo build -o iis-logs-parser main.go\n```\n\n## Configuration\n\nCopy `.env.local` and configure your environment:\n\n```bash\n# Environment\nGO_ENV=development              # development or production\n\n# Server\nSERVER_PORT=8090\n\n# JWT\nJWT_SECRET=\"ReplaceWithStrongSecret\"\n\n# Database\nDB_USER=postgres\nDB_PASS=password\nDB_HOST=localhost\nDB_PORT=5432\nDB_NAME=postgres-dev\n\n# Email (SMTP)\nFROM_EMAIL=\"your-email@example.com\"\nFROM_EMAIL_PASSWORD=\"\"\nFROM_EMAIL_SMTP=\"smtp.gmail.com\"\nFROM_EMAIL_PORT=587\n```\n\n## Usage\n\n### Development\n\n```bash\nsource .env.local\n./iis-logs-parser \"$FROM_EMAIL_PASSWORD\"\n```\n\n### Production\n\n```bash\nGO_ENV=production ./iis-logs-parser\n```\n\nThe email password is read from the `FROM_EMAIL_PASSWORD` environment variable in production mode.\n\n## API Reference\n\nAll endpoints return JSON. Protected endpoints require `Authorization: Bearer \u003ctoken\u003e` header.\n\n### Authentication\n\n| Method | Endpoint                             | Description             |\n| ------ | ------------------------------------ | ----------------------- |\n| POST   | `/api/v1/users/register`             | Register new user       |\n| POST   | `/api/v1/users/login`                | Login and get JWT token |\n| GET    | `/api/v1/users/verify?token=\u003ctoken\u003e` | Verify email address    |\n\n**Register Request:**\n\n```json\n{\n  \"email\": \"user@example.com\",\n  \"password\": \"SecurePass123!\",\n  \"first_name\": \"John\",\n  \"last_name\": \"Doe\"\n}\n```\n\nPassword requirements: 8+ characters, uppercase, lowercase, number, special character.\n\n**Login Response:**\n\n```json\n{\n  \"message\": \"Logged in\",\n  \"token\": \"eyJhbGciOiJIUzI1NiIs...\"\n}\n```\n\n### Domains (Protected)\n\n| Method | Endpoint              | Description         |\n| ------ | --------------------- | ------------------- |\n| GET    | `/api/v1/domains/`    | List user's domains |\n| POST   | `/api/v1/domains/`    | Create domain       |\n| PUT    | `/api/v1/domains/:id` | Update domain       |\n| DELETE | `/api/v1/domains/:id` | Delete domain       |\n\n**Create Domain Request:**\n\n```json\n{\n  \"name\": \"example.com\",\n  \"description\": \"Production server\"\n}\n```\n\n### Log Files (Protected)\n\n| Method | Endpoint                  | Description                 |\n| ------ | ------------------------- | --------------------------- |\n| GET    | `/api/v1/logs/`           | List all user's log files   |\n| GET    | `/api/v1/logs/domain/:id` | List log files for a domain |\n| POST   | `/api/v1/logs/upload`     | Upload log files            |\n| DELETE | `/api/v1/logs/:id`        | Delete log file             |\n\n**Upload Request:**\n\n```bash\ncurl -X POST http://localhost:8090/api/v1/logs/upload \\\n  -H \"Authorization: Bearer \u003ctoken\u003e\" \\\n  -F \"domain=1\" \\\n  -F \"logfiles=@/path/to/logfile.log\"\n```\n\n## Log File Processing\n\nUploaded log files are processed asynchronously by a background scheduler running every 20 seconds. The processing pipeline:\n\n1. File status changes from `pending` to `processing`\n2. Log file is parsed line-by-line (IIS W3C format)\n3. Entries are batch-inserted to PostgreSQL using `COPY` command\n4. Status changes to `completed` (or `failed` on error)\n\n### Supported Log Format\n\nIIS W3C Extended Log Format with fields:\n\n```\ndate time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken\n```\n\n## Running Tests\n\n```bash\n# Run all tests\ngo test ./tests -v\n\n# Run specific test\ngo test ./tests -v -run TestParseLogLine\n\n# Run benchmarks\ngo test ./tests -bench=. -benchmem\n```\n\n## Performance\n\nThe processor uses `pgx` with PostgreSQL `COPY` command instead of GORM for batch inserts, achieving:\n\n- ~60% faster processing time\n- ~85% memory reduction\n\nBenchmark processing a 1.7GB log file: ~37 seconds.\n\nSee `task2-brainstorming.md` for detailed benchmark analysis.\n\n## Project Structure\n\n```\n.\n├── main.go              # Entry point, background scheduler\n├── config/              # Application constants\n├── database/            # PostgreSQL connection (GORM + pgx)\n├── middleware/          # JWT authentication middleware\n├── models/              # Data models (User, Domain, LogFile, LogEntry)\n├── parser/              # IIS log line parser\n├── processor/           # Batch processing with pgx COPY\n├── routes/              # API route handlers\n├── utils/               # JWT, password hashing, email, validation\n└── tests/               # Unit tests and benchmarks\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshehab910%2Fiis-logs-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshehab910%2Fiis-logs-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshehab910%2Fiis-logs-parser/lists"}