{"id":28388250,"url":"https://github.com/portwatcher/prerender-url-shortener","last_synced_at":"2026-04-13T11:01:49.364Z","repository":{"id":295386909,"uuid":"986767407","full_name":"portwatcher/prerender-url-shortener","owner":"portwatcher","description":"url shortener with prerendered spa content cached","archived":false,"fork":false,"pushed_at":"2025-09-08T13:29:35.000Z","size":74,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-09-08T14:36:50.037Z","etag":null,"topics":["bitly","chromium","go","opengraph","opengraph-generator","opengraph-images","opengraph-tags","prerender","prerenderio","puppeteer","rod","spa","ssr","tinyurl","url-shortener"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/portwatcher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-20T05:17:46.000Z","updated_at":"2025-09-08T13:29:38.000Z","dependencies_parsed_at":"2025-05-25T09:34:35.311Z","dependency_job_id":"831ba72d-0e85-425f-80ed-e76cc58ba78f","html_url":"https://github.com/portwatcher/prerender-url-shortener","commit_stats":null,"previous_names":["portwatcher/prerender-url-shortener"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/portwatcher/prerender-url-shortener","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portwatcher%2Fprerender-url-shortener","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portwatcher%2Fprerender-url-shortener/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portwatcher%2Fprerender-url-shortener/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portwatcher%2Fprerender-url-shortener/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/portwatcher","download_url":"https://codeload.github.com/portwatcher/prerender-url-shortener/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/portwatcher%2Fprerender-url-shortener/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31749763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T09:16:15.125Z","status":"ssl_error","status_checked_at":"2026-04-13T09:16:05.023Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitly","chromium","go","opengraph","opengraph-generator","opengraph-images","opengraph-tags","prerender","prerenderio","puppeteer","rod","spa","ssr","tinyurl","url-shortener"],"created_at":"2025-05-30T21:09:18.808Z","updated_at":"2026-04-13T11:01:49.356Z","avatar_url":"https://github.com/portwatcher.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prerender URL Shortener\n\nThis project is a Go-based URL shortener with a prerendering feature for search engine bots and crawlers.\n\n## Features\n\n### 1. Web Server\n\nThe web server handles two main types of requests:\n\n#### 1.1. `GET /\u003cshort-code\u003e`\n   - Retrieves a record from the database associated with the provided `\u003cshort-code\u003e`.\n   - **User Agent (UA) Detection:**\n     - If the UA indicates a regular user browser, the server issues a redirect to the original URL.\n     - If the UA indicates a bot or crawler:\n       - **Returns pre-rendered HTML** only if rendering is complete AND contains valid Open Graph image metadata\n       - **Returns 404 Not Found** if content is not ready, rendering failed, or missing og:image tag\n       - **Auto-triggers re-rendering** when 404 is due to missing og:image tag in completed content\n       - **Never redirects bots** - they only get valid HTML or 404\n\n#### 1.2. `POST /generate`\n   - Accepts a JSON request body with the following structure:\n     ```json\n     {\n       \"url\": \"string\"\n     }\n     ```\n   - **Response behavior:**\n     - **Returns 200 OK** if URL already exists with valid rendered HTML (contains og:image tag)\n     - **Returns 202 Accepted** for async rendering in all other cases (new URLs, invalid/missing HTML content)\n   - Triggers the backend process to generate a short code and prerender the content asynchronously when needed.\n\n### 2. Prerendering and Shortening Logic (Rod Integration with Async Queue)\n\nWhen a URL is submitted via the `/generate` endpoint:\n   - A unique `short-code` is generated for the given URL.\n   - The system checks if this URL is already cached/stored in the PostgreSQL database.\n   - **If cached and rendering complete:**\n     - **Returns 200 OK** if rendered HTML contains valid og:image tag (no re-rendering needed)\n     - **Returns 202 Accepted and re-queues for fresh rendering** if HTML is missing or lacks og:image tag\n   - **If not cached:**\n     - The link is immediately saved to the database with a \"pending\" render status.\n     - Returns 202 Accepted with the short code.\n     - The URL is queued for background rendering using a worker pool.\n   - **If already being rendered:**\n     - Returns 202 Accepted with the existing short code.\n     - Prevents duplicate rendering of the same URL using URL as the task key.\n   \n   **Background Rendering Process:**\n   - Configurable number of worker goroutines process the render queue.\n   - Each worker uses the `rod` library to launch a headless browser instance.\n   - `rod` navigates to the original URL and renders its content, ensuring support for Single Page Applications (SPAs).\n   - The rendered HTML content and status are updated in the database upon completion.\n\n### 3. PostgreSQL Database\n\n   - A PostgreSQL database is used to store the following information:\n     - `short_code` (Primary Key)\n     - `original_url` (Indexed for efficient lookups)\n     - `rendered_html_content`\n     - `render_status` (pending, rendering, completed, failed)\n     - Timestamps (e.g., `created_at`, `updated_at`)\n\n### 4. Additional Endpoints\n\n#### 4.1. `GET /health`\n   - Simple health check endpoint returning `{\"status\": \"UP\"}`\n\n#### 4.2. `GET /status`\n   - Detailed status endpoint including render queue information:\n     ```json\n     {\n       \"status\": \"UP\",\n       \"render_queue\": {\n         \"worker_count\": 3,\n         \"queue_length\": 2,\n         \"in_progress_count\": 1,\n         \"in_progress_urls\": [\"https://example.com\"]\n       }\n     }\n     ```\n\n## Technology Stack\n\n- **Language:** Go\n- **Web Framework:** Gin\n- **Database:** PostgreSQL\n- **ORM/DB Library:** GORM\n- **Web Automation/Prerendering:** `go-rod/rod`\n\n## Getting Started\n\n### Prerequisites\n\n- Go (version 1.24.1 or higher)\n- PostgreSQL database\n- Docker (optional, for containerized deployment)\n\n### Manual Installation \u0026 Running\n\n1.  **Clone the repository:**\n\n```bash\ngit clone \u003crepository-url\u003e\ncd prerender-url-shortener\n```\n2.  **Create a `.env` file** in the project root with your configuration. See `.env.example` for a template (if one exists). Current configuration options:\n\n```env\n# Required\nDATABASE_URL=\"postgres://user:password@host:port/dbname?sslmode=disable\"\n\n# Server\nSERVER_PORT=\":8080\"                       # Optional, default :8080\nALLOWED_DOMAINS=\"example.com,another.org\" # Optional, comma-separated; empty means allow all\n\n# Renderer / Rod\nROD_BIN_PATH=\"\"                           # Optional, path to Chrome/Chromium binary\nRENDER_WORKER_COUNT=\"3\"                   # Optional, default 3 workers\nRENDER_TIMEOUT_SECONDS=\"90\"               # Optional, default 90s overall per-page timeout\n\n# Prerender stabilization\nMETA_WAIT_TIMEOUT_SECONDS=\"20\"            # Optional, default 20s max wait for meta stabilization\nMETA_STABLE_CONSECUTIVE_CHECKS=\"3\"        # Optional, default 3 equal reads for stability\nPRERENDER_READY_MARKER=\"data-prerender-ready=\\\"true\\\"\"  # Optional marker that, if present in HTML, ends waiting early\n```\n\nNotes:\n- PRERENDER_READY_MARKER: leave it as default or set to an empty value to disable marker-based early exit. The renderer still returns once og:image stabilizes or after a brief loop if no og:image is present.\n- Without a ready marker: behavior is generic — waits for load + network idle; then only loops for og:image stabilization if an og:image exists.\n\n3.  **Install dependencies:**\n```bash\ngo mod tidy\n```\n4.  **Run the application:**\n\n```bash\ngo run cmd/server/main.go\n```\nThe server will start, typically on port 8080.\n\n### Running with Docker\n\nA multi-architecture Docker image (supporting `linux/amd64` and `linux/arm64`) is available on Docker Hub at `juryschon/prerender-url-shortener:latest`.\n\n1.  **Pull the image (optional, `docker run` will do it automatically):**\n\n```bash\ndocker pull juryschon/prerender-url-shortener:latest\n```\n\n2.  **Run the container:**\n\n```bash\ndocker run -d -p 8080:8080 \\\n  --name prerender-shortener \\\n  -e DATABASE_URL=\"postgres://your_user:your_password@your_db_host:5432/your_dbname?sslmode=disable\" \\\n  -e ALLOWED_DOMAINS=\"example.com,another.org\" \\\n  -e SERVER_PORT=\":8080\" \\\n  # -e ROD_BIN_PATH=\"/usr/bin/google-chrome-stable\" # Optional: if you bake chrome into your image and rod can't find it\n  --restart unless-stopped \\\n  juryschon/prerender-url-shortener:latest\n```\n\n### Building the Docker Image\n\nIf you want to build the image yourself:\n\n1.  Ensure Docker Buildx is enabled (often default in Docker Desktop, or run `docker buildx create --use`).\n2.  Log in to Docker Hub if you intend to push:\n\n```bash\ndocker login\n```\n\n3.  Run the build command:\n\n```bash\n# For multi-platform (amd64, arm64) and pushing to your Docker Hub (replace \u003cyour_username\u003e)\ndocker buildx build --platform linux/amd64,linux/arm64 -t \u003cyour_username\u003e/prerender-url-shortener:latest --push .\n\n# For a local build (current platform only)\ndocker build -t prerender-url-shortener .\n``` \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fportwatcher%2Fprerender-url-shortener","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fportwatcher%2Fprerender-url-shortener","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fportwatcher%2Fprerender-url-shortener/lists"}