https://github.com/portwatcher/prerender-url-shortener
url shortener with prerendered spa content cached
https://github.com/portwatcher/prerender-url-shortener
bitly chromium go opengraph opengraph-generator opengraph-images opengraph-tags prerender prerenderio puppeteer rod spa ssr tinyurl url-shortener
Last synced: 8 months ago
JSON representation
url shortener with prerendered spa content cached
- Host: GitHub
- URL: https://github.com/portwatcher/prerender-url-shortener
- Owner: portwatcher
- Created: 2025-05-20T05:17:46.000Z (9 months ago)
- Default Branch: develop
- Last Pushed: 2025-06-15T16:48:14.000Z (8 months ago)
- Last Synced: 2025-06-15T17:47:12.520Z (8 months ago)
- Topics: bitly, chromium, go, opengraph, opengraph-generator, opengraph-images, opengraph-tags, prerender, prerenderio, puppeteer, rod, spa, ssr, tinyurl, url-shortener
- Language: Go
- Homepage:
- Size: 57.6 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Prerender URL Shortener
This project is a Go-based URL shortener with a prerendering feature for search engine bots and crawlers.
## Features
### 1. Web Server
The web server handles two main types of requests:
#### 1.1. `GET /`
- Retrieves a record from the database associated with the provided ``.
- **User Agent (UA) Detection:**
- If the UA indicates a regular user browser, the server issues a redirect to the original URL.
- If the UA indicates a bot or crawler:
- **Returns pre-rendered HTML** only if rendering is complete AND contains valid Open Graph image metadata
- **Returns 404 Not Found** if content is not ready, rendering failed, or missing og:image tag
- **Auto-triggers re-rendering** when 404 is due to missing og:image tag in completed content
- **Never redirects bots** - they only get valid HTML or 404
#### 1.2. `POST /generate`
- Accepts a JSON request body with the following structure:
```json
{
"url": "string"
}
```
- **Response behavior:**
- **Returns 200 OK** if URL already exists with valid rendered HTML (contains og:image tag)
- **Returns 202 Accepted** for async rendering in all other cases (new URLs, invalid/missing HTML content)
- Triggers the backend process to generate a short code and prerender the content asynchronously when needed.
### 2. Prerendering and Shortening Logic (Rod Integration with Async Queue)
When a URL is submitted via the `/generate` endpoint:
- A unique `short-code` is generated for the given URL.
- The system checks if this URL is already cached/stored in the PostgreSQL database.
- **If cached and rendering complete:**
- **Returns 200 OK** if rendered HTML contains valid og:image tag (no re-rendering needed)
- **Returns 202 Accepted and re-queues for fresh rendering** if HTML is missing or lacks og:image tag
- **If not cached:**
- The link is immediately saved to the database with a "pending" render status.
- Returns 202 Accepted with the short code.
- The URL is queued for background rendering using a worker pool.
- **If already being rendered:**
- Returns 202 Accepted with the existing short code.
- Prevents duplicate rendering of the same URL using URL as the task key.
**Background Rendering Process:**
- Configurable number of worker goroutines process the render queue.
- Each worker uses the `rod` library to launch a headless browser instance.
- `rod` navigates to the original URL and renders its content, ensuring support for Single Page Applications (SPAs).
- The rendered HTML content and status are updated in the database upon completion.
### 3. PostgreSQL Database
- A PostgreSQL database is used to store the following information:
- `short_code` (Primary Key)
- `original_url` (Indexed for efficient lookups)
- `rendered_html_content`
- `render_status` (pending, rendering, completed, failed)
- Timestamps (e.g., `created_at`, `updated_at`)
### 4. Additional Endpoints
#### 4.1. `GET /health`
- Simple health check endpoint returning `{"status": "UP"}`
#### 4.2. `GET /status`
- Detailed status endpoint including render queue information:
```json
{
"status": "UP",
"render_queue": {
"worker_count": 3,
"queue_length": 2,
"in_progress_count": 1,
"in_progress_urls": ["https://example.com"]
}
}
```
## Technology Stack
- **Language:** Go
- **Web Framework:** Gin
- **Database:** PostgreSQL
- **ORM/DB Library:** GORM
- **Web Automation/Prerendering:** `go-rod/rod`
## Getting Started
### Prerequisites
- Go (version 1.24.1 or higher)
- PostgreSQL database
- Docker (optional, for containerized deployment)
### Manual Installation & Running
1. **Clone the repository:**
```bash
git clone
cd prerender-url-shortener
```
2. **Create a `.env` file** in the project root with your configuration. See `.env.example` for a template (if one exists, otherwise define the following):
```env
DATABASE_URL="postgres://user:password@host:port/dbname?sslmode=disable"
SERVER_PORT=":8080" # Optional, defaults to :8080
ALLOWED_DOMAINS="example.com,another.org" # Optional, comma-separated, empty means allow all
ROD_BIN_PATH="" # Optional, path to Chrome/Chromium binary if not in system PATH or for specific version
RENDER_WORKER_COUNT="3" # Optional, number of background rendering workers, defaults to 3
```
3. **Install dependencies:**
```bash
go mod tidy
```
4. **Run the application:**
```bash
go run cmd/server/main.go
```
The server will start, typically on port 8080.
### Running with Docker
A multi-architecture Docker image (supporting `linux/amd64` and `linux/arm64`) is available on Docker Hub at `juryschon/prerender-url-shortener:latest`.
1. **Pull the image (optional, `docker run` will do it automatically):**
```bash
docker pull juryschon/prerender-url-shortener:latest
```
2. **Run the container:**
```bash
docker run -d -p 8080:8080 \
--name prerender-shortener \
-e DATABASE_URL="postgres://your_user:your_password@your_db_host:5432/your_dbname?sslmode=disable" \
-e ALLOWED_DOMAINS="example.com,another.org" \
-e SERVER_PORT=":8080" \
# -e ROD_BIN_PATH="/usr/bin/google-chrome-stable" # Optional: if you bake chrome into your image and rod can't find it
--restart unless-stopped \
juryschon/prerender-url-shortener:latest
```
### Building the Docker Image
If you want to build the image yourself:
1. Ensure Docker Buildx is enabled (often default in Docker Desktop, or run `docker buildx create --use`).
2. Log in to Docker Hub if you intend to push:
```bash
docker login
```
3. Run the build command:
```bash
# For multi-platform (amd64, arm64) and pushing to your Docker Hub (replace )
docker buildx build --platform linux/amd64,linux/arm64 -t /prerender-url-shortener:latest --push .
# For a local build (current platform only)
docker build -t prerender-url-shortener .
```