{"id":50711945,"url":"https://github.com/ba0f3/gocrawl","last_synced_at":"2026-06-09T16:03:17.714Z","repository":{"id":348230202,"uuid":"1101019951","full_name":"ba0f3/gocrawl","owner":"ba0f3","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-02T07:30:05.000Z","size":14300,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-02T08:15:23.145Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ba0f3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-11-21T04:40:28.000Z","updated_at":"2026-06-02T07:30:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ba0f3/gocrawl","commit_stats":null,"previous_names":["ba0f3/gocrawl"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/ba0f3/gocrawl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ba0f3%2Fgocrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ba0f3%2Fgocrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ba0f3%2Fgocrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ba0f3%2Fgocrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ba0f3","download_url":"https://codeload.github.com/ba0f3/gocrawl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ba0f3%2Fgocrawl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34114437,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-09T16:03:14.064Z","updated_at":"2026-06-09T16:03:17.706Z","avatar_url":"https://github.com/ba0f3.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoCrawl - Firecrawl-like Web Crawler\n\nA production-grade Go web crawler that provides a Firecrawl-compatible API for web scraping and content extraction.\n\n## Features\n\n- Web crawling using Colly (queued jobs, configurable workers)\n- HTML to Markdown conversion\n- Multiple backends: MongoDB or SQL (PostgreSQL / SQLite via GORM)\n- Optional user authentication with API keys\n- REST API under `/v1` (Firecrawl-style)\n- Optional HTTP rate limiting, crawl retries, and chromedp/Lightpanda (auto fallback for CSR, HTTP 401/403/429/503, challenge pages, or `forceBrowser`)\n- Optional **Chrome-like TLS** (uTLS) for Colly fetches (`ENABLE_CHROME_TLS`), **webclaw-style** main-content extraction (noise/scoring), optional **inline JS data** extraction via goja (`extractJsData`), and optional **LLM summaries** (OpenAI-compatible API)\n- Automatic data cleanup and crawl job progress tracking\n\n## Quick start (Docker / GHCR)\n\nRun the prebuilt image (no database needed with `ENABLE_AUTH=false`):\n\n```bash\ndocker pull ghcr.io/ba0f3/gocrawl:latest\n\ndocker run --rm -p 8151:8151 \\\n  -e PORT=8151 \\\n  -e HOST=0.0.0.0 \\\n  -e ENABLE_AUTH=false \\\n  ghcr.io/ba0f3/gocrawl:latest\n```\n\nThe API is at `http://localhost:8151/v1/...` (for example `POST /v1/scrape`). For private images, `docker login ghcr.io` first; for auth and databases, see [Docker installation](#docker-installation) and **Configuration**.\n\n## Prerequisites\n\n- **From source:** Go 1.26 or higher (see `go.mod`)\n- **Docker:** optional — see [Docker installation](#docker-installation) to run a prebuilt image or build locally\n- When `ENABLE_AUTH=true`: a database (MongoDB, PostgreSQL, or SQLite) per configuration\n\n## Installation\n\n1. Clone the repository:\n\n```bash\ngit clone \u003crepository-url\u003e\ncd gocrawl\n```\n\n2. Install dependencies:\n\n```bash\ngo mod tidy\n```\n\n3. Set up environment variables by copying `.env` and modifying as needed:\n\n```bash\ncp .env .env.local\n```\n\n4. Run the application:\n\n```bash\ngo run ./cmd\n```\n\nOr use the Makefile:\n\n```bash\nmake run      # development\nmake build    # produces ./bin/gocrawl\n```\n\n## Docker installation\n\nRequires [Docker](https://docs.docker.com/get-docker/). Optional: [Docker Compose](https://docs.docker.com/compose/) for `docker-compose.yml`.\n\n### Pull a prebuilt image (GHCR)\n\nCI publishes images to GitHub Container Registry on pushes to `main` and version tags (`v*`). Replace `\u003cowner\u003e` with your GitHub user or organization (lowercase).\n\n```bash\ndocker pull ghcr.io/\u003cowner\u003e/gocrawl:latest\n```\n\nIf the package is private, sign in first:\n\n```bash\necho \"$GITHUB_TOKEN\" | docker login ghcr.io -u USERNAME --password-stdin\n```\n\nRun the container (auth disabled; in-memory crawl jobs — same behavior as a local dev run without a database):\n\n```bash\ndocker run --rm -p 8151:8151 \\\n  -e PORT=8151 \\\n  -e HOST=0.0.0.0 \\\n  -e ENABLE_AUTH=false \\\n  ghcr.io/\u003cowner\u003e/gocrawl:latest\n```\n\nThe API is available at `http://localhost:8151/v1/...`. For `ENABLE_AUTH=true`, pass database settings (`DATABASE_DRIVER`, `MONGO_URI` or `DATABASE_DSN`, etc.) — see **Configuration** below.\n\n### Build the image locally\n\nFrom the repository root:\n\n```bash\ndocker build -t gocrawl:local .\ndocker run --rm -p 8151:8151 -e PORT=8151 -e HOST=0.0.0.0 -e ENABLE_AUTH=false gocrawl:local\n```\n\n### Docker Compose\n\nBase stack (`docker-compose.yml`): **gocrawl** on port `8151` and **lightpanda** on `9222` (optional JS/CDP fallback). MongoDB is **not** included.\n\n1. Copy the environment template and edit values (see **Configuration** below):\n\n```bash\ncp .env.example .env\n```\n\n2. Start:\n\n```bash\ndocker compose up --build -d\n```\n\nCompose loads `.env` when present (`env_file`); variables also support defaults in `docker-compose.yml`. For a typical no-auth scrape API, keep `ENABLE_AUTH=false` in `.env` (no database required). The **gocrawl** service defaults **`LIGHTPANDA_HTTP_URL=http://lightpanda:9222`** so chromedp can resolve the CDP WebSocket from Lightpanda’s `/json/version` without pasting a full `ws://…` URL (override or clear the variable if you do not run Lightpanda).\n\n#### Optional MongoDB (`docker-compose.mongo.yml`)\n\nUse this **only** when you want MongoDB in Compose (for example `ENABLE_AUTH=true` with `DATABASE_DRIVER=mongo`).\n\n1. In `.env`, set at least:\n\n```bash\nDATABASE_DRIVER=mongo\nMONGO_URI=mongodb://mongo:27017\nENABLE_AUTH=true\nJWT_SECRET=\u003ca-long-random-secret\u003e\n```\n\n2. Start both files:\n\n```bash\ndocker compose -f docker-compose.yml -f docker-compose.mongo.yml up --build -d\n```\n\nThe overlay adds the **mongo** service and makes **gocrawl** wait until Mongo is healthy. For Postgres or SQLite instead, run your database elsewhere and set `DATABASE_DRIVER` / `DATABASE_DSN` (or `SQLITE_PATH`) in `.env` without the Mongo overlay.\n\n## Configuration\n\nEnvironment variables can be set in the `.env` file or as system environment variables.\n\n### Server\n\n| Variable | Description |\n|----------|-------------|\n| `PORT` | Server port (default: `8151`) |\n| `HOST` | Bind address (default: empty; use `0.0.0.0` to listen on all interfaces) |\n| `GIN_MODE` | `debug` for verbose Gin logging; otherwise release mode (default) |\n| `SERVER_READ_HEADER_TIMEOUT` | Max time to read request headers (default: `10s`; mitigates slow clients) |\n| `SERVER_READ_TIMEOUT` | Max time to read the full request including body (default: `60s`) |\n| `SERVER_WRITE_TIMEOUT` | Max time to write the response; `0` or unset means no limit (default), which suits long scrapes and SSE |\n| `SERVER_IDLE_TIMEOUT` | Keep-alive idle timeout (default: `120s`) |\n| `SERVER_MAX_HEADER_BYTES` | Maximum request header size in bytes (default: `1048576`) |\n| `SERVER_SHUTDOWN_TIMEOUT` | Graceful shutdown deadline after SIGINT/SIGTERM (default: `30s`) |\n\n### Database\n\n| Variable | Description |\n|----------|-------------|\n| `DATABASE_DRIVER` | `mongo` (default), `postgres`, or `sqlite` |\n| `MONGO_URI` | MongoDB URI (default: `mongodb://localhost:27017`) |\n| `DB_NAME` | Database name (default: `gocrawl`) |\n| `DATABASE_DSN` | Postgres connection string or SQLite file path |\n| `SQLITE_PATH` | Alternative SQLite path if `DATABASE_DSN` is empty |\n\n### Security\n\n| Variable | Description |\n|----------|-------------|\n| `JWT_SECRET` | Secret for JWT-related use |\n| `ENABLE_AUTH` | `true` to require API keys on protected routes and to use a real database; `false` runs without `Authorization` and uses an **in-memory** crawl job store (jobs are lost on restart; register/login stay unavailable) |\n\n### Crawler (selected)\n\n| Variable | Description |\n|----------|-------------|\n| `CRAWL_WORKERS` | Goroutines that drain the crawl job queue (default: same as `MAX_CONCURRENT_CRAWLS`) |\n| `MAX_CONCURRENT_CRAWLS` | Default per-job Colly parallelism (default: `10`) |\n| `CRAWL_TIMEOUT` | Default per-request timeout (e.g. `30s`) |\n| `USER_AGENT` | User-Agent for Colly and chromedp |\n| `CRAWL_MAX_RETRIES` | HTTP retries for 429/5xx (default: `3`) |\n| `ENABLE_CHROME_TLS` | When `true`, Colly uses uTLS with a Chrome TLS fingerprint (HTTP/2 via ALPN). Default User-Agent becomes a Chrome string unless you set `USER_AGENT` explicitly. Default `false` |\n| `LLM_ENABLED` | When `true`, allows `summarize: true` on scrape requests (requires `LLM_BASE_URL` and `LLM_MODEL`) |\n| `LLM_BASE_URL` | OpenAI-compatible API root (no trailing slash; e.g. `http://host:11434/v1` for Ollama) |\n| `LLM_API_KEY` | Optional bearer token for the LLM API |\n| `LLM_MODEL` | Default model id for summaries (e.g. `qwen2.5:7b`, `gpt-oss-20b`) |\n| `LLM_TIMEOUT` | HTTP client timeout for LLM calls (default `120s`) |\n| `LIGHTPANDA_WS_URL` | Optional full CDP WebSocket URL (e.g. from `GET http://host:9222/json/version` → `webSocketDebuggerUrl`) |\n| `LIGHTPANDA_HTTP_URL` | Optional HTTP base (e.g. `http://lightpanda:9222`); first chromedp use resolves and caches `webSocketDebuggerUrl` |\n| `CHROMEDP_AUTO_FALLBACK` | When `true` (default), automatically use chromedp after Colly when heuristics match (WAF/antibot signals via [is-antibot-go](https://github.com/ba0f3/is-antibot-go), thin main text, SPA shell, challenge HTML, configured status codes, errors). Set `false` to only use chromedp with JSON `forceBrowser: true` |\n| `CHROMEDP_FALLBACK_STATUS_CODES` | Comma-separated HTTP statuses that trigger auto fallback (default `401,403,429,503` if unset) |\n| `CHROMEDP_MAX_CONCURRENT` | Max concurrent chromedp sessions (default `8` if unset or `0`) |\n| `CHROMEDP_LOAD_WAIT_TIMEOUT` | Max time to poll `document.readyState` after navigation (default `30s`; capped under the scrape timeout). Stops on **`complete`** or on two consecutive **`interactive`** reads (SPAs often never reach `complete` after XHRs, which used to spin until Lightpanda’s CDP timeout). Short `Evaluate` calls only |\n| `CHROMEDP_NAV_WAIT` | Post-load settle before hydration (default `500ms` if unset; set `0s` to disable). Longer waits are **paced** with tiny CDP pings so idle sessions are less likely to hit Lightpanda’s **CDP timeout** |\n| `CHROMEDP_HYDRATION_POLL_INTERVAL` | Delay between hydration polls (default `300ms`, also paced with CDP pings) |\n| `CHROMEDP_HYDRATION_MAX_POLLS` | Max hydration polls (default `22`) |\n| `CHROMEDP_HYDRATION_MIN_TEXT_RUNES` | Stop polling when main-selector text length reaches this (default `80`) |\n| `RATE_LIMIT_REQUESTS` / `RATE_LIMIT_WINDOW` | **Off by default** (`0` or unset). If both are positive (e.g. `100` and `1h`), enables per-client rate limiting on `/v1`. The limiter uses a token bucket: burst equals `RATE_LIMIT_REQUESTS`, then refill is spread evenly across `RATE_LIMIT_WINDOW` (so `100`/`1h` averages about one request every 36 seconds after the burst). |\n\n**Chromedp limitations:** Running pages in Lightpanda helps with **client-side rendering** and some **simple bot or challenge pages**, but it does **not** guarantee bypass of advanced anti-bot (CAPTCHA vendors, strict TLS/JA3 fingerprinting, etc.). Difficult sites may need proxies, a full browser profile, or manual steps.\n\n### Data retention\n\n- `DATA_RETENTION_DAYS`, `CLEANUP_INTERVAL` — see `internal/config/config.go` for defaults.\n\n## API base URL\n\nAll HTTP APIs are mounted at **`/v1`** (not `/api/v1`).\n\nExample base:\n\n```bash\nexport BASE_URL=\"http://localhost:8151\"\n```\n\n## API overview\n\n| Method | Path | Auth (if enabled) | Description |\n|--------|------|---------------------|-------------|\n| POST | `/v1/auth/register` | No | Create user (requires DB) |\n| POST | `/v1/auth/login` | No | Login; response includes `apiKey` |\n| POST | `/v1/scrape` | Bearer API key | Single-page scrape |\n| POST | `/v1/crawl` | Bearer API key | Enqueue multi-page crawl |\n| GET | `/v1/crawl/{id}` | Bearer API key | Job status and results |\n\n## Optional CSS selectors\n\n### Scrape body (`POST /v1/scrape`)\n\n| Field | JSON | Description |\n|-------|------|-------------|\n| Output shapes | `formats` | Array: `\"markdown\"`, `\"html\"`, `\"rawHtml\"`. Only requested fields are filled; others are omitted from JSON (`omitempty`). **If `formats` is omitted or empty**, all three are returned (backward compatible). Example: `[\"markdown\"]` returns only `markdown` (plus `metadata`, `links`). |\n| Main vs full page | `onlyMainContent` | **Omitted** or `true`: try built-in selectors (`main`, `article`, …) then `body` if none match. **`false`**: use the entire `\u003cbody\u003e` (full page). Omitting the field now defaults to main-content mode (previously a JSON quirk made omitted behave like `false`). |\n| Content root | `contentSelector` | One CSS selector for the node whose HTML is converted to markdown / stored as `html`. |\n| Content root | `contentSelectors` | More selectors, tried in order after `contentSelector`. |\n| Links list | `linkSelector` | Optional. **Omitted:** collect links only inside the same DOM subtree as the extracted content (e.g. only under `\u003carticle\u003e` when that block is used for markdown), with **URLs de-duplicated**. **Set:** run this selector against the **full page** (e.g. `a[href]` for every anchor on the page). |\n| Browser-only | `forceBrowser` | When `true` and `LIGHTPANDA_WS_URL` or `LIGHTPANDA_HTTP_URL` is set, **skips the Colly HTTP fetch** and loads the page with chromedp only. Metadata includes `chromedpTrigger: force_browser`. |\n| Exclusions | `excludeSelectors` | CSS selectors; matching subtrees are excluded from **scored** main-content extraction (webclaw-style). |\n| Advanced extraction | `useAdvancedExtractor` | Default follows `onlyMainContent`: when main-content mode is on, uses noise/scoring extraction; set `false` for legacy selector-only behavior. |\n| JS blobs | `extractJsData` | When `true`, runs inline scripts in a sandbox (goja), collects large `window.__*` JSON blobs (e.g. Next.js `__next_f`), and appends a `## Additional Content` markdown section. Metadata may include `jsExtracted`, `jsBlobCount`. |\n| LLM summary | `summarize` | When `true` and `LLM_*` env is configured, adds a plain-text `summary` field (default sentence count `3`). Optional: `summaryMaxSentences`, `summaryModel` (overrides `LLM_MODEL`). Failures are logged and do not fail the scrape. |\n\nIf `contentSelector` or `contentSelectors` are set, they **replace** the built-in main-content list (`main`, `article`, …). If no selector matches, the extractor falls back to `body`. `onlyMainContent` applies only when you do **not** set custom content selectors.\n\nWhen chromedp runs (auto fallback or `forceBrowser`), **`metadata`** may include `extractor: chromedp` and **`chromedpTrigger`** (`antibot_\u003cprovider\u003e` when a WAF/challenge is detected, e.g. `antibot_cloudflare`, `antibot_hcaptcha`; `thin_markdown`, `spa_shell`, `csr_framework` for Vue/React/Next/Nuxt/Angular/SvelteKit/Remix/Astro/Vite-style shells, `challenge_html`, `status_403`, `visit_error`, etc.).\n\n### Crawl body (`POST /v1/crawl`)\n\n| Field | JSON | Description |\n|-------|------|-------------|\n| Link discovery | `linkSelector` | Single CSS selector for the same purpose as `linkSelectors` (convenience). |\n| Link discovery | `linkSelectors` | Only enqueue links that match these selectors (e.g. `article a[href]`, `.post-list a`). When both `linkSelector` and `linkSelectors` are set, `linkSelector` is tried first. When empty, the crawler uses the default article/main heuristics plus all `a[href]`. |\n| Per page | `scrapeOptions` | Same fields as scrape: `contentSelector`, `contentSelectors`, `linkSelector`, `onlyMainContent`, `formats`, etc., applied to each fetched page. If you set `linkSelectors` but omit `scrapeOptions.linkSelector`, the same selectors are reused when collecting `links` on each scraped page (Colly debug logs will show that selector instead of only `a[href]`). |\n\n## Testing with curl\n\nThe examples below use `$BASE_URL` (default `http://localhost:8151`). With authentication enabled, set `API_KEY` after login.\n\n### 1. Scrape a single page (auth disabled)\n\nWhen `ENABLE_AUTH=false`, omit `Authorization`:\n\n```bash\ncurl -sS -X POST \"${BASE_URL:-http://localhost:8151}/v1/scrape\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"url\": \"https://hehemetal.com\",\n    \"onlyMainContent\": true,\n    \"formats\": [\"markdown\", \"html\"]\n  }' | jq .\n```\n\n### 2. Register and log in (auth enabled)\n\nRequires MongoDB or SQL configured and `ENABLE_AUTH=true`.\n\n```bash\ncurl -sS -X POST \"${BASE_URL:-http://localhost:8151}/v1/auth/register\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"username\":\"demo\",\"password\":\"demo-secret\"}' | jq .\n\ncurl -sS -X POST \"${BASE_URL:-http://localhost:8151}/v1/auth/login\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"username\":\"demo\",\"password\":\"demo-secret\"}' | jq .\n```\n\nSave the API key from the login response:\n\n```bash\nexport API_KEY=\"$(curl -sS -X POST \"${BASE_URL:-http://localhost:8151}/v1/auth/login\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"username\":\"demo\",\"password\":\"demo-secret\"}' | jq -r .data.apiKey)\"\necho \"$API_KEY\"\n```\n\n### 3. Scrape with API key\n\n```bash\ncurl -sS -X POST \"${BASE_URL:-http://localhost:8151}/v1/scrape\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer ${API_KEY}\" \\\n  -d '{\n    \"url\": \"https://hehemetal.com\",\n    \"onlyMainContent\": true,\n    \"formats\": [\"markdown\"]\n  }' | jq .\n```\n\nSuccessful responses use the wrapper: `{\"success\":true,\"data\":{...}}` (or `success:false` with `error`).\n\n### 4. Start a crawl job\n\nReturns `id` immediately; workers process the queue in the background.\n\n```bash\ncurl -sS -X POST \"${BASE_URL:-http://localhost:8151}/v1/crawl\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer ${API_KEY}\" \\\n  -d '{\n    \"url\": \"https://hehemetal.com\",\n    \"limit\": 5,\n    \"maxDepth\": 1,\n    \"maxConcurrency\": 2,\n    \"delay\": 500\n  }' | jq .\n```\n\nExample response shape:\n\n```json\n{\"success\":true,\"id\":\"\u003cuuid\u003e\",\"url\":\"https://hehemetal.com\"}\n```\n\n### 5. Poll crawl status and results\n\nUse the **`id` from the crawl POST response** (not another UUID). A wrong or expired id returns HTTP 404.\n\n```bash\nJOB_ID=\"\u003cpaste-id-from-crawl-response\u003e\"\n\ncurl -sS \"${BASE_URL:-http://localhost:8151}/v1/crawl/${JOB_ID}\" \\\n  -H \"Authorization: Bearer ${API_KEY}\" | jq .\n```\n\nResponse includes `status` (`queued`, `crawling`, `completed`, …), `total`, `completed`, and `data` (array of page results).\n\n### 6. Quick one-liner (auth off)\n\n```bash\ncurl -sS -X POST \"http://localhost:8151/v1/scrape\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\":\"https://hehemetal.com\",\"onlyMainContent\":true,\"formats\":[\"markdown\"]}'\n```\n\n### 7. Scrape with custom selectors\n\n```bash\ncurl -sS -X POST \"http://localhost:8151/v1/scrape\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"url\": \"https://hehemetal.com/news/buc-tuong-duoc-duc-bang-bac-cua-helios\",\n    \"contentSelector\": \"article\",\n    \"linkSelector\": \"article a[href]\",\n    \"formats\": [\"markdown\",\"html\"]\n  }' | jq .\n```\n\n### 8. Crawl with restricted link discovery\n\n```bash\ncurl -sS -X POST \"http://localhost:8151/v1/crawl\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"url\": \"https://hehemetal.com/category/news\",\n    \"limit\": 20,\n    \"maxDepth\": 2,\n    \"linkSelectors\": [\"article a[href]\", \".pagination a[href]\"],\n    \"scrapeOptions\": {\n      \"onlyMainContent\": true,\n      \"contentSelector\": \"article\",\n      \"formats\": [\"markdown\"]\n    }\n  }' | jq .\n```\n\n## Response shapes (reference)\n\n**Scrape** (`writeResponse`):\n\n```json\n{\n  \"success\": true,\n  \"data\": {\n    \"markdown\": \"...\",\n    \"html\": \"...\",\n    \"rawHtml\": \"...\",\n    \"links\": [\"https://hehemetal.com/...\"],\n    \"metadata\": { \"title\": \"...\", \"sourceURL\": \"...\" }\n  }\n}\n```\n\n**Start crawl** (raw JSON, not wrapped):\n\n```json\n{ \"success\": true, \"id\": \"\u003cjob-uuid\u003e\", \"url\": \"https://hehemetal.com\" }\n```\n\n**Crawl status** (raw JSON):\n\n```json\n{\n  \"status\": \"completed\",\n  \"total\": 3,\n  \"completed\": 3,\n  \"creditsUsed\": 0,\n  \"expiresAt\": \"2026-03-30T12:00:00Z\",\n  \"data\": [ { \"markdown\": \"...\", \"metadata\": { } } ]\n}\n```\n\n## Project structure (abbreviated)\n\n```\ngocrawl/\n├── cmd/main.go              # Entrypoint, routes, DB wiring\n├── internal/\n│   ├── api/                 # HTTP handlers, crawl manager, middleware\n│   ├── config/\n│   ├── crawler/             # Colly scrape, retries, chromedp fallback\n│   ├── extractor/           # HTML → Markdown\n│   ├── db/                  # Store interface, Mongo + GORM SQL\n│   ├── user/\n│   └── mcp/\n├── Makefile                 # build, test, run, fmt, vet\n├── go.mod\n└── README.md\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Run `make all` (or `make fmt vet test build`)\n5. Submit a pull request\n\n## License\n\nThis project is licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fba0f3%2Fgocrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fba0f3%2Fgocrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fba0f3%2Fgocrawl/lists"}