{"id":18669978,"url":"https://github.com/blahah/vercel-headless-scraper-api","last_synced_at":"2025-06-19T07:07:59.959Z","repository":{"id":66276046,"uuid":"469769835","full_name":"blahah/vercel-headless-scraper-api","owner":"blahah","description":"Examples of using vercel workers to turn any website into an API","archived":false,"fork":false,"pushed_at":"2022-03-14T14:31:07.000Z","size":3531,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-18T08:35:24.637Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blahah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-14T14:29:42.000Z","updated_at":"2022-04-01T04:57:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"ef5b341e-4c98-4e58-9a9c-b44dd902783a","html_url":"https://github.com/blahah/vercel-headless-scraper-api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/blahah/vercel-headless-scraper-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blahah%2Fvercel-headless-scraper-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blahah%2Fvercel-headless-scraper-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blahah%2Fvercel-headless-scraper-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blahah%2Fvercel-headless-scraper-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blahah","download_url":"https://codeload.github.com/blahah/vercel-headless-scraper-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blahah%2Fvercel-headless-scraper-api/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260705874,"owners_count":23049493,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T08:49:22.274Z","updated_at":"2025-06-19T07:07:54.820Z","avatar_url":"https://github.com/blahah.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# memex-feed-parser\n\nLive API: `https://memex-feed-parser.vercel.app/`\n\n- [memex-feed-parser](#memex-feed-parser)\n  - [API](#api)\n    - [Endpoint: `GET /api/memex/[sharedListId]`](#endpoint-get-apimemexsharedlistid)\n      - [response](#response)\n      - [pages](#pages)\n      - [annotations](#annotations)\n      - [example](#example)\n      - [try it](#try-it)\n    - [Endpoint: `get /api/scrape/[url]`](#endpoint-get-apiscrapeurl)\n      - [example](#example-1)\n      - [try it](#try-it-1)\n  - [Development](#development)\n    - [Deploy config](#deploy-config)\n\n## API\n\n### Endpoint: `GET /api/memex/[sharedListId]`\n\nRetrieve the Memex shared feed with the provided ID, and return the entries as structured data.\n\n#### response\n\nThe response is a JSON object which has the form:\n\n```json\n{\n  \"listId\": \"\", // the same listId you provided\n  \"pages\": [],\n  \"annotations\": []\n}\n```\n\n#### pages\n\nEach entry in `pages` takes the form:\n\n```json\n{\n  \"source\": \"memex\",\n  \"type\": \"link\",\n  \"url\": \"\", // the URL\n  \"title\": \"\", // the page title\n  \"created\": 1618345214065, // as Unix Timestamp (milliseconds)\n  \"updated\": 1618345214065\n}\n```\n\n#### annotations\n\nEach entry in `annotations` takes the form:\n\n```json\n{\n  \"source\": \"memex\",\n  \"type\": \"annotation\",\n  \"url\": \"\", // the URL\n  \"quote\": \"\", // the quote part of the annotation\n  \"comment\": \"\", // the comment part of the annotation\n  \"created\": 1616803866655, // as Unix Timestamp (milliseconds)\n  \"updated\": 1618346658307\n}\n```\n\n#### example\n\nThe query:\n\n`GET /api/memex/tEr22YvmUnYZ30vFiOL0`\n\nReturns the response:\n\n\u003e note: entries omitted for brevity\n\n```json\n{\n  \"listId\": \"tEr22YvmUnYZ30vFiOL0\",\n  \"pages\": [{\n    \"source\": \"memex\",\n    \"type\": \"link\",\n    \"url\": \"https://jackrusher.com/journal/what-does-it-mean-to-buy-a-gif.html\",\n    \"title\": \"What Does It Mean To Buy a Gif? | Jack Rusher\",\n    \"created\": 1618345214065,\n    \"updated\": 1618345214065\n  },\n  ...\n  ],\n  \"annotations\": [{\n    \"source\": \"memex\",\n    \"type\": \"annotation\",\n    \"url\": \"https://jackrusher.com/journal/what-does-it-mean-to-buy-a-gif.html\",\n    \"quote\": \"The top performers are so homogeneous that NYC’s galleries show more art by people who went to Yale than by all minorities combined. In addition to inequalities along race, class, and gender lines for those born in the rich world, artists born in countries without elite institutions are essentially barred from success.\\n\\nWe might well ask what service the gatekeepers who operate these markets provide to make up for these shortcomings. One argument is that they have the taste and education to allocate capital wisely, thus rewarding the highest quality art.\\n\\n\",\n    \"comment\": \"\",\n    \"created\": 1616803866655,\n    \"updated\": 1618346658307\n  },\n  ...\n  ]\n})\n```\n\n#### try it\n\n[Try this example on the live API](https://memex-feed-parser.vercel.app/api/memex/tEr22YvmUnYZ30vFiOL0)\n\n### Endpoint: `get /api/scrape/[url]`\n\nScrape the URL and return structured metadata.\n\n#### example\n\nThe query:\n\n`GET /api/scrape/https%3A%2F%2Fwww.theguardian.com%2Fenvironment%2F2021%2Fapr%2F28%2Fspeed-at-which-worlds-glaciers-are-melting-has-doubled-in-20-years`\n\nReturns the response:\n\n```json\n{\n  \"url\": \"https://www.theguardian.com/environment/2021/apr/28/speed-at-which-worlds-glaciers-are-melting-has-doubled-in-20-years\",\n  \"metadata\": {\n    \"author\": \"Jonathan Watts\",\n    \"date\": \"2021-04-28T16:22:42.000Z\",\n    \"description\": \"Glacier melt contributing more to sea-level rise than loss of Greenland and Antarctic ice sheets, say experts\",\n    \"image\": \"https://i.guim.co.uk/img/media/7ee51e31092088a3079012acf57ddc71c45930b5/0_117_3500_2100/master/3500.jpg?width=1200\u0026height=630\u0026quality=85\u0026auto=format\u0026fit=crop\u0026overlay-align=bottom%2Cleft\u0026overlay-width=100p\u0026overlay-base64=L2ltZy9zdGF0aWMvb3ZlcmxheXMvdGctZGVmYXVsdC5wbmc\u0026enable=upscale\u0026s=3a8bd91a62cd8ca8dcf21e83feaae84a\",\n    \"logo\": \"https://uploads.guim.co.uk/2018/01/31/TheGuardian_AMP.png\",\n    \"publisher\": \"The Guardian\",\n    \"title\": \"Speed at which world’s glaciers are melting has doubled in 20 years\",\n    \"url\": \"http://www.theguardian.com/environment/2021/apr/28/speed-at-which-worlds-glaciers-are-melting-has-doubled-in-20-years\"\n  }\n}\n```\n\n#### try it\n\n[Try this example on the live API](https://memex-feed-parser.vercel.app/api/scrape/https%3A%2F%2Fwww.theguardian.com%2Fenvironment%2F2021%2Fapr%2F28%2Fspeed-at-which-worlds-glaciers-are-melting-has-doubled-in-20-years)\n\n## Development\n\nRequirements: `npm install --global vercel`\nLogin with `vercel login`\n\nTo run locally:\n\n```bash\nvercel dev\n```\n\nTo deploy as a test deployment:\n\n```bash\nvercel\n```\n\nTo deploy to production:\n\n```bash\nvercel --prod\n```\n\n### Deploy config\n\nGood cache settings to add to `.vercel/project.json` are:\n\n```\n\"headers\": [\n  {\n    \"source\": \"/api/memex\",\n    \"headers\" : [\n      {\n        \"key\" : \"Cache-Control\",\n        \"value\" : \"public, s-maxage=20, stale-while-revalidate=40\"\n      }\n    ]\n  },\n  {\n    \"source\": \"/api/scrape\",\n    \"headers\" : [\n      {\n        \"key\" : \"Cache-Control\",\n        \"value\" : \"public, s-maxage=2592000, stale-while-revalidate\"\n      }\n    ]\n  }\n]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblahah%2Fvercel-headless-scraper-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblahah%2Fvercel-headless-scraper-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblahah%2Fvercel-headless-scraper-api/lists"}