{"id":13622774,"url":"https://github.com/brahma-dev/metafetch","last_synced_at":"2026-03-04T06:03:32.930Z","repository":{"id":24357506,"uuid":"27756104","full_name":"brahma-dev/metafetch","owner":"brahma-dev","description":"NodeJS package that fetches a given URL's title, description, images, links etc.","archived":false,"fork":false,"pushed_at":"2026-02-28T14:39:35.000Z","size":280,"stargazers_count":21,"open_issues_count":0,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-28T14:44:09.979Z","etag":null,"topics":["javascript","meta-tags","scraper","scraping-websites"],"latest_commit_sha":null,"homepage":"https://www.npmjs.org/package/metafetch","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brahma-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2014-12-09T08:05:25.000Z","updated_at":"2026-02-28T14:39:39.000Z","dependencies_parsed_at":"2024-04-03T16:41:52.899Z","dependency_job_id":"91a3564f-ff7a-4f52-9be8-eecdd8175b06","html_url":"https://github.com/brahma-dev/metafetch","commit_stats":{"total_commits":118,"total_committers":6,"mean_commits":"19.666666666666668","dds":0.5,"last_synced_commit":"86f6b20245d77d821c76bab290058d1b1e6cb3bc"},"previous_names":["afzaalace/metafetch"],"tags_count":60,"template":false,"template_full_name":null,"purl":"pkg:github/brahma-dev/metafetch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahma-dev%2Fmetafetch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahma-dev%2Fmetafetch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahma-dev%2Fmetafetch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahma-dev%2Fmetafetch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brahma-dev","download_url":"https://codeload.github.com/brahma-dev/metafetch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brahma-dev%2Fmetafetch/sbom","scorecard":{"id":171356,"data":{"date":"2025-08-11","repo":{"name":"github.com/brahma-dev/metafetch","commit":"9123c780bb94ebf1fef0f933c4fd3579006adbb1"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.6,"checks":[{"name":"Maintained","score":5,"reason":"7 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 5","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/28 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/build.yml:1","Warn: no topLevel permission defined: .github/workflows/npm-publish.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":1,"reason":"dependency not pinned by hash detected -- score normalized to 1","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:27: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/build.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:50: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/build.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:56: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/build.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:61: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:71: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/build.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/build.yml:74: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/npm-publish.yml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/npm-publish.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/npm-publish.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/npm-publish.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/npm-publish.yml:35: update your workflow using https://app.stepsecurity.io/secureworkflow/brahma-dev/metafetch/npm-publish.yml/main?enable=pin","Warn: npmCommand not pinned by hash: .github/workflows/build.yml:37","Info:   0 out of   6 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   4 third-party GitHubAction dependencies pinned","Info:   1 out of   2 npmCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":6,"reason":"4 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-fjxv-7rqg-78g4","Warn: Project is vulnerable to: GHSA-p8p7-x288-28g6","Warn: Project is vulnerable to: GHSA-72xf-g2v4-qvf3"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-16T16:34:09.927Z","repository_id":24357506,"created_at":"2025-08-16T16:34:09.927Z","updated_at":"2025-08-16T16:34:09.927Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30073702,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T05:31:57.858Z","status":"ssl_error","status_checked_at":"2026-03-04T05:31:38.462Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["javascript","meta-tags","scraper","scraping-websites"],"created_at":"2024-08-01T21:01:23.906Z","updated_at":"2026-03-04T06:03:32.917Z","avatar_url":"https://github.com/brahma-dev.png","language":"TypeScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Node metafetch\n\n[![Build Status](https://github.com/brahma-dev/metafetch/actions/workflows/build.yml/badge.svg)](https://github.com/brahma-dev/metafetch/actions/workflows/build.yml)\n[![codecov](https://codecov.io/gh/brahma-dev/metafetch/branch/main/graph/badge.svg)](https://codecov.io/gh/brahma-dev/metafetch)\n[![coveralls](https://coveralls.io/repos/github/brahma-dev/metafetch/badge.svg?branch=main)](https://coveralls.io/github/brahma-dev/metafetch?branch=main)\n[![Known Vulnerabilities](https://snyk.io/test/npm/metafetch/badge.svg?style=flat)](https://snyk.io/test/npm/metafetch)\n[![NPM version](https://img.shields.io/npm/v/metafetch.svg?style=flat)](https://www.npmjs.org/package/metafetch)\n\n**Metafetch** is a library to fetch and parse metadata from a web page. It can extract standard meta tags, Open Graph data, JSON-LD, favicons, and feeds, and even render client-side JavaScript to get data from Single-Page Applications (SPAs).\n\n## Table of Contents\n\n-   [Key Features](#key-features)\n-   [Installation](#installation)\n-   [Usage](#usage)\n    -   [Basic Example](#basic-example)\n    -   [Advanced Example (Rendering SPAs \u0026 Retries)](#advanced-example-rendering-spas--retries)\n-   [API](#api)\n    -   [`metafetch.fetch(url, [options])`](#metafetchfetchurl-options)\n    -   [Available Flags](#available-flags)\n-   [Instance Management](#instance-management)\n-   [Response Data](#response-data)\n-   [License](#license)\n\n## Key Features\n\n*   **Comprehensive Metadata:** Extracts title, description, URL, site name, images, links, and more.\n*   **Client-Side Rendering:** Uses **Puppeteer** to render JavaScript-heavy sites, ensuring you get the metadata even from SPAs (e.g., React, Vue, Svelte).\n*   **Robust Network Handling:** Features built-in **request retries with exponential backoff** to handle transient network errors gracefully.\n*   **Rich Content Discovery:**\n    *   Finds the best-quality **favicon** by prioritizing Apple touch icons and largest sizes.\n    *   Discovers **RSS/Atom feeds** linked in the page.\n    *   Parses and flattens structured **JSON-LD** data.\n*   **Advanced Encoding Detection:** Accurately detects character encoding via BOM, HTTP headers, and meta tags to prevent garbled text.\n*   **Highly Configurable:** Fine-tune requests with custom headers, user agents, and feature flags to parse only what you need.\n\n## Installation\n\nUse NPM to install:\n\n```bash\nnpm install metafetch\n```\n\n*Note: **`puppeteer`** is a peer dependency and must be installed separately (`npm install puppeteer`) if you want to use the SPA / client-side rendering feature.*\n\n## Usage\n\n### Basic Example\n\nHere's a simple example using `async/await`.\n\n```javascript\nimport metafetch from 'metafetch';\n\nasync function getMeta(url) {\n  try {\n    const meta = await metafetch.fetch(url);\n    console.log(meta);\n  } catch (err) {\n    console.error(err);\n  }\n}\n\ngetMeta('https://example.com');\n```\n\nThis will output a representative object like this:\n\n```js\n{\n  title: 'Awesome Web Page',\n  description: 'A compelling description of the page content, often used by search engines.',\n  type: 'article',\n  url: 'https://example.com/article-path',\n  originalURL: 'https://example.com',\n  ampURL: 'https://example.com/article-path/amp',\n  siteName: 'Example News',\n  charset: 'utf-8',\n  image: 'https://example.com/images/featured-image.png',\n  favicon: 'https://example.com/favicons/apple-touch-icon.png',\n  feeds: [ 'https://example.com/rss.xml' ],\n  meta: {\n     'og:title': 'Awesome Web Page',\n     'og:description': 'A compelling description...',\n     'ld:headline': 'Awesome Web Page',\n     /* ... other meta and JSON-LD tags ... */\n  },\n  images: [ 'https://example.com/images/featured-image.png', /* ... */ ],\n  links: [ 'https://example.com/another-page', /* ... */ ],\n  headers: { 'content-type': 'text/html; charset=utf-8', /* ... */ },\n  language: 'en'\n}\n```\n\n### Advanced Example (Rendering SPAs \u0026 Retries)\n\nTo get metadata from a client-rendered page or to make your request more robust, use the advanced options.\n\n```javascript\nimport metafetch from 'metafetch';\n\nasync function getSpaMeta(url) {\n  try {\n    const meta = await metafetch.fetch(url, {\n      // Use puppeteer to render the page\n      render: true,\n\n      // Retry up to 2 times on failure\n      retries: 2,\n      retryDelay: 1000, // 1 second base delay\n\n      // Disable parsing of things you don't need\n      flags: {\n        images: false,\n        links: false\n      },\n      \n      // Pass custom options to the underlying fetch call\n      fetch: {\n        headers: {\n          'Accept-Language': 'en-US,en;q=0.9'\n        }\n      }\n    });\n\n    console.log('Title:', meta.title);\n    console.log('Description:', meta.description);\n\n  } catch (err) {\n    console.error(err);\n  }\n}\n\ngetSpaMeta('https://my-single-page-app.com');\n```\n\n### Dynamic User-Agent Example\n\nThe default User-Agent is static. To avoid being blocked and to mimic a real, up-to-date browser, you can supply a fresh User-Agent for each request using a helper library like `user-agents`.\n\nFirst, install the library:\n\n```bash\nnpm install user-agents\n```\n\nThen, generate a new User-Agent and pass it in the `fetch` options:\n\n```javascript\nimport metafetch from 'metafetch';\nimport UserAgent from 'user-agents';\n\nasync function getMetaWithFreshAgent(url) {\n  try {\n    // Generate a new, random, and up-to-date User-Agent string.\n    const userAgent = new UserAgent();\n    console.log('Using User-Agent:', userAgent.toString());\n\n    const meta = await metafetch.fetch(url, {\n      userAgent: userAgent.toString()\n    });\n\n    console.log(meta.title);\n  } catch (err) {\n    console.error(err);\n  }\n}\n\ngetMetaWithFreshAgent('https://example.com');\n```\n\n## API\n\n### `metafetch.fetch(url, [options])`\n\nFetches and parses metadata from the given `url`.\n\n#### Options (`FetchOptions`)\n\n| Option | Type | Default | Description |\n|---|---|---|---|\n| `render` | `boolean` | `false` | If `true`, uses Puppeteer to render the page's JavaScript before parsing. |\n| `retries` | `number` | `0` | Number of times to retry the request on failure. |\n| `retryDelay` | `number` | `1000` | Base delay in milliseconds for retries (uses exponential backoff). |\n| `userAgent`| `string` | Firefox | The User-Agent string to use for the request. |\n| `flags` | `object` | `{...}` | An object to enable/disable parsing specific fields. All flags are `true` by default. See the list below for all available flags. |\n| `fetch` | `object` | `{}` | `RequestInit` options passed directly to the `fetch` call (e.g., `{ headers: {...} }`). |\n\n### Available Flags\n\nYou can pass any of these boolean flags in the `flags` object to optimize parsing. For example, `flags: { links: false, images: false }` will skip extracting links and images.\n\n*   `title`\n*   `description`\n*   `type`\n*   `url`\n*   `siteName`\n*   `charset`\n*   `image`\n*   `meta` (includes all meta tags and flattened JSON-LD)\n*   `images`\n*   `links`\n*   `headers`\n*   `language`\n*   `favicon`\n*   `feeds`\n\n### Instance Management\n\nYou can create multiple instances of `Metafetch`, each with its own default User-Agent.\n\n```javascript\nimport { Metafetch } from 'metafetch';\n\n// Create an instance with a custom default User-Agent\nconst myBot = new Metafetch(\"MyPersonalBot/1.0\");\nawait myBot.fetch('https://example.com');\n\n// You can also change it later\nmyBot.setUserAgent(\"MyPersonalBot/2.0\");\nconsole.log(myBot.userAgent); // \"MyPersonalBot/2.0\"\n```\n\n## Response Data\n\n| Field | Type | Description |\n|---|---|---|\n| `title` | `string` | The page's `\u003ctitle\u003e` tag. |\n| `description` | `string` | The `og:description` or `description` meta tag. |\n| `type` | `string` | The `og:type` meta tag (e.g., \"website\", \"article\"). |\n| `url` | `string` | The final URL after redirects. Prioritizes `canonical` or `og:url`. |\n| `originalURL`| `string` | The URL that was originally passed to the fetch method. |\n| `ampURL` | `string` | The URL from a `\u003clink rel=\"amphtml\"\u003e` tag, if present. |\n| `siteName` | `string` | The `og:site_name` meta tag. |\n| `charset` | `string` | The detected character encoding of the page. |\n| `image` | `string` | The `og:image` or `twitter:image` meta tag. |\n| `favicon` | `string` | The best-quality favicon URL found on the page. |\n| `feeds` | `string[]`| An array of RSS/Atom feed URLs discovered on the page. |\n| `meta` | `object` | A key-value object of all meta tags and flattened JSON-LD data. |\n| `images` | `string[]`| An array of all absolute image URLs on the page. |\n| `links` | `string[]`| An array of all absolute hyperlink (`\u003ca\u003e`) URLs on the page. |\n| `headers` | `object` | An object containing the final response's HTTP headers. |\n| `language` | `string` | The content language (ISO 639-1) from the `\u003chtml\u003e` tag or headers. |\n\n\n## License\n\n(The MIT License)\n\nCopyright (c) 2025 Brahma Dev;\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrahma-dev%2Fmetafetch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrahma-dev%2Fmetafetch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrahma-dev%2Fmetafetch/lists"}