{"id":20692340,"url":"https://github.com/kazedevid/scraped-core","last_synced_at":"2025-07-13T15:35:33.230Z","repository":{"id":204218107,"uuid":"711366168","full_name":"KazeDevID/scraped-core","owner":"KazeDevID","description":"Serves to scrape the core of the website.","archived":false,"fork":false,"pushed_at":"2023-10-29T03:17:18.000Z","size":9,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-28T22:40:21.911Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KazeDevID.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-29T03:04:35.000Z","updated_at":"2023-10-29T03:29:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"e97162c6-faf8-4247-ab5e-3f1745075c07","html_url":"https://github.com/KazeDevID/scraped-core","commit_stats":null,"previous_names":["kazedevid/scraped-core"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KazeDevID%2Fscraped-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KazeDevID%2Fscraped-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KazeDevID%2Fscraped-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KazeDevID%2Fscraped-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KazeDevID","download_url":"https://codeload.github.com/KazeDevID/scraped-core/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242956179,"owners_count":20212449,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T23:20:46.071Z","updated_at":"2025-03-11T01:33:36.205Z","avatar_url":"https://github.com/KazeDevID.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scraped-core\n\n\u003e Serves to scrape the core of the website.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n## :cloud: Installation\n\n```sh\n# Using npm\nnpm install --save scraped-core\n\n# Using yarn\nyarn add scraped-core\n```\n\n\n\n\n\n\n\n\n\n\n\n\n\n## :clipboard: Example\n\n\n\n```js\nconst scrapedCore = require(\"scraped-core\")\n\n// Promise interface\nconst data = scrapedCore(`\n    \u003ch1 class='heading'\u003eHello World\u003c/h1\u003e\n    \u003cimg src=\"test.jpg\" /\u003e\n    \u003cp\u003eTest\u003c/p\u003e\n    \u003cul\u003e\n        \u003cli\u003e1\u003c/li\u003e\n        \u003cli\u003e2\u003c/li\u003e\n        \u003cli\u003e3\u003c/li\u003e\n        \u003cli\u003e4\u003c/li\u003e\n        \u003cli\u003e5\u003c/li\u003e\n        \u003cli\u003e\u003ci\u003e6\u003c/i\u003e\u003c/li\u003e\n    \u003c/ul\u003e\n`, {\n    title: \".heading\"\n  , desc: \"p\"\n  , avatar: {\n        selector: \"img\"\n      , attr: \"src\"\n    }\n  , items: {\n        listItem: \"ul \u003e li\"\n      , data: {\n            content: {\n                how: \"text\"\n            }\n        }\n    }\n})\n\nconsole.log(data)\n// { title: 'Hello World',\n//   desc: 'Test',\n//   avatar: 'test.jpg',\n//   items:\n//    [ { content: '1' },\n//      { content: '2' },\n//      { content: '3' },\n//      { content: '4' },\n//      { content: '5' },\n//      { content: '6' } ] }\n```\n\n\n\n\n\n\n\n\n\n\n\n## :question: Get Help\n\nThere are few ways to get help:\n\n\n\n 1. Please [post questions on Stack Overflow](https://stackoverflow.com/questions/ask). You can open issues with questions, as long you add a link to your Stack Overflow question.\n 2. For bug reports and feature requests, open issues. :bug:\n\n\n## :memo: Documentation\n\n\n### `scrapedCore($, opts)`\nScrapes the data in the provided element.\n\nFor the format of the selector, please refer to the [Selectors section of the Cheerio library](https://github.com/cheeriojs/cheerio#-selector-context-root-)\n\n#### Params\n\n- **Cheerio|String** `$`: The input element or the HTML as a string.\n- **Object** `opts`: An object containing the scraping information.\n  If you want to scrape a list, you have to use the `listItem` selector:\n\n   - `listItem` (String): The list item selector.\n   - `data` (Object): The fields to include in the list objects:\n      - `\u003cfieldName\u003e` (Object|String): The selector or an object containing:\n         - `selector` (String): The selector.\n         - `convert` (Function): An optional function to change the value.\n         - `how` (Function|String): A function or function name to access the\n           value.\n         - `attr` (String): If provided, the value will be taken based on\n           the attribute name.\n         - `trim` (Boolean): If `false`, the value will *not* be trimmed\n           (default: `true`).\n         - `closest` (String): If provided, returns the first ancestor of\n           the given element.\n         - `eq` (Number): If provided, it will select the *nth* element.\n         - `texteq` (Number): If provided, it will select the *nth* direct text child.\n           Deep text child selection is not possible yet.\n           Overwrites the `how` key.\n         - `listItem` (Object): An object, keeping the recursive schema of\n           the `listItem` object. This can be used to create nested lists.\n\n  **Example**:\n  ```js\n  {\n     articles: {\n         listItem: \".article\"\n       , data: {\n             createdAt: {\n                 selector: \".date\"\n               , convert: x =\u003e new Date(x)\n             }\n           , title: \"a.article-title\"\n           , tags: {\n                 listItem: \".tags \u003e span\"\n             }\n           , content: {\n                 selector: \".article-content\"\n               , how: \"html\"\n             }\n           , traverseOtherNode: {\n                 selector: \".upperNode\"\n               , closest: \"div\"\n               , convert: x =\u003e x.length\n             }\n         }\n     }\n  }\n  ```\n\n  If you want to collect specific data from the page, just use the same\n  schema used for the `data` field.\n\n  **Example**:\n  ```js\n  {\n       title: \".header h1\"\n     , desc: \".header h2\"\n     , avatar: {\n           selector: \".header img\"\n         , attr: \"src\"\n       }\n  }\n  ```\n\n#### Return\n- **Object** The scraped data.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n## :yum: How to contribute\nHave an idea? Found a bug? See [how to contribute][contributing].\n\n\n## :sparkling_heart: Support my projects\nI open-source almost everything I can, and I try to reply to everyone needing help using these projects. Obviously,\nthis takes time. You can integrate and use these projects in your applications *for free*! You can even change the source code and redistribute (even resell it).\n\n\n\nThanks!\n\n\n\n\n\n\n\n\n\n\n## License\n\n[MIT][license] © [KazeDevID][website]\n\n\n\n\n\n\n[license]: /LICENSE\n[website]: https://github.com/KazeDevID","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkazedevid%2Fscraped-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkazedevid%2Fscraped-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkazedevid%2Fscraped-core/lists"}