{"id":24255922,"url":"https://github.com/3on/scrap.js","last_synced_at":"2025-09-23T18:30:25.598Z","repository":{"id":2341152,"uuid":"3303412","full_name":"3on/scrap.js","owner":"3on","description":"Scrapping tool for node.js","archived":false,"fork":false,"pushed_at":"2012-02-03T10:11:10.000Z","size":189,"stargazers_count":5,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-07T08:18:36.244Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"SchemaStore/schemastore","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/3on.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-01-30T08:48:42.000Z","updated_at":"2014-03-23T04:03:45.000Z","dependencies_parsed_at":"2022-09-01T13:31:25.194Z","dependency_job_id":null,"html_url":"https://github.com/3on/scrap.js","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/3on/scrap.js","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3on%2Fscrap.js","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3on%2Fscrap.js/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3on%2Fscrap.js/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3on%2Fscrap.js/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/3on","download_url":"https://codeload.github.com/3on/scrap.js/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/3on%2Fscrap.js/sbom","scorecard":{"id":3657,"data":{"date":"2025-08-11","repo":{"name":"github.com/3on/scrap.js","commit":"4a0dab19b1710337a3b641f8bfb9cd9ab5564e8d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Code-Review","score":0,"reason":"Found 0/25 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-14T13:06:11.776Z","repository_id":2341152,"created_at":"2025-08-14T13:06:11.776Z","updated_at":"2025-08-14T13:06:11.776Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276047144,"owners_count":25575876,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-20T02:00:10.207Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-15T04:22:22.147Z","updated_at":"2025-09-23T18:30:25.230Z","avatar_url":"https://github.com/3on.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Scrap.js\n========\n\nScrapping websites made easy. It handles redirections, cookies, get/post, string/binary/dom, jquery ... for you!\n\n```\nnpm install Scrap\n```\n\n\nExamples\n========\n\n\nDownload images from Wikipedia front-page\n-----------------------------------------\n\nThis example makes use of jQuery to traverse the page, and shows how to download binary files.\n\n```coffeescript\n# Create a new Scrap with the base for all the following requests\nwikipedia = new Scrap\n  path: 'http://en.wikipedia.org/'\n\n# Download the front page as HTML\nwikipedia.get '/', type: 'html', (window) -\u003e\n\n\t# We get a window element with jQuery\n\t$ = window.$\n\n\t# Using jQuery we iterate over all the images\n\t$('img').each -\u003e\n\n\t\t# We get the url and basename\n\t\turl = $(this).attr('src')\n\t\tbasename = path.basename url\n\n\t\t# We get the image as binary\n\t\twikipedia.get url, type: 'binary', (file) -\u003e\n\n\t\t\t# And save it to the disk\n\t\t\tfs.writeFile 'images/' + basename, file\n```\n\nDownload all the excerpts from Wikipedia front-page links\n---------------------------------------------------------\n\nThis example shows how to download the page as string and use regular expressions with [jsMatch](https://github.com/vjeux/jsMatch) to extract meaningful parts.\n\n```coffeescript\n# Create a new Scrap with the base for all the following requests\nwikipedia = new Scrap\n\tpath: 'http://en.wikipedia.org/'\n\n# Download the front page as string\nwikipedia.get '/', (page) -\u003e\n\n\t# Get all the wiki article links using a regex\n\turls = match.all(page, '\u003ca href=\"(/wiki/[^:\"]+)\"')\n\n\t# Request all the urls at once\n\twikipedia.get urls, (page, url) -\u003e\n\n\t\t# Display useful information from the page\n\t\tconsole.log\n\t\t\turl: url\n\t\t\ttitle: match(page, '\u003ch1[^\u003e]+\u003e(.*?)\u003c\\/h1\u003e')\n\t\t\texcerpt: match(page, '\u003cp\u003e(.*?)\u003c\\/p\u003e').replace(/\u003c[^\u003e]+\u003e/g, '')\n```\n\nThanks to NodeJS asynchronous download, this example runs 3 times faster than the [same version written synchronously in php](https://gist.github.com/1724677).\n\nAPI\n===\n\nAPI\n---\n\n- **new Scrap**([options])\n- **scrap.get**(url, [options], [callback])\n\nAll the 4 HTTP methods ```get```, ```put```, ```post``` and ```delete``` share the same definition.\n\nOptions\n-------\n\n- **path**: Base path of the website. The following request paths will be relative to this path.\n- **type**: You can chose several post filters \n  - ```'string'```(default) Raw string.\n  - ```'binary'``` [NodeJS Buffer](http://nodejs.org/docs/latest/api/buffers.html).\n  - ```'html'``` [jsdom](https://github.com/tmpvar/jsdom) using [htmlparser](https://github.com/tautologistics/node-htmlparser).\n  - ```'html5'``` [jsdom](https://github.com/tmpvar/jsdom) using [html5 parser](https://github.com/aredridel/html5). Beware, it is very slow.\n- **filter**: A function used to edit the html string before it is parsed.\n- **cookies**: An object to provide additional cookies.\n- **headers**: An object to provide additional headers.\n- **data**: The content of a POST request.\n  - String: The content as-is.\n  - Object: Converted to text with [querystring](http://nodejs.org/docs/latest/api/querystring.html).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3on%2Fscrap.js","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F3on%2Fscrap.js","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F3on%2Fscrap.js/lists"}