{"id":19589103,"url":"https://github.com/popcorn-official/pop-api-scraper","last_synced_at":"2025-08-08T13:10:30.536Z","repository":{"id":57327400,"uuid":"115551152","full_name":"popcorn-official/pop-api-scraper","owner":"popcorn-official","description":"The base modules for the popcorn-api scraper","archived":false,"fork":false,"pushed_at":"2022-12-06T19:14:14.000Z","size":799,"stargazers_count":19,"open_issues_count":6,"forks_count":25,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-07-29T23:24:26.802Z","etag":null,"topics":["cheerio","http","popcorn","popcorn-api","popcorn-time"],"latest_commit_sha":null,"homepage":"https://popcorn-official.github.io/pop-api-scraper/manual/index.html","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/popcorn-official.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-27T19:28:04.000Z","updated_at":"2025-04-24T04:08:34.000Z","dependencies_parsed_at":"2023-01-23T15:15:13.854Z","dependency_job_id":null,"html_url":"https://github.com/popcorn-official/pop-api-scraper","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/popcorn-official/pop-api-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/popcorn-official%2Fpop-api-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/popcorn-official%2Fpop-api-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/popcorn-official%2Fpop-api-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/popcorn-official%2Fpop-api-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/popcorn-official","download_url":"https://codeload.github.com/popcorn-official/pop-api-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/popcorn-official%2Fpop-api-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269425732,"owners_count":24414745,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cheerio","http","popcorn","popcorn-api","popcorn-time"],"created_at":"2024-11-11T08:17:07.977Z","updated_at":"2025-08-08T13:10:30.502Z","avatar_url":"https://github.com/popcorn-official.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pop-api-scraper\n\n[![Build Status](https://travis-ci.org/popcorn-official/pop-api-scraper.svg?branch=master)](https://travis-ci.org/popcorn-official/pop-api-scraper)\n[![Windows Build](https://img.shields.io/appveyor/ci/ChrisAlderson/pop-api-scraper/master.svg?label=windows)](https://ci.appveyor.com/project/ChrisAlderson/pop-api-scraper)\n[![Coverage Status](https://coveralls.io/repos/github/popcorn-official/pop-api-scraper/badge.svg?branch=master)](https://coveralls.io/github/popcorn-official/pop-api-scraper?branch=master)\n[![Dependency Status](https://david-dm.org/popcorn-official/pop-api-scraper.svg)](https://david-dm.org/popcorn-official/pop-api-scraper)\n[![devDependencies Status](https://david-dm.org/popcorn-official/pop-api-scraper/dev-status.svg)](https://david-dm.org/popcorn-official/pop-api-scraper?type=dev)\n\n## Features\n\nThe pop-api-scraper project aims to provide the core modules for the\n[`popcorn-api`](https://github.com/popcorn-official/popcorn-api) scraper, but\ncan also be used for other purposes by using middleware.\n - Strategy pattern with providers\n - Cronjobs\n - Scraper wrapper class\n - HttpService with [`got`](https://github.com/sindresorhus/got)\n\n## Installation\n\n```\n $ npm install --save pop-api-scraper pop-api\n```\n\n## Documentation\n\n - [General documentation](https://popcorn-official.github.io/pop-api-scraper/manual/index.html)\n - [Api docs](https://popcorn-official.github.io/pop-api-scraper/identifiers.html)\n - [Usage](https://popcorn-official.github.io/pop-api-scraper/manual/usage.html)\n - [Middleware](https://popcorn-official.github.io/pop-api-scraper/manual/middleware.html)\n\n## Usage\n\nFor the basic setup you need to create a `Provider` (strategy) the\n`PopApiScraper` instance can use. The `PopApiScraper` implements the strategy\npattern, where the providers are the strategies.\n\nThe example below makes a HTTP GET request to a web service or website. from\nthere on you are free to implement how and what data you want to get from it.\n\n```js\n// ./ExampleProvider.js\nimport { AbstractProvider, HttpService } from 'pop-api-scraper'\n\n// Extend from the internal AbstractProvider.\nexport default class ExampleProvider extends AbstractProvider {\n\n  constructor(PopApiScraper, {name, configs, maxWebRequests = 2}) {\n    super(PopApiScraper, {name, configs, maxWebRequests})\n  }\n\n  // Override the `scrapeConfig` method to get the content from one\n  // configuration.\n  scrapeConfig(config) {\n    // A HTTP service to send HTTP requests.\n    this.httpService = new HttpService({\n      baseUrl: config.baseUrl\n    })\n\n    // HTTP  GET request to: https://jsonplaceholder.typicode.com/posts?foo=bar\n    return this.httpService.get('/posts', config.httpOptions)\n      .then(res =\u003e res.data)\n  }\n\n}\n```\n\nBundle it all up together with\n[`pop-api`](https://github.com/popcorn-official/pop-api):\n\n```js\n// ./index.js\nimport os from 'os'\nimport { PopApi } from 'pop-api'\nimport { join } from 'path'\nimport { Cron, PopApiScraper } from 'pop-api-scraper'\n\nimport ExampleProvider from './ExampleProvider'\n\n(async () =\u003e {\n  try {\n    // Let the PopApiScraper use the ExampleProvider o scrape data.\n    PopApiScraper.use(ExampleProvider, {\n      name: 'example-provider',\n      configs: [{\n        baseUrl: 'https://jsonplaceholder.typicode.com',\n        httpOptions: {\n          query: {\n            foo: 'bar'\n          }\n        }\n      }],\n      maxWebRequests: 2\n    })\n\n    // Register the PopApiScraper middleware to the pop-api instance.\n    PopApi.use(PopApiScraper, {\n      statusPath: join(...[os.tmpdir(), 'status.json']),\n      updatedPath: join(...[os.tmpdir(), 'updated.json'])\n    })\n    // Optionally you can use the Cron middleware to scrape for content on a\n    // regulat basis.\n    PopApi.use(Cron, {\n      cronTime: '0 0 */6 * * *',\n      start: false\n    })\n\n    // PopApi now has a `scraper` instance.\n    const res = await PopApi.scraper.scrape()\n    console.info(res[0])\n  } catch (err) {\n    console.error(err)\n  }\n})()\n```\n\n## License\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpopcorn-official%2Fpop-api-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpopcorn-official%2Fpop-api-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpopcorn-official%2Fpop-api-scraper/lists"}