{"id":13510200,"url":"https://github.com/redco/goose-parser","last_synced_at":"2025-04-09T21:14:08.506Z","repository":{"id":40722413,"uuid":"43803895","full_name":"redco/goose-parser","owner":"redco","description":"Universal scraping tool, which allows you to extract data using multiple environments","archived":false,"fork":false,"pushed_at":"2019-04-17T14:58:12.000Z","size":955,"stargazers_count":227,"open_issues_count":3,"forks_count":13,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-09T21:14:08.136Z","etag":null,"topics":["browser","crawler","docker","goose","jsdom","nodejs","parser","parsing","phantomjs","scraper","scraping"],"latest_commit_sha":null,"homepage":"https://andrew.red/posts/goose-parser-the-beginning","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-10-07T08:38:07.000Z","updated_at":"2024-12-05T05:31:24.000Z","dependencies_parsed_at":"2022-07-14T06:30:35.618Z","dependency_job_id":null,"html_url":"https://github.com/redco/goose-parser","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redco%2Fgoose-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redco%2Fgoose-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redco%2Fgoose-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redco%2Fgoose-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redco","download_url":"https://codeload.github.com/redco/goose-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248111972,"owners_count":21049578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["browser","crawler","docker","goose","jsdom","nodejs","parser","parsing","phantomjs","scraper","scraping"],"created_at":"2024-08-01T02:01:28.444Z","updated_at":"2025-04-09T21:14:08.479Z","avatar_url":"https://github.com/redco.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","browser"],"sub_categories":[],"readme":"[![mr.Goose](https://i.imgur.com/e0CPF7C.png)](http://goose.show)\n\n# goose-parser \n\n[![CircleCI (all branches)](https://img.shields.io/circleci/project/github/redco/goose-parser.svg)](https://circleci.com/gh/redco/goose-parser)\n[![Codecov](https://img.shields.io/codecov/c/github/redco/goose-parser.svg)](https://codecov.io/gh/redco/goose-parser)\n[![Latest Stable Version](https://img.shields.io/npm/v/goose-parser.svg?style=flat)](https://www.npmjs.com/package/goose-parser)\n[![Total Downloads](https://img.shields.io/npm/dt/goose-parser.svg?style=flat)](https://www.npmjs.com/package/goose-parser)\n[![NPM downloads](https://badgen.net/npm/dm/goose-parser)](https://npmjs.com/package/goose-parser)\n\nThis tool moves routine crawling process to the new level.\nNow it's possible to parse a web page for a moment. \nAll you need is to specify parsing rules based on css selectors. It's so simple as Goose can do it.\nThis library allows to parse such data types as Grid, Collections, and Simple objects.\nParser has support of pagination by extension [goose-paginator](https://github.com/redco/goose-paginator).\nAlso it offers you following features: *actions* to interact with the page and *transforms* to convert parsed data to friendly format.\n\n## Goose Starter Kit\nNow it's easy to start with Goose, just try to use [goose-starter-kit](https://github.com/redco/goose-starter-kit) for it.\n\n## Key features\n* Declarative approach for definition of parsing rules, actions and transformations.\n* Multi environments to run parser on the browser, PhantomJS, Chrome, JsDOM and more.\n* Clear code with the latest features of ES6.\n* Clear and consistent API with promises all the way.\n* Improved [Sizzle](https://sizzlejs.com) format of selectors.\n* Ajax and multi-pages parsing modes.\n* Docker Support.\n* It's easy extendable.\n\n## Installation\n\n```bash\nyarn add goose-parser goose-chrome-environment\n```\n\n## Usage\n\n```JS\nconst Parser = require('goose-parser');\nconst ChromeEnvironment = require('goose-chrome-environment');\n\nconst env = new ChromeEnvironment({\n  url: 'https://www.google.com/search?q=goose-parser',\n});\n\nconst parser = new Parser({ environment: env });\n\n(async function () {\n  try {\n    const results = await parser.parse({\n      actions: [\n        {\n          type: 'wait',\n          timeout: 10 * 1000,\n          scope: '.srg\u003e.g',\n          parentScope: 'body'\n        }\n      ],\n      rules: {\n        scope: '.srg\u003e.g',\n        collection: [[\n          {\n            name: 'url',\n            scope: 'h3.r\u003ea',\n            attr: 'href',\n          },\n          {\n            name: 'text',\n            scope: 'h3.r\u003ea',\n          }\n        ]]\n      }\n    });\n    console.log(results);\n  } catch (e) {\n    console.log('Error occurred:');\n    console.log(e.stack);\n  }\n})();\n```\n\n## Environment\nThis is a special atmosphere where Parser has to be executed. The main purpose of an environment is to provide a method for evaluating JS on the page.\nGoose supports following environments:\n* [PhantomJS](https://github.com/redco/goose-phantom-environment) (executes in NodeJS)\n* [Chrome](https://github.com/redco/goose-chrome-environment) (executes in NodeJS)\n* [JSDom](https://github.com/redco/goose-jsdom-environment) (executes in NodeJS)\n* FireFox (coming soon)\n* [Browser](https://github.com/redco/goose-browser-environment) (executes in Browser)\n\n## Docker usage\n\nFor now it's available to run goose-parser as a docker service.\n\n**Params:**\n\n* *url* - first param is an url to parser\n* *Parsing rules* [optional] - Rules to parse. It's optional, if *--rules-file* specified.\n\n**Options:**\n\n* -e \"DEBUG=*\" - to enable debug mode and see all what happens inside the goose-parser. Reed more about debug [here](https://www.npmjs.com/package/debug).\n* *--rules-file* - to specify rules file. Be aware that you need to mount a folder with rules as a volume to the docker container.\n\nThere are two options to run it:\n\n### Process parsing from the user input\n\n```bash\ndocker run -it --rm -e \"DEBUG=*,-puppeteer:*\" redcode/goose-parser:chrome-1.1.3-parser-0.6.0\\\n    https://www.google.com/search?q=goose-parser\\\n    '{\n      \"actions\": [\n        {\n          \"type\": \"wait\",\n          \"scope\": \".g\"\n        }\n      ],\n      \"rules\": {\n        \"scope\": \".g\",\n        \"collection\": [\n          [\n            {\n              \"scope\": \".r\u003ea h3\",\n              \"name\": \"name\"\n            },\n            {\n              \"scope\": \".r\u003ea:eq(0)\",\n              \"name\": \"link\",\n              \"attr\": \"href\"\n            }\n          ]\n        ]\n      }\n    }'\n```\n\n### Process parsing from the mounted file with parsing rules\n\nCreate a file `rules/rules.json` which contains parser rules and run following command:\n\n```bash\ndocker run -it --rm --volume=\"`pwd`/rules:/app/rules:ro\" -e \"DEBUG=*,-puppeteer:*\" redcode/goose-parser:chrome-1.1.3-parser-0.6.0 --rules-file=\"/app/rules/rules.json\" 'https://www.google.com/search?q=goose-parser'\n```\n\n## Documentation\nBased on the code you can find detailed documentation about [actions](https://github.com/redco/goose-parser/tree/master/lib/actions) and [transformations](https://github.com/redco/goose-parser/tree/master/lib/transforms)\n\nAPI reference - coming soon\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredco%2Fgoose-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredco%2Fgoose-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredco%2Fgoose-parser/lists"}