{"id":21296449,"url":"https://github.com/mawngo/html-parser-js","last_synced_at":"2026-03-06T09:03:53.359Z","repository":{"id":57126230,"uuid":"454028946","full_name":"mawngo/html-parser-js","owner":"mawngo","description":"Html parsing engine inspired by x-ray and scrape-it","archived":false,"fork":false,"pushed_at":"2024-07-14T05:56:00.000Z","size":322,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-19T05:26:35.520Z","etag":null,"topics":["html","js","json","parse","scraper"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/@mawngo/html-parser","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mawngo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-31T14:03:47.000Z","updated_at":"2024-07-14T05:55:52.000Z","dependencies_parsed_at":"2024-06-19T18:08:28.350Z","dependency_job_id":"756337b0-a6ac-46f2-a339-1f5bbe4af95b","html_url":"https://github.com/mawngo/html-parser-js","commit_stats":null,"previous_names":["mawngo/html-parser-js","lana-collections/html-parser-lib"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/mawngo/html-parser-js","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mawngo%2Fhtml-parser-js","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mawngo%2Fhtml-parser-js/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mawngo%2Fhtml-parser-js/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mawngo%2Fhtml-parser-js/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mawngo","download_url":"https://codeload.github.com/mawngo/html-parser-js/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mawngo%2Fhtml-parser-js/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30168609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T07:56:45.623Z","status":"ssl_error","status_checked_at":"2026-03-06T07:55:55.621Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["html","js","json","parse","scraper"],"created_at":"2024-11-21T14:26:39.648Z","updated_at":"2026-03-06T09:03:53.341Z","avatar_url":"https://github.com/mawngo.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Html Parser\n\nParse html to json object using predefined schema. Inspired by [x-ray](https://github.com/matthewmueller/x-ray)\nand [scrape-it](https://github.com/IonicaBizau/scrape-it)\n\n## Installation\n\n```sh\n# Using npm\nnpm install --save @mawngo/html-parser\n```\n\n## Usage\n\nScraping using html-parser with axios\n\n```ts\nimport { Parser } from \"@mawngo/html-parser\";\nimport axios from \"axios\";\n\nconst parser = new Parser();\n\nasync function main() {\n  const html = (await axios.get(\"https://github.com/topics/html\")).data;\n  const data = await parser.parseHtml(html, {\n    selector: {\n      title: \"h3 a:last-child\",\n      user: {\n        selector: \"a:first-child\",\n        scope: \"h3\" // find first element using this scope selector. then apply the selector on this element\n      },\n      stars: {\n        selector: \"span#repo-stars-counter-star\",\n        number: true // parse value as number\n      }\n    },\n    scope: [\".col-md-8 \u003e article\"] // find all element using this scope selector. then apply the selector on each element\n  });\n\n  console.log(data);\n  // [\n  //   { title: \"bootstrap\", user: \"twbs\", stars: 155000 },\n  //   { title: \"electron\", user: \"electron\", stars: 100000 },\n  //   { title: \"storybook\", user: \"storybookjs\", stars: 68600 },\n  //   { title: \"Front-End-Checklist\", user: \"thedaviddias\", stars: 58000 },\n  //   { title: \"html5-boilerplate\", user: \"h5bp\", stars: 52300 },\n  //    ...\n  // ];\n}\n\nmain();\n```\n\nSimplify selector schema using helpers\n\n```ts\nimport { num, obj, Parser, str } from \"@mawngo/html-parser\";\nimport axios from \"axios\";\n\nconst parser = new Parser();\n\nasync function main() {\n  const html = (await axios.get(\"https://github.com/topics/html\")).data;\n  const data = await parser.parseHtml(html,\n    obj({\n      title: \"h3 a:last-child\",\n      user: str(\"a:first-child\", \"h3\"),\n      stars: num(\"span#repo-stars-counter-star\")\n    }, [\".col-md-8 \u003e article\"])\n  );\n\n  console.log(data);\n  // [\n  //   { title: \"bootstrap\", user: \"twbs\", stars: 155000 },\n  //   { title: \"electron\", user: \"electron\", stars: 100000 },\n  //   { title: \"storybook\", user: \"storybookjs\", stars: 68600 },\n  //   { title: \"Front-End-Checklist\", user: \"thedaviddias\", stars: 58000 },\n  //   { title: \"html5-boilerplate\", user: \"h5bp\", stars: 52300 },\n  //    ...\n  // ];\n}\n\nmain();\n\n```\n\n## Documentation\n\n### API\n\n### ```new Parser\u003cP\u003e(options: ParserOptions\u003cP\u003e)```\n\nCreate new parser instance\n\n#### ```options: ParserOptions\u003cP\u003e```\n\n- ```engines?: ParserEngine\u003cP\u003e[]```: Array of custom engines\n- ```nodeFactory?: NodeFactory```: factory class to create nodes. default parser use Cheerio backed node factory\n- ```transforms?: { [key: string]: TransformFunction }```: map of transform function (```built-in transforms```)\n- ```objTransforms?: { [key: string]: TransformFunction }```: map of transform function that apply to object selector\n- ``P`` additional custom selector to support\n\n```ts\nimport { Parser, BasicParser } from \"@mawngo/html-parser\"\n// Default parser\n// included ObjectParserEngine, BooleanParserEngine, NumberParserEngine, DateParserEngine, DefaultParserEngine (StringParserEngine)\nnew Parser();\n\n// Basic parser, only include ObjectParserEngine and DefaultParserEngine (StringParserEngine)\nnew BasicParser();\n\n// Advanced usage with custom engine\nnew Parser\u003cMyCustomSelector\u003e({\n  engines: [new MyCustomEngine()]\n});\n\ninterface MyCustomSelector extends ValueSelector {\n  me: true;\n}\n\nclass MyCustomEngine extends ValueParserEngine\u003cMyCustomSelector\u003e {\n  match(selector: any): boolean {\n    return selector?.me === true;\n  }\n\n  protected parseValue(value: any, _: MyCustomSelector): Promise\u003cany\u003e {\n    // just an example. this engine does nothing\n    return Promise.resolve(value);\n  }\n}\n```\n\n### ```parser.parseHtml\u003cT\u003e(html, selector): Promise\u003cT\u003e```\n\nParse html\n\n```T``` specify the return type\n\n```html: string``` the html to parse\n\n```selector: string | string[] | GeneralSelector\u003cP\u003e``` the selector schema\n\n### Selector Schema\n\nThe selector can be a string, or array of single string follow the\nformat ```[selector]@[attribute] | [built-in transform]```\nwhere the ``[selector]`` is a Jquery selector. ```[built-in transform]``` will be shifted to\nthe ```transforms?: (string | TransformFunction)[]``` array\n\n```ts\nimport { Parser } from \"@mawngo/html-parser\"\n\nconst html = \"\u003ch1\u003eHello\u003c/h1\u003e \u003ch1\u003eWorld\u003c/h1\u003e\";\nconst parser = new Parser();\n\nparser.parseHtml(html, \"h1\"); // \"Hello\"\nparser.parseHtml(html, \"h1@text\"); // \"Hello\"\nparser.parseHtml(html, [\"h1\"]); // [\"Hello\", \"World\"]\nparser.parseHtml(html, \"h1@innerHTML\"); // \"Hello\" (the innerHTML of h1 element)\nparser.parseHtml(html, \"h1@html\"); // \"Hello\" (the innerHTML of h1 element)\nparser.parseHtml(html, \"h1@outerHTML\"); // \"\u003ch1\u003eHello\u003c/h1\u003e\"\n```\n\nThe selector can also an object. Structure of the selector object based on the Selector that parser support or the\nengines that passed to engines options. Common selector properties are:\n\n- ```selector: string | string[] | MapSelector```: The actual selector\n- ```scope?: string | string []```: scope of selector. transform to ```$(scope).find(selector)```\n- ```trim?: boolean```: trim the value before process. default to ```true```\n- ```transforms?: (string | TransformFunction)[]```: list of transform to apply. transform can be a function that take\n  current value and return another value, or name of ```built-in transforms```. if applied when the ```selector``` is an\n  array (select all), the transform will be applied on each item in result list\n- ```arrTransforms?: (string | TransformFunction)[]```: list of arrTransform to apply. arrTransform only apply if the\n  selector is array (select all) .arrTransform can be a function that take current value and return another value, or\n  name of ```built-in transforms```\n\n### Built-in engines\n\nBuilt-in engines to convert parsed data into various data types like date, number, boolean\n\n#### DefaultParserEngine / StringParserEngine\n\nParse value into string, or match string value\n\n```ts\ninterface DefaultSelector {\n  selector: string | string[];\n  string?: boolean; // require true if using StringParserEngine. optional if using DefaultParserEngine\n  default?: string | null; // default value\n  defaultIfEmpty?: string | null; // default value if value is empty\n  match?: string | RegExp;  // match the value and return match\n  defaultIfNoMatch?: string | null; // default value if there is no match\n}\n```\n\n- if complex type or null passed, the parser return default value\n\n#### ObjectParserEngine\n\nEnable support for nested selector map\n\n```ts\ninterface ObjectSelector {\n  selector: { [key: string]: GeneralSelector };// GeneralSelector can be any supported selector\n  trim?: boolean; // apply trim for all sub selector (only if sub selector trim property is not specified)\n  transforms?: (string | TransformFunction)[]; // apply transforms for all sub selector (apply after sub selector transform)\n  arrTransforms?: (string | TransformFunction)[]; // apply arrTransforms for all sub selector (apply after sub selector arrTransform, and only if sub selector is array selector)\n  objTransforms?: (string | TransformFunction)[]; // apply transform to the object result;\n  flat?: boolean; // flatten the object, merge its keys with its parent keys\n}\n```\n\n- arrTransforms will be applied on all object's array fields, and run after fields arrTransform\n- transforms will be applied on all object's fields, and run after fields transform\n- objTransforms will be applied on the object after all of its fields resolved\n- if flat is true, the parser will merge current object keys with its parent (do nothing if specified in root schema)\n\n#### BooleanParserEngine\n\nParse value into boolean\n\n```ts\ninterface BooleanSelector {\n  selector: string | string[];\n  boolean: true; // required\n  truthy?: string | string[]; // list of value to parse as true.\n  falsy?: string | string[];  // list of value to parse as false\n  default?: boolean | null;   // default value\n}\n```\n\n- if complex type or null passed, the parser return default value\n- if number passed the parser parse 0 as false, others as true\n- if falsy, and truthy not provided, the parser parse non-empty string as true, empty string as false\n- if only falsy provided, the parser parse falsy string as false, otherwise true\n- if only truthy provided, the parser parse truthy string as true, otherwise false\n- if both passed, the parser parse truthy string as true, falsy string as false, otherwise default value\n\n#### NumberParserEngine\n\nParse value into number. Use [numeral.js](https://www.npmjs.com/package/numeral) for numeric parsing and formatting, can\nhandler various input string format like \"$10,000.00\", \"3.467TB\", \"76%\", ...\n\n```ts\ninterface NumberSelector {\n  selector: string | string[];\n  number: true; // required\n  int?: boolean; // round value to int\n  default?: number | null; // default value\n  format?: \"number\" | string; // if format string passed then format number to that format (as string)\n  roundMode?: \"round\" | \"floor\" | \"ceil\"; // mode used to round value to int\n}\n```\n\n- if invalid number passed, the parser return default value\n- if complex type or null passed, the parser return default value\n- if boolean passed, the parser return 0 for false and 1 for true\n- if format options is 'number', the parser return value as number (default)\n- if format options is other string, the parser return formatted number string\n- [list of number format](http://numeraljs.com/#format)\n\n#### DateParserEngine\n\nParse value into date. use [day.js](https://www.npmjs.com/package/dayjs) for date parsing and formatting\n\n```ts\ninterface NumberSelector {\n  selector: string | string[];\n  date: true; // required\n  parse?: string | string[]; // format for parsing input string into date\n  format?: \"iso\" | \"date\" | \"dayjs\" | \"timestamp\" | string;  // output format\n  default?: string | Date | number; // default value\n}\n```\n\n- if invalid date passed, the parser return default value\n- if complex type or null passed, the parser return default value\n- the parse options can be an array of string, for parsing different format\n- if number passed, the parser parse the number as timestamp\n- if format options is 'iso', the parser return value as iso date string (default)\n- if format options is 'date', the parser return value as Date object\n- if format options is 'timestamp', the parser return value as timestamp number\n- if format options is 'dayjs', the parser return value as dayjs instance\n- if format options is other string, the parser return formatted date string\n- [list of date format token](https://day.js.org/docs/en/parse/string-format#list-of-all-available-parsing-tokens)\n\n### Schema Helpers\n\nUsing schema helpers function to reduce nesting of your selector. example\n\n```ts\nimport { date, num, obj, Parser } from \"@mawngo/html-parser\";\n\nconst parser = new Parser();\n\n// using schema only\nparser.parseHtml(html, {\n  selector: {\n    number: {\n      selector: \"pagination li\",\n      number: true // required for parsing number\n    },\n    date: {\n      selector: \"span.date\",\n      date: true // required for parsing date\n    },\n    scoped: {\n      selector: \".title\",\n      scope: [\"article\"]\n    }\n  }\n});\n\n// using helpers\nparser.parseHtml(html,\n  obj({\n    number: num(\"pagination li\"),\n    date: date(\"span.date\"),\n    scoped: str(\".title\", [\"article\"])\n  })\n);\n```\n\nList of available helpers\n\n- ```obj(selector: { [key: string]: GeneralSelector }, scopeOrOptions?): ObjectSelector```\n- ```flat(selector: { [key: string]: GeneralSelector }, scopeOrOptions?): ObjectSelector \u0026 { flat: true }```\n- ```num(selector: string | string[], scopeOrOptions?, default?): NumberSelector```\n- ```int(selector: string | string[], scopeOrOptions?, default?): NumberSelector```\n- ```str(selector: string | string[], scopeOrOptions?, default?): StringSelector```\n- ```match(regex: string | RegExp, selector: string | string[], scopeOrOptions?, default?): StringSelector \u0026 { match: /regex/ }```\n- ```bool(selector: string | string[], scopeOrOptions?, truthy?, falsy?): BooleanSelector```\n- ```date(selector: string | string[], scopeOrOptions?, parse?, format?, default?): DateSelector```\n\n### Built-in transforms\n\nAlmost all built-in transforms can be applied on single value, array and object. For array and object, the transform\nwill auto applied to each item (in array) or field (in object)\n\nMarked built-in transforms (* at the start) only support some specific type. They will return the input value if type is\nnot supported\n\n- ```replace(input, match, replaceBy)```: replace matched in string (match is treated as regex).\n- ```match(input, token, flag?, defaultIfNoMatch?)```: match string (match is treated as regex), return matched value.\n- ```matchAll(input, token, flag?, defaultIfNoMatch?)```: like ```match``` but return all matched value as array (empty\n  array if not match).\n- ```split(input, token, flag?)```: split string by token (token is treated as regex). return array of string.\n- ```lowercase(input)```:  convert input to lowercase.\n- ```uppercase(input)```:  convert input to uppercase.\n- ```title(input)```:  convert input to title (upper case first letter of each word).\n\n- ```def(input, default, mode?: \"blank\", values?: [])```: return default if input is in values, or input is blank |\n  empty | falsy | null (based on mode)\n- ```empty(input, default, mode?: \"blank\", values?: [])```: like ```def``` but return empty string instead of default\n\n- ```*join(input, glue?)```: join array into string. Only support array or object (join all object values into string).\n- ```*flat(input, depth?)```: flatten array. Only support array.\n- ```*unique(input)```: remove duplicated element in array. Only support array.\n\n- ```wrap(input, key?)```: if key not specified, return array contain input. if key is specified, return object which\n  have 'key' value is input\n- ```*del(input, match)```: return null if value equal to match. For object, remove all key which have value equal to\n  match. For array, remove all item equal to match\n\n- ```str(input)```: convert input to string\n- ```toString(input)```: call ```toString``` on input (basically convert input to string, but without special treatment\n  for array and object). return string\n- ```json(input)```: call ```JSON.stringify``` on input. return json string\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmawngo%2Fhtml-parser-js","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmawngo%2Fhtml-parser-js","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmawngo%2Fhtml-parser-js/lists"}