{"id":13961765,"url":"https://github.com/nberlette/dql","last_synced_at":"2025-07-17T05:32:05.399Z","repository":{"id":44371899,"uuid":"512306480","full_name":"nberlette/dql","owner":"nberlette","description":"Web Scraping with Deno: DOM + GraphQL","archived":false,"fork":false,"pushed_at":"2024-08-06T22:42:13.000Z","size":66,"stargazers_count":13,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-25T22:16:32.344Z","etag":null,"topics":["deno","deno-deploy","denoland","dom","dom-parser","dql","graphql","graphql-scraper","scraper","webscraping"],"latest_commit_sha":null,"homepage":"https://deno.land/x/dql","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nberlette.png","metadata":{"funding":{"ko_fi":"nberlette"},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-07-09T23:40:52.000Z","updated_at":"2024-11-05T21:03:51.000Z","dependencies_parsed_at":"2024-08-07T01:44:05.615Z","dependency_job_id":"90a850a2-1a70-4c73-91e3-cdea5bc9b1a7","html_url":"https://github.com/nberlette/dql","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/nberlette/dql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nberlette%2Fdql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nberlette%2Fdql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nberlette%2Fdql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nberlette%2Fdql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nberlette","download_url":"https://codeload.github.com/nberlette/dql/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nberlette%2Fdql/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265568991,"owners_count":23789640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deno","deno-deploy","denoland","dom","dom-parser","dql","graphql","graphql-scraper","scraper","webscraping"],"created_at":"2024-08-08T17:01:25.463Z","updated_at":"2025-07-17T05:32:05.247Z","avatar_url":"https://github.com/nberlette.png","language":"TypeScript","funding_links":["https://ko-fi.com/nberlette"],"categories":["TypeScript"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# [🦕 DQL](https://deno.land/x/dql)\n\n### _**Web Scraping with Deno  –  DOM + GraphQL**_\n\n\u003c/div\u003e\n\n---\n\n**`DQL`** is a web scraping module for Deno and Deno Deploy that integrates the power of [**GraphQL Queries**](https://graphql.org/learn/queries) with the DOM tree of a remote webpage or HTML document fragment. This is a fork of [**DenoQL**](https://deno.land/x/denoql) with some heavy refactoring and some additional features:\n\n- [x] Compatibility with the [**Deno Deploy**](https://deno.com/deploy) architecture\n- [x] Ability to pass variables alongside all queries\n- [x] New state-management class with additional methods\n- [x] Modular project structure (as opposed to a mostly single-file design)\n- [x] Improved types and schema structure\n\n\u003e **Note**: _This is a work-in-progress and there is still a lot to be done._\n\n### 🛝  [**`GraphQL Playground`**](https://dql.deno.dev)\n\n### 📝  [**`HackerNews Scraper`**](https://dash.deno.com/playground/dql-hn)\n\n### 🚛  [**`Junkyard Scraper`**](https://dash.deno.com/playground/dirty-sparrow-69)\n\n---\n\n## `useQuery`\n\nThe primary function exported by the module is the workhorse named `useQuery`:\n\n```ts\nimport { useQuery } from \"https://deno.land/x/dql/mod.ts\";\n\nconst data = await useQuery(`query { ... }`);\n```\n\n### `QueryOptions`\n\nYou can also provide a `QueryOptions` object as the second argument of `useQuery`, to further control the behavior of your query requests. All properties are optional.\n\n```ts\nconst data = await useQuery(`query { ... }`, {\n  concurrency: 8, // passed directly to PQueue initializer\n  fetch_options: { // passed directly to Fetch API requests\n    headers: {\n      \"Authorization\": \"Bearer ghp_a5025a80a24defd0a7d06b4fc215bb5635a167c6\",\n    },\n  },\n  variables: {}, // variables defined in your queries\n  operationName: \"\", // when using multiple queries\n});\n```\n\n## `createServer`\n\nWith [**Deno Deploy**](https://dash.deno.com/new), you can deploy **`DQL`** with a GraphQL Playground in **only 2 lines of code**:\n\n```ts\nimport { createServer } from \"https://deno.land/x/dql/mod.ts\";\n\ncreateServer(80, { endpoint: \"https://dql.deno.dev\" });\n```\n\n`🛝` [Try the **GraphQL Playground** at **`dql.deno.dev`**](https://dql.deno.dev)\\\n`🦕` [View the source code in the **`Deno Playground`**](https://dash.deno.com/playground/dql)\n\n## Command Line Usage (CLI)\n\n```bash\ndeno run -A --unstable https://deno.land/x/dql/serve.ts\n```\n\n#### Custom port (default is `8080`)\n\n```bash\ndeno run -A https://deno.land/x/dql/serve.ts --port 3000\n```\n\n\u003e **Warning**: you need to have the [**Deno CLI**](https://deno.land) installed first.\n\n---\n\n## 💻 Examples\n\n### `🚛` Junkyard Scraper · [**`Deno Playground 🦕`**](https://dash.deno.com/playground/dirty-sparrow-69)\n\n```ts\nimport { useQuery } from \"https://deno.land/x/dql/mod.ts\";\nimport { serve } from \"https://deno.land/std@0.147.0/http/server.ts\";\n\nserve(async (res: Request) =\u003e\n  await useQuery(\n    `\n  query Junkyard (\n    $url: String\n    $itemSelector: String = \"table \u003e tbody \u003e tr\"\n  ) {\n    vehicles: page(url: $url) {\n      totalCount: count(selector: $itemSelector)\n      nodes: queryAll(selector: $itemSelector) {\n        id: index\n        vin:   text(selector: \"td:nth-child(7)\", trim: true)\n        sku:   text(selector: \"td:nth-child(6)\", trim: true)\n        year:  text(selector: \"td:nth-child(1)\", trim: true)\n        model: text(selector: \"td:nth-child(2) \u003e .notranslate\", trim: true)\n        aisle: text(selector: \"td:nth-child(3)\", trim: true)\n        store: text(selector: \"td:nth-child(4)\", trim: true)\n        color: text(selector: \"td:nth-child(5)\", trim: true)\n        date:  attr(selector: \"td:nth-child(8)\", name: \"data-value\")\n        image: src(selector: \"td \u003e a \u003e img\")\n      }\n    }\n  }`,\n    {\n      variables: {\n        \"url\": \"http://nvpap.deno.dev/action=getVehicles\u0026makes=BMW\",\n      },\n    },\n  )\n    .then((data) =\u003e JSON.stringify(data, null, 2))\n    .then((json) =\u003e\n      new Response(json, {\n        headers: { \"content-type\": \"application/json;charset=utf-8\" },\n      })\n    )\n);\n```\n\n### 📝 HackerNews Scraper · [**`Deno Playground 🦕`**](https://dash.deno.com/playground/dql-hn)\n\n```ts\nimport { useQuery } from \"https://deno.land/x/dql/mod.ts\";\nimport { serve } from \"https://deno.land/std@0.147.0/http/server.ts\";\n\nserve(async (res: Request) =\u003e\n  await useQuery(`\n  query HackerNews (\n    $url: String = \"http://news.ycombinator.com\"\n    $rowSelector: String = \"tr.athing\"\n  ) {\n    page(url: $url) {\n      title\n      totalCount: count(selector: $rowSelector)\n      nodes: queryAll(selector: $rowSelector) {\n        rank: text(selector: \"td span.rank\", trim: true)\n        title: text(selector: \"td.title a\", trim: true)\n        site: text(selector: \"span.sitestr\", trim: true)\n        url: href(selector: \"td.title a\")\n        attrs: next {\n          score: text(selector: \"span.score\", trim: true)\n          user: text(selector: \"a.hnuser\", trim: true)\n          date: attr(selector: \"span.age\", name: \"title\")\n        }\n      }\n    }\n  }`)\n    .then((data) =\u003e JSON.stringify(data, null, 2))\n    .then((json) =\u003e\n      new Response(json, {\n        headers: { \"content-type\": \"application/json;charset=utf-8\" },\n      })\n    )\n);\n```\n\n## License\n\nMIT © [**Nicholas Berlette**](https://github.com/nberlette), based on [DenoQL](https://deno.land/x/denoql).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnberlette%2Fdql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnberlette%2Fdql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnberlette%2Fdql/lists"}