{"id":16398126,"url":"https://github.com/hebilicious/graphql-crawler","last_synced_at":"2026-07-18T04:35:16.299Z","repository":{"id":92704804,"uuid":"287851882","full_name":"Hebilicious/graphql-crawler","owner":"Hebilicious","description":"Crawl URLs and generate Sitemaps","archived":false,"fork":false,"pushed_at":"2023-12-05T05:26:20.000Z","size":244,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-23T09:29:28.270Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Hebilicious.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["Hebilicious"]}},"created_at":"2020-08-16T01:19:40.000Z","updated_at":"2023-03-07T05:49:57.000Z","dependencies_parsed_at":"2024-11-09T14:52:11.023Z","dependency_job_id":null,"html_url":"https://github.com/Hebilicious/graphql-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Hebilicious/graphql-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hebilicious%2Fgraphql-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hebilicious%2Fgraphql-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hebilicious%2Fgraphql-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hebilicious%2Fgraphql-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Hebilicious","download_url":"https://codeload.github.com/Hebilicious/graphql-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hebilicious%2Fgraphql-crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265178516,"owners_count":23723275,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T05:11:54.868Z","updated_at":"2025-10-15T10:50:43.539Z","avatar_url":"https://github.com/Hebilicious.png","language":"TypeScript","funding_links":["https://github.com/sponsors/Hebilicious"],"categories":[],"sub_categories":[],"readme":"![GraphQL Crawler CI](https://github.com/Hebilicious/graphql-crawler/workflows/GraphQL%20Crawler%20CI/badge.svg)\n\n# DeepCrawl take-home task\n\n## 🧾Requirements\n\n1. Crawl should be limited to one domain address.\n2. Crawled page URLs should be fetchable either using a query or subscription.\n3. URLs could be used for building sitemaps.\n4. Crawling multiple pages should run in parallel.\n\n### 📚 Stories\n\n1. As a user, I want to be able to start a crawl for my website, so that I would be able to generate a sitemap.\n2. As a user, I want to be able to limit the speed of the crawl. So that I would not take down my website.\n3. As a user, I want to be able to monitor my crawl using GraphQL subscriptions, what URLs it is crawling at the moment and how many he crawled total.\n4. As a user, I want to be able to fetch a paginated list of URLs found, to be able to generate sitemap using the API.\n\n## 👩‍💻Technology choices\n\n**We'll leverage mostyl native JavaScript functionalities to implements this crawler.**\n\n**We'll also use the following runtime libraries :**\n\n-   [Puppeteer](https://github.com/puppeteer/puppeteer) (Page rendering)\n\n    -   We're using a headless puppeteer to render pages before the crawler runs, in order to handle SPA/JavaScript rendered pages.\n    -   Even though pupeteer is a node dependency, you might have to manually install some libs like libxss1. Refer to their documentation if needed.\n\n-   [Apollo Server Fastify](https://github.com/apollographql/apollo-server/blob/main/packages/apollo-server-fastify/README.md) (GraphQL Server)\n\n    -   We're using Fastify over Express as it gives way better performances 🚀 The adoption is increasing, and the [ecosystem](https://www.fastify.io/ecosystem/) is maturing.\n    -   Fastify v3 doesn't work with Apollo [yet](https://github.com/apollographql/apollo-server/pull/4356) 😥, so we're using v2\n    -   We'll use the in-memory PubSub for our websocket implementation, which is fine for this PoC.\n    -   Fastify already uses Pino for logging, and out of the box it can accept a custom pino instance if needed.\n\n-   [Nexus Schema](https://github.com/graphql-nexus/schema) (GraphQL schema building)\n\n    -   In order to build our schema we'll use `@nexus/schema`.\n    -   It is an interesting alternative to type-graphql. It is more recent and you can find an article that compares [both here](https://novvum.io/post/typegraphql-and-graphql-nexus-a-look-at-code-first-apis/).\n    -   The way nexus/schema works is closer to the way graphql.js works, while giving full Typescript support. `nexus:generate` will generate the schema.graphql and the typescript types at once.\n    -   Having used both, I personnally think they're both good and production ready. While type-graphql has a better documentation and is more popular, I find nexus/schema more flexible and readable, and more pleasant to work with.\n\n-   [Threads.js](https://github.com/andywer/threads.js) (Multi-threading)\n    -   In order to implement the 4th requirement, we'll leverage threads.js, which has a syntax closer to the WebWorker API while using `worker_threads` under the hood (when supported).\n    -   By adding a `ts-node` dependency we can write our Worker in typescript without configuring anything.\n\n## 🔬 How to use\n\n**It is recommended that you clone the repository and use an IDE with Typescript support (like VSCode) to review the code. Thanks to [type inference and contextual typings](https://www.typescriptlang.org/docs/handbook/type-inference.html), Typescript provides us with a lot of information. For example most of the return types are inferred and aren't specified in the JSdocs or written out. Functional Programming ❤**\n\n**That's how it should looks in VSCode :**\n![image](./crawlPage.png)\n\n-   Tested with Node 14+ and yarn\n-   The `yarn nexus:generate` script needs to run to generate Typescript types and the graphQL schema. It runs by default as a postinstall script.\n-   All the tests are in the tests directory, use `yarn test` to run everything.\n-   Static documenation is available with `yarn docs`. This is generated directly from the schema.graphql.\n-   The graphQL playground is available using `yarn dev`.\n-   The entry point is `src/server.ts`\n-   The source code will be extensively documented to explain the reasoning behind most choices.\n\n_Types troubleshoot: If some of the nexus related types are not working, try to run `yarn nexus:generate`, restart the TS Server, and restart your IDE._\n\n\u003e CLI instructions\n\n```bash\n# Install the dependencies, this will run `nexus-generate`\nyarn\n\n# Run the test suite. Note that the services.test.ts can fail\n# if your system doesn't have the necessary libs to run puppeteer.\n# If that happens you  probably needs to run something like\n# `apt install libss1` (depends on your OS).\nyarn test\n\n# Start the dev server. Note that this uses nodemon,\n# So the in memory stuff isn't lost when you add/edit the source code.\nyarn dev\n\n#You can now play with the playground 🎉🎉🎉\n\n# This serve the static documentation\nyarn docs\n\n# If you want to build and run with node :\nyarn build \u0026\u0026 node dist/server.js\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhebilicious%2Fgraphql-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhebilicious%2Fgraphql-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhebilicious%2Fgraphql-crawler/lists"}