{"id":15697219,"url":"https://github.com/cestef/webscraper-exporter","last_synced_at":"2025-05-08T23:45:10.926Z","repository":{"id":40242185,"uuid":"445413443","full_name":"cestef/webscraper-exporter","owner":"cestef","description":"A simple yet powerful prometheus exporter for website performance metrics built using puppeteer","archived":false,"fork":false,"pushed_at":"2023-05-15T16:24:39.000Z","size":5767,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-08T23:45:05.206Z","etag":null,"topics":["metrics-exporter","prometheus","prometheus-exporter","puppeteer","scraper"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cestef.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-07T05:51:36.000Z","updated_at":"2024-10-30T09:22:26.000Z","dependencies_parsed_at":"2024-10-24T02:53:02.548Z","dependency_job_id":"5f01706f-b641-4797-8fda-018ae0c07dfb","html_url":"https://github.com/cestef/webscraper-exporter","commit_stats":null,"previous_names":["cstefflexin/webscraper-exporter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cestef%2Fwebscraper-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cestef%2Fwebscraper-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cestef%2Fwebscraper-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cestef%2Fwebscraper-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cestef","download_url":"https://codeload.github.com/cestef/webscraper-exporter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253166474,"owners_count":21864467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["metrics-exporter","prometheus","prometheus-exporter","puppeteer","scraper"],"created_at":"2024-10-03T19:14:13.485Z","updated_at":"2025-05-08T23:45:10.907Z","avatar_url":"https://github.com/cestef.png","language":"TypeScript","readme":"# Webscraper Exporter\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/53212129/168236403-3a8cc17f-3f1f-4d94-86b9-44bbe1b41a44.png\" style=\"height:70%; width:70%;margin-bottom:20px;margin-top:20px;\"\u003e\n\u003c/p\u003e\n\nA simple yet powerful [`prometheus`](https://prometheus.io) exporter for website performance metrics built using [`puppeteer`](https://pptr.dev/).\n\n## Table of contents \n\n- [Webscraper Exporter](#webscraper-exporter)\n  - [Table of contents](#table-of-contents)\n  - [CLI](#cli)\n    - [Getting started](#getting-started)\n      - [Installation of the CLI](#installation-of-the-cli)\n      - [Configuration](#configuration)\n      - [Using the CLI](#using-the-cli)\n        - [Starting](#starting)\n        - [Creating a project](#creating-a-project)\n        - [Adding, Removing and Listing Templates](#adding-removing-and-listing-templates)\n  - [NodeJS module](#nodejs-module)\n      - [Installation of the module](#installation-of-the-module)\n      - [Using the module](#using-the-module)\n  - [Docker](#docker)\n  - [Exported Data](#exported-data)\n  - [Customization](#customization)\n  - [Contributing](#contributing)\n\n## CLI \n\nCurrently only tested on `macOS` and `Linux`, should support `Windows`\n\n### Getting started\n\n#### Installation of the CLI\n\n\u003cimg src=\"./docs/media/wsce-install.gif\" style=\"height:50%; width:50%;margin-bottom:20px;margin-top:20px\"\u003e\n\nYou can install the cli included to get started quickly with barely no configuration:\n\n```bash\nnpm install --global webscraper-exporter \n```\n\nor \n\n```bash\nyarn global add webscraper-exporter\n```\n\nYou can then check if everything went OK by running:\n```bash\nwsce --version\n```\nThis should print the package's version\n\n#### Configuration\n\nIn your current working directory, create a file named `wsce.config.js` with the following example content:\n\n```js\nmodule.exports = {\n    scraper: {\n        urls: [\"https://google.com\"],\n        puppeteerOptions: {},\n        addons: [],\n        interval: 60_000,\n    },\n    exporter: {\n        port: 9924\n    },\n};\n```\n\nFor further information on the configuration, see [CONFIGURING.md](docs/CONFIGURING.md)\n\n#### Using the CLI\n\nTo see all available options for a command, see `wsce \u003ccommand\u003e -h`;\n\n##### Starting\nYou can start the exporter by simply executing:\n\n```\nwsce start\n```\n\nYou can then access the metrics at http://localhost:9924\n\nIf you want more detailled logs, you can pass the `-v` argument.\n\nTo pass a custom config file, use `-c path/to/file`\n\n##### Creating a project\n\nCreate a project from one of the included templates by running:\n\n```\nwsce init my-project \n```\n\nYou will then be prompted for the project template. You can also directly pass the template to `wsce init`, e.g. : \n\n```\nwsce init my-project --template javascript\n```\n\nBy default, `wsce` includes a typings file for the config so it's easier to fill. If you don't want that, you can pass `--typings=false` to the `init` command.\n\n##### Adding, Removing and Listing Templates\n\n`wsce template` has 3 subcommands:\n\n- `add`: Add a template from a local directory / remote repository\n- `remove`: Remove a saved template\n- `list`: List all saved templates\n\nFor more information on templates, see [TEMPLATES.md](docs/TEMPLATES.md)\n\n## NodeJS module\n\n#### Installation of the module\n\n```\nnpm install --save webscraper-exporter \n```\n\nor \n\n```\nyarn add webscraper-exporter\n```\n\n#### Using the module\n\n```js\nimport { Exporter, Scraper } from \"webscraper-exporter\";\n\nconst scraper = new Scraper({\n    urls: [\"https://cstef.dev\"],\n    addons: [\n        {\n            name: \"Logger\",\n            when: \"before\",\n            run: (browser, page, URL) =\u003e {\n                console.log(`I am running on ${URL}`);\n            },\n        },\n    ],\n    interval: 60_000,\n});\nscraper.start();\nconst exporter = new Exporter({ \n    scraper, \n    port: 9924, \n});\nexporter.start();\n```\n\nFor more examples, see the `examples` folder.\n\n## Docker\n\nThere is a [`Dockerfile`](Dockerfile) included in this repository so you can build your own image for `wsce` (based on the `node:alpine-16` image).\n\nSee [DOCKER.md](docs/DOCKER.md) for more information.\n\n## Exported Data\n\nYou can see the metrics that are exposed in: [EXPORTED_DATA.md](docs/EXPORTED_DATA.md)\n\n## Customization\n\nYou can easily write your own plugins in Javascript to expand `webscraper-exporter`'s functionalities. \n\nSee [ADDONS.md](docs/ADDONS.md) for more information.\n\n## Contributing\n\nPull requests are always welcome if you feel like fixing / expanding something in this project !","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcestef%2Fwebscraper-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcestef%2Fwebscraper-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcestef%2Fwebscraper-exporter/lists"}