{"id":19383981,"url":"https://github.com/leandro-bertoluzzi/scrap-to-api","last_synced_at":"2026-04-13T06:49:55.752Z","repository":{"id":159506464,"uuid":"484229749","full_name":"Leandro-Bertoluzzi/scrap-to-api","owner":"Leandro-Bertoluzzi","description":"API to get the content of web pages via scraping","archived":false,"fork":false,"pushed_at":"2022-09-25T01:45:02.000Z","size":116,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-07T07:14:31.337Z","etag":null,"topics":["docker","docker-compose","nodejs","puppeteer","scraper"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Leandro-Bertoluzzi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-21T23:06:11.000Z","updated_at":"2023-05-12T22:17:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"b07dcf48-c189-4435-81b8-4d3586bb7b1a","html_url":"https://github.com/Leandro-Bertoluzzi/scrap-to-api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Leandro-Bertoluzzi%2Fscrap-to-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Leandro-Bertoluzzi%2Fscrap-to-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Leandro-Bertoluzzi%2Fscrap-to-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Leandro-Bertoluzzi%2Fscrap-to-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Leandro-Bertoluzzi","download_url":"https://codeload.github.com/Leandro-Bertoluzzi/scrap-to-api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240521037,"owners_count":19814694,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","nodejs","puppeteer","scraper"],"created_at":"2024-11-10T09:28:35.392Z","updated_at":"2026-04-13T06:49:55.747Z","avatar_url":"https://github.com/Leandro-Bertoluzzi.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eScrap to API\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Github top language\" src=\"https://img.shields.io/github/languages/top/Leandro-Bertoluzzi/scrap-to-api?color=56BEB8\"\u003e\n\n  \u003cimg alt=\"Github language count\" src=\"https://img.shields.io/github/languages/count/Leandro-Bertoluzzi/scrap-to-api?color=56BEB8\"\u003e\n\n  \u003cimg alt=\"Repository size\" src=\"https://img.shields.io/github/repo-size/Leandro-Bertoluzzi/scrap-to-api?color=56BEB8\"\u003e\n\n  \u003cimg alt=\"License\" src=\"https://img.shields.io/github/license/Leandro-Bertoluzzi/scrap-to-api?color=56BEB8\"\u003e\n\u003c/p\u003e\n\n\u003c!-- Status --\u003e\n\n\u003ch4 align=\"center\"\u003e\n\t🚧 Scrap to API 🚀 Under construction...  🚧\n\u003c/h4\u003e\n\n\u003chr\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#dart-about\"\u003eAbout\u003c/a\u003e \u0026#xa0; | \u0026#xa0;\n  \u003ca href=\"#sparkles-features\"\u003eFeatures\u003c/a\u003e \u0026#xa0; | \u0026#xa0;\n  \u003ca href=\"#rocket-technologies\"\u003eTechnologies\u003c/a\u003e \u0026#xa0; | \u0026#xa0;\n  \u003ca href=\"#white_check_mark-requirements\"\u003eRequirements\u003c/a\u003e \u0026#xa0; | \u0026#xa0;\n  \u003ca href=\"#checkered_flag-starting\"\u003eStarting\u003c/a\u003e \u0026#xa0; | \u0026#xa0;\n  \u003ca href=\"#development\"\u003eDevelopment\u003c/a\u003e \u0026#xa0; | \u0026#xa0;\n  \u003ca href=\"#memo-license\"\u003eLicense\u003c/a\u003e \u0026#xa0; | \u0026#xa0;\n  \u003ca href=\"https://github.com/Leandro-Bertoluzzi\" target=\"_blank\"\u003eAuthor\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cbr\u003e\n\n## :dart: About\n\nProxy API to get info from MAL (My Anime List) website via web scraping.\n\n## :sparkles: Features\n\n:heavy_check_mark: REST API\\\n:heavy_check_mark: Scraping of websites via Puppeteer\n\n## :rocket: Technologies\n\nThe following tools were used in this project:\n\n-   [Node.js](https://nodejs.org/en/)\n-   [Docker](https://www.docker.com/)\n-   [Puppeteer](https://github.com/puppeteer/puppeteer)\n\n## :white_check_mark: Requirements\n\nBefore starting :checkered_flag:, you need to have [Node](https://nodejs.org/en/) installed. You would also need [Docker](https://www.docker.com/) if you want to run the containers.\n\n## :checkered_flag: Starting\n\n```bash\n# Clone this project\n$ git clone https://github.com/Leandro-Bertoluzzi/scrap-to-api\n\n# Access\n$ cd scrap-to-api\n\n# Set up environment variables\n$ cp .env.example .env\n# Edit .env and adjust values as needed\n\n# Install dependencies\n$ npm install\n\n# Run the project (option 1: With Docker compose)\n$ docker-compose up\n\n# Run the project (option 2: Locally with Node.js)\n$ npm run start:dev\n\n# In any case, the API will initialize in \u003chttp://localhost:8000\u003e\n```\n\n### Environment configuration\n\nThe project uses environment variables for configuration. You can set these in a `.env` file at the root of the project. An example `.env.example` file is provided for reference.\n\nThe environment variables include:\n\n| Variable        | Description                                             | Default Value             |\n| --------------- | ------------------------------------------------------- | ------------------------- |\n| `NODE_ENV`      | The environment mode (development, production, etc.)    | `development`             |\n| `HOST`          | The host address for the API server                     | `0.0.0`                   |\n| `PORT`          | The port number for the API server                      | `8000`                    |\n| `MAL_BASE_URL`  | The base URL for MyAnimeList                            | `https://myanimelist.net` |\n\n## Development\n\n### Testing\n\nThe project includes unit and integration tests. To run the tests, use the following commands:\n\n```bash\n# Run unit tests\n$ npm run test:unit\n\n# Run integration tests\n$ npm run test:integration\n\n# Run all tests\n$ npm test\n\n# Run tests with coverage report\n$ npm run test:coverage\n```\n\n## :memo: License\n\nThis project is under license from MIT. For more details, see the [LICENSE](LICENSE.md) file.\n\nMade with :heart: by \u003ca href=\"https://github.com/Leandro-Bertoluzzi\" target=\"_blank\"\u003eLeandro Bertoluzzi\u003c/a\u003e\n\n\u003ca href=\"#top\"\u003eBack to top\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleandro-bertoluzzi%2Fscrap-to-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleandro-bertoluzzi%2Fscrap-to-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleandro-bertoluzzi%2Fscrap-to-api/lists"}