{"id":15159602,"url":"https://github.com/nazemmahmud/symfony-scraping","last_synced_at":"2026-01-27T21:51:45.926Z","repository":{"id":192228297,"uuid":"671681546","full_name":"NazemMahmud/symfony-scraping","owner":"NazemMahmud","description":"web scraping with symfony 6, puppeteer, redis, docker with crud operation, some unit tests","archived":false,"fork":false,"pushed_at":"2024-01-28T23:56:13.000Z","size":545,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T16:52:39.028Z","etag":null,"topics":["cloudflare","crud-api","docker","docker-compose","phpunit","proxy-server","puppeteer","redis","symfony6","unit-test"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NazemMahmud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-27T22:27:24.000Z","updated_at":"2024-04-07T05:49:20.000Z","dependencies_parsed_at":"2023-09-03T14:50:47.636Z","dependency_job_id":"c6825041-d2be-433b-a0bd-e3798850e922","html_url":"https://github.com/NazemMahmud/symfony-scraping","commit_stats":null,"previous_names":["nazemmahmud/symfony-scraping"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NazemMahmud%2Fsymfony-scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NazemMahmud%2Fsymfony-scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NazemMahmud%2Fsymfony-scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NazemMahmud%2Fsymfony-scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NazemMahmud","download_url":"https://codeload.github.com/NazemMahmud/symfony-scraping/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247694892,"owners_count":20980731,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudflare","crud-api","docker","docker-compose","phpunit","proxy-server","puppeteer","redis","symfony6","unit-test"],"created_at":"2024-09-26T21:40:41.318Z","updated_at":"2026-01-27T21:51:45.897Z","avatar_url":"https://github.com/NazemMahmud.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [UPDATED: CORS, Docker Redis \u0026 Unit test](#updated)\r\n- [Updated Installation](#updated-installation)\r\n- [CORS](#cors)\r\n- [Redis in Docker](#redis-in-docker)\r\n- [Unit test](#unit-test)\r\n\r\n# Table of Contents\r\n- [Installation](#installation)\r\n- [CORS](#cors)\r\n- [APIs / Postman Collection](#apis--postman-collection)\r\n  - [Documentation](#Documentation)\r\n  - [Collection](#Collection)\r\n- [Implementation](#implementation)\r\n   - [Docker Environment Setup](#docker-environment-setup)\r\n   - [Scrapper Implementation](#scrapper-implementation)\r\n   - [APIs](#apis)\r\n      - [Create/Add Info](#createadd-info)\r\n      - [Index](#index)\r\n      - [Update](#update)\r\n      - [Delete](#delete)\r\n   - [Additionally Implemented](#additionally-implemented)\r\n- [Remaining Tasks](#remaining-tasks)\r\n- [NOTE](#note)\r\n  - [Scraping](#scraping)\r\n  - [Cache Clear](#cache-clear)\r\n\r\n\r\n# Installation:\r\nAfter pulling from the repository\r\n- Copy `.env.example` and paste as `.env`. Change anything such as port number or db configuration if you feel necessary\r\n- Run command `docker compose build --no-cache` for a fresh build\r\n- Run command: `docker compose up -d`\r\n- Go to the bash script of the php related container: run commnad: `docker compose exec php-service bash`\r\n- **Inside the container:** \r\n  - To install composer packages, run command: `composer install`\r\n  - To install npm packages, run command: `npm install`\r\n\r\n# APIS / POSTMAN COLLECTION:\r\n\r\n## Documentation\r\nThere are 4 APIs. This is postman collection link: https://documenter.getpostman.com/view/1449184/2s9Y5cu1WN \r\n\r\nUse it to understand the APIs in details.\r\n\r\n## Collection\r\n- This is the json file of the postman collection:  **Nordstreet Scrape.postman_collection.json** You can find it in the root directory\r\n\r\n\r\n# Implementation\r\n1. Docker environment setup is done for Nginx, PHP 8.2, MySQL 8 \u0026 Redis\r\n2. Also, added a Postman image in docker to visualize the database.\r\n3. Implemented Scrapper using puppeteer stealth (npm package) and scrape.do for free proxy server\r\n4. Company profile information is extracted from scrapped data and stored in the database, using  Symfony 6.2 framework.\r\n5. 4 APIs are implemented: \r\n   1. Create/Add info: \r\n      1. While storing, the registration codes are stored in redis. So that, next time for the existing codes, can reduce the unnecessary scrapping time.\r\n      2. Because of this we don't need any duplicate data checking in DB. Hence, it also reduces DB query time.\r\n   2. Index: \r\n      1. Pagination added. \r\n      2. Also make sure that, if the same pagination information is passed, then no DB operation will take place. It will take data from redis cache\r\n   3. Update:\r\n      1. Checked if the registration code is already exist in another row in DB. Because codes are unique. So, no 2 rows will have same code.\r\n   4. Delete: If ID exist in the DB, then soft delete mechanism is used so that no data is completely removed, but only updated the deleted time.\r\n6. Unit testing using phpunit and guzzle client.\r\n\r\n**Additionally implemented:**\r\n- **Custom validation rule** for each request.\r\n- **Custom Exception handler** for individual error.\r\n- **Log:** customized log handler for the error log, inside the project directory for error response. It helps to identify each error properly. Every day a new file (named with the datetime) will generate when the error occurs.\r\n- **Added indexing** for column registration code\r\n\r\n\r\n# NOTE:\r\n\r\n### Scraping\r\nI used [scrape.do](scrape.do). It provides a limited free plan with a token. \\\r\nFor easy testing the API, I provide my scraping token in the env file. \\\r\nEach scrapping takes mane requests, because for each time 3 pages need to scrape in order to get correct company info. \\\r\nSo, you can use your own token by opening an account in [scrape.do](scrape.do site.\r\n\r\n\r\n### Cache clear:\r\nIf you face this error:\r\n```bash\r\nFailed to remove directory \u0026quot;/var/www/project/var/cache/dev/ContainerCWmhSbR\u0026quot;: rmdir(/var/www/project/var/cache/dev/._Ed+): Directory not empty (500 Internal Server Error) \r\n```\r\nFor this, **clear cache**: \r\n```shell\r\nphp bin/console cache:clear\r\n```\r\n\r\n\r\n# [UPDATED]\r\n- If you don't have this updated code, please update, by pulling the latest code.\r\n- And also, check the env file for the update, in case you missed.\r\n\r\n# Updated Installation\r\n- The dockerfile is updated, so build the docker file again.\r\n```shell\r\ndocker compose build --no-cache\r\n```\r\n\r\n- Then run: `docker compose up -d`\r\n\r\n- new package is installed for this, you have to run `composer install` or `composer update` command again to install the new packages.\r\n```shell\r\ndocker compose exec php-service bash\r\n```\r\n```shell\r\ncomposer install # or run, composer update\r\n```\r\n\r\n- **After installing everything run a cache clear command, because many things are updated now**\r\n```shell\r\nphp bin/console cache:clear\r\n```\r\n\r\n## CORS\r\n- Run `composer install` to install the package for cors control.\r\n- For this backend, allow CORS origin from specific site. \\\r\n  You can see this in `.env.example` file. The variable for this is: `CORS_ALLOW_ORIGIN=` \\\r\n  Update it in `.env` if you feel necessary.\r\n- Then clear the cache: `php bin/console cache:clear`\r\n\r\n\r\n## Redis in Docker\r\n- updated docker compose and dockerfile for redis setup\r\n\r\n## Unit test\r\n- Latest phpunit package has conflict with latest symfony version. So, phpunit package is downgraded for this\r\n- To run unit test, run command from inside docker container:\r\n```shell\r\nphp bin/phpunit\r\n```\r\n- If you want to see each individual test case result, run command:\r\n```shell\r\nphp bin/phpunit --testdox\r\n```\r\n\r\n### Note on Unit Test\r\n- Faced API calling using WebTestCase class. So, instead, used guzzle client.\r\n- The variable value `TEST_BASE_URL` in `.env.test` file used the nginx service name because of docker container. \r\nIf you want to use the project directly then, you may use localhost\r\n- **The proper way** should be, create a mock db so that it does not affect the real DB. But for this time being, I hit the main DB to test.\r\n- 6 unit test cases are written. One of them is written as to be failed purposely. So, 5 test cases pass, 1 get failure.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnazemmahmud%2Fsymfony-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnazemmahmud%2Fsymfony-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnazemmahmud%2Fsymfony-scraping/lists"}