{"id":21492349,"url":"https://github.com/muhamed-didovic/scrapping-node","last_synced_at":"2026-04-10T07:37:15.819Z","repository":{"id":84129450,"uuid":"136208586","full_name":"muhamed-didovic/scrapping-node","owner":"muhamed-didovic","description":"scrapping with node","archived":false,"fork":false,"pushed_at":"2018-06-06T16:53:08.000Z","size":502,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-17T11:27:55.749Z","etag":null,"topics":["express","mocha","mongodb","nodejs","pm2","selenium"],"latest_commit_sha":null,"homepage":null,"language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muhamed-didovic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-05T16:57:55.000Z","updated_at":"2018-06-06T16:53:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"52356e34-612d-417a-b656-e457e263ea66","html_url":"https://github.com/muhamed-didovic/scrapping-node","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/muhamed-didovic/scrapping-node","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhamed-didovic%2Fscrapping-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhamed-didovic%2Fscrapping-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhamed-didovic%2Fscrapping-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhamed-didovic%2Fscrapping-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muhamed-didovic","download_url":"https://codeload.github.com/muhamed-didovic/scrapping-node/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhamed-didovic%2Fscrapping-node/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279017675,"owners_count":26086125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["express","mocha","mongodb","nodejs","pm2","selenium"],"created_at":"2024-11-23T15:28:28.058Z","updated_at":"2025-10-14T02:18:16.619Z","avatar_url":"https://github.com/muhamed-didovic.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Water scraping\n\n## Installation\n\n### Requirements\nIn order to run locally you need to install and have running MongoDB and Node.js as services on your machine.\nDetails how to connect to MongoDB you can find in config.js file which is in root directory of the Micro-service.\n\nOther important packages(best installed globally) that you need are gulp, mocha, nodemon\n\n### Gitlab\nAfter you clone or checked out from Gitlab repository, go to the project's directory and run commands\n```sh\n$ npm install (this will install all dependencies needed)\n$ npm start   (this will start the server)\nor\n$ nodemon\nor\n$ gulp\n```\n### Run tests\n   ```sh\n$ npm test\n```\n\n## PM2 package\nInfo: https://github.com/Unitech/pm2, http://pm2.keymetrics.io/\n\nEnsure that package pm2 is installed on your environment.\n\nImportant file is ecosystem.json, which holds configuration for app:\n1. watcher for files, whenever there is change within a file restart server\n2. which files not to watch\n3. make clustering and load balancer\n4. set the environments\n\nIn order to check state of the app or start the app use this steps:\n   ```sh\n$ cd /your-folder/\n$ pm2 list (check if there is anything running, if not go to 3. step)\n$ pm2 start ecosystem.json (start server and cronjob)\n```\nIf you change the ecosystem.json file do the following\n   ```sh\n$ cd /your-folder/\n$ pm2 delete ecosystem.json\n$ pm2 start ecosystem.json\n```\n\n## MongoDB\n\nDevices collection and indexing:\n\n```\nconst DeviceSchema = new Schema({\n  name: String,\n  data: Schema.Types.Mixed,\n  date: {type: Date, default: Date.now, get: transformDate}\n});\n```\n\n## API\n\n### GET /\nHome route\n\n### GET /reports\nRoute to get form for reports\n\n### POST /reports\nRoute to post data to backend and get in response Excel file\n\n### GET /scrape\nRoute to start scraping data from website\n\n### GET /devices\nRoute to get all data from database and show them in table to the user\n\n### GET /devices/:id\nRoute to get data of device\n\n### PUT /devices/:id\nRoute to edit data of device\n\n### DELETE /devices/:id\nRoute to delete device\n\n### GET /healthcheck\nInvoked when health check is needed\nRoute with information about the health of MongoDB and Scrape links,\ntest if we can connect to those services\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhamed-didovic%2Fscrapping-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuhamed-didovic%2Fscrapping-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhamed-didovic%2Fscrapping-node/lists"}