{"id":20275434,"url":"https://github.com/tapiocaaberta/datasus_node_scrap","last_synced_at":"2025-04-11T05:24:06.004Z","repository":{"id":19988198,"uuid":"23255419","full_name":"TapiocaAberta/datasus_node_scrap","owner":"TapiocaAberta","description":"Nodejs web crawler to get informations from http://cnes.datasus.gov.br/Lista_Tot_Es_Estado.asp","archived":false,"fork":false,"pushed_at":"2015-08-03T12:49:01.000Z","size":63607,"stargazers_count":7,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T03:27:30.097Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TapiocaAberta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-23T12:35:53.000Z","updated_at":"2022-05-18T19:00:36.000Z","dependencies_parsed_at":"2022-08-28T08:40:36.403Z","dependency_job_id":null,"html_url":"https://github.com/TapiocaAberta/datasus_node_scrap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TapiocaAberta%2Fdatasus_node_scrap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TapiocaAberta%2Fdatasus_node_scrap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TapiocaAberta%2Fdatasus_node_scrap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TapiocaAberta%2Fdatasus_node_scrap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TapiocaAberta","download_url":"https://codeload.github.com/TapiocaAberta/datasus_node_scrap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248346568,"owners_count":21088478,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T13:09:28.235Z","updated_at":"2025-04-11T05:24:05.983Z","avatar_url":"https://github.com/TapiocaAberta.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"CNES web scraper\n================\n\n[![Build Status](https://travis-ci.org/transparenciasjc/datasus_node_scrap.svg)](https://travis-ci.org/transparenciasjc/datasus_node_scrap)\n[![Coverage Status](https://coveralls.io/repos/transparenciasjc/datasus_node_scrap/badge.svg)](https://coveralls.io/r/transparenciasjc/datasus_node_scrap)\n[![bitHound Score](https://www.bithound.io/github/transparenciasjc/datasus_node_scrap/badges/score.svg?)](https://www.bithound.io/github/transparenciasjc/datasus_node_scrap/master)\n[![Dependency Status](https://david-dm.org/transparenciasjc/datasus_node_scrap.svg \"Dependencies Checked \u0026 Updated Regularly (Security is Important!)\")](https://david-dm.org/transparenciasjc/datasus_node_scrap)\n\n\n## Table of Contents\n\u003c!-- toc --\u003e\n* [How to debug:](#how-to-debug)\n* [How to run:](#how-to-run)\n* [Limitations:](#limitations)\n* [How to export](#how-to-export)\n  * [To CSV:](#to-csv)\n  * [To Database dump:](#to-database-dump)\n\n\u003c!-- toc stop --\u003e\n\u003c!--\n\tps: table of contents generated by [readme-toc](https://www.npmjs.com/package/readme-toc) plugin.\n--\u003e\n\nYou have to use nodejs version 0.10.x\n\nDownload the dependencies:\n\n\tnpm install\n\nInstall MongoDB:\n\n\tsudo apt-get install mongodb\n\n# How to debug:\n\n\tnode-inspector\n\tnode --debug-brk crawler.js\n\nThen, go to http://localhost:8080/debug?port=5858\n\n# How to run:\n\n\tnode --max-old-space-size=8192 --expose-gc crawler.js\n\n# Limitations:\n\nThe script consumes a lot of memory in order of their execution time. On the first minute of their execution, about 150 registers are downloaded, netherless, this measure is going down in order of the memory consumption.\n\nA real fixes to this problem is to study how the V8 garbage collector works, and pay attention to remove the closure variables to improve less memory consumption.\n\nSo, an work around to this problem is to kill and reopen the script in determined cycle of time using CRON. To do that, run the following instructions:\n\ncreate a file with the following content on `/etc/cron.d/crawler` (without the extension '.sh')\n\n\t#!/bin/sh\n\tpkill node\n\tcd \"\u003cPATH_OF_SOURCE\u003e/node_scrap/\"\n\t\u003cYOUR_NODE_PATH\u003e/node --max-old-space-size=8192 index.js \u003e /tmp/crawler.log \u0026\n\nrun:\n\n\tcrontab -e\n\nadd this on the last line of the file:\n\n\t*/2 * * * * /bin/sh /etc/cron.d/crawler\n\nThis will run automatically the script `/etc/cron.d/crawler` on the interval of 2 in 2 minute, It would be kill and re-execute the crawler script.\n\n# Running the crawler\n## First Step\n\nFirst of all you need to change the function \"initialize\" of the class index.js, which the content is something like that:\n\n```js\ndownloadModule.processStates();\n//downloadModule.processEntities();\n```\n\nThe first step is to execute the function `processStates()` this function will download all the urls of the entities, in order to make the process synchronous, and it will maintain the control of what register was downloaded.\n\nMore of 300.000 url's will be downloaded. You can check It on database:\n\n```shell\n mongo\n use cnes2015\n show collections\n db.entitytodownloads.count();\n```\n\n## Second step\n\nYou must backup the collection `entityurls` to `entityurls_bak` by using the following command:\n\n`db.entityurls.copyTo('entityurls_bak')`\n\nThen, you have to change the function `initialize` in order to make it call the function that download the entity details:\n\n```js\n//downloadModule.processStates();\ndownloadModule.processEntities();\n\n```\n\nYou can now check your log with tail: `tail -f /tmp/crawler.log`\n\n# How to export\n\n## To a Unique CSV:\n\n\tcd output\n\tmongoexport --db cnes --collection entities --csv --fieldFile entities_fields.txt --out entities.csv\n\n## To a separated CSV file with the UF as a file name (like SP, RJ, MG etc)\n\n\trm -rf output/*.csv \u0026\u0026 node crawler/exporter.js\n\n## To Database dump:\n\nto generate the dump:\n\n\tmongodump -d cnes -o output\n\nto restore:\n\n\tmongorestore cnes","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftapiocaaberta%2Fdatasus_node_scrap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftapiocaaberta%2Fdatasus_node_scrap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftapiocaaberta%2Fdatasus_node_scrap/lists"}