https://github.com/luciopaiva/dicio-crawler

Node.js crawler for dicio.com.br.
https://github.com/luciopaiva/dicio-crawler

crawler nodejs scraper

Last synced: 24 days ago
JSON representation

Node.js crawler for dicio.com.br.

• Host: GitHub
• URL: https://github.com/luciopaiva/dicio-crawler
• Owner: luciopaiva
• License: other
• Created: 2018-10-13T22:29:42.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2022-12-08T02:42:09.000Z (almost 2 years ago)
• Last Synced: 2024-04-14T12:59:41.724Z (7 months ago)
• Topics: crawler, nodejs, scraper
• Language: JavaScript
• Homepage:
• Size: 86.9 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        

# dicio.com.br crawler

An experimental crawler for dicio.com.br. Makes throttled, concurrent requests and saves results to local sqlite database.

Hint: if you just want to grab the dictionary for offline use, simply download Dicio's [mobile app](https://play.google.com/store/apps/details?id=com.setegraus.dicio).

## License

You should use this only for educational purposes. Please see `LICENSE.md`. I am not involved with `dicio.com.br` in any way, so run it at your own risk.

## How to install and run the crawler

    nvm install

    npm install

## How the crawler works

Starting with some seed words (see `crawler.js::getOpenUrls()`), it crawls `dicio.com.br`, fetching each word's page, which in turn contains links to other words.

For each word, the crawler persists that word's definition and uses linked words found in the page to continue crawling the website (I call it "open URLs").

Whenever the crawler stops (be it because it was commanded to, via `crawler.js::NUMBER_OF_REQUESTS_TO_MAKE`, be it because the user hit Ctrl+C to stop the application), it persists the current list of open URLs to the database before terminating.

The list of open URLs has its size constrained by two limits: one in-memory and the other on the database. This is to prevent memory from being totally depleted and also to avoid increasingly big delays when saving data to the database. There's no need to keep huge amounts of open URLs anyway.

## Future improvements

Later I found out that dicio's mobile app uses a JSON API to fetch word definitions:

    https://www.dicio.com.br/api/indexv2.php?p=cadeira

And another one for synonyms:

    https://www.sinonimos.com.br/api/?method=getSinonimos&palavra=cadeira

An improvement would be to start using them instead of parsing HTML. One drawback is that we loose links to other words. Synonyms could be used, but we'd end up trapped in a subset of known words (e.g.: starting with "cadeira", you would probably never find "mesa" through its synonyms and their synonyms, and so on and so on.