https://github.com/erickrus/js-web-crawler
https://github.com/erickrus/js-web-crawler
Last synced: 7 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/erickrus/js-web-crawler
- Owner: Erickrus
- Created: 2014-09-18T01:54:57.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2014-09-20T21:22:29.000Z (about 11 years ago)
- Last Synced: 2025-01-11T22:29:20.661Z (9 months ago)
- Language: JavaScript
- Size: 328 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
js-web-crawler
==============
Implement a simple web crawler based on node.js.
It is a light-weighted crawler or data extraction tools.
The data will be stored in mysql server or filesystem finally.
3 DEMOs
-------
* Baidu traffic map crawler (binary crawler ,5 min/fetch)
* bdt_main.js - cralwer main program
* baidu-traffic.js - implements the main logic to process the data)
* PNGMerge.exe - A C# program written to support merge of 7x7 map into 1 bigger map)
* Kitco Forex/Metal crawler (encrypted csv crawler, 30 min/fetch)
* Based on the recent decompile result of amstock.swf, csv parser revealed the details of the data source and format
* amstock_main.js - cralwer main program
* amstock.js - implements the main logic to translate the data
* AQICN China AQI city index crawler (json crawler, 30 min/fetch)
* Based on the json service for rectangle area selection on longitude, latitude
* aqicn_main.js - cralwer main program
* aqicn.js - implements the utilities for the crawler