https://github.com/qduc/hn-downloader
NodeJS CLI app to download HackerNews database
https://github.com/qduc/hn-downloader
nodejs
Last synced: about 1 month ago
JSON representation
NodeJS CLI app to download HackerNews database
- Host: GitHub
- URL: https://github.com/qduc/hn-downloader
- Owner: qduc
- License: mit
- Created: 2017-07-31T17:31:42.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2017-08-07T15:58:09.000Z (almost 9 years ago)
- Last Synced: 2025-07-27T16:51:21.759Z (11 months ago)
- Topics: nodejs
- Language: JavaScript
- Homepage:
- Size: 7.81 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Hacker News downloader
Simple CLI app to download [Hacker News](https://news.ycombinator.com/) database using API from
## Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
### Prerequisites
- Node.js
- A MySQL server
### Configuration
Install the dependencies
```
npm install
```
Prepare database schema: Import `database.sql` file that came with this repo
_( Note: you must create an empty database first then `USE` it )_
Change `database.js` according to your database credential
Update `config.js` to suit your preference _(optional)_
```
concurrentFetch : 1000
maxRetry : 3
```
- `concurrentFetch` number of items to download before insert into database
- `maxretry` if a network error occured, how many times it will attempt to retry before giving up
You are all set!
### Usage
```
node index.js [largestId]
```
- `largestId` _(optional)_ largest ID the script will download then stop, in case you just want to download a part of it.
Leave it blank to download the whole database (takes several hours!)
**Warning:** [Aug/2017] Whole database is nearly 7GB. Make sure you have enough room for it.
## License
This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details