https://github.com/suddi/xe-crawler

Collect currency exchange rates from xe.com
https://github.com/suddi/xe-crawler

beanstalkd nodejs redis xe

Last synced: 3 months ago
JSON representation

Collect currency exchange rates from xe.com

• Host: GitHub
• URL: https://github.com/suddi/xe-crawler
• Owner: suddi
• License: mit
• Created: 2015-03-04T16:54:35.000Z (about 10 years ago)
• Default Branch: master
• Last Pushed: 2019-03-24T22:12:50.000Z (about 6 years ago)
• Last Synced: 2025-01-11T19:12:36.169Z (4 months ago)
• Topics: beanstalkd, nodejs, redis, xe
• Language: JavaScript
• Homepage:
• Size: 51.8 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# XE Crawler #

[![Codacy Badge](https://api.codacy.com/project/badge/Grade/46bec21af4304765a45656bcb51a3a00)](https://www.codacy.com/app/Suddi/xe-crawler?utm_source=github.com&utm_medium=referral&utm_content=suddi/xe-crawler&utm_campaign=badger)

[![license](https://img.shields.io/github/license/suddi/xe-crawler.svg?maxAge=2592000)](https://github.com/suddi/xe-crawler/blob/master/LICENSE)

This program was developed on and for `Node.js 0.10.25` on `Ubuntu 14.04 LTS 64 bit`.

The program stores key-value pairs in `Redis`, for this you will need to install `redis-server`.

---

### Installation ###

---

To install the necessary `node_modules`, type in from the project directory:

    $ npm install

---

### Settings ###

---

We may change the settings for the program by amending `settings.js`. The following settings are provided:

* `fivebeans_settings` - settings pertaining to the creation of a beanstalkd client using the fivebeans library.

* `mongo_settings` - settings pertaining to the creation of a MongoDB connection using the mongoose library.

* `queue_settings` - settings for jobs to be created to the tube in beanstalkd.

* `payload_settings` - a list of payloads to be run, more payloads can be added by appending objects to the list.

* `general_settings` - general settings such as success_limit and fail_limit which provide a threshold to end the program.

---

### Seed a Job ###

---

After setting the `payload_settings` in `settings.js`, we may seed a job by:

    $ node seeder.js

---

### Consuming Jobs ###

---

Once a job has been seeded, we may run the worker with the following:

    $ node worker.js

---

### Destroying Jobs ###

---

Currently, if the beanstalkd client reports an error, the program will shut down.

If this happens, some jobs may be left behind in the tube, to remove these we can run:

	$ node destroy.js

---

### Remove Program Relevant Key-Value Pairs ###

---

In the scenario that a program is stopped mid-run, it may be necessary to remove program relevant key-value pairs from redis.

To do this:

	$ node clear_redis.js