Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ArchiveTeam/universal-tracker

A configurable, reusable tracker with dashboard
https://github.com/ArchiveTeam/universal-tracker

Last synced: 3 months ago
JSON representation

A configurable, reusable tracker with dashboard

Awesome Lists containing this project

README 

        
A configurable multi-project tracker with dashboards.  [![Build Status](https://secure.travis-ci.org/ArchiveTeam/universal-tracker.png)](http://travis-ci.org/ArchiveTeam/universal-tracker)



Quick Start

===========



To start running the Sinatra app:



1. Install a recent version of Redis.

2. Install Ruby 1.9 or 2.0 using `rvm`.

3. Install project dependencies using `bundle install`.

4. Copy `config/redis.json.example` and set the Redis parameters.

5. Launch the app using something like nginx+passenger or `rackup` for development.



You'll now have an empty application. The admin pages are accessible to anyone without logging in.



Create a user account and make yourself a global admin:



1. Go to http://yourtracker/global-admin/

2. Go to Users.

3. Create a new account with global admin powers.



From now on, you'll have to log in to access the admin pages.



If you ever forget your password, remove the `admins` key from the Redis database to reopen access and create a new user account.



For detailed step-by-step instructions, see http://archiveteam.org/index.php?title=Dev/Tracker which describes additional steps required for a full install.



To run tests, use `bundle exec rake`.



Terminology

===========



- `items`: users, members or another type of unit that is to be saved. Each item is identified by a unique string, e.g., the username.

- `domains`: identify parts of an item (e.g., mobileme is is divided in web, gallery, public.me.com and homepage.mac.com). This is only used for statistics.



Basics / Redis structure

========================



1. The new items are added to the `todo` set.

2. When a downloader requests an item, the tracker removes a random item from the `todo` set and adds it to the `out` zset and the `claims` hash.

3. When a downloader completes an item, the tracker removes the item from `out` and `claims` and adds it to the `done` set. The statistics about the item are appended to the `log` list.



The main Redis structures used by this process:



- `todo`: a set with the unclaimed items.

- `out`: a zset with the claimed items, with for each item the time when it was claimed.

- `claims`: a hash with the downloader name and ip for every claimed item.

- `done`: a set with the items that have been completed.

- `log`: a list of JSON objects, with details of each completed item.



The tracker checks more than one `todo` queue. It will return the item from the first queue in this order:



1. It checks the downloader-specific queue `todo:d:#{ downloader }`.

2. Then it checks the general queue `todo`.

3. The lower-priority queue `todo:secondary`.

4. The redo queue `todo:redo` (this isn't actually used; the idea was to assign released claims to another user).



If no item is found in any of these queues, the tracker returns a 404 Not Found response.



The tracker maintains several statistics to feed the dashboard:



- `downloader_count`: a hash (downloader -> item count) indicating the number of downloaded items per downloader.

- `downloader_bytes`: a hash (downloader -> total bytes) indicating the downloaded bytes per downloader.

- `downloader_version`: a hash (downloader -> script version) with the version number reported by the downloader.

- `downloader_chartdata:#{ downloader }`: a list of [timestamp,bytes] pairs, logging the growth of `downloader_bytes` over time.

- `items_done_chartdata`: a list of [timestamp,item count] pairs, logging the progress over time.

- `domain_bytes`: a hash (domain -> total bytes) with the total number of bytes for each domain.



Rate-limiting is implemented as follows:



1. The tracker counts the number of items given out in this minute by incrementing `requests_processed:#{ minute }`.

2. If `requests_per_minute` is set and `requests_per_minute` < `requests_processed:#{ minute }`, no new items will be given out until the next minute begins.



(This may not be ideal, since it can generate a large burst of activity in the first seconds of a minute and then nothing.)



Downloaders can be blocked by adding the IP address to `blocked`. Blocked downloaders will not receive a username. Downloaders that send invalid requests will be blocked automatically.



Multiple trackers with one instance

-----------------------------------



One instance can manage trackers for multiple projects. In the Redis database, the keys of each tracker are prefixed with "#{ slug }:".



Heroku

------



Can run on Heroku, with a Redis server somewhere.



HTTP API

========



The clients communicate with the tracker over HTTP. Payloads are in JSON format, so the client should add a "Content-Type: application/json" header.



### Requesting a new item



    POST /:slug/request

    {"downloader":"#{ downloader_name }"}



The tracker will respond with the name of the next item:



    #{ item_name }



Response 404 with an empty body indicates that no item is available. The client should wait for a while and try again. Similarly, response 420 with an empty body indicates that the rate limiting is active. The client should wait for a while and try again.



### Completing an item



    POST /:slug/done

    {"downloader":"#{ downloader_name }","item":"#{ item_name }","bytes":{"#{ domain_a }":#{ bytes_a },"#{ domain_b }":#{ bytes_b }},"version":#{ script_version }","id":"#{ checksum }"}



The `bytes` field contains an entry with the number of bytes for each domain of the item. The `id` field can contain an arbitrary checksum; this value is stored in the log and can be used to check results.



The tracker responds with HTTP status 200 and a body of two characters:



    OK



If the tracker does not respond with 200 / OK, the client should wait for a little while and send the request again.