{"id":21035410,"url":"https://github.com/archiveteam/universal-tracker","last_synced_at":"2025-05-15T14:31:15.743Z","repository":{"id":1831383,"uuid":"2755688","full_name":"ArchiveTeam/universal-tracker","owner":"ArchiveTeam","description":"A configurable, reusable tracker with dashboard","archived":false,"fork":false,"pushed_at":"2023-12-15T17:49:43.000Z","size":665,"stargazers_count":36,"open_issues_count":38,"forks_count":17,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-05-07T23:27:35.444Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"emilybache/GildedRose-Refactoring-Kata","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ArchiveTeam.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2011-11-11T12:34:12.000Z","updated_at":"2025-04-26T09:30:04.000Z","dependencies_parsed_at":"2023-01-11T16:06:10.007Z","dependency_job_id":"b13b5a79-5a0a-4b74-ad62-c3ae0ffb5f42","html_url":"https://github.com/ArchiveTeam/universal-tracker","commit_stats":{"total_commits":236,"total_committers":6,"mean_commits":"39.333333333333336","dds":0.2542372881355932,"last_synced_commit":"2b6629033a837a9803501485c1f172bf3bdf1052"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Funiversal-tracker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Funiversal-tracker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Funiversal-tracker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArchiveTeam%2Funiversal-tracker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ArchiveTeam","download_url":"https://codeload.github.com/ArchiveTeam/universal-tracker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254358677,"owners_count":22057958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T13:14:49.158Z","updated_at":"2025-05-15T14:31:15.368Z","avatar_url":"https://github.com/ArchiveTeam.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"A configurable multi-project tracker with dashboards.  [![Build Status](https://secure.travis-ci.org/ArchiveTeam/universal-tracker.png)](http://travis-ci.org/ArchiveTeam/universal-tracker)\n\n\nQuick Start\n===========\n\nTo start running the Sinatra app:\n\n1. Install a recent version of Redis.\n2. Install Ruby 1.9 or 2.0 using `rvm`.\n3. Install project dependencies using `bundle install`.\n4. Copy `config/redis.json.example` and set the Redis parameters.\n5. Launch the app using something like nginx+passenger or `rackup` for development.\n\nYou'll now have an empty application. The admin pages are accessible to anyone without logging in.\n\nCreate a user account and make yourself a global admin:\n\n1. Go to http://yourtracker/global-admin/\n2. Go to Users.\n3. Create a new account with global admin powers.\n\nFrom now on, you'll have to log in to access the admin pages.\n\nIf you ever forget your password, remove the `admins` key from the Redis database to reopen access and create a new user account.\n\nFor detailed step-by-step instructions, see http://archiveteam.org/index.php?title=Dev/Tracker which describes additional steps required for a full install.\n\nTo run tests, use `bundle exec rake`.\n\n\nTerminology\n===========\n\n- `items`: users, members or another type of unit that is to be saved. Each item is identified by a unique string, e.g., the username.\n- `domains`: identify parts of an item (e.g., mobileme is is divided in web, gallery, public.me.com and homepage.mac.com). This is only used for statistics.\n\n\nBasics / Redis structure\n========================\n\n1. The new items are added to the `todo` set.\n2. When a downloader requests an item, the tracker removes a random item from the `todo` set and adds it to the `out` zset and the `claims` hash.\n3. When a downloader completes an item, the tracker removes the item from `out` and `claims` and adds it to the `done` set. The statistics about the item are appended to the `log` list.\n\nThe main Redis structures used by this process:\n\n- `todo`: a set with the unclaimed items.\n- `out`: a zset with the claimed items, with for each item the time when it was claimed.\n- `claims`: a hash with the downloader name and ip for every claimed item.\n- `done`: a set with the items that have been completed.\n- `log`: a list of JSON objects, with details of each completed item.\n\nThe tracker checks more than one `todo` queue. It will return the item from the first queue in this order:\n\n1. It checks the downloader-specific queue `todo:d:#{ downloader }`.\n2. Then it checks the general queue `todo`.\n3. The lower-priority queue `todo:secondary`.\n4. The redo queue `todo:redo` (this isn't actually used; the idea was to assign released claims to another user).\n\nIf no item is found in any of these queues, the tracker returns a 404 Not Found response.\n\nThe tracker maintains several statistics to feed the dashboard:\n\n- `downloader_count`: a hash (downloader -\u003e item count) indicating the number of downloaded items per downloader.\n- `downloader_bytes`: a hash (downloader -\u003e total bytes) indicating the downloaded bytes per downloader.\n- `downloader_version`: a hash (downloader -\u003e script version) with the version number reported by the downloader.\n- `downloader_chartdata:#{ downloader }`: a list of [timestamp,bytes] pairs, logging the growth of `downloader_bytes` over time.\n- `items_done_chartdata`: a list of [timestamp,item count] pairs, logging the progress over time.\n- `domain_bytes`: a hash (domain -\u003e total bytes) with the total number of bytes for each domain.\n\nRate-limiting is implemented as follows:\n\n1. The tracker counts the number of items given out in this minute by incrementing `requests_processed:#{ minute }`.\n2. If `requests_per_minute` is set and `requests_per_minute` \u003c `requests_processed:#{ minute }`, no new items will be given out until the next minute begins.\n\n(This may not be ideal, since it can generate a large burst of activity in the first seconds of a minute and then nothing.)\n\nDownloaders can be blocked by adding the IP address to `blocked`. Blocked downloaders will not receive a username. Downloaders that send invalid requests will be blocked automatically.\n\nMultiple trackers with one instance\n-----------------------------------\n\nOne instance can manage trackers for multiple projects. In the Redis database, the keys of each tracker are prefixed with \"#{ slug }:\".\n\nHeroku\n------\n\nCan run on Heroku, with a Redis server somewhere.\n\nHTTP API\n========\n\nThe clients communicate with the tracker over HTTP. Payloads are in JSON format, so the client should add a \"Content-Type: application/json\" header.\n\n### Requesting a new item\n\n    POST /:slug/request\n    {\"downloader\":\"#{ downloader_name }\"}\n\nThe tracker will respond with the name of the next item:\n\n    #{ item_name }\n\nResponse 404 with an empty body indicates that no item is available. The client should wait for a while and try again. Similarly, response 420 with an empty body indicates that the rate limiting is active. The client should wait for a while and try again.\n\n### Completing an item\n\n    POST /:slug/done\n    {\"downloader\":\"#{ downloader_name }\",\"item\":\"#{ item_name }\",\"bytes\":{\"#{ domain_a }\":#{ bytes_a },\"#{ domain_b }\":#{ bytes_b }},\"version\":#{ script_version }\",\"id\":\"#{ checksum }\"}\n\nThe `bytes` field contains an entry with the number of bytes for each domain of the item. The `id` field can contain an arbitrary checksum; this value is stored in the log and can be used to check results.\n\nThe tracker responds with HTTP status 200 and a body of two characters:\n\n    OK\n\nIf the tracker does not respond with 200 / OK, the client should wait for a little while and send the request again.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchiveteam%2Funiversal-tracker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farchiveteam%2Funiversal-tracker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchiveteam%2Funiversal-tracker/lists"}