{"id":16427897,"url":"https://github.com/zemke/relrank","last_synced_at":"2026-06-10T01:31:22.938Z","repository":{"id":63626485,"uuid":"563931716","full_name":"Zemke/relrank","owner":"Zemke","description":"Predicting players’ strengths from unevenly distributed games.","archived":false,"fork":false,"pushed_at":"2023-01-21T08:24:13.000Z","size":1086,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-25T05:13:48.761Z","etag":null,"topics":["ranking","ranking-algorithm","rating-system"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Zemke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-09T16:28:40.000Z","updated_at":"2022-12-03T21:40:09.000Z","dependencies_parsed_at":"2023-01-23T19:15:25.812Z","dependency_job_id":null,"html_url":"https://github.com/Zemke/relrank","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Zemke/relrank","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zemke%2Frelrank","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zemke%2Frelrank/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zemke%2Frelrank/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zemke%2Frelrank/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Zemke","download_url":"https://codeload.github.com/Zemke/relrank/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zemke%2Frelrank/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34133404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ranking","ranking-algorithm","rating-system"],"created_at":"2024-10-11T08:14:26.228Z","updated_at":"2026-06-10T01:31:22.918Z","avatar_url":"https://github.com/Zemke.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Relative Ranking System\n\nRanking system for ladder rankings where games are won in best out of X rounds.\n\n## Usage\n\nThe input is supplied to stdin in a comma-separated format:\n\n`user_a_id,user_b_id,user_a_rounds,user_b_rounds`\n\nYou can provide one game per line or have rounds between two users already\naccumulated.\n\n**It is important to end with a new line, otherwise the last line is ignored**\n\n```console\n$ printf \"1,2,3,0\\n2,1,3,2\\n3,1,2,1\\n\" | go run .\n3,38.7940217999804307425334034878722275274658\n1,94.4733759805052229464094241552471635973446\n2,54.2679612557235192475522802224982500417932\n```\n\nIt returns the user ID and its calculated rating as in `user,rating` per line.\n\nThe ranking system relies on rounds won and doesn’t care about games.\nYou can have even drawn games. \\\n\n### Config\n\nThere are multiple environment variables to tweak the execution:\n\n`DEBUG=1` logs debug information to stderr. \\\nSet `RELRANK_RELREL`, `RELRANK_RELSTEPS` to change `relRel`, `relSteps`\nsettings. \\\n`RELRANK_PREC` to change decimal precision. \\\n`RELRANK_SCALE_MAX` to rescale the ratings by setting the maximum rating.\n\nThe detailed configuration of each of these settings can be understood by\nreading on.\n\n## Implementation\n\nAn original implementation has been as part of\n[Worms League](https://github.com/Zemke/worms-league).\nIt was battle-tested and performed convincingly.\n\nNow there is a self-contained CLI application written in Go.\nThis application should allow for usage outside of Worms League or any\nparticular league so that it can supplied only with the games that were played\nand then output the ranking.\n\nGo is also faster than PHP and a better suit for a standalone CLI application.\n\n## Problem\n\nIn the public world one is accustomed to absolute ranking system.\nThese are self-contained perfect.\nEveryone plays everyone the same amount of times.\nTherefore there’s no need in complexity to assess a points pattern other than\naddition.\nThe fixtures are inherently perfect.\n\nThis is not typically the case in online games where there’s a fixed schedule\nand not everyone is necessarily playing everyone the same amount of times.\nSo players may never clash, others multiple times.\nAn absolute ranking would make players end up top who win most of their games.\nIn other words, possibly the most active or the one playing the weakest players.\n\n## Solution\n\nTherefore the series of a user has to be evaluated within a context that’s\n**relative** to all the other contenders in.\n\nThe most blatant example is a victory against a higher ranked other player\nshould get one more points than against a lower ranked player.\n\n## Increasingly Relative\n\nIn a relative ranking system the points of a user are determined by the points\nof the opponents one has played against.\nThe question is how to you initially determine the points of another user when\nthey are not yet ranked.\n\n### Starting Absolute\n\nThe most fundamental information that an absolute ranking could be generated\nfrom is the total won rounds.\nOne gets one point per won round.\n\nThis is now an absolute state.\nThe idea of the relative ranking system is to put this absolute rating of each\nplayers into perspective — into context.\n\n### Relativizers\n\nLooking at the number of won rounds of each player under which circumstances\nwere they actually attained? \\\nOne is more likely to add won rounds based on these factors:\n\n* level of opponents (`quality`)\n* number of rounds (`farming`)\n* total number of rounds (`effort`)\n\nThese three factors are relative among themselves again.\nFor instance the more number of rounds against the same opponent of a weaker\nskill level, the more likely one is to gain won rounds and vice versa. \\\nIf one is generally playing more rounds, it’s more likely to accumulate won\nrounds, too.\n\nThese are the three main factors used in the relative ranking system.\nThese can be referenced to put the absolute value of won rounds into a context.\nTherefore they relativize an absolute number and are thus called “relativizers.”\n\n#### `quality`\n\nThe `quality` relativizer relativizes won rounds in the context of how well the\nopponents of that user are rated.\nThis is perhaps the most obvious relativizer valuing won rounds according to the\nrating of the respective opponent.\n\n$o$ = number of opponents \\\n$n$ = distinct ratings of all users \\\n$n_i$ = position of opponent in $n$ \\\n$w_i$ = won rounds against opponent \\\n$t$ = total won rounds\n\n$$ \\sum_{i=1}^{o}{((n_i+1)/n)^3 * w_i/t} $$\n\nThis is averaged so that number of rounds per opponent are accounted for\nproportionally.\n\nHere is the opponent’s position to relativization plot for a total of $n=16$\ndistinct ratings.\n\n![Quality relativizer](images/quality.png)\n\nThe highest rated player — being at position 16 in $n$ — is worth a 100\npercent so that the result for $max(n)$ is always $1$.\nWon rounds are exponentially less relativized the higher the opponent is rated.\n\nExponential makes sense here because ratings for players at the top tend to be \nmore robust.\nFor example the difference in skill of a player is likely to be greater between\nfirst and third place than it is for 20th and 23rd.\n\nOne fact special to this relativizer is that its input parameters possibly\nchange with each step.\nSince an input parameter is the opponent’s rating and it may change with each\nstep.\n\n#### `farming`\n\nThe `farming` relativizer relativizes won rounds in the context of how often\nthat user played against the same opponent.\nEach won round weighing less than the previous.\nThis retaliates the infamous “noob bashing” where a better\nplayer repeatedly defeats a very low rated player to farm points from.\n\nAdditionally this promotes users to have diverse pairings.\nCertainly a user’s rating is easier to assess the more data is available. \\\nTheoretically you can gain more points from defeating a low rated player for the\nfirst time than gaining a hundredth won round against a high rated player.\n\nThe relativizer is made so that per opponent the first round is always worth $1$\nas in 100 percent.\nThen the value of each next won round decreases logarithmically.\nThe logarithm is negative natural.\nIt decreases until $0.01$ which is the least value a round can have as per the\n`farming` relativizer.\nIt is given to the round of the maximum won rounds any player has against another.\nThis value is $a$.\n\nThis is the important concept here.\nScaling the value of won rounds from $1$ to $0.01$ negatively logarithmically.\n\nStarting with the fundamental logarithmic function:\n\n$$ y=-ln(x)+1 $$\n\n![Farming fundamental logarithm](images/farming_fundamental.png)\n\nAdding $1$ controls where $y$ intercepts at $x=1$.\nLet’s make it variable to make it clearer: $y=-ln(x)+k$\nIt is desired to make it so that when $x=1$ and $k=1$ then $y$ should resolve\nto $1$.\nThis is because $x$ is the round and the first round should always be valued at\na 100 percent, $1$ respectively.\n\nNow the curve needs to be adjusted so that $y=0.01$ when $x=a$.\nSince $-ln(x)+1 = -z * ln(x)+1$ where $a=1$, it can resolve for $z$ for when\n$y=0.01$.\n\n$$ 0.01=-z * ln(x)+1 $$\n\n$$ z={99 \\over 100ln(a)} $$\n\nThe formula ends up like this:\n\n$$ y = -{99 \\over {100 ln(a)}}ln(x)+1 $$\n\nThe boundaries are for the first round\n\n$$ a=x \\to y=0.01 $$\n\nand maximum round respectively\n\n$$ x=1 \\to y=1 $$\n\nwith everything in between decreasing towars $0.01$ logarithmically.\n\n![Farming relativizer](images/farming.png)\n\nThis formula is applied to each round per opponent so that $x$ is a set where\neach number represents the $n$-th won round.\n\n$$ X = \\lbrace 1, 2, 3, 4, … \\rbrace $$\n\nThis is then averaged across the total number of won rounds.\nThe final formula to output the factor of relativization:\n\n$$ {\\sum_{x=1}{-{99 \\over {100 ln(a)}}ln(x)+1}} \\over z $$\n\n#### `effort`\n\nThe `quality` relativizer relativizes won rounds in the context of how many\nrounds in total were played to attain the portion of won rounds.\n\nThe total rounds played of each user $x$ are rescaled so that\n\n$$ max(x) \\to 1 $$\n\n$$ min(x) \\to 0.01 $$\n\n$$ y = a + \\frac{(x - {min}(x))(b-a)}{\\max(x)-{min}(x)} $$\n\n![Effort relativizer](images/effort.png)\n\n### Relativization Steps\n\nEach calculation of a relative ranking starts with an absolute ranking.\nThey’re put into relation using relativizers.\n\nSince two perspectives — an absolute and a relative — are put against each\nother, it is necessary to balance them out.\n\nThe more relativity is “applied” to the absolute information, the less the\noriginal absolute value has any weight in the final outcome.\nPotentially leading to an entirely unfounded result as the relativization\nprocess to relativize itself more and more.\n\nThe relative ranking system applies relativization steps. With each new\nstep the previous value is relativized further. Each output is the input\nof the next step. As aforementioned the initial input is the absolute value of\nwon rounds.\n\nEach step is the same relativization algorithm applied to the input.\n\n#### Configuration\n\nTo further fine-tune the balancing of relativization relative ranking system\nuses two configuration parameters.\n\n##### `relSteps`\n\n`relSteps` is used in a formula to determine the total number of relativization\nsteps. \\\nConsider $a$ to be `relSteps` and $x$ is the maximum total number of\nsteps played of any user.\n\n$$ S = \\max(\\min(\\left\\lfloor -1+\\log(x*.13)*a \\right\\rceil,21),1) $$\n\n`S` is solved for the number of relativization steps to take.\n\nAssuming `relSteps` is $36$, it would generate the following relativization\nsteps per maximum rounds played of any user.\n\n![Relativization steps per max rounds played of any user](images/relsteps.png)\n\nThe user with the maximum number of rounds played on the y-axis and what number\nof relativization steps it would cause on the y-axis.\n\nThe graph shows how the number of steps rises logarithmically.\nThe greater `relSteps` the steeper the rise.\nHence a greater `relSteps` factor causes the relativization to be more effect\nthus weighing in more in the balancing ob the absolute number versus the\nrelativized result.\n\nThere has got to be at least one relativization step and $21$ at most.\nThe result is rounded down.\n\n$20$ for the $relSteps$ parameter and maxing out at $21$ have proven to deliver the\nbest results.\nMore on that later.\n\n##### `relRel`\n\n`relRel` is used in a formula to influence the impact of each individual\nrelativization step.\n\nA relativizer returns a decimal that the number to be relativized is multiplied\nwith.\nThe number to be relativized is always the current rating of that user.\nIn the first relativization step that is the absolute number of won rounds of\nthat user in all subsequent steps it is the rating of that user.\n\nIn other words: Prior to any relativization, the rating of a user is simply\nthat user’s won rounds, in all subsequent steps it is the relativization\nresult of the previous step.\nIn the first step the won rounds are relativized and that result is to be\nrelativized in the next step and so on and so forth.\n\nIn that example the three won rounds of a user are passed through four\nrelativization steps determining the final rating of the user.\n\n$$ 3 * 0.83 * 1.2 * 0.5 * 3.2 = 1.7808 $$\n\nHere the decimals $0.83$, $1.2$, $0.5$ and $3.2$ represent a step of\nrelativization each. The greater the difference to $1$ the greater the impact\nof relativization as $1$ would not impact the result at all.\n\nHere `relRel` configuration comes into play.\nIt is added to the relativization and then the average of these result in the\nfinal relativization.\n\nIn practice the average of three relativizers (as mentioned earlier) and\n`relRel` form the final decimal that the rating of the current step is \nrelativized by.\n\nTherefore within reach step the rating of the previous round is multiplied by\nthe average of all the relativizers including `relRel` to form the new rating\nwhich is the final rating depending on whether all steps have completed.\n\n$$ R_i = R_{i-1} * {\\sum_{k=1}^{3}{rel_k} + a \\over 4} $$\n\n## Output Scaling\n\nThe algorithm produces rather insane numbers. For example a user with $163$\nrounds played total of which are $127$ won could potentially end up with a\nrating of\n\n$$ 18790682468510068.970103417500681036366 $$\n\nand even more decimal places depending on the scale that is set for decimals.\n\nTo make the numbers more pleasing for displaying to the end-user in a ladder\nthe numbers can be rescaled using min-max normalization.\n\nConsider $a$ to be the number of points the lowest rated player should have\nand $b$ the number of points the highest rated player should have.\n\n$$ a + \\frac{(x - {min}(x))(b-a)}{\\max{x}-{min}(x)} $$\n\nTo make numbers increase as more rounds are played one can make $b$ the\nmaximum number of rounds of any player. This adds to points being more\nrelatable over time.\n\n**In the relative ranking system everyone’s points may change with just one new\nround played.**\n\nThe maximum rating of a use according to which to scale all other users’\npoints can be set with the `RELRANK_SCALE_MAX` environment variable.\n\n## Validation Testing\n\nThe algorithm is tested against another ranking system that is regarded as the\nbest in Worms Armageddon online play.\n\nThe current specifics is that the root mean-squared error is\n$23.54168716924731$ and the mean absolute error is $16.515357399195885$.\n\nMany of the magic numbers in the formulas provided in this documentation\noriginate from experimentation to drive down the root mean-squared error toward\nthat target system.\n\n## Mean Absolute Error\n\nThis metric is interesting and human-readable.\nThis is the average difference of a user’s rating compared to the target system.\n\n$$ 16.515357399195885 = \\sum_{i=1}^{D}|x_i-y_i| $$\n\n## Root Mean-Squared Error\n\nThis metric isn’t really human-readable because like mean absolute error but\nit’s defining metric to measure how well the algorithm performs. \\\nThe edge it has over MAE that due to it weighs big differences heavier than\nsmall ones.\n\nThere is testing in place making sure RMSE doesn’t increase.\n\n$$ 23.54168716924731 = \\sqrt{(\\frac{1}{n})\\sum_{i=1}^{n}(y_{i} - x_{i})^{2}} $$\n\n### Run Validation Test\n\nTests are run by the `validate.py` script.\n\nHere is is run with ranking of users with at least one won round (`MINW=1`) for\ntarget seasons 39, 40, 41 and 42.\n\nNote that the number of games played per season is very different.\nSeasons with many games played generally have greater ratings and lead to\ngreater MAE and RMSE.\n\n```console\n$ DEBUG=0 PERM=0 MINW=1 python3 validate.py\nminw 1\nenv DEBUG=0 RELRANK_ROUND=2\nseason 39\n39 MAE 25.55278493305227\n39 RMSE 36.456688788966446\nseason 40\n40 MAE 14.376024564718678\n40 RMSE 20.091997886647718\nseason 41\n41 MAE 0.7738193555490799\n41 RMSE 0.8859498890892478\nseason 42\n42 MAE 25.358800743463515\n42 RMSE 36.73211211228584\naverage\nMAE 16.515357399195885\nRMSE 23.54168716924731\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzemke%2Frelrank","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzemke%2Frelrank","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzemke%2Frelrank/lists"}