Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/nadiamoe/refractor

Refractor is linux mirror load-balancer, which parallelizes requests between an extremely dynamic pool of mirrors.
https://github.com/nadiamoe/refractor

Last synced: 24 days ago
JSON representation

Refractor is linux mirror load-balancer, which parallelizes requests between an extremely dynamic pool of mirrors.

Awesome Lists containing this project

README 

        
# 🪞 Refractor



Refractor is linux mirror load-balancer, which parallelizes requests between an extremely dynamic pool of mirrors. Mirrors in the pool are constantly monitored for throughput, and slowest mirrors are continuously rotated out of the pool and replaced by new ones.



## Working principle



The core of Refractor is a pool of workers, to which HTTP requests are routed. A worker mapped to a particular mirror performs the request to said mirror and proxies the response to the user.



Before considering a request, workers look how well they are performing compared to their peers. If they are on the bottom two positions of the ranking, they will resign and get out of the pool. The pool will automatically add a worker for a different mirror to compensate.



This way, the pool of active mirrors is constantly rotating slow mirrors out of the pool, based on their current performance. This eliminates the need of continuously benchmarking mirrors, and avoids having to assume that mirrors' bandwidth is constant in time.



## Intended usage



Refractor is intended to be run either locally, or in a local network where linux machines reside. This is because Refractor drops mirrors aggressively based on mirror-to-client throughput, and therefore it will not be effective if clients with different effective throughput to the host running Refractor connect to it. Moreover, for this same reason, bad actors could deliberately simulate bad latencies and kick good mirrors out of the pool, degrading service quality for others.



## Providers



Refractor is designed to be distribution-agnostic, as long as a Provider that can fetch a mirror and feed it to the pool is implemented. Refractor automatically sorts the pool of mirrors automatically by the throughput they provide as request come by. This means that providers do not need to sort or benchmark mirrors before supplying them to the pool.



It is recommended, however, for providers to apply coarse-grain filter such as physical location, as doing so will allow the pool to stabilize faster.



For the moment, the following providers exist:



### Arch Linux (`archlinux`)



The Arch Linux provider feeds mirrors from `https://archlinux.org/mirrors/status/json/`, after applying some user-defined filters. For now, filtering by country and by score is allowed.



```yaml

workers: 8

goodThroughputMiBs: 10



provider:

  archlinux:

    maxScore: 5

    countries:

      - ES

      - IT

      - FR

      - PT

```



### Command (`command`)



The Command provider allows to feed to the pool mirror URLs obtained from running an user-defined command. This should help as an stop-gap for supporting distros without coding providers from them.



> ⚠️ Refractor rotates mirrors from the pool very aggressively, which means the specified command will be called multiple times and very often. Please make sure this command is not hammering any public API without appropriate caching.



```yaml

workers: 8

goodThroughputMiBs: 10



provider:

  command:

    #shell: /bin/bash # Defaults to $SHELL, then to /bin/sh

    command: |

      cat <