{"id":40115872,"url":"https://github.com/dlcs/echo-fs","last_synced_at":"2026-01-19T12:10:05.731Z","repository":{"id":9074506,"uuid":"60691657","full_name":"dlcs/echo-fs","owner":"dlcs","description":"Echo is a highly available NFS rig for Amazon Web Services.","archived":false,"fork":false,"pushed_at":"2025-11-18T12:59:26.000Z","size":64,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-11-18T15:49:25.483Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dlcs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2016-06-08T11:03:27.000Z","updated_at":"2022-05-23T15:23:46.000Z","dependencies_parsed_at":"2022-08-18T20:53:57.984Z","dependency_job_id":null,"html_url":"https://github.com/dlcs/echo-fs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dlcs/echo-fs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fecho-fs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fecho-fs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fecho-fs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fecho-fs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dlcs","download_url":"https://codeload.github.com/dlcs/echo-fs/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Fecho-fs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28567863,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T08:53:44.001Z","status":"ssl_error","status_checked_at":"2026-01-19T08:52:40.245Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-19T12:10:04.316Z","updated_at":"2026-01-19T12:10:05.725Z","avatar_url":"https://github.com/dlcs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# echo-fs\n\nEcho is a highly available NFS rig for Amazon Web Services.\n\nIt uses Redis as a backing store to maintain a [Sorted Set](https://redis.io/topics/data-types#sorted-sets) maintaining last accessed time for a given file.\n\n## Scripts\n\nThere are 3 different 'modes' that echo-fs can be run in, corresponding to the 3 different entry points:\n\n* Listen - `echo_listener.py`: listens for messages from SNS or SQS. Sets SortedSet score, or copies file from S3-\u003eNas depending on message.\n* Populate - `echo_populate.py`: starting from folder walks repo and updates SortedSet to record path and set score = last access time (`atime`).\n* Scavenger - `echo_scavenger.py`: if diskspace below threshold get a % of items from SortedSet and delete corresponding file if older than threshold.\n\n## Configuration\n\nThe follow envvars can be supplied to control operation\n\n| Name                           | Description                                                                                     | Default | Services  |\n| ------------------------------ | ----------------------------------------------------------------------------------------------- | ------- | --------- |\n| DEBUG                          | If `True` debug logging is enabled                                                              | `False` | All       |\n| ECHO_REDIS_HOST                | Host for redis connection                                                                       |         | All       |\n| ECHO_REDIS_PORT                | Redis port                                                                                      | 6379    | All       |\n| ECHO_REDIS_DB                  | Redis logical DB                                                                                | 0       | All       |\n| ECHO_CACHE_ROOT                | Dir to serve as root for operations                                                             |         | All       |\n| ECHO_QUEUE_REGION              | AWS region for listener                                                                         |         | Listen    |\n| ECHO_INPUT_QUEUE               | SQS queue name to monitor for listener                                                          |         | Listen    |\n| ECHO_ERROR_QUEUE               | SQS queue to send error messages to                                                             |         | Listen    |\n| ECHO_SCAVENGER_CACHE_THRESHOLD | Scavenger will only run when less than this % of free disk space                                | 50      | Scavenger |\n| ECHO_SCAVENGER_CHUNK_SIZE      | % of available files to delete in each pass                                                     | 10      | Scavenger |\n| ECHO_SCAVENGER_SLEEP_SECONDS   | Number of seconds to sleep between scavenger runs                                               | 30      | Scavenger |\n| ECHO_SCAVENGER_MIN_AGE_SECONDS | If \u003e 0, only files older than this will be considered for deletion. Age determined from `mtime` | 0       | Scavenger |\n| ECHO_POPULATE_CACHE_THRESHOLD  | Populate will only run when less than this % of free disk space                                 | 60      | Populate  |\n| ECHO_POPULATE_LOOP             | If `True`, populate script will continuously run.                                               | `False` | Populate  |\n| ECHO_POPULATE_SLEEP_SECONDS    | If continuously running, umber of seconds to sleep between populate runs sleep                  | 300     | Populate  |\n\n\u003e [!NOTE]\n\u003e Scavenger should run against a fully populated Redis DB, therefore `ECHO_POPULATE_CACHE_THRESHOLD` should be greater than `ECHO_SCAVENGER_CACHE_THRESHOLD`.\n\n## Initially populating Echo\n\nIf you want to run Echo on a volume that already has files in it, i.e. that Echo hasn't been made aware of during normal operations, you can populate Echo's Redis with the `echo-populate.py` script.\n\nThe following example assumes that you are considering a volume called `scratch`.\n\nCreate a Redis instance:\n\n```bash\ndocker run -d --name echo-redis redis:latest\n```\n\nBuild the Echo image:\n\n```bash\ndocker build -t echo-fs .\n```\n\nPopulate Echo with contents of volume `/scratch`:\n\n```bash\ndocker run -t -i --name echo-populate --rm \\\n  -e ECHO_REDIS_HOST=\"echo-redis\" \\\n  -e ECHO_REDIS_PORT=\"6379\" \\\n  -e ECHO_REDIS_DB=\"0\" \\\n  -e ECHO_CACHE_ROOT=\"/scratch\" \\\n  --link echo-redis:echo-redis \\\n  -v /scratch:/scratch \\\n  echo-fs \\\n  python -u echo_populate.py\n```\n\nYou can then see if that is working by running the Scavenger in isolation:\n\n```bash\ndocker run -t -i --name echo-scavenger --rm \\\n  -e ECHO_REDIS_HOST=\"echo-redis\" \\\n  -e ECHO_REDIS_PORT=\"6379\" \\\n  -e ECHO_REDIS_DB=\"0\" \\\n  -e ECHO_CACHE_ROOT=\"/scratch\" \\\n  -e ECHO_SCAVENGER_CACHE_THRESHOLD=\"50\" \\\n  -e ECHO_SCAVENGER_CHUNK_SIZE=\"2\" \\\n  -e ECHO_SCAVENGER_SLEEP_SECONDS=\"1\" \\\n  --link echo-redis:echo-redis \\\n  -v /scratch:/scratch \\\n  echo-fs \\\n  python -u echo_scavenger.py\n```\n\n### Docker Compose\n\nFor ease of running locally the included `docker-compose.yaml` file will run 3 services:\n\n* `echo-redis` latest Redis image. The 2 services below have a dependency on this starting successfully.\n* `echo-populate` - Will iterate over `./scratch` and populate redis. Every 60 seconds the populate loop will run.\n* `echo-scavenger` - Runs every 10 seconds, will delete anything older than 180seconds when the overall disk has \u003c 50% free space\n\nThis can be run via\n\n```bash\ndocker compose up\n```\n\n\n\u003e [!NOTE]\n\u003e This is intended for familiarisation / testing purposes only!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlcs%2Fecho-fs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdlcs%2Fecho-fs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlcs%2Fecho-fs/lists"}