{"id":22621415,"url":"https://github.com/cca/equella_clean_staging","last_synced_at":"2025-07-07T08:36:53.671Z","repository":{"id":84610383,"uuid":"178487247","full_name":"cca/equella_clean_staging","owner":"cca","description":"clean up files from openEQUELLA staging directory","archived":false,"fork":false,"pushed_at":"2024-11-25T23:42:01.000Z","size":43,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-02-03T12:35:06.634Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cca.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-29T23:21:10.000Z","updated_at":"2024-11-25T23:42:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"2694060b-801b-4595-953e-1035a7d68fc5","html_url":"https://github.com/cca/equella_clean_staging","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fequella_clean_staging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fequella_clean_staging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fequella_clean_staging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fequella_clean_staging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cca","download_url":"https://codeload.github.com/cca/equella_clean_staging/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246127738,"owners_count":20727777,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-08T23:08:44.442Z","updated_at":"2025-03-29T02:24:26.046Z","avatar_url":"https://github.com/cca.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Clean up EQUELLA Staging Folder\n\nThere's some 1.3TB of files left in /mnt/equelladata01/Staging many of which are years old and should have been deleted a long time ago.\n\n**Algorithm**:\n\n1. from list of `staging` database tuples\n1. check that `staging.user_session` is not in `cached_value.key`\n1. check that `staging.user_session` is not in `entity_lock.user_session`\n1. check that `staging.user_session` is not in `item_lock.user_session`\n1. enter {{7-bit hash of staging.stagingid}}/{{staging.stagingid}} directory\n1. check that the last modified date is over a year ago\n    + _NOTE_: just does this for the first file/dir in `os.listdir` which should be fine\n1. check that all files are in a list of duplicates (if we have such a list)\n1. if all the checks above passed, then\n    + delete the directory and its contents\n    + remove the row from the `staging` database table\n\nThat's clean.py, which looks at the EQUELLA database then to the files. The dircheck.py script goes in the other direction and looks at UUID-named directories under the Staging path, trying to find references to them in the database. Neither will delete files or database rows if `config.debug` is `True`.\n\n## Setup\n\nWe _should_ be able to set this up and run it using python 3 on our servers, but it was originally written for 2.7. Python on the server is 3.4.3. We can't use f-strings and the psycopg2 version is chosen to work with this older python. We may need to `apt-get install libpq-dev` to get it to work.\n\n```sh\n\u003e pip install -r requirements.txt\n\u003e cp example.config.py config.py\n\u003e vim config.py\n```\n\n**config.py**:\n\n```python\nhost = \"db.example.com\"\nuser = \"username\"\npassword = \"sup3rsecr3t\"\ndbname = \"equella\"\nport = 5432\nfilestore = \"/path/to/Staging\"\nduplicates_file = \"dupes.txt\"\ndebug = True\n```\n\nThe \"duplicates_file\" is an optional list of files known to be duplicated somewhere else in storage. This lets us check that it's 100% safe to remove a file before doing so. It is an optional parameter; if not provided, the script simply skips over this check. I've used [fdupes](https://github.com/adrianlopezroche/fdupes) to generate this text file.\n\nThe script will not delete files in `debug` mode. Outputs a lot of text about each entry in the \"staging\" database table. There are no command line flags, just run `python3 clean.py` or `python3 dircheck.py`. The script needs to run as root, otherwise it cannot access the mounted filestore. I recommend making config.py readable only by root (`chmod 400 config.py`).\n\n## LICENSE\n\n[ECL Version 2.0](https://opensource.org/licenses/ECL-2.0)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fequella_clean_staging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcca%2Fequella_clean_staging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fequella_clean_staging/lists"}