{"id":22621448,"url":"https://github.com/cca/vault_indus_dedupe","last_synced_at":"2025-10-04T02:03:44.429Z","repository":{"id":84610445,"uuid":"599268787","full_name":"cca/vault_indus_dedupe","owner":"cca","description":null,"archived":false,"fork":false,"pushed_at":"2023-02-08T23:20:38.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-02-03T12:35:18.677Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cca.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-08T19:49:38.000Z","updated_at":"2023-02-08T19:51:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"b175ce55-e9ab-4473-a6e4-59717d676fb7","html_url":"https://github.com/cca/vault_indus_dedupe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_indus_dedupe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_indus_dedupe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_indus_dedupe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_indus_dedupe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cca","download_url":"https://codeload.github.com/cca/vault_indus_dedupe/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246127769,"owners_count":20727784,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-08T23:09:13.821Z","updated_at":"2025-10-04T02:03:39.399Z","avatar_url":"https://github.com/cca.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dedupe Industrial Design Items in VAULT\n\nIn March, 2019 we moved the Industrial Design Program collection to a custom filestore location outside the main filestore. openEQUELLA saves INDUS items to this location with one additional intermediary directory, named after the collection's UUID, in the attachments path:\n\n- main filestore: {{ ROOT }}/Institutions/cca2012/Attachments/{{ 7-bit hash of item UUID }}/{{ item UUID }}/{{ item version }}\n- custom filestore: {{ ROOT }}/Institutions/cca2012/Attachments/{{ collection UUID }}/{{ 7-bit hash of item UUID }}/{{ item UUID }}/{{ item version }}\n\nHowever, openEQUELLA did not _move_ INDUS item attachments to the new location, it _copied_ them. Thus every INDUS item contributed before March, 2019 is duplicated at a corresponding path in the main filestore. I discovered this when trying to confirm that our retention procedures were actually purging items—I kept running across supposedly purged items with their attachments still present, then noticed they were all INDUS items.\n\n## Outline\n\n- identify relevant INDUS items\n- remove their main filestore directory if\n    1. the attachments dir exists in the INDUS filestore\n    2. both attachments dirs have the same set of files\n- handle exceptions that the script runs across\n\nTo collect a JSON array of INDUS items, we can iterate over `eq search` commands:\n\n```sh\n# how many items are there?\n\u003e eq search --collections 5b07c041-2353-4712-92d0-a71eed9201da --showall --modifiedBefore 2019-03-31 | jq .available\n# ≈750 items, max search length is 50, iterate 14 times until we have all items\n\u003e for i in (seq 0 14); eq search --collections 5b07c041-2353-4712-92d0-a71eed9201da --showall --info detail --length 50 --mb 2019-03-31 --start (math 50 x $i) | jq '.results[]' \u003e\u003e items.txt; end\n# items.txt is invalid JSON because there are no commas in between array members\n# gsed is gnu sed (from homebrew), edits add commas \u0026 fix first \u0026 last lines\n# could also do this in a text editor\ncat items.txt | gsed -e 's/^}$/},/' -e '1c[{' -e '$c}]' \u003e items.json\n```\n\nThat will collect items that _still exist_ in VAULT, but items have were purged previously are not accessible via search. To find previously purged items from our retention procedures, we can use the `jq` command below (the output file will need to be wrapped in array brackets afterwards):\n\n```sh\njq '.[] | select(.collection.uuid == \"5b07c041-2353-4712-92d0-a71eed9201da\")' purged.json | sed 's/^}$/},/' \u003e INDUS.json\n```\n\nOnce we have the items JSON, just sync this data to the file server and run the script with enough permissions to operate on the files in the filestore.\n\n```sh\n\u003e ./sync.sh\n\u003e ssh v2\n\u003e cd indus-dedupe\n# test first to make sure script works \u0026 output makes sense\n\u003e sudo python3 app.py --dry-run items.json\n\u003e sudo python3 app.py items.json\n```\n\nIncluded is a test.json file of example items that demonstrate different scenarios, meant to be processed with the `--dry-run` flag. It should show that one item exists only in the INDUS filestore, one in the main, and one in both with no differences in the file list.\n\n## Results\n\nLive items: of 743 items, all but one had a duplicate attachments directory under the main storage area. Of those 742 items, only 3 had differences in their file lists: 1 item (a 2016 test of character-encoding issues of uploaded filenames which has since been fixed) had 2 of the same file with different names and 2 items had an additional thumbnail image in the INDUS filestore which the main directory did not have, probably because it was generated after the storage change. I manually removed the attachments from these 3 items and then reran the script to delete the 740 duplicate attachments directories. There was no noticeable impact on disk usage per `df -h` before and after the script.\n\nPurged items: of 818 items, 595 only existed in the main filestore, 202 items existed in both filestores, and 21 only existed in the INDUS filestore. 5 items had storage differences; all were related to derivative/preview files and not indicative of any real problem.\n\nIt is odd that so many items have entries in the INDUS filestore, and that some _only_ have files in the INDUS location. I reran the script to remove 595 + 202 = 797 directories from main storage, but that still leaves 202 + 21 = 243 directories under the INDUS filestore that should not be there.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fvault_indus_dedupe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcca%2Fvault_indus_dedupe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fvault_indus_dedupe/lists"}