https://github.com/cca/vault_indus_dedupe
https://github.com/cca/vault_indus_dedupe
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/cca/vault_indus_dedupe
- Owner: cca
- Created: 2023-02-08T19:49:38.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-02-08T23:20:38.000Z (over 3 years ago)
- Last Synced: 2025-02-03T12:35:18.677Z (over 1 year ago)
- Language: Python
- Size: 8.79 KB
- Stars: 0
- Watchers: 6
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Dedupe Industrial Design Items in VAULT
In March, 2019 we moved the Industrial Design Program collection to a custom filestore location outside the main filestore. openEQUELLA saves INDUS items to this location with one additional intermediary directory, named after the collection's UUID, in the attachments path:
- main filestore: {{ ROOT }}/Institutions/cca2012/Attachments/{{ 7-bit hash of item UUID }}/{{ item UUID }}/{{ item version }}
- custom filestore: {{ ROOT }}/Institutions/cca2012/Attachments/{{ collection UUID }}/{{ 7-bit hash of item UUID }}/{{ item UUID }}/{{ item version }}
However, openEQUELLA did not _move_ INDUS item attachments to the new location, it _copied_ them. Thus every INDUS item contributed before March, 2019 is duplicated at a corresponding path in the main filestore. I discovered this when trying to confirm that our retention procedures were actually purging items—I kept running across supposedly purged items with their attachments still present, then noticed they were all INDUS items.
## Outline
- identify relevant INDUS items
- remove their main filestore directory if
1. the attachments dir exists in the INDUS filestore
2. both attachments dirs have the same set of files
- handle exceptions that the script runs across
To collect a JSON array of INDUS items, we can iterate over `eq search` commands:
```sh
# how many items are there?
> eq search --collections 5b07c041-2353-4712-92d0-a71eed9201da --showall --modifiedBefore 2019-03-31 | jq .available
# ≈750 items, max search length is 50, iterate 14 times until we have all items
> for i in (seq 0 14); eq search --collections 5b07c041-2353-4712-92d0-a71eed9201da --showall --info detail --length 50 --mb 2019-03-31 --start (math 50 x $i) | jq '.results[]' >> items.txt; end
# items.txt is invalid JSON because there are no commas in between array members
# gsed is gnu sed (from homebrew), edits add commas & fix first & last lines
# could also do this in a text editor
cat items.txt | gsed -e 's/^}$/},/' -e '1c[{' -e '$c}]' > items.json
```
That will collect items that _still exist_ in VAULT, but items have were purged previously are not accessible via search. To find previously purged items from our retention procedures, we can use the `jq` command below (the output file will need to be wrapped in array brackets afterwards):
```sh
jq '.[] | select(.collection.uuid == "5b07c041-2353-4712-92d0-a71eed9201da")' purged.json | sed 's/^}$/},/' > INDUS.json
```
Once we have the items JSON, just sync this data to the file server and run the script with enough permissions to operate on the files in the filestore.
```sh
> ./sync.sh
> ssh v2
> cd indus-dedupe
# test first to make sure script works & output makes sense
> sudo python3 app.py --dry-run items.json
> sudo python3 app.py items.json
```
Included is a test.json file of example items that demonstrate different scenarios, meant to be processed with the `--dry-run` flag. It should show that one item exists only in the INDUS filestore, one in the main, and one in both with no differences in the file list.
## Results
Live items: of 743 items, all but one had a duplicate attachments directory under the main storage area. Of those 742 items, only 3 had differences in their file lists: 1 item (a 2016 test of character-encoding issues of uploaded filenames which has since been fixed) had 2 of the same file with different names and 2 items had an additional thumbnail image in the INDUS filestore which the main directory did not have, probably because it was generated after the storage change. I manually removed the attachments from these 3 items and then reran the script to delete the 740 duplicate attachments directories. There was no noticeable impact on disk usage per `df -h` before and after the script.
Purged items: of 818 items, 595 only existed in the main filestore, 202 items existed in both filestores, and 21 only existed in the INDUS filestore. 5 items had storage differences; all were related to derivative/preview files and not indicative of any real problem.
It is odd that so many items have entries in the INDUS filestore, and that some _only_ have files in the INDUS location. I reran the script to remove 595 + 202 = 797 directories from main storage, but that still leaves 202 + 21 = 243 directories under the INDUS filestore that should not be there.