{"id":22621421,"url":"https://github.com/cca/vault_migration","last_synced_at":"2025-07-06T02:33:17.233Z","repository":{"id":200011049,"uuid":"703775921","full_name":"cca/vault_migration","owner":"cca","description":"Tools, ideas, and data for repository migration","archived":false,"fork":false,"pushed_at":"2025-03-07T23:49:27.000Z","size":640,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-08T00:26:49.888Z","etag":null,"topics":["equella","institutional-repository","invenio-rdm","metadata"],"latest_commit_sha":null,"homepage":"https://cca.github.io/vault_migration/crosswalk.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cca.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-11T22:32:48.000Z","updated_at":"2025-03-07T23:49:31.000Z","dependencies_parsed_at":"2024-02-12T20:27:55.201Z","dependency_job_id":"edb65062-6920-449e-8b8e-3c99bf379c20","html_url":"https://github.com/cca/vault_migration","commit_stats":null,"previous_names":["cca/vault_migration"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_migration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_migration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_migration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cca%2Fvault_migration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cca","download_url":"https://codeload.github.com/cca/vault_migration/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246127720,"owners_count":20727776,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["equella","institutional-repository","invenio-rdm","metadata"],"created_at":"2024-12-08T23:08:46.337Z","updated_at":"2025-07-06T02:33:17.226Z","avatar_url":"https://github.com/cca.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Repository Migrations\n\nTools, ideas, and data.\n\nSemantics: EQUELLA objects are _items_ with _attachments_. Invenio objects are _records_ with _files_. EQUELLA has _taxonomies_; Invenio has _vocabularies_. We use these terms consistently so it's clear what format an object is in (e.g. `python migrate/record.py item.json \u003e record.json` converts an _item_ into a _record_).\n\n## Setup \u0026 Tests\n\n```sh\nuv install # get dependencies\nsource .venv/bin/activate.fish # enter venv\npython -m spacy download en_core_web_lg # download spaCy model for Named Entity Recognition\npytest -v migrate/tests.py # run tests\n```\n\nMigrate scripts that create records require an `INVENIO_TOKEN` or `TOKEN` variable in our environment or .env file. To create a token: sign in as an admin and go to Applications \u003e Personal access tokens.\n\n## Vocabularies\n\nInvenio uses [vocabularies](https://inveniordm.docs.cern.ch/customize/vocabularies/) to represent a number of fixtures beyond just subject headings, like names, description types, and creator roles. They're stored under the app_data directory and loaded when an instance is initialized. Many of our controlled lists in contribution wizards and EQUELLA taxonomies will be mapped to vocabularies.\n\nThe **taxos** dir contains exported EQUELLA taxonomies and tools for working with them. The **vocab** dir contains YAML files for Invenio vocabularies.\n\n### Subjects\n\nWe create two subject vocabularies: one for Library of Congress subjects with URIs from one of their authorities and one for CCA local subjects not present in any LC authority.\n\nDownload our [subjects sheet](https://docs.google.com/spreadsheets/d/1la_wsFPOkHLjpv4-f3tWwMsCd0_xzuqZ5xp_p1zAAoA/edit#gid=1465207925) and run `python migrate/mk_subjects.py data/subjects.csv` to create the YAML vocabularies in the vocab dir (lc.yaml and cca_local.yaml) as well as migrate/subjects_map.json which is used to convert the text of VAULT subject terms into Invenio identifiers or ID-less keyword subjects.\n\nCopy the YAML vocabularies into the app_data/vocabularies directory of our Invenio instance. The site needs to be rebuilt to load the changes (`invenio-cli services destroy` and then `invenio-cli services setup` again). Eventually (Invenio v12) there will be a CLI command to alter vocabularies without rebuilding the site.\n\n## Creating Records in Invenio\n\n- **migrate/record.py**: Converts EQUELLA item JSON into Invenio record JSON\n- **migrate/api.py**: Converts an item and `POST`s it to Invenio to create a record\n- **migrate/import.py**: Imports an item _directory_ (created by [the export tool](https://github.com/cca/equella_scripts/tree/main/collection-export)) with its attachments to Invenio\n\nTo use these scripts, we must create a personal access token for an administrator account in Invenio:\n\n1. Sign in as an admin\n2. Go to **Applications** \u003e **Personal access tokens**\n3. Create one—its name and the `user:email` scope (as of v12) do not matter\n4. Copy it to clipboard and **Save**\n5. Paste in .env and/or set it as an env var, e.g. `set -x INVENIO_TOKEN=xyz` in fish\n\nBelow, we migrate a VAULT item to an Invenio record and post it to Invenio.\n\n```sh\nset -x INVENIO_TOKEN your_token_here\npython migrate/api.py items/item.json # example output below\nHTTP 201\nhttps://127.0.0.1:5000/api/records/k7qk8-fqq15/draft\nHTTP 202\n{\"id\": \"k7qk8-fqq15\", \"created\": \"2024-05-31T15:26:17.972009+00:00\", ...\nhttps://127.0.0.1:5000/records/k7qk8-fqq15\n```\n\nYou can sometimes trip over yourself if the `.env` file in the project root is loaded and contains an outdated personal access token. If API calls fail with 403 errors, check that the `TOKEN` and/or `INVENIO_TOKEN` environment variables are set correctly.\n\nRerunning a \"migrate\" script with the same input creates a new record, it doesn't update the existing one.\n\n## Items\n\nWe could write scripts to directly take an item from EQUELLA using its API, perform a metadata crosswalk, and post it to Invenio. Alternatively, we could work with local copies of items, perhaps created by the equella_scripts collection export tool.\n\nWe need to load the necessary fixtures, including user accounts, before adding to Invenio. For instance, the item owner needs to already be in Invenio before we can add them as owner of a record. If we attempt to load a record with a subject `id` that doesn't exist yet, we get a 500 error.\n\nWe download metadata for all items using equella-cli and a script like this:\n\n```sh\n#!/usr/bin/env fish\nset total (eq search -l 1 | jq '.available')\nset length 50 # can only download up to 50 at a time\nset pages (math floor $total / $length)\nfor i in (seq 0 $pages)\n  set start (math $i x $length)\n  echo \"Downloading items $start to\" (math $start + $length)\n  # NOTE: no attachment info, use \"--info all\" for both attachments \u0026 metadata\n  eq search -l $length --info metadata --start $start \u003e json/$i.json\nend\n```\n\n## Metadata Crosswalk\n\nWe can use the `item.metadata` XML of existing VAULT items for testing. Generally, `python migrate/record.py items/item.json | jq` to see the JSON Invenio record. See [our crosswalk diagrams](https://cca.github.io/vault_migration/crosswalk.html).\n\nSchemas:\n\n- https://cca.github.io/vault_schema/\n- https://inveniordm.docs.cern.ch/reference/metadata/\n\nIt's likely our schema is outdated/inaccurate in places.\n\nHow to map a field:\n\n- Add a brief description to the mermaid diagram in [docs/crosswalk.html](docs/crosswalk.html)\n- Write a test in tests.py with your input XML and expected record output\n- Write a `Record` method in migrate.py \u0026 use it in the `Record::get()` dict\n- Run tests, optionally run a record migration as described above\n\n## LICENSE\n\n[ECL Version 2.0](https://opensource.org/licenses/ECL-2.0)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fvault_migration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcca%2Fvault_migration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcca%2Fvault_migration/lists"}