https://github.com/libris/swepub-redux
https://github.com/libris/swepub-redux
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/libris/swepub-redux
- Owner: libris
- Created: 2022-01-10T13:40:34.000Z (over 4 years ago)
- Default Branch: develop
- Last Pushed: 2025-09-16T08:02:21.000Z (9 months ago)
- Last Synced: 2025-09-16T10:17:00.513Z (9 months ago)
- Language: Python
- Size: 17.1 MB
- Stars: 3
- Watchers: 8
- Forks: 0
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Swepub
Swepub consists of two programs
1. A pipeline, which fetches and cleans up publication metadata from a set of sources, using OAI-PMH. From this data an sqlite3 database is created which serves as the source for the service (the other program).
1. A service, the "Swepub website", which makes the aggregate Swepub data available for analysis.
## Setup
Assuming https://github.com/astral-sh/uv is installed, just clone this repo.
### Setup: Annif
For automated subject classification we use [Annif](https://annif.org/). This is optional in the default DEV environment;
if Annif is not detected, autoclassification will be automatically disabled.
To set it up, follow the instructions in https://github.com/libris/swepub-annif.
## Pipeline
To run the pipeline and harvest a few sources do:
```bash
uv run python -m pipeline.harvest --update --skip-unpaywall mdh miun mau
```
(`--skip-unpaywall` avoids hitting a non-public Unpaywall mirror; alternatively, you could set `SWEPUB_SKIP_REMOTE` which skips both Unpaywall and other remote services (e.g. shortdoi.org, issn.org).)
Expect this to take a few minutes. If you don't specify source(s) you instead get the full production data which takes a lot longer (~8 hours). Sources must exist in `pipeline/sources.json`. If the database doesn't exist, it will be created; if it already exists, sources will be incrementally updated (harvesting records added/updated/deleted since the previous execution of `pipeline.harvest --update`).
To forcibly create a new database, run `uv run python -m pipeline.harvest --force` (or `-f`).
Run `uv run python -m pipeline.harvest -h` to see all options.
There are no running "services", nor any global state. Each time the pipeline is executed, an sqlite3 database is either created or updated. You may even run more than one harvester (with a different database path) in parallel if you like.
You can `purge` (delete) one or more sources. In combination with a subsequent `update` command, this lets you completely remove a source and then harvest it fully, while keeping records from other sources in the database intact:
```bash
uv run python -m pipeline.harvest --purge uniarts
uv run python -m pipeline.harvest --update uniarts
```
(If you omit the source name, all sources' records will be purged and fully harvested.)
For sources that keep track of deleted records, a much quicker way is:
```bash
uv run python -m pipeline.harvest --reset-harvest-time uniarts
uv run python -m pipeline.harvest --reset-harvest-time uniarts
```
`--reset-harvest-time` removes the `last_harvest` entry for the specified source(s), meaning the next `--update` will trigger a full harvest.
The resulting sqlite3 database has roughly the following structure (see storage.py for details):
| Table | Description |
| --- | --- |
|original| Original XML data for every harvested record. |
|converted| Converted+validated+normalized versions of each record. There is foreign key for each row which references the 'original' table |
|cluster| Clusters of converted records that are all considered to be duplicates of the same publication. There is a foreign key which references the 'converted' table. |
|finalized| A union or "master" record for each cluster, containing a merger of all the records in the cluster. There is a foreign key which references the 'cluster' table. |
## Service
If you want the frontend, make sure Nodejs/npm/yarn are installed, and then:
```bash
npm ci --prefix service/vue-client/
yarn --cwd service/vue-client/ build
```
For development, you can run `yarn --cwd service/vue-client/ build --mode=development` instead; this lets you use e.g. Vue Devtools.
To start the Swepub service (which provides the API and serves the static frontend files, if they exist):
```bash
uvr run python -m service.swepub
```
Then visit http://localhost:5000. API docs are available on http://localhost:5000/api/v2/apidocs.
To use port other than 5000, set `SWEPUB_SERVICE_PORT`.
## Training/updating Annif
See https://github.com/libris/swepub-annif
## Tests
Unit tests:
```bash
uv run pytest
# Also test embedded doctests:
uv run pytest --doctest-modules
```
To harvest specific test records, first start the mock API server:
```bash
uv run python -m tests.mock_api
```
Then, in another terminal:
```bash
export SWEPUB_SOURCE_FILE=tests/sources_test.json
uv run python -m pipeline.harvest -f dedup
uv run python -m service.swepub
```
To add a new test record, for example `foobar.xml`:
First, add `foobar.xml` to `tests/test_data`
Then, edit `tests/sources_test.json` and add an entry for the file:
```json
"foobar": {
"code": "whatever",
"name": "TEST - whatever",
"sets": [
{
"url": "http://localhost:8549/foobar",
"subset": "SwePub-whatever",
"metadata_prefix": "swepub_mods"
}
]
}
```
Make sure `foobar` in `url` corresponds to the filename (minus the `.xml` suffix).
Now you should be able to harvest it:
```bash
# Again, make sure the mock API server is rnning and that you've set
# export SWEPUB_SOURCE_FILE=tests/sources_test.json
uv run python -m pipeline.harvest -f foobar
```
## Working with local XML files
To quickly test changes without having to hit real OAI-PMH servers over and over,
you can download the XML files locally and start a local OAI-PMH server from which
you can harvest.
```bash
# Harvest to disk
uv run python -m misc.fetch_records uniarts ths # Saves to ./_xml by default; -h for options
# _or_, to fetch all sources: uv run python -m misc.fetch_records
# Start OAI-PMH server in the background or in another terminal
uv run python -m misc.oai_pmh_server # See -h for options
# Now harvest (--local-server defaults to http://localhost:8383/oai)
uv run python -m pipeline.harvest -f uniarts ths --local-server
```
This "OAI-PMH server" supports only the very bare minimum for `pipeline.harvest` to work
in (non-incremental mode). Remember that if you run `misc.fetch_records` again, you need
to restart `misc.oai_pmh_server` for it to pick up the changes.
You can also download only specific records (and when downloading specific records,
the XML will be pretty-printed):
```bash
uv run python -m misc.fetch_records oai:DiVA.org:uniarts-1146 oai:DiVA.org:lnu-108145
```
## Testing XML->JSON-LD conversion
Having downloaded the XML of a record (see "Working with local XML files" above):
```bash
uv run python -m misc.fetch_records oai:DiVA.org:uniarts-1146
```
...you can then test conversion from Swepub MODS XML to KBV/JSON-LD like so:
```bash
uv run python -m misc.mods_to_json resources/mods_to_xjsonld.xsl _xml/uniarts/oaiDiVA.orguniarts-1146.xml
```
Then you can edit `resources/mods_to_xjsonld.xsl` and/or `_xml/uniarts/oaiDiVA.orguniarts-1146.xml`,
run `misc.mods_to_json` again, and see what happens.
(With `xsltproc` you can also see the intermediary post-XSLT, pre-JSON-conversion XML:
`xsltproc resources/mods_to_xjsonld.xsl _xml/uniarts/oaiDiVA.orguniarts-1146.xml`)
## Testing conversion, enrichment and "legacy" export pipeline
```bash
uv run python -m misc.testpipe _xml/sometestfile.xml /tmp/swepub-testpipe/
```
Produces 3 files corresponding to conversion, audit and legacy steps:
```bash
/tmp/swepub-testpipe/sometestfile-out-1.jsonld
/tmp/swepub-testpipe/sometestfile-out-2-audited.jsonld
/tmp/swepub-testpipe/sometestfile-out-3-legacy.jsonld
```
## Testing changes to Swepub "legacy" export
Install `mysql-server` or `mariadb-server`. Follow the instructions in `pipeline/legacy_sync.py`.
Then, if you've set the relevant env vars (`SWEPUB_LEGACY_SEARCH_USER`, etc.), `pipeline.harvest`
will update the MySQL database. You can then inspect the JSON data of a specific record:
```bash
mysql -u -p swepub_legacy -sN -e "select data from enriched where identifier = 'oai:DiVA.org:uniarts-1146';" | jq
```
## Updating Resources
Some resources are generated from external data, and may eventually become stale and in need of a refresh. See `resources/Makefile`.