{"id":51490393,"url":"https://github.com/epfllibrary/infoscience-imports","last_synced_at":"2026-07-07T11:04:23.072Z","repository":{"id":267100173,"uuid":"865990511","full_name":"epfllibrary/infoscience-imports","owner":"epfllibrary","description":"Infoscience Imports uses modular Python scripts for distinct tasks: harvesting publications and files from different sources : OpenAlex, unpaywall, WoS and Scopus, deduplicating and enriching metadata via local API, and uploading records to Infoscience using the DSpace API.","archived":false,"fork":false,"pushed_at":"2026-06-10T16:51:24.000Z","size":2836,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-06-10T17:10:19.269Z","etag":null,"topics":["import","institutional-repository","open-access","openalex","research-output"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epfllibrary.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-01T13:13:18.000Z","updated_at":"2026-05-26T11:36:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"48b70964-1173-4943-ae1f-ff7aa3ea06bc","html_url":"https://github.com/epfllibrary/infoscience-imports","commit_stats":null,"previous_names":["epfllibrary/infoscience-imports"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/epfllibrary/infoscience-imports","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfllibrary%2Finfoscience-imports","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfllibrary%2Finfoscience-imports/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfllibrary%2Finfoscience-imports/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfllibrary%2Finfoscience-imports/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epfllibrary","download_url":"https://codeload.github.com/epfllibrary/infoscience-imports/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfllibrary%2Finfoscience-imports/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35225044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["import","institutional-repository","open-access","openalex","research-output"],"created_at":"2026-07-07T11:04:22.344Z","updated_at":"2026-07-07T11:04:23.053Z","avatar_url":"https://github.com/epfllibrary.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Infoscience Import Pipeline\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-007480.svg)](LICENSE)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)\n\nAutomated harvesting, deduplication, enrichment, and loading of publication data from multiple external sources into **[Infoscience](https://infoscience.epfl.ch) / [DSpace-CRIS](https://wiki.lyrasis.org/display/DSDOC7x)**.\n\n**Sources:** [Scopus](https://dev.elsevier.com/) · [Web of Science](https://developer.clarivate.com/) · [Crossref](https://www.crossref.org/documentation/retrieve-metadata/rest-api/) · [OpenAlex](https://docs.openalex.org/) · [Zenodo](https://developers.zenodo.org/) · [EPO OPS](https://developers.epo.org/)\n\n---\n\n## Table of contents\n\n1. [What it does](#what-it-does)\n2. [Prerequisites](#prerequisites)\n3. [Installation](#installation)\n4. [Quick start](#quick-start)\n5. [Streamlit supervision UI](#streamlit-supervision-ui)\n6. [Authentication](#authentication)\n7. [Scheduled runs](#scheduled-runs)\n8. [CLI reference](#cli-reference)\n9. [Environments](#environments-dev--test--prod)\n10. [Environment variables](#environment-variables)\n11. [Architecture](#architecture)\n12. [Output structure](#output-structure)\n13. [Incremental logic](#incremental-logic)\n14. [License](#license)\n15. [Citation](#citation)\n\n---\n\n## What it does\n\nThe pipeline runs a linear sequence of stages on every execution:\n\n| Stage | Description |\n|---|---|\n| **Harvest** | Queries each enabled source API within the configured time window |\n| **Deduplicate** | Cross-source dedup (DOI, then title + year with type-aware rules for preprints and datasets), then type-scoped dedup against existing Infoscience items; ambiguous cases are flagged and forwarded to the DSpace workspace |\n| **Enrich** | EPFL author reconciliation (People API, ORCID); OA/full-text metadata ([Unpaywall](https://unpaywall.org/), OpenAlex) |\n| **Load** | Builds DSpace-CRIS item payloads and ingests them (skipped in `--dry-run`) |\n| **Report** | Generates a timestamped Excel report and optionally sends it by email |\n| **Persist** | Writes run history, per-source stats, publications, and EPFL author data to [DuckDB](https://duckdb.org/) |\n\nThe pipeline is **stateless**: incrementality comes from the sliding time window and deduplication against what is already in Infoscience.\n\n---\n\n## Prerequisites\n\n- Python **3.11+**\n- A `.env.dev` / `.env.test` / `.env.prod` credentials file at the project root (see [Environment variables](#environment-variables))\n- Network access to the APIs you intend to harvest\n\n---\n\n## Installation\n\n```bash\ngit clone \u003crepo-url\u003e \u0026\u0026 cd infoscience-imports\npython3 -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\n\n# Create your first credentials file\ncp .sample.env .env.dev   # edit and fill in values\n```\n\n---\n\n## Quick start\n\n```bash\n# Dry-run on dev — harvests and enriches but does not load into DSpace\npython3 data_pipeline/main.py --env dev --dry-run --no-email -vv\n\n# Real run on dev with a 7-day window\npython3 data_pipeline/main.py --env dev --window-days 7\n\n# Production run\npython3 data_pipeline/main.py --env prod\n```\n\n---\n\n## Streamlit supervision UI\n\nA web dashboard for monitoring run history, browsing and curating publications, and launching pipeline runs without touching the CLI.\n\n```bash\n./run_ui.sh          # default port 8501\n./run_ui.sh 8502     # custom port\n# or directly:\nstreamlit run app.py\n```\n\n### Pages\n\n| Page | Role | Description |\n|---|---|---|\n| 🏠 Tableau de bord | all | KPIs, 30-day trend chart, recent runs, per-source breakdown, charts by type / OA status / year / unit / journal / PDF proportion |\n| 🚀 Lancer un run | admin | Form to configure and launch the pipeline; live log streaming with stop button |\n| ⏰ Programmation | admin | Create and manage scheduled runs; cron-based triggers; enable/disable toggle; run-now button; scheduler status indicator |\n| 📋 Publications | all | Paginated, sortable, filterable datatable; OA / licence / PDF badges; EPFL author + unit aggregation; weak-status flag; Excel report download |\n| 📊 Statistiques | all | Per-run funnel by source, publication type breakdown, EPFL author and unit tabs |\n| ⚙️ Configuration | admin | Environment variable status, DuckDB info, `.env` template |\n\n### Dashboard charts\n\nThe dashboard includes six analytical visualisations, all scoped to the selected run or global:\n\n- **Types de documents** — horizontal bar, top-15\n- **Statut Open Access** — donut (OA + PDF / OA sans PDF / OA non-libre / Non-OA / Non défini)\n- **Publications par année** — bar chart by publication year\n- **Proportion PDF récupéré** — donut with summary count\n- **Top unités EPFL** — horizontal bar, top-15\n- **Top journaux** — horizontal bar, top-15\n\n### Publication curation features\n\nThe Publications page supports detailed curation:\n\n- **OA column** — `OA` / `Non-OA` / `Non-libre` (publisher-specific licences are never treated as OA)\n- **Licence column** — CC-BY variant displayed (e.g. `CC-BY-NC-ND`); prefix-matched so all `cc-*` variants are covered\n- **PDF ✓ column** — checkbox; only `True` when a valid PDF was retrieved under an open licence\n- **Auteurs EPFL** — reconciled EPFL authors with status/position; for rejected publications, shows pre-detected unreconciled authors\n- **Unités** — EPFL units with type in parentheses\n- **⚠️ column** — flags publications where *all* matched EPFL authors have a \"weak\" status (Hôte, Hors EPFL, Étudiant, or Personnel with non-permanent position)\n- **Note dédup** (`dedup_note`) — set when a record was let through despite a potential conflict in Infoscience; possible values: `supersedes_preprint`, `published_version_exists`, `cross_type_doi`, `dataset_in_other_collection`\n- **🚩 Doublon Infoscience** (`flagged_publication`) — JSON list of existing Infoscience items that triggered the flag, each with `uuid`, `doi`, and `dc_type`; use the UUID to locate the item directly in Infoscience\n\nUse the **Signalement dédup** filter (options: *Tous* / *🚩 Flaggés* / specific note value) to isolate flagged records for curation. Flagged records also appear in the dedicated **Flagged Publications** sheet of the Excel report.\n\nThe environment selector in the sidebar switches between `dev`, `test`, and `prod` — the choice is persisted and automatically passed to any run launched from the UI.\n\n---\n\n## Authentication\n\nThe Streamlit UI requires login. Two roles are available:\n\n| Role | Pages |\n|---|---|\n| `admin` | All pages |\n| `reporting` | Tableau de bord · Publications · Statistiques |\n\nThe CLI pipeline is **not** protected by authentication — credentials are only required for the web UI.\n\n### Setup\n\n```bash\n# Create the first users (passwords are prompted interactively)\npython -m ui.auth add admin    admin\npython -m ui.auth add reporter reporting\n\n# Other management commands\npython -m ui.auth list\npython -m ui.auth passwd \u003cusername\u003e\npython -m ui.auth remove \u003cusername\u003e\n```\n\nCredentials are stored as [bcrypt](https://pypi.org/project/bcrypt/) hashes in `.streamlit/auth.yaml` (gitignored). Copy `.streamlit/auth.yaml.example` as a reference for the file structure.\n\n---\n\n## Scheduled runs\n\nThe **⏰ Programmation** page (admin only) lets you define pipeline runs that fire automatically on a cron schedule, without manual intervention.\n\n### How it works\n\n`run_ui.sh` starts `scheduler.py` as a background daemon alongside Streamlit. The daemon reads `data/schedules.json` every 15 seconds and registers or updates APScheduler jobs accordingly — any change made in the UI takes effect without a restart.\n\nWhen a scheduled job fires, the scheduler:\n1. Acquires the per-environment run lock (`data/run_active_{env}.json`). If a run is already active, the execution is silently skipped.\n2. Spawns `data_pipeline/main.py` as an independent OS subprocess.\n3. Updates `last_run_at`, `last_run_id`, and `last_run_status` in `schedules.json` on completion.\n\nBecause jobs run as separate OS processes, **stopping or restarting the UI does not interrupt a run already in progress** — it continues to completion and releases the lock itself.\n\n### Creating a schedule\n\n1. Open **⏰ Programmation** in the sidebar (admin role required).\n2. Check the scheduler status indicator at the top — it should show 🟢. If it shows ⚪, start the UI via `./run_ui.sh`.\n3. Click **➕ Nouveau schedule** and fill in:\n\n| Field | Description |\n|---|---|\n| **Nom** | Label shown in the schedule list |\n| **Environnement** | `dev`, `test`, or `prod` |\n| **Sources** | One or more sources to harvest (default: `scopus`, `crossref`, `openalex`) |\n| **Fenêtre glissante** | Days back from today (default: 20) |\n| **Fréquence** | Preset (daily, weekly, …) or \"Personnalisé…\" for a manual cron expression |\n| **Expression cron** | Standard 5-field cron — pre-filled from the preset, always editable |\n| **Dry-run** | Skip DSpace ingestion |\n| **Désactiver l'envoi d'e-mail** | Suppress the Excel report email |\n\n4. Click **Créer le schedule**. The next execution time is shown immediately.\n\n### Managing schedules\n\nEach schedule card shows the next scheduled execution and the last run result (✅ completed / ⏳ running / ❌ failed / 🛑 killed).\n\n- **Toggle \"Actif\"** — enable or disable without deleting the schedule. Takes effect within 15 seconds.\n- **▶ Now** — fire the run immediately, using the schedule's configuration.\n- **🗑** — permanently delete the schedule.\n\nThe last 50 lines of `logs/scheduler.log` are accessible at the bottom of the page.\n\n---\n\n## CLI reference\n\n### Time window\n\n| Argument | Default | Description |\n|---|---|---|\n| `--window-days N` | `15` | Sliding window: *today − N days* → *today* |\n| `--start-date YYYY-MM-DD` | — | Fixed start date (requires `--end-date`) |\n| `--end-date YYYY-MM-DD` | — | Fixed end date (requires `--start-date`) |\n\n### Sources\n\n```bash\n# All sources (default)\npython3 data_pipeline/main.py\n\n# Specific subset\npython3 data_pipeline/main.py --sources scopus,wos,openalex\n```\n\nAvailable: `wos`, `scopus`, `crossref`, `openalex`, `zenodo`, `epo`\n\n### Query overrides\n\nOverride the default institution-wide query for one or more sources:\n\n```bash\npython3 data_pipeline/main.py \\\n  --query-wos    \"EPFL OR Lausanne\" \\\n  --query-scopus \"AFFIL(EPFL)\"\n```\n\nQuery overrides are also available in the UI via the **Requêtes (optionnel)** expander on the run page.\n\n#### Crossref — flexible query format\n\nThe Crossref query field accepts three formats:\n\n**Plain string** — uses the generic `query` parameter:\n```bash\n--query-crossref \"EPFL machine learning\"\n```\n\n**JSON object** — spread directly as API parameters (supports any [Crossref query index](https://api.crossref.org/swagger-ui/index.html) or [filter](https://www.crossref.org/documentation/retrieve-metadata/rest-api/rest-api-filters/)):\n```bash\n--query-crossref '{\"query.affiliation\": \"EPFL SV\", \"filter\": \"type:journal-article\"}'\n--query-crossref '{\"filter\": \"orcid:0000-0002-1825-0097\"}'\n```\n\n**JSON array** — each element runs as a separate API call; results are merged and deduplicated by DOI:\n```bash\n--query-crossref '[{\"query.affiliation\": \"EPFL\"}, {\"filter\": \"ror-id:02s376052\"}]'\n```\n\n\u003e **Reserved filters:** `from-created-date` and `until-created-date` are always injected by the harvester (from the configured time window) and cannot be overridden. Any other [Crossref filter](https://www.crossref.org/documentation/retrieve-metadata/rest-api/rest-api-filters/) can be used freely.\n\n### Author ID-based harvesting\n\nWhen any of these flags are provided, the pipeline switches to **ID-based harvesting** for the relevant sources instead of institution-wide queries.\n\n```bash\n# Scopus Author IDs\npython3 data_pipeline/main.py --scopus-ids \"7004212771,57201854951\"\n\n# Web of Science ResearcherIDs\npython3 data_pipeline/main.py --wos-ids \"A-1234-2010,R-5678-2017\"\n\n# ORCID iDs (used by Crossref and OpenAlex)\npython3 data_pipeline/main.py --orcid-ids \"0000-0002-1825-0097\"\n\n# OpenAlex Author IDs\npython3 data_pipeline/main.py --openalex-ids \"A12345678\"\n\n# All combined\npython3 data_pipeline/main.py \\\n  --scopus-ids \"7004212771\" \\\n  --orcid-ids  \"0000-0002-1825-0097\" \\\n  --sources scopus,openalex,crossref\n```\n\nEach flag also accepts a **file path** (one ID per line):\n\n```bash\npython3 data_pipeline/main.py --scopus-ids ./ids/scopus_ids.txt\n```\n\n### Run modes\n\n| Flag | Effect |\n|---|---|\n| `--dry-run` | Skip DSpace load and email — safe for inspection |\n| `--no-email` | Generate the report but do not send it |\n| `-v` / `-vv` | Increase log verbosity |\n| `--env {dev,test,prod}` | Select the target environment (see [Environments](#environments-dev--test--prod)) |\n| `--output-dir PATH` | Override the output directory (default: `data/`) |\n\n### Common recipes\n\n```bash\n# Inspect last 7 days without touching anything\npython3 data_pipeline/main.py --dry-run --no-email --window-days 7 -vv\n\n# Backfill a specific month on prod\npython3 data_pipeline/main.py --env prod \\\n  --start-date 2025-01-01 --end-date 2025-01-31\n\n# Harvest one author across all relevant sources\npython3 data_pipeline/main.py \\\n  --scopus-ids \"7004212771\" \\\n  --orcid-ids  \"0000-0002-1825-0097\" \\\n  --wos-ids    \"A-1234-2010\" \\\n  --sources scopus,wos,openalex,crossref --dry-run\n```\n\n---\n\n## Environments (dev / test / prod)\n\nThe pipeline supports three fully isolated environments. Each has its own credentials file, DuckDB database, and run-lock — a production run never interferes with development.\n\n| Environment | Credentials | Database | Run lock |\n|---|---|---|---|\n| `dev` | `.env.dev` | `data/pipeline_dev.duckdb` | `data/run_active_dev.json` |\n| `test` | `.env.test` | `data/pipeline_test.duckdb` | `data/run_active_test.json` |\n| `prod` | `.env.prod` | `data/pipeline_prod.duckdb` | `data/run_active_prod.json` |\n\nThe active environment defaults to `dev`. All `.env.*` files are git-ignored.\n\n### Setup\n\n```bash\ncp .sample.env .env.dev   # fill in dev credentials\ncp .sample.env .env.test  # fill in test credentials\ncp .sample.env .env.prod  # fill in prod credentials\n```\n\n### Switching environments\n\n**CLI** — applies to that invocation only, not persisted:\n```bash\npython3 data_pipeline/main.py --env prod --dry-run\n```\n\n**Streamlit UI** — use the selector at the top of the sidebar; the choice is persisted to `data/active_env` and passed automatically to any run launched from the UI. A coloured badge (green / orange / red) and a warning banner indicate the active environment.\n\n**Shell variable** — highest priority, useful for cron or CI/CD:\n```bash\nAPP_ENV=prod python3 data_pipeline/main.py\nAPP_ENV=test streamlit run app.py\n```\n\nPriority order: `APP_ENV` \u003e `--env` \u003e persisted `data/active_env` \u003e `dev`\n\n**Fallback:** if `.env.{env}` does not exist, the pipeline falls back to the generic `.env` at the project root.\n\n---\n\n## Environment variables\n\nCopy `.sample.env` to the appropriate `.env.*` file(s) and fill in the values.\n\n### Required\n\n| Variable | Description |\n|---|---|\n| `DS_API_ENDPOINT` | DSpace REST API base URL — `https://\u003cdomain\u003e/server/api` |\n| `DS_API_TOKEN` | DSpace REST API static token |\n\n### Authentication (alternative)\n\n| Variable | Description |\n|---|---|\n| `DS_ACCESS_TOKEN` | DSpace session cookie token — alternative auth set after login |\n\n### EPFL People API\n\n| Variable | Description |\n|---|---|\n| `API_EPFL_USER` | [EPFL People API](https://api.epfl.ch/) username (author reconciliation) |\n| `API_EPFL_PWD` | EPFL People API password |\n\n### Harvesting sources\n\n| Variable | Source | Description |\n|---|---|---|\n| `SCOPUS_API_KEY` | [Scopus](https://dev.elsevier.com/) | Elsevier API key |\n| `SCOPUS_INST_TOKEN` | Scopus | Elsevier institutional token |\n| `ELS_API_KEY` | [Unpaywall](https://unpaywall.org/) | Elsevier key for full-text PDF retrieval |\n| `WOS_TOKEN` | [Web of Science](https://developer.clarivate.com/) | Clarivate API token |\n| `EPO_OPS_KEY` | [EPO OPS](https://developers.epo.org/) | Open Patent Services consumer key |\n| `EPO_OPS_SECRET` | EPO OPS | Open Patent Services consumer secret |\n| `OPENALEX_API_KEY` | [OpenAlex](https://docs.openalex.org/) | Authenticated access (higher rate limits) |\n| `OPENALEX_DATA_VERSION` | OpenAlex | API data version (default: `2`) |\n| `ZENODO_API_KEY` | [Zenodo](https://developers.zenodo.org/) | Authenticated rate limit |\n| `ORCID_API_TOKEN` | [ORCID](https://info.orcid.org/documentation/api-tutorials/) | Bearer token for author reconciliation |\n\n### Polite pool / HTTP\n\n| Variable | Description |\n|---|---|\n| `CONTACT_API_EMAIL` | Email sent as `mailto` in requests to [Crossref](https://www.crossref.org/documentation/retrieve-metadata/rest-api/), [Unpaywall](https://unpaywall.org/), [OpenAlex](https://docs.openalex.org/) — strongly recommended |\n| `USER_AGENT` | HTTP `User-Agent` header (defaults to a sensible EPFL string if unset) |\n\n### Email report (optional)\n\n| Variable | Description |\n|---|---|\n| `RECIPIENT_EMAIL` | Report recipient address |\n| `SENDER_EMAIL` | Report sender address |\n| `SMTP_SERVER` | SMTP hostname |\n\n---\n\n## Architecture\n\nThe pipeline follows a strict linear sequence: **Harvest → Deduplicate → Enrich → Load → Report → Persist**.\n\n```\ndata_pipeline/\n├── main.py          Entry point — CLI args, orchestration, DuckDB persistence\n├── harvester.py     One Harvester subclass per source (WoS, Scopus, Crossref, …)\n├── deduplicator.py  Cross-source dedup + DSpace-aware dedup\n├── enricher.py      EPFL author reconciliation, OA/full-text enrichment\n├── loader.py        Builds DSpace-CRIS payloads, calls DSpaceClientWrapper\n└── reporting.py     Excel report generation + SMTP delivery\n\nclients/             One module per external API\nconfig/\n├── __init__.py      YAML loader — exposes source_order, default_queries, unit_types, …\n├── pipeline.yaml    Default harvest queries, source priority order, unit filters, Scopus AF-IDs\n└── mappings/\n    ├── collections.yaml      Infoscience collection names → UUIDs + DSpace Submission Forms section names\n    ├── doctypes.yaml         Source doc-types → collection + dc.type (active + commented-out)\n    ├── licenses.yaml         OA licence identifiers → DSpace display values\n    ├── versions.yaml         OA version identifiers → COAR URIs\n    └── types_authority.yaml  dc.type values → COAR authority identifiers\nmappings.py          Loads the above YAML files; exposes classify_record_type, get_version_mapping, …\nenv_loader.py        Environment selection and .env.* loading\ndb/pipeline_db.py    DuckDB persistence layer (run history, publications, authors)\nui/\n├── run_state.py     File-based mutex (one pipeline run at a time, per environment)\n├── auth.py          Streamlit authentication + role-based ACL\n└── styles.css       External stylesheet (colour tokens injected from app.py)\napp.py               Streamlit supervision UI\n```\n\n**Key design points:**\n\n- `env_loader.py` is the single source of truth for environment selection. It sets `APP_ENV` in `os.environ` on load so every downstream module (`PipelineDB`, `run_state`) picks up the correct environment without re-reading the state file.\n- `PipelineDB` opens a new [DuckDB](https://duckdb.org/) connection for every operation and closes it immediately — no persistent connections, which avoids write-lock conflicts. All DDL uses `CREATE/ALTER … IF NOT EXISTS` so schema migrations are idempotent.\n- The run-lock file (`data/run_active_{env}.json`) is created atomically with `open(..., 'x')` so two simultaneous UI submissions cannot both start a run.\n- The UI is authenticated via bcrypt-hashed credentials in `.streamlit/auth.yaml`. The CLI is authentication-free.\n- All data-driven configuration (queries, mappings, collection UUIDs) lives in `config/pipeline.yaml` and `config/mappings/*.yaml`. To add a new document type, update `doctypes.yaml`; to update a collection UUID after a DSpace migration, update `collections.yaml` — no Python changes required.\n- Source priority for deduplication merging is defined in `config/pipeline.yaml → source_order`.\n- The stylesheet lives in `ui/styles.css` (pure CSS); `app.py` only injects colour tokens as CSS custom properties (`var(--canard)`, etc.) via a small inline `\u003cstyle\u003e` block.\n\n---\n\n## Output structure\n\nEach run produces a timestamped subfolder under `data/`:\n\n```\ndata/\n└── 2025-10-08_03-15/\n    ├── Raw_WosItems.csv\n    ├── Raw_ScopusItems.csv\n    ├── Raw_CrossrefItems.csv\n    ├── Raw_OpenalexItems.csv\n    ├── Raw_ZenodoItems.csv\n    ├── Raw_EpoItems.csv\n    ├── DeduplicatedItems.csv\n    ├── UnloadedItems.csv          # clear duplicates found in Infoscience (discarded)\n    ├── Items.csv\n    ├── AuthorsAndAffiliations.csv\n    ├── EpflAuthors.csv\n    ├── ItemsWithOAMetadata.csv\n    ├── ImportedItems.csv\n    ├── RejectedItems.csv\n    └── Report_2025-10-08_03-15.xlsx\n```\n\nRun history, per-source statistics, publications, EPFL authors, and unit links are also written to DuckDB (`data/pipeline_{env}.duckdb`) and are browsable through the Streamlit UI.\n\n---\n\n## Incremental logic\n\nThe pipeline is intentionally stateless. Re-running it is always safe because:\n\n1. **Sliding window** — only publications within the configured date range are harvested.\n2. **DSpace-aware dedup** — the deduplicator queries Infoscience for existing items before loading, so already-imported records are never duplicated. Deduplication is **type-aware**:\n   - A dataset is only deduplicated against other datasets (scoped to the *Datasets and Code* collection); a title match in another collection is flagged but not discarded.\n   - A preprint whose published version already exists in Infoscience is forwarded to the DSpace workspace rather than silently dropped, so it can be reviewed and linked.\n   - Clear duplicates (same DOI, same type, same title + year within the same collection) are discarded without flagging.\n3. **Stable `row_id`** — each record gets a deterministic hash of its key fields, ensuring consistent matching across runs.\n\nRunning daily with a 15-day window (the default) catches late-indexed publications while the overlap with previous windows is handled entirely by deduplication.\n\n---\n\n## License\n\nThis project is licensed under the **MIT License** — see [LICENSE](LICENSE) for the full text.\n\nThe bundled [DSpace REST Python Client](https://github.com/the-library-code/dspace-rest-python) (`dspace/dspace_rest_client/`) is licensed under the **BSD 3-Clause License** (© The Library Code GmbH). MIT and BSD-3-Clause are compatible permissive licences; the BSD-3-Clause copyright notice is preserved in the LICENSE file and in the bundled source.\n\n---\n\n## Citation\n\nIf you use this software in your research or institutional work, please cite it using the metadata in [CITATION.cff](CITATION.cff) or the reference below:\n\n```bibtex\n@software{infoscience_import_pipeline,\n  author    = {Sicot, Julien and Borel, Alain and Geoffroy, Géraldine},\n  title     = {Infoscience Import Pipeline},\n  year      = {2026},\n  publisher = {EPFL Library},\n  url       = {https://github.com/epfllibrary/infoscience-imports},\n  license   = {MIT}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfllibrary%2Finfoscience-imports","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepfllibrary%2Finfoscience-imports","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfllibrary%2Finfoscience-imports/lists"}