{"id":50141416,"url":"https://github.com/opencitations/oc_pruner","last_synced_at":"2026-05-24T01:34:01.638Z","repository":{"id":340775375,"uuid":"1167538527","full_name":"opencitations/oc_pruner","owner":"opencitations","description":"A tool for removing rows from an OpenCitations metadata or citations table based on the table's validation report, with support for running complete validation and pruning pipelines.","archived":false,"fork":false,"pushed_at":"2026-04-29T15:32:59.000Z","size":210,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-24T01:33:51.526Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opencitations.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-26T12:03:26.000Z","updated_at":"2026-04-29T16:02:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/opencitations/oc_pruner","commit_stats":null,"previous_names":["eliarizzetto/oc_pruner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/opencitations/oc_pruner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_pruner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_pruner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_pruner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_pruner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opencitations","download_url":"https://codeload.github.com/opencitations/oc_pruner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opencitations%2Foc_pruner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33418547,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-23T22:14:44.296Z","status":"ssl_error","status_checked_at":"2026-05-23T22:14:43.778Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-24T01:34:00.561Z","updated_at":"2026-05-24T01:34:01.631Z","avatar_url":"https://github.com/opencitations.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# oc_pruner\n\nA tool for removing rows from an OpenCitations metadata or citations table based on the table's validation report, with support for running complete validation and pruning pipelines.\n\n## Features\n\n- **Selective filtering**: Filter by error type (error/warning) and/or specific error labels\n- **Flexible configuration**: Configure via CLI arguments or configuration files\n- **Row-level deletion**: Removes entire rows containing issues\n- **Verbose output**: Detailed information about processing when needed\n- **Complete pipeline**: Run validation + pruning pipeline with multiple rounds for thorough cleaning\n- **Configurable pipeline**: Customise validation and pruning options when running the pipeline via CLI flags or config files\n\n## Installation\nThe library can be installed from **PyPI**:\n```\npip install oc_pruner\n```\n\n## Contributing / Development\n\nThis project uses [uv](https://docs.astral.sh/uv/) for dependency management and building. To set up a development environment:\n\n```bash\n# Clone the repository\ngit clone https://github.com/opencitations/oc_pruner.git\ncd oc_pruner\n\n# Create a virtual environment and install dependencies\nuv sync\n```\n\n## Quick Start\n\n### Run the Complete Pipeline\n\nRun a full validation and pruning pipeline for metadata and citations files:\n\n```bash\noc_pruner pipeline --meta metadata.csv --cits citations.csv --out-dir output_dir\n```\n\nThis will:\n\n  1. Validate both files\n  2. Remove invalid rows\n  3. Re-validate the cleaned files\n  4. Repeat the process to catch any newly exposed issues\n  5. Perform a final validation check\n\nYou can customise the pipeline behaviour (which errors to ignore, whether to verify ID existence, etc.) via CLI flags or a configuration file:\n\n```bash\n# Using CLI flags\noc_pruner pipeline -m metadata.csv -c citations.csv -o output_dir --ignore-labels br_id_syntax --verify-id-existence\n\n# Using a config file\noc_pruner pipeline -m metadata.csv -c citations.csv -o output_dir --config pipeline_config.yaml\n```\n\nSee the [Configuration](#configuration) section for details on the available options.\n\n### Prune a Single Table Based On Its Existing Validation Report\n\nRemove all issues (errors and warnings) from a CSV file:\n\n```bash\noc_pruner --csv input.csv --report report.json --output output.csv\n```\n\nOr use the explicit `prune` subcommand:\n\n```bash\noc_pruner prune --csv input.csv --report report.json --output output.csv\n```\n\n### With Verbose Output\n\nSee detailed information about what's being processed:\n\n```bash\noc_pruner prune --csv input.csv --report report.json --output output.csv --verbose\n```\n\n## Configuration\n\n### CLI Arguments for `pipeline` mode (`pipeline` subcommand)\n\n| Argument                  | Abbreviation | Required | Description                                                        |\n|---------------------------|--------------|----------|--------------------------------------------------------------------|\n| `--meta PATH`             | `-m`         | Yes      | Path to the input metadata CSV file                                |\n| `--cits PATH`             | `-c`         | Yes      | Path to the input citations CSV file                               |\n| `--out-dir PATH`          | `-o`         | Yes      | Path to the output directory where to write the output (pruned) files |\n| `--config PATH`           | —            | No       | Path to a YAML/JSON configuration file for pipeline options        |\n| `--error-type`            | `-e`         | No       | Filter issues by error type: `all` or `error`                     |\n| `--ignore-labels LABELS`  | `-i`         | No       | Comma-separated list of error labels to ignore                     |\n| `--verify-id-existence`   | —            | No       | Verify that bibliographic IDs exist via API lookup                 |\n| `--use-meta-endpoint`     | —            | No       | Use the OC Meta endpoint for ID existence checks                   |\n| `--strict-sequentiality`  | —            | No       | Skip closure check when individual validations report errors       |\n| `--help`                  | `-h`         | No       | Show help message                                                  |\n\n### CLI Arguments for single document mode (`prune` subcommand)\n\n| Argument          | Abbreviation | Required | Description                               |\n|-------------------|--------------|----------|-------------------------------------------|\n| `--csv PATH`      | `-t`         | Yes      | Path to the input CSV file                |\n| `--report PATH`   | `-r`         | Yes      | Path to the validation report JSON file   |\n| `--output PATH`   | `-o`         | Yes      | Path for the output CSV file              |\n| `--config PATH`   | `-c`         | No       | Path to configuration file (YAML or JSON) |\n| `--error-type`    | `-e`         | No       | Filter by error type: all or error        |\n| `--ignore-labels` | `-i`         | No       | Comma-separated error labels to ignore    |\n| `--verbose`       | `-v`         | No       | Show detailed processing information      |\n| `--init-config`   | —            | No       | Generate a configuration file template    |\n| `--list-labels`   | —            | No       | List all valid error labels               |\n| `--help`          | `-h`         | No       | Show help message                         |\n\n\n### Configuration File\n\nCreate a configuration file for default settings. The tool looks for:\n\n  1. Explicitly specified file (via `--config`)\n  2. `oc_pruner_config.yaml` or `oc_pruner_config.json` in current directory\n  3. `~/.oc_pruner_config.yaml` in home directory\n\nGenerate a template:\n\n```bash\noc_pruner --init-config\n```\n\nExample `oc_pruner_config.yaml`:\n\n```yaml\n# oc_pruner Configuration File\n\n# ============================================================\n# Pruning options (used by both 'prune' and 'pipeline')\n# ============================================================\n\n# Filter by error type: \"all\" (errors and warnings) or \"error\" (errors only)\nerror_type_filter: \"all\"\n\n# List of error labels to ignore (rows with these issues will be kept, unless interested by other issues)\nignore_error_labels:\n- \"extra_space\"\n- \"br_id_format\"\n\n# ============================================================\n# Validation options (used by 'pipeline')\n# ============================================================\n\n# Whether to verify that bibliographic IDs exist via API lookup\nverify_id_existence: false\n\n# Whether to use the OC Meta endpoint for ID existence checks\nuse_meta_endpoint: false\n\n# Whether to skip closure check when individual validations report errors\nstrict_sequentiality: false\n\n# Whether to use LMDB for caching (recommended for large files)\nuse_lmdb: false\n\n# Maximum size in bytes for LMDB environments (default: 1 GB)\n# map_size: 1073741824\n\n# Base directory for LMDB caches\n# cache_dir: null\n```\n\n### Configuration Priority\n\nSettings are applied in this order (later override earlier):\n\n  1. **Default values** from the code\n  2. **Configuration file** if found\n  3. **CLI arguments** (highest priority)\n\n## Usage Examples\n\n### Run the Complete Validation + Pruning Pipeline from CLI\n\nFor thorough cleaning of OpenCitations metadata and citations files, use the `pipeline` command:\n\n```bash\noc_pruner pipeline -m metadata.csv -c citations.csv -o output_dir\n```\n\n**Pipeline Arguments:**\n\n| Argument       | Abbreviation | Required | Description                          |\n|----------------|--------------|----------|--------------------------------------|\n| `--meta PATH`  | `-m`         | Yes      | Path to original metadata CSV        |\n| `--cits PATH`  | `-c`         | Yes      | Path to original citations CSV       |\n| `--out-dir`    | `-o`         | Yes      | Base output directory for results    |\n| `--config PATH` | —           | No       | Path to a YAML/JSON config file for pipeline options |\n| `--error-type` | `-e`         | No       | Filter issues by error type: `all` or `error` |\n| `--ignore-labels` | `-i`      | No       | Comma-separated error labels to ignore |\n| `--verify-id-existence` | — | No       | Verify bibliographic IDs via API lookup |\n| `--use-meta-endpoint` | —   | No       | Use OC Meta endpoint for ID checks   |\n| `--strict-sequentiality` | — | No      | Skip closure check on validation errors |\n\n**What the pipeline does:**\n\n  1. **First validation**: Validates both metadata and citations files\n  2. **First pruning**: Removes rows with validation errors\n  3. **Second validation**: Re-validates the cleaned files to catch new issues\n  4. **Second pruning**: Removes any newly exposed errors\n  5. **Third validation**: Re-validates again (removing citations may expose further metadata issues)\n  6. **Third pruning**: Final cleanup of any remaining errors\n  7. **Final validation**: Performs a sanity check on the final cleaned files\n\nYou can customise the pipeline via CLI flags or a config file. CLI flags override the config file:\n\n```bash\n# Using CLI flags\noc_pruner pipeline -m metadata.csv -c citations.csv -o output_dir --ignore-labels br_id_syntax --verify-id-existence\n\n# Using a config file\noc_pruner pipeline -m metadata.csv -c citations.csv -o output_dir --config pipeline_config.yaml\n```\n\nThe pipeline creates the following structure in the output directory:\n\n```\noutput_dir/\n├── cleaned/\n│   ├── metadata.csv       # Final cleaned metadata\n│   └── citations.csv      # Final cleaned citations\n└── validation_reports/\n    ├── first_round/\n    │   ├── metadata/\n    │   └── citations/\n    ├── second_round/\n    │   ├── metadata/\n    │   └── citations/\n    ├── third_round/\n    │   ├── metadata/\n    │   └── citations/\n    └── final_round/\n        ├── metadata/\n        └── citations/\n```\n\nAll operations are logged to `logs/pipeline_YYYYMMDD_HHMMSS.log`.\n\n### Remove Only Errors (Single Document)\n\nIgnore warnings and only remove rows with errors:\n\n```bash\noc_pruner --csv data.csv --report report.json --output clean.csv --error-type error\n```\n\n### Ignore Specific Error Labels (Single Document)\n\nKeep rows that have specific issues:\n\n```bash\noc_pruner --csv data.csv --report report.json --output clean.csv \\\n --ignore-labels extra_space,br_id_format\n```\n\n### Use Configuration File (Single Document)\n\nCreate a config file and use it:\n\n```bash\noc_pruner --init-config\n# Edit oc_pruner_config.yaml\noc_pruner --csv data.csv --report report.json --output clean.csv\n```\n\n### Combine Filters (Single Document)\n\nRemove only errors except for specific labels:\n\n```bash\noc_pruner --csv data.csv --report report.json --output clean.csv \\\n  --error-type error \\\n  --ignore-labels extra_space,type_format\n```\n\n### List Available Error Labels\n\nSee all valid error labels:\n\n```bash\noc_pruner --list-labels\n```\n\n## Validation Report Model\n\nThe validation report is a JSON file following the [validation report schema](schema.json). It consists of a list of issue objects, where each object represents a validation issue tied to specific locations in the CSV table.\n\n### Issue Object Structure\n\n```json\n{\n  \"validation_level\": \"csv_wellformedness\",\n  \"error_type\": \"error\",\n  \"error_label\": \"extra_space\",\n  \"message\": \"The value in this field is not expressed in compliance with the syntax...\",\n  \"valid\": false,\n  \"position\": {\n    \"located_in\": \"item\",\n    \"table\": {\n      \"0\": {\n        \"id\": [1]\n      }\n    }\n  }\n}\n```\n\n\n### Error Labels Reference\n\nThe supported issue labels are listed in the [validation report schema](schema.json) and the associated issues are explained [in this summary table](errors_map.csv).\n\n## How It Works\n\n  1. **Load Files**: Reads the CSV file and validation report\n  2. **Filter Issues**: Based on configuration, determines which issues to consider\n     - `--error-type error`: Only considers \"error\" type issues\n     - `--ignore-labels`: Ignores issues with specified labels\n  3. **Extract Affected Rows**: For each relevant issue, extracts row numbers from the position data\n  4. **Remove Rows**: Removes entire rows that contain any non-ignored issue\n  5. **Write Output**: Saves the cleaned CSV file\n\n**Important**: If a row has both an ignorable issue and a non-ignorable issue, the entire row is removed (the non-ignorable issue takes precedence).\n\n## API Usage\n\nYou can also use oc_pruner as a Python library:\n\n### Prune a Single Document\n\n```python\nfrom oc_pruner import prune\nfrom oc_pruner.config import PrunerConfig\n\n# Create configuration\nconfig = PrunerConfig(\n    error_type_filter=\"all\",\n    ignore_error_labels=[\"extra_space\"]\n)\n\n# Prune the CSV file\nprune(\n    csv_path=\"input.csv\",\n    report_path=\"report.json\",\n    output_path=\"output.csv\",\n    config=config,\n    verbose=True\n)\n```\n\n### Run the Pipeline\n\n```python\nfrom oc_pruner.pipeline import run_pruning_pipeline\nfrom oc_pruner.config import PipelineConfig\n\n# Create pipeline configuration\nconfig = PipelineConfig(\n    error_type_filter=\"all\",\n    ignore_error_labels=[\"extra_space\"],\n    verify_id_existence=False,\n    use_meta_endpoint=False,\n    strict_sequentiality=False,\n)\n\n# Run the pipeline\nrun_pruning_pipeline(\n    original_fp_meta=\"metadata.csv\",\n    original_fp_cits=\"citations.csv\",\n    base_out_dir=\"output\",\n    pipeline_config=config,\n)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencitations%2Foc_pruner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopencitations%2Foc_pruner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopencitations%2Foc_pruner/lists"}