{"id":46893555,"url":"https://github.com/ywatanabe1989/scitex-io","last_synced_at":"2026-06-07T01:04:30.670Z","repository":{"id":343559850,"uuid":"1178235774","full_name":"ywatanabe1989/scitex-io","owner":"ywatanabe1989","description":"Universal scientific data I/O with plugin registry — save/load 30+ formats with one API. Part of SciTeX.","archived":false,"fork":false,"pushed_at":"2026-03-25T02:38:00.000Z","size":4562,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-26T04:51:00.586Z","etag":null,"topics":["cli","csv","data-io","hdf5","mcp","numpy","openscience","pandas","plugin-registry","python","research","scientific-computing","scitex"],"latest_commit_sha":null,"homepage":"https://scitex.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ywatanabe1989.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-10T20:36:28.000Z","updated_at":"2026-03-24T23:12:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ywatanabe1989/scitex-io","commit_stats":null,"previous_names":["ywatanabe1989/scitex-io"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/ywatanabe1989/scitex-io","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fscitex-io","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fscitex-io/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fscitex-io/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fscitex-io/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ywatanabe1989","download_url":"https://codeload.github.com/ywatanabe1989/scitex-io/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywatanabe1989%2Fscitex-io/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31307468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","csv","data-io","hdf5","mcp","numpy","openscience","pandas","plugin-registry","python","research","scientific-computing","scitex"],"created_at":"2026-03-10T23:19:03.053Z","updated_at":"2026-04-02T14:04:19.119Z","avatar_url":"https://github.com/ywatanabe1989.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scitex-io\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://scitex.ai\"\u003e\n    \u003cimg src=\"docs/scitex-logo-blue-cropped.png\" alt=\"SciTeX\" width=\"400\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cb\u003eUniversal scientific data I/O with plugin registry\u003c/b\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://badge.fury.io/py/scitex-io\"\u003e\u003cimg src=\"https://badge.fury.io/py/scitex-io.svg\" alt=\"PyPI version\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://scitex-io.readthedocs.io/\"\u003e\u003cimg src=\"https://readthedocs.org/projects/scitex-io/badge/?version=latest\" alt=\"Documentation\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/ywatanabe1989/scitex-io/actions/workflows/ci.yml\"\u003e\u003cimg src=\"https://github.com/ywatanabe1989/scitex-io/actions/workflows/ci.yml/badge.svg\" alt=\"Tests\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.gnu.org/licenses/agpl-3.0\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-AGPL--3.0-blue.svg\" alt=\"License: AGPL-3.0\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://scitex-io.readthedocs.io/\"\u003eFull Documentation\u003c/a\u003e · \u003ccode\u003epip install scitex-io\u003c/code\u003e\n\u003c/p\u003e\n\n---\n\n## Problem\n\nThree problems recur in every scientific Python project:\n\n1. **Format fragmentation.** Loading a CSV requires `pandas.read_csv()`, an HDF5 file requires `h5py.File()`, a NumPy array requires `numpy.load()`. Each format demands its own library, its own API, and its own boilerplate. Operating systems solved this decades ago — double-click any file and the OS dispatches to the right application. Python has no equivalent.\n\n2. **Hard-coded parameters scattered across scripts.** Sample rates, thresholds, model hyperparameters, plot dimensions — magic numbers buried in code, duplicated across files, impossible to track or share. Changing one parameter means grepping through the entire project.\n\n3. **Figures without provenance.** A saved PNG has no record of the code, parameters, or session that produced it. Months later, reproducing a figure means reverse-engineering which script with which settings generated it.\n\n## Solution\n\nscitex-io addresses all three:\n\n- **`save()`/`load()`** — One interface for 30+ formats with automatic extension-based dispatch. A plugin registry lets you add custom formats without modifying the library.\n- **`load_configs()`** — Loads all YAML files from a `config/` directory into a single `DotDict` with dot-notation access. Parameters are version-controlled, centralized, and separate from code.\n- **`embed_metadata()`/`read_metadata()`** — Embeds provenance (timestamps, session IDs, parameters) directly into image and PDF files. The figure carries its own history.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSupported Formats (30+)\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\n| Category | Extensions |\n|----------|-----------|\n| Spreadsheet | `.csv`, `.tsv`, `.xlsx`, `.xls`, `.xlsm`, `.xlsb` |\n| Scientific | `.npy`, `.npz`, `.mat`, `.hdf5`, `.h5`, `.zarr` |\n| Serialization | `.pkl`, `.pickle`, `.pkl.gz`, `.joblib` |\n| ML/DL | `.pth`, `.pt`, `.cbm` |\n| Config | `.json`, `.yaml`, `.yml`, `.xml` |\n| Database | `.db` (SQLite3) |\n| Documents | `.txt`, `.md`, `.pdf`, `.docx`, `.tex`, `.log` |\n| Code | `.py`, `.sh`, `.css`, `.js` |\n| Images | `.png`, `.jpg`, `.jpeg`, `.gif`, `.tiff`, `.tif`, `.svg` |\n| Media | `.mp4` |\n| Web | `.html` |\n| Bibliography | `.bib` |\n| EEG | `.vhdr`, `.vmrk`, `.edf`, `.bdf`, `.gdf`, `.cnt`, `.egi`, `.eeg`, `.set`, `.con` |\n\n\u003c/details\u003e\n\n## Installation\n\nRequires Python \u003e= 3.9.\n\n```bash\npip install scitex-io\n```\n\nFor MCP server support:\n\n```bash\npip install scitex-io[mcp]\n```\n\n\u003e **SciTeX users**: `pip install scitex` already includes scitex-io.\n\n## Quickstart\n\n### Save and Load\n\n```python\nfrom scitex_io import save, load\n\n# Universal save/load — format auto-detected from extension\nimport pandas as pd\ndf = pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]})\nsave(df, \"data.csv\")\nloaded = load(\"data.csv\")\n\n# 30+ formats work the same way\nimport numpy as np\nsave(np.array([1, 2, 3]), \"data.npy\")\nsave({\"key\": \"value\"}, \"config.yaml\")\nsave({\"nested\": [1, 2]}, \"data.json\")\n```\n\n### Project Configuration\n\nHard-coded parameters belong in config files, not in code. Use **UPPER_CASE** keys — Python's convention for constants — to signal that these are user-defined values:\n\n```\nproject/\n  config/\n    PATHS.yaml          # DATA_DIR: /data/experiment_01\n    PREPROCESS.yaml     # SAMPLE_RATE: 1000, BANDPASS: [0.5, 40]\n    MODEL.yaml          # HIDDEN_DIM: 256, DROPOUT: 0.3\n    PLOT.yaml           # FIGSIZE: [180, 60], DPI: 300\n    IS_DEBUG.yaml       # IS_DEBUG: true\n```\n\n```python\nfrom scitex_io import load_configs\n\nCONFIG = load_configs()          # loads ./config/*.yaml\nCONFIG.PATHS.DATA_DIR            # \"/data/experiment_01\"\nCONFIG.PREPROCESS.SAMPLE_RATE    # 1000\nCONFIG.MODEL.HIDDEN_DIM          # 256\n\n# Debug mode: DEBUG_ prefixed keys override their counterparts\n# In MODEL.yaml: { HIDDEN_DIM: 256, DEBUG_HIDDEN_DIM: 32 }\nCONFIG = load_configs(IS_DEBUG=True)\nCONFIG.MODEL.HIDDEN_DIM          # 32 (debug value promoted)\n```\n\nReturns a `DotDict` — a nested dictionary with dot-notation access. Parameters become version-controlled, shareable, and separate from code.\n\n### Metadata Embedding\n\nEmbed provenance into figures so they carry their own history:\n\n```python\nfrom scitex_io import embed_metadata, read_metadata, has_metadata\n\n# Embed metadata into an image\nembed_metadata(\"figure.png\", {\n    \"experiment\": \"exp_042\",\n    \"model\": \"resnet50\",\n    \"accuracy\": 0.94,\n    \"timestamp\": \"2026-03-11\",\n})\n\n# Read it back — months later, from the file alone\nmeta = read_metadata(\"figure.png\")\nprint(meta[\"experiment\"])    # \"exp_042\"\n\n# Check if a file has embedded metadata\nhas_metadata(\"figure.png\")   # True\n```\n\nSupports PNG (tEXt chunks), JPEG (EXIF), SVG (XML metadata), and PDF (XMP metadata).\n\n### Advanced Save Features\n\n`save()` auto-routes relative paths based on execution context and supports symlinks and dry runs:\n\n```python\nfrom scitex_io import save\n\n# Auto path routing — relative paths resolve based on context:\n#   Script analysis.py  → analysis_out/results.csv\n#   Notebook exp.ipynb  → exp_out/results.csv\n#   Interactive/IPython → /tmp/{USER}/results.csv\n#   Absolute paths      → used as-is\nsave(df, \"results.csv\")\n\n# Create symlink from cwd to the auto-routed save location\nsave(df, \"results.csv\", symlink_from_cwd=True)\n\n# Create symlink at a specific path\nsave(fig, \"fig1.png\", symlink_to=\"/data/latest/fig1.png\")\n\n# Skip auto CSV export for image saves\nsave(fig, \"plot.png\", no_csv=True)\n\n# use_caller_path=True — resolve path from the calling script,\n# not the immediate caller. Essential when save() is wrapped by a library.\nsave(df, \"results.csv\", use_caller_path=True)\n\n# Dry run — print resolved path without writing\nsave(df, \"results.csv\", dry_run=True)\n```\n\n### Glob and Caching\n\n```python\nfrom scitex_io import glob, parse_glob, load\n\n# Natural-sorted file matching (1, 2, 10 — not 1, 10, 2)\npaths = glob(\"data/**/*.csv\")\npaths = glob(\"results/{exp1,exp2}/*.npy\")  # brace expansion\n\n# Parse named placeholders from paths\npaths, parsed = parse_glob(\"sub_{id}/ses_{session}/*.vhdr\")\n# parsed = [{'id': '001', 'session': 'pre'}, ...]\n\n# Glob patterns work directly in load()\ndfs = load(\"results/*.csv\")  # → list of DataFrames\n\n# Caching is automatic (by path + mtime)\ndata = load(\"large.hdf5\")        # disk read\ndata = load(\"large.hdf5\")        # cache hit (instant)\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eCustom Format Registration\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\n```python\nfrom scitex_io import register_saver, register_loader, save, load\n\n@register_saver(\".custom\")\ndef save_custom(obj, path, **kwargs):\n    with open(path, \"w\") as f:\n        f.write(str(obj))\n\n@register_loader(\".custom\")\ndef load_custom(path, **kwargs):\n    with open(path) as f:\n        return f.read()\n\nsave(\"hello\", \"data.custom\")\nassert load(\"data.custom\") == \"hello\"\n```\n\n\u003c/details\u003e\n\n## Four Interfaces\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003ePython API\u003c/strong\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\n```python\nfrom scitex_io import save, load, list_formats, register_saver, register_loader\nfrom scitex_io import load_configs, DotDict\nfrom scitex_io import embed_metadata, read_metadata, has_metadata\n\nsave(obj, \"path.ext\")        # Save any object\ndata = load(\"path.ext\")      # Load any file\nfmts = list_formats()        # Show all registered formats\ncfg  = load_configs()        # Load ./config/*.yaml as DotDict\nembed_metadata(\"fig.png\", d) # Embed provenance into figure\n```\n\n\u003e **[Full API reference](https://scitex-io.readthedocs.io/)**\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eCLI Commands\u003c/strong\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\n```bash\nscitex-io --help-recursive          # Show all commands\nscitex-io info                      # Show registered formats\nscitex-io configs                   # Load and display project configs\nscitex-io configs -d ./my_configs   # Custom config directory\nscitex-io configs --json            # Output as JSON\nscitex-io list-python-apis -vv      # List Python APIs with signatures\nscitex-io version                   # Show version\nscitex-io mcp start                 # Start MCP server\nscitex-io mcp doctor                # Check MCP health\nscitex-io mcp list-tools -vv        # List MCP tools with parameters\n```\n\n\u003e **[Full CLI reference](https://scitex-io.readthedocs.io/)**\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eMCP Server — for AI Agents\u003c/strong\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nAI agents can save, load, and discover formats autonomously.\n\n| Tool | Description |\n|------|-------------|\n| `io_list_formats` | List all registered save/load formats |\n| `io_load` | Load data from any supported format |\n| `io_save` | Save data to any supported format |\n| `io_load_configs` | Load YAML project configurations |\n| `io_register_info` | Show how to register custom formats |\n\n```bash\nscitex-io mcp start\n```\n\n\u003e **[Full MCP specification](https://scitex-io.readthedocs.io/)**\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eSkills — for AI Agent Discovery\u003c/strong\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nSkills provide structured documentation that AI agents can query to discover package capabilities, API signatures, and usage patterns.\n\n```bash\nscitex-io skills list              # List available skill pages\nscitex-io skills get save-and-load # Get detailed save/load documentation\nscitex-io skills get glob          # Get glob/parse_glob patterns\nscitex-io skills get supported-formats  # Get all format tables\n```\n\n| Skill | Content |\n|-------|---------|\n| `save-and-load` | Core API, path routing, symlinks, `use_caller_path` |\n| `centralized-config` | `load_configs()`, DotDict, DEBUG_ override |\n| `metadata-embedding` | Provenance in PNG/JPEG/SVG/PDF |\n| `cache` | Load caching, reload, flush |\n| `glob` | Pattern matching with natural sort and parsing |\n| `linting-rules` | STX-IO001–007 lint rules |\n| `supported-formats` | All 30+ format tables |\n| `path-resolution` | Auto save-path routing, `scitex.path` utilities |\n\nAlso available via MCP: `io_skills_list()` / `io_skills_get(name)`.\n\n\u003c/details\u003e\n\n## Lint Rules\n\nDetected by [scitex-linter](https://github.com/ywatanabe1989/scitex-linter) when this package is installed.\n\n| Rule | Severity | Message |\n|------|----------|---------|\n| `STX-IO001` | warning | `np.save()` detected — use `stx.io.save()` for provenance tracking |\n| `STX-IO002` | warning | `np.load()` detected — use `stx.io.load()` for provenance tracking |\n| `STX-IO003` | warning | `pd.read_csv()` detected — use `stx.io.load()` for provenance tracking |\n| `STX-IO004` | warning | `.to_csv()` detected — use `stx.io.save()` for provenance tracking |\n| `STX-IO005` | warning | `pickle.dump()` detected — use `stx.io.save()` for provenance tracking |\n| `STX-IO006` | warning | `json.dump()` detected — use `stx.io.save()` for provenance tracking |\n| `STX-IO007` | warning | `.savefig()` detected — use `stx.io.save(fig, path)` for metadata embedding |\n\n## Part of SciTeX\n\nscitex-io is part of [**SciTeX**](https://scitex.ai). When used inside the SciTeX framework, I/O is seamless:\n\n```python\nimport scitex\n\n@scitex.session\ndef main(CONFIG=scitex.INJECTED):\n    data = scitex.io.load(\"input.csv\")     # auto-tracked by clew\n    result = process(data)\n    scitex.io.save(result, \"output.csv\")   # auto-tracked by clew\n    return 0\n```\n\n`scitex.io` delegates to `scitex_io` — they share the same API and registry.\n\nThe SciTeX system follows the Four Freedoms for Research below, inspired by [the Free Software Definition](https://www.gnu.org/philosophy/free-sw.en.html):\n\n\u003eFour Freedoms for Research\n\u003e\n\u003e0. The freedom to **run** your research anywhere — your machine, your terms.\n\u003e1. The freedom to **study** how every step works — from raw data to final manuscript.\n\u003e2. The freedom to **redistribute** your workflows, not just your papers.\n\u003e3. The freedom to **modify** any module and share improvements with the community.\n\u003e\n\u003eAGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://scitex.ai\" target=\"_blank\"\u003e\u003cimg src=\"docs/scitex-icon-navy-inverted.png\" alt=\"SciTeX\" width=\"40\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003c!-- EOF --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fywatanabe1989%2Fscitex-io","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fywatanabe1989%2Fscitex-io","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fywatanabe1989%2Fscitex-io/lists"}