{"id":18942006,"url":"https://github.com/lucidfrontier45/pydanticio","last_synced_at":"2026-01-16T11:36:30.636Z","repository":{"id":246536056,"uuid":"821336839","full_name":"lucidfrontier45/pydanticio","owner":"lucidfrontier45","description":"Tiny IO utility library for Python with Pydantic inspired by Serdeio of Rust","archived":false,"fork":false,"pushed_at":"2026-01-13T15:48:00.000Z","size":52,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-13T18:20:41.598Z","etag":null,"topics":["file","file-reading","file-writing","input-output","io","pydantic","python","utility"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidfrontier45.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2024-06-28T10:07:51.000Z","updated_at":"2026-01-13T15:22:04.000Z","dependencies_parsed_at":"2024-11-08T12:42:57.098Z","dependency_job_id":null,"html_url":"https://github.com/lucidfrontier45/pydanticio","commit_stats":null,"previous_names":["lucidfrontier45/pydanticio"],"tags_count":5,"template":false,"template_full_name":"lucidfrontier45/python-rye-template","purl":"pkg:github/lucidfrontier45/pydanticio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidfrontier45%2Fpydanticio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidfrontier45%2Fpydanticio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidfrontier45%2Fpydanticio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidfrontier45%2Fpydanticio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidfrontier45","download_url":"https://codeload.github.com/lucidfrontier45/pydanticio/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidfrontier45%2Fpydanticio/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478350,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T06:30:42.265Z","status":"ssl_error","status_checked_at":"2026-01-16T06:30:16.248Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["file","file-reading","file-writing","input-output","io","pydantic","python","utility"],"created_at":"2024-11-08T12:30:48.441Z","updated_at":"2026-01-16T11:36:30.631Z","avatar_url":"https://github.com/lucidfrontier45.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"logo.png\" alt=\"PydanticIO Logo\" width=\"500\" /\u003e\n\n[![PyPI Version](https://img.shields.io/pypi/v/pydanticio)](https://pypi.org/project/pydanticio/)\n[![Python Versions](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/pypi/l/pydanticio)](https://github.com/lucidfrontier45/pydanticio/blob/main/LICENSE)\n\nA tiny file IO utility library for Python powered by [Pydantic](https://docs.pydantic.dev/). This library is a port of the Rust library [SerdeIO](https://github.com/lucidfrontier45/serdeio).\n\n## Features\n\n- **Type-safe**: Read and write Pydantic models with full type inference\n- **Format support**: CSV, JSON, JSON Lines, TOML, YAML, and MessagePack (optional)\n- **Auto-detection**: Automatically detects format from file extension\n- **Simple API**: Intuitive functions for single records and lists\n- **Zero dependencies**: Core library only requires Pydantic\n\n## Installation\n\n```sh\n# Standard distribution\npip install pydanticio\n\n# With YAML support\npip install pydanticio[yaml]\n\n# With MessagePack support\npip install pydanticio[messagepack]\n\n# With TOML support\npip install pydanticio[toml]\n```\n\n## Quick Start\n\n```python\nfrom pydantic import BaseModel\nfrom pydanticio import read_records_from_file, write_records_to_file\n\nclass User(BaseModel):\n    name: str\n    age: int\n\n# Read from any supported format (auto-detected from extension)\nusers = read_records_from_file(\"users.csv\", User)\n\n# Or specify format explicitly (overrides file extension)\nusers = read_records_from_file(\"data.txt\", User, data_format=\"csv\")\n\n# Write to any supported format\nwrite_records_to_file(\"output.json\", users)\n```\n\n## Supported Formats\n\n| Format      | File Extensions                        | Single Record | List of Records |\n| ----------- | -------------------------------------- | ------------- | --------------- |\n| CSV         | `.csv`                                 | No            | Yes             |\n| JSON        | `.json`                                | Yes           | Yes             |\n| JSON Lines  | `.jsonl`, `.jl`, `.jsl`, `.json_lines` | No            | Yes             |\n| MessagePack | `.msgpack`                             | Yes           | Yes             |\n| TOML        | `.toml`                                | Yes           | No              |\n| YAML        | `.yaml`, `.yml`                        | Yes           | Yes             |\n\nAll text-based formats use UTF-8 encoding.\n\n### Newline Handling\n\nAll text-based formats handle newlines automatically:\n\n- **On read**: Any newline style (`\\n`, `\\r\\n`, or `\\r`) is accepted and normalized\n- **On write**: Each format uses its appropriate line ending per specification\n\n| Format     | Line Ending | Notes    |\n| ---------- | ----------- | -------- |\n| CSV        | `\\r\\n`      | RFC 4180 |\n| JSON       | `\\n`        |          |\n| JSON Lines | `\\n`        | RFC 7464 |\n| TOML       | Platform    |          |\n| YAML       | `\\n`        |          |\n\n## API Reference\n\n### Reading\n\n| Function                                                 | Description                          | Supported Formats             |\n| -------------------------------------------------------- | ------------------------------------ | ----------------------------- |\n| `read_record_from_reader(reader, model, format)`         | Read single record from `BinaryIO`   | JSON, MessagePack, TOML, YAML |\n| `read_record_from_file(path, model, data_format=None)`   | Read single record from file path    | JSON, MessagePack, TOML, YAML |\n| `read_records_from_reader(reader, model, format)`        | Read list of records from `BinaryIO` | All formats except for TOML   |\n| `read_records_from_file(path, model, data_format=None)`  | Read list of records from file path  | All formats except for TOML   |\n\n### Writing\n\n| Function                                                  | Description                         | Supported Formats             |\n| --------------------------------------------------------- | ----------------------------------- | ----------------------------- |\n| `write_record_to_writer(writer, record, format)`          | Write single record to `BinaryIO`   | JSON, MessagePack, TOML, YAML |\n| `write_record_to_file(path, record, data_format=None)`    | Write single record to file path    | JSON, MessagePack, TOML, YAML |\n| `write_records_to_writer(writer, records, format)`        | Write list of records to `BinaryIO` | All formats except for TOML   |\n| `write_records_to_file(path, records, data_format=None)`  | Write list of records to file path  | All formats except for TOML   |\n\n### Format Specification\n\nWhen using `*_from_file` or `*_to_file` functions, you can optionally specify the data format explicitly using the `data_format` parameter. If not specified, the format is automatically detected from the file extension.\n\n```python\nfrom pydanticio import read_records_from_file, write_records_to_file\n\n# Auto-detects CSV format from .csv extension\nusers = read_records_from_file(\"data/users.csv\", User)\n\n# Explicit format overrides file extension\nusers = read_records_from_file(\"data/file.xyz\", User, data_format=\"csv\")\nwrite_records_to_file(\"data/output.txt\", users, data_format=\"json\")\n```\n\n**Valid format values:**\n\n| Value          | Description    |\n| -------------- | -------------- |\n| `\"json\"`       | JSON format    |\n| `\"yaml\"`       | YAML format    |\n| `\"messagepack\"`| MessagePack    |\n| `\"toml\"`       | TOML format (single record only) |\n| `\"csv\"`        | CSV format (records only) |\n| `\"json_lines\"` | JSON Lines format (records only) |\n\nWhen `data_format` is `None` (default), the format is automatically detected from the file extension. When explicitly specified, it overrides the automatic detection.\n\n### Explicit Format Specification\n\nUse the `data_format` parameter to override automatic format detection from file extensions:\n\n```python\nfrom pydantic import BaseModel\nfrom pydanticio import (\n    read_records_from_file,\n    write_records_to_file,\n    read_record_from_file,\n    write_record_to_file,\n)\n\nclass User(BaseModel):\n    name: str\n    age: int\n\n# Override file extension - read CSV from .txt file\nusers = read_records_from_file(\"data/users.txt\", User, data_format=\"csv\")\n\n# Write JSON to file with non-standard extension\nwrite_records_to_file(\"data/export.xyz\", users, data_format=\"json\")\n\n# Single record with explicit format\nclass Config(BaseModel):\n    setting: str\n    value: int\n\nconfig = read_record_from_file(\"config.data\", Config, data_format=\"yaml\")\nwrite_record_to_file(\"config.out\", config, data_format=\"toml\")\n```\n\nThis is useful when:\n- Working with files that have non-standard extensions\n- Converting between formats while preserving original file\n- Ensuring consistent format regardless of file naming\n\n## Examples\n\n### Reading and Writing Lists\n\n```python\nfrom pydantic import BaseModel\nfrom pydanticio import read_records_from_file, write_records_to_file\n\nclass User(BaseModel):\n    name: str\n    age: int\n\n# Convert between formats\nusers = read_records_from_file(\"users.csv\", User)\nwrite_records_to_file(\"users.json\", users)\n```\n\n### Reading Single Records\n\n```python\nfrom pydantic import BaseModel\nfrom pydanticio import read_record_from_file\n\nclass Config(BaseModel):\n    name: str\n    version: int\n    enabled: bool\n\nconfig = read_record_from_file(\"config.toml\", Config)\nprint(config.name, config.version)\n```\n\n### Using Streams\n\n```python\nfrom pydantic import BaseModel\nfrom pydanticio import read_records_from_reader, write_records_to_writer\n\nclass Item(BaseModel):\n    id: int\n    value: str\n\n# Read from a file stream\nwith open(\"data.json\", \"rb\") as f:\n    items = read_records_from_reader(f, Item)\n\n# Write to a BytesIO stream\nfrom io import BytesIO\nbuffer = BytesIO()\nwrite_records_to_writer(buffer, items, \"json\")\n```\n\n### Converting Between Formats\n\n```bash\n# Command line usage example\npython examples/convert_format.py input.csv output.json\n```\n\nSee `examples/convert_format.py` for the full source code.\n\n## Requirements\n\n- Python 3.12+\n- Pydantic 2.5.0+\n\n## License\n\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidfrontier45%2Fpydanticio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidfrontier45%2Fpydanticio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidfrontier45%2Fpydanticio/lists"}