{"id":31871577,"url":"https://github.com/khive-ai/pydapter","last_synced_at":"2026-04-24T20:01:02.863Z","repository":{"id":291481445,"uuid":"977305537","full_name":"khive-ai/pydapter","owner":"khive-ai","description":"adapt data to and from every format","archived":false,"fork":false,"pushed_at":"2026-04-24T18:09:56.000Z","size":21552,"stargazers_count":28,"open_issues_count":13,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-24T19:30:20.132Z","etag":null,"topics":["ai","data","database","pydantic","schema","vector"],"latest_commit_sha":null,"homepage":"https://khive-ai.github.io/pydapter/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/khive-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-03T22:20:09.000Z","updated_at":"2026-04-24T18:09:37.000Z","dependencies_parsed_at":"2025-08-24T17:13:36.719Z","dependency_job_id":"ce17cd74-b407-4db5-8e80-b3fff6c94cb3","html_url":"https://github.com/khive-ai/pydapter","commit_stats":null,"previous_names":["ohdearquant/pydapter","agenticsorg/pydapter","khive-ai/pydapter"],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/khive-ai/pydapter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khive-ai%2Fpydapter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khive-ai%2Fpydapter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khive-ai%2Fpydapter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khive-ai%2Fpydapter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/khive-ai","download_url":"https://codeload.github.com/khive-ai/pydapter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khive-ai%2Fpydapter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32238748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","data","database","pydantic","schema","vector"],"created_at":"2025-10-12T20:26:36.443Z","updated_at":"2026-04-24T20:01:02.843Z","avatar_url":"https://github.com/khive-ai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🔄 Pydapter: Infrastructure for the Post-Database World\n\n[![codecov](https://codecov.io/github/khive-ai/pydapter/graph/badge.svg?token=FAE47FY26T)](https://codecov.io/github/khive-ai/pydapter)\n[![PyPI version](https://img.shields.io/pypi/v/pydapter.svg)](https://pypi.org/project/pydapter/)\n![PyPI - Downloads](https://img.shields.io/pypi/dm/pydapter?color=blue)\n![Python Version](https://img.shields.io/badge/python-3.10%2B-blue)\n[![License](https://img.shields.io/github/license/ohdearquant/pydapter.svg)](https://github.com/ohdearquant/pydapter/blob/main/LICENSE)\n[![Discord](https://img.shields.io/discord/YOUR_SERVER_ID?color=5865F2\u0026logo=discord\u0026logoColor=white)](https://discord.gg/JDj9ENhUE8)\n\n---\n\n## The Problem We're Solving\n\nModern applications don't fit in one database anymore.\n\nYour LLM extracts insights that need PostgreSQL for transactions, Qdrant for\nvector search, Neo4j for relationships. Each integration means different APIs,\ndifferent patterns, weeks of custom code.\n\n**We calculated it: engineers spend 40% of their time on data plumbing instead\nof building intelligence.**\n\n## Why Pydapter Exists\n\nWe believe the data layer should be invisible. When you build an application,\nyou should think about your domain models and business logic, not about cursor\nmanagement or connection pooling.\n\n**Pydapter makes storage a deployment decision, not an architecture decision.**\n\n---\n## The Paradigm Shift\n\nTraditional thinking: Choose a database, design around its constraints, live with the tradeoffs forever.\n\n**The new reality: Data flows where it provides the most value.**\n\n```python\n# One model, multiple destinations based on need\nfrom pydantic import BaseModel\n\nclass CustomerInsight(BaseModel):\n    \"\"\"Your domain model - doesn't care about storage\"\"\"\n    customer_id: str\n    insight_text: str\n    embedding: list[float]\n    relationships: dict\n\n# Same model flows to different systems based on purpose\nPostgresAdapter.to_obj(insight, table=\"insights\")        # Transactional storage\nQdrantAdapter.to_obj(insight, collection=\"vectors\")      # Similarity search\nNeo4jAdapter.to_obj(insight, label=\"Insight\")           # Relationship mapping\n```\n\n**When integration friction disappears, architecture becomes liquid.**\n---\n\n## What This Enables\n\n### 1. **Multi-Modal AI Pipelines**\n\nYour LLM processes a document and extracts entities. Those entities need to be\nstored for compliance (PostgreSQL), searchable by meaning (vector database), and\nanalyzed for relationships (graph database). With Pydapter, it's the same three\nlines of code for each destination.\n\n### 2. **Storage-Agnostic Applications**\n\nDeploy the same application on-premise with PostgreSQL, in AWS with DynamoDB, or\nat the edge with SQLite. Just change configuration, not code.\n\n### 3. **Evolutionary Architecture**\n\nStart with PostgreSQL. Add vector search when needed. Migrate to a graph\ndatabase for complex relationships. Your application code doesn't change - only\ndeployment configuration evolves.\n\n### 4. **True Vendor Independence**\n\nNo more lock-in through proprietary APIs. Switch between cloud providers,\ndatabases, or storage systems without touching application logic.\n\n---\n\n## How It Works\n\nOne consistent interface for everything:\n\n```python\n# Read from any source\ndata = Adapter.from_obj(YourModel, source_config, many=True)\n\n# Write to any destination\nAdapter.to_obj(data, destination_config, many=True)\n```\n\nWhether it's a local JSON file or a distributed database cluster, the pattern\nnever changes.\n\n### Real Example: AI Content Pipeline\n\n```python\nfrom pydantic import BaseModel\nfrom pydapter.extras.postgres_ import PostgresAdapter\nfrom pydapter.extras.qdrant_ import QdrantAdapter\nfrom pydapter.extras.neo4j_ import Neo4jAdapter\n\n\nclass Document(BaseModel):\n    id: str\n    content: str\n    summary: str\n    embedding: list[float]\n    entities: list[str]\n\n# Your AI pipeline extracts information\ndoc = ai_pipeline.process(raw_document)\n\n# One model, multiple destinations\nPostgresAdapter.to_obj(doc,\n    engine_url=\"postgresql://localhost/app\",\n    table=\"documents\"\n)\n\nQdrantAdapter.to_obj(doc,\n    collection=\"documents\",\n    url=\"http://localhost:6333\"\n)\n\nNeo4jAdapter.to_obj(doc,\n    url=\"bolt://localhost:7687\",\n    label=\"Document\"\n)\n```\n\n---\n\n## Why Framework Agnostic Matters\n\nPydapter works with **any** model framework, not just Pydantic:\n\n```python\n# Works with dataclasses\n@dataclass\nclass Product:\n    name: str\n    price: float\n\n    def to_dict(self):\n        return {\"name\": self.name, \"price\": self.price}\n\n# Use your existing serialization methods\nPostgresAdapter.to_obj(product,\n    adapt_meth=\"to_dict\",  # Your method\n    table=\"products\"\n)\n\n# Works with custom classes, attrs, or any validation framework\n```\n\n**No need to rewrite your models. Pydapter adapts to you.**\n\n---\n\n## Production Ready\n\n- **Battle Tested**: \u003e90% test coverage, used in production systems\n- **Enterprise Features**: Full async support, connection pooling, comprehensive\n  error handling\n- **Type Safe**: Complete type hints and protocol-based design\n- **Performance**: Minimal overhead - the bottleneck is your database, not\n  Pydapter\n\n---\n\n## Data Source Coverage\n\n| **Category**        | **Supported**              | **Async** |\n| ------------------- | -------------------------- | --------- |\n| **SQL Databases**   | PostgreSQL, MySQL, SQLite  | ✅        |\n| **NoSQL**           | MongoDB, DynamoDB          | ✅        |\n| **Vector Stores**   | Qdrant, Weaviate, Pinecone | ✅        |\n| **Graph Databases** | Neo4j, ArangoDB            | ✅        |\n| **Files**           | JSON, CSV, Excel, Parquet  | ✅        |\n\n---\n\n## Installation\n\n```bash\npip install pydapter\n\n# Add specific integrations as needed\npip install \"pydapter[postgres,mongo,qdrant]\"\n```\n\n---\n\n## Real-World Patterns\n\n### Pattern 1: Multi-Stage AI Pipeline\n\n```python\n# LLM processes document → multiple specialized storage systems\nclass ProcessedDocument(BaseModel):\n    doc_id: str\n    content: str\n    summary: str\n    entities: list[str]\n    embedding: list[float]\n\n# Stage 1: Store for compliance\nPostgresAdapter.to_obj(doc, table=\"documents\")\n\n# Stage 2: Enable semantic search\nQdrantAdapter.to_obj(doc, collection=\"docs\")\n\n# Stage 3: Map entity relationships\nNeo4jAdapter.to_obj(doc, label=\"Document\")\n```\n\n### Pattern 2: Environment-Specific Deployment\n\n```python\nimport os\n\n# Same code, different storage based on environment\nif os.getenv(\"ENVIRONMENT\") == \"local\":\n    adapter = SqliteAdapter\n    config = {\"db_path\": \"local.db\"}\nelif os.getenv(\"ENVIRONMENT\") == \"aws\":\n    adapter = DynamoAdapter\n    config = {\"table\": \"prod-data\", \"region\": \"us-east-1\"}\nelse:\n    adapter = PostgresAdapter\n    config = {\"engine_url\": os.getenv(\"DATABASE_URL\")}\n\n# Application code remains unchanged\ndata = adapter.from_obj(Model, config, many=True)\n```\n\n### Pattern 3: Gradual Migration\n\n```python\n# Migrate from MongoDB to PostgreSQL without downtime\nasync def migrate_with_dual_writes():\n    # Read from old system\n    data = await AsyncMongoAdapter.from_obj(Model, mongo_config, many=True)\n\n    # Write to both during migration\n    await AsyncMongoAdapter.to_obj(data, mongo_config, many=True)\n    await AsyncPostgresAdapter.to_obj(data, postgres_config, many=True)\n\n    # Later: switch reads to PostgreSQL, remove MongoDB\n```\n\n---\n\n## Advanced Features\n\n### Async-First Design\n\nAll major adapters support async operations for modern applications:\n\n```python\nasync with AsyncPostgresAdapter() as adapter:\n    results = await adapter.from_obj(Model, config, many=True)\n    await adapter.to_obj(new_data, config, many=True)\n```\n\n### Custom Serialization\n\nUse your existing model methods:\n\n```python\nclass CustomModel:\n    def to_dict(self): ...\n\n    @classmethod\n    def from_dict(cls, data): ...\n\n# Pydapter uses your methods\ndata = Adapter.from_obj(CustomModel, source, adapt_meth=\"from_dict\")\nAdapter.to_obj(data, destination, adapt_meth=\"to_dict\")\n```\n\n### Error Handling\n\nRich exceptions with context for debugging:\n\n```python\nfrom pydapter.exceptions import ValidationError, ConnectionError\n\ntry:\n    data = PostgresAdapter.from_obj(Model, config)\nexcept ConnectionError as e:\n    # Detailed connection diagnostics\n    logger.error(f\"Database unreachable: {e}\")\nexcept ValidationError as e:\n    # Field-level validation errors\n    for error in e.errors():\n        logger.error(f\"{error['loc']}: {error['msg']}\")\n```\n\n---\n\n## Creating Custom Adapters\n\nExtend Pydapter for your specific needs:\n\n```python\nfrom pydapter.core import Adapter\n\nclass RedisAdapter(Adapter[T]):\n    \"\"\"Example: Redis key-value adapter\"\"\"\n\n    @classmethod\n    def from_obj(cls, model_cls, config, /, *, many=False, **kw):\n        client = redis.from_url(config[\"url\"])\n        if many:\n            keys = client.keys(config.get(\"pattern\", \"*\"))\n            return [model_cls.model_validate_json(client.get(k)) for k in keys]\n        return model_cls.model_validate_json(client.get(config[\"key\"]))\n\n    @classmethod\n    def to_obj(cls, data, /, *, many=False, **kw):\n        client = redis.from_url(kw[\"url\"])\n        items = data if many else [data]\n        for item in items:\n            client.set(f\"{kw['prefix']}:{item.id}\", item.model_dump_json())\n```\n\n---\n\n## Why v1.0.0 Matters\n\nThis isn't just another release. It's a commitment.\n\n**API Stability**: The patterns you learn today will work in v2.0. We've spent\ntwo years refining the interface to be both powerful and permanent.\n\n**Production Trust**: Battle-tested in real systems handling millions of\nrecords. When you build on Pydapter, you build on solid ground.\n\n**The Standard**: We're establishing how modern applications should handle data.\nOne interface, everywhere.\n\n---\n\n## The Philosophy\n\nWe're not building another ORM or database wrapper. We're building\ninfrastructure for a fundamental shift in how applications relate to data.\n\n**Three principles guide everything:**\n\n1. **Storage is tactical, not strategic** - Choose databases based on current\n   needs, not future fears\n2. **Data should flow like water** - Between systems, across boundaries, without\n   friction\n3. **Applications outlive their databases** - Your business logic shouldn't be\n   coupled to storage decisions\n\n---\n\n## Documentation\n\n- **[Full Documentation](https://khive-ai.github.io/pydapter/)** - Complete API\n  reference and guides\n- **[Architecture Overview](https://khive-ai.github.io/pydapter/guides/architecture/)** -\n  Understanding the design\n- **[Getting Started](https://khive-ai.github.io/pydapter/getting_started/)** -\n  From zero to production\n\n---\n\n## Contributing\n\nPydapter is built by the community, for the community. We believe in the vision\nof invisible data layers and need your help to make it reality.\n\nSee our\n[Contributing Guide](https://github.com/khive-ai/pydapter/blob/main/CONTRIBUTING.md)\nto get started.\n\n---\n\n## License\n\nApache-2.0 License - see [LICENSE](LICENSE) for details.\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n## The Future is Fluid\n\nIn the post-database world, data flows where it provides value. Storage adapts\nto your needs. Architecture evolves without rewrites.\n\n**Join us in building infrastructure for applications that outlive their\ndatabases.**\n\n```bash\npip install pydapter\n```\n\n- **[GitHub](https://github.com/khive-ai/pydapter)**\n- **[Documentation](https://khive-ai.github.io/pydapter/)**\n- **[Discussions](https://github.com/khive-ai/pydapter/discussions)**\n\n---\n\n_What will you build when the data layer disappears?_\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhive-ai%2Fpydapter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkhive-ai%2Fpydapter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhive-ai%2Fpydapter/lists"}