{"id":20446371,"url":"https://github.com/ray-project/deltacat","last_synced_at":"2026-01-16T14:00:35.856Z","repository":{"id":37693741,"uuid":"395142441","full_name":"ray-project/deltacat","owner":"ray-project","description":"A portable Multimodal Lakehouse powered by Ray that brings exabyte-level scalability and fast, ACID-compliant, change-data-capture to your ML and analytics workloads.","archived":false,"fork":false,"pushed_at":"2026-01-14T23:28:44.000Z","size":7913,"stargazers_count":265,"open_issues_count":24,"forks_count":44,"subscribers_count":10,"default_branch":"2.0","last_synced_at":"2026-01-15T05:43:19.564Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ray-project.png","metadata":{"files":{"readme":"README-development.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-08-11T23:42:44.000Z","updated_at":"2026-01-12T11:13:43.000Z","dependencies_parsed_at":"2024-01-17T00:48:02.165Z","dependency_job_id":"b8ec61ac-d66f-4300-9e15-21fd12dc3853","html_url":"https://github.com/ray-project/deltacat","commit_stats":{"total_commits":169,"total_committers":11,"mean_commits":"15.363636363636363","dds":0.5680473372781065,"last_synced_commit":"b586cb3752b502623c6f67272e2162236a15e245"},"previous_names":[],"tags_count":102,"template":false,"template_full_name":null,"purl":"pkg:github/ray-project/deltacat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdeltacat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdeltacat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdeltacat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdeltacat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ray-project","download_url":"https://codeload.github.com/ray-project/deltacat/tar.gz/refs/heads/2.0","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ray-project%2Fdeltacat/sbom","scorecard":{"id":763366,"data":{"date":"2025-08-11","repo":{"name":"github.com/ray-project/deltacat","commit":"236cb5e22671ffd3d1020a3acb5bae9e7f3d459d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":5,"checks":[{"name":"Maintained","score":10,"reason":"16 commit(s) and 11 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":8,"reason":"Found 26/30 approved changesets -- score normalized to 8","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Warn: no topLevel permission defined: .github/workflows/publish-to-pypi.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Info: Possibly incomplete results: error parsing shell code: LitWord cannot be followed by a word: dev/deploy/aws/scripts/s3-build-and-deploy.sh:0","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/ci.yml/2.0?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/ci.yml/2.0?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:34: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/ci.yml/2.0?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:38: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/ci.yml/2.0?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:56: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/ci.yml/2.0?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:61: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/ci.yml/2.0?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish-to-pypi.yml:11: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/publish-to-pypi.yml/2.0?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/publish-to-pypi.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/publish-to-pypi.yml/2.0?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/publish-to-pypi.yml:39: update your workflow using https://app.stepsecurity.io/secureworkflow/ray-project/deltacat/publish-to-pypi.yml/2.0?enable=pin","Warn: containerImage not pinned by hash: dev/iceberg-integration/Dockerfile:16: pin your Docker image by updating python:3.9-bullseye to python:3.9-bullseye@sha256:3dcaf39bbe2d6d8031a4a7ce68a20f2d494419aca18ec064f9e82e146e97a0ed","Warn: pipCommand not pinned by hash: dev/iceberg-integration/Dockerfile:65","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:22","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:23","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:43","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:49","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:50","Warn: pipCommand not pinned by hash: .github/workflows/publish-to-pypi.yml:20","Warn: pipCommand not pinned by hash: .github/workflows/publish-to-pypi.yml:26","Warn: pipCommand not pinned by hash: .github/workflows/publish-to-pypi.yml:27","Warn: pipCommand not pinned by hash: .github/workflows/publish-to-pypi.yml:28","Info:   0 out of   7 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of  10 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during GetBranch(main): error during branchesHandler.query: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/publish-to-pypi.yml:6"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 29 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"18 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-48 / GHSA-fj7x-q9j7-g6q6","Warn: Project is vulnerable to: PYSEC-2024-161","Warn: Project is vulnerable to: PYSEC-2021-47 / GHSA-5jqp-qgf6-3pvh","Warn: Project is vulnerable to: GHSA-mr82-8j83-vxmv","Warn: Project is vulnerable to: GHSA-3749-ghw9-m3mg","Warn: Project is vulnerable to: PYSEC-2025-41 / GHSA-53q9-r3pm-6pq6","Warn: Project is vulnerable to: PYSEC-2024-252 / GHSA-5pcm-hx3q-hm94","Warn: Project is vulnerable to: GHSA-887c-mr87-cxwp","Warn: Project is vulnerable to: PYSEC-2024-251 / GHSA-pg7h-5qx3-wjr3","Warn: Project is vulnerable to: PYSEC-2024-250","Warn: Project is vulnerable to: PYSEC-2024-259","Warn: Project is vulnerable to: GHSA-37mw-44qp-f5jm","Warn: Project is vulnerable to: GHSA-9356-575x-2w9m","Warn: Project is vulnerable to: GHSA-fpwr-67px-3qhx","Warn: Project is vulnerable to: GHSA-jjph-296x-mrcr","Warn: Project is vulnerable to: GHSA-phhr-52qp-3mj4","Warn: Project is vulnerable to: GHSA-q2wp-rjmx-x6x9","Warn: Project is vulnerable to: PYSEC-2025-40 / GHSA-qq3j-4f4f-9583"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-23T00:02:58.778Z","repository_id":37693741,"created_at":"2025-08-23T00:02:58.778Z","updated_at":"2025-08-23T00:02:58.778Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28479072,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T10:19:44.539Z","updated_at":"2026-01-16T14:00:35.836Z","avatar_url":"https://github.com/ray-project.png","language":"Python","funding_links":[],"categories":["Models and Projects","Table of Contents","Python"],"sub_categories":["Ray Data (Data Processing)","Metadata Service"],"readme":"# DeltaCAT Developer Quickstart Guide\n\n## Local Development\n### Install UV\nDeltaCAT uses [UV](https://docs.astral.sh/uv/) for package and project management. Before\nproceeding, [ensure that UV is installed](https://docs.astral.sh/uv/getting-started/installation/)\nand that the `uv` command is available in your path.\n\n### Verify Code Changes\nBefore publishing a pull request, ensure that the following validations pass:\n#### Unit Tests\n```shell\nmake test\n```\n\n#### Code-Style Checks\n```shell\nmake lint\n```\n\n### Makefile Targets\nWe use `make` to automate common development tasks. If you feel that any recurring development routine is\nmissing from the current set of Makefile targets, please propose a new one!\n\n#### build\n```shell\nmake build\n```\nBuilds a redistributable wheel to the `build` directory. Stores intermediate artifacts in the `dist` directory.\n\n#### clean-build\n```shell\nmake clean-build\n```\nRemoves all non-virtual-environment artifacts created by the `build` Makefile target.\n\n#### rebuild\n```shell\nmake rebuild\n```\nRuns `clean-build` followed by `build`.\n\n#### clean\n```shell\nmake clean\n```\nRemoves all artifacts created by the `build` target and removes the virtual environment.\n\n#### deploy-s3\n```shell\nmake deploy-s3\n```\nBuilds and uploads a wheel to S3. See [Build and Deploy an S3 Wheel](#build-and-deploy-an-s3-wheel).\n\n#### install\n```shell\nmake install\n```\nCreates a virtual environment if it doesn't exist and installs all build and runtime dependencies from pyproject.toml.\n\n#### lint\n```shell\nmake lint\n```\nRuns the linter to ensure that code in your local workspace conforms to code-style guidelines.\n\n#### test\n```shell\nmake test\n```\nRuns all unit tests.\n\n\u003e [!NOTE]\n\u003e To run an individual unit test where `my_deltacat_test` exists in either the test file, class,\n\u003e or function/method name, run a command of the form:\n\u003e ```shell\n\u003e uv run -m pytest -k \"my_deltacat_test\" -s -vv\n\u003e ```\n\u003e Note that the `-s` flag disables output capturing so that you can see stdout from `print`\n\u003e and other statements in real time, and `-vv` let's you see each test version and its input\n\u003e parameters to ease debugging.\n\n#### benchmark-aws\n```shell\nmake benchmark-aws\n```\nRun AWS benchmarks.\n\n#### type-mappings\n```shell\nmake type-mappings\n```\nRegenerates type mapping documentation and corresponding Python module. Specifically this:\n1. Regenerates the markdown documentation at `docs/schema/README.md`\n2. Regenerates the writer/reader compatibility mapping file at `utils/reader_compatibility_mapping.py`\nThis should be run after any changes to PyArrow, Polars, Pandas, Ray, or Daft dependency versions.\n\n## Integration Testing\n### AWS S3\nYou can deploy your local DeltaCAT changes to S3 and test them on any environment that can run Ray applications.\n\n#### Usage Prerequisites\n1. Install and configure the latest version of the AWS CLI:\n   * https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#getting-started-install-instructions\n2. Install and configure boto3:\n   * https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html\n   * https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html\n\n#### Install Wheel from a Signed S3 URL\nFor integration testing in any runtime environment that can access S3, you can run a single command to package your\nlocal changes in a wheel, upload it to S3, then install it on your Ray cluster from a signed S3 URL.\n\n###### Default S3 Bucket\nSimply run `make deploy-s3` to upload your local workspace to a wheel at\n`s3://deltacat-packages-{stage}/deltacat-{version}-{timestamp}-{python}-{abi}-{platform}.whl`.\n\nIf the deploy succeeds, you should see some text printed telling you how to install this wheel from a signed S3 URL:\n```\nto install run:\npip install deltacat @ `s3://deltacat-packages-{stage}/deltacat-{version}-{timestamp}-{python}-{abi}-{platform}.whl`\n```\nThe variables in the above S3 URL will be replaced as follows:\n\n\u003e **stage**: The runtime value of the `$DELTACAT_STAGE` environment variable if defined or the `$USER` environment\n\u003e variable if not.\n\n\u003e **version**: The current DeltaCAT distribution version. See https://peps.python.org/pep-0491/.\n\n\u003e **timestamp**: Second-precision epoch timestamp build tag. See https://peps.python.org/pep-0491/.\n\n\u003e **python**: Language implementation and version tag (e.g. ‘py27’, ‘py2’, ‘py3’). See https://peps.python.org/pep-0491/.\n\n\u003e **abi**: ABI tag (e.g. ‘cp33m’, ‘abi3’, ‘none’). See https://peps.python.org/pep-0491/.\n\n\u003e **platform**: Platform tag (e.g. ‘linux_x86_64’, ‘any’). See https://peps.python.org/pep-0491/.\n\n###### Custom S3 Bucket\nUse the `$DELTACAT_STAGE` environment variable to change the S3 bucket that your workspace wheel is uploaded to:\n```shell\nexport DELTACAT_STAGE=dev\nmake deploy-s3\n```\nThis uploads a wheel to\n`s3://deltacat-packages-dev/deltacat-{version}-{timestamp}-{python}-{abi}-{platform}.whl`.\n\n###### What Does it Do?\n1. Creates an S3 bucket at `s3://deltacat-packages-{stage}` if it doesn't already exist.\n2. Builds a wheel containing your local workspace changes and uploads it to\n`s3://deltacat-packages-{stage}/`.\n\n#### Benchmarks\nYou can also benchmark your DeltaCAT changes against public AWS S3 datasets by running:\n```shell\nmake benchmark-aws\n```\n\u003e [!NOTE]\n\u003e We recommend running benchmarks in an environment configured for high bandwidth access to cloud storage.\n\u003e For example, on an EC2 instance with enhanced networking support: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html.\n\n###### Adding Benchmarks\n**Parquet Reads**: Modify the `SINGLE_COLUMN_BENCHMARKS` and `ALL_COLUMN_BENCHMARKS` fixtures in `deltacat/benchmarking/benchmark_parquet_reads.py`\nto add more files and benchmark test cases.\n\n## Coding Quirks \u0026 Conventions\n### Storage and Catalog APIs\nDeltaCAT defines the signature for its internal `storage` and `catalog` APIs in their respective `interface.py` files.\nAll functions in `interface.py` raise a `NotImplementedError` if invoked directly. Implementations of each are expected\nto conform to the function signatures defined in `interface.py`, with conformance validated through unit tests.\n\nSome reasons we made these \"classless\" interface signatures are:\n1. **STATELESS**: All `catalog` and `storage` implementations should be stateless (e.g., to support\nwrapping in stateless web services), but classes encourage tracking ephemeral state in class properties.\n2. **SERDE LIMITATIONS**: In early DeltaCAT test cases, distributed Ray applications using equivalent `storage` and\n`catalog` classes produced oversized serialized payloads via Ray cloudpickle. This resulted in application stability\nissues and/or severe runtime performance penalties.\n\n### Storage Models\nDeltaCAT's base metadata `storage` model (`Metafile`) and all child classes inherit from a standard Python\n`Dict`. Other `storage` models like `SortKey` inherit from other standard Python collections like `Tuple`.\n\nThere are a few reasons for this:\n1. **SERDE**: `Dict` and other Python collections support standardized serialization/deserialization via\n`json`, `msgpack`, `pickle`, and Ray `cloudpickle` `dumps`/`loads` functions. They also support standardized output\nto a wide variety of human-readable and/or pretty-printed string formats (e.g., via `pprint`) to simplify log message\nevaluation and debugging.\n2. **EXTENSIBLE**: `Dict` and other Python collections make it easy to add new properties to models over time, can\nstore/fetch any valid Python objects with known time \u0026 space complexities, and simplify delineation between when\nmodel validation is required (e.g., during write/read to/from disk via custom serde methods) and not (e.g., during\nin-memory instantiation before all final property states are known).\n3. **PERFORMANT**: We prioritize model performance over pure object-oriented design principals, and Python's base\ncollections (e.g., `Dict`, `List`, `Set`, `Tuple`) avoid many performance penalties otherwise incurred by Python OOD\nextensions like abstract base classes (`ABC`).\n\nHere are a few guiding principles to keep in mind when creating or modifying DeltaCAT's internal storage models:\n1. **CORRECTNESS**: The general tenet is \"don't make it easy for users to create invalid models\" not \"make it\nimpossible to create invalid models\". To that end, use properties \u0026 setters to validate reads/writes of in-memory model\nstate, and override `Metafile` `to_serializable`/`from_serializable` methods to validate model correctness during\nreads/writes to/from disk.\n    - **How hard should it be to create an invalid model?** A developer set on modifying a model's internal state\n   will do so, regardless of guardrails put in place. However, the act of creating an invalid model should look obvious,\n   not accidental (e.g., directly modifying the bytes of a persisted metadata file on disk, directly modifying\n   an in-memory model's key/value pairs in its underlying `Dict`, etc.).\n   - **Should I interact with a model's base collection directly?** If you're directly accessing key/value pairs\n   of a model's underlying `Dict`, then it's assumed that you know what you're doing, have intentionally bypassed all\n   property-based guardrails (e.g., for performance reasons), and will assume responsibility for leaving the model in a\n   valid state. This isn't implicitly a bad thing, provided that you understand the trade-offs being made.\n   - **Should I make my model immutable to prevent accidental modification?** DeltaCAT model performance,\n   flexibility, and SerDe compatibility take priority over trying to create immutable models. Don't worry about\n   trying to freeze your model via NamedTuple, frozendict, frozen pydantic ConfigDict, etc. Users that want to mutate\n   the model will do so anyway. An immutable type can be copied into new immutable types with the desired changes\n   applied, and forcing all nested objects to be immutable creates unnecessary limitations on the types of properties\n   that can be modeled.\n2. **DECORATORS**: Models should follow the decorator design pattern. In other words, they should only extend their\nbase Python collections and wrap their underlying methods, but shouldn't override base Python collection methods with\ndifferent/unexpected behaviors.\n3. **PROPERTIES**: Telegraph read-only model properties by just creating a `@property` decorator with no corresponding\nsetter. Telegraph mutable model properties by creating a corresponding `@property-name.setter` decorator.\n4. **PERSISTENCE**: Models should be validated before being written to durable storage by their `to_serializable`\nmethod. They should also be validated on every read from durable storage via their `from_serializable` method. Only model\nproperties that are persisted in the model's base Python collection will be persisted post-serialization. For example,\nif a model's base Python collection is `Dict`, then durable (written-to-disk) model properties must persist a\ncorresponding key/value pair in their underlying `Dict`, while ephemeral (in-memory-only) model properties should not.\n5. **INTERFACES**: Interface API declarations and abstract methods should simply raise a `NotImplementedError` in the\nbase class where they're defined, but not implemented.\n\n### Cloudpickle\nSome DeltaCAT compute functions interact with Ray `cloudpickle` differently than the typical Ray application. This\nallows us to improve compute stability and efficiency at the cost of managing our own distributed object garbage\ncollection instead of relying on Ray's automatic distributed object reference counting and garbage collection. For\nexample, see the comment at `deltacat/compute/compactor/utils/primary_key_index.py` for an explanation of our custom\n`cloudpickle.dumps` usage.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fray-project%2Fdeltacat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fray-project%2Fdeltacat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fray-project%2Fdeltacat/lists"}