https://github.com/0x26res/ptars
Protobuf to Arrow, using Rust
https://github.com/0x26res/ptars
python rust
Last synced: 5 months ago
JSON representation
Protobuf to Arrow, using Rust
- Host: GitHub
- URL: https://github.com/0x26res/ptars
- Owner: 0x26res
- License: apache-2.0
- Created: 2023-10-26T08:45:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2026-02-18T17:47:15.000Z (5 months ago)
- Last Synced: 2026-02-18T20:57:58.761Z (5 months ago)
- Topics: python, rust
- Language: Rust
- Homepage:
- Size: 1.52 MB
- Stars: 23
- Watchers: 1
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# ptars
[![PyPI Version][pypi-image]][pypi-url]
[![Python Version][versions-image]][versions-url]
[![PyPI Wheel][wheel-image]][wheel-url]
[![Documentation][docs-image]][docs-url]
[![Downloads][downloads-image]][downloads-url]
[![Downloads][downloads-month-image]][downloads-month-url]
[![Crates.io][crates-image]][crates-url]
[![Crates.io Downloads][crates-downloads-image]][crates-url]
[![docs.rs][docsrs-image]][docsrs-url]
[![Build Status][build-image]][build-url]
[![codecov][codecov-image]][codecov-url]
[![License][license-image]][license-url]
[![Ruff][ruff-image]][ruff-url]
[![snyk][snyk-image]][snyk-url]
[![Github Stars][stars-image]][stars-url]
[![GitHub issues][github-issues-image]][github-issues-url]
[![GitHub Release][release-image]][release-url]
[![Release Date][release-date-image]][release-url]
[![Last Commit][last-commit-image]][commits-url]
[![Commit Activity][commit-activity-image]][commits-url]
[![Open PRs][open-prs-image]][prs-url]
[![Contributors][contributors-image]][contributors-url]
[![Contributing][contributing-image]][contributing-url]
[![FOSSA Status][fossa-image]][fossa-url]
[![Repo Size][repo-size-image]][repo-size-url]
[![Rust][rust-image]][rust-url]
[![Apache Arrow][apache-arrow-image]][apache-arrow-url]
[![prek][prek-image]][prek-url]
[Repository](https://github.com/0x26res/ptars) |
[Python Documentation](https://ptars.readthedocs.io/) |
[Python Installation](https://ptars.readthedocs.io/en/latest/getting-started/#installation) |
[PyPI](https://pypi.org/project/ptars/) |
[Rust Crate](https://crates.io/crates/ptars) |
[Rust Documentation](https://docs.rs/ptars)
Fast convertion from Protocol Buffers to Arrow, and back, using Rust, with Python bindings.
## Example
Take a protobuf:
```protobuf
message SearchRequest {
string query = 1;
int32 page_number = 2;
int32 result_per_page = 3;
}
```
And convert serialized messages directly to `pyarrow.RecordBatch`:
```python
from ptars import HandlerPool
messages = [
SearchRequest(
query="protobuf to arrow",
page_number=0,
result_per_page=10,
),
SearchRequest(
query="protobuf to arrow",
page_number=1,
result_per_page=10,
),
]
payloads = [message.SerializeToString() for message in messages]
pool = HandlerPool([SearchRequest.DESCRIPTOR.file])
handler = pool.get_for_message(SearchRequest.DESCRIPTOR)
record_batch = handler.list_to_record_batch(payloads)
```
| query | page_number | result_per_page |
|:------------------|--------------:|------------------:|
| protobuf to arrow | 0 | 10 |
| protobuf to arrow | 1 | 10 |
You can also convert a `pyarrow.RecordBatch` back to serialized protobuf messages:
```python
array: pa.BinaryArray = handler.record_batch_to_array(record_batch)
messages_back: list[SearchRequest] = [
SearchRequest.FromString(s.as_py()) for s in array
]
```
## Configuration
Customize Arrow type mappings with `PtarsConfig`:
```python
from ptars import HandlerPool, PtarsConfig
config = PtarsConfig(
timestamp_unit="us", # microseconds instead of nanoseconds
timestamp_tz="America/New_York",
)
pool = HandlerPool([SearchRequest.DESCRIPTOR.file], config=config)
```
## Benchmark against protarrow
[Ptars](https://github.com/0x26res/ptars) is a rust implementation of
[protarrow](https://github.com/tradewelltech/protarrow),
which is implemented in plain python.
It is:
- 2.5 times faster when converting from proto to arrow.
- 3 times faster when converting from arrow to proto.
```benchmark
---- benchmark 'to_arrow': 2 tests ----
Name (time in ms) Mean
---------------------------------------
protarrow_to_arrow 9.4863 (2.63)
ptars_to_arrow 3.6009 (1.0)
---------------------------------------
---- benchmark 'to_proto': 2 tests -----
Name (time in ms) Mean
----------------------------------------
protarrow_to_proto 20.8297 (3.20)
ptars_to_proto 6.5013 (1.0)
----------------------------------------
```
[pypi-image]: https://img.shields.io/pypi/v/ptars
[pypi-url]: https://pypi.org/project/ptars/
[versions-image]: https://img.shields.io/pypi/pyversions/ptars
[versions-url]: https://pypi.org/project/ptars/
[wheel-image]: https://img.shields.io/pypi/wheel/ptars
[wheel-url]: https://pypi.org/project/ptars/
[docs-image]: https://readthedocs.org/projects/ptars/badge/?version=latest
[docs-url]: https://ptars.readthedocs.io/en/latest/
[downloads-image]: https://pepy.tech/badge/ptars
[downloads-url]: https://static.pepy.tech/badge/ptars
[downloads-month-image]: https://pepy.tech/badge/ptars/month
[downloads-month-url]: https://static.pepy.tech/badge/ptars/month
[build-image]: https://github.com/0x26res/ptars/actions/workflows/ci.yaml/badge.svg
[build-url]: https://github.com/0x26res/ptars/actions/workflows/ci.yaml
[codecov-image]: https://codecov.io/gh/0x26res/ptars/branch/main/graph/badge.svg?token=XMFH27IL70
[codecov-url]: https://codecov.io/gh/0x26res/ptars
[license-image]: http://img.shields.io/:license-Apache%202-blue.svg
[license-url]: https://github.com/0x26res/ptars/blob/master/LICENSE
[ruff-image]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
[ruff-url]: https://github.com/astral-sh/ruff
[snyk-image]: https://snyk.io/test/github/0x26res/ptars/badge.svg
[snyk-url]: https://snyk.io/advisor/python/ptars
[stars-image]: https://img.shields.io/github/stars/0x26res/ptars
[stars-url]: https://github.com/0x26res/ptars
[github-issues-image]: https://img.shields.io/badge/issue_tracking-github-blue.svg
[github-issues-url]: https://github.com/0x26res/ptars/issues
[contributing-image]: https://img.shields.io/badge/PR-Welcome-%23FF8300.svg?
[contributing-url]: https://github.com/0x26res/ptars/blob/main/DEVELOPMENT.md
[fossa-image]: https://app.fossa.com/api/projects/git%2Bgithub.com%2F0x26res%2Fptars.svg?type=shield
[fossa-url]: https://app.fossa.com/projects/git%2Bgithub.com%2F0x26res%2Fptars?ref=badge_shield
[repo-size-image]: https://img.shields.io/github/repo-size/0x26res/ptars
[repo-size-url]: https://img.shields.io/github/repo-size/0x26res/ptars
[crates-image]: https://img.shields.io/crates/v/ptars
[crates-url]: https://crates.io/crates/ptars
[crates-downloads-image]: https://img.shields.io/crates/d/ptars
[docsrs-image]: https://docs.rs/ptars/badge.svg
[docsrs-url]: https://docs.rs/ptars
[last-commit-image]: https://img.shields.io/github/last-commit/0x26res/ptars
[commits-url]: https://github.com/0x26res/ptars/commits/main
[commit-activity-image]: https://img.shields.io/github/commit-activity/m/0x26res/ptars
[open-prs-image]: https://img.shields.io/github/issues-pr/0x26res/ptars
[prs-url]: https://github.com/0x26res/ptars/pulls
[contributors-image]: https://img.shields.io/github/contributors/0x26res/ptars
[contributors-url]: https://github.com/0x26res/ptars/graphs/contributors
[release-image]: https://img.shields.io/github/v/release/0x26res/ptars
[release-date-image]: https://img.shields.io/github/release-date/0x26res/ptars
[release-url]: https://github.com/0x26res/ptars/releases
[rust-image]: https://img.shields.io/badge/rust-%23000000.svg?logo=rust&logoColor=white
[rust-url]: https://www.rust-lang.org/
[apache-arrow-image]: https://img.shields.io/badge/Apache%20Arrow-powered-orange.svg
[apache-arrow-url]: https://arrow.apache.org/
[prek-image]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/j178/prek/master/docs/assets/badge-v0.json
[prek-url]: https://github.com/j178/prek