https://github.com/drorspei/arrow-csv-benchmark
Short benchmark for arrow's read_csv
https://github.com/drorspei/arrow-csv-benchmark
Last synced: 6 months ago
JSON representation
Short benchmark for arrow's read_csv
- Host: GitHub
- URL: https://github.com/drorspei/arrow-csv-benchmark
- Owner: drorspei
- License: apache-2.0
- Created: 2020-10-14T17:44:42.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2021-02-27T19:26:03.000Z (over 5 years ago)
- Last Synced: 2024-12-30T00:14:30.926Z (over 1 year ago)
- Language: Python
- Size: 143 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# arrow-csv-benchmark
Short benchmark for arrow's read_csv
# Why
I made this repo after experiencing low read speeds (0.5GiB/s) on real work csvs.
# What this does
1. Generates a big csv with many string, float, and null columns, using joblib for parallelization,
2. Puts the csv into a `BytesIO` object,
3. Calls `pyarrow.csv.read_csv` a few times on the csv bytes.
The Dockerfile sets up a minimal container for running the benchmark.
# My Results
Running this on Azure, machine size `Standard E48s_v3 (48 vcpus, 384 GiB memory)`, on Linux (ubuntu 18.04), unused other than this benchmark, consistently shows speeds of less than 1GiB/s, and often below 0.5GiB/s.
Included in the repo are profiling dumps, made manually with py-spy. I started them 5 seconds after the beginning of each `read_csv`, and stopped them after about 15 seconds. This was always more than 5 seconds before the `read_csv` finished.
If the profiles are to be trusted, there is considerable time spent in the shared pointer's lock mechanisms. As for the reading of the bytes, I'm not sure what goes into this or why it takes time.