{"id":15043601,"url":"https://github.com/se7entyse7en/ohmyfpg","last_synced_at":"2025-06-12T17:36:00.344Z","repository":{"id":60864567,"uuid":"507608910","full_name":"se7entyse7en/ohmyfpg","owner":"se7entyse7en","description":"Oh My Fast Postgres!","archived":false,"fork":false,"pushed_at":"2023-02-04T00:35:16.000Z","size":362,"stargazers_count":11,"open_issues_count":3,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-14T23:12:46.789Z","etag":null,"topics":["numpy","pandas","postgresql","python","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/se7entyse7en.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-26T15:16:19.000Z","updated_at":"2023-07-19T12:35:26.000Z","dependencies_parsed_at":"2023-02-18T12:45:56.741Z","dependency_job_id":null,"html_url":"https://github.com/se7entyse7en/ohmyfpg","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"purl":"pkg:github/se7entyse7en/ohmyfpg","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se7entyse7en%2Fohmyfpg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se7entyse7en%2Fohmyfpg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se7entyse7en%2Fohmyfpg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se7entyse7en%2Fohmyfpg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/se7entyse7en","download_url":"https://codeload.github.com/se7entyse7en/ohmyfpg/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/se7entyse7en%2Fohmyfpg/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259512938,"owners_count":22869526,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["numpy","pandas","postgresql","python","rust"],"created_at":"2024-09-24T20:49:19.607Z","updated_at":"2025-06-12T17:36:00.321Z","avatar_url":"https://github.com/se7entyse7en.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Oh My Fast Postgres!\n\n`ohmyfpg` is a Postgres client library for Python that aims to return data as columns. This is often needed when working with numerical data. Usually this is achieved by taking the output of the client library and then convert it either into `numpy` arrays or `pandas` dataframes. When dealing with a large amount of data, this conversion is not much performant.\n\nThe goal of this library is to return data already as `numpy` arrays without sacrificing performance. The section of \"Performance comparison\" goes more in-depth on this topic.\n\nIn order to squeeze performance the underlying implementation is written in Rust. The Python layer on top is very thin.\n\n## Why `ohmyfpg`?\n\nWhen working with Postgres at work we faced multiple times performance issues. Most of the times our reactions were along the lines of: \"OMG\", \"F*****g PG\", etc. So `ohmyfpg` is kinda a mix of the two, but where the `f` now stands for `fast`.\n\n(To be fair, when we faced performance issues with Postgres was most of the times because of our inexperience with tuning the server configurations.)\n\n## Installation\n\n```\npip install ohmyfpg\n```\n\n## Quickstart\n\n```\nimport asyncio\nimport ohmyfpg\n\nDSN = 'postgres://postgres:postgres@postgres:5432/postgres'\nQUERY = 'SELECT * FROM performance_test'\n\nasync def main():\n    conn = await ohmyfpg.connect(DSN)\n    print(await conn.fetch(QUERY))\n\nif __name__ == '__main__':\n    asyncio.run(main())\n```\n\n## Performance comparison\n\nThe image below compares the performance of `ohmyfpg`, `asyncpg`, and `psycopg`. The 6 bars have the following meaning:\n- `ohmyfpg`: plain fetch,\n- `asyncpg`: plain fetch,\n- `psycopg`: plain fetch,\n- `ohmyfpg-pandas`: plain fetch + conversion to `pandas` Dataframe,\n- `asyncpg-pandas`: plain fetch + conversion to `pandas` Dataframe,\n- `psycopg-pandas`: plain fetch + conversion to `pandas` Dataframe,\n\nSee details [here](performance/compare.py), especially how the conversion to `pandas` Dataframe has been implemented.\n\nThe query is a `SELECT *` that has been run on a table with 1mln rows and the following schema:\n```\n(id INT, foo_bar_int2 INT2, foo_bar_int4 INT4, foo_bar_int8 INT8, foo_bar_float4 FLOAT4, foo_bar_float8 FLOAT8)\n```\n\nIt has been run inside docker with 8 CPU and 8GB of RAM allocated to the daemon on a MBP with 2.2 GHz 6-Core Intel Core i7 and 16GB 2400 MHz DDR4.\n\n![Performance comparison](performance/performance-comparison.png)\n\n### Detailed summary\n\n#### Plain fetch\n\n- `ohmyfpg` vs. `asyncpg` =\u003e **38.4%** (or **1.6x**) faster\n- `ohmyfpg` vs. `psycopg` =\u003e **58.0%** (or **2.4x**) faster\n\n```\n--------------------------------------------------\nohmyfpg\navg: 856.2ms\nmin: 747ms\np25: 782.25ms\nmedian: 819.0ms\np75: 871.0ms\nmax: 1335ms\n--------------------------------------------------\nasyncpg\navg: 1375.3ms\nmin: 1136ms\np25: 1261.75ms\nmedian: 1330.5ms\np75: 1406.25ms\nmax: 1925ms\n--------------------------------------------------\npsycopg\navg: 2023.7ms\nmin: 1777ms\np25: 1886.75ms\nmedian: 1951.0ms\np75: 2073.0ms\nmax: 3039ms\n--------------------------------------------------\n```\n\n#### Plain fetch + conversion to `pandas` Dataframe\n\n- `ohmyfpg-pandas` vs. `asyncpg-pandas` =\u003e **64.7%** (or **2.8x**) faster\n- `ohmyfpg-pandas` vs. `psycopg-pandas` =\u003e **69.8%** (or **3.3x**) faster\n\n```\n--------------------------------------------------\nohmyfpg-pandas\navg: 970.6666666666666ms\nmin: 852ms\np25: 924.25ms\nmedian: 948.0ms\np75: 988.5ms\nmax: 1292ms\n--------------------------------------------------\nasyncpg-pandas\navg: 2754.1666666666665ms\nmin: 2569ms\np25: 2642.75ms\nmedian: 2683.0ms\np75: 2737.75ms\nmax: 4044ms\n--------------------------------------------------\npsycopg-pandas\navg: 3193.6ms\nmin: 2945ms\np25: 3067.0ms\nmedian: 3141.0ms\np75: 3236.75ms\nmax: 4040ms\n--------------------------------------------------\n```\n\n## Limitations\n\nThis library is highly experimental and has many limitations:\n- no support for `NULL`s with unpredictable outcome,\n- no support for non-numerical types,\n- limited support for authentication,\n- no proper logging,\n- no support for insert operations,\n- no paremeters support for prepared statements,\n- etc.\n\n## Development\n\n\u003cdetails\u003e\n\u003csummary\u003eExpand\u003c/summary\u003e\n\n### How to run the performance comparison\n\n```\ndocker compose build script\ndocker compose up -d postgres\ndocker compose exec -- postgres psql -U postgres\n\nCREATE TABLE performance_test (id INT, foo_bar_int2 INT2, foo_bar_int4 INT4, foo_bar_int8 INT8, foo_bar_float4 FLOAT4, foo_bar_float8 FLOAT8);\nINSERT INTO performance_test (\n    id,\n    foo_bar_int2,\n    foo_bar_int4,\n    foo_bar_int8,\n    foo_bar_float4,\n    foo_bar_float8\n) VALUES (\n    generate_series(1, 1000000),\n    trunc(random() * (2*32768) - 32768),\n    trunc(random() * (2*2147483648) - 2147483648),\n    trunc(random() * (2*9223372036854775808) - 9223372036854775808),\n    trunc(random()),\n    trunc(random())\n);\n\n\ndocker compose up script\ndocker compose cp script:/usr/src/app/performance-comparison.png ./performance\n```\n\n### How to do basic benchmarking\n\n```\ndocker run -p 5432:5432 --name rust-postgres -e POSTGRES_PASSWORD=postgres -d postgres -c log_min_messages=DEBUG5\n```\n\nData preparation:\n```\nCREATE TABLE performance_test (id INT, foo_bar_int2 INT2, foo_bar_int4 INT4, foo_bar_int8 INT8, foo_bar_float4 FLOAT4, foo_bar_float8 FLOAT8);\nINSERT INTO performance_test (\n    id,\n    foo_bar_int2,\n    foo_bar_int4,\n    foo_bar_int8,\n    foo_bar_float4,\n    foo_bar_float8\n) VALUES (\n    generate_series(1, 1000000),\n    trunc(random() * (2*32768) - 32768),\n    trunc(random() * (2*2147483648) - 2147483648),\n    trunc(random() * (2*9223372036854775808) - 9223372036854775808),\n    trunc(random()),\n    trunc(random())\n);\n```\n\n```\nmaturin develop --release --manifest-path ohmyfpg/Cargo.toml\npython python/examples/simple_query.py\n```\n\n```\nRUST_BACKTRACE=1 cargo run -r -p ohmyfpg_core --example simple_query\n```\n\n### How to do basic profiling\n\n```\nsudo CARGO_PROFILE_BENCH_DEBUG=true RUST_BACKTRACE=1 cargo flamegraph -p ohmyfpg_core --example simple_query\n```\n\n```\nCARGO_PROFILE_BENCH_DEBUG=true RUST_BACKTRACE=1 cargo instruments --release -p ohmyfpg_core --example simple_query -t time\n```\n\n```\nCARGO_PROFILE_BENCH_DEBUG=true RUST_BACKTRACE=1 cargo instruments --release -p ohmyfpg_core --example simple_query -t Allocations\n```\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fse7entyse7en%2Fohmyfpg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fse7entyse7en%2Fohmyfpg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fse7entyse7en%2Fohmyfpg/lists"}