{"id":15297429,"url":"https://github.com/benchling/sqlalchemy_batch_inserts","last_synced_at":"2025-04-13T22:36:27.538Z","repository":{"id":55642084,"uuid":"161857514","full_name":"benchling/sqlalchemy_batch_inserts","owner":"benchling","description":"Batch inserts for psycopg2 on SQLAlchemy","archived":false,"fork":false,"pushed_at":"2020-12-16T02:46:28.000Z","size":15,"stargazers_count":32,"open_issues_count":1,"forks_count":6,"subscribers_count":46,"default_branch":"master","last_synced_at":"2025-03-27T12:52:34.547Z","etag":null,"topics":["psycopg2","python2","python3","sqlalchemy"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/sqlalchemy-batch-inserts/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benchling.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-15T01:14:00.000Z","updated_at":"2024-11-28T16:34:11.000Z","dependencies_parsed_at":"2022-08-15T05:20:22.509Z","dependency_job_id":null,"html_url":"https://github.com/benchling/sqlalchemy_batch_inserts","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchling%2Fsqlalchemy_batch_inserts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchling%2Fsqlalchemy_batch_inserts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchling%2Fsqlalchemy_batch_inserts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchling%2Fsqlalchemy_batch_inserts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benchling","download_url":"https://codeload.github.com/benchling/sqlalchemy_batch_inserts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248345263,"owners_count":21088242,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["psycopg2","python2","python3","sqlalchemy"],"created_at":"2024-09-30T19:17:27.898Z","updated_at":"2025-04-13T22:36:27.451Z","avatar_url":"https://github.com/benchling.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Benchling uses SQLAlchemy and psycopg2 to talk to PostgreSQL.\nTo save on round-trip latency, we batch our inserts using this code.\n\nTypically, creating and flushing N models in SQLAlchemy does N roundtrips to the database if the model has an autoincrementing primary key.\nThis module **improves creating N models to only require 2 roundtrips**, without requiring any other changes to your code.\n\nHere's our accompanying blog post that describes how we investigated and improved insert performance:\n[`sqlalchemy_batch_inserts`: a module for when you’re inserting thousands of rows and it’s slow](https://benchling.engineering/sqlalchemy-batch-inserts-a-module-for-when-youre-inserting-thousands-of-rows-and-it-s-slow-16ece0ef5bf7)\n\n## Is this for me?\nYou may find use for this module if:\n- You are using SQLAlchemy\n- You are using Postgres\n- You sometimes need to create several models at once and care about performance\n\n## Usage\n\n```python\nfrom sqlalchemy import create_engine\nfrom sqlalchemy.orm import sessionmaker\nfrom sqlalchemy_batch_inserts import enable_batch_inserting\n\nengine = create_engine(\"postgresql+psycopg2://postgres@localhost\", executemany_mode=\"values\")  # SQLAlchemy \u003c 1.3.7 needs use_batch_mode=True instead\nSession = sessionmaker(bind=engine)\nsession = Session()\nenable_batch_inserting(session)\n```\n\nIf you use [Flask-SQLAlchemy](https://flask-sqlalchemy.palletsprojects.com/),\n\n```python\nfrom flask_sqlalchemy import SignallingSession\nfrom sqlalchemy_batch_inserts import enable_batch_inserting\n\n# Make sure that you've specified executemany_mode or use_batch_mode when creating your engine! Otherwise\n# this library will not have any effect.\nenable_batch_inserting(SignallingSession)\n```\n\n## Demo\n\n```\ndocker build -t sqla_batch .\ndocker run --rm -v $PWD:/src --name sqla_batch sqla_batch\n# Wait for it to finish spinning up\n# Switch to another shell\ndocker exec -it sqla_batch src/demo.py no 100\ndocker exec -it sqla_batch src/demo.py yes 100\n```\n\nTo simulate 100 * 3 inserts with 20 ms latency,\nfirst change the connection string in demo.py from\n`postgresql+psycopg2://postgres@localhost` to `postgresql+psycopg2://postgres@db`.\nThen,\n```\ndocker network create sqla_batch\ndocker run --rm --network sqla_batch --network-alias db --name db sqla_batch\n# Switch to another shell\ndocker run -it -v /var/run/docker.sock:/var/run/docker.sock --network sqla_batch gaiaadm/pumba netem --duration 15m --tc-image gaiadocker/iproute2 delay --time 20 --jitter 0 db\n# Switch to another shell\n# This should take 100 * 3 * 20 ms = 6 seconds\ndocker run -it --rm -v $PWD:/src --network sqla_batch sqla_batch src/demo.py no 100\n# This should take 4 * 20 ms = 0.08 seconds\ndocker run -it --rm -v $PWD:/src --network sqla_batch sqla_batch src/demo.py yes 100\n```\n\n## Acknowledgements\n\nThis is all possible thanks to @dvarrazzo's psycopg2 [execute_batch](http://initd.org/psycopg/docs/extras.html#fast-execution-helpers)\nand @zzzeek's SQLAlchemy [support for the same](https://docs.sqlalchemy.org/en/13/dialects/postgresql.html#psycopg2-fast-execution-helpers)\nand [helpful](https://groups.google.com/forum/#!topic/sqlalchemy/GyAZTThJi2I)\n[advice](https://groups.google.com/forum/#!msg/sqlalchemy/l02TH_m1DkU/7PMlF8HzAgAJ) on the mailing list.\n\n## Maintainer notes\n\nAfter bumping the `version` in `setup.py` and `__version__` in `__init__.py`,\n\n```\n$ ./setup.py sdist\n$ twine upload --repository-url https://test.pypi.org/legacy/ dist/*\n# Check https://test.pypi.org/project/sqlalchemy-batch-inserts/\n$ twine upload dist/*\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenchling%2Fsqlalchemy_batch_inserts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenchling%2Fsqlalchemy_batch_inserts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenchling%2Fsqlalchemy_batch_inserts/lists"}