{"id":19707912,"url":"https://github.com/nyov/postgres-bindata-batchimport","last_synced_at":"2026-06-12T19:32:19.684Z","repository":{"id":144730723,"uuid":"223866221","full_name":"nyov/postgres-bindata-batchimport","owner":"nyov","description":"PostgreSQL batch-importing binary files, from disk to database table. Using the Large Objects interface.","archived":false,"fork":false,"pushed_at":"2019-11-25T13:52:58.000Z","size":5,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-27T18:19:55.116Z","etag":null,"topics":["batch-processing","command-line","data-import","large-object","postgresql","psql","shell-script"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nyov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-25T05:12:15.000Z","updated_at":"2019-11-25T13:53:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"d621624e-e0dd-4b17-ace6-3d3661bb1823","html_url":"https://github.com/nyov/postgres-bindata-batchimport","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nyov/postgres-bindata-batchimport","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyov%2Fpostgres-bindata-batchimport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyov%2Fpostgres-bindata-batchimport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyov%2Fpostgres-bindata-batchimport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyov%2Fpostgres-bindata-batchimport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nyov","download_url":"https://codeload.github.com/nyov/postgres-bindata-batchimport/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nyov%2Fpostgres-bindata-batchimport/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34260309,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","command-line","data-import","large-object","postgresql","psql","shell-script"],"created_at":"2024-11-11T21:40:19.793Z","updated_at":"2026-06-12T19:32:19.659Z","avatar_url":"https://github.com/nyov.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"postgres-bindata-batchimport\n----------------------------\n\nPostgreSQL batch-importing binary files, from disk to database table.\nUsing the [Large Objects][1] interface.\n(PostgreSQL 9.3+)\n\nThe shell scripts `pg-*-batchimport-cs.sh` and `pg-*-batchimport-ss.sh`\nare very similar.\nThe former (*client-side*) can be used with a remote database cluster and only\nuses `psql`-local functions, while the latter (*server-side*) executes commands\non the db cluster and thus requires cluster-local file access (read permission)\nby the postgres user.\n\nThe server-local variant is on average 50% faster than the client-side (remote)\nversion (even with a local cluster), since the client-side code must execute a\nlarge-object import and `UPDATE` statement for each file.\nWith a remote cluster, bandwidth limitations for the file uploads can further\nimpact the execution speed.\n\n\n##### Usage #####\n\nImporting files as Large Objects only:\n```sh\n./pg-lo-batchimport-cs.sh    \u003cdatabasename\u003e \u003csource-directory\u003e [\u003ctablename\u003e]\n./pg-lo-batchimport-ss.sh    \u003cdatabasename\u003e \u003csource-directory\u003e [\u003ctablename\u003e]\n```\nUsing large objects gives all the gimmicks that the [Large Objects][1]\nfile-like interface supports (such as seeking). Reference them by the table's\n`oid` column.\n\nConverting files into BYTEA table rows after import:\n```sh\n./pg-bytea-batchimport-cs.sh \u003cdatabasename\u003e \u003csource-directory\u003e [\u003ctablename\u003e]\n./pg-bytea-batchimport-ss.sh \u003cdatabasename\u003e \u003csource-directory\u003e [\u003ctablename\u003e]\n```\nThe `source-directory` will be walked recursively, using `find`, by default.\n`find` supports many modifier options to grab files selectively, e.g. limited\nto depth level, or files newer than `$last-run` minutes, for recurring jobs.\n\n##### Example #####\n\n```sh\n#!/bin/sh\n\n# Generate 20 files with some text, or use your own:\nmkdir /tmp/randomtext\nfor OF in `seq 101 120`; do curl -s http://metaphorpsum.com/paragraphs/5/5 \u003e /tmp/randomtext/${OF}; done\n\n# DB to use, possibly replace 'postgres' with an existing test-database\nDB=\"postgres\"\nTABLE=\"tmp_docs\"\n\n# Run script\n./pg-bytea-batchimport-cs.sh $DB /tmp/randomtext $TABLE\n# (or) Server-side version; faster, but requires a local db cluster\n#./pg-bytea-batchimport-ss.sh $DB /tmp/randomtext $TABLE\n\n# Verify contents in db; should count 20 rows\npsql -c \"SELECT count(*) FROM $TABLE\" $DB\npsql -c \"SELECT file_name, encode(doc::bytea, 'escape')::text AS data FROM $TABLE LIMIT 1\" $DB\n\n# clean up\nrm -r /tmp/randomtext\npsql -c \"DROP TABLE $TABLE\" $DB\nunset TABLE\nunset DB\n```\n(This code snippet can be pasted into a file and executed as a shell script,\n as is.)\n\n##### Notes \u0026 Caveats #####\n\nIf the script aborts before the created large objects have been cleaned up\nand the table is then dropped without unlinking them, orphaned objects stay\naround. The `\\lo_list` (local to database) command can help finding them.\nIf the current DB contains no valid objects beside the failed import, the\nnuke-em command is `SELECT lo_unlink(oid) FROM pg_largeobject_metadata;`.\n\nBecause the workflow used in the `pg-bytea-batchimport-*.sh` scripts makes a\nround-trip through large-objects-storage land, the DB cluster may use up to\ndouble the space of the source file directory, for the runtime duration of the\nscript.\n\nDepending on the type of files, this may not be true, however; with\ntext-files I've seen a compression ratio of 2 GiB raw on-disk to 400 MiB\nin-db of the same data. Which means even with two copies, the database was\nusing less storage than the source directory data.\n\n---\n\n![Public Domain Icon](https://upload.wikimedia.org/wikipedia/commons/thumb/6/62/PD-icon.svg/16px-PD-icon.svg.png)\nPublic Domain. No warranties of any kind, express or implied.\n\nFurther information and alternative methods to handle data importing in\nPostgreSQL can be found in the PDF presentation\n[PGOpen2018_data_loading.pdf][2], which inspired this code; and on\n[StackOverflow][3].\n\n  [1]: https://www.postgresql.org/docs/current/largeobjects.html\n  [2]: https://www.postgis.us/presentations/PGOpen2018_data_loading.pdf\n  [3]: https://dba.stackexchange.com/q/253425\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnyov%2Fpostgres-bindata-batchimport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnyov%2Fpostgres-bindata-batchimport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnyov%2Fpostgres-bindata-batchimport/lists"}