{"id":31782825,"url":"https://github.com/hogwai/spring-boot-postgresql-copy","last_synced_at":"2026-05-06T00:38:27.872Z","repository":{"id":318234909,"uuid":"1069486829","full_name":"Hogwai/spring-boot-postgresql-copy","owner":"Hogwai","description":"Several ways to leverage the COPY command of PostgreSQL (WIP)","archived":false,"fork":false,"pushed_at":"2025-10-05T23:22:26.000Z","size":54,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-06T01:13:50.895Z","etag":null,"topics":["copy-command","java-25","pgbulkinsert","postgresql","springboot","sql"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Hogwai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-04T03:05:25.000Z","updated_at":"2025-10-05T23:24:13.000Z","dependencies_parsed_at":"2025-10-06T01:24:26.816Z","dependency_job_id":null,"html_url":"https://github.com/Hogwai/spring-boot-postgresql-copy","commit_stats":null,"previous_names":["hogwai/spring-boot-postgresql-copy"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Hogwai/spring-boot-postgresql-copy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hogwai%2Fspring-boot-postgresql-copy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hogwai%2Fspring-boot-postgresql-copy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hogwai%2Fspring-boot-postgresql-copy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hogwai%2Fspring-boot-postgresql-copy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Hogwai","download_url":"https://codeload.github.com/Hogwai/spring-boot-postgresql-copy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hogwai%2Fspring-boot-postgresql-copy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279003386,"owners_count":26083582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["copy-command","java-25","pgbulkinsert","postgresql","springboot","sql"],"created_at":"2025-10-10T09:56:50.077Z","updated_at":"2026-05-06T00:38:27.854Z","avatar_url":"https://github.com/Hogwai.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Using the PostgreSQL COPY command with Spring Boot\n\nThis repository demonstrates how to leverage the PostgreSQL `COPY` command for high-performance bulk inserts in a Spring\nBoot application.\n\nTwo approaches are compared against standard JPA `saveAll()`:\n\n| Strategy                   | 10 000 rows | Speedup  |\n|----------------------------|-------------|----------|\n| JPA `saveAll()`            | ~25s        | baseline |\n| **CopyManager** (custom)   | ~1.1s       | **x22**  |\n| **pgbulkinsert** (library) | ~1.6s       | **x15**  |\n\n## Getting started\n\n### Prerequisites\n\n- Java 25+\n- Docker\n\n### Run\n\n```bash\ndocker compose up -d\n./gradlew bootRun\n```\n\n### API\n\n```\nGET    /customers              # list all customers\nPOST   /customers/copy?customerNumber=10000   # insert via CopyManager\nPOST   /customers/bulk?customerNumber=10000   # insert via pgbulkinsert\n```\n\nBoth POST endpoints return an `InsertResult` with the row count and elapsed time:\n\n```json\n{\n  \"rowCount\": 10000,\n  \"elapsedSeconds\": 1.153\n}\n```\n\n### Tests\n\n```bash\ndocker compose up -d\n./gradlew test\n```\n\nUnit tests run without a database. Integration benchmarks (`InsertBenchmarkIT`) require the PostgreSQL container.\n\n---\n\n## Approach 1 — Custom CopyManager implementation\n\n### Architecture\n\nThe custom implementation is built around four components:\n\n**`CopyMapper\u003cT\u003e`** — defines the mapping contract for any entity:\n\n```java\npublic interface CopyMapper\u003cT\u003e {\n    String[] toCopyRow(T entity);\n\n    String tableName();\n\n    List\u003cString\u003e columns();\n}\n```\n\n**`CopyOptions`** — configurable COPY options (format, delimiter, null string, batch size):\n```java\nCopyOptions options = CopyOptions.builder()\n                                 .format(CopyOptions.Format.CSV)\n                                 .delimiter(\",\")\n                                 .batchSize(5000)\n                                 .build();\n```\n\n**`CopyInputStream\u003cT\u003e`** — streams rows on-the-fly without buffering the entire dataset in memory. Only the current row\nis held in memory at any time.\n\n**`CopyUtils`** — orchestrates the COPY operation:\n```java\n\n@Component\npublic class CopyUtils {\n\n    public \u003cT\u003e void insertWithCopy(CopyMapper\u003cT\u003e mapper, List\u003cT\u003e entities) { ...}\n\n    public \u003cT\u003e void insertWithCopy(CopyMapper\u003cT\u003e mapper, List\u003cT\u003e entities, CopyOptions options) { ...}\n}\n```\n\n### Key design decisions\n\n- **Spring transaction participation** — connections are obtained via `DataSourceUtils.getConnection()` so the COPY\n  operation participates in the current `@Transactional` context\n- **Streaming** — `CopyInputStream` feeds rows to the `CopyManager` one by one, avoiding loading the entire payload into\n  memory\n- **Batching** — when `batchSize \u003e 0`, entities are split into chunks and each chunk is sent in a separate COPY call\n- **SQL injection prevention** — table and column names are quoted with `\"` identifiers\n- **Escaping** — `escapeTextValue()` handles `\\`, `\\t`, `\\n`, `\\r`, `\\b`, `\\f` and null; `escapeCsvValue()` handles CSV\n  double-quote escaping\n\n### Usage example\n\n```java\nprivate static final CopyMapper\u003cCustomer\u003e CUSTOMER_MAPPER = new CopyMapper\u003c\u003e() {\n    @Override\n    public String[] toCopyRow(Customer customer) {\n        return new String[]{\n                String.valueOf(customer.getId()),\n                CopyUtils.escapeTextValue(customer.getFirstName()),\n                CopyUtils.escapeTextValue(customer.getLastName()),\n                customer.getCreationDate().toString()\n        };\n    }\n\n    @Override\n    public String tableName() {\n        return \"customer\";\n    }\n\n    @Override\n    public List\u003cString\u003e columns() {\n        return List.of(\"id\", \"first_name\", \"last_name\", \"creation_date\");\n    }\n};\n\n// Simple call with default options (TEXT format, tab delimiter)\ncopyUtils.\n\ninsertWithCopy(CUSTOMER_MAPPER, customers);\n\n// Or with custom options\nCopyOptions options = CopyOptions.builder()\n                                 .format(CopyOptions.Format.CSV)\n                                 .delimiter(\",\")\n                                 .batchSize(5000)\n                                 .build();\ncopyUtils.\n\ninsertWithCopy(CUSTOMER_MAPPER, customers, options);\n```\n\n---\n\n## Approach 2 — [PgBulkInsert](https://github.com/PgBulkInsert/PgBulkInsert) library\n\nA higher-level alternative using the `pgbulkinsert` library. Define a mapping:\n\n```java\npublic class CustomerMapping extends AbstractMapping\u003cCustomer\u003e {\n    public CustomerMapping() {\n        super(\"public\", \"customer\");\n        mapLong(\"id\", Customer::getId);\n        mapText(\"first_name\", Customer::getFirstName);\n        mapText(\"last_name\", Customer::getLastName);\n        mapText(\"country\", Customer::getCountry);\n        mapText(\"address\", Customer::getAddress);\n        mapText(\"city\", Customer::getCity);\n        mapTimeStamp(\"creation_date\", Customer::getCreationDate);\n        mapTimeStamp(\"update_date\", Customer::getUpdateDate);\n    }\n}\n```\n\nThen use it:\n\n```java\nPgBulkInsert\u003cCustomer\u003e bulkInsert = new PgBulkInsert\u003c\u003e(new CustomerMapping());\n\ntry(\nConnection conn = dataSource.getConnection()){\n        bulkInsert.\n\nsaveAll(PostgreSqlUtils.getPGConnection(conn),customers);\n        }\n```\n\n---\n\n## Project structure\n\n```\nsrc/main/java/.../\n├── controller/\n│   └── CustomerController.java          # REST endpoints\n├── model/\n│   ├── Customer.java                    # JPA entity\n│   └── CustomerMapping.java             # pgbulkinsert mapping\n├── repository/\n│   ├── CustomerRepository.java          # JPA + custom repos\n│   ├── CustomerCopyRepository.java      # CopyManager interface\n│   ├── PgbulkinsertCopyRepository.java  # pgbulkinsert interface\n│   ├── impl/\n│   │   ├── CustomerCopyRepositoryImpl.java\n│   │   └── PgbulkinsertCopyRepositoryImpl.java\n│   └── util/\n│       ├── CopyUtils.java               # COPY orchestration\n│       ├── CopyMapper.java              # Entity-to-row contract\n│       ├── CopyOptions.java             # COPY configuration\n│       ├── CopyInputStream.java         # Streaming InputStream\n│       └── CopyInsertException.java     # Dedicated exception\n├── service/\n│   ├── CustomerService.java\n│   ├── InsertResult.java                # Response DTO\n│   └── impl/\n│       └── CustomerServiceImpl.java\n└── util/\n    ├── CustomerFactory.java             # Test data generator\n    └── StringUtil.java\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhogwai%2Fspring-boot-postgresql-copy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhogwai%2Fspring-boot-postgresql-copy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhogwai%2Fspring-boot-postgresql-copy/lists"}