{"id":18547647,"url":"https://github.com/vincentlaucsb/pgreaper","last_synced_at":"2025-09-09T15:41:24.061Z","repository":{"id":57452177,"uuid":"87692461","full_name":"vincentlaucsb/pgreaper","owner":"vincentlaucsb","description":"A Python library for loading data from various formats into PostgreSQL databases.","archived":false,"fork":false,"pushed_at":"2017-10-18T23:05:22.000Z","size":37359,"stargazers_count":12,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T11:21:32.276Z","etag":null,"topics":["convert-data","csv-converter","html-parser","postgresql","python","sql","sql-database","sql-table","sqlite3-database"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vincentlaucsb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-04-09T08:00:34.000Z","updated_at":"2022-12-26T13:44:29.000Z","dependencies_parsed_at":"2022-09-02T06:43:52.411Z","dependency_job_id":null,"html_url":"https://github.com/vincentlaucsb/pgreaper","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentlaucsb%2Fpgreaper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentlaucsb%2Fpgreaper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentlaucsb%2Fpgreaper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vincentlaucsb%2Fpgreaper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vincentlaucsb","download_url":"https://codeload.github.com/vincentlaucsb/pgreaper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247744328,"owners_count":20988784,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convert-data","csv-converter","html-parser","postgresql","python","sql","sql-database","sql-table","sqlite3-database"],"created_at":"2024-11-06T20:30:22.335Z","updated_at":"2025-04-09T21:31:49.308Z","avatar_url":"https://github.com/vincentlaucsb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgreaper\n[![Build Status](https://travis-ci.org/vincentlaucsb/pgreaper.svg?branch=master)](https://travis-ci.org/vincentlaucsb/pgreaper)\n[![Coverage Status](https://coveralls.io/repos/github/vincentlaucsb/pgreaper/badge.svg?branch=master)](https://coveralls.io/github/vincentlaucsb/pgreaper?branch=master)\n\nPGReaper is the easy Pythonic way to upload data from CSV, JSON, HTML, and\nSQLite sources to PostgreSQL databases. Interally, it uses the fast `COPY`\nstreaming protocol, but wraps it in a way that makes it more flexible, robust and\neasier to use. Features include:\n * Automatic schema inference\n * Ability to parse and normalize `\u003ctable\u003e`s in HTML\n * Ability to flatten and/or extract nested keys from JSON before copying\n * Ability to copy files in `.zip` archives without decompressing them\n * Ability to copy over networks\n \nLastly, but certainly not least, PGReaper ships with its own `Table` data structure\nwhile also being able to copy `pandas` DataFrames. These allows programmatic creation\nor updating of SQL tables without the verbosity associated with traditional ORMs.\nFurthermore, when uploading Python data structures, PGReaper automatically infers the schema,\nincluding for `JSONB` (from dict or list) and `timestamp` (from datetime) objects.\n\n## Benchmarks\nSpeed is one of `pgreaper`'s main design goals. A list of benchmarks may be found under the `benchmarks` subdirectory.\n\n## Installation\nI've been using PGReaper heavily for my own projects, such as Twitter and web scraping, but I have only recently started polishing up the documentation and API for public consumption. PGReaper will be released on PyPI when I feel it is mature enough, but if you would like to use it now, you can clone this repository and run the following command where the files are extracted.\n\n```pip install .```\n\nCurrently, you may need Cython to build this project.\n\n### Dependencies\n`PGReaper` requires a minimum of dependencies. Namely, these are Python 3.5+, `psycopg2`, and obviously PostgreSQL (use 9.3+ for JSON features).\n\n## Documentation\nFull documentation (or at least that's the goal!) is [available here](http://vincela.com/pgreaper/).\nAs a user of software, I frequently get frustrated with inconsistent, inaccurate\n or unclear documentation. However, as a writer of software I also realize\n writing documentation isn't always fun and what is obvious to you isn't always \n obvious to others. If you see something that needs\n improvement, feel free to submit an issue or pull request.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvincentlaucsb%2Fpgreaper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvincentlaucsb%2Fpgreaper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvincentlaucsb%2Fpgreaper/lists"}