{"id":18960826,"url":"https://github.com/omniti-labs/pg_extractor","last_synced_at":"2025-08-20T08:33:20.916Z","repository":{"id":2036652,"uuid":"2973489","full_name":"omniti-labs/pg_extractor","owner":"omniti-labs","description":"PG Extractor - Advanced PostgreSQL Dump Filter","archived":false,"fork":false,"pushed_at":"2021-11-30T14:24:26.000Z","size":177,"stargazers_count":162,"open_issues_count":1,"forks_count":33,"subscribers_count":36,"default_branch":"master","last_synced_at":"2024-11-26T11:22:47.295Z","etag":null,"topics":["backup","postgresql","schemas"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/omniti-labs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.txt","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-12-13T16:20:59.000Z","updated_at":"2024-11-15T21:41:22.000Z","dependencies_parsed_at":"2022-08-29T02:31:46.857Z","dependency_job_id":null,"html_url":"https://github.com/omniti-labs/pg_extractor","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omniti-labs%2Fpg_extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omniti-labs%2Fpg_extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omniti-labs%2Fpg_extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omniti-labs%2Fpg_extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/omniti-labs","download_url":"https://codeload.github.com/omniti-labs/pg_extractor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230408171,"owners_count":18220974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup","postgresql","schemas"],"created_at":"2024-11-08T14:09:42.169Z","updated_at":"2024-12-19T09:07:58.622Z","avatar_url":"https://github.com/omniti-labs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"PGExtractor is both a command line script and python class that can be used to provide more finely \ndetailed filtering options for PostgreSQL's pg_dump program. Each database object is extracted into \nits own file organized into folders by type. This makes having the database schema as a reference \neasier and allows for better checking into version control. By default data is not dumped, but \nis easily done with a single option and can be done as plaintext or pg_dump's custom format. Object \nfiltering can be done directly via command line options or fed in with external text files. Regex \npattern matching is also possible.\n\nSee --help \u0026 --examples for a full list of available options and how to use the script.\n\nThe script only uses pg_dump/all to touch the database. \npg_restore is only used for generating ddl and does not ever touch the database.\n\nThis script natively requires Python 3. The 3to2 script can be used to allow it work work with\nPython 2.7, but it will not always be guarenteed to work. https://pypi.python.org/pypi/3to2\n\n````\n$ 3to2 -w pg_extractor.py\n````\n\nPython 3 was chosen for its more consistent treatment of plaintext and binary file formats. \nSince this is a text processing script, that consistency makes development easier and more \npredictable. Also, Python 3 has been out since 2008 and all major OS distributions have packages \navailable, so I'm doing my small part to help drive adoption to the new major version.\n\nSeveral of the class methods are public and can be used to inspect a custom format binary dump \nfile or apply some of the editing options.\n\n````\nPython 3.3.1 (default, Sep 25 2013, 19:29:01) \n[GCC 4.7.3] on linux\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n\u003e\u003e\u003e from pg_extractor import PGExtractor\n\u003e\u003e\u003e p = PGExtractor()\n\u003e\u003e\u003e object_list = p.build_main_object_list(\"dumpfile.pgr\")\n\u003e\u003e\u003e table_object_list = p.build_type_object_list(object_list, ['TABLE'])\n\u003e\u003e\u003e for t in table_object_list:\n...     print(t)\n... \n{'objname': 'job_detail_p0', 'objid': '238; 1259 596233', 'objowner': 'keith', 'objtype': 'TABLE', 'objschema': 'jobmon'}\n{'objname': 'job_detail_p10', 'objid': '239; 1259 596244', 'objowner': 'keith', 'objtype': 'TABLE', 'objschema': 'jobmon'}\n...\n````\n\nRemove the password hashes from an existing \"pg_dumpall -r\" roles file:\n````\n\u003e\u003e\u003e p.remove_passwords(\"pg_dumpall_roles.sql\")\n````\n\n### New Version 2.x\n\nVersion 2.x is a complete rewrite of PG Extractor in python. Most of the configuration options are the same,\nbut many have been changed for clarity, so please check the --help. \n\nNon-compatibilities with 1.x to be aware of when dropping in 2.x to replace it\n * Requires Python 3\n * The \"hostname\" is no longer a default part of the directory structure created. If this is still desired, set the --hostnamedir option with whatever the existing directory is.\n * Built in version control options are gone. They were rather fragile options and could easily lead to a whole lot of things getting checked into version control that should not have been. I've found it's easier (and safer) to manage version control check-ins separately. If these are really wanted please create an Issue on github and I'll consider it if there's enough interest.\n * Removed --rolesdir option\n\nNew features:\n * Full Python 3 class object with public methods that may possibly be useful on existing dump files\n * --jobs option to allow parallel object extraction\n * --remove_passwords option can remove the password hashes from an extracted roles file\n * --getdefaultprivs extracts the default privileges set for any roles that used ALTER DEFAULT PRIVILEGES\n * --delete cleans up empty folders properly\n * --wait option to allow a pause in object extraction. Helps reduce load when data is included in extraction.\n * --temp option to allow setting custom temporary working space\n * Sequences files can now include the statement to set the current value if data is output\n *  Better support for when objects have mixed case names or special characters. Special characters in an object name turn into ***,hexcode,*** to allow a valid system filename.\n * Rules \u0026 Triggers on views are now always included in the view file itself properly.\n\n*The version 1.x series written in perl will no longer be developed. Only bug fixes to the existing code will be accepted.*\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomniti-labs%2Fpg_extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fomniti-labs%2Fpg_extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomniti-labs%2Fpg_extractor/lists"}