Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dimitri/pgloader
Migrate to PostgreSQL in a single command!
https://github.com/dimitri/pgloader
clozure-cl common-lisp csv database dbase ixf migration mssql mysql pgloader postgresql sbcl sqlite
Last synced: 25 days ago
JSON representation
Migrate to PostgreSQL in a single command!
- Host: GitHub
- URL: https://github.com/dimitri/pgloader
- Owner: dimitri
- License: other
- Created: 2010-04-04T11:10:22.000Z (over 14 years ago)
- Default Branch: master
- Last Pushed: 2024-04-01T08:08:55.000Z (7 months ago)
- Last Synced: 2024-05-02T00:07:17.276Z (6 months ago)
- Topics: clozure-cl, common-lisp, csv, database, dbase, ixf, migration, mssql, mysql, pgloader, postgresql, sbcl, sqlite
- Language: Common Lisp
- Homepage: http://pgloader.io
- Size: 19.9 MB
- Stars: 5,076
- Watchers: 79
- Forks: 525
- Open Issues: 377
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
- awesome - pgloader - Migrate to PostgreSQL in a single command! (Common Lisp)
- jimsghstars - dimitri/pgloader - Migrate to PostgreSQL in a single command! (Common Lisp)
README
# PGLoader
[](https://travis-ci.org/dimitri/pgloader)
[](https://gitter.im/dimitri/pgloader?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
[](http://pgloader.readthedocs.io/en/latest/)
pgloader is a data loading tool for PostgreSQL, using the `COPY` command.
Its main advantage over just using `COPY` or `\copy`, and over using a
*Foreign Data Wrapper*, is its transaction behaviour, where *pgloader*
will keep a separate file of rejected data, but continue trying to
`copy` good data in your database.
The default PostgreSQL behaviour is transactional, which means that
*any* erroneous line in the input data (file or remote database) will
stop the entire bulk load for the table.
pgloader also implements data reformatting, a typical example of that
being the transformation of MySQL datestamps `0000-00-00` and
`0000-00-00 00:00:00` to PostgreSQL `NULL` value (because our calendar
never had a *year zero*).
## Documentation
Full documentation is available online, including manual pages of all the
pgloader sub-commands. Check out
[https://pgloader.readthedocs.io/](https://pgloader.readthedocs.io/en/latest/).
```
$ pgloader --help
pgloader [ option ... ] SOURCE TARGET
--help -h boolean Show usage and exit.
--version -V boolean Displays pgloader version and exit.
--quiet -q boolean Be quiet
--verbose -v boolean Be verbose
--debug -d boolean Display debug level information.
--client-min-messages string Filter logs seen at the console (default: "warning")
--log-min-messages string Filter logs seen in the logfile (default: "notice")
--summary -S string Filename where to copy the summary
--root-dir -D string Output root directory. (default: #P"/tmp/pgloader/")
--upgrade-config -U boolean Output the command(s) corresponding to .conf file for v2.x
--list-encodings -E boolean List pgloader known encodings and exit.
--logfile -L string Filename where to send the logs.
--load-lisp-file -l string Read user code from files
--dry-run boolean Only check database connections, don't load anything.
--on-error-stop boolean Refrain from handling errors properly.
--no-ssl-cert-verification boolean Instruct OpenSSL to bypass verifying certificates.
--context -C string Command Context Variables
--with string Load options
--set string PostgreSQL options
--field string Source file fields specification
--cast string Specific cast rules
--type string Force input source type
--encoding string Source expected encoding
--before string SQL script to run before loading the data
--after string SQL script to run after loading the data
--self-upgrade string Path to pgloader newer sources
--regress boolean Drive regression testing
```
## Usage
You can either give a command file to pgloader or run it all from the
command line, see the
[pgloader quick start](https://pgloader.readthedocs.io/en/latest/tutorial/tutorial.html#pgloader-quick-start) on
for more details.
$ ./build/bin/pgloader --help
$ ./build/bin/pgloader
For example, for a full migration from SQLite:
$ createdb newdb
$ pgloader ./test/sqlite/sqlite.db postgresql:///newdb
Or for a full migration from MySQL, including schema definition (tables,
indexes, foreign keys, comments) and parallel loading of the corrected data:
$ createdb pagila
$ pgloader mysql://user@localhost/sakila postgresql:///pagila
## LICENCE
pgloader is available under [The PostgreSQL
Licence](http://www.postgresql.org/about/licence/).
## INSTALL
Please see full documentation at
[https://pgloader.readthedocs.io/](https://pgloader.readthedocs.io/en/latest/install.html).
If you're using debian, it's already available:
$ apt-get install pgloader
If you're using docker, you can use the latest version built by the CI at
each commit to the master branch:
$ docker pull ghcr.io/dimitri/pgloader:latest
$ docker run --rm -it ghcr.io/dimitri/pgloader:latest pgloader --version