{"id":20264887,"url":"https://github.com/gavincabbage/chiv","last_synced_at":"2025-04-11T02:31:58.817Z","repository":{"id":57494824,"uuid":"176830666","full_name":"gavincabbage/chiv","owner":"gavincabbage","description":"gavincabbage.com/chiv | Archive relational database tables to Amazon S3","archived":false,"fork":false,"pushed_at":"2019-10-28T21:52:44.000Z","size":311,"stargazers_count":8,"open_issues_count":5,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T23:51:32.603Z","etag":null,"topics":["archive","database","s3","sql"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gavincabbage.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-20T23:07:55.000Z","updated_at":"2024-06-05T23:06:42.000Z","dependencies_parsed_at":"2022-08-28T19:40:28.131Z","dependency_job_id":null,"html_url":"https://github.com/gavincabbage/chiv","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gavincabbage%2Fchiv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gavincabbage%2Fchiv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gavincabbage%2Fchiv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gavincabbage%2Fchiv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gavincabbage","download_url":"https://codeload.github.com/gavincabbage/chiv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248329631,"owners_count":21085570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive","database","s3","sql"],"created_at":"2024-11-14T11:45:01.164Z","updated_at":"2025-04-11T02:31:58.780Z","avatar_url":"https://github.com/gavincabbage.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Title](img/chiv.png)\n\n---\n\n\u003cp align=\"center\"\u003e\n    Archive relational data to Amazon S3.\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://github.com/gavincabbage/chiv/actions?workflow=build\"\u003e\n        \u003cimg src=\"https://github.com/gavincabbage/chiv/workflows/build/badge.svg\" alt=\"build\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://goreportcard.com/report/gavincabbage.com/chiv\"\u003e\n        \u003cimg src=\"https://goreportcard.com/badge/gavincabbage.com/chiv\" alt=\"go report\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://godoc.org/gavincabbage.com/chiv\"\u003e\n        \u003cimg src=\"https://godoc.org/gavincabbage.com/chiv?status.svg\" alt=\"godoc\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://gavincabbage.com/chiv/blob/master/LICENSE\"\u003e\n        \u003cimg src=\"http://img.shields.io/badge/License-MIT-blue.svg\" alt=\"license\" /\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\n---\n\nThis project provides the `chiv` package and a simple CLI wrapper of the same name. It requires Go 1.13.\n\nUse `chiv` to download relational data from a database and format it before uploading to Amazon S3.\nBuilt-in formats include CSV, YAML and JSON. \nCustom formats are also supported through the `Formatter` interface.\n\n# Getting Started\n\n```go\nimport \"gavincabbage.com/chiv\"\n```\n\nProvide a database and upload manager to upload a table to an S3 bucket in the default CSV format.\n\n```go\ndb, _ := sql.Open(config.driver, config.url)\n\nclient := s3.New(session.NewSessionWithOptions(session.Options{}))\nuploader := s3manager.NewUploaderWithClient(client)\n\nchiv.Archive(db, uploader, \"table\", \"bucket\")\n``` \n\nUse [options](https://github.com/gavincabbage/chiv/blob/master/options.go) to configure the archival format,\nupload key, null placeholder, etc.\n\n```go\nchiv.Archive(db, uploader, \"table\", \"bucket\"\n    chiv.WithFormat(chiv.JSON),\n    chiv.WithKey(\"2019/september/monthly_archive.json\"),\n    chiv.WithNull(\"empty\"),\n)\n```\n\nFor multiple uploads using the same database and S3 clients, construct an `Archiver`. Options provided during\nconstruction of an `Archiver` can be overridden in individual archival calls.\n\n```go\na := chiv.NewArchiver(db, uploader, chiv.WithFormat(chiv.YAML))\na.Archive(\"first_table\", \"bucket\")\na.Archive(\"second_table\", \"bucket\")\na.Archive(\"second_table\", \"bucket\", chiv.WithFormat(chiv.JSON), chiv.WithKey(\"second_table.json\"))\n``` \n\nCustom queries can be archived using the `ArchiveRows` family of functions.\n\n```go\nrows, _ := db.Exec(\"SELECT * FROM table and JOIN all the things...\")\n\nchiv.ArchiveRows(rows, uploader, \"bucket\")\n``` \n\nContext-aware versions are also provided, e.g. `ArchiveWithContext`, `ArchiveRowsWithContext`, etc.\n\nSee the [unit](https://github.com/gavincabbage/chiv/blob/master/chiv_test.go)\nand [integration](https://github.com/gavincabbage/chiv/blob/master/chiv_integration_test.go)\ntests for additional examples.\n\n# Custom Formats\n\nCustom formats can be used by implementing the `FormatterFunc` and `Formatter` interfaces.\nThe optional `Extensioner` interface can be implemented to allow a `Formatter` to provide a default extension.\n\nSee the three [built-in formats](https://github.com/gavincabbage/chiv/blob/master/chiv_formatters.go)\nfor examples.\n\n# CLI\n\nA simple CLI wrapping the package is also included.\n\n```text\nNAME:\n   chiv - Archive relational data to Amazon S3\n\nUSAGE:\n   chiv [flags...]\n\nVERSION:\n   vX.Y.Z\n\nGLOBAL OPTIONS:\n   --database value, -d value   database connection string [$DATABASE_URL]\n   --table value, -t value      database table to archive\n   --bucket value, -b value     upload S3 bucket name\n   --driver value, -r value     database driver type: postgres or mysql (default: \"postgres\")\n   --columns value, -c value    database columns to archive, comma-separated\n   --format value, -f value     upload format: csv, yaml or json (default: \"csv\")\n   --key value, -k value        upload key\n   --extension value, -e value  upload extension\n   --null value, -n value       upload null value\n   --help, -h                   show usage details\n   --version, -v                print the version\n```\n\n# Design\n\nThis package ties together three components of what is essentially an ETL operation:\n\n- **Extract** database rows with a `Database` or `Rows`\n  - `Database` may be a `*sql.DB`, `*sql.Conn`, `*sql.Tx`, etc.\n- **Transform** data into an upload format with a `Formatter`\n- **Load** the formatted data into S3 with a `Uploader`\n  - `Uploader` is typically an `*s3manager.Uploader`\n  \nAn `io.Pipe` is used to guarantee a cap on total memory usage by `chiv` itself.\nA [benchmark](https://github.com/gavincabbage/chiv/blob/master/chiv_benchmark_test.go)\nis provided to profile the package's memory usage.\nTotal memory usage of the archival process can be further controlled by \nconfiguring the S3 upload manager.\n\n![Architecture](img/arch.png)\n\n# Testing \u0026 Development\n\nTests are segregated by build tags.\n\nRun unit tests:\n\n```bash\ngo test -tags=unit ./...\n```\n\nUse docker-compose to run the full suite of unit and integration tests:\n\n```bash\ndocker-compose up --exit-code-from test --build\n```\n\nRun benchmarks:\n\n```bash\ngo test -v -p 1 -tags=benchmark,unit -run=Benchmark -benchmem -bench=.\n```\n\nThis project uses GitHub Actions for CI. \nLinting, unit tests, integration tests and benchmarks are run for each push or pull request.\n\nTagged releases are built automatically by [GoReleaser](https://goreleaser.com/).\n\n# Contributing\n\nContributions in the form of comments, issues or pull requests are very welcome.\n\nIf you use this package I would love to hear from you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgavincabbage%2Fchiv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgavincabbage%2Fchiv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgavincabbage%2Fchiv/lists"}