{"id":19096607,"url":"https://github.com/sapcc/swift-http-import","last_synced_at":"2025-04-30T14:13:17.791Z","repository":{"id":38420131,"uuid":"76873293","full_name":"sapcc/swift-http-import","owner":"sapcc","description":"Mirror files from an HTTP server into OpenStack Swift (or from one Swift account to another)","archived":false,"fork":false,"pushed_at":"2025-04-11T01:34:49.000Z","size":8092,"stargazers_count":11,"open_issues_count":7,"forks_count":3,"subscribers_count":53,"default_branch":"master","last_synced_at":"2025-04-11T02:38:42.001Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sapcc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-12-19T15:25:21.000Z","updated_at":"2025-04-04T09:28:40.000Z","dependencies_parsed_at":"2023-02-18T05:16:04.095Z","dependency_job_id":"5de0b15d-4af9-4729-8f84-467071b1fc63","html_url":"https://github.com/sapcc/swift-http-import","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fswift-http-import","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fswift-http-import/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fswift-http-import/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sapcc%2Fswift-http-import/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sapcc","download_url":"https://codeload.github.com/sapcc/swift-http-import/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249514453,"owners_count":21284548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T03:37:13.199Z","updated_at":"2025-04-18T15:33:51.206Z","avatar_url":"https://github.com/sapcc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# swift-http-import\n\n* [Why this instead of rclone?](#why-this-instead-of-rclone)\n* [Do NOT use if\\.\\.\\.](#do-not-use-if)\n* [Implicit assumptions](#implicit-assumptions)\n* [Installation](#installation)\n* [Usage](#usage)\n  * [Specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables)\n  * [Alternative authentication options](#alternative-authentication-options)\n  * [Source specification](#source-specification)\n    * [Yum](#yum)\n    * [Debian](#debian)\n    * [Github Releases](#github-releases)\n    * [Swift](#swift)\n  * [File selection](#file-selection)\n    * [By name](#by-name)\n    * [By age](#by-age)\n    * [Simplistic file comparison](#simplistic-file-comparison)\n  * [Transfer behavior: Segmenting on the source side](#transfer-behavior-segmenting-on-the-source-side)\n  * [Transfer behavior: Segmenting on the target side](#transfer-behavior-segmenting-on-the-target-side)\n  * [Transfer behavior: Expiring objects](#transfer-behavior-expiring-objects)\n  * [Transfer behavior: Symlinks](#transfer-behavior-symlinks)\n  * [Transfer behavior: Delete objects on the target side](#transfer-behavior-delete-objects-on-the-target-side)\n  * [Performance](#performance)\n* [Log output](#log-output)\n* [StatsD metrics](#statsd-metrics)\n* [GPG keyserver selection](#gpg-keyserver-selection)\n\nThis tool imports files from an HTTP server into a Swift container. Given an input URL, it recurses through the directory\nlistings on that URL, and mirrors all files that it finds into Swift. It will take advantage of `Last-Modified` and\n`Etag` response headers to avoid repeated downloads of the same content, so best performance is ensured if the HTTP server\nhandles `If-Modified-Since` and `If-None-Match` request headers correctly.\n\n## Why this instead of rclone?\n\n[rclone](https://github.com/ncw/rclone/) is a similar tool that supports a much wider range of sources, and supports\ntargets other than Swift. Users who need to sync from or to different clouds might therefore prefer rclone. If your only\ntarget is Swift, swift-http-import is better than rclone because it is built by people who operate Swift clusters for a\nliving and know all the intricacies of the system, and how to optimize Swift clients for maximum performance and\nstability:\n\n* rclone does not support Swift symlinks. Symlinks will be copied as regular files, thereby potentially wasting space on\n  the target storage. swift-http-import recognizes symlinks and copies them as a symlink.\n* rclone transfers large objects using the Dynamic Large Object strategy. DLO suffers from severe eventual-consistency\n  problems when the Swift cluster is under high load. swift-http-import uses the much more resilient Static Large Object\n  strategy instead.\n* rclone does not propagate expiration dates when transferring between Swift containers. swift-http-import does.\n* swift-http-import uses customized transfer strategies to ensure a stable transfer over narrow or flaky network\n  connections.\n\n## Do NOT use if...\n\n* ...you have access to the source filesystem. Just use the normal [`swift upload`](https://docs.openstack.org/python-swiftclient/latest/) instead, it's much more efficient.\n* ...you need to import a lot of small files exactly once. Download them all and pack them into a tarball, and send them to Swift in one step with a [bulk upload](https://www.swiftstack.com/docs/admin/middleware/bulk.html#uploading-archives).\n\n## Implicit assumptions\n\nThe HTTP server must present the contents of directories using standard directory listings, as can be generated\nautomatically by many HTTP servers. In detail, that means that `GET` on a directory should return an HTML page that\n\n- links to all files in this directory with an `\u003ca\u003e` tag whose `href` is just the filename (i.e. relative, not absolute URL)\n- links to all directories below this directory with an `\u003ca\u003e` tag whose `href` is just the directory name plus\n  a trailing slash (i.e. relative, not absolute URL)\n\nAbsolute URLs containing a protocol and domain are ignored, as are relative URLs containing `..` path elements.\n\n## Installation\n\nTo build the binary:\n\n```bash\nmake\n```\n\nThe binary can also be installed with `go get`:\n```bash\ngo get github.com/sapcc/swift-http-import\n```\n\nOr just grab a pre-compiled binary from the [release list](https://github.com/sapcc/swift-http-import/releases).\n\nTo build the Docker container:\n\n```bash\ndocker build .\n```\n\n## Usage\n\nCall with the path to a configuration file. The file should look like this:\n[(Link to full example config file)](./examples/basic.yaml)\n\n```yaml\nswift:\n  auth_url: https://my.keystone.local:5000/v3\n  user_name: uploader\n  user_domain_name: Default\n  project_name: datastore\n  project_domain_name: Default\n  password: 20g82rzg235oughq\n\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n```\n\nThe first paragraph contains the authentication parameters for\nOpenStack's Identity v3 API. Optionally a `region_name` can be specified, but this is only\nrequired if there are multiple regions to choose from. You can also specify the `tls_client_certificate_file` and `tls_client_key_file` for creating a TLS client.\n\nYou can use the `fromEnv` special syntax for the `to.container`, `to.object_prefix`, and\nthe Swift fields (options under the `swift` key).\nSee [specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables) for more details.\n\nEach sync job contains the source URL as `from.url`, and `to.container` has the target container name, optionally paired with an\nobject name prefix in the target container. For example, in the case above, the file\n\n```\nhttp://de.archive.ubuntu.com/ubuntu/pool/main/p/pam/pam_1.1.8.orig.tar.gz\n```\n\nwould be synced to the `mirror` container as\n\n```\nubuntu-repos/pool/main/p/pam/pam_1.1.8.orig.tar.gz\n```\n\nThe order of jobs is significant: Source trees will be scraped in the order indicated by the `jobs` list.\n\n### Specifying sensitive info as environment variables\n\nFor some config fields, instead of specifying the value as plain text, you can use the\nspecial `fromEnv` syntax to read the respective value from an exported environment\nvariable.\n\nFor example, instead of specifying the `swift.password` as plain text, you can use the\nfollowing syntax to retrieve the password from the `OS_PASSWORD` environment variable:\n\n```yaml\nswift:\n  password: { fromEnv: OS_PASSWORD }\n  ...\n```\n\n### Alternative authentication options\n\nInstead of password-based authentication, [application credentials][app-cred] can also be used, for example:\n[(Link to full example config file)](./examples/application-credential-auth.yaml)\n```yaml\nswift:\n  auth_url: https://my.keystone.local:5000/v3\n  application_credential_id: 80e810bf385949ae8f0e251f90269515\n  application_credential_secret: eixohMoo1on5ohng\n\njobs: ...\n```\n\n[app-cred]: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/application-credentials.html\n\n\n### Source specification\n\nIn `jobs[].from`, you can pin the server's CA certificate, and specify a TLS client certificate (including private key)\nthat will be used by the HTTP client.\n[(Link to full example config file)](./examples/source-clientcert.yaml)\n\n```yaml\njobs:\n  - from:\n      url:  http://de.archive.ubuntu.com/ubuntu/\n      cert: /path/to/client.pem\n      key:  /path/to/client-key.pem\n      ca:   /path/to/server-ca.pem\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n```\n\n#### Yum\n\nIf `jobs[].from.url` refers to a Yum repository (as used by most RPM-based Linux distributions), setting\n`jobs[].from.type` to `yum` will cause `swift-http-import` to parse repository metadata to discover files to transfer,\ninstead of looking at directory listings.\n\n*Warning:* With this option set, files below the given URL which are not\nreferenced by the Yum repository metadata will **not** be picked up.\nConversely, files mentioned in the repo metadata that don't exist in the repo\nwill result in `404` errors.\n\nIf the optional `jobs[].from.arch` field is given, the Yum repository metadata reader will only consider packages for\nthese architectures. Special values include \"noarch\" for architecture-independent packages and \"src\" for source\npackages.\n\nThe GPG signature for the repository's metadata file is verified by default and\nthe job will be skipped if the verification is unsuccessful. This behavior can be disabled by\nspecifying the `jobs[].from.verify_signature` option. See [\"GPG keyserver selection\"](#gpg-keyserver-selection) for how\nto control how `swift-http-import` retrieves the required public keys for signature verification.\n\n[Link to full example config file](./examples/source-yum.yaml)\n\n```yaml\njobs:\n  - from:\n      url:  https://dl.fedoraproject.org/pub/epel/7Server/x86_64/\n      type: yum\n      arch: [x86_64, noarch]\n      verify_signature: true\n      # SSL certs are optionally supported here, too\n      cert: /path/to/client.pem\n      key:  /path/to/client-key.pem\n      ca:   /path/to/server-ca.pem\n    to:\n      container: mirror\n      object_prefix: redhat/server/7/epel\n```\n\n#### Debian\n\nIf `jobs[].from.url` refers to a Debian repository (or an Ubuntu repository),\nsetting `jobs[].from.type` to `debian` will cause `swift-http-import` to parse\npackage and source metadata files to discover which respective files to\ntransfer, instead of looking at directory listings.\n\nIf the optional `jobs[].from.arch` field is given, the Debian repository\nmetadata reader will only consider package and source files for these\narchitectures.\n\n*Warning:* If `arch` field is omitted then files mentioned in the repository\nmetadata that don't actually exist in the repository will result in `404`\nerrors.\n\nThe GPG signature for the repository's metadata file is verified by default and\nthe job will be skipped if the verification is unsuccessful. This behavior can be disabled by\nspecifying the `jobs[].from.verify_signature` option. See [\"GPG keyserver selection\"](#gpg-keyserver-selection) for how\nto control how `swift-http-import` retrieves the required public keys for signature verification.\n\n [Link to full example config file](./examples/source-debian.yaml)\n\n```yaml\njobs:\n  - from:\n      url:  http://de.archive.ubuntu.com/ubuntu/\n      type: debian\n      dist: [xenial, xenial-updates, disco, cosmic]\n      arch: [amd64, i386]\n      verify_signature: true\n      # SSL certs are optionally supported here, too\n      cert: /path/to/client.pem\n      key:  /path/to/client-key.pem\n      ca:   /path/to/server-ca.pem\n    to:\n      container: mirror\n      object_prefix: ubuntu\n```\n\n#### Github Releases\n\nIf `jobs[].from.url` refers to a GitHub repository, setting `jobs[].from.type` to\n`github-releases` will cause `swift-http-import` to use GitHub's API to discover and\ndownload GitHub releases for the repository. Repositories hosted on a GitHub Enterprise\ninstance are also supported.\n\nSince GitHub's API rate limits the number of requests per IP therefore it is\n**recommended** that you specify a GitHub personal access token in the `jobs[].from.token`\nfield. Refer to the [GitHub API docs](https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting)\nfor more info on its rate limiting. This field is **required** if the repository is hosted on a Github Enterprise instance instead of `github.com`.\nInstead of providing your token as plain text in the config file, you can use the\n`fromEnv` special syntax for the `jobs[].from.token` field. See\n[specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables) for more details.\n\nIf a repository publishes GitHub releases using different tags, e.g. server components at\n`server-x.y.z` and client at `client-x.y.z`, and you only want to get releases whose tag\nmatches a specific pattern then you can specify a regex for this in the\n`jobs[].from.tag_name_pattern` field. Only releases whose tag name matches this regex will\nbe transferred.\n\nPrerelease and draft releases are not transferred by default. You can override this\nbehavior by setting the `jobs[].from.include_prerelease` and `jobs[].from.include_draft`\nfield to `true`.\n\n[Link to full example config file](./examples/source-debian.yaml)\n\n```yaml\njobs:\n  - from:\n      url: https://github.com/sapcc/limesctl\n      type: github-releases\n    to:\n      container: mirror\n      object_prefix: sapcc/limesctl\n```\n\n#### Swift\n\nAlternatively, the source in `jobs[].from` can also be a private Swift container if Swift credentials are specified\ninstead of a source URL.\n[(Link to full example config file)](./examples/source-swift.yaml)\n\n```yaml\njobs:\n  - from:\n      auth_url:            https://my.keystone.local:5000/v3\n      user_name:           uploader\n      user_domain_name:    Default\n      project_name:        datastore\n      project_domain_name: Default\n      password:            20g82rzg235oughq\n      container:           upstream-mirror\n      object_prefix:       repos/ubuntu\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n```\n\nFor defining Swift options, you can use the `fromEnv` special syntax for all the fields\nunder the `from` key instead of specifying these fields as plain text. See\n[specifying sensitive info as environment variables](#specifying-sensitive-info-as-environment-variables)\n\n### File selection\n\n#### By name\n\nFor each job, you may supply three [regular expressions](https://golang.org/pkg/regexp/syntax/) to influence which files\nare transferred:\n\n* `except`: Files and subdirectories whose path matches this regex will not be transferred.\n* `only`: Only files and subdirectories whose path matches this regex will be transferred. (Note that `except` takes\n  precedence over `only`. If both are supplied, a file or subdirectory matching both regexes will be excluded.)\n* `immutable`: Files whose path matches this regex will be considered immutable, and `swift-http-import` will not check\n  them for updates after having synced them once.\n\nFor `except` and `only`, you can distinguish between subdirectories and files because directory paths end with a slash,\nwhereas file paths don't.\n\nFor example, with the configuration below, directories called `sub_dir` and files with a `.gz` extension are excluded on\nevery level in the source tree:\n[(Link to full example config file)](./examples/filter-except.yaml)\n\n```yaml\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n    except: \"sub_dir/$|.gz$\"\n```\n\nWhen using `only` to select files, you will usually want to include an alternative `/$` in the regex to match all\ndirectories. Otherwise all directories will be excluded and you will only include files in the toplevel directory.\n[(Link to full example config file)](./examples/filter-only.yaml)\n\n```yaml\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n    only: \"/$|.amd64.deb$\"\n```\n\nThe `immutable` regex is especially useful for package repositories because package files, once uploaded, will never change:\n[(Link to full example config file)](./examples/filter-immutable.yaml)\n\n```yaml\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n    immutable: '.*\\.deb$'\n```\n\n#### By age\n\nThe `jobs[].match.not_older_than` configuration option can be used to exclude objects that are older than some\nthreshold, as indicated by the `Last-Modified` header of the source object.\n[(Link to full example config file)](./examples/filter-not-older-than.yaml)\n\n```yaml\njobs:\n  - from:\n      ... # not shown: credentials for Swift source\n      container: on-site-backup\n    to:\n      container: off-site-backup\n    match:\n      not_older_than: 3 days # ignore old backups, focus on the recent ones\n```\n\nThe value of `jobs[].match.not_older_than` is a value with one of the following units:\n\n- `seconds` (`s`)\n- `minutes` (`m`)\n- `hours` (`h`)\n- `days` (`d`)\n- `weeks` (`w`)\n\n*Warning:* As of this version, this configuration option only works with Swift sources.\n\n\n#### Simplistic file comparison\n\nThe `jobs[].match.simplistic_comparison` configuration option can be used to\nforce `swift-http-import` to only use last modified time to determine file\ntransfer eligibility. This option can be used for compatibility with other\nsimilar tools (e.g. `rclone`). Without it, `swift-http-import` is likely to\nretransfer files that were already transferred by other tools due to its strict\ncomparison constraints.\n\n**Note**: This option is only _99.9% reliable_, if you need 100% reliability\nfor file transfer eligibility then you should not use this option and let\n`swift-http-import` default to its strong comparison constraints.\n\n[(Link to full example config file)](./examples/filter-simplistic-comparison.yaml)\n\n```yaml\njobs:\n  - from:\n      ...\n    to:\n      container: off-site-backup\n    match:\n      simplistic_comparison: true\n```\n\n\n### Transfer behavior: Segmenting on the source side\n\nBy default, `swift-http-import` will download source files in segments of at most 500 MiB, using [HTTP range\nrequests](https://tools.ietf.org/html/rfc7233). Range requests are supported by most HTTP servers that serve static\nfiles, and servers without support will fallback to regular HTTP and send the whole file at once. **Note that** range\nrequests are currently not supported for Swift sources that require authentication.\n\nIn the unlikely event that range requests confuse the HTTP server at the source side, they can be disabled by setting\n`jobs[].from.segmenting` to `false`:\n[(Link to full example config file)](./examples/transfer-no-source-segmenting.yaml)\n\n```yaml\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n      segmenting: false\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n```\n\nThe default segment size of 500 MiB can be changed by setting `jobs[].from.segment_bytes` like so:\n[(Link to full example config file)](./examples/transfer-source-segmenting.yaml)\n\n```yaml\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n      segment_bytes: 1073741824 # 1 GiB\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n```\n\n### Transfer behavior: Segmenting on the target side\n\nSwift rejects objects beyond a certain size (usually 5 GiB). To import larger files,\n[segmenting](https://docs.openstack.org/swift/latest/overview_large_objects.html) must be used. The configuration\nsection `jobs[].segmenting` enables segmenting for the given job:\n[(Link to full example config file)](./examples/transfer-target-segmenting.yaml)\n\n```yaml\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n    segmenting:\n      min_bytes:     2147483648      # import files larger than 2 GiB...\n      segment_bytes: 1073741824      # ...as segments of 1 GiB each...\n      container:     mirror_segments # ...which are stored in this container (optional, see below)\n```\n\nSegmenting behaves like the standard `swift` CLI client with the `--use-slo` option:\n\n- The segment container's name defaults to the target container's name plus a `_segments` prefix.\n- Segments are uploaded with the object name `$OBJECT_PATH/slo/$UPLOAD_TIMESTAMP/$OBJECT_SIZE_BYTES/$SEGMENT_SIZE_BYTES/$SEGMENT_INDEX`.\n- The target object uses an SLO manifest. DLO manifests are not supported.\n\n### Transfer behavior: Expiring objects\n\nSwift allows for files to be set to\n[expire at a user-configured time](https://docs.openstack.org/swift/latest/overview_expiring_objects.html), at which\npoint they will be deleted automatically. When transferring files from a Swift source, `swift-http-import` will copy any\nexpiration dates to the target, unless the `jobs[].expiration.enabled` configuration option is set to `false`.\n[(Link to full example config file)](./examples/transfer-no-expiring.yaml)\n\n```yaml\njobs:\n  - from:\n      ... # not shown: credentials for Swift source\n      container: source-container\n    to:\n      container: target-container\n    expiration:\n      enabled: false\n```\n\nIn some cases, it may be desirable for target objects to live longer than source objects. For example, when syncing from\nan on-site backup to an off-site backup, it may be useful to retain the off-site backup for a longer period of time than\nthe on-site backup. Use the `jobs[].expiration.delay_seconds` configuration option to shift all expiration dates on the\ntarget side by a fixed amount of time compared to the source side.\n[(Link to full example config file)](./examples/transfer-expiring.yaml)\n\n```yaml\njobs:\n  - from:\n      ... # not shown: credentials for Swift source\n      container: on-site-backup\n    to:\n      container: off-site-backup\n    expiration:\n      delay_seconds: 1209600 # retain off-site backups for 14 days longer than on-site backup\n```\n\n### Transfer behavior: Symlinks\n\nStarting with the Queens release, Swift optionally supports symlinks. A symlink is a light-weight reference to some\nother object (possibly in a different container and/or account). When a HEAD or GET request is sent to retrieve the\nsymlink's metadata or content, the linked object's metadata or content is returned instead.\n\nWhen swift-http-import transfers from a Swift source (i.e., Swift credentials are given in `jobs[].from`), and when the\ntarget side supports symlinks, symlinks in the source side will be copied as symlinks. No extra configuration is\nnecessary for this behavior.\n\nHowever, the link target must be transferred **in the same job**. (This restriction may be lifted in a later version.)\nOtherwise, the symlink will be transferred as a regular object, possibly resulting in duplication of file contents on\nthe target side.\n\n### Transfer behavior: Delete objects on the target side\n\nBy default, swift-http-import will only copy files from the source side to the target side. To enable the deletion of\nobjects that exist on the target side, but not on the source side, set the `jobs[].cleanup.strategy` configuration\noption to `delete`.\n[(Link to full example config file)](./examples/transfer-delete-on-target.yaml)\n\n```yaml\njobs:\n  - from:\n      url: http://de.archive.ubuntu.com/ubuntu/\n    to:\n      container: mirror\n      object_prefix: ubuntu-repos\n    cleanup:\n      strategy: delete\n```\n\nAnother possible value for `jobs[].cleanup.strategy` is `report`, which will log objects that `delete` would clean\nup without actually touching them.\n\nWhen combined with `jobs[].only` and/or `jobs[].except`, cleanup will delete all files excluded by those filters, even\nif the same file exists on the source side. This is the same behavior as if `--delete-excluded` is given to rsync.\n\n### Performance\n\nBy default, only a single worker thread will be transferring files. You can scale this up by including a `workers` section at the top level like so:\n[(Link to full example config file)](./examples/basic-multiple-workers.yaml)\n\n```yaml\nworkers:\n  transfer: 10\n```\n\n## Log output\n\nLog output on `stderr` is very sparse by default. Errors are always reported, and a final count will appear at the end like this:\n\n```\n2016/12/19 14:28:23 INFO: 103 dirs scanned, 0 failed\n2016/12/19 14:28:23 INFO: 1496 files found, 167 transferred, 3 failed\n```\n\nIn this case, all sources contain 103 directories and 1496 files. 170 files were found to be newer on the source and\nthus need transfer. Of those, 167 were successfully transferred, and the remaining 3 file transfers failed (so there\nshould be 3 errors in the log).\n\nWhen the environment variable `LOG_TRANSFERS=true` is set, a log line will be printed for each transferred file:\n\n```\n2016/12/19 14:28:19 INFO: transferring to container-name/path/to/object\n```\n\n## StatsD metrics\n\nAdding an optional statsd config section enables submitting StatsD metrics.\n```yaml\nstatsd:\n  hostname: localhost\n  port:     8125\n  prefix:   swift_http_import\n```\n\nThe following metrics are sent:\n\n| Kind    | Name                         | Description\n| ------- | ---------------------------- | --------------------------------------------\n| Gauge   | `last_run.success`           | `1` if no error occurred, otherwise 0\n| Gauge   | `last_run.success_timestamp` | UNIX timestamp of last successful run\n| Gauge   | `last_run.duration_seconds`  | Runtime in seconds\n| Gauge   | `last_run.jobs_skipped`      | Number of jobs skipped\n| Gauge   | `last_run.dirs_scanned`      | Number of directories scanned\n| Gauge   | `last_run.files_found`       | Number of files found\n| Gauge   | `last_run.files_transfered`  | Number of files actually transferred\n| Gauge   | `last_run.files_failed`      | Number of files failed (download or upload)\n| Gauge   | `last_run.bytes_transfered`  | Number of bytes transferred\n\n## GPG keyserver selection\n\nWhen verifying GPG signatures on repos, the respective public keys are downloaded from a key server.\n[keyserver.ubuntu.com](https://keyserver.ubuntu.com) and [pgp.mit.edu](https://pgp.mit.edu) are used by default.\n\nYou can manually specify which keyserver to use by providing a list of keyserver URL patterns, in which the string\n`{keyid}` will be replaced with the 16-hexdigit ID of the key that shall be retrieved. The default value for this is:\n\n```yaml\ngpg:\n  keyserver_urls:\n    - \"https://keyserver.ubuntu.com/pks/lookup?search=0x{keyid}\u0026options=mr\u0026op=get\"\n    - \"https://pgp.mit.edu/pks/lookup?search=0x{keyid}\u0026options=mr\u0026op=get\"\n```\n\nThe keyservers will be tried in order until one returns a public key.\n\nAdditionally, you can also specify a Swift container name with `gpg.cache_container_name` to cache the downloaded keys. The\ncache will be loaded into memory on startup in order to avoid downloading the same keys every time.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsapcc%2Fswift-http-import","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsapcc%2Fswift-http-import","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsapcc%2Fswift-http-import/lists"}