Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/deb-s3/deb-s3
Easily create and manage an APT repository on S3
https://github.com/deb-s3/deb-s3
Last synced: about 2 months ago
JSON representation
Easily create and manage an APT repository on S3
- Host: GitHub
- URL: https://github.com/deb-s3/deb-s3
- Owner: deb-s3
- License: mit
- Created: 2020-07-08T06:23:38.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2024-07-09T20:14:28.000Z (3 months ago)
- Last Synced: 2024-07-10T00:46:07.264Z (3 months ago)
- Language: Ruby
- Homepage:
- Size: 312 KB
- Stars: 99
- Watchers: 3
- Forks: 25
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
- awesome - deb-s3/deb-s3 - Easily create and manage an APT repository on S3 (others)
README
# deb-s3
[](https://travis-ci.org/deb-s3/deb-s3)
**This repository is a fork of [krobertson/deb-s3](https://github.com/krobertson/deb-s3).**
`deb-s3` is a simple utility to make creating and managing APT repositories on
S3.
Most existing guides on using S3 to host an APT repository have you
using something like [reprepro](http://mirrorer.alioth.debian.org/) to generate
the repository file structure, and then [s3cmd](http://s3tools.org/s3cmd) to
sync the files to S3.
The annoying thing about this process is it requires you to maintain a local
copy of the file tree for regenerating and syncing the next time. Personally,
my process is to use one-off virtual machines with
[Vagrant](http://vagrantup.com), script out the build process, and then would
prefer to just upload the final `.deb` from my Mac.
With `deb-s3`, there is no need for this. `deb-s3` features:
* Downloads the existing package manifest and parses it.
* Updates it with the new package, replacing the existing entry if already
there or adding a new one if not.
* Uploads the package itself, the Packages manifest, and the Packages.gz
manifest. It will skip the uploading if the package is already there.
* Updates the Release file with the new hashes and file sizes.
## Getting Started
Install the package via gem
```console
$ gem install deb-s3
```
or via APT (Debian 11 "Bullseye" and Ubuntu 22.04 "Jammy Jellyfish" and newer):
```console
# Add repository key
$ sudo wget -O /etc/apt/trusted.gpg.d/deb-s3-archive-keyring.gpg https://raw.githubusercontent.com/deb-s3/deb-s3/master/deb-s3-archive-keyring.gpg
# Add repository
$ echo "deb http://deb-s3-repo.s3.us-east-2.amazonaws.com/$(lsb_release -is | tr A-Z a-z)/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list > /dev/null
# Install package
$ sudo apt-get update
$ sudo apt-get install deb-s3
```
To run the code directly, just check out the repo and run bundler to ensure
all dependencies are installed:
```console
$ git clone https://github.com/deb-s3/deb-s3.git
$ cd deb-s3
$ bundle install
```
Now to upload a package, simply use:
```console
$ deb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb
>> Examining package file my-deb-package-1.0.0_amd64.deb
>> Retrieving existing package manifest
>> Uploading package and new manifests to S3
-- Transferring pool/m/my/my-deb-package-1.0.0_amd64.deb
-- Transferring dists/stable/main/binary-amd64/Packages
-- Transferring dists/stable/main/binary-amd64/Packages.gz
-- Transferring dists/stable/Release
>> Update complete.
```
```
Usage:
deb-s3 upload FILES
Options:
-a, [--arch=ARCH] # The architecture of the package in the APT repository.
-p, [--preserve-versions], [--no-preserve-versions] # Whether to preserve other versions of a package in the repository when uploading one.
-l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
[--fail-if-exists], [--no-fail-if-exists] # Whether to overwrite any existing package that has the same filename in the pool or the same name and version in the manifest but different contents.
[--skip-package-upload], [--no-skip-package-upload] # Whether to skip all package uploads.This is useful when hosting .deb files outside of the bucket.
-b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
[--prefix=PREFIX] # The path prefix to use when storing on S3.
-o, [--origin=ORIGIN] # The origin to use in the repository Release file.
[--suite=SUITE] # The suite to use in the repository Release file.
-c, [--codename=CODENAME] # The codename of the APT repository.
# Default: stable
-m, [--component=COMPONENT] # The component of the APT repository.
# Default: main
[--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
[--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
[--s3-region=S3_REGION] # The region for connecting to S3.
# Default: us-east-1
[--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
[--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
-v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, authenticated, or nil to avoid setting an ACL.
# Default: public
[--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
[--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
-e, [--encryption], [--no-encryption] # Use S3 server side encryption.
-q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
-C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
Uploads the given files to a S3 bucket as an APT repository.
```
You can also delete packages from the APT repository. Please keep in mind that
this does NOT delete the .deb file itself (the `clean` command does that), it
only removes it from the list of packages in the specified component, codename
and architecture.
Now to delete the package:
```console
$ deb-s3 delete my-deb-package --arch amd64 --bucket my-bucket --versions 1.0.0
>> Retrieving existing manifests
-- Deleting my-deb-package version 1.0.0
>> Uploading new manifests to S3
-- Transferring dists/stable/main/binary-amd64/Packages
-- Transferring dists/stable/main/binary-amd64/Packages.gz
-- Transferring dists/stable/Release
>> Update complete.
```
```
Usage:
deb-s3 delete PACKAGE
Options:
-a, [--arch=ARCH] # The architecture of the package in the APT repository.
[--versions=one two three] # The space-delimited versions of PACKAGE to delete. If not specified, ALL VERSIONS will be deleted. Fair warning. E.g. --versions "0.1 0.2 0.3"
-b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
[--prefix=PREFIX] # The path prefix to use when storing on S3.
-o, [--origin=ORIGIN] # The origin to use in the repository Release file.
[--suite=SUITE] # The suite to use in the repository Release file.
-c, [--codename=CODENAME] # The codename of the APT repository.
# Default: stable
-m, [--component=COMPONENT] # The component of the APT repository.
# Default: main
[--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
[--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
[--s3-region=S3_REGION] # The region for connecting to S3.
# Default: us-east-1
[--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
[--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
-v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
# Default: public
[--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
[--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
-e, [--encryption], [--no-encryption] # Use S3 server side encryption.
-q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
-C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
Remove the package named PACKAGE. If --versions is not specified, deleteall versions of PACKAGE. Otherwise, only the specified versions will be deleted.
```
Dangling `.deb` files left by the `delete` command (or uploading new versions) can be removed using the `clean` command:
```console
$ deb-s3 clean --bucket my-bucket
>> Retrieving existing manifests
>> Searching for unreferenced packages
-- pool/m/my/my-deb-package-1.0.0_amd64.deb
```
```
Usage:
deb-s3 clean
Options:
-l, [--lock], [--no-lock] # Whether to check for an existing lock on the repository to prevent simultaneous updates
-b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
[--prefix=PREFIX] # The path prefix to use when storing on S3.
-o, [--origin=ORIGIN] # The origin to use in the repository Release file.
[--suite=SUITE] # The suite to use in the repository Release file.
-c, [--codename=CODENAME] # The codename of the APT repository.
# Default: stable
-m, [--component=COMPONENT] # The component of the APT repository.
# Default: main
[--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
[--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
[--session-token=SESSION_TOKEN] # The (optional) session token for connecting to S3.
[--endpoint=ENDPOINT] # The URL endpoint to the S3 API.
[--s3-region=S3_REGION] # The region for connecting to S3.
# Default: us-east-1
[--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
[--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
-v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
# Default: public
[--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
[--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
-e, [--encryption], [--no-encryption] # Use S3 server side encryption.
-q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
-C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
Delete packages from the pool which are no longer referenced
```
You can also verify an existing APT repository on S3 using the `verify` command:
```console
deb-s3 verify -b my-bucket
>> Retrieving existing manifests
>> Checking for missing packages in: stable/main i386
>> Checking for missing packages in: stable/main amd64
>> Checking for missing packages in: stable/main all
```
```
Usage:
deb-s3 verify
Options:
-f, [--fix-manifests], [--no-fix-manifests] # Whether to fix problems in manifests when verifying.
-b, [--bucket=BUCKET] # The name of the S3 bucket to upload to.
[--prefix=PREFIX] # The path prefix to use when storing on S3.
-o, [--origin=ORIGIN] # The origin to use in the repository Release file.
[--suite=SUITE] # The suite to use in the repository Release file.
-c, [--codename=CODENAME] # The codename of the APT repository.
# Default: stable
-m, [--component=COMPONENT] # The component of the APT repository.
# Default: main
[--access-key-id=ACCESS_KEY_ID] # The access key for connecting to S3.
[--secret-access-key=SECRET_ACCESS_KEY] # The secret key for connecting to S3.
[--s3-region=S3_REGION] # The region for connecting to S3.
# Default: us-east-1
[--force-path-style], [--no-force-path-style] # Use S3 path style instead of subdomains.
[--proxy-uri=PROXY_URI] # The URI of the proxy to send service requests through.
-v, [--visibility=VISIBILITY] # The access policy for the uploaded files. Can be public, private, or authenticated.
# Default: public
[--sign=SIGN] # GPG Sign the Release file when uploading a package, or when verifying it after removing a package. Use --sign with your GPG key ID to use a specific key (--sign=6643C242C18FE05B).
[--gpg-options=GPG_OPTIONS] # Additional command line options to pass to GPG when signing.
-e, [--encryption], [--no-encryption] # Use S3 server side encryption.
-q, [--quiet], [--no-quiet] # Doesn't output information, just returns status appropriately.
-C, [--cache-control=CACHE_CONTROL] # Add cache-control headers to S3 objects.
Verifies that the files in the package manifests exist
```
#### Example S3 IAM Policy
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::BUCKETNAME",
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:GetObjectAcl",
"s3:GetObjectTagging",
"s3:GetObjectTorrent",
"s3:GetObjectVersion",
"s3:GetObjectVersionAcl",
"s3:GetObjectVersionTagging",
"s3:GetObjectVersionTorrent",
"s3:PutObjectAcl",
"s3:PutObjectTagging",
"s3:PutObjectVersionAcl",
"s3:PutObjectVersionTagging",
"s3:ReplicateObject",
"s3:RestoreObject"
],
"Resource": [
"arn:aws:s3:::BUCKETNAME/*"
]
}
]
}
```
## FAQ
## "The bucket does not allow ACLs"
Set visiblity to `nil` to avoid trying to set ACL: `--visibility=nil`. You will need to configure permissions through the bucket policy.