Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/nlnetlabs/rrdpit

RRDP IT: A disk to RPKI Delta Protocol tool.
https://github.com/nlnetlabs/rrdpit

rpki rrdp rust

Last synced: about 2 months ago
JSON representation

RRDP IT: A disk to RPKI Delta Protocol tool.

Host: GitHub
URL: https://github.com/nlnetlabs/rrdpit
Owner: NLnetLabs
License: bsd-3-clause
Created: 2019-07-30T07:13:15.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2024-03-26T09:09:10.000Z (9 months ago)
Last Synced: 2024-05-01T10:25:30.224Z (8 months ago)
Topics: rpki, rrdp, rust
Language: Rust
Size: 44.9 KB
Stars: 9
Watchers: 13
Forks: 2
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# RRDPIT

"rrdpit" is a small little tool that can be pointed at a directory on your
system, and produce RPKI RRDP (RFC 8182) notification, snapshot, and
delta files. You will need to use an http server of your preferred
flavour to deliver these files to the world.

WARNING:
========

This tool is designed to be run *in between* publication runs, but not
during. As part of the syncing process the source directory is crawled
recursively. If there are any changes made to the paths during the sync
then this can result in RRDP snapshots and delta containing inconsistent
repository state.

This situation would resolve itself when rrdpit runs again while there
are no changes being made, but it could cause errors and noise for RPKI
validators.

So, the best option is to run this when it is known that there will be
no changes made to the source. E.g. when a non RRDP capable publication
server knows that it's done writing content, it could trigger rrdpit,
and wait with writing new content until rrdpit is done.

Of course, the safest option might still be to use an RRDP capable RPKI
Publication Server instead so this extra helper tool would not be needed.

## Changelog

### Release 0.0.4

Updated _ring_ to 0.17.

### Release 0.0.3

Add option to limit the maximum number of deltas using --max_deltas.
Keeping too many deltas will result in large RRDP notification files if
the individual deltas are much smaller than the snapshot. This can have
a big impact on the server if many RPs request a large notification file.

The default limit is set to 25. This value will work well if rrdpit runs
every minute as it's more than twice the number of the typical RP fetch
interval (10 minutes). If rrdpit runs less frequently then this number
can be lowered. Essentially, one should keep enough deltas so that returning
RPs never need to load the snapshot.

The minimum value of this setting is 1.

### Release 0.0.2

Ignore hidden files in the source directory. I.e. exclude any and all files
and folders starting with a '.' character.

### Release 0.0.1

Initial release.

## Installing rrdpit

Assuming you have rsync and the C toolchain but not yet Rust, here’s how
you get rust installed.

```bash
curl https://sh.rustup.rs -sSf | sh
source ~/.cargo/env
```

If you have an old version of rust installed you may have to update it.
```bash
rustup update
```

To install 'rrdpit' under your user's home directory, use this:

```bash
git clone [email protected]:NLnetLabs/rrdpit.git
cd rrdpit
cargo install
```

If you have an older version of rrdpit installed, you can update via

```bash
cargo install -f
```

## Using rrdpit

You can ask rrdpit for help.

```bash
rrdpit --help
rrdpit 0.0.2
Dist to RPKI RRDP

USAGE:
rrdpit --https --rsync --source --target [clean]

FLAGS:
--help Prints help information
-V, --version Prints version information

OPTIONS:
-h, --https base rrdp uri
-r, --rsync base rsync uri
-s, --source source directory
-t, --target target directory

ARGS:
Clean up target dir (handle with care!)
```

Note that 'clean' is optional. If used rrdpit will try to clean out the target
dir, i.e. it will remove unused session id dirs, and unused version directories
for delta files which are no longer referenced.

Use this option with care. You do NOT want to use this and accidentally use a
system directory for the `--target` option. Especially if you run this as root,
which would be ill-advised as well.

### Examples

Sync the entire ARIN RPKI repository:
```bash

$ mkdir -p tmp/arin
$ cd tmp/arin/
$ mkdir source
$ rsync -zarvh rsync://rpki.arin.net/repository ./source/
receiving file list ... done
./
arin-rpki-ta.cer
arin-rpki-ta/
arin-rpki-ta/5e4a23ea-e80a-403e-b08c-2171da2157d3.cer
arin-rpki-ta/arin-rpki-ta.crl
arin-rpki-ta/arin-rpki-ta.mft
arin-rpki-ta/5e4a23ea-e80a-403e-b08c-2171da2157d3/
.....
.....

sent 158.78K bytes received 11.08M bytes 976.85K bytes/sec
total size is 14.25M speedup is 1.27
```

Now create RRDP files in a target dir:
```bash
$ mkdir target
$ time rrdpit --https https://rpki.arin.net/rrdp/ \
--rsync rsync://rpki.arin.net/repository/ \
--source ./source/ \
--target ./target/

real 0m0.848s
user 0m0.385s
sys 0m0.258s
```

Check that all expected files are there, or well, at least the number:
```bash
$ find ./source/ -type f | wc -l
7031

$ grep uri ./target/8e142e20-236c-4694-8430-b05693fab150/1/snapshot.xml | wc -l
7031
```

(note that that uuid is a randomly generated session id, used when the target dir is empty)

```bash
$ rm source/arin-rpki-ta.cer
$ time rrdpit --https https://rpki.arin.net/rrdp/ \
--rsync rsync://rpki.arin.net/repository/ \
--source ./source/ \
--target ./target/

real 0m1.484s
user 0m1.285s
sys 0m0.186s

$ find target
target
target/8e142e20-236c-4694-8430-b05693fab150
target/8e142e20-236c-4694-8430-b05693fab150/1
target/8e142e20-236c-4694-8430-b05693fab150/1/snapshot.xml
target/8e142e20-236c-4694-8430-b05693fab150/2
target/8e142e20-236c-4694-8430-b05693fab150/2/snapshot.xml
target/8e142e20-236c-4694-8430-b05693fab150/2/delta.xml
target/notification.xml

$ cat ./target/notification.xml

$ cat ./target/8e142e20-236c-4694-8430-b05693fab150/2/delta.xml

```

Note that if you sync again, and there are no changes in the source dir, no deltas will be written:

```bash
$ time rrdpit --https https://rpki.arin.net/rrdp/ \
--rsync rsync://rpki.arin.net/repository/ \
--source ./source/ \
--target ./target/

real 0m1.495s
user 0m1.292s
sys 0m0.190s

rrdpit will also perform some sanity checks on the existing RRDP files, and if it finds an issue it will use a new session:

```bash
$ find target -type f
target/8e142e20-236c-4694-8430-b05693fab150/1/snapshot.xml
target/8e142e20-236c-4694-8430-b05693fab150/2/snapshot.xml
target/8e142e20-236c-4694-8430-b05693fab150/2/delta.xml
target/notification.xml

$ echo "corrupt" > target//8e142e20-236c-4694-8430-b05693fab150/2/delta.xml

$ rrdpit --https https://rpki.arin.net/rrdp/ \
--rsync rsync://rpki.arin.net/repository/ \
--source ./source/ \
--target ./target/

$ find target -type f
target/07cfc1ce-e7d9-4bec-8a70-9feb76778700/1/snapshot.xml
target/8e142e20-236c-4694-8430-b05693fab150/1/snapshot.xml
target/8e142e20-236c-4694-8430-b05693fab150/2/snapshot.xml
target/8e142e20-236c-4694-8430-b05693fab150/2/delta.xml
target/notification.xml
```

Optionally you can let rrdpit clean up old files as well:
```bash
$ rrdpit --https https://rpki.arin.net/rrdp/ \
--rsync rsync://rpki.arin.net/repository/ \
--source ./source/ \
--target ./target/ \
clean

$ find target -type f
target/07cfc1ce-e7d9-4bec-8a70-9feb76778700/1/snapshot.xml
target/notification.xml
```

## Future

This code can possibly use more testing. And some things can be cleaned up. However, it seems to
work well from the testing we have done.

Of course you can create issues, but given that our main effort is directed at Krill for the
moment, which includes its own RRDP server, we cannot guarantee that issues will get a high
priority. Pull requests may get more mileage ;)