https://github.com/xiph/rd_tool
https://github.com/xiph/rd_tool
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/xiph/rd_tool
- Owner: xiph
- License: bsd-2-clause
- Created: 2014-07-15T17:37:17.000Z (almost 12 years ago)
- Default Branch: master
- Last Pushed: 2025-02-26T22:50:13.000Z (over 1 year ago)
- Last Synced: 2025-03-25T01:37:19.314Z (over 1 year ago)
- Language: Python
- Homepage: http://arewecompressedyet.com/
- Size: 495 KB
- Stars: 20
- Watchers: 9
- Forks: 29
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
rd_tool
=======
rd_tool.py is a script for collecting rate-distortion curves across a series of either local or Amazon AWS nodes.
This script is run by arewecompressedyet.com, but can also be run locally to produce the same data.
Dependencies
============
You will need Python 3.4 or later, as well as [boto3](https://github.com/boto/boto3).
On Ubuntu, install with:
```
sudo apt install python3-tornado python3-boto3
```
Node dependencies
=================
Individual build machines do not need Python, but do need bash. Each machine
should be configured with a user and work_root (such as that user's home
directory). This work directory must be populated with folders called
daalatool, dump\_ciede2000, and, optionally, dav1d and vmaf, which need to
contain checkouts of their respective git repositories, each with tools built.
To compile Daala tools:
```
sudo apt install build-essential autoconf libogg-dev libjpeg-dev libpng-dev check python3-numpy python3-scipy libtool pkg-config
```
```
git clone https://git.xiph.org/daala.git daalatool
cd daalatool
./autogen.sh
./configure --disable-player
make tools -j4
```
For dump\_ciede2000:
Install rust if you haven't already. You only need rust to compile the binary
and don't need it on the individual machines.
```
curl -sf -L https://static.rust-lang.org/rustup.sh | sh
```
Build dump_ciede2000.
```
git clone https://github.com/KyleSiefring/dump_ciede2000
cd dump_ciede2000
cargo build --release
```
Exit that directory
```
cd ../
```
For dav1d (optional):
```
sudo apt install meson
```
```
git clone https://code.videolan.org/videolan/dav1d.git
cd dav1d
mkdir build && cd build
meson ..
ninja
```
For vmaf (optional):
```
git clone https://github.com/Netflix/vmaf.git
cd vmaf
cd ptools; make; cd ../wrapper; make; cd ..;
```
rd_tool will automatically create one slot directory per core, and upload
codec binaries into that directory.
Using AWS nodes
===============
You will need a ~/.aws configuration for boto with your AWS information and credentials.
Specify the autoscaling groukp to use with -awsgroup.
Using local nodes
=================
You can specify all of the machines you want to use in a JSON file:
```json
[
{
"host": "localhost",
"user": "thomas",
"cores": 4,
"port": 22,
"work_root": "/home/thomas/tmp",
"media_path": "/home/thomas/sets"
},
{
...
}
]
```
Specify this configuration on the command line with -machineconf.
Builds
======
Specify the path to the git checkout of thetested codec with the -bindir
argument. rd_tool.py expects to find prebuilt binaries of the codec - it will
not build the codec automatically.
Output files
============
Output files are written into the current directory. The format of the .out
files is one quantizer per line, of the format
...
See RDRun.reduce() in work.py for an up-to-date list of metrics.