{"id":20267121,"url":"https://github.com/xiph/rd_tool","last_synced_at":"2025-04-11T03:36:22.428Z","repository":{"id":18661555,"uuid":"21869318","full_name":"xiph/rd_tool","owner":"xiph","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-26T22:50:13.000Z","size":507,"stargazers_count":20,"open_issues_count":18,"forks_count":29,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-25T01:37:19.314Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://arewecompressedyet.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xiph.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-07-15T17:37:17.000Z","updated_at":"2025-02-26T22:50:17.000Z","dependencies_parsed_at":"2023-12-29T15:26:56.650Z","dependency_job_id":"fef18922-a825-49f3-91ed-81f76e9afa0d","html_url":"https://github.com/xiph/rd_tool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Frd_tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Frd_tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Frd_tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Frd_tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xiph","download_url":"https://codeload.github.com/xiph/rd_tool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248337955,"owners_count":21087134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T12:13:18.519Z","updated_at":"2025-04-11T03:36:22.392Z","avatar_url":"https://github.com/xiph.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"rd_tool\n=======\n\nrd_tool.py is a script for collecting rate-distortion curves across a series of either local or Amazon AWS nodes.\n\nThis script is run by arewecompressedyet.com, but can also be run locally to produce the same data.\n\nDependencies\n============\n\nYou will need Python 3.4 or later, as well as [boto3](https://github.com/boto/boto3).\n\nOn Ubuntu, install with:\n```\nsudo apt install python3-tornado python3-boto3\n```\n\nNode dependencies\n=================\n\nIndividual build machines do not need Python, but do need bash. Each machine\nshould be configured with a user and work_root (such as that user's home\ndirectory). This work directory must be populated with folders called\ndaalatool, dump\\_ciede2000, and, optionally, dav1d and vmaf, which need to\ncontain checkouts of their respective git repositories, each with tools built.\n\nTo compile Daala tools:\n\n```\nsudo apt install build-essential autoconf libogg-dev libjpeg-dev libpng-dev check python3-numpy python3-scipy libtool pkg-config\n```\n\n```\ngit clone https://git.xiph.org/daala.git daalatool\ncd daalatool\n./autogen.sh\n./configure --disable-player\nmake tools -j4\n```\n\nFor dump\\_ciede2000:\n\nInstall rust if you haven't already. You only need rust to compile the binary\nand don't need it on the individual machines.\n\n```\ncurl -sf -L https://static.rust-lang.org/rustup.sh | sh\n```\n\nBuild dump_ciede2000.\n\n```\ngit clone https://github.com/KyleSiefring/dump_ciede2000\ncd dump_ciede2000\ncargo build --release\n```\n\nExit that directory\n\n```\ncd ../\n```\n\nFor dav1d (optional):\n\n```\nsudo apt install meson\n```\n\n```\ngit clone https://code.videolan.org/videolan/dav1d.git\ncd dav1d\nmkdir build \u0026\u0026 cd build\nmeson ..\nninja\n```\n\nFor vmaf (optional):\n\n```\ngit clone https://github.com/Netflix/vmaf.git\ncd vmaf\ncd ptools; make; cd ../wrapper; make; cd ..;\n```\n\nrd_tool will automatically create one slot directory per core, and upload\ncodec binaries into that directory.\n\nUsing AWS nodes\n===============\n\nYou will need a ~/.aws configuration for boto with your AWS information and credentials.\n\nSpecify the autoscaling groukp to use with -awsgroup.\n\nUsing local nodes\n=================\n\nYou can specify all of the machines you want to use in a JSON file:\n\n```json\n[\n  {\n    \"host\": \"localhost\",\n    \"user\": \"thomas\",\n    \"cores\": 4,\n    \"port\": 22,\n    \"work_root\": \"/home/thomas/tmp\",\n    \"media_path\": \"/home/thomas/sets\"\n  },\n  {\n    ...\n  }\n]\n\n```\n\nSpecify this configuration on the command line with -machineconf.\n\nBuilds\n======\n\nSpecify the path to the git checkout of thetested codec with the -bindir\nargument. rd_tool.py expects to find prebuilt binaries of the codec - it will\nnot build the codec automatically.\n\nOutput files\n============\n\nOutput files are written into the current directory. The format of the .out\nfiles is one quantizer per line, of the format\n\n\u003cquantizer\u003e \u003cnumber of pixels\u003e \u003cfile size in bytes\u003e \u003cmetric1\u003e \u003cmetric2\u003e ...\n\nSee RDRun.reduce() in work.py for an up-to-date list of metrics.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiph%2Frd_tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxiph%2Frd_tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiph%2Frd_tool/lists"}