{"id":28163991,"url":"https://github.com/psy-ex/metrics","last_synced_at":"2026-03-02T22:03:29.756Z","repository":{"id":277755895,"uuid":"923981128","full_name":"psy-ex/metrics","owner":"psy-ex","description":"Perceptual video metrics toolkit","archived":false,"fork":false,"pushed_at":"2025-04-22T05:50:19.000Z","size":108,"stargazers_count":18,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T07:11:35.714Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/psy-ex.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-29T07:31:20.000Z","updated_at":"2025-04-22T05:50:22.000Z","dependencies_parsed_at":"2025-02-15T22:20:44.599Z","dependency_job_id":"bfb42736-365a-4d68-b070-02d3aa321949","html_url":"https://github.com/psy-ex/metrics","commit_stats":null,"previous_names":["psy-ex/metrics"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psy-ex%2Fmetrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psy-ex%2Fmetrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psy-ex%2Fmetrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/psy-ex%2Fmetrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/psy-ex","download_url":"https://codeload.github.com/psy-ex/metrics/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254328393,"owners_count":22052634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-15T11:14:55.590Z","updated_at":"2026-03-02T22:03:29.745Z","avatar_url":"https://github.com/psy-ex.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PSY-EX Metrics\n\nThe Psychovisual Experts group presents `metrics`, a video quality assessment\ntoolkit that provides a suite of scripts for measuring and comparing video\ncodecs using metrics such as:\n\n- Average SSIMULACRA2 across frames\n- Average Butteraugli (3pnorm) across frames\n- Weighted XPSNR\n- VMAF NEG (Harmonic Mean)\n- VMAF\n- SSIM\n- PSNR\n\nWe include modules for processing video files, running video encoding, and\ngenerating both numerical and visual reports from quality metrics.\n\nThe four main scripts are:\n\n- `scores.py`: Compute detailed quality scores for a given source/distorted\n  video pair.\n- `encode.py`: Encode videos using various codecs (e.g., x264, x265, svtav1,\n  aomenc, libvpx/VP9) and print metrics.\n- `stats.py`: Encode videos at various quality settings and log metric\n  statistics to a CSV file.\n- `plot.py`: Generate visual plots from CSV metric data for side-by-side codec\n  comparison.\n\nRead more below on how to install and use these utilities.\n\n## Sample Graphs\n\n![SSIMULACRA2](./static/ssimu2_hmean_plot.svg)\n\n![Butteraugli](./static/butter_distance_plot.svg)\n\n![W-XPSNR](./static/wxpsnr_plot.svg)\n\n## Overview\n\nPSY-EX Metrics enables you to:\n\n- Encode videos using various codecs (e.g., x264, x265, svtav1, aomenc,\n  libvpx/VP9).\n- Calculate various metrics.\n- Generate CSV files with computed metric statistics for further analysis.\n- Visualize the metrics side-by-side, comparing codec results through\n  customizable plots.\n\n## Installation\n\n### Dependencies\n\n- [uv](https://github.com/astral-sh/uv/blob/main/README.md), a Python project\n  manager\n- FFmpeg \u003e= 7.1 (required for XPSNR)\n- VapourSynth, and required plugins:\n  - ffms2\n  - [vszip](https://github.com/dnjulek/vapoursynth-zip)\n  - [vship](https://github.com/Line-fr/Vship) [Optional: provides GPU support]\n\n### Install Steps\n\n0. Install required dependencies outlined in the previous section\n\n1. Clone the repository\n\n```bash\ngit clone https://github.com/psy-ex/metrics.git\ncd metrics/\n```\n\n2. Enter the scripts directory \u0026 mark the scripts as executable\n\n```bash\ncd scripts/\nchmod a+x stats.py scores.py plot.py encode.py\n```\n\n3. Run a script, no Python package installation required\n\n```bash\n./scores.py source.mkv distorted.mkv\n```\n\n## Usage\n\n### scores.py\n\n```bash\n% ./scores.py --help\nusage: scores.py [-h] [-e EVERY] [-g GPU_STREAMS] [-t THREADS] source distorted\n\nRun metrics given a source video \u0026 a distorted video.\n\npositional arguments:\n  source                Source video path\n  distorted             Distorted video path\n\noptions:\n  -h, --help            show this help message and exit\n  -e, --every EVERY     Only score every nth frame. Default 1 (every frame)\n  -g, --gpu-streams GPU_STREAMS\n                        Number of GPU streams for SSIMULACRA2/Butteraugli\n  -t, --threads THREADS\n                        Number of threads for SSIMULACRA2/Butteraugli\n```\n\nExample:\n\n```bash\n./scores.py source.mkv distorted.mkv -e 3\n```\n\nThis command compares a reference `source.mkv` with `distorted.mkv`, scoring\nevery 3rd frame.\n\n### encode.py\n\n```bash\n% ./encode.py --help\nusage: encode.py [-h] -i INPUT -q QUALITY [-b KEEP] [-e EVERY] [-g GPU_STREAMS] [-t THREADS] [-n] {x264,x265,svtav1,aomenc,vpxenc} ...\n\nGenerate statistics for a single video encode.\n\npositional arguments:\n  {x264,x265,svtav1,aomenc,vpxenc}\n                        Which video encoder to use\n  encoder_args          Additional encoder arguments (pass these after a '--' delimiter)\n\noptions:\n  -h, --help            show this help message and exit\n  -i, --input INPUT     Path to source video file\n  -q, --quality QUALITY\n                        Desired CRF value for the encoder\n  -b, --keep KEEP       Output video file name\n  -e, --every EVERY     Only score every nth frame. Default 1 (every frame)\n  -g, --gpu-streams GPU_STREAMS\n                        Number of GPU streams for SSIMULACRA2/Butteraugli\n  -t, --threads THREADS\n                        Number of threads for SSIMULACRA2/Butteraugli\n  -n, --no-metrics      Skip metrics calculations\n```\n\nExamples:\n\n```bash\n./encode.py -i source.mkv --keep video.ivf -q 29 svtav1 -- --preset 2\n```\n\nThis command encodes `source.mkv` at a CRF of 29 using the SVT-AV1 encoder with\nthe `--preset 2` argument. It will print metrics after encoding.\n\n```bash\n./encode.py -i source.mkv --keep video.ivf -q 29 -g 4 svtav1 -- --preset 8\n```\n\nThis command does the same as the previous command, but uses 4 GPU streams\n(instead of using the CPU) as well as passing a higher preset value to SVT-AV1.\n\n### stats.py\n\n```bash\nusage: stats.py [-h] -i INPUTS [INPUTS ...] -q QUALITY -o OUTPUT [-e EVERY] [-g GPU_STREAMS] [-t THREADS] [-k] {x264,x265,svtav1,aomenc,vpxenc} ...\n\nGenerate statistics for a series of video encodes.\n\npositional arguments:\n  {x264,x265,svtav1,aomenc,vpxenc}\n                        Which video encoder to use\n  encoder_args          Additional encoder arguments (pass these after a '--' delimiter)\n\noptions:\n  -h, --help            show this help message and exit\n  -i, --inputs INPUTS [INPUTS ...]\n                        Path(s) to source video file(s)\n  -q, --quality QUALITY\n                        List of quality values to test (e.g. 20 30 40 50)\n  -o, --output OUTPUT   Path to output CSV file\n  -e, --every EVERY     Only score every nth frame. Default 1 (every frame)\n  -g, --gpu-streams GPU_STREAMS\n                        Number of GPU streams for SSIMULACRA2/Butteraugli\n  -t, --threads THREADS\n                        Number of threads for SSIMULACRA2/Butteraugli\n  -k, --keep            Keep output video files\n```\n\nExample:\n\n```bash\n./stats.py \\\n  -i source.mkv \\\n  -q \"20 25 30 35 40\" \\\n  -o ./svtav1_2.3.0-B_p8.csv \\\n  -e 3 \\\n  svtav1 -- --preset 8 --tune 2\n```\n\nThis command processes `source.mkv` at quality levels 20, 25, 30, 35, \u0026 40 using\nthe SVT-AV1 encoder, scoring every 3rd frame, and writes the results to an\noutput `svtav1_2.3.0-B_p8.csv`.\n\nYou can also script this to run across multiple speed presets:\n\n```bash\n#!/bin/bash -eu\n\nfor speed in {2..6}; do\n  ./stats.py \\\n  -i ~/Videos/reference/*.y4m \\\n  -q \"20 25 30 35 40\" \\\n  -o ./svtav1_2.3.0-B_p${speed}.csv \\\n  -e 3 \\\n  svtav1 -- --preset $speed --tune 2\ndone\n```\n\nThis snippet here will run the same command as before, but across all speed\npresets from 2 to 6 (naming the output CSV files accordingly). It will also\nencode all of the files ending in `.y4m` in the `~/Videos/reference` directory.\n\nYou can also use libvpx for VP9 benchmarking:\n\n```bash\n./stats.py \\\n  -i source.mkv \\\n  -q \"20 25 30 35 40\" \\\n  -o ./libvpx_vp9.csv \\\n  -g 4 \\\n  vpxenc -- --cpu-used=4\n```\n\n### plot.py\n\n```bash\n./plot.py --help\nusage: plot.py [-h] -i INPUT [INPUT ...] [-f FORMAT]\n\nPlot codec metrics from one or more CSV files (one per codec) for side-by-side comparison.\n\noptions:\n  -h, --help            show this help message and exit\n  -i, --input INPUT [INPUT ...]\n                        Path(s) to CSV file(s). Each CSV file should have the same columns.\n  -f, --format FORMAT   Save the plot as 'svg', 'png', or 'webp'\n```\n\nExample:\n\n```bash\n./plot.py -i codec1_results.csv codec2_results.csv -f webp\n```\n\nThis command reads `codec1_results.csv` and `codec2_results.csv`, generating\nseparate plots (one for each metric) as WebP images.\n\nIt will also print BD-rate statistics for each metric, comparing the two results\nfrom the CSV files.\n\n`plot.py` also outputs a CSV file containing the average encode time for an\ninput file accompanied by the corresponding average BD-rate. These statistics\ncan assist in looking at overall encoder efficiency across multiple speed\npresets or configurations.\n\n### Run via Docker\n\n\u003ci\u003e See the pre-requisites for host machine:\nhttps://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html\n\u003c/i\u003e\n\n1. Build image against your GPU architechture:\n   ```bash\n   GPU_ARCH=$(amdgpu-arch) docker-compose --build\n   ```\n\n2. Run `docker-compose`:\n   ```bash\n   docker-compose -f \u003cdocker-compose-file\u003e run -v \u003chost-path\u003e:/videos metrics-rocm \u003cscores|plot|encode|stats\u003e \u003c..args\u003e\n   ```\n\n## License\n\nThis project was originally authored by @gianni-rosato \u0026 is provided as FOSS\nthrough the Psychovisual Experts Group.\n\nUsage is subject to the terms of the [LICENSE](LICENSE). Please refer to the\nlinked file for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpsy-ex%2Fmetrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpsy-ex%2Fmetrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpsy-ex%2Fmetrics/lists"}