{"id":20267123,"url":"https://github.com/xiph/awcy","last_synced_at":"2025-04-05T10:09:35.878Z","repository":{"id":19486497,"uuid":"22732331","full_name":"xiph/awcy","owner":"xiph","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-10T18:59:39.000Z","size":3976,"stargazers_count":72,"open_issues_count":62,"forks_count":47,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-29T09:11:53.360Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://arewecompressedyet.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xiph.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-08-07T18:42:21.000Z","updated_at":"2024-11-10T18:59:44.000Z","dependencies_parsed_at":"2023-12-29T15:27:08.155Z","dependency_job_id":"e454b322-604f-4e56-9a8e-fd19a1a5f6e3","html_url":"https://github.com/xiph/awcy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Fawcy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Fawcy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Fawcy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiph%2Fawcy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xiph","download_url":"https://codeload.github.com/xiph/awcy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247318745,"owners_count":20919484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T12:13:18.819Z","updated_at":"2025-04-05T10:09:35.856Z","avatar_url":"https://github.com/xiph.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\nAre We Compressed Yet?\n====\nThis repository contains the arewecompressedyet.com website source code.\n\nRunning your own local copy of the website\n===\nTo run a local copy, you will need a copy of node and npm.\n\nYou will need node, npm, etc. For Ubuntu:\n```\nsudo apt install nodejs npm nodejs-legacy libicu-dev\n```\nYou'll also need tools to build AV1 or other codecs being tested:\n```\nsudo apt install yasm libtool\n```\n\nFirst, run the `./setup.sh` script. It will create directories needed for AWCY to run.\n\nNext, create a configuration file called `config.json`.\n\nHere is an example config.json:\n\n```\n{ \"channel\": \"#daalatest\",\n  \"have_aws\": false,\n  \"port\": 3000,\n  \"rd_server_url\": \"http://localhost:4000\"\n}\n```\n\nYou will also need a file called `secret_key` which contains the key needed to use the website.\n\n```\necho 'fake_password_to_compile' \u003e secret_key\n```\n\nThese commands will create the configuration files and install the node.js modules that get used by\nawcy. Open a node command line and run the following:\n\n```\nmkdir runs\ncd www\nnpm install\ncd ..\nnpm install\nnpm run tsc\nnode generate_list.js\nnpm run start\n```\n\nTo run the server, execute the run_awcy.bat file\nor run the following in your command line:\n```\n  node awcy_server.js\n```\nNow you can open localhost:3000 with your browser to see your local version of the website.\n\nSetting up repositories\n===\nFor the website to build codecs, you need local checkouts of every codec. While the `./setup.sh` script\nautomatically adds AV1, you can add your own. For example:\n```\ngit clone https://aomedia.googlesource.com/aom av1\nln -s av1 av1-rt\n```\n\nSetting up rd_server\n===\nThe AWCY web server manages the repositories and runs/ directory, and compiles and builds the codecs. Once this is done, it hands off the job of actually talking to all the AWS machines to rd_server.\n\nTo install rd_server, checkout the rd_tool repository in the same directory as awcy:\n```\ngit clone https://github.com/tdaede/rd_tool.git\n```\n\nThen start the rd_server.py daemon:\n```\n./rd_server.py\n```\nThe rd_server.py daemon listens on port 4000 by default.\n\nMore documentation on rd_server.py can be found in its README.\n\nSetting up workers\n===\nWorkers are Linux machines accessible over ssh. They need to be the same architecture as the AWCY server.\n\nA worker needs a AWCY work root, which has a directory structure as follows:\n```\ndump_ciede2000/\ndaalatool/\nslot0/\nvmaf/\n```\n\nThe slot* directores are created by AWCY for jobs.\n\ndaalatool must be pre-populated by a clone of the daala repo, with tools built (make tools):\n\n```\ngit clone https://github.com/xiph/daala.git daalatool\ncd daalatool\n./autogen.sh\n./configure\nmake\nmake tools\n```\n\ndump_ciede2000 must be populated by a built version of the dump_ciede2000 tool:\n\n```\ngit clone https://github.com/KyleSiefring/dump_ciede2000\ncd dump_ciede2000\ncargo build --release\n```\n\nvmaf is a clone of the Netflix VMAF repository, also built.\n\n```\ngit clone https://github.com/Netflix/vmaf.git\ncd vmaf/libvmaf\nmeson build --buildtype release\nninja -C build\n```\n\nIn addition, a copy of the test media must be accessible to the worker.\n\nThe workers should be configured on the main server with a machines.json file, which contains the host, user and port (for ssh). In addition, it contains the number of cores (which controlls the number of slots), plus the work root (containing the above mentioned directores) and the test media path. For example:\n\n```\n[\n  {\n    \"host\": \"localhost\",\n    \"user\": \"xiph\",\n    \"cores\": 8,\n    \"port\": 22,\n    \"work_root\": \"/home/xiph/awcy_temp\",\n    \"media_path\": \"/home/xiph/sets/\"\n  }\n]\n```\n\nEach worker must have a unique work root - usually local storage on the worker. One worker is intended to be one physical machine or VM. For testing purposes, one machine can act as multiple workers, but the work_root must be independent. The media path is read-only and can be shared, e.g. via NFS.\n\nThe machines should be accesible via passwordless SSH authentication. If needed, the SSH_PRIVKEY_FILE environment variable can inform rd_server.py of a custom private key (.pem format) to use for logging into workers.\n\nRun database format\n===\nThe runs/ directory will contain all of the output files generated from a job. There is a info.json file that specifies what options were used by that particular run. Here is an example of an info.json file:\n\n  {\"codec\":\"daala\",\"commit\":\"\",\"run_id\":\"2014-09-19T22-00-08.196Z\",\"task\":\"video-1-short\",\"nick\":\"AWCY\",\"task_type\":\"video\"}\n\nThere is also an output.txt file that contains the output of the rd_tool.py script.\n\nAfter each run, a cache file called list.json file is generated by the generate_list.js script. This contains all of the info.json files, as an ordered list. This should probably be replaced by a \"real\" database at some point.\n\nDocker support\n===\n\nAWCY can be started as a docker container for local usage or development.\n\nBuild (server)\n----\n\nThe docker image is pretty large (3.3 GB), as it includes:\n\n * build dependencies for all codecs to be built (gcc/g++, autotools, make, rust, ...)\n * build dependencies for the WebUI (nodejs, emscripten, ...)\n * runtime dependencies for build and analysis scripts (python, daalatool, ...)\n\nNote: this image has not been optimized for size yet, and may be split in the future.\n\nTo build the image:\n\n```sh\ndocker build --tag xiph/awcy:latest .\n```\n\nBuild (worker)\n----\n\nAnother dockerfile (`Dockerfile.worker`) is available to create worker nodes.\n\nTo build the worker image:\n\n```sh\ndocker build --file Dockerfile.worker --tag xiph/awcy-worker:latest .\n```\n\nRun (all-in-one)\n----\n\nYou can run an all-in-one container which will start:\n\n * awcy server\n * rd_server\n * sshd (to emulate a local rd_tool worker)\n\nIn addition, a git clone of every codecs' source is going to be done from upstream at initial startup.\n\nConfiguration is driven by a few environment variables (with their default values):\n\n * `CONFIG_DIR=/data/conf`: configuration files, sqlite database, generate SSH keys, set list, ...\n * `CODECS_SRC_DIR=/data/src`: where codecs' sources are going to be cloned (and compiled)\n * `RUNS_DST_DIR=/data/runs`: rd_tool jobs runs output\n * `WORK_DIR=/data/work`: rd_tool job workdir directory\n * `MEDIAS_SRC_DIR=/data/media`: directory containing media files (get them from https://media.xiph.org/sets/)\n * `LOCAL_WORKER_ENABLED=false`: if a local worker need to be started and configured\n * `LOCAL_WORKER_SLOTS=$(nproc)`: number of slots for the local worker\n * `IRC_CHANNEL=none`: IRC notifications target channel (`none` means disabled)\n * `AWCY_API_KEY=awcy_api_key`: WebUI/API key\n * `AWCY_SERVER_PORT=3000`: awcy server listening port\n * `RD_SERVER_PORT=4000`: rd_server listening port\n\nSimple run example (all data will be contained on the docker host in `/tmp/awcy`):\n\n```sh\ndocker run \\\n\t-it \\\n\t--rm \\\n\t--name xiph-awcy \\\n\t--publish 3000:3000 \\\n\t--volume /tmp/awcy:/data \\\n\t--volume ${HOME}/xiph-media-files:/media \\\n\t--env MEDIAS_SRC_DIR=/media \\\n\t--env AWCY_API_KEY=complicated_password \\\n\t--env LOCAL_WORKER_ENABLED=true \\\n\t--env LOCAL_WORKER_SLOTS=4 \\\n\txiph/awcy:latest\n```\n\nOutput:\n\n```\n--2019-01-16 22:46:11--  https://people.xiph.org/~tdaede/sets/subset1-monochrome/Claudette_14_july_2003_1920Z.y4m\nResolving people.xiph.org (people.xiph.org)... 140.211.15.28, 2001:470:eb26:54::1\nConnecting to people.xiph.org (people.xiph.org)|140.211.15.28|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 1485050 (1.4M) [application/octet-stream]\nSaving to: ‘/media/awcy-builder-quicktest/test_frame.y4m’\n\n/media/awcy-builder-quicktest/test_frame.y4m         100%[=====================================================================================================================\u003e]   1.42M  1.14MB/s    in 1.2s\n\n2019-01-16 22:46:13 (1.14 MB/s) - ‘/media/awcy-builder-quicktest/test_frame.y4m’ saved [1485050/1485050]\n\nCloning into '/data/src/av1'...\nremote: Sending approximately 132.14 MiB ...\nremote: Counting objects: 8, done\nremote: Finding sources: 100% (5/5)\nremote: Total 193273 (delta 156479), reused 193273 (delta 156479)\nReceiving objects: 100% (193273/193273), 132.11 MiB | 2.33 MiB/s, done.\nResolving deltas: 100% (156479/156479), done.\nCloning into '/data/src/daala'...\nremote: Enumerating objects: 12671, done.\nremote: Total 12671 (delta 0), reused 0 (delta 0), pack-reused 12671\nReceiving objects: 100% (12671/12671), 8.70 MiB | 2.37 MiB/s, done.\nResolving deltas: 100% (9442/9442), done.\nCloning into '/data/src/rav1e'...\nremote: Enumerating objects: 1, done.\nremote: Counting objects: 100% (1/1), done.\nremote: Total 4426 (delta 0), reused 0 (delta 0), pack-reused 4425\nReceiving objects: 100% (4426/4426), 2.61 MiB | 2.33 MiB/s, done.\nResolving deltas: 100% (3108/3108), done.\nCloning into '/data/src/thor'...\n\n(...)\n\nGenerating SSH host keys\nGenerating public/private rsa key pair.\nYour identification has been saved in /data/conf/awcy.pem.\nYour public key has been saved in /data/conf/awcy.pem.pub.\nThe key fingerprint is:\nSHA256:TJa3T7FZhByGZ3owJOtV/OXk75nyBJa8I7pq2BwNEaU xiph@4ee6bf6c8ffe\nThe key's randomart image is:\n+---[RSA 2048]----+\n|       .+o.++o.  |\n|       ..++o*.  o|\n|       E* o*...= |\n|       * o..o=o o|\n|        S ..+=  .|\n|       . . o. o .|\n|      + .  ..o oo|\n|     . +  . ..oo.|\n|      ...o.   o. |\n+----[SHA256]-----+\nSTARTING SSHD SERVICE\nSTARTING AWCY SERVICE\nSTARTING RDTOOL SERVICE\nAWCY server started! Open a browser at http://172.20.0.12:3000\nAWCY server directory: /opt/app\nConfiguration directory: /data/conf\nCodecs git repositories location: /data/src\nMedia samples directory: /media\nRuns output directory: /data/runs\n```\n\nIf you open your browser at http://localhost:3000 (or the one in the container output), you should see the AWCY WebUI.\n\nOn initial startup, the following configuration files are generated:\n\n```\n/tmp/awcy/conf/awcy.pem\n/tmp/awcy/conf/awcy.pub\n/tmp/awcy/conf/config.json\n/tmp/awcy/conf/list.json\n/tmp/awcy/conf/machines.json\n/tmp/awcy/conf/secret_key\n/tmp/awcy/conf/sets.json\n/tmp/awcy/conf/subjective.sqlite3\n```\n\nRun (with workers)\n----\n\nYou first need to start an AWCY server instance using the [all-in-one instructions](#run-all-in-one), but setting `LOCAL_WORKER_ENABLED` to false.\n\nNext, run once instance of the worker container on each worker machine. FOr testing, you can also run multiple workers on your local machine. In a second terminal, start one or more worker instances:\n\n```sh\ndocker run \\\n\t-it \\\n\t--detach \\\n\t--name xiph-awcy-worker-1 \\\n\t--volume ${HOME}/xiph-media-files:/media \\\n\t--env SSH_PUBKEY=\"$(cat /tmp/awcy/conf/awcy.pub)\"\n\txiph/awcy-worker:latest\n\ndocker run \\\n\t-it \\\n\t--detach \\\n\t--name xiph-awcy-worker-2 \\\n\t--volume ${HOME}/xiph-media-files:/media \\\n\t--env SSH_PUBKEY=\"$(cat /tmp/awcy/conf/awcy.pub)\"\n\txiph/awcy-worker:latest\n\n...\n```\n\nNote that the given public key is the server-generated one.\n\nThen stop the server instance by hitting `CTRL+C` in its terminal.\n\nGet IP addresses of your worker nodes:\n\n```sh\ndocker inspect xiph-awcy-worker-1 | jq -r '.[].NetworkSettings.IPAddress'\n172.20.0.12\n\ndocker inspect xiph-awcy-worker-2 | jq -r '.[].NetworkSettings.IPAddress'\n172.20.0.13\n...\n```\n\nYou can now edit the /tmp/awcy/conf/machines.json (paths should be left as-is):\n\n```\ncat \u003e/tmp/awcy/conf/machines.json \u003c\u003cEOF\n[\n  {\n    \"host\": \"172.20.0.12\",\n    \"user\": \"xiph\",\n    \"cores\": 8,\n    \"port\": 22,\n    \"work_root\": \"/data/work\",\n    \"media_path\": \"/media\"\n  },\n  {\n    \"host\": \"172.20.0.13\",\n    \"user\": \"xiph\",\n    \"cores\": 8,\n    \"port\": 22,\n    \"work_root\": \"/data/work\",\n    \"media_path\": \"/media\"\n  }\n]\nEOF\n```\n\nThen start the server again using the previously used command.\n\nYou should now have two (or more) workers accessible.\n\nSamples\n----\n\nTo make sure that everything is working properly, you can try a run using the `awcy-builder-quicktest` set, which runs in one minute or so.\n\nCleanup\n----\n\nYou can stop the container by hitting `CTRL+C`.\n\nIf you have started your container using the above example, all data is contained in `/tmp/awcy`, so you can removed this directory from your host.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiph%2Fawcy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxiph%2Fawcy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiph%2Fawcy/lists"}