{"id":26664614,"url":"https://github.com/gatlenculp/homebrew-vivaria","last_synced_at":"2025-03-25T16:37:09.738Z","repository":{"id":256094063,"uuid":"854309134","full_name":"GatlenCulp/homebrew-vivaria","owner":"GatlenCulp","description":"Homebrew tap for vivarium, METR's AI evaluation tool","archived":false,"fork":false,"pushed_at":"2024-09-15T03:49:18.000Z","size":69,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-09-16T03:24:59.696Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GatlenCulp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-08T23:25:22.000Z","updated_at":"2024-09-15T03:49:21.000Z","dependencies_parsed_at":"2024-09-15T03:22:04.801Z","dependency_job_id":null,"html_url":"https://github.com/GatlenCulp/homebrew-vivaria","commit_stats":null,"previous_names":["gatlenculp/homebrew-vivarium"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GatlenCulp%2Fhomebrew-vivaria","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GatlenCulp%2Fhomebrew-vivaria/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GatlenCulp%2Fhomebrew-vivaria/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GatlenCulp%2Fhomebrew-vivaria/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GatlenCulp","download_url":"https://codeload.github.com/GatlenCulp/homebrew-vivaria/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245501301,"owners_count":20625753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-25T16:37:09.068Z","updated_at":"2025-03-25T16:37:09.719Z","avatar_url":"https://github.com/GatlenCulp.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# homebrew-vivaria\n\n![Homebrew Tests](https://github.com/gatlenculp/homebrew-vivaria/actions/workflows/tests.yml/badge.svg)\n![GitHub stars](https://img.shields.io/github/stars/gatlenculp/homebrew-vivaria?style=social)\n\n\u003e [!WARNING]\n\u003e This is currently a pre-production Forumla that has not been thoroughly tested and which installs a currently non-official version of Vivaria\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/metr_logo.svg\" alt=\"METR Logo\" style=\"max-width: 200px; margin-right: 20px;\"\u003e\n  \u003cimg src=\"./assets/logo.png\" alt=\"Vivaria Logo\" style=\"max-width: 200px;\"\u003e\n\u003c/div\u003e\n\n[Vivaria](https://vivaria.metr.org/) is [METR](https://metr.org/)'s tool for running evaluations and conducting agent elicitation research. This package contains a web app which is used for running and organzing evaluations as well as a command line interface to aid in the development of tasks. More information can be found on the [website](https://vivaria.metr.org/).\n\nFor prototyping purposes, Gatlen has created [his own fork of Vivaria](https://github.com/GatlenCulp/vivaria/) which this formulae installs. See the original repo [here](https://github.com/METR/vivaria)\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/homebrew-logo.jpg\" alt=\"Homebrew Logo\" style=\"max-width: 300px; margin-bottom: 20px;\"\u003e\n\u003c/div\u003e\n\n*[Homebrew (\"brew\")](https://brew.sh/) is a macOS (and Linux) package manager. New contributers to this Homebrew formulae (especially those new to Homebrew formula development) should see [CONTRIBUTING.md](./CONTRIBUTING.md).*\n\n---\n## 00 TOC\n\n- [homebrew-vivaria](#homebrew-vivaria)\n  - [00 TOC](#00-toc)\n  - [01 Installation](#01-installation)\n  - [02 Post-install Setup](#02-post-install-setup)\n  - [03 Getting Started](#03-getting-started)\n    - [03.01 Starting the Web GUI](#0301-starting-the-web-gui)\n    - [03.02 Accessing the Web GUI](#0302-accessing-the-web-gui)\n    - [03.03 Starting and Testing a Task via the CLI](#0303-starting-and-testing-a-task-via-the-cli)\n    - [03.04 Evaluating an Agent on a Task using the CLI and Web GUI](#0304-evaluating-an-agent-on-a-task-using-the-cli-and-web-gui)\n    - [03.05 Shutting Down the Web Server](#0305-shutting-down-the-web-server)\n    - [03.06 Experimenting with Our Examples](#0306-experimenting-with-our-examples)\n    - [03.07 Learn More](#0307-learn-more)\n  - [04 Uninstalling](#04-uninstalling)\n  - [05 Updating and Reinstalling](#05-updating-and-reinstalling)\n  - [06 Known Issues](#06-known-issues)\n  - [07 Contact the Maintainer](#07-contact-the-maintainer)\n\n---\n## 01 Installation\n\n**00. Install Requirements (Docker)**\nMake sure to have `docker compose` version \u003e 2.0. You can check this by running:\n```bash\ndocker compose version\n```\n\n\u003e [!TIP]\n\u003e If you don't have `docker compose`, you can install docker desktop with:\n\u003e ```bash\n\u003e brew install --cask docker\n\u003e ```\n\n\n**01. Tap this repository**\n```bash\nbrew tap GatlenCulp/vivaria\n```\n\n**02. Install Vivaria**\n```bash\nbrew install vivaria\n```\n\n---\n## 02 Post-install Setup\n\n**03. Run the post-installation setup** (This will ask you for a valid [OpenAI API Key](https://medium.com/@lorenzozar/how-to-get-your-own-openai-api-key-f4d44e60c327))\n\n*Be cautious running this command multiple times as it will overwrite your current configuration and will require you to follow all the instructions from here onward*\n```bash\nviv setup\n```\n\u003cdetails\u003e\n\u003csummary\u003eExample Output\u003c/summary\u003e\n\n\u003e Please enter your OpenAI API key: sk-Hk[REDACTED]\n\u003e Using output directory: /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria\n\u003e Creating new file /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/.env.server\n\u003e Successfully wrote to /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/.env.server\n\u003e Creating new file /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/.env.db\n\u003e Successfully wrote to /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/.env.db\n\u003e Creating new file /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/.env\n\u003e Successfully wrote to /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/.env\n\u003e Created /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/docker-compose.override.yml\n\u003e Updated /opt/homebrew/Cellar/vivaria/HEAD-6cc4707/vivaria/docker-compose.dev.yml: Changed 'user: node:docker' to 'user: node:0'\n\u003e viv CLI configuration completed successfully.\n\u003e Vivaria setup completed successfully. To finish installation, run:\n\u003e         viv docker compose up --detach --wait\n\u003e Building the docker image may take upwards of an hour.\n\n\u003c/details\u003e\n\n---\n## 03 Getting Started\n\n### 03.01 Starting the Web GUI\n\n**04. Open docker**\n\nOpen Docker Desktop automatically with:\n```bash\nopen -a Docker\n```\n![./assets/docker-desktop.png](./assets/docker-desktop.png)\n\n**05. Build and run the server images** (This may take a while)\n```bash\nviv docker compose up --detach --wait\n```\n\n**06. Check that the containers are running**\n```bash\nviv docker compose ps\n```\n\u003cdetails\u003e\n\u003csummary\u003eExample Output\u003c/summary\u003e\n\n\u003e 🪴 Handing over execution to docker. Running command:\n🪴       docker compose ps (at /opt/homebrew/Cellar/vivaria/0.1.5/vivaria)\nNAME                                  IMAGE                               COMMAND                  SERVICE                     CREATED       STATUS                        PORTS\nvivaria-background-process-runner-1   vivaria-background-process-runner   \"docker-entrypoint.s…\"   background-process-runner   8 hours ago   Up 58 seconds                 4001/tcp\nvivaria-database-1                    vivaria-database                    \"docker-entrypoint.s…\"   database                    8 hours ago   Up About a minute (healthy)   0.0.0.0:5432-\u003e5432/tcp\nvivaria-proxy-1                       quay.io/panubo/sshd                 \"/entry.sh /usr/sbin…\"   proxy                       8 hours ago   Up About a minute             0.0.0.0:2222-\u003e22/tcp\nvivaria-server-1                      vivaria-server                      \"docker-entrypoint.s…\"   server                      8 hours ago   Up 58 seconds (healthy)       0.0.0.0:4001-\u003e4001/tcp\nvivaria-ui-1                          vivaria-ui                          \"docker-entrypoint.s…\"   ui                          8 hours ago   Up About a minute (healthy)   0.0.0.0:4000-\u003e4000/tcp\n\n\u003c/details\u003e\n\n\n### 03.02 Accessing the Web GUI\n\n**07. Check that the task server is running**\n```bash\ncurl http://localhost:4001/health\n```\n\n\u003e ```json\n\u003e {\"result\":{\"data\":\"ok\"}}\n\u003e ```\n\n**08. Open the Vivaria web GUI**\n\nYou can access the web GUI at [https://localhost:4000/](https://localhost:4000/) and continue past the \"Your connecton is not private\" warning.\n\n![./assets/cert-error.png](./assets/cert-error.png)\n\n**09. The website will prompt you for your `ACCESS_TOKEN` and `ID_TOKEN` from `.env.server`.**\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./assets/token-prompt.png\" alt=\"Token prompt\" style=\"max-width: 300px;\"\u003e\n\u003c/div\u003e\n\nYou can get these by running\n```bash\ncat \"$(brew --prefix vivaria)/vivaria/.env.server\" | grep -E \"ACCESS_TOKEN=|ID_TOKEN=\"\n```\n\u003e ACCESS_TOKEN=[REDACTED]\n\u003e ID_TOKEN=[REDACTED]\n\n\u003c!-- TODO: Explain why these prompts exist in the first place. --\u003e\n\nYou should then have access to the web interface.\nThe landing page should look like this:\n\n![./assets/home-page.png](./assets/home-page.png)\n\n\n### 03.03 Starting and Testing a Task via the CLI\n\n**10. Start your first task environment**\n\nLoad up the example task \"abandon\" from the \"reverse_hash\" task family (this may take a while):\n```bash\nviv task start reverse_hash/abandon \\\n    --task-family-path \"$(brew --prefix vivaria)/vivaria/task-standard/examples/reverse_hash\"\n```\n\n**11. Check that the task is running**\n```bash\nviv task list\n```\n\u003e `task-environment--reverse_hash--abandon--4222326789--8628449513`\n\n**12. Enter the task environment container**\n```bash\nviv task enter --user agent\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eExample Output\u003c/summary\u003e\n\n\u003e No task environment specified. Using the task environment from the previous command: `task-environment--reverse_hash--abandon--446756826--4480279478`\n\u003e\n\u003e `agent@2789651c36fd:~$ `\n\n\u003c/details\u003e\n\n**13. Look at the instructions in `/home/agent/`**\n\n```bash\ncat /home/agent/instructions.txt\n```\nYou should see something like:\n\u003e Find the word whose SHA-256 hash is as follows: `df864c0596aa1a432205ccfb6e659f6cbd0f5535b358ad1f0dae24b5002b4894`\n\u003e\n\u003e Return only the word as your answer.\n\n**13. Simulate an agent by submitting a `submission.txt` document**\n\nThe solution to this task is \"abandon\"\n```bash\necho \"abandon\" \u003e /home/agent/submission.txt\n```\n\n**14. Exit the task environment container**\n```bash\nexit\n```\n\n**15. Check your score from the `submission.txt`**\n```bash\nviv task score\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eExample Output\u003c/summary\u003e\n\n\n\u003e No task environment specified. Using the task environment from the previous command: `task-environment--reverse_hash--abandon--446756826--4480279478`\n\u003e === Scoring submission ===\n\u003e SEP_MUfKWkpuVDn9E\n\u003e 1.0\n\u003e === Score ===\n\u003e Task scored. Score: `1`\n\u003e === Task finished ===\n\u003e Leaving the task environment running. You can destroy it with:\n\u003e\n\u003e  `viv task destroy task-environment--reverse_hash--abandon--446756826--4480279478`\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eScore other answers\u003c/summary\u003e\n\nYou can try other answers using:\n```bash\nviv task score --submission \"wrong answer\"\n```\n\u003e No task environment specified. Using the task environment from the previous command: `task-environment--reverse_hash--abandon--446756826--4480279478`\n\u003e === Scoring submission ===\n\u003e SEP_MUfKWkpuVDn9E\n\u003e 0.0\n\u003e === Score ===\n\u003e Task scored. Score: `0`\n\u003e === Task finished ===\n\u003e Leaving the task environment running. You can destroy it with:\n\u003e \n\u003e `viv task destroy task-environment--reverse_hash--abandon--446756826--4480279478`\n\n\u003c/details\u003e\n\n\n\n**16. Stop the task**\n```bash\nviv task destroy\n```\n\n### 03.04 Evaluating an Agent on a Task using the CLI and Web GUI\n\n**17. Download an agent to your computer**\n\nUnfortunately, Vivaria does not come included with an example agent, but we can add one easily to our installation directory. We will add the public [modular agent](https://github.com/poking-agents/modular-public), developed by METR:\n```bash\nmkdir -p \"$(brew --prefix vivaria)/agents\"\n```\n\n```bash\ngit clone https://github.com/poking-agents/modular-public \\\n  \"$(brew --prefix vivaria)/agents/modular-public\"\n```\n\n\n**18. Run a task with the agent you downloaded**\nWe will now run this agent on the same `reverse_hash/abandon` task we did above.\n\n```bash\nviv run reverse_hash/abandon \\\n  --task-family-path $(brew --prefix vivaria)/vivaria/task-standard/examples/reverse_hash \\\n  --agent-path $(brew --prefix vivaria)/agents/modular-public\n```\n\n\u003e 1289838120\n\u003e https://localhost:4000/run/#1289838120/uq\n\n\u003e [!TIP]\n\u003e The syntax for creating and running task environments is different from the syntax for creating and running agents in tasks environments. The former uses `viv task ...` while the latter uses `viv ...`\n\u003e\n\u003e Ex: `viv task run` vs `viv run`\n\n**19. Track the agent's progress with the web GUI**\n\nThe last command prints a link to [https://localhost:4000/run/#\u003cRUN_ID\u003e](https://localhost:4000/...). Follow that link to see the run's trace and track the agent's progress on the task. Enable `Show generations`, and the run page should update live as the agent takes actions. It should look something like this:\n\n![./assets/run-page2.png](./assets/run-page2.png)\n\nWe recommend playing with the interface a bit to get an understanding of the tool.\n\n**20. Check the runs page for your most recent evaluation**\n\nHead back to the homepage at [https://localhost:4000/](https://localhost:4000/) and check out the runs page, and run the default query. This is where you can view the summaries of your ran tasks. It should look a bit like this (with less items):\n\n![./assets/runs-page2.png](./assets/runs-page2.png)\n\nYou can get a similar response without the GUI using\n```bash\nviv query\n```\n\n\u003e ```json\n\u003e {\"id\": 1170869829, \"taskId\": \"reverse_hash/abandon\", \"agent\": null, \"runStatus\": \"running\", \"isContainerRunning\": true, \"createdAt\": 1727879299676, \"isInteractive\": false, \"submission\": null, \"score\": null, \"username\": \"me\", \"metadata\": {}}\n\u003e ```\n\n**21. Kill the task**\n\nBefore killing the task, you may want to revisit entering the task environment and poking around with the too\n\n```bash\nviv kill \u003cRUN_ID from run step\u003e\n```\n\n\u003e run killed\n\n### 03.05 Shutting Down the Web Server\n\n**22. Stop the containers**\n\n```bash\nviv docker compose down\n```\n\n**23. Confirm there are no more active images**\n\n```bash\nviv docker compose ps\n```\n\u003e (You should see an empty table.)\n\u003e NAME      IMAGE     COMMAND   SERVICE   CREATED   STATUS    PORTS\n\n### 03.06 Experimenting with Our Examples\n\nLocated in `$(brew --prefix vivaria)/vivaria/task-standard/examples` are a variety of example tasks you can examine, run, and test to understand how to create your own tasks\n\n```bash\nls $(brew --prefix vivaria)/vivaria/task-standard/examples\n```\n\u003e agentbench              crypto                  gaia                    gpu_inference           humaneval               machine_learning_local  reverse_hash            vm_test count_odds              days_since              gpqa_diamond            hello_world             local_research          pico_ctf                swe_bench\n\n### 03.07 Learn More\n\nThis getting started guide is meant to be a quick introduction on using your Brew installation of Vivaria. To learn more about the project or API, please visit the [project homepage](https://vivaria.metr.org/)\n\n---\n## 04 Uninstalling\n\nTo uninstall, run:\n```bash\nbrew uninstall vivaria\n```\n\nThis will not delete your `~/.config/viv-cli/` directory. Remove it with:\n```bash\nrm -r ~/.config/viv-cli/\n```\n\n---\n## 05 Updating and Reinstalling\n\nTo update Vivaria to the latest version:\n\n**1. Update the Homebrew formulae**\n```bash\nbrew update\n```\n\n**2. Upgrade Vivaria**\n```bash\nbrew upgrade vivaria\n```\n\n**3. Restart the Docker containers**\n```bash\nviv docker compose down --rmi all\n```\n\u003e [!TIP]\n\u003e More docker image/build/cache/etc. removal commands may be necessary if you run into any errors.*\n\n```bash\nviv docker compose up --detach --wait\n```\n\n**4. Check that the server is running with the new version**\n\n```bash\ncurl http://localhost:4001/health\n```\n\u003e ```json\n\u003e {\"result\":{\"data\":\"ok\"}}\n\u003e ```\n\n**5. Clear your browser cache for localhost and refresh the page**\n\nIf you get the error `Unable to transform response from server` on the Web GUI screen, this means the `access_token` and `id_token` stored in your browser cache don't match the ones in the server. This is common if your docker environment variables change due to a new install, your previous tokens will be outdated and must be reset and the page reloaded.\n\n![./assets/clear-browser-cache.png](./assets/clear-browser-cache.png)\n\n---\n## 06 Known Issues\n\n**ISSUE: Install failed due to docker**\n```text\nError: An exception occurred within a child process:\n  RuntimeError: /opt/homebrew/opt/docker not present or broken\nPlease reinstall docker. Sorry :(\n```\nThis may be fixed by running `brew link docker` and trying the installation again.\n\n---\n## 07 Contact the Maintainer\n\nGatlen Culp, METR Contractor \\\nEmail: gatlen.culp@metr.org \\\nPortfolio: [gatlen.notion.site](https://gatlen.notion.site)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatlenculp%2Fhomebrew-vivaria","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgatlenculp%2Fhomebrew-vivaria","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatlenculp%2Fhomebrew-vivaria/lists"}