{"id":19830724,"url":"https://github.com/crytic/cloudexec","last_synced_at":"2025-07-13T07:36:02.076Z","repository":{"id":196061858,"uuid":"693341280","full_name":"crytic/cloudexec","owner":"crytic","description":"A general purpose foundation for cloud-based fuzzing and mutation testing jobs","archived":false,"fork":false,"pushed_at":"2024-07-26T20:49:07.000Z","size":274,"stargazers_count":19,"open_issues_count":21,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-06T15:06:46.870Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crytic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-18T20:47:48.000Z","updated_at":"2025-02-21T09:27:59.000Z","dependencies_parsed_at":"2024-01-12T23:24:01.259Z","dependency_job_id":"61cd6ee1-fd4b-4226-9d2c-8cbc4db1edfd","html_url":"https://github.com/crytic/cloudexec","commit_stats":null,"previous_names":["crytic/cloudexec"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crytic%2Fcloudexec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crytic%2Fcloudexec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crytic%2Fcloudexec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crytic%2Fcloudexec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crytic","download_url":"https://codeload.github.com/crytic/cloudexec/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251898518,"owners_count":21661838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T11:25:01.106Z","updated_at":"2025-05-01T15:30:35.497Z","avatar_url":"https://github.com/crytic.png","language":"Go","funding_links":[],"categories":["Tooling"],"sub_categories":["Utils"],"readme":"# CloudExec\n\nCloudExec is a management tool for running computation jobs on DigitalOcean via the command line. It is general purpose; Cloudexec can set up the server with arbitrary dependencies and then run an arbitrary workload process, but it is designed to run a single long-running code analysis job such as a fuzz testing or mutation testing campaign. Output data and runtime logs are uploaded to DigitalOcean's S3-style object storage for later analysis and, when the job is complete, the server is automatically destroyed. The client-side management engine of CloudExec is written in Golang and the server itself is managed by a Bash script.\n\nFeatures:\n\n- 1Password CLI support for secure DigitalOcean API key management. CloudExec will help you configure these credentials and verify that they are valid.\n- Launch config file allows specification of:\n  - An input folder which is uploaded to the runtime server and also to DigitalOcean's S3-style object storage for later reference. This folder is zipped for speedy uploads.\n  - A job name, providing human-readable tags for each job.\n  - A timeout, after which the workload process will be terminated if it hasn't finished already, output will be uploaded to persistent storage, and the server will be destroyed so you will stop being charged for it.\n  - A setup command which uses bash to install dependencies and prepare the server to run the workload process.\n  - A run command which will kick off the workload process\n- An `init` subcommand for creating a new, default launch config file\n- Output is periodically uploaded to DigitalOcean's S3-style object storage so you can pull results incrementally from a running job\n- Jobs can be cancelled early if the workload process hasn't completed or the timeout hasn't been reached yet\n- Monitoring the runtime logs of a specific job or the status of all jobs\n- Automatic `ssh_config` additions allowing you to access a running server by executing `ssh cloudexec`\n- Tracks the cumulative costs incurred for running processes and the total cost of completed processes\n\nRun `cloudexec help` to list available subcommands or `cloudexec \u003csubcommand\u003e --help` for information regarding a specific subcommand:\n\n```text\n$ cloudexec help\nNAME:\n   cloudexec - easily run cloud based jobs\n\nUSAGE:\n   cloudexec [global options] command [command options] [arguments...]\n\nCOMMANDS:\n   check, c    Verifies cloud authentication\n   configure   Configure credentials\n   init        Create a new cloudexec.toml launch configuration in the current directory\n   launch, l   Launch a droplet and start a job\n   logs        Stream logs from a running job\n   cancel      Cancels any running cloudexec jobs\n   clean       Cleans up any running cloudexec droplets and clears the spaces bucket\n   pull        Pulls down the results of the latest successful job\n   status, s   Get status of running jobs\n   state       Manage state file\n   attach, a   Attach to a running job\n   version, v  Gets the version of the app\n   help, h     Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --help, -h  show help\n```\n\nExample job status output:\n\n```text\n$ cloudexec status --all\n+--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+\n| JOB ID |    JOB NAME    |  STATUS   |   DROPLET IP   | MEMORY  | CPUS | DISK  |     STARTED AT      |     UPDATED AT      | TIME ELAPSED | HOURLY COST | TOTAL COST |\n+--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+\n| 1      | medusa fuzz    | completed | 12.34.56.78    | 4096 MB | 4    | 50 GB | 2024-01-01 13:55:53 | 2024-01-02 14:05:29 | 2 days       | $0.125      | $6.0100    |\n+--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+\n| 2      | experiment     | failed    | 12.34.56.79    | 2048 MB | 2    | 25 GB | 2024-01-01 14:04:50 | 2024-01-01 14:08:03 | 3 minutes    | $0.0625     | $0.0034    |\n+--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+\n| 3      | slither-mutate | running   | 12.34.56.80    | 2048 MB | 2    | 25 GB | 2024-01-02 10:04:50 | 2024-01-02 12:08:03 | 2 hours      | $0.0625     | $0.125     |\n+--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+\n```\n\n## Getting Started\n\n### Installation\n\n#### Install with Brew\n\n```bash\nbrew tap trailofbits/tools\nbrew install cloudexec\n```\n\n#### Upgrade with Brew\n\n```bash\nbrew update \u0026\u0026 brew upgrade cloudexec\n```\n\nalternatively, you can install from a GitHub release:\n\n### Install from a GitHub release\n\nDownload the latest release for your platform from the [releases page](https://github.com/crytic/cloudexec/releases).\n\n#### Release verification\n\nReleases are signed with sigstore. You can verify using [`cosign`](https://github.com/sigstore/cosign) with the following example command:\n\n```bash\ncosign verify-blob \\\n    --certificate-identity-regexp \"https://github.com/crytic/cloudexec.*\" \\\n    --certificate-oidc-issuer https://token.actions.githubusercontent.com  \\\n    --bundle cloudexec-\u003cversion\u003e-\u003cos\u003e-\u003carch\u003e.tar.gz.bundle \\\n    cloudexec-\u003cversion\u003e-\u003cos\u003e-\u003carch\u003e.tar.gz\n```\n\n#### Install from a tarball\n\n```bash\ntar -xzf cloudexec-\u003cversion\u003e-\u003cos\u003e-\u003carch\u003e.tar.gz\nmv cloudexec /usr/local/bin\n```\n\n#### Install from source\n\nRunning the command below will build the CLI tool from source with a binary named `cloudexec` in a `dist` folder:\n\n```bash\nmake build\n```\n\nThen, move the resulting binary from `./dist/clouexec` into your `PATH`.\n\nNix users can run `nix build` and then `nix profile install ./result` to install `cloudexec`. A helper command `make nix-install` is available which performs these steps for you and also upgrades an existing version of `cloudexec` that might already be installed.\n\n### Configure credentials\n\nCloudExec requires DigitalOcean API credentials to manage droplets, and Spaces credentials to store state and job data. The recommended method for storing and providing your credentials securely is by using the 1Password CLI.\n\nCloudExec supports natively integrating with 1Password, allowing you to reference your credentials stored in your 1Password vault. However, you can also choose to provide plaintext credentials using the `cloudexec configure` command. Additionally, you can override individual values or the entire configuration by setting the corresponding environment variables.\n\n#### Get credentials from DigitalOcean\n\n[API Token](https://cloud.digitalocean.com/account/api/tokens)\n\n[Spaces Token](https://cloud.digitalocean.com/account/api/spaces)\n\n#### Configure 1password CLI (optional)\n\nSave the above tokens in your 1Password vault and [install the 1password CLI](https://developer.1password.com/docs/cli/get-started/#step-1-install-1password-cli).\n\n```bash\nbrew install --cask 1password/tap/1password-cli # see the link above for installation instructions on other platforms\n```\n\n[Sign in to your 1Password account](https://developer.1password.com/docs/cli/sign-in-manually/).\n\n```bash\neval $(op signin)\n```\n\nNote what your [1Password secret references](https://developer.1password.com/docs/cli/secret-references/) are and use them in place of your actual secret values during the `cloudexec configure` or env var setup steps described in the next section.\n\nThese references generally follow the format: `op://\u003cvault-name\u003e/\u003citem-name\u003e/\u003cfield-name\u003e`. For example, if you saved your keys to a vault called `Private`, in an item called `DigitalOcean` and the api key field is called `ApiKey`, then the secret reference to use is `op://Private/DigitalOcean/ApiKey`.\n\n#### Configure CloudExec credentials\n\n```bash\ncloudexec configure\n```\n\nor set environment variables:\n\n```bash\nDIGITALOCEAN_API_KEY\nDIGITALOCEAN_SPACES_ACCESS_KEY\nDIGITALOCEAN_SPACES_SECRET_ACCESS_KEY\nDIGITALOCEAN_SPACES_REGION\n```\n\nRemember, if you save secret values to a `.env` file, never commit it to any version control system. Add such `.env` files to your project's `.gitignore` file to help prevent mistakes. Even when not committed, plaintext secrets in a `.env` file can pose security risks so we recommend using a dedicated secret management tool such as 1Password.\n\nConfirm `cloudexec` is authorized to access to DigitalOcean.\n\n```bash\ncloudexec check\n```\n\n### Configure the new job\n\nGenerate a `cloudexec.toml` configuration file in the current directory.\n\n```bash\ncloudexec init\n```\n\nUpdate this `cloudexec.toml` file as needed. The following fields are available:\n\n`[input]`:\n\n- `jobName`: an arbitrary, human-readable label that can help identify this job\n- `directory`: the path to the input directory which will be uploaded to the cloud runner and from which the run command will be executed\n- `timeout`: a string specifying a maximum duration for which the job can run. After this timeout is reached, results will be uploaded to s3-style storage and the server will be destroyed. For example, \"6h\" for six hours or \"3d\" for three days.\n\n`[commands]`:\n\n- `setup`: A bash string that can be used to instal arbitrary software prior to the start of the job. These setup commands are run at the beginning of each job and time elapsed does not count towards the timeout.\n- `run`: A bash string that executes the workload command\n\n### Launch a new remote job\n\nRun `cloudexec launch` from the directory containing the launch config.\n\n```bash\n# default nyc3 region and c-2 size droplet, using a cloudexec.toml file in the current directory\ncloudexec launch\n# Or, specify a custom region and droplet size\ncloudexec launch --size c-4 --region sfo2\n```\n\n### Stream logs from the provisioning script\n\n```bash\ncloudexec logs\n```\n\nNote that the `logs` subcommand will continue to stream logs until you stop with ctrl-c, even after the job is finished and stops producing new logs. This is a read-only command and it is safe to kill it at any point.\n\n### Get logs from a previous run\n\n```bash\ncloudexec logs --job 1\n```\n\n### Attach to the running job\n\n```bash\ncloudexec attach\n\n# or\nssh -t cloudexec tmux attach -s cloudexec\n```\n\n### SSH to your droplet\n\n```bash\nssh cloudexec\n```\n\n### Check on the status of your jobs\n\n```bash\n# show only runnning jobs, and the last completed job\ncloudexec status\n# show all jobs\ncloudexec status --all\n```\n\nThe DigitalOcean dashboard will also provide helpful info including the droplet status, cpu and memory usage, and more; look for a droplet with a name that starts with `cloudexec-`.\n\n### Sync files from a completed job to a local path\n\n```bash\n# pull from the latest successful job\ncloudexec pull example/output\n# pull from any job ID\ncloudexec pull --job 1 example/output\n\n```\n\n### Cancel any in progress jobs\n\n```bash\ncloudexec cancel\n```\n\n### Cleanup all bucket contents and reset state (destructive)\n\n```bash\ncloudexec clean\n```\n\nNote that there is a delay of up to 2 weeks while deleting files from Digital Ocean Spaces buckets. Be aware that during this delay, the files will continue to occupy space in your bucket and may incur storage costs. If you need to manage your storage usage or costs, consider this delay when planning your data management strategy. This delay is managed by Digital Ocean, cloudexec is not able to influence it.\n\n## Optional: Create a CloudExec DigitalOcean image\n\nBuilding and uploading a dedicated DigitalOcean image for `cloudexec` will simplify your launch configuration and improve startup times.\n\nTo do so, install `packer` with `brew install packer`. If you're using `nix` and `direnv`, it's added to your PATH via the flake's dev shell.\n\nTo build and upload a docker image, run the following command. Make sure your DigitalOcean API key is either in your env vars or replace it with the actual token.\n\n`packer build -var do_api_token=$DIGITALOCEAN_API_KEY cloudexec.pkr.hcl`\n\nThis will take care of everything and if you visit the [DigitalOcean snapshots page](https://cloud.digitalocean.com/images/snapshots/droplets), you'll see a snapshot called `cloudexec-20230920164605` or similar. `cloudexec` will search for snapshots starts with a `cloudexec-` prefix and it will use the one with the most recent timestamp string.\n\nNow, you can remove everything from the setup command in the example launch config or replace it to install additional tools.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrytic%2Fcloudexec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrytic%2Fcloudexec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrytic%2Fcloudexec/lists"}