{"id":34520099,"url":"https://github.com/rustyrazorblade/easy-db-lab","last_synced_at":"2026-05-23T05:18:23.432Z","repository":{"id":89236402,"uuid":"253851902","full_name":"rustyrazorblade/easy-db-lab","owner":"rustyrazorblade","description":"Create lab environments for database testing in AWS.","archived":false,"fork":false,"pushed_at":"2026-04-20T07:41:47.000Z","size":6084,"stargazers_count":33,"open_issues_count":103,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-04-23T05:33:30.140Z","etag":null,"topics":["aws","cassandra","clickhouse","kubernetes","opensearch"],"latest_commit_sha":null,"homepage":"https://rustyrazorblade.github.io/easy-db-lab/","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rustyrazorblade.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-04-07T16:32:31.000Z","updated_at":"2026-04-13T13:56:30.000Z","dependencies_parsed_at":"2026-02-21T05:00:42.302Z","dependency_job_id":null,"html_url":"https://github.com/rustyrazorblade/easy-db-lab","commit_stats":null,"previous_names":["rustyrazorblade/easy-cass-lab","rustyrazorblade/tlp-cluster"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/rustyrazorblade/easy-db-lab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyrazorblade%2Feasy-db-lab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyrazorblade%2Feasy-db-lab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyrazorblade%2Feasy-db-lab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyrazorblade%2Feasy-db-lab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rustyrazorblade","download_url":"https://codeload.github.com/rustyrazorblade/easy-db-lab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustyrazorblade%2Feasy-db-lab/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32523637,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T01:12:54.858Z","status":"online","status_checked_at":"2026-05-02T02:00:05.923Z","response_time":132,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cassandra","clickhouse","kubernetes","opensearch"],"created_at":"2025-12-24T04:44:13.148Z","updated_at":"2026-05-05T05:06:23.996Z","avatar_url":"https://github.com/rustyrazorblade.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"# easy-db-lab\n\n**[Documentation](https://rustyrazorblade.github.io/easy-db-lab/)**\n\nA tool for creating database lab environments in AWS. Designed for testing, benchmarking, and learning.\n\n## Features\n\n### Databases\n\n- **Apache Cassandra** - Versions 2.2 through trunk, mixed version clusters, custom branches\n- **ClickHouse** - Sharded clusters with configurable replication, distributed tables, S3 storage\n- **Any K8s-compatible database** - Deploy via Kubernetes manifests\n\n### Infrastructure\n\n- **AWS EC2** - Automated provisioning with configurable instance types\n- **Kubernetes (K3s)** - Lightweight K8s on all nodes for deploying databases and tools\n- **Spark EMR** - Provision EMR clusters for analytics workloads\n- **S3 Integration** - Object storage for ClickHouse and data pipelines\n\n### Observability\n\n- **Prometheus and Grafana** - Pre-configured dashboards for all supported databases\n- **OpenTelemetry** - Distributed tracing and metrics collection\n- **Flame Graphs** - CPU and wall-clock profiling with async-profiler\n- **Kernel Metrics** - bcc-tools for low-level performance analysis\n\n### Developer Experience\n\n- **Server** - AI assistant integration, REST status endpoints, live metrics\n- **Interactive CLI** - REPL mode for reduced typing\n- **Homebrew and Docker** - Multiple installation options\n\n## Custom AMI\n\nWe use packer to create a single AMI with the following:\n\n* Multiple versions of Cassandra\n* ClickHouse (deployed via K3s)\n* [bcc tools](https://github.com/iovisor/bcc), [learn about these tools here](https://rustyrazorblade.com/post/2023/2023-11-14-bcc-tools/)\n* [async-profiler](https://github.com/async-profiler/async-profiler), [learn about it here](https://rustyrazorblade.com/post/2023/2023-11-07-async-profiler/)\n* [cassandra-easy-stress](https://github.com/apache/cassandra-easy-stress) (Apache project, formerly easy-cass-stress)\n* [AxonOps agent](https://axonops.com/) (free monitoring up to six nodes)\n* K3s distribution of Kubernetes\n\n## Pre-requisites\n\nThe following must be set up before using this project:\n\n* [Setup AWS Account API Credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)\n* [Install Docker Locally](https://www.docker.com/products/docker-desktop/) for Packer and Terraform\n\n## AWS Account Setup\n\n**IMPORTANT: We strongly recommend using a separate AWS account under an organization for lab environments.**\n\nThis tool provisions and destroys AWS infrastructure! Using a dedicated account provides:\n- **Resource isolation** - No risk of accidentally affecting production resources\n- **Cost isolation** - Lab costs separated from production\n- **Clean billing** - Easy to see lab-related costs\n\n### Setting Up Your Profile\n\nThis part is a bit clunky still, but it's a one time event.  You will need a user account and credentials.\n\n1. Create a User for easy-db-lab and get the credentials.\n2. Create a group and add the user to the group.\n3. Create 3 managed policies\n4. Attach managed policies to the group\n\n### Viewing Required IAM Policies\n\nTo see the IAM policies required for easy-db-lab with your account ID populated:\n\n```shell\neasy-db-lab show-iam-policies\n```\n\nYou can filter by policy type:\n\n```shell\neasy-db-lab show-iam-policies ec2   # Show only EC2 policy\neasy-db-lab show-iam-policies iam   # Show only IAM policy\neasy-db-lab show-iam-policies emr   # Show only EMR policy\n```\n\nSee `bin/set-policies`.\n\n### Interactive Setup\n\nRun the interactive setup to configure your AWS credentials and create necessary resources:\n\n```shell\neasy-db-lab setup-profile\n```\n\nThis will:\n- Prompt for your AWS credentials\n- Validate your credentials\n- Create an EC2 key pair for SSH access\n- Create an IAM role for instance permissions\n- Set up Packer VPC infrastructure for building AMIs\n\n## Usage\n\nTo install `easy-db-lab`, you can use Homebrew, download a release, or clone the project and build it.\n\n### Install A Release using Homebrew\n\n```shell\nbrew tap rustyrazorblade/rustyrazorblade\nbrew install easy-db-lab\n```\n\n`easy-db-lab` should now be available for you to use.\n\nSkip ahead to Read The Help.\n\n### Use Container Image\n\nA containerized version is available from GitHub Container Registry. This is useful for CI/CD pipelines or environments where you prefer container-based tools.\n\n#### Container Tags\n\nAvailable tags:\n- `latest` - Most recent build from main branch\n- `v12` - Specific version (e.g., v12, v13, etc.)\n- `12` - Version without 'v' prefix\n\n```shell\n# Pull latest version\ndocker pull ghcr.io/rustyrazorblade/easy-db-lab:latest\n\n# Pull specific version\ndocker pull ghcr.io/rustyrazorblade/easy-db-lab:v12\n```\n\n#### Running Commands\n\nTo run commands using the container:\n\n```shell\ndocker run --rm \\\n  -v ~/.aws:/root/.aws:ro \\                              # Read-only: AWS credentials\n  -v ~/.ssh:/root/.ssh:ro \\                              # Read-only: SSH keys\n  -v $(pwd):/workspace \\                                 # Read-write: Working directory for cluster state\n  -v /var/run/docker.sock:/var/run/docker.sock \\         # Required for Docker operations\n  ghcr.io/rustyrazorblade/easy-db-lab:latest --help\n```\n\n**Important notes for container usage:**\n- Mount your AWS credentials (`~/.aws`) as read-only for authentication\n- Mount your SSH keys (`~/.ssh`) as read-only for instance access\n- Mount a working directory (`$(pwd):/workspace`) for storing cluster state and configuration\n- Mount the Docker socket to allow the tool to use Docker for Terraform/Packer operations\n- For convenience, create a shell alias:\n\n```shell\nalias easy-db-lab='docker run --rm \\\n  -v ~/.aws:/root/.aws:ro \\\n  -v ~/.ssh:/root/.ssh:ro \\\n  -v $(pwd):/workspace \\\n  -v /var/run/docker.sock:/var/run/docker.sock \\\n  ghcr.io/rustyrazorblade/easy-db-lab:latest'\n```\n\nThen use it like the native command:\n\n```shell\neasy-db-lab setup-profile\neasy-db-lab init my-cluster --cassandra 5.0\n```\n\n#### Container Limitations\n\nWhen using the container version, note the following:\n- The `build-image` command requires access to your AWS credentials and Docker socket\n- Terraform state is stored in the working directory (mount `/workspace` to persist it)\n- SSH keys must be mounted and accessible inside the container for instance access\n- Performance may be slightly slower than native installation due to container overhead\n- The container runs with root privileges to access the Docker socket (required for Terraform/Packer operations)\n\n#### Security Note\n\nThe container requires root privileges to access the Docker socket, which is necessary for Terraform and Packer operations. For improved security:\n- Use Docker socket access control (e.g., Docker socket proxy) in production environments\n- Run containers only in isolated environments\n- Never use with untrusted inputs\n- Consider using native installation if Docker socket access is a concern\n\n#### Troubleshooting Container Issues\n\n**Permission errors accessing Docker socket:**\n```shell\n# Ensure your user has Docker permissions\nsudo usermod -aG docker $USER\n# Log out and back in for changes to take effect\n```\n\n**AWS credential mounting issues:**\n```shell\n# Verify credentials directory exists and is readable\nls -la ~/.aws\n# Ensure AWS credentials are properly configured\naws configure list\n```\n\n**SSH key permissions in container:**\n```shell\n# SSH keys should have correct permissions (600)\nchmod 600 ~/.ssh/id_rsa\n# If using non-default key, specify it in container command\ndocker run --rm \\\n  -v ~/.ssh:/root/.ssh:ro \\\n  -e SSH_KEY=/root/.ssh/my-custom-key \\\n  ghcr.io/rustyrazorblade/easy-db-lab:latest ...\n```\n\n### Download A Release\n\nDownload the latest [release](https://github.com/rustyrazorblade/easy-db-lab/releases) and add the project's bin directory\nto your PATH.\n\n```shell\nexport PATH=\"$PATH:/Users/username/path/to/easy-db-lab/bin\"\n```\n\nYou can skip ahead to Read The Help.\n\n### Build the Project\n\nThe following command should be run from the root directory of the project.\nDocker will need to be running for this step.\n\n```bash\ngit clone https://github.com/rustyrazorblade/easy-db-lab.git\ncd easy-db-lab\n./gradlew shadowJar\n```\n\n\n#### Build the Universal AMI\n\nUsing easy-db-lab requires building an AMI if you're building from source.\n\nYou can skip this if you're using us-west-2\nThis can be done once, and reused many times.\nThe AMI should be rebuilt when updating easy-db-lab.\n\nIf you haven't run `easy-db-lab setup-profile` yet, you'll be prompted to set up your profile before building.\n\n```shell\nbin/easy-db-lab build-image\n```\n\nAt the end, you'll see something like this:\n\n```text\n==\u003e Builds finished. The artifacts of successful builds are:\n--\u003e cassandra.amazon-ebs.ubuntu: AMIs were created:\nus-west-2: ami-abcdeabcde1231231\n```\n\nThat means you're ready!\n\n### Read the Help\n\nRun `easy-db-lab` without any parameters to view all the commands and all options.\n\n### Create The Environment\n\nNote: If you haven't run `easy-db-lab setup-profile` yet, you'll be prompted to set up your profile.\n\nImportant: If you've installed the project via homebrew or downloaded a release,\nplease use the `us-west-2` region.  This limitation will be lifted soon.\n\nFirst create a directory for the environment, then initialize it, and start the instances.\n\nThis directory is your working space for the cluster.\n\n```bash\nmkdir cluster\ncd cluster\neasy-db-lab init -c 3 -s 1 myclustername # 3 node cluster with 1 stress instance\n```\n\nYou can start your instances now.\n\n```shell\neasy-db-lab up\n```\n\nTo access the cluster, follow the instructions at the end of the output of the `up` command:\n\n```bash\nsource env.sh # to setup local environment with commands to access the cluster\n\n# ssh to a node\nssh db0\nssh db1 # number corresponds to an instance\nc0 # shortcut to ssh db0\nssh app0 # ssh to app/stress node\ns0 # shortcut to ssh app0\n```\n\n### Select The Cassandra Version\n\nWhile the nodes in the cluster are up, a version isn't yet selected.  Since the AMI contains multiple versions, you'll\nneed to pick one.\n\nTo see what versions are supported, you can do the following:\n\n```shell\neasy-db-lab list\n````\n\nYou'll see 3.0, 3.11, 4.0, 4.1, and others.\n\nChoose your cassandra version.\n\n```shell\neasy-db-lab use 4.1\n```\n\neasy-db-lab will automatically configure the right Python and Java versions on the instances for you.\n\nThis will also create a local directory corresponding to the name of the version, and will download most of the files in the conf directory to your local dir.  You can edit them, and upload with:\n\n```shell\neasy-db-lab update-config\n```\n\nYou can override the java version by passing the `-j` flag:\n\n```shell\neasy-db-lab use 5.0 -j 17\n```\n\nDoing this will update each nodes local copy of `/etc/cassandra_versions.yaml`.\n\nYou can switch just one host:\n\n```shell\neasy-db-lab use 5.0 -j 17 --hosts db0\n```\n\nUnlike production tools, easy-db-lab is designed for testing and breaking things, which I find is the best way to learn.\n\nFor deploying other databases, see [ClickHouse](https://rustyrazorblade.github.io/easy-db-lab/user-guide/clickhouse/), [OpenSearch](https://rustyrazorblade.github.io/easy-db-lab/user-guide/opensearch/), and [Spark](https://rustyrazorblade.github.io/easy-db-lab/user-guide/spark/).\n\n### Modify the YAML Configuration\n\nYou'll see a file called `cassandra.patch.yaml` in your directory.  You can add any valid cassandra.yaml parameters,\nand the changes will be applied to your cluster.  The `listen_address` is handled for you,\nyou do not need to supply it.  The data directories are set up for you.\n\nYou can also edit the JVM options files under the different local version directories. Different versions of Cassandra use different\nnames for jvm.options.  Edit the ones in the directory that corresponds to the version you're using.\n\n```shell\neasy-db-lab update-config # uc for short\n```\n\n### Start The Cluster\n\nStart the cluster.  It will take about a minute to go through the startup process\n\n```shell\neasy-db-lab start\n```\n\n### Log In and Have Fun!\n\n**Important Directories:**\n\n```shell\n# The ephemeral or EBS disk is automatically formatted as XFS and mounted here.\n/mnt/db1\n\n# Database-specific subdirectories\n/mnt/db1/cassandra    # Cassandra data\n/mnt/db1/clickhouse   # ClickHouse data\n/mnt/db1/otel         # OpenTelemetry logs\n\n# Cassandra data files\n/mnt/db1/cassandra/data\n\n# hints\n/mnt/db1/cassandra/hints\n\n# commitlogs\n/mnt/db1/cassandra/commitlog\n\n# flame graphs and artifacts\n/mnt/db1/cassandra/artifacts\n\n# Note: /mnt/cassandra symlinks to /mnt/db1/cassandra for backwards compatibility\n\n# axonops agents for different versions of Cassandra\n/usr/local/share/axonops\n```\n\nMultiple cassandra versions are installed at /usr/local/cassandra.\n\nThe current version is symlinked as /usr/local/cassandra/current:\n\n```shell\nubuntu@db0:~$ readlink /usr/local/cassandra/current\n/usr/local/cassandra/4.1\n```\n\nThis allows us to support updating, mixed version clusters, A/B version testing, etc.\n\n\n**Profiling with Flame Graphs**\n\nhttps://rustyrazorblade.com/post/2023/2023-11-07-async-profiler/\n\nUsing easy-db-lab `env.sh`, you can run a profile and generate a flamegraph,\nwhich will automatically download after it's complete by doing the following:\n\n```shell\nc-flame db0\n```\n\nThe data will be saved in `artifacts/db0`\n\nOr on a node, generate flame graphs with `flamegraph`.\n\nThere are several convenient aliases defined in env.sh.\nYou may substitute any cassandra host.\nYou may pass extra parameters, they will be passed along automatically.\n\n\n| Command                 | Description                                                                              |\n|-------------------------|------------------------------------------------------------------------------------------|\n| `c-flame`      | CPU Flame graph                                                                          |\n| `c-flame-wall` | wall clock profiling, picks up I/O, parked threads filtered out                          |\n| `c-flame-compaction` | More specific wall clock profiling to compaction                                         |\n| `c-flame-offcpu ` | Just tracks time spent when cpu is unscheduled, mostly I/O                               |\n| `c-flame-sepworker`       | Request handling, by default this is CPU time. You can add -e wall to make it wall time. |\n\n\n**Aliases**\n\nOn each node there are several aliases for commonly run commands:\n\n| command | action                                     |\n| --------|---------------------------------------------|\n | c | cqlsh (auto use the correct hostname)      |\n| ts | tail cassandra system log                  |\n| nt | nodetool                                   |\n| d | cd to /mnt/db1/cassandra/data directory    |\n| l | cd to /mnt/db1/cassandra/logs directory    |\n| v | ls -lahG (friendly output)                 |\n\n\n### Shut it Down\n\nTo tear down the entire environment, simply run the following and confirm:\n\n```shell\neasy-db-lab down\n```\n\n## Tools\n\nbcc-tools is a useful package of tools\n\nhttps://rustyrazorblade.com/post/2023/2023-11-14-bcc-tools/\n\n## Server Integration\n\neasy-db-lab includes a server mode that enables AI assistants like Claude Code to interact directly with your database clusters via MCP (Model Context Protocol), and provides REST status endpoints for programmatic access.\n\n### Starting the Server\n\nTo start the server:\n\n```shell\neasy-db-lab server --port 8888\n```\n\nThis starts the server on port 8888 (you can use any available port).\n\n### Integrating with Claude Code\n\nOnce the server is running, add it to Claude Code:\n\n```shell\nclaude mcp add --transport sse easy-db-lab http://127.0.0.1:8888/sse\n```\n\nThis establishes a Server-Sent Events (SSE) connection between Claude Code and the server.\n\n### What You Can Do\n\nWith the server integration, Claude Code can:\n\n* Manage and provision clusters directly\n* Configure and deploy Cassandra, ClickHouse, and OpenSearch\n* Run performance tests and analyze results\n* Submit Spark jobs and check their status\n* Troubleshoot issues by analyzing logs and metrics\n* Automate complex multi-step cluster operations\n\nFor detailed documentation, see the [Server section in the user manual](https://rustyrazorblade.github.io/easy-db-lab/integrations/server/).\n\n## Sanity Check Test\n\nThis initializes then shuts down a cluster.  Useful after major refactors and before a release.\n\n```bash\ngw clean test shadowjar installdist  \u0026\u0026 easy-db-lab init test --up \u0026\u0026 source env.sh \u0026\u0026 edl use 5.0 \u0026\u0026 edl start \u0026\u0026 edl down --yes\n```\n\n## Contributing\n\nInterested in contributing?  Check out the [good first issue tag](https://github.com/rustyrazorblade/easy-db-lab/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) first!  Please read the [development documentation](https://rustyrazorblade.github.io/easy-db-lab/development/overview/) before getting started.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustyrazorblade%2Feasy-db-lab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frustyrazorblade%2Feasy-db-lab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustyrazorblade%2Feasy-db-lab/lists"}