{"id":13820238,"url":"https://github.com/pinterest/soundwave","last_synced_at":"2025-05-16T07:33:37.661Z","repository":{"id":66170238,"uuid":"79304145","full_name":"pinterest/soundwave","owner":"pinterest","description":"A searchable EC2 Inventory store","archived":true,"fork":false,"pushed_at":"2019-11-13T18:40:06.000Z","size":1992,"stargazers_count":96,"open_issues_count":5,"forks_count":17,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-11-19T19:44:06.889Z","etag":null,"topics":["aws","cloud-management","configuration-management","ec2","infrastructure","tools"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pinterest.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-18T04:44:05.000Z","updated_at":"2024-04-28T09:07:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"e5c30b0a-fee3-4980-a3ed-ad3803340dc7","html_url":"https://github.com/pinterest/soundwave","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2Fsoundwave","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2Fsoundwave/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2Fsoundwave/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinterest%2Fsoundwave/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pinterest","download_url":"https://codeload.github.com/pinterest/soundwave/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254488631,"owners_count":22079462,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud-management","configuration-management","ec2","infrastructure","tools"],"created_at":"2024-08-04T08:00:59.930Z","updated_at":"2025-05-16T07:33:37.656Z","avatar_url":"https://github.com/pinterest.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"**Note:** This project is no longer actively maintained by Pinterest.\n\n---\n\n# Soundwave\n![Demo](https://github.com/pinterest/soundwave/blob/master/soundwavedemo.png)\n# What is soundwave\nSoundwave is a searchable EC2 inventory store. It tracks all the current and historic EC2 instances with their metadata.\nIt has three parts:\n\n* A java based worker system to synchronize EC2 data and push to the elastic search store.\n* A RESTful API layer to provide access to instance data and search.\n* A UI dashboard for end users to perform ad hoc search.\n\nAt Pinterest, Soundwave is the foundation of our Configuration Management Database which serves as the source of truth for our EC2 instance information.\nThe data is used for service management, visibility, and finance purposes. This engineering blog (https://medium.com/@Pinterest_Engineering/soundwave-an-open-source-configuration-management-database-df75b31b8f5a) contains the detail informations\n\n\n# Why use soundwave\n\nSoundwave is useful for companies which operate EC2 at scale and require to track usage and instance meta data.\nWhile the AWS console and EC2 API can already be used to access the metadata of EC2 instances, there are some limitations that prompted us to build an internal data store. \n1. AWS EC2 API has a rate limit which makes it unsuitable for direct querying of machine information from automation systems.\n2. AWS doesn’t persist metadata and instance information for terminated instances. This makes it difficult to track usage history and diagnostic issues of these terminated instances.\n3. In AWS, EC2 schema can only be extended in a limited way with the use of tags.\n4. EC2 API has very limited support for querying beyond simple filtering.\n\nIn addition, We built Soundwave to bring to life the following benefits:\n1. A dedicated tracking of EC2 instances in our elasticsearch store helps us achieve a better query performance than EC2 API. This means getting information about all running instances takes roughly 5 seconds in Soundwave, but more than a minute from EC2 API.\n2. It creates a cloud agnostic abstraction layer for configuration management that makes it easier to support hybrid cloud scenarios\n\n\n# Getting Started\n\n#### 1. Clone the repository\n\n```\ngit clone https://github.com/pinterest/soundwave.git\n```\n\n#### 2. Setup AWS Credentials\n\n\nSoundwave uses [Terraform](https://www.terraform.io/) to provision the required infrastructure for running the service.\n\n[Terraform](https://www.terraform.io/) requires AWS credentials (Access and Secret Key) of a privileged user. \n```\n$ vi soundwave/terraform/soundwaveaws.tf\n----------------------------------------\n\nprovider \"aws\" {\n  access_key = \"\u003c\u003cYour Access Key\u003e\u003e\"\n  secret_key = \"\u003c\u003cYour Secret Key\u003e\u003e\"\n  region     = \"\u003c\u003cYour AWS Region\u003e\u003e\"\n} \n```\n#### 3. Run Terraform\n\n\n```\n$ cd soundwave/terraform\n$ terraform apply\n```\n\n#### 4. Update Properties\n\n\nUse the AWS region and your account number in the fields shown below\n\n(Note: Account Number is available on the AWS console under MyAccount section)\n```\n$ vi soundwave/worker/config/soundwaveworker.properties\n-------------------------------------------------------\n\naws_region=us-east-1\nupdate_queue=https://sqs.us-east-1.amazonaws.com/\u003caccountnumber\u003e/soundwave-events\n```\n\n#### 5. Build the project\n\n\n```\nmvn clean package\n```\n\n#### 6. Update docker-compose\n\n\nUpdate the docker-compose.yml file to use your AWS Access and Secret keys\n```\n$ vi soundwave/docker-compose.yml\n---------------------------------\n\nversion: '2'\nservices:\n  soundwave-worker:\n    build: ./worker\n    environment:\n      - CONFIG_FILE=config/soundwaveworker.properties\n      - TEST_CONFIG=1\n      - AWS_ACCESS_KEY_ID=\u003c\u003ckey_id\u003e\u003e\n      - AWS_SECRET_ACCESS_KEY=\u003c\u003caccess_key\u003e\u003e\n```\n\n#### 7. Run docker-compose\n\n\n```\ndocker-compose up\n```\n\n#### 8. Create Elastic search indexes\n\n\n```\n$ cd worker/scripts\n$ ./provision_index.sh http://localhost:9200/soundwave_prod\n$ ./provision_index.sh http://localhost:9200/soundwave_ss\n```\n\n#### 9. Launch an EC2 instance through the AWS console or aws-cli command line\n\n#### 10. See the results\n\nResults can be seen on one of the following:\n\n##### UI [http://localhost:80](http://localhost:80)\n##### API [http://localhost:8080/v2/instance/\u003cinstance_id\u003e](http://localhost:8080/v2/instance/\u003cinstance_id\u003e) \n##### Elasticsearch [http://localhost:9200/soundwave_prod/_search](http://localhost:9200/soundwave_prod/_search)\n\n```\n{\n   \"took\": 38,\n   \"timed_out\": false,\n   \"_shards\": {\n      \"total\": 1,\n      \"successful\": 1,\n      \"failed\": 0\n   },\n   \"hits\": {\n      \"total\": 1,\n      \"max_score\": 1,\n      \"hits\": [\n         {\n            \"_index\": \"soundwave_prod\",\n            \"_type\": \"instance\",\n            \"_id\": \"i-01c085b645bffcf6f\",\n            \"_score\": 1,\n            \"_source\": {\n               \"id\": \"i-01c085b645bffcf6f\",\n               \"region\": \"us-east-1\",\n               \"location\": \"us-east-1b\",\n               \"state\": \"running\",\n               \"created_time\": \"2017-03-21T18:57:56.571+0000\",\n               \"updated_time\": \"2017-03-21T18:58:27.577+0000\",\n               \"terminated_time\": null,\n               \"vpc_id\": \"vpc-b58320d0\",\n               \"subnet_id\": \"subnet-dcfc3cab\",\n               \"aws_launch_time\": \"2017-03-22T03:49:18.000+0000\",\n               \"security_groups\": null,\n               \"security_group_ids\": null,\n               \"tags\": null,\n               \"cloud\": {\n                  \"aws\": {\n                     \"subnetId\": \"subnet-dcfc3cab\",\n                     \"kernelId\": null,\n                     \"ramdiskId\": null,\n                     \"virtualizationType\": \"hvm\",\n                     \"amiLaunchIndex\": 0,\n                     \"enaSupport\": true,\n                     \"stateReason\": null,\n                     \"sourceDestCheck\": true,\n                     \"platform\": null,\n                     \"instanceId\": \"i-01c085b645bffcf6f\",\n                     \"vpcId\": \"vpc-b58320d0\",\n                     \"hypervisor\": \"xen\",\n                     \"rootDeviceName\": \"/dev/xvda\",\n                     \"state\": {\n                        \"code\": 16,\n                        \"name\": \"running\"\n                     },\n                     \"productCodes\": [],\n                     \"architecture\": \"x86_64\",\n                     \"ebsOptimized\": false,\n                     \"imageId\": \"ami-0b33d91d\",\n                     \"blockDeviceMappings\": [\n                        {\n                           \"ebs\": {\n                              \"volumeId\": \"vol-0305a977aad819e7f\",\n                              \"deleteOnTermination\": true,\n                              \"attachTime\": 1490154559000,\n                              \"status\": \"attached\"\n                           },\n                           \"deviceName\": \"/dev/xvda\"\n                        }\n                     ],\n                     \"stateTransitionReason\": \"\",\n                     \"publicIpAddress\": \"54.242.222.37\",\n                     \"clientToken\": \"EtnOD1490154557345\",\n                     \"instanceType\": \"t2.micro\",\n                     \"keyName\": \"lidaec2key\",\n                     \"publicDnsName\": \"\",\n                     \"monitoring\": \"disabled\",\n                     \"iamInstanceProfile\": null,\n                     \"privateIpAddress\": \"172.30.1.227\",\n                     \"instanceLifecycle\": null,\n                     \"rootDeviceType\": \"ebs\",\n                     \"tags\": [],\n                     \"launchTime\": 1490154558000,\n                     \"spotInstanceRequestId\": null,\n                     \"networkInterfaces\": [\n                        {\n                           \"networkInterfaceId\": \"eni-0c2febee\",\n                           \"subnetId\": \"subnet-dcfc3cab\",\n                           \"privateIpAddresses\": [\n                              {\n                                 \"association\": {\n                                    \"ipOwnerId\": \"amazon\",\n                                    \"publicIp\": \"54.242.222.37\",\n                                    \"publicDnsName\": \"\"\n                                 },\n                                 \"privateDnsName\": null,\n                                 \"privateIpAddress\": \"172.30.1.227\",\n                                 \"primary\": true\n                              }\n                           ],\n                           \"description\": \"Primary network interface\",\n                           \"groups\": [\n                              {\n                                 \"groupName\": \"launch-wizard-8\",\n                                 \"groupId\": \"sg-7465ce0b\"\n                              }\n                           ],\n                           \"association\": {\n                              \"ipOwnerId\": \"amazon\",\n                              \"publicIp\": \"54.242.222.37\",\n                              \"publicDnsName\": \"\"\n                           },\n                           \"ipv6Addresses\": [],\n                           \"ownerId\": \"093574427811\",\n                           \"sourceDestCheck\": true,\n                           \"privateIpAddress\": \"172.30.1.227\",\n                           \"macAddress\": \"0a:f0:41:07:19:aa\",\n                           \"attachment\": {\n                              \"attachmentId\": \"eni-attach-3a2f6802\",\n                              \"deleteOnTermination\": true,\n                              \"deviceIndex\": 0,\n                              \"attachTime\": 1490154558000,\n                              \"status\": \"attached\"\n                           },\n                           \"vpcId\": \"vpc-b58320d0\",\n                           \"privateDnsName\": null,\n                           \"status\": \"in-use\"\n                        }\n                     ],\n                     \"sriovNetSupport\": null,\n                     \"privateDnsName\": \"ip-172-30-1-227.ec2.internal\",\n                     \"securityGroups\": [\n                        {\n                           \"groupName\": \"launch-wizard-8\",\n                           \"groupId\": \"sg-7465ce0b\"\n                        }\n                     ],\n                     \"placement\": {\n                        \"availabilityZone\": \"us-east-1b\",\n                        \"groupName\": \"\",\n                        \"tenancy\": \"default\",\n                        \"hostId\": null,\n                        \"affinity\": null,\n                        \"availability_zone\": \"us-east-1b\"\n                     }\n                  }\n               },\n               \"aws_status\": null,\n               \"token\": null,\n               \"cached\": 0\n            }\n         }\n      ]\n   }\n}\n```\n# Some example queries in UI (follows ES syntax):\n\n* id:i-123456789abce\n* cloud.aws.instanceType:c3.4xlarge\n* aws_launch_time:\u003enow-1d OR terminated_time:\u003enow-1d\n\n# License\n[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinterest%2Fsoundwave","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpinterest%2Fsoundwave","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinterest%2Fsoundwave/lists"}