https://github.com/cloud-gov/aws-broker
Cloud Foundry AWS Service Broker for RDS, OpenSearch, and Elasticache
https://github.com/cloud-gov/aws-broker
Last synced: 6 months ago
JSON representation
Cloud Foundry AWS Service Broker for RDS, OpenSearch, and Elasticache
- Host: GitHub
- URL: https://github.com/cloud-gov/aws-broker
- Owner: cloud-gov
- License: other
- Created: 2015-12-22T16:02:08.000Z (over 10 years ago)
- Default Branch: main
- Last Pushed: 2026-01-07T17:54:36.000Z (6 months ago)
- Last Synced: 2026-01-12T03:42:12.237Z (6 months ago)
- Language: Go
- Homepage:
- Size: 13.5 MB
- Stars: 18
- Watchers: 15
- Forks: 17
- Open Issues: 22
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
README
# Cloud Foundry AWS Service Broker
Cloud Foundry Service Broker to manage instances of various AWS services.
## Current Services Supported
- RDS
- AWS Elasticache for Redis
- AWS OpenSearch
## Setup
### Environment Variables
There are important environment variables that should be overriden inside the `manifest.yml` file
> Note: All environment variables prefixed with `DB_` refer to attributes for the database the broker itself will use for internal uses.
1. `DB_URL`: The hostname / IP address of the database.
1. `DB_PORT`: The port number to access the database.
1. `DB_NAME`: The database name.
1. `DB_USER`: Username to access the database.
1. `DB_PASS`: Password to access the database.
1. `DB_TYPE`: The type of database. Currently supported types: `postgres` and `sqlite3`.
1. `DB_SSLMODE`: The type of SSL Mode to use when connecting to the database. Supported modes: `disabled`, `require` and `verify-ca`.
1. `AWS_ACCESS_KEY_ID`: The id credential (treat like a password) with access to make requests to the Amazon RDS .
1. `AWS_SECRET_ACCESS_KEY`: The secret key (treat like a password) credential to access Amazon RDS.
1. `AWS_DEFAULT_REGION`: Region you wish to provision services in.
1. `AUTH_USER`: The username used by cf to authenticate to the broker
1. `AUTH_PASS`: The password used by cf to authenticate to the broker
1. `ENC_KEY`: This is an string that must be 16, 24, or 32 bytes long. It is an AES key that is used to encrypt the password.
1. `CF_API_URL`: URL for CloudFoundry API in this environment
1. `CF_API_CLIENT_ID`: UAA client ID that will be used for requests to the CloudFoundry API
1. `CF_API_CLIENT_SECRET`: UAA client secret that will be used for requests to the CloudFoundry API
1. `ENVIRONMENT`: the current environment name (e.g. "development")
> Note the AWS Environment Variables should be generated by following the instructions [here](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html)
>
> Make sure the account has write access to RDS and EC2 (particularly for VPC and Subnet).
>
> Example of permissions that suffice: `AmazonRDSFullAccess` and `AmazonEC2FullAccess`
>
> You may need to adjust VPC routing, security groups, public accessibility (DNS names, etc) as well, depending on your needs.
#### Creating a UAA Client
In order to run the app, you will need to create a UAA client application.
To create the client, log in to the jumpbox for the target environment and run:
```shell
uaac client add aws_broker \
--authorized_grant_types client_credentials \
--authorities cloud_controller.global_auditor \
-s
```
### Optional Environment Variables
There are some feature flags that you can turn on as well:
1. `ENABLE_FUNCTIONS`: If this environment variable exists, it will enable users to create mysql databases like
`cf create-service _servicename_ production my-mysql-service -c '{"enable_functions": true}'`,
which will set the `log_bin_trust_function_creators=1` parameter for their db,
enabling the creation of functions in their databases.
1. `PUBLICLY_ACCESSIBLE`: If this environment variable exists, it will enable users to create databases with
`PubliclyAccessible: true` by doing something like
`cf create-service _servicename_ production my-mysql-service -c '{"publicly_accessible": true}'`.
This is probably not something you want to set unless you really know what you are doing.
### Catalog.yml
Catalog.yml contains a list of service(s) offered with plans. It contains no secrets.
Prior to pushing, complete the catalog.yml for your environment. It is architected where the service name (e.g. rds) is the mapping between it and the service details.
### Secrets.yml
secrets.yml contains the all of the secrets for the different resources.
## Testing and development
Make sure you have a valid `secrets.yml` and `catalog.yml`:
```shell
cp catalog-test.yml catalog.yml
cp secrets-test.yml secrets.yml
```
Once you have these in place, run `go test ./...` to run the tests.
### Testing with PostgreSQL database
1. Copy `.env-sample` to `.env`
1. Add a value for `POSTGRES_USER` to the `.env` file
1. Add a value for `POSTGRES_PASSWORD` to the `.env` file
1. Start the PostgreSQL docker container:
```shell
cd docker && docker compose up -d && cd -
```
1. **OPTIONAL**: For `.env` file integration with the VSCode Go test runner, create `.vscode/settings.json` with:
```json
{
"go.testEnvFile": "/path/to/aws-broker/.env"
}
```
1. Run the tests using `godotenv`:
```shell
go install github.com/joho/godotenv/cmd/godotenv@latest
godotenv -f ./.env go test ./...
```
### How to deploy it
1. `cf push`
1. `cf create-service-broker BROKER_NAME AUTH_USER AUTH_PASS https://BROKER-URL`
1. `cf enable-service-access SERVICE_NAME`
In this case BROKER_NAME would be `aws` and it would contain many service names (one for `rds`, one for `s3`). Then SERVICE_NAME would be `rds` for example.
### How to use it
To use the service you need to create a service instance and bind it:
1. `cf create-service SERVICE_NAME micro-psql MYDB`
1. `cf bind-service APP MYDB`
When you do that you will have all the credentials in the
`VCAP_SERVICES` environment variable with the JSON key `rds`.
Also, you will have a `DATABASE_URL` environment variable that will
be the connection string to the DB.
## Credential handling
This section is primarily for auditors who need to understand how the broker, and related components, handle credentials so that they aren't stored or transmitted in the clear. All calls between entities are made over HTTPS, unless otherwise specified.
### Instantiation
The broker is deployed by Concourse CI onto CloudFoundry, using a manifest that is built by the cloud.gov secrets management system to specify the environment variables. When the app is deployed, Concourse [registers](https://docs.cloudfoundry.org/services/managing-service-brokers.html#register-broker) the broker, specifying the AUTH_USER and AUTH_PASS.
The CF Cloud Controller stores the configuration for the app, including these environment variables, in an encrypted database table on the CCDB, as described in [Cloud Foundry security concepts](https://docs.cloudfoundry.org/concepts/security.html). The `aws-broker` app does not write these to static storage since Cloud Foundry makes them available as environment variables.
### Provision a new instance
When an authenticated, authorized CloudFoundry user runs `cf create-service aws-rds _plan_name_ _service_name_`, the CloudFoundry platform uses the [OSBAPI (open-service broker API)](https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md) to call the registered broker with a `PUT` request, `/v2/service_instances/:instance_id` where instance_id is a GUID. The request uses BASIC AUTH, e,g.:
```shell
curl -X PUT https://username:password@aws-broker..../v2/service_instances/:instance_id
```
The broker expects the `AUTH_PASS`and `AUTH_USER` as specified in the environment, which the Platform has provided (see above)
The response indicates if the provisioning request has been accepted.
### Broker to AWS to provision an instance
The broker application calls the AWS API with the AWS Access Key and Secret Key, which were provided as environment variables at instantiation.
When the provisioning is complete, the broker takes the following actions:
- For RDS and Redis, it creates a username/password in the AWS service, and stores the credentials in the broker database
- For AWS Elasticsearch, it creates an IAM user with privileges to the new instance, then stores the credentials in the broker database
### Storing credentials in the broker database
The broker uses a dedicated AWS RDS PostgreSQL database. The RDS instance data are encrypted at rest using AWS storage encryption. The communication between the broker and the database is over postgres StartTLS with TLS 1.2 enabled.
The broker is instantiated with encryption key, `ENC_KEY`, and all credentials are written to the database encrypted with that key and a random salt, as in the `setPassword` function of each _service_instance.go file, e.g.:
### Providing credentials to CloudFoundry applications
The CloudFoundry applications have access to the credentials only if the user `binds` an app to a service instance, as specified at of the OSBAPI standard. The credentials are fetched from the service broker and are stored in the environment of the application container, and not written the static storage. If the application instance is re-instantiated, the platform fetches the credentials for the application container from the broker.
## Public domain
This project is in the worldwide [public domain](LICENSE.md). As stated in [CONTRIBUTING](CONTRIBUTING.md):
> This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
>
> All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.