Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/michaelpalmer1/scoutr-go
A simple way to put an API in front of a NoSQL backend.
https://github.com/michaelpalmer1/scoutr-go
api dynamo firestore mongo nosql rbac
Last synced: 8 days ago
JSON representation
A simple way to put an API in front of a NoSQL backend.
- Host: GitHub
- URL: https://github.com/michaelpalmer1/scoutr-go
- Owner: MichaelPalmer1
- License: gpl-3.0
- Created: 2020-01-12T23:59:38.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2024-04-17T21:32:15.000Z (8 months ago)
- Last Synced: 2024-11-17T04:14:52.895Z (about 1 month ago)
- Topics: api, dynamo, firestore, mongo, nosql, rbac
- Language: Go
- Homepage:
- Size: 10.4 MB
- Stars: 5
- Watchers: 3
- Forks: 1
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Security: SECURITY.md
Awesome Lists containing this project
README
# Scoutr Go
A simple way to put an API in front of a DynamoDB, Firestore, or Azure CosmosDB (MongoDB) backend.
This is based off of the Python implementation of the [scoutr](https://github.com/GESkunkworks/scoutr).
## Sample implementation
An sample implementation of this project is provided in the [examples](examples) folder.
## Requirements
At minimum, two tables are required for this to work: an auth table and a groups table. Additionally, an optional
audit log table can be used to track all API calls and changes to records in the data table. The configuration of
each table is detailed next.
### Auth Table
The auth table must have a primary key of `id`. The table name does not matter, as this is passed in during
instantiation.
### Groups Table
The groups table must have a primary key of `group_id`. The table name does not matter, as this is passed in during
instantiation.
### Audit Log Table
The audit log table must have a primary key of `time`. For DynamoDB, it should also have a TTL attribute
of `expire_time` configured. The table name does not matter, as this is passed in during instantiation. If a value is
not specified, it is assumed that no audit logs should be kept.
## Access Control
Scoutr provides full access control over the endpoints a set of users is permitted to call and the output that is
returned. This is done using field filters, field exclusions, and permitted endpoints, which are outlined in the next
section.
This access control functionality is implemented at both a user and a group level. A user can be a member of zero or
more groups. The implementation of [auth identifiers](#auth-identifier) and [groups](#groups) is outlined in their
respective sections.
The two types of access control supported are via API Gateway or via OIDC. Helper functions have been created for
each access control type to assist with passing the correct request format into Scoutr.
### API Gateway Authentication
For API Gateway authentication, the request format is generated by the [`InitAPIGateway`](helpers/apigateway.go#L16)
function.
#### Example
Refer to the [example serverless endpoint](examples/apigateway/list/main.go)
### OIDC Authentication
It is assumed that there is an Apache server running in front of the application that performs OIDC authentication
and passes the OIDC claims as headers.
The simplest method to setup the API is to use net/http. Helper functions have been
provided to make the setup as simple as possible. The [`InitHTTPServer`](helpers/http.go#L38) function
automatically generates the belows endpoints:
- GET `/user/` - Returns information about the authenticated user
- POST `/user/has-permission/` - Determine if user has permission to access an endpoint. The body of this request should
contain `method` and `path` keys as JSON.
- GET `//` - Primary endpoint used to list data. The value of `primary_list_endpoint` is
determined by an argument passed to `InitHTTPServer()`
- GET `/audit/` - List and search all audit logs
- GET `/audit//` - List audit logs for a particular resource
- GET `/history//` - Show history for a particular resource
- POST `/search//` - Search endpoint that allows searching by any key for one or more values. The body of
this request should be a JSON list of values.
#### Example
Refer to the [example net/http applications](examples/oidc)
### Concepts
#### Field filters
List of field filters to apply to queries by this group. Each item in this list must be structured as:
If the type of `value` is a string, it will be filtered using a `field = value` operation. To support multiple
values for a single field, tf the type of `value` is a list, it will be filtered using a
`field IN ['value1', 'value2', ..., 'valueN']` operation. When multiple field filters are specified, they are
combined together using an `AND` operation.
##### Syntax
```json
[
{"field": "field1", "value": "filter_value"},
{"field": "field2", "value": ["value1", "value2"]},
]
```
#### Field exclusions
Field exclusions allow for excluding one or more fields from the output of all queries. These fields are from any output
during the post-processing phase of all queries. Additionally, if a user attempts to create or update an item that
contains a field from this list, the operation will be denied.
##### Syntax
```json
[
"field1",
"field2"
]
```
#### Permitted endpoints
Before taking any action, every call from API gateway is validated to ensure the user has permissions to
perform the call. For convenience, regular expressions can be used within the `endpoint` field.
##### Syntax
```json
[
{"method": "GET|POST|PUT|DELETE", "endpoint": "/endpoint"},
{"method": "GET|POST|PUT|DELETE", "endpoint": "^/endpoint2/.+$"}
]
```
### Groups
A group object be made up of:
- `group_id` - Identifier for the group
- `permitted_endpoints` - Optional list of permitted endpoints
- `filter_fields` - Optional list of field filters
- `exclude_fields` - Optional list of field exclusions
- `update_fields_permitted` - Optional list of the only fields that can be updated
- `update_fields_restricted` - Optional list of fields to restrict updates for
The name of the group table must be passed in to the [Config](config/config.go) struct.
#### Example
```json
{
"group_id": "read-only",
"permitted_endpoints": [
{
"endpoint": "^/item/.+$",
"method": "GET"
},
{
"endpoint": "^/items.*$",
"method": "GET"
},
{
"endpoint": "^/search/.+$",
"method": "POST"
}
],
"exclude_fields": [
"supersecret"
],
"update_fields_permitted": [
"comments"
],
"update_fields_restricted": [
"type"
],
"filter_fields": [
{
"field": "provider",
"value": "Provider A"
},
{
"field": "product",
"value": [
"Product A",
"Product B"
]
}
]
}
```
### Auth Identifier
#### Types
There are three types of accepted authentication identifiers:
- USERNAME
- OIDC_GROUP
- API_KEY
Though not required, it is recommended for each object type to have a `type` key that corresponds to its
authentication type (OIDC_GROUP, USERNAME, or API_KEY).
The field requirements for each object type are outlined in the following sections
##### USERNAME
- id (primary key) - this is the user's username (i.e. johndoe)
Though not required, it is recommended to also include a `name` field containing the user's full name to make it
easier to identify the user at a glance.
##### OIDC_GROUP
- id (primary key) - this is expected to be the group id (i.e. group123) from the OIDC header
Though not required, it is recommended to also include a `name` field containing the group's display name to make it
easier to identify the group at a glance.
If a user is a member of more than one OIDC group, the permissions granted by each configured group will be combined
together to generate the effective permissions applied to the user.
##### API_KEY
- id (primary key) - this is the api key id
- name
- username
- email
#### Groups
Optionally, each auth object can include a `groups` object, which should be a list of group ids that the user is a
member of:
```
{
"groups": [
"read-only",
"product-a-only"
]
}
```
Any permissions defined in the groups are combined together to make up the user's permissions. In addition, the same
permissions that a group defines (`filter_fields`, `exclude_fields`, `update_fields_permitted`,
`update_fields_restricted`, `permitted_endpoints`) can be expressed at the user level. These permissions will be
combined together with the permissions outlined in the groups the user is a member of. Permissions defined at the user
level **DO NOT** override those specified at the group level - they are combined.
The name of the user table must be passed in to the constructor.
### Audit Logs
For every authorized, successful call to the API, an entry will be logged in the audit log table. Each record will
follow the below format:
```json
{
"action": "CREATE|UPDATE|DELETE|GET|LIST|SEARCH|{CUSTOM-ACTION}",
"body": {
"key": "value"
},
"method": "HTTP method from API gateway",
"path": "/endpoint/path",
"path_params": {
"key": "value"
},
"query_params": {
"key": "value"
},
"resource": {
"key": "value"
},
"time": "2019-10-04T18:44:30.166635",
"user": {
"api_key_id": "ID",
"name": "John Doe",
"source_ip": "1.2.3.4",
"username": "johndoe",
"user_agent": "curl"
}
}
```
The following fields may not be included or may not have values for all types of actions:
- body
- query_params
- path_params
- resource
## Endpoint Structure
The helper methods within Scoutr assume that your API consists of the following endpoint types:
- [List all records](#list)
- [List all unique values for a key](#list-by-unique-key)
- [Search multiple values for a single search key](#search)
- [Get single item by key](#get)
- [Update single item by key](#update)
- [Delete single item by key](#delete)
- [List all audit logs](#list-audit-logs)
- [View item history](#history)
### List
The list all items endpoint will return a list of all items within the backend that the user has permission to see
and that meet any specified filter criteria.
### List by Unique Key
The list by unique key endpoint provides a means to display all unique values for a single search key. It is
implemented by specifying a value for the `uniqueKey` argument of the `ListUniqueValues()` function. This is only
supported in `DynamoAPI` currently.
#### Serverless Example
```yml
# Unique listing of all values of the `status` key that the user is permitted to see
list-statuses:
handler: listUnique
events:
- http:
path: statuses
method: get
private: true
environment:
UniqueKey: status
```
#### Implementation Example
```go
func handler(event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
// Convert log retention to int
logRetention, err := strconv.Atoi(os.Getenv("LogRetentionDays"))
if err != nil {
panic(err)
}
// Build config
config := config.Config{
DataTable: os.Getenv("DataTable"),
AuthTable: os.Getenv("AuthTable"),
AuditTable: os.Getenv("AuditTable"),
GroupTable: os.Getenv("GroupTable"),
LogRetentionDays: logRetention,
}
// Initialize api gateway
api, request := helpers.InitAPIGateway(event, config)
// List the data
data, err := api.List(request)
// Handle any errors
if errorResponse := helpers.APIGatewayErrorHandler(err); errorResponse != nil {
return *errorResponse, nil
}
// Send response
return helpers.ProcessAPIGatewayResponse(data)
}
```
### Search
Lookup information about multiple items (POST `/search/{search_key}`)
```
[
"record-a",
"record-b"
]
```
### Get
Retrieve a single record from the backend. The `Get()` function accepts two arguments:
- `req` - the [Request](models/models.go#L13) object containing information about the request
- `id` - the id of the item to lookup
If this returns more than one record, it will throw a `BadRequest` error. If no records are
returned, a `NotFound` error will be thrown.
### Create
The `Create()` function accepts the `req` argument, with `req` being the [Request](models/models.go#L13) object, an
`item` argument, with `item` being a `map[string]string` of the data to be inserted, and a `validation` argument in
order to perform validation on all the supplied data. Refer to the [data validation](#data-validation) section for more
information.
### Update
The `Update()` function accepts a couple of arguments:
**`req`**
[Request](models/models.go#L13) object
**`partitionKey`**
Mapping of the partition key to value. For instance, if the table's partition key is `id`, it is expected this mapping
would be:
```go
map[string]string{
"id": "value"
}
```
**`item`**
`map[string]string` of fields to be updated
**`validation`**
`map[string]utils.FieldValidation` of fields to perform validation against. Refer to the
[data validation](#data-validation) section for more information.
**`auditAction**`
A string value to use as the Action in the audit logs. This should be set to `UPDATE` in most cases.
### Delete
The `Delete()` function accepts a couple of arguments:
**`req`**
[Request](models/models.go#L13) object
**`partitionKey`**
Mapping of the partition key to value. For instance, if the table's partition key is `id`, it is expected this mapping
would be:
```python
map[string]string{
"id": "value"
}
```
### List audit logs
The `ListAuditLogs()` function accepts:
**`req`**
[Request](models/models.go#L13) object
**`pathParams`**
Any search parameters to apply
**`queryParams`**
Query parameters from API Gateway
### History
**`req`**
[Request](models/models.go#L13) object
**`key`**
Resource key to search on
**`value`**
Resource value to search on
**`queryParams`**
Query parameters from API Gateway
**`actions`**
List of actions to filter on
## Filtering
There are two levels of filtering that are supported:
- Path-based filtering
- Querystring-based filtering
The `List()` function accepts a single `req` argument as a [Request](models/models.go#L13) object. For filtering to be
applied, its `PathParams` and `QueryParams` fields should contain values. These are intended to contain the values
of `PathParameters` and `QueryStringParameters`, respectively, that API Gateway passed into Lambda. In the case of
net/http, these values should be set using the `request.URL.Query()` function and any path parameters that
are set by the `httprouter` package. Refer to the [net/http example](examples/oidc/aws/main.go) and the
[InitHTTPServer](helpers/http.go#L38) function to see usage.
### Dynamic path filters
The `List()` function also supports dynamic path filtering. When `search_key` and `search_value` are passed into
the method as `PathParams`, it will dynamically modify the path parameters to construct a search filter where
```
search_key = search_value
```
To configure this in API Gateway, setup path parameters on the resource:
```
/endpoint/{search_key}/{search_value}
```
Or when using serverless:
```yml
events:
- http:
path: endpoint
method: get
private: true
- http:
path: endpoint/{search_key}/{search_value}
method: get
private: true
```
When using the dynamic path filters, there is no need to construct additional endpoints that support filtering by a
specific key. However, using this method provides no limitations over what fields can be used as a filter. If that is a
concern for your API, you will need to construct static path filters.
### Static path filters
Static path filters can be constructed in a similar manner to the dynamic path filters, except that the search key is
manually specified:
```
/endpoint/status/{status}
```
In order to properly work, the path variable must _exactly_ match the key in the backend table that you want to perform
the filter against.
### Querystring Filters
In addition to path filters, querystring filtering is also supported. The `List()` endpoint accepts all
querystrings via the `QueryParams` field of the request object. Each querystring should be a
`field_name=search_value` format:
```
/endpoint?status=Active&type=ABC
```
Path parameters **always** take precedence over querystring parameters. The below query:
```
/endpoint/type/ABC?status=Active&type=Azure
```
Would result in this filter criteria:
```
type = ABC AND status = Active
```
#### Magic Operators
For more complex queries, querystring search supports the below magic operations:
- `in` (value is in list)
- `notin` (value is not in list)
- `ne` (not equal)
- `startswith` (string starts with)
- `contains` (string contains)
- `notcontains` (string does not contain)
- `exists` (attribute exists)
- `gt` (greater than)
- `lt` (less than)
- `ge` (greater than or equal)
- `le` (less than or equal)
- `between` (value is between)
Note that DynamoDBAPI does not support the `in` operation and FirestoreAPI only accepts the following magic operations:
- in
- gt
- lt
- ge
- le
- between
The MongoDBAPI supports these operations:
- in
- notin
- ne
- startswith
- contains
- exists
- gt
- ge
- lt
- le
- between
To use a magic operator, append `__operator` to the key name. For example:
To search for all items with the `product` key containing the word "Product"
```
/items?product__contains=Product
```
To search for all items with the `product` key starting with the word "Product"
```
/items?sku__startswith=Product
```
Usage of all the magic operators is straightforward, with the exception of the `in` and `between` operators. The `in`
operators checks to see if the the value is included in a list of options. It should follow the JSON list syntax:
```
/items?product__in=["Product A", "Product B"]
```
The `between` operator checks to see if the value is, inclusively, between a low and high value. It should also follow
a JSON list syntax:
```
/items?num__between=[0, 3]
```
It also works for string values, such as two dates:
```
/items?date__between=["2019-01-01", "2019-12-31"]
```
To find items that have an attribute:
```
/items?name__exists=true
```
To search for items that do not have an attribute:
```
/items?name__exists=false
```
## Data validation
For convenience, support for data validation on all create and update calls is supported. In order to implement the
validation, a `map[string]utils.FieldValidation` should be passed to the `validation` map of the `Create()` or
`Update()` functions. The syntax of this object is outlined below.
On `Create()` calls, all items specified in the `validation` map are assumed to be required fields. If a
field is missing from the user input, an error will be thrown saying that the field is required.
### Syntax
```go
validation := map[string]utils.FieldValidation{
"field1": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
if value != "hello" {
return false, fmt.Sprintf("Invalid value '%s' for attribute 'field1'", value), nil
}
return true, "", nil
},
"field2": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
if value != "world" {
return false, fmt.Sprintf("Invalid value '%s' for attribute 'field2'", value), nil
}
return true, "", nil
},
}
```
The key of each item in the dictionary should match a field name that you want to perform validation against. The
corresponding value for the key should be a callable that returns a boolean, string, and error. The boolean should be
`true` if the field validated successfully, or `false` if it did not. The `string` should contain the error message
that should be displayed to the user. The `error` should be `nil` if there were not any errors while running validation.
If an error was encountered, this error value will be returned to the user.
The callable that you provide must accept three arguments:
- `value` - Contains the input value for this field
- `item` - Contains the entire data object that was passed from the user
- `existingItem` - Contains the existing data object. This will only have a value on update calls. For
create calls, this will be `None`.
### Example
```go
func validateUser(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
var itemType string
if existingItem != nil {
itemType = existingItem["type"]
} else {
itemType = item["type"]
}
if _, ok := item["type"]; !ok {
return false, "Type field is required", nil
}
if itemType == "Type1" {
re := regexp.MustCompile("^\d{10}$")
if re.MatchString(value) {
return true, "", nil
} else {
return false, "Value does not match pattern", nil
}
} else if itemType == "Type2" {
re := regexp.MustCompile("^[a-z]+$")
if re.MatchString(value) {
return true, "", nil
} else {
return false, "Value does not match pattern", nil
}
} else {
return false, "Validation failed", nil
}
}
fieldValidation := map[string]utils.FieldValidation{
"user": validateUser,
"type": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {
validOptions := []string{"ABC", "DEF"}
found := false
for _, item := range validOptions {
if item == value {
found = true
break
}
}
if !found {
return false, fmt.Sprintf("Invalid value. Supported options are %s", validOptions), nil
}
}
}
```
## [Sentry](https://sentry.io) support
Coming soon