{"id":23187509,"url":"https://github.com/michaelpalmer1/scoutr-go","last_synced_at":"2026-03-15T20:05:20.722Z","repository":{"id":37894456,"uuid":"233477533","full_name":"MichaelPalmer1/scoutr-go","owner":"MichaelPalmer1","description":"A simple way to put an API in front of a NoSQL backend.","archived":false,"fork":false,"pushed_at":"2024-04-17T21:32:15.000Z","size":10871,"stargazers_count":5,"open_issues_count":6,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-17T04:14:52.895Z","etag":null,"topics":["api","dynamo","firestore","mongo","nosql","rbac"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MichaelPalmer1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-12T23:59:38.000Z","updated_at":"2024-03-14T16:00:23.000Z","dependencies_parsed_at":"2023-02-18T22:45:57.751Z","dependency_job_id":"9c45baee-41fb-406b-afd1-375485116f53","html_url":"https://github.com/MichaelPalmer1/scoutr-go","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelPalmer1%2Fscoutr-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelPalmer1%2Fscoutr-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelPalmer1%2Fscoutr-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelPalmer1%2Fscoutr-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MichaelPalmer1","download_url":"https://codeload.github.com/MichaelPalmer1/scoutr-go/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230252772,"owners_count":18197285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","dynamo","firestore","mongo","nosql","rbac"],"created_at":"2024-12-18T10:19:38.344Z","updated_at":"2026-03-15T20:05:20.646Z","avatar_url":"https://github.com/MichaelPalmer1.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scoutr Go\n\nA simple way to put an API in front of a DynamoDB, Firestore, or Azure CosmosDB (MongoDB) backend.\n\nThis is based off of the Python implementation of the [scoutr](https://github.com/GESkunkworks/scoutr).\n\n## Sample implementation\n\nAn sample implementation of this project is provided in the [examples](examples) folder.\n\n## Requirements\n\nAt minimum, two tables are required for this to work: an auth table and a groups table. Additionally, an optional\naudit log table can be used to track all API calls and changes to records in the data table. The configuration of\neach table is detailed next.\n\n### Auth Table\nThe auth table must have a primary key of `id`. The table name does not matter, as this is passed in during\ninstantiation.\n\n### Groups Table\nThe groups table must have a primary key of `group_id`. The table name does not matter, as this is passed in during\ninstantiation.\n\n### Audit Log Table\nThe audit log table must have a primary key of `time`. For DynamoDB, it should also have a TTL attribute\nof `expire_time` configured. The table name does not matter, as this is passed in during instantiation. If a value is\nnot specified, it is assumed that no audit logs should be kept.\n\n## Access Control\n\nScoutr provides full access control over the endpoints a set of users is permitted to call and the output that is\nreturned. This is done using field filters, field exclusions, and permitted endpoints, which are outlined in the next\nsection.\n\nThis access control functionality is implemented at both a user and a group level. A user can be a member of zero or\nmore groups. The implementation of [auth identifiers](#auth-identifier) and [groups](#groups) is outlined in their\nrespective sections.\n\nThe two types of access control supported are via API Gateway or via OIDC. Helper functions have been created for\neach access control type to assist with passing the correct request format into Scoutr.\n\n### API Gateway Authentication\nFor API Gateway authentication, the request format is generated by the [`InitAPIGateway`](helpers/apigateway.go#L16)\nfunction.\n\n#### Example\nRefer to the [example serverless endpoint](examples/apigateway/list/main.go)\n\n### OIDC Authentication\nIt is assumed that there is an Apache server running in front of the application that performs OIDC authentication\nand passes the OIDC claims as headers.\n\nThe simplest method to setup the API is to use net/http. Helper functions have been\nprovided to make the setup as simple as possible. The [`InitHTTPServer`](helpers/http.go#L38) function\nautomatically generates the belows endpoints:\n- GET `/user/` - Returns information about the authenticated user\n- POST `/user/has-permission/` - Determine if user has permission to access an endpoint. The body of this request should\n    contain `method` and `path` keys as JSON.\n- GET `/\u003cprimary_list_endpoint\u003e/` - Primary endpoint used to list data. The value of `primary_list_endpoint` is\n    determined by an argument passed to `InitHTTPServer()`\n- GET `/audit/` - List and search all audit logs\n- GET `/audit/\u003citem\u003e/` - List audit logs for a particular resource\n- GET `/history/\u003citem\u003e/` - Show history for a particular resource\n- POST `/search/\u003csearch_key\u003e/` - Search endpoint that allows searching by any key for one or more values. The body of\n    this request should be a JSON list of values.\n\n#### Example\nRefer to the [example net/http applications](examples/oidc)\n\n### Concepts\n\n#### Field filters\n\nList of field filters to apply to queries by this group. Each item in this list must be structured as:\n\nIf the type of `value` is a string, it will be filtered using a `field = value` operation. To support multiple\nvalues for a single field, tf the type of `value` is a list, it will be filtered using a\n`field IN ['value1', 'value2', ..., 'valueN']` operation. When multiple field filters are specified, they are\ncombined together using an `AND` operation.\n\n##### Syntax\n```json\n[\n    {\"field\": \"field1\", \"value\": \"filter_value\"},\n    {\"field\": \"field2\", \"value\": [\"value1\", \"value2\"]},\n]\n```\n\n#### Field exclusions\n\nField exclusions allow for excluding one or more fields from the output of all queries. These fields are from any output\nduring the post-processing phase of all queries. Additionally, if a user attempts to create or update an item that\ncontains a field from this list, the operation will be denied.\n\n##### Syntax\n```json\n[\n    \"field1\",\n    \"field2\"\n]\n```\n\n#### Permitted endpoints\n\nBefore taking any action, every call from API gateway is validated to ensure the user has permissions to\nperform the call. For convenience, regular expressions can be used within the `endpoint` field.\n\n##### Syntax\n```json\n[\n    {\"method\": \"GET|POST|PUT|DELETE\", \"endpoint\": \"/endpoint\"},\n    {\"method\": \"GET|POST|PUT|DELETE\", \"endpoint\": \"^/endpoint2/.+$\"}\n]\n```\n\n### Groups\n\nA group object be made up of:\n- `group_id` - Identifier for the group\n- `permitted_endpoints` - Optional list of permitted endpoints\n- `filter_fields` - Optional list of field filters\n- `exclude_fields` - Optional list of field exclusions\n- `update_fields_permitted` - Optional list of the only fields that can be updated\n- `update_fields_restricted` - Optional list of fields to restrict updates for\n\nThe name of the group table must be passed in to the [Config](config/config.go) struct.\n\n#### Example\n```json\n{\n    \"group_id\": \"read-only\",\n    \"permitted_endpoints\": [\n        {\n            \"endpoint\": \"^/item/.+$\",\n            \"method\": \"GET\"\n        },\n        {\n            \"endpoint\": \"^/items.*$\",\n            \"method\": \"GET\"\n        },\n        {\n            \"endpoint\": \"^/search/.+$\",\n            \"method\": \"POST\"\n        }\n    ],\n    \"exclude_fields\": [\n        \"supersecret\"\n    ],\n    \"update_fields_permitted\": [\n        \"comments\"\n    ],\n    \"update_fields_restricted\": [\n        \"type\"\n    ],\n    \"filter_fields\": [\n        {\n            \"field\": \"provider\",\n            \"value\": \"Provider A\"\n        },\n        {\n            \"field\": \"product\",\n            \"value\": [\n                \"Product A\",\n                \"Product B\"\n            ]\n        }\n    ]\n}\n```\n\n### Auth Identifier\n\n#### Types\n\nThere are three types of accepted authentication identifiers:\n- USERNAME\n- OIDC_GROUP\n- API_KEY\n\nThough not required, it is recommended for each object type to have a `type` key that corresponds to its\nauthentication type (OIDC_GROUP, USERNAME, or API_KEY).\n\nThe field requirements for each object type are outlined in the following sections\n\n##### USERNAME\n- id (primary key) - this is the user's username (i.e. johndoe)\n\nThough not required, it is recommended to also include a `name` field containing the user's full name to make it\neasier to identify the user at a glance.\n\n##### OIDC_GROUP\n- id (primary key) - this is expected to be the group id (i.e. group123) from the OIDC header\n\nThough not required, it is recommended to also include a `name` field containing the group's display name to make it\neasier to identify the group at a glance.\n\nIf a user is a member of more than one OIDC group, the permissions granted by each configured group will be combined\ntogether to generate the effective permissions applied to the user.\n\n##### API_KEY\n- id (primary key) - this is the api key id\n- name\n- username\n- email\n\n#### Groups\n\nOptionally, each auth object can include a `groups` object, which should be a list of group ids that the user is a\nmember of:\n```\n{\n    \"groups\": [\n        \"read-only\",\n        \"product-a-only\"\n    ]\n}\n```\n\nAny permissions defined in the groups are combined together to make up the user's permissions. In addition, the same\npermissions that a group defines (`filter_fields`, `exclude_fields`, `update_fields_permitted`,\n`update_fields_restricted`, `permitted_endpoints`) can be expressed at the user level. These permissions will be\ncombined together with the permissions outlined in the groups the user is a member of. Permissions defined at the user\nlevel **DO NOT** override those specified at the group level - they are combined.\n\nThe name of the user table must be passed in to the constructor.\n\n### Audit Logs\n\nFor every authorized, successful call to the API, an entry will be logged in the audit log table. Each record will\nfollow the below format:\n\n```json\n{\n  \"action\": \"CREATE|UPDATE|DELETE|GET|LIST|SEARCH|{CUSTOM-ACTION}\",\n  \"body\": {\n    \"key\": \"value\"\n  },\n  \"method\": \"HTTP method from API gateway\",\n  \"path\": \"/endpoint/path\",\n  \"path_params\": {\n    \"key\": \"value\"\n  },\n  \"query_params\": {\n    \"key\": \"value\"\n  },\n  \"resource\": {\n    \"key\": \"value\"\n  },\n  \"time\": \"2019-10-04T18:44:30.166635\",\n  \"user\": {\n    \"api_key_id\": \"ID\",\n    \"name\": \"John Doe\",\n    \"source_ip\": \"1.2.3.4\",\n    \"username\": \"johndoe\",\n    \"user_agent\": \"curl\"\n  }\n}\n```\n\nThe following fields may not be included or may not have values for all types of actions:\n- body\n- query_params\n- path_params\n- resource\n\n## Endpoint Structure\n\nThe helper methods within Scoutr assume that your API consists of the following endpoint types:\n- [List all records](#list)\n- [List all unique values for a key](#list-by-unique-key)\n- [Search multiple values for a single search key](#search)\n- [Get single item by key](#get)\n- [Update single item by key](#update)\n- [Delete single item by key](#delete)\n- [List all audit logs](#list-audit-logs)\n- [View item history](#history)\n\n### List\n\nThe list all items endpoint will return a list of all items within the backend that the user has permission to see\nand that meet any specified filter criteria.\n\n### List by Unique Key\n\nThe list by unique key endpoint provides a means to display all unique values for a single search key. It is\nimplemented by specifying a value for the `uniqueKey` argument of the `ListUniqueValues()` function. This is only\nsupported in `DynamoAPI` currently.\n\n#### Serverless Example\n```yml\n# Unique listing of all values of the `status` key that the user is permitted to see\nlist-statuses:\n  handler: listUnique\n  events:\n    - http:\n        path: statuses\n        method: get\n        private: true\n  environment:\n    UniqueKey: status\n```\n\n#### Implementation Example\n```go\nfunc handler(event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {\n\t// Convert log retention to int\n\tlogRetention, err := strconv.Atoi(os.Getenv(\"LogRetentionDays\"))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\t// Build config\n\tconfig := config.Config{\n\t\tDataTable:        os.Getenv(\"DataTable\"),\n\t\tAuthTable:        os.Getenv(\"AuthTable\"),\n\t\tAuditTable:       os.Getenv(\"AuditTable\"),\n\t\tGroupTable:       os.Getenv(\"GroupTable\"),\n\t\tLogRetentionDays: logRetention,\n\t}\n\n\t// Initialize api gateway\n\tapi, request := helpers.InitAPIGateway(event, config)\n\n\t// List the data\n\tdata, err := api.List(request)\n\n\t// Handle any errors\n\tif errorResponse := helpers.APIGatewayErrorHandler(err); errorResponse != nil {\n\t\treturn *errorResponse, nil\n\t}\n\n\t// Send response\n\treturn helpers.ProcessAPIGatewayResponse(data)\n}\n```\n\n### Search\n\nLookup information about multiple items (POST `/search/{search_key}`)\n```\n[\n    \"record-a\",\n    \"record-b\"\n]\n```\n\n### Get\n\nRetrieve a single record from the backend. The `Get()` function accepts two arguments:\n- `req` - the [Request](models/models.go#L13) object containing information about the request\n- `id` - the id of the item to lookup\n\nIf this returns more than one record, it will throw a `BadRequest` error. If no records are\nreturned, a `NotFound` error will be thrown.\n\n### Create\n\nThe `Create()` function accepts the `req` argument, with `req` being the [Request](models/models.go#L13) object, an\n`item` argument, with `item` being a `map[string]string` of the data to be inserted, and a `validation` argument in\norder to perform validation on all the supplied data. Refer to the [data validation](#data-validation) section for more\ninformation.\n\n### Update\n\nThe `Update()` function accepts a couple of arguments:\n\n**`req`**\n[Request](models/models.go#L13) object\n\n**`partitionKey`**\nMapping of the partition key to value. For instance, if the table's partition key is `id`, it is expected this mapping\nwould be:\n\n```go\nmap[string]string{\n    \"id\": \"value\"\n}\n```\n\n**`item`**\n`map[string]string` of fields to be updated\n\n**`validation`**\n`map[string]utils.FieldValidation` of fields to perform validation against. Refer to the\n[data validation](#data-validation) section for more information.\n\n**`auditAction**`\nA string value to use as the Action in the audit logs. This should be set to `UPDATE` in most cases.\n\n### Delete\n\nThe `Delete()` function accepts a couple of arguments:\n\n**`req`**\n[Request](models/models.go#L13) object\n\n**`partitionKey`**\nMapping of the partition key to value. For instance, if the table's partition key is `id`, it is expected this mapping\nwould be:\n\n```python\nmap[string]string{\n    \"id\": \"value\"\n}\n```\n\n### List audit logs\n\nThe `ListAuditLogs()` function accepts:\n\n**`req`**\n[Request](models/models.go#L13) object\n\n**`pathParams`**\nAny search parameters to apply\n\n**`queryParams`**\nQuery parameters from API Gateway\n\n### History\n\n**`req`**\n[Request](models/models.go#L13) object\n\n**`key`**\nResource key to search on\n\n**`value`**\nResource value to search on\n\n**`queryParams`**\nQuery parameters from API Gateway\n\n**`actions`**\nList of actions to filter on\n\n## Filtering\n\nThere are two levels of filtering that are supported:\n- Path-based filtering\n- Querystring-based filtering\n\nThe `List()` function accepts a single `req` argument as a [Request](models/models.go#L13) object. For filtering to be\napplied, its `PathParams` and `QueryParams` fields should contain values. These are intended to contain the values\nof `PathParameters` and `QueryStringParameters`, respectively, that API Gateway passed into Lambda. In the case of\nnet/http, these values should be set using the `request.URL.Query()` function and any path parameters that\nare set by the `httprouter` package. Refer to the [net/http example](examples/oidc/aws/main.go) and the\n[InitHTTPServer](helpers/http.go#L38) function to see usage.\n\n### Dynamic path filters\n\nThe `List()` function also supports dynamic path filtering. When `search_key` and `search_value` are passed into\nthe method as `PathParams`, it will dynamically modify the path parameters to construct a search filter where\n\n```\nsearch_key = search_value\n```\n\nTo configure this in API Gateway, setup path parameters on the resource:\n```\n/endpoint/{search_key}/{search_value}\n```\n\nOr when using serverless:\n\n```yml\nevents:\n  - http:\n      path: endpoint\n      method: get\n      private: true\n  - http:\n      path: endpoint/{search_key}/{search_value}\n      method: get\n      private: true\n```\n\nWhen using the dynamic path filters, there is no need to construct additional endpoints that support filtering by a\nspecific key. However, using this method provides no limitations over what fields can be used as a filter. If that is a\nconcern for your API, you will need to construct static path filters.\n\n### Static path filters\n\nStatic path filters can be constructed in a similar manner to the dynamic path filters, except that the search key is\nmanually specified:\n\n```\n/endpoint/status/{status}\n```\n\nIn order to properly work, the path variable must _exactly_ match the key in the backend table that you want to perform\nthe filter against.\n\n### Querystring Filters\n\nIn addition to path filters, querystring filtering is also supported. The `List()` endpoint accepts all\nquerystrings via the `QueryParams` field of the request object. Each querystring should be a\n`field_name=search_value` format:\n\n```\n/endpoint?status=Active\u0026type=ABC\n```\n\nPath parameters **always** take precedence over querystring parameters. The below query:\n\n```\n/endpoint/type/ABC?status=Active\u0026type=Azure\n```\n\nWould result in this filter criteria:\n\n```\ntype = ABC AND status = Active\n```\n\n#### Magic Operators\n\nFor more complex queries, querystring search supports the below magic operations:\n- `in` (value is in list)\n- `notin` (value is not in list)\n- `ne` (not equal)\n- `startswith` (string starts with)\n- `contains` (string contains)\n- `notcontains` (string does not contain)\n- `exists` (attribute exists)\n- `gt` (greater than)\n- `lt` (less than)\n- `ge` (greater than or equal)\n- `le` (less than or equal)\n- `between` (value is between)\n\nNote that DynamoDBAPI does not support the `in` operation and FirestoreAPI only accepts the following magic operations:\n- in\n- gt\n- lt\n- ge\n- le\n- between\n\nThe MongoDBAPI supports these operations:\n- in\n- notin\n- ne\n- startswith\n- contains\n- exists\n- gt\n- ge\n- lt\n- le\n- between\n\nTo use a magic operator, append `__operator` to the key name. For example:\n\nTo search for all items with the `product` key containing the word \"Product\"\n\n```\n/items?product__contains=Product\n```\n\nTo search for all items with the `product` key starting with the word \"Product\"\n\n```\n/items?sku__startswith=Product\n```\n\nUsage of all the magic operators is straightforward, with the exception of the `in` and `between` operators. The `in`\noperators checks to see if the the value is included in a list of options. It should follow the JSON list syntax:\n\n```\n/items?product__in=[\"Product A\", \"Product B\"]\n```\n\nThe `between` operator checks to see if the value is, inclusively, between a low and high value. It should also follow\na JSON list syntax:\n\n```\n/items?num__between=[0, 3]\n```\n\nIt also works for string values, such as two dates:\n\n```\n/items?date__between=[\"2019-01-01\", \"2019-12-31\"]\n```\n\nTo find items that have an attribute:\n\n```\n/items?name__exists=true\n```\n\nTo search for items that do not have an attribute:\n\n```\n/items?name__exists=false\n```\n\n## Data validation\n\nFor convenience, support for data validation on all create and update calls is supported. In order to implement the\nvalidation, a `map[string]utils.FieldValidation` should be passed to the `validation` map of the `Create()` or\n`Update()` functions. The syntax of this object is outlined below.\n\nOn `Create()` calls, all items specified in the `validation` map are assumed to be required fields. If a\nfield is missing from the user input, an error will be thrown saying that the field is required.\n\n### Syntax\n```go\nvalidation := map[string]utils.FieldValidation{\n    \"field1\": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {\n        if value != \"hello\" {\n            return false, fmt.Sprintf(\"Invalid value '%s' for attribute 'field1'\", value), nil\n        }\n\n        return true, \"\", nil\n    },\n    \"field2\": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {\n        if value != \"world\" {\n            return false, fmt.Sprintf(\"Invalid value '%s' for attribute 'field2'\", value), nil\n        }\n\n        return true, \"\", nil\n    },\n}\n```\n\nThe key of each item in the dictionary should match a field name that you want to perform validation against. The\ncorresponding value for the key should be a callable that returns a boolean, string, and error. The boolean should be\n`true` if the field validated successfully, or `false` if it did not. The `string` should contain the error message\nthat should be displayed to the user. The `error` should be `nil` if there were not any errors while running validation.\nIf an error was encountered, this error value will be returned to the user.\n\n\nThe callable that you provide must accept three arguments:\n- `value` - Contains the input value for this field\n- `item` - Contains the entire data object that was passed from the user\n- `existingItem` - Contains the existing data object. This will only have a value on update calls. For\n    create calls, this will be `None`.\n\n### Example\n```go\nfunc validateUser(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {\n    var itemType string\n    if existingItem != nil {\n        itemType = existingItem[\"type\"]\n    } else {\n        itemType = item[\"type\"]\n    }\n\n    if _, ok := item[\"type\"]; !ok {\n        return false, \"Type field is required\", nil\n    }\n\n    if itemType == \"Type1\" {\n        re := regexp.MustCompile(\"^\\d{10}$\")\n        if re.MatchString(value) {\n            return true, \"\", nil\n        } else {\n            return false, \"Value does not match pattern\", nil\n        }\n    } else if itemType == \"Type2\" {\n        re := regexp.MustCompile(\"^[a-z]+$\")\n        if re.MatchString(value) {\n            return true, \"\", nil\n        } else {\n            return false, \"Value does not match pattern\", nil\n        }\n    } else {\n        return false, \"Validation failed\", nil\n    }\n}\n\nfieldValidation := map[string]utils.FieldValidation{\n    \"user\": validateUser,\n    \"type\": func(value string, item map[string]string, existingItem map[string]string) (bool, string, error) {\n        validOptions := []string{\"ABC\", \"DEF\"}\n\n        found := false\n        for _, item := range validOptions {\n            if item == value {\n                found = true\n                break\n            }\n        }\n\n        if !found {\n            return false, fmt.Sprintf(\"Invalid value. Supported options are %s\", validOptions), nil\n        }\n    }\n}\n```\n\n## [Sentry](https://sentry.io) support\n\nComing soon\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelpalmer1%2Fscoutr-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelpalmer1%2Fscoutr-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelpalmer1%2Fscoutr-go/lists"}