https://github.com/aserto-dev/ds-load
CLI pipeline for populating the directory
https://github.com/aserto-dev/ds-load
Last synced: about 1 year ago
JSON representation
CLI pipeline for populating the directory
- Host: GitHub
- URL: https://github.com/aserto-dev/ds-load
- Owner: aserto-dev
- License: apache-2.0
- Created: 2023-05-04T09:41:26.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2025-04-22T08:41:48.000Z (about 1 year ago)
- Last Synced: 2025-04-22T09:48:13.901Z (about 1 year ago)
- Language: Go
- Size: 988 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ds-load
`ds-load` is a CLI pipeline for populating the Topaz directory (or directories that are contract-compatible, such as the Aserto directory).
## Arguments
```
Usage: ds-load
Directory loader
Commands:
exec import data in directory by running fetch, transform and publish
publish load data from stdin into directory
get-plugin download plugin
set-default-plugin sets a plugin as default
list-plugins list available plugins
version version information
Flags:
-h, --help Show context-sensitive help.
-c, --config=CONFIG-FLAG Path to the config file. Any argument provided to the CLI will take precedence.
-v, --verbosity=INT Use to increase output verbosity.
```
The `ds-load` pipeline has three stages: `fetch`, `transform`, and `publish`:
* `fetch` retrieves plugin-specific data from the source in a native JSON format
* `transform` converts this data into directory objects and relations
* `publish` loads the objects and relations data into the directory
The default command for ds-load is `exec`, which executes all three stages. When running `ds-load` without any command, the exec parameters need to be passed.
Note: the plugin examples below all use `auth0`, but every one of the plugins (`azuread`, `cognito`, `google`, `okta`, etc) follow the same patterns.
### ds-load vs plugin parameters
The ds-load CLI parameters need to be passed first, and can be followed by an arbitrary list of positional parameters. The first positional parameter is the plugin name we want to invoke followed by the plugin's parameters.
Example: `ds-load --host= auth0 --domain=`
* `--host` is a CLI parameter for `ds-load`
* `auth0` is the plugin name
* `--domain` is a parameter for the `auth0` plugin
For viewing the plugin help, use the following format: `ds-load auth0 --help`.
Tip: when running `ds-load auth0 --key`, `auth0` is a positional parameter, so `--key` will be run in the context of the plugin. If we run `ds-load --key auth0`, `--key` would be a parameter to `ds-load exec`.
### ds-load exec
`exec` is the default command. it will invoke a plugin with the specified parameters reading its output and importing the resulting data into the directory.
```
Usage: ds-load exec ...
import data in directory by running fetch, transform and publish
Arguments:
... available commands are: auth0|azuread|cognito|google|okta
Flags:
-h, --help Show context-sensitive help.
-c, --config=CONFIG-FLAG Path to the config file. Any argument provided to the CLI will take precedence.
-v, --verbosity=INT Use to increase output verbosity.
-s, --host=STRING Directory host address ($DIRECTORY_HOST)
-k, --api-key=STRING Directory API Key ($DIRECTORY_API_KEY)
-i, --insecure Disable TLS verification
-t, --tenant-id=STRING Directory Tenant ID ($DIRECTORY_TENANT_ID)
-p, --print print output to stdout
```
`-p/--print` is enabled by default when invoking a plugin with `fetch/version/export-transform/--help`
### Environment variables
Parameters can also be passed using environment variables, as seen in the help message of each command, but the ones from config files and command line take precedence.
## Config files
Config files are in yaml format:
```yaml
---
arg: value
another-arg: value
:
arg: value
```
When passing custom config files to both the cli and the plugin, use `ds-load -c `
### CLI config
The default location for the configuration file is `~/.config/ds-load/cfg/config.yaml`. It can be overridden using the `-c/--config` flag.
#### example with auth0 plugin
```yaml
---
host: directory.prod.aserto.com:8443
api-key: "secretapikey"
tenant-id: your-tenant-id
auth0:
domain: "domain.auth0.com"
client-id: "clientid"
client-secret: "clientsupersecret"
template: "/path/to/transform.file"
```
### Plugin config
The default location for plugin configuration files is `~/.config/ds-load/cfg/.yaml`. It can be overridden using the `-c/--config` flag.
#### example for auth0
```yaml
---
auth0:
domain: "domain.auth0.com"
client-id: "clientid"
client-secret: "clientsupersecret"
template: "/path/to/transform.file"
```
## Transform
The data received from the fetcher is transformed into objects and relations using a transformation template, which is uses the go template syntax.
The default transformation template can be exported using `ds-load export-transform`.
A custom transformation file can be provided when running the plugin in `exec` or `transform` mode via the `--template` parameter.
More information on the transformation template language can be found in the [tranform template docs](./docs/templates.md).
## Logs
Logs are printed to `stdout`. You can increase detail using the verbosity flag (e.g. `-vvv`).
## Fetching source data
Most plugins fetch entire objects that are available in the transform template. `azuread` and `azureadb2c` use the msgraph api to query the data and by default only query only properties that are needed in the default transform template.
### AzureAD and AzureADB2C
To use a custom porperty list in the query, you can use the CLI parameters or configure them in your config:
```
azuread:
tenant: "tenant-id"
client-id: "client-id"
client-secret: "secret"
groups: true
user-properties: ["id", "displayName"]
group-properties: ["id", "displayName"]
```
AzureAD Groups:
- displayName
- id
- mail
- createdDateTime
- mailNickname
AzureADB2C Users:
- displayName
- id
- mail
- createdDateTime
- mobilePhone
- userPrincipalName
- accountEnabled
- identities
- creationType
AzureADB2C Groups:
- displayName
- id
- mail
- createdDateTime
- mailNickname
- members
- transitiveMembers
A list of all available properties is available on the Microsoft website for [user object type](https://learn.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0#properties) and [group object type](https://learn.microsoft.com/en-us/graph/api/resources/group?view=graph-rest-1.0#properties)
## Usage examples
### Import from auth0 into the directory
```
ds-load --host= --api-key= --tenant-id= auth0 --domain= --client-id= --client-secret=
```
### Import data with a custom transformation file
```
ds-load --host= --api-key= --tenant-id= auth0 --domain= --client-id= --client-secret= --template=
```
### Fetch data from auth0 without importing it
```
ds-load auth0 fetch --domain= --client-id= --client-secret=
```
### Transform data from a previously saved auth0 fetch
Note: we use `-p` in order to just print the transform data.
```
ds-load auth0 fetch --domain= --client-id= --client-secret= > auth0.json
cat auth0.json | ds-load -p auth0 transform
```
### Transform and import data from a previously saved auth0 fetch
```
ds-load auth0 fetch --domain= --client-id= --client-secret= > auth0.json
cat auth0.json | ds-load --host= --api-key= --tenant-id= auth0 transform
```
### Pipe data from fetch to transform
```
ds-load auth0 fetch --domain= --client-id= --client-secret= | ds-load -p auth0 transform
```
### Use config file to import data from auth0 into the directory
config.yaml
```yaml
---
host: "directory.prod.aserto.com:8443"
api-key: "secretapikey"
tenant-id: "your-tenant-id"
auth0:
domain: "domain.auth0.com"
client-id: "clientid"
client-secret: "clientsupersecret"
```
```
ds-load -c ./config.yaml auth0
```
### Load directory data from a file
```
ds-load -p auth0 --domain= --client-id= --client-secret= > auth0.json
cat auth0.json | ds-load publish --host= --api-key= --tenant-id=
```