{"id":22787398,"url":"https://github.com/aserto-dev/ds-load","last_synced_at":"2025-04-30T12:45:09.275Z","repository":{"id":177037941,"uuid":"636182203","full_name":"aserto-dev/ds-load","owner":"aserto-dev","description":"CLI pipeline for populating the directory","archived":false,"fork":false,"pushed_at":"2025-04-22T08:41:48.000Z","size":1012,"stargazers_count":2,"open_issues_count":7,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-22T09:48:13.901Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aserto-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-04T09:41:26.000Z","updated_at":"2025-04-11T13:59:51.000Z","dependencies_parsed_at":"2024-03-08T20:37:33.656Z","dependency_job_id":"387b8afe-0c0e-4b94-8d06-6be82e59d876","html_url":"https://github.com/aserto-dev/ds-load","commit_stats":null,"previous_names":["aserto-dev/ds-load"],"tags_count":49,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aserto-dev%2Fds-load","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aserto-dev%2Fds-load/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aserto-dev%2Fds-load/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aserto-dev%2Fds-load/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aserto-dev","download_url":"https://codeload.github.com/aserto-dev/ds-load/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251703190,"owners_count":21630194,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-12T00:56:11.891Z","updated_at":"2025-04-30T12:45:09.267Z","avatar_url":"https://github.com/aserto-dev.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ds-load\n\n`ds-load` is a CLI pipeline for populating the Topaz directory (or directories that are contract-compatible, such as the Aserto directory).\n\n## Arguments\n\n```\nUsage: ds-load \u003ccommand\u003e\n\nDirectory loader\n\nCommands:\n  exec                  import data in directory by running fetch, transform and publish\n  publish               load data from stdin into directory\n  get-plugin            download plugin\n  set-default-plugin    sets a plugin as default\n  list-plugins          list available plugins\n  version               version information\n\nFlags:\n  -h, --help                  Show context-sensitive help.\n  -c, --config=CONFIG-FLAG    Path to the config file. Any argument provided to the CLI will take precedence.\n  -v, --verbosity=INT         Use to increase output verbosity.\n```\n\nThe `ds-load` pipeline has three stages: `fetch`, `transform`, and `publish`:\n* `fetch` retrieves plugin-specific data from the source in a native JSON format\n* `transform` converts this data into directory objects and relations\n* `publish` loads the objects and relations data into the directory\n\nThe default command for ds-load is `exec`, which executes all three stages. When running `ds-load` without any command, the exec parameters need to be passed.\n\nNote: the plugin examples below all use `auth0`, but every one of the plugins (`azuread`, `cognito`, `google`, `okta`, etc) follow the same patterns.\n\n### ds-load vs plugin parameters\nThe ds-load CLI parameters need to be passed first, and can be followed by an arbitrary list of positional parameters. The first positional parameter is the plugin name we want to invoke followed by the plugin's parameters.\n\nExample: `ds-load --host=\u003cdirectory host\u003e auth0 --domain=\u003cauth0 domain\u003e`\n* `--host` is a CLI parameter for `ds-load`\n* `auth0` is the plugin name\n* `--domain` is a parameter for the `auth0` plugin\n\nFor viewing the plugin help, use the following format: `ds-load auth0 --help`.\n\nTip: when running `ds-load auth0 --key`, `auth0` is a positional parameter, so `--key` will be run in the context of the plugin. If we run `ds-load --key auth0`, `--key` would be a parameter to `ds-load exec`.\n\n### ds-load exec\n`exec` is the default command. it will invoke a plugin with the specified parameters reading its output and importing the resulting data into the directory.\n\n```\nUsage: ds-load exec \u003ccommand\u003e ...\n\nimport data in directory by running fetch, transform and publish\n\nArguments:\n  \u003ccommand\u003e ...    available commands are: auth0|azuread|cognito|google|okta\n\nFlags:\n  -h, --help                  Show context-sensitive help.\n  -c, --config=CONFIG-FLAG    Path to the config file. Any argument provided to the CLI will take precedence.\n  -v, --verbosity=INT         Use to increase output verbosity.\n\n  -s, --host=STRING           Directory host address ($DIRECTORY_HOST)\n  -k, --api-key=STRING        Directory API Key ($DIRECTORY_API_KEY)\n  -i, --insecure              Disable TLS verification\n  -t, --tenant-id=STRING      Directory Tenant ID ($DIRECTORY_TENANT_ID)\n  -p, --print                 print output to stdout\n```\n\n`-p/--print` is enabled by default when invoking a plugin with `fetch/version/export-transform/--help`\n\n### Environment variables\nParameters can also be passed using environment variables, as seen in the help message of each command, but the ones from config files and command line take precedence.\n\n## Config files\n\nConfig files are in yaml format:\n```yaml\n---\narg: value\nanother-arg: value\n\u003cplugin-name\u003e:\n  arg: value\n```\n\nWhen passing custom config files to both the cli and the plugin, use `ds-load -c \u003cconfig-path\u003e \u003cplugin-name\u003e \u003ccommand\u003e` \n\n### CLI config\nThe default location for the configuration file is `~/.config/ds-load/cfg/config.yaml`. It can be overridden using the `-c/--config` flag.\n\n#### example with auth0 plugin\n```yaml\n---\nhost: directory.prod.aserto.com:8443\napi-key: \"secretapikey\"\ntenant-id: your-tenant-id\nauth0:\n  domain: \"domain.auth0.com\"\n  client-id: \"clientid\"\n  client-secret: \"clientsupersecret\"\n  template: \"/path/to/transform.file\"\n```\n\n### Plugin config\nThe default location for plugin configuration files is `~/.config/ds-load/cfg/\u003cplugin-name\u003e.yaml`. It can be overridden using the `-c/--config` flag.\n\n#### example for auth0\n```yaml\n---\nauth0:\n  domain: \"domain.auth0.com\"\n  client-id: \"clientid\"\n  client-secret: \"clientsupersecret\"\n  template: \"/path/to/transform.file\"\n```\n\n## Transform\nThe data received from the fetcher is transformed into objects and relations using a transformation template, which is uses the go template syntax.\n\nThe default transformation template can be exported using `ds-load \u003cplugin-name\u003e export-transform`.\n\nA custom transformation file can be provided when running the plugin in `exec` or `transform` mode via the `--template` parameter.\n\nMore information on the transformation template language can be found in the [tranform template docs](./docs/templates.md).\n\n## Logs\n\nLogs are printed to `stdout`. You can increase detail using the verbosity flag (e.g. `-vvv`).\n\n## Fetching source data\n\nMost plugins fetch entire objects that are available in the transform template. `azuread` and `azureadb2c` use the msgraph api to query the data and by default only query only properties that are needed in the default transform template.\n\n### AzureAD and AzureADB2C\n\nTo use a custom porperty list in the query, you can use the CLI parameters or configure them in your config:\n\n```\nazuread:\n  tenant: \"tenant-id\"\n  client-id: \"client-id\"\n  client-secret: \"secret\"\n  groups: true\n  user-properties: [\"id\", \"displayName\"]\n  group-properties: [\"id\", \"displayName\"]\n```\n\nAzureAD Groups:\n\n- displayName\n- id\n- mail\n- createdDateTime\n- mailNickname\n\nAzureADB2C Users:\n\n- displayName\n- id\n- mail\n- createdDateTime\n- mobilePhone\n- userPrincipalName\n- accountEnabled\n- identities\n- creationType\n\nAzureADB2C Groups:\n\n- displayName\n- id\n- mail\n- createdDateTime\n- mailNickname\n- members\n- transitiveMembers\n\nA list of all available properties is available on the Microsoft website for [user object type](https://learn.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0#properties) and [group object type](https://learn.microsoft.com/en-us/graph/api/resources/group?view=graph-rest-1.0#properties)\n\n## Usage examples\n\n### Import from auth0 into the directory\n```\nds-load --host=\u003cdirectory-host\u003e --api-key=\u003cdirectory-api-key\u003e --tenant-id=\u003ctenant-id\u003e auth0 --domain=\u003cauth0-domain\u003e --client-id=\u003cauth0-client-id\u003e --client-secret=\u003cauth0-client-secret\u003e\n```\n\n### Import data with a custom transformation file\n```\nds-load --host=\u003cdirectory-host\u003e --api-key=\u003cdirectory-api-key\u003e --tenant-id=\u003ctenant-id\u003e auth0 --domain=\u003cauth0-domain\u003e --client-id=\u003cauth0-client-id\u003e --client-secret=\u003cauth0-client-secret\u003e --template=\u003ctemplate-path\u003e\n```\n\n### Fetch data from auth0 without importing it\n```\nds-load auth0 fetch --domain=\u003cauth0-domain\u003e --client-id=\u003cauth0-client-id\u003e --client-secret=\u003cauth0-client-secret\u003e\n```\n\n### Transform data from a previously saved auth0 fetch\nNote: we use `-p` in order to just print the transform data.\n```\nds-load auth0 fetch --domain=\u003cauth0-domain\u003e --client-id=\u003cauth0-client-id\u003e --client-secret=\u003cauth0-client-secret\u003e \u003e auth0.json\ncat auth0.json | ds-load -p auth0 transform\n```\n\n### Transform and import data from a previously saved auth0 fetch\n```\nds-load auth0 fetch --domain=\u003cauth0-domain\u003e --client-id=\u003cauth0-client-id\u003e --client-secret=\u003cauth0-client-secret\u003e \u003e auth0.json\n\ncat auth0.json | ds-load --host=\u003cdirectory-host\u003e --api-key=\u003cdirectory-api-key\u003e --tenant-id=\u003ctenant-id\u003e auth0 transform\n```\n\n### Pipe data from fetch to transform\n```\nds-load auth0 fetch --domain=\u003cauth0-domain\u003e --client-id=\u003cauth0-client-id\u003e --client-secret=\u003cauth0-client-secret\u003e | ds-load -p auth0 transform\n```\n\n### Use config file to import data from auth0 into the directory\n\nconfig.yaml\n```yaml\n---\nhost: \"directory.prod.aserto.com:8443\"\napi-key: \"secretapikey\"\ntenant-id: \"your-tenant-id\"\nauth0:\n  domain: \"domain.auth0.com\"\n  client-id: \"clientid\"\n  client-secret: \"clientsupersecret\"\n```\n\n```\nds-load -c ./config.yaml auth0\n```\n\n### Load directory data from a file\n```\nds-load -p auth0 --domain=\u003cauth0-domain\u003e --client-id=\u003cauth0-client-id\u003e --client-secret=\u003cauth0-client-secret\u003e \u003e auth0.json\n\ncat auth0.json | ds-load publish --host=\u003cdirectory-host\u003e --api-key=\u003cdirectory-api-key\u003e --tenant-id=\u003ctenant-id\u003e\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faserto-dev%2Fds-load","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faserto-dev%2Fds-load","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faserto-dev%2Fds-load/lists"}