{"id":20661847,"url":"https://github.com/mirantis/dataops-dataeng","last_synced_at":"2025-04-19T15:48:35.462Z","repository":{"id":63635670,"uuid":"379382201","full_name":"Mirantis/dataops-dataeng","owner":"Mirantis","description":"A tool to scrape data from multiple integrations","archived":false,"fork":false,"pushed_at":"2024-10-01T14:00:12.000Z","size":187,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-10T10:17:25.141Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mirantis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-06-22T19:40:33.000Z","updated_at":"2023-08-03T19:36:44.000Z","dependencies_parsed_at":"2024-04-19T15:10:54.328Z","dependency_job_id":null,"html_url":"https://github.com/Mirantis/dataops-dataeng","commit_stats":null,"previous_names":["mirantis/dataeng"],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mirantis%2Fdataops-dataeng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mirantis%2Fdataops-dataeng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mirantis%2Fdataops-dataeng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mirantis%2Fdataops-dataeng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mirantis","download_url":"https://codeload.github.com/Mirantis/dataops-dataeng/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249730636,"owners_count":21317327,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T19:11:51.868Z","updated_at":"2025-04-19T15:48:35.444Z","avatar_url":"https://github.com/Mirantis.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Engineering - dataeng\n\n![Image](https://github.com/Mirantis/dataeng/blob/main/.assets/logos/dataeng_logo_1.png)\n\n## Status Checks\n\n[![Greetings](https://github.com/Mirantis/dataeng/actions/workflows/greetings.yml/badge.svg)](https://github.com/Mirantis/dataeng/actions/workflows/greetings.yml)\n[![Labeler](https://github.com/Mirantis/dataeng/actions/workflows/label.yml/badge.svg)](https://github.com/Mirantis/dataeng/actions/workflows/label.yml)\n\nThe dataeng project is dedicated to storing all data interactions across an organization as IaC.\nIt provides a Command Line Interface to interact witht the various software tools that are used on a daily basis.\nIt also provides an anlaysis perspective, and offers a wide variety of tooling in terms of analyzing overall operational status of the entire organization for all products.\n\n## Repository Structure\n\nThis repository is loosely organized into 11 main categories:\n\n\u003c!-- TODO: REFACTOR --\u003e\n1. [.assets](https://github.com/Mirantis/dataeng/tree/main/.assets)\n    - An assets directory housing images and assets used throughout the directory (EACH FILE HAS AN MD5SUM TO VERIFY FILE INTEGRITY)\n\n2. [.docker](https://github.com/Mirantis/dataeng/tree/main/.docker)\n    - A configuration and secrets directory for all things docker\n\n4. [.github](https://github.com/Mirantis/dataeng/tree/main/.github/)\n    - A configuration and secrets directory for all things github\n\n5. [.kube](https://github.com/Mirantis/dataeng/tree/main/.kube)\n    - A configuration and secrets directory for all things kubernetes\n\n6. [.secrets](https://github.com/Mirantis/dataeng/tree/main/.secrets)\n    - A configuration and secrets directory for all things secret\n\n7. [Analyses](https://github.com/Mirantis/dataeng/tree/main/Analyses)\n    - The Analyses directory that houses a variety of analyses across multiple toolings\n\n8. [Applications](https://github.com/Mirantis/dataeng/tree/main/Applications)\n    - The Applications directory houses any and all affiliated applications to the housing repository\n\n9. [Automation](https://github.com/Mirantis/dataeng/tree/main/Automation)\n    - The Automation directory will contain automation and tooling for deploying and utilizing the repository\n\n10. [Charts](https://github.com/Mirantis/dataeng/tree/main/Charts)\n    - The Charts directory will container Helm Charts related to this repository\n\n11. [Infrastructure](https://github.com/Mirantis/dataeng/tree/main/Infrastructure)\n    - The Infrastrucutre directory contains information relate to Ansible and Terraform for continuously deployed infrastructure\n\n12. [Lakes](https://github.com/Mirantis/dataeng/tree/main/Lakes)\n    - The Lakes Directory will house the information related to any and all data lakes within the organization\n\n13. [Pipelines](https://github.com/Mirantis/dataeng/tree/main/Pipelines)\n    - This will contain any pipelines related to the overall repisitory and is tooling dependent\n\n14. [Queries](https://github.com/Mirantis/dataeng/tree/main/Queries)\n    - The Queries directory contains SQL Like queries that are stored for IaC purposes\n\n15. [Warehouse](https://github.com/Mirantis/dataeng/tree/main/Warehouse)\n    - The Warehouse directory contains information on how to interact with the Data Warehouse for the respective organization\n\n16. [dataengctl](https://github.com/Mirantis/dataeng/tree/main)\n    - The dataengctl houses the binary that allows you to work with a variety of tooling at the organization's disposal\n\n## Navigating Repository\n\nThe above showcases the directories that are contained within the repository.\nAll directories labeled with `.` in front, such as `.secrets` are meant to be ignored, and nothing should ever be commited inside of these directories. The main directories are each provided with a `README.md` that tells you how you should interact with the repository in that particular directory.\n\n## Getting Started with dataenctl\n\nTo get started with `dataengctl`, you should create a configuration file wihtin your home directory.\nIt should be named `~/.dataeng` and should be a hidden directory. In this directory you should create a file called `config.yaml`, which contains the secrets and credentials needed to work with `dataengctl`. The config should look something like this:\n\n```bash\njiraConfig:\n  token: YOURJIRATOKEN\n  url: YOURJIRAURL\n  username: YOURUSERNAME\nsalesForceConfig:\n  url: YOURSALESFORCEURL\n  username: YOURSALESFORCEUSERNAME\n  password: YOURSALESFORCEPASSWORD\n  token: YOURSALESFORCETOKEN\n  clientID: YOURSALESFORCEORGCLIENTID\n  apiVersion: YOURSALESFORCEDEFUALTAPI\n```\n\n## Installing dataengctl\n\nTo install dataengctl you need to make sure you have two things installed\n\n1. You Want `jq` installed on your local host\n2. You want `go` installed on your local host\n\n## Working with dataengctl\n\nTo work with dataengctl you can use the `makefile` that is housed within the dataengctl directory, or you can build it directly in that directory. All Binary files are ignored by default within that directory.\n\n1. Build the dataengctl binary\n    - `go build -o dataengctl`\n2. Afterwards run the binary specifying it with `--help`\n\n```bash\n rbarrett@MacBook-Pro-2  ~/Git/dataeng   DATAENG-19 ●  . --help                                                      1 ↵  10130  11:38:37\ngets you data from different sources\n\nUsage:\n  dataengctl [command]\n\nAvailable Commands:\n  analyze     Analyze something\n  completion  generate the autocompletion script for the specified shell\n  help        Help about any command\n  jira        Interact with jira\n  salesforce  Interact with salesforce\n\nFlags:\n      --config-file string   path to config file\n      --debug                specify debug level\n  -h, --help                 help for dataengctl\n\nUse \"dataengctl [command] --help\" for more information about a command.\n```\n\nAs a result, you can see that there are several commands that are available\n\n- 1. analyze\n- 2. completion\n- 3. help\n- 4. jira\n- 5. salesforce\n\nTo interact with Jira and Salesforce or anything that is using the `config.yaml` mappings, you will need to specify the command as follows:\n\n- 1. Intreracting with `analyze` command specifying the config path\n\n```bash\n. analyze --issue-type Escalation --project-key FIELD --config-file ${HOME}/.dataeng/config.yaml | jq \".\"\n```\n\n- 2. Interacting with `jira` command specifying the config path\n\n```bash\n. jira issue list --issue-type Escalation --project-key FIELD --config-file ${HOME}/.dataeng/config.yaml | jq \".pri\"\n```\n\n- 3. Interacting with `salesforce` command specifying the config path\n\n```bash\n. salesforce query \"SELECT+name+from+Account\" --config-file ${HOME}/.dataeng/config.yaml | jq \".\"\n```\n\nWithout `jq` installed on your machine the default `JSON` output will not look pretty. All output is default to `JSON` format.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirantis%2Fdataops-dataeng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmirantis%2Fdataops-dataeng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmirantis%2Fdataops-dataeng/lists"}