{"id":18810406,"url":"https://github.com/absaoss/hermes","last_synced_at":"2025-04-13T20:31:03.579Z","repository":{"id":43058514,"uuid":"211824332","full_name":"AbsaOSS/hermes","owner":"AbsaOSS","description":"A E2E test tool for Enceladus. Also general dataframe comparison tool","archived":false,"fork":false,"pushed_at":"2023-05-05T12:51:13.000Z","size":408,"stargazers_count":8,"open_issues_count":15,"forks_count":3,"subscribers_count":16,"default_branch":"develop","last_synced_at":"2024-04-12T07:05:52.422Z","etag":null,"topics":["atum","dataset-comparison","e2e-tests","enceladus","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AbsaOSS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-30T09:29:26.000Z","updated_at":"2023-04-11T15:27:15.000Z","dependencies_parsed_at":"2023-02-19T06:30:25.471Z","dependency_job_id":null,"html_url":"https://github.com/AbsaOSS/hermes","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fhermes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fhermes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fhermes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fhermes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AbsaOSS","download_url":"https://codeload.github.com/AbsaOSS/hermes/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223603279,"owners_count":17172073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atum","dataset-comparison","e2e-tests","enceladus","spark"],"created_at":"2024-11-07T23:20:06.184Z","updated_at":"2024-11-07T23:20:07.334Z","avatar_url":"https://github.com/AbsaOSS.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Enceladus TestUtils\n\n- [Enceladus TestUtils](#enceladus-testutils)\n  - [To Build](#to-build)\n  - [Dataset Comparison](#dataset-comparison)\n    - [Running](#running)\n      - [Where](#where)\n  - [Info File Comparison](#info-file-comparison)\n    - [Running](#running-1)\n  - [E2E Runner](#e2e-runner)\n\nHermes is an E2E testing tool created mainly for the use in [ABSA OSS][gh-absa] ecosystem but still provides some tools/utils that are usable in other projects and are quite generic. For more information, please look at our [Hermes Github Pages][gh-pages].\n\n## To Build\n\nUse either of the commands below. Depending on your versions.\n```bash\nsbt ++2.11.12 assembly -DSPARK_VERSION=2.4.7\nsbt ++2.12.12 assembly -DSPARK_VERSION=3.2.2\n```\n\n## Known to work with:\n\n- Spark 2.4.2 - 3.2.2 [1]\n- Java 1.8.0_191-b12\n- Scala 2.11.12 and 2.12.12\n\n[1] There are now spark version guards to protect from false positives. \nIf there is someone willing to test for older versions, we are happy to extend these.\nThese are applicable only to use as a spark-job not as a library\n\n## How to generate Code coverage report\n```\nsbt ++2.11.12 jacoco -DSPARK_VERSION=2.4.7\nsbt ++2.12.12 jacoco -DSPARK_VERSION=3.2.2\n```\nCode coverage will be generated on path:\n```\n{project-root}/{module}/target/scala-{scala_version}/jacoco/report/html\n```\n\n\n## Dataset Comparison\n\nSpark job for the comparison of data sets. As it leverages spark, there are almost no limitations to data sources and size of the data.\n\n### Running\n\nBasic running example\n```bash\nspark-submit \\\n/path/to/jar/file \\\n--format \u003cformat of the reference and new data sets\u003e \\\n--new-path /path/to/new/data/set \\\n--ref-path /path/to/referential/data/set \\\n--out-path /path/to/diff/output\n--keys key1,key2,key3\n```\n\n#### Where\n```bash\nDatasets Comparison \nUsage: spark-submit [spark options] --class za.co.absa.hermes.datasetComparison.DatasetComparisonJob hermes.jar [options]\n\n  --[ref|new]-format            Format of the raw data (csv, xml, parquet,fixed-width, etc.). Use prefix only in case\n                                    comparison of two different formats. Mandatory.\n  --new-path|--new-dbtable      Path to the new dataset or dbtable (jdbc), just generated and to be tested. Mandatory.\n  --ref-path|--ref-dbtable      Path to supposedly correct data set or dbtable (jdbc). Mandatory.\n  --out-path.                    Path to where the `ComparisonJob` will save the differences. \n                                    This will efectivly creat a folder in which you will find two \n                                    other folders. expected_minus_actual and actual_minus_expected.\n                                    Both hold parque data sets of differences. (minus as in is \n                                    relative complement. Mandatory.\n  --keys                        If there are know unique keys, they can be specified for better\n                                   output. Keys should be specified one by one, with , (comma) \n                                   between them. Optional.\n  others                        Other options depends on selected format specifications (e.g. --delimiter and --header for\n                                   csv, --rowTag for xml). For case comparison of two different formats use prefix ref|new\n                                   for each of this options. For more information, check sparks documentation on what all\n                                   the options for the format you are using. Optional.\n  \n  --help                   prints similar text to this one.\n  \n```\n\nOther configurations are Spark dependant and are out of scope of this README.\n\n## Info File Comparison\n\n[Atum][atum]'s (and it's derivatives) Info file comparison. Ran as part of the E2E Runner. Can be run as a plain old jar file.\n\n### Running\n\nBasic running example\n\n```bash\njava -jar \\\n/path/to/jar/file \\\n--new-path /path/to/new/data/set \\\n--ref-path /path/to/referential/data/set \\\n--out-path /path/to/diff/output\n```\n\n## E2E Runner\n\nCurrently runs both Standardization and Conformance of [Enceladus][enceladus] project on the data provided. After each, a comparison job is run to check the results against expected reference data.\n\nThis tool is planed for an upgrade in the nearest future to be a general E2E Runner for user defined runes.\n\nBasic running example:\n\n```bash\nspark-submit \\\n/path/to/jar/file \\\n--menas-credentials-file /path/to/credentials/file \\\n--dataset-name \u003cdatasetName\u003e \\\n--dataset-version \u003cdatasetVersion\u003e \\\n--report-date \u003creportData\u003e \\\n--report-version \u003creportVersion\u003e \\\n--raw-format \u003crawFormat\u003e\n--keys \u003ckey1,key2,...\u003e\n```\n\n[gh-absa]: https://github.com/AbsaOSS\n[gh-pages]: https://absaoss.github.io/hermes/\n[atum]: https://github.com/AbsaOSS/atum\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fhermes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabsaoss%2Fhermes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fhermes/lists"}