{"id":13680799,"url":"https://github.com/FRosner/drunken-data-quality","last_synced_at":"2025-04-30T00:30:56.289Z","repository":{"id":62579816,"uuid":"39070233","full_name":"FRosner/drunken-data-quality","owner":"FRosner","description":"Spark package for checking data quality","archived":false,"fork":false,"pushed_at":"2020-02-28T16:20:55.000Z","size":446,"stargazers_count":221,"open_issues_count":25,"forks_count":68,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-22T08:01:08.060Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FRosner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-14T11:10:51.000Z","updated_at":"2024-10-12T01:40:55.000Z","dependencies_parsed_at":"2022-11-03T20:49:17.051Z","dependency_job_id":null,"html_url":"https://github.com/FRosner/drunken-data-quality","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FRosner%2Fdrunken-data-quality","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FRosner%2Fdrunken-data-quality/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FRosner%2Fdrunken-data-quality/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FRosner%2Fdrunken-data-quality/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FRosner","download_url":"https://codeload.github.com/FRosner/drunken-data-quality/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251607413,"owners_count":21616754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T13:01:22.095Z","updated_at":"2025-04-30T00:30:51.272Z","avatar_url":"https://github.com/FRosner.png","language":"Scala","funding_links":[],"categories":["Table of Contents"],"sub_categories":["Frameworks and Libraries"],"readme":"# Drunken Data Quality (DDQ) \u003cimg src=\"https://raw.githubusercontent.com/FRosner/drunken-data-quality/master/logo/DDQ_small.png\" alt=\"Logo\" height=\"25\"\u003e\n\n[![Join the chat at https://gitter.im/FRosner/drunken-data-quality](https://badges.gitter.im/FRosner/drunken-data-quality.svg)](https://gitter.im/FRosner/drunken-data-quality?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge) [![Build Status](https://travis-ci.org/FRosner/drunken-data-quality.svg?branch=master)](https://travis-ci.org/FRosner/drunken-data-quality) [![Codacy Badge](https://api.codacy.com/project/badge/grade/b738700bad0a4b6da14e06c0dd508a21)](https://www.codacy.com/app/frank_7/drunken-data-quality) [![codecov.io](https://codecov.io/github/FRosner/drunken-data-quality/coverage.svg?branch=master)](https://codecov.io/github/FRosner/drunken-data-quality?branch=master) [![Known Vulnerabilities](https://snyk.io/test/github/frosner/drunken-data-quality/badge.svg)](https://snyk.io/test/github/frosner/drunken-data-quality)\n\n## Description\n\nDDQ is a small library for checking constraints on Spark data structures. It can be used to assure a certain data quality, especially when continuous imports happen.\n\n## Getting DDQ\n\n### Spark Package\n\nDDQ is available as a [spark package](http://spark-packages.org/package/FRosner/drunken-data-quality). You can add it to your spark-shell, spark-submit or pyspark using the `--packages` command line option:\n\n```sh\nspark-shell --packages FRosner:drunken-data-quality:4.1.1-s_2.11\n```\n\n### Python API\n\nDDQ also comes with a Python API. It is available via the Python Package Index, so you have to install it once using `pip`:\n\n```\npip install pyddq==4.1.1\n```\n\n### Project Dependency [![Latest Release](https://img.shields.io/github/tag/FRosner/drunken-data-quality.svg?label=JitPack)](https://jitpack.io/#FRosner/drunken-data-quality)\n\nIn order to use DDQ in your project, you can add it as a library dependency. This can be done through the [SBT spark package plugin](https://github.com/databricks/sbt-spark-package), or you can add it using [JitPack.io](https://jitpack.io/#FRosner/drunken-data-quality).\n\nKeep in mind that you might need to add additional resolvers as DDQ has some external dependencies starting from version 4.1.0:\n\n```\nresolvers += \"lightshed-maven\" at \"http://dl.bintray.com/content/lightshed/maven\"\n```\n\nIf neither of the above-mentioned ways work for you, feel free to download one of the compiled artifacts in the [release section](https://github.com/FRosner/drunken-data-quality/releases). Alternatively you may of course also build from source.\n\n## Using DDQ\n\n### Getting Started\n\nCreate two example tables to play with (or use your existing ones).\n\n```scala\ncase class Customer(id: Int, name: String)\ncase class Contract(id: Int, customerId: Int, duration: Int)\n\nval customers = spark.createDataFrame(List(\n  Customer(0, \"Frank\"),\n  Customer(1, \"Alex\"),\n  Customer(2, \"Slavo\")\n))\n\nval contracts = spark.createDataFrame(List(\n  Contract(0, 0, 5),\n  Contract(1, 0, 10),\n  Contract(0, 1, 6)\n))\n```\n\nRun some checks and see the results on the console.\n\n```scala\nimport de.frosner.ddq.core._\n\nCheck(customers)\n  .hasNumRows(_ \u003e= 3)\n  .hasUniqueKey(\"id\")\n  .run()\n\nCheck(contracts)\n  .hasNumRows(_ \u003e 0)\n  .hasUniqueKey(\"id\", \"customerId\")\n  .satisfies(\"duration \u003e 0\")\n  .hasForeignKey(customers, \"customerId\" -\u003e \"id\")\n  .run()\n```\n\n### Custom Reporters\n\nBy default the check result will be printed to stdout using ANSI escape codes to highlight the output. To have a report in another format, you can specify one or more [custom reporters](https://github.com/FRosner/drunken-data-quality/wiki).\n\n```scala\nimport de.frosner.ddq.reporters.MarkdownReporter\n\nCheck(customers)\n  .hasNumRows(_ \u003e= 3)\n  .hasUniqueKey(\"id\")\n  .run(MarkdownReporter(System.err))\n```\n\n### Running multiple checks\n\nYou can use a runner to generate reports for multiple checks at once. It will execute all the checks and report the results to the specified reporters.\n\n```scala\nimport de.frosner.ddq.reporters.ConsoleReporter\nimport java.io.{PrintStream, File}\n\nval check1 = Check(customers)\n  .hasNumRows(_ \u003e= 3)\n  .hasUniqueKey(\"id\")\n\nval check2 = Check(contracts)\n  .hasNumRows(_ \u003e 0)\n  .hasUniqueKey(\"id\", \"customerId\")\n  .satisfies(\"duration \u003e 0\")\n  .hasForeignKey(customers, \"customerId\" -\u003e \"id\")\n\nval consoleReporter = new ConsoleReporter(System.out)\nval markdownMd = new PrintStream(new File(\"report.md\"))\nval markdownReporter = new MarkdownReporter(markdownMd)\n\nRunner.run(Seq(check1, check2), Seq(consoleReporter, markdownReporter))\n\nmarkdownMd.close()\n```\n\n### Unit Tests\n\nYou can also use DDQ to write automated quality tests for your data. After running a check or a series of checks, you can inspect the results programmatically.\n\n```scala\ndef allConstraintsSatisfied(checkResult: CheckResult): Boolean =\n  checkResult.constraintResults.map {\n    case (constraint, ConstraintSuccess(_)) =\u003e true\n    case (constraint, ConstraintFailure(_)) =\u003e false\n  }.reduce(_ \u0026\u0026 _)\n\nval results = Runner.run(Seq(check1, check2), Seq.empty)\nassert(allConstraintsSatisfied(results(check1)))\nassert(allConstraintsSatisfied(results(check2)))\n```\n\nIf you want to fail the data load if the number of rows and the unique key constraints are not satisfied, but the duration constraint can be violated, you can write individual assertions for each constraint result.\n\n```scala\nval numRowsConstraint = Check.hasNumRows(_ \u003e= 3)\nval uniqueKeyConstraint = Check.hasUniqueKey(\"id\", \"customerId\")\nval durationConstraint = Check.satisfies(\"duration \u003e 0\")\n\nval check = Check(contracts)\n  .addConstraint(numRowsConstraint)\n  .addConstraint(uniqueKeyConstraint)\n  .addConstraint(durationConstraint)\n\nval results = Runner.run(Seq(check), Seq.empty)\nval constraintResults = results(check).constraintResults\nassert(constraintResults(numRowsConstraint).isInstanceOf[ConstraintSuccess])\nassert(constraintResults(uniqueKeyConstraint).isInstanceOf[ConstraintSuccess])\n```\n\n### Python API\n\nIn order to use the Python API, you have to start PySpark with the DDQ jar added. Unfortunately, using the `--packages` way is [not working in Spark \u003c 2.0](https://issues.apache.org/jira/browse/SPARK-5185).\n\n```\npyspark --driver-class-path drunken-data-quality_2.11-x.y.z.jar\n```\n\nThen you can create a dummy dataframe and run a few checks.\n\n```python\nfrom pyddq.core import Check\n\ndf = spark.createDataFrame([(1, \"a\"), (1, None), (3, \"c\")])\ncheck = Check(df)\ncheck.hasUniqueKey(\"_1\", \"_2\").isNeverNull(\"_1\").run()\n```\n\nJust as the Scala version of DDQ, PyDDQ supports multiple reporters.\nIn order to facilitate them, you can use `pyddq.streams`, which wraps the Java streams.\n\n```python\nfrom pyddq.reporters import MarkdownReporter, ConsoleReporter\nfrom pyddq.streams import FileOutputStream, ByteArrayOutputStream\nimport sys\n\n# send the report in a console-friendly format the standard output\n# and in markdown format to the bytearray\nstdout_stream = FileOutputStream(sys.stdout)\nbytearray_stream = ByteArrayOutputStream()\n\nCheck(df)\\\n    .hasUniqueKey(\"_1\", \"_2\")\\\n    .isNeverNull(\"_1\")\\\n    .run([MarkdownReporter(bytearray_stream), ConsoleReporter(stdout_stream)])\n\n# print markdown report\nprint bytearray_stream.get_output()\n```\n\n## Spark Version Compatibility\n\nAlthough we try to maintain as much compatibility between all available Spark versions we cannot guarantee that everything works smoothly for every possible combination of DDQ and Spark versions. The following matrix shows you what version of DDQ is built and tested against what version of Spark.\n\nDDQ Version | Spark Version\n--- | ---\n5.x | 2.2.x\n4.x | 2.0.x\n3.x | 1.6.x\n2.x | 1.3.x\n1.x | 1.3.x\n\n## Documentation\n\nFor a comprehensive list of available constraints, please refer to the [Wiki](https://github.com/FRosner/drunken-data-quality/wiki).\n\n## Authors\n\n- [Frank Rosner](https://github.com/FRosner) (Core, Reporters, Scala API)\n- [Aleksandr Sorokoumov](https://github.com/Gerrrr) (Python API)\n\nThanks also to everyone who submitted pull requests, bug reports and feature requests.\n\n## License\n\nThis project is licensed under the Apache License Version 2.0. For details please see the file called LICENSE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFRosner%2Fdrunken-data-quality","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFRosner%2Fdrunken-data-quality","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFRosner%2Fdrunken-data-quality/lists"}