{"id":22353530,"url":"https://github.com/dp6/pipeline-penguin","last_synced_at":"2026-02-28T05:22:12.479Z","repository":{"id":38189920,"uuid":"308403136","full_name":"DP6/pipeline-penguin","owner":"DP6","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-14T17:43:54.000Z","size":1520,"stargazers_count":6,"open_issues_count":0,"forks_count":3,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-11-28T23:32:27.037Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://dp6.github.io/pipeline-penguin","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DP6.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-29T17:34:44.000Z","updated_at":"2024-08-26T01:40:15.000Z","dependencies_parsed_at":"2022-08-19T05:30:27.091Z","dependency_job_id":null,"html_url":"https://github.com/DP6/pipeline-penguin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fpipeline-penguin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fpipeline-penguin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fpipeline-penguin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DP6%2Fpipeline-penguin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DP6","download_url":"https://codeload.github.com/DP6/pipeline-penguin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228114910,"owners_count":17871742,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-04T13:08:55.334Z","updated_at":"2026-02-28T05:22:12.411Z","avatar_url":"https://github.com/DP6.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Description\n\nPipeline Penguin is a versatile python library for data quality.\n\n## Documentation\n\n- [Reference](https://dp6.github.io/pipeline-penguin/pipeline_penguin.html)\n\n## Getting Started\n\n### How to install\n\nYou can use PyPI Test to install the early develoment build by executing this command:\n\n```\npip install pipeline-penguin\n```\n\n### Core Concepts\n\nBefore you start executing validations on you data, firstly you need to understand a few concepts:\n\n#### Data Node\nAny database your pipeline needs to process. It has a Node Type, that identifies what is the source of that data, like BigQuery. \n\n#### Data Premise\nAny premise on a Data Node, for example, the column \"name\" on the database \"employers\" must not have a number on it. Data Premises also have a type called Premise Type. If an Premise Type is SQL, you cannot execute a validation on a Data Node with a Node Type that does not have a SQL engine to process that query.\n\n#### Connector\nThe way you access an Data Node to check a Data Premise. \n\n#### Premise Output\nIs the result of a Data Premise validation on a Data Node.\n\n### Implementing a test\n\n- Importing and instantiating PP\n\n```python\nfrom pipeline_penguin import PipelinePenguin\npp = PipelinePenguin()\n```\n\n- Defining the default connector\n\n```python\nbq_connector = ConnectorSQLBigQuery('/config/service_account.json')\npp.connectors.define_default(bq_connector)\n```\n\n- Creating a Data Node\n\n```python\nnode = pp.nodes.create_node('Node Name', DataNodeBigQuery, project_id='example', dataset_id='example', table_id='example')\npp.nodes.list_nodes()\n```\n\n- Creating a Data Premise\n\n```python\nnode.insert_premise('Premise Name', DataPremiseSQLCheckIsNull, \"Column Name\")\n```\n\n- Executing a validation\n\n```python\npp.nodes.run_premises()\n```\n\n- Checking Logs\n\n```python\nlog_formatter = OutputFormatterLog()\noutputs.format_outputs(log_formatter)\n```\n\n### Implementing a custom Data Premise\n\n- Implementing a new DataPremise class\n\n```python\n\nfrom pipeline_penguin.core.data_premise.sql import DataPremiseSQL\nfrom pipeline_penguin.core.premise_output.premise_output import PremiseOutput\nfrom pipeline_penguin.data_node.sql.bigquery import DataNodeBigQuery\n\nclass CheckBanana(DataPremiseSQL):\n    def __init__(\n        self,\n        name: str,\n        data_node: DataNodeBigQuery,\n        column: str\n    ):\n        self.query_template = \"SELECT {column} AS result FROM `{project}.{dataset}.{table}` WHERE LOWER({column}) = 'banana')\"\n        super().__init__(name, data_node, column)\n\n    def query_args(self):\n        \"\"\"Method for returning the arguments to be passed on the query template of this\n        validation.\n\n        Returns:\n            A `dictionary` with the query parameters.\n        \"\"\"\n        return {\n            \"project\": self.data_node.project_id,\n            \"dataset\": self.data_node.dataset_id,\n            \"table\": self.data_node.table_id,\n            \"column\": self.column\n        }\n\n    def validate(self) -\u003e PremiseOutput:\n        \"\"\"Method for executing the validation over the DataNode.\n\n        Returns:\n            PremiseOutput: Object storeing the results for this validation.\n        \"\"\"\n\n        query = self.query_template.format(**self.query_args())\n        connector = self.data_node.get_connector(self.type)\n        data_frame = connector.run(query)\n\n        failed_count = len(data_frame[\"result\"])\n        passed = failed_count == 0\n\n        output = PremiseOutput(\n            self, self.data_node, self.column, passed, failed_count, data_frame\n        )\n        return output\n```\n\n- Testing a DataNode with a custom Data Premise\n\n```python\nfrom pipeline_penguin import PipelinePenguin\nfrom CheckBanana import CheckBanana\n\npp = PipelinePenguin()\n\nbq_connector = ConnectorSQLBigQuery('/config/service_account.json')\npp.connectors.define_default(bq_connector)\n\nnode = pp.nodes.create_node('Node Name', DataNodeBigQuery, project_id='example', dataset_id='example', table_id='example')\n\nnode.insert_premise('Check Contains Banana', CheckBanana, \"Column Name\")\n\nlog_formatter = OutputFormatterLog()\nlog_formatter.export_output(node.premises['Check Is Null'].validate())\n```\n\n## Collaborate\n### Installation\n\n```\npipenv install\n```\n\n### Tests\n\n```\npipenv install --dev\n```\n\nRunning tests\n\n```\npipenv run test\n```\n\n### Style format\n#### Running format\n\n```\npipenv run format\n```\n\n#### Checking format\n\n```\npipenv run format-check\n```\n\n### Developing documentation\n#### Running local build\n```\npipenv run docs\n```\n\n#### Bulding docs\n```\npipenv run pipenv run build-docs\n```\n\n## Support or Contact\nHaving trouble with PP? Check out our [documentation](https://dp6.github.io/pipeline-penguin/pipeline_penguin.html) or contact support and we’ll help you sort it out.\n\nDP6 Koopa-Troopa Team\ne-mail: [koopas@dp6.com.br](koopas@dp6.com.br)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdp6%2Fpipeline-penguin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdp6%2Fpipeline-penguin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdp6%2Fpipeline-penguin/lists"}