{"id":46259732,"url":"https://github.com/gopidesupavan/qualink","last_synced_at":"2026-04-01T22:19:06.604Z","repository":{"id":341119779,"uuid":"1168873665","full_name":"gopidesupavan/qualink","owner":"gopidesupavan","description":"Data quality validation, profiling, anomaly detection, and YAML-driven checks for Python on Apache DataFusion.","archived":false,"fork":false,"pushed_at":"2026-03-21T21:38:47.000Z","size":609,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-22T09:54:30.962Z","etag":null,"topics":["analytics","data-profiling","data-quality","data-validation","etl"],"latest_commit_sha":null,"homepage":"https://gopidesupavan.github.io/qualink/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gopidesupavan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-27T22:30:21.000Z","updated_at":"2026-03-21T21:46:39.000Z","dependencies_parsed_at":"2026-03-05T02:00:59.998Z","dependency_job_id":null,"html_url":"https://github.com/gopidesupavan/qualink","commit_stats":null,"previous_names":["gopidesupavan/dq-tool","gopidesupavan/qualink"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/gopidesupavan/qualink","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gopidesupavan%2Fqualink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gopidesupavan%2Fqualink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gopidesupavan%2Fqualink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gopidesupavan%2Fqualink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gopidesupavan","download_url":"https://codeload.github.com/gopidesupavan/qualink/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gopidesupavan%2Fqualink/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31292639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","data-profiling","data-quality","data-validation","etl"],"created_at":"2026-03-04T01:21:11.353Z","updated_at":"2026-04-01T22:19:06.594Z","avatar_url":"https://github.com/gopidesupavan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# qualink\n\n[Official Website](https://gopidesupavan.github.io/qualink/)\n\nBlazing fast data quality framework for Python, built on Apache DataFusion.\n\n## Features\n\n- **High Performance**: Leverages Apache DataFusion for fast data processing and validation.\n- **Flexible Constraints**: Supports various data quality constraints including completeness, uniqueness, and custom assertions.\n- **YAML Configuration**: Define validation suites declaratively using YAML files.\n- **CLI – `qualinkctl`**: Run YAML-driven validations from the terminal — no Python script required.\n- **Cloud Object Stores**: Read data directly from Amazon S3 (and S3-compatible services).\n- **Multiple Output Formats**: Results can be formatted as human-readable text, JSON, or Markdown.\n- **Async Support**: Built with asyncio for non-blocking operations.\n- **Analyzers**: Compute reusable dataset and column metrics independent of pass/fail checks.\n- **Metrics Repository**: Persist analyzer outputs over time using tagged result keys.\n- **Anomaly Detection**: Detect unexpected metric shifts from historical baselines.\n- **Intelligent Rule Suggestions**: Generate candidate validation rules from column profiles.\n- **Easy Integration**: Simple API for defining and running validation suites.\n\n## Installation\n\nInstall qualink using uv:\n\n```bash\nuv add qualink\n```\n\nOr using pip:\n\n```bash\npip install qualink\n```\n\n## Quick Start\n\nHere's a basic example of using qualink to validate a CSV file:\n\n```python\nimport asyncio\nfrom datafusion import SessionContext\nfrom qualink.checks import Check, Level\nfrom qualink.constraints import Assertion\nfrom qualink.core import ValidationSuite\nfrom qualink.formatters import MarkdownFormatter\n\n\nasync def main() -\u003e None:\n    ctx = SessionContext()\n    ctx.register_csv(\"users\", \"examples/users.csv\")\n\n    result = await (\n        ValidationSuite()\n        .on_data(ctx, \"users\")\n        .with_name(\"User Data Quality\")\n        .add_check(Check.builder(\"Critical Checks\").with_level(Level.ERROR).is_complete(\"user_id\").build())\n        .add_check(\n            Check.builder(\"Data Quality\")\n            .with_level(Level.WARNING)\n            .has_completeness(\"name\", Assertion.greater_than_or_equal(0.95))\n            .build()\n        )\n        .run()\n    )\n\n    print(MarkdownFormatter().format(result))\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## YAML Configuration\n\nYou can also define validation suites using YAML files for a declarative approach:\n\n```yaml\nsuite:\n  name: \"User Data Quality\"\n\ndata_sources:\n  - name: users_source\n    format: csv\n    path: \"examples/users.csv\"\n    table_name: users\n\nchecks:\n  - name: \"Critical Checks\"\n    level: error\n    rules:\n      - is_complete: user_id\n      - is_unique: email\n      - has_size:\n          gt: 0\n  - name: \"Data Quality\"\n    level: warning\n    rules:\n      - has_completeness:\n          column: name\n          gte: 0.95\n```\n\nRun the YAML configuration:\n\n```python\nimport asyncio\nfrom qualink.config import run_yaml\nfrom qualink.formatters import HumanFormatter\n\n\nasync def main() -\u003e None:\n    result = await run_yaml(\"path/to/your/config.yaml\")\n    print(HumanFormatter().format(result))\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n`run_yaml()` also accepts filesystem URIs such as `s3://my-bucket/checks.yaml` or\n`file:///absolute/path/to/checks.yaml`, in addition to local file paths and inline YAML strings.\n\n## CLI – `qualinkctl`\n\nThe simplest way to run a YAML validation is with `qualinkctl`:\n\n```bash\n# Human-readable output (default)\nuv run qualinkctl checks.yaml\n\n# JSON output\nuv run qualinkctl checks.yaml -f json\n\n# Markdown report saved to file\nuv run qualinkctl checks.yaml -f markdown -o report.md\n\n# JSON report written to object storage\nuv run qualinkctl checks.yaml -f json -o s3://my-bucket/qualink/results.json\n\n# Show all constraints (including passed) with debug logging\nuv run qualinkctl checks.yaml --show-passed -v\n```\n\n`qualinkctl` exits with code `0` on success and `1` on failure, making it easy to use in CI/CD pipelines:\n\n```bash\nuv run qualinkctl checks.yaml -f json -o results.json || echo \"Validation failed!\"\n```\n\nRun `uv run qualinkctl --help` for a full list of options.\n\n## Advanced Features\n\nRunnable end-to-end examples are available in:\n\n- `examples/adbc_sqlite_example.py`\n- `examples/analyzers_example.py`\n- `examples/metrics_repository_example.py`\n- `examples/anomaly_detection_example.py`\n- `examples/intelligent_rule_suggestions_example.py`\n- `examples/output_results_example.py`\n- `examples/file_uri_validation.py`\n\n### ADBC Datasources\n\nqualink can also register database-backed sources through ADBC and materialize them into DataFusion tables before running checks.\n\nSQLite example shape:\n\n```yaml\nconnections:\n  sqlite_local:\n    uri: sqlite:///tmp/users.db\n\ndata_sources:\n  - name: users_source\n    connection: sqlite_local\n    table: users\n    table_name: users\n```\n\nTo run the SQLite example after installing the optional ADBC packages:\n\n```bash\nuv sync --group adbc\nuv run python examples/adbc_sqlite_example.py\n```\n\n### Secret-backed Connections\n\nSensitive connection values can be resolved inline from environment variables, AWS Systems Manager Parameter Store, AWS Secrets Manager, or GCP Secret Manager.\n\nExample:\n\n```yaml\nconnections:\n  sqlite_local:\n    uri:\n      from: env\n      key: QUALINK_SQLITE_URI\n\ndata_sources:\n  - name: users_source\n    connection: sqlite_local\n    table: users\n    table_name: users\n```\n\nAWS SSM example:\n\n```yaml\nconnections:\n  postgres_prod:\n    uri:\n      from: aws_ssm\n      key: /qualink/prod/postgres/uri\n      region: us-east-1\n```\n\nAWS Secrets Manager JSON field extraction:\n\n```yaml\nconnections:\n  snowflake_prod:\n    uri:\n      from: aws_secretsmanager\n      key: qualink/prod/snowflake\n      field: uri\n      region: eu-west-1\n```\n\nThe checked-in reference config is [examples/secret_backed_connections.yaml](/Users/gopidesupavan/qualink/examples/secret_backed_connections.yaml).\n\n### Result Outputs to Filesystems\n\nValidation results can be written to local paths or filesystem URIs backed by PyArrow filesystems such as S3, GCS, and Azure Blob/Data Lake.\n\nCLI example:\n\n```bash\nuv run qualinkctl checks.yaml -f json -o s3://my-bucket/qualink/results.json\nuv run qualinkctl checks.yaml -f markdown -o gs://my-bucket/qualink/report.md\n```\n\nYAML-driven outputs:\n\n```yaml\noutputs:\n  - path: reports/results.json\n    format: json\n    show_passed: true\n  - uri: s3://my-bucket/qualink/results.md\n    format: markdown\n```\n\nPython API example:\n\n```python\nfrom qualink.config import run_yaml\nfrom qualink.config.parser import load_yaml\nfrom qualink.output import OutputService, normalize_output_specs\n\nconfig = load_yaml(\"examples/output_results.yaml\")\nresult = await run_yaml(\"examples/output_results.yaml\")\nOutputService().emit_many(result, normalize_output_specs(config))\n```\n\n### S3 Object Store Sources\n\nqualink can read data directly from Amazon S3 using DataFusion's built-in `AmazonS3`:\n\n```yaml\nsuite:\n  name: \"Cloud Data Quality\"\n\ndata_sources:\n  - name: users_source\n    format: parquet\n    path: s3://my-data-lake/data/users.parquet\n    table_name: users\n\nchecks:\n  - name: \"Completeness\"\n    level: error\n    rules:\n      - is_complete: user_id\n      - is_unique: email\n```\n\nUse the standard AWS credential chain. On Glue, ECS, EKS, or EC2 with an attached role, explicit keys are usually not required.\n\n## Constraints\n\nqualink supports the following constraint types:\n\n- **Completeness**: Ensures a column has no null values or meets a minimum completeness ratio.\n- **Uniqueness**: Checks for duplicate values in a column.\n- **Assertion**: Custom assertions using SQL expressions.\n\n## Formatters\n\nResults can be formatted using:\n\n- `HumanFormatter`: Human-readable text output.\n- `JsonFormatter`: JSON format for programmatic processing.\n- `MarkdownFormatter`: Markdown tables for documentation.\n\n## Benchmarks\n\nqualink ships with a real-world benchmark suite that validates **~42 million NYC Yellow Taxi trip records** (654 MB of Parquet data) through 12 check groups and 92 constraints — in **under 1.5 seconds**.\n\n```\n========================================================================\n  qualink Benchmark — NYC Taxi Trips\n========================================================================\n  Parquet files : 3\n  Total size    : 654.3 MB\n  Data dir      : benchmarks/data\n  YAML config   : benchmarks/nyc_taxi_validation.yaml\n\n    • data-200901.parquet  (211.9 MB)\n    • data-201206.parquet  (231.1 MB)\n    • data-201501.parquet  (211.3 MB)\n========================================================================\n\n⏱  Running benchmark with 'human' formatter …\n\nVerification PASSED: NYC Taxi Trips – qualink Benchmark Suite\n\nChecks          12\nConstraints     92\nPassed          91\nFailed          1\nSkipped         0\nPass rate       98.9%\nExecution time  1440 ms\n\nStatus    Check       Message\n--------  ----------  ---------------------------------------------\n[FAIL]    Uniqueness  Uniqueness of (id) is 0.0000, expected \u003e= 1.0\n\n========================================================================\n  Status         : ✅ PASSED\n  Total records  : 41.94M\n  Wall-clock     : 1.455s\n  Checks         : 12\n  Constraints    : 92\n  Passed         : 91\n  Failed         : 1\n  Pass rate      : 98.9%\n  Engine time    : 0.02m\n========================================================================\n```\n\n### Run it yourself\n\n```bash\n# 1. Download data (parquet files from public S3)\n./benchmarks/download_data.sh 3\n\n# 2. Run the benchmark\nuv run python benchmarks/run_benchmark.py\n\n# Other output formats\nuv run python benchmarks/run_benchmark.py --format markdown\nuv run python benchmarks/run_benchmark.py --format json\n```\n\nSee [`benchmarks/README.md`](benchmarks/README.md) for full dataset details and configuration.\n\n## Development\n\nTo set up the development environment:\n\n```bash\ngit clone https://github.com/gopidesupavan/qualink.git\ncd qualink\nuv sync\n```\n\nRun tests:\n\n```bash\nuv run pytest\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- [Apache DataFusion](https://datafusion.apache.org/) for the query engine\n- [AWS Deequ](https://github.com/awslabs/deequ/) for the inspiration\n- [Term Guard](https://github.com/withterm/term-guard)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgopidesupavan%2Fqualink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgopidesupavan%2Fqualink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgopidesupavan%2Fqualink/lists"}