{"id":23511193,"url":"https://github.com/percolate/sdic","last_synced_at":"2025-04-18T15:04:22.093Z","repository":{"id":45912880,"uuid":"60359314","full_name":"percolate/sdic","owner":"percolate","description":"SQL Data Integrity Checker","archived":false,"fork":false,"pushed_at":"2023-03-16T19:25:58.000Z","size":40,"stargazers_count":9,"open_issues_count":2,"forks_count":0,"subscribers_count":41,"default_branch":"master","last_synced_at":"2024-04-26T08:21:41.858Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/percolate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-03T15:44:32.000Z","updated_at":"2023-10-08T05:13:16.000Z","dependencies_parsed_at":"2024-12-25T12:13:45.222Z","dependency_job_id":"4f8cdd86-2db2-4d1e-9364-64e63bbe11f1","html_url":"https://github.com/percolate/sdic","commit_stats":{"total_commits":35,"total_committers":3,"mean_commits":"11.666666666666666","dds":0.4571428571428572,"last_synced_commit":"02f4ff219272f875319e0561fe5d13f2195d984a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percolate%2Fsdic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percolate%2Fsdic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percolate%2Fsdic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percolate%2Fsdic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/percolate","download_url":"https://codeload.github.com/percolate/sdic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249509720,"owners_count":21283626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-25T12:13:41.350Z","updated_at":"2025-04-18T15:04:22.061Z","avatar_url":"https://github.com/percolate.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sdic\n\nA.K.A. __SQL Data Integrity Checker__\n\n[![CircleCI](https://circleci.com/gh/percolate/sdic.svg?style=svg)](https://circleci.com/gh/percolate/sdic)\n\n## One line purpose\n\n`sdic` executes all the SQL queries found in a folder and displays its output.\n\n## More detailed purpose: Soft Constraints\n\n### Usual Constraints\n\nIn any RDBMS, you can set constraints to prevent the application to save the\ndata in a way that's not consistent. E.g. if you want all your users to have an\nemail, you can set the email column to `NOT NULL`.\n\nThis works for simple constraints:\n\n1. It's easy to implement\n1. It's cheap for the database to check on every change\n\nBut for more complex constraints that you'd like to set, it'd be either very\nexpensinve to check on every write, or even impossible to write as a\nconstraint.\n\n### Soft Constraints\n\nWith `sdic`, you can write you complex constraints as simple queries, and have\nthe database run them asynchronously at the occurrence you want.\n\nWe call them \"soft constraints\".\n\n#### Example\n\nLet's say that you have a `users` table, defined like this:\n\n- `id` Primary Key `NOT NULL`\n- `firstname` `NULL`\n- `lastname` `NULL`\n- `email` `NOT NULL`\n\nNow, let's suppose your application allows users to register just with their\n`email` but can fill in their `firstname` and `lastname` later on, but we don't\nwant our users to have only a firstname or a lastname.\n\nSimply put, our constraint is: Make sure every users has either a `firstname`\nand a `lastname` set, or both set to NULL.\n\nWith `sdic`, you can add this `enforce_fullname.sql` file and let `sdic` check\nthat every user comply nightly.\n\n```sql\n-- Make sure every user with a name has both a firstname and a lastname\nSELECT id, firstname, lastname\nFROM users\nWHERE\n    (firstname IS NULL AND lastname IS NOT NULL) OR\n    (firstname IS NOT NULL AND lastname IS NULL)\nLIMIT 10\n;\n-- Could also be written as firstname IS NULL \u003c\u003e lastname IS NULL but this is\n-- for people to understand the use case.\n```\n\nPut this file in `your-environment/your-server/enforce_fullname.sql`.\n\nEdit the `your-environment/servers.ini` file to tell sdic how to connect to\nyour server.\n\nNow run `sdic your-environment` and it will output any user that do no comply\nwith your soft constraint.\n\nYou can have as many soft constraints on as many servers and as many\nenvironments as you need.\n\n### Constraints that can be temporary violated\n\nAnother use case is that there are times that certain business rules can be\nviolated for short periods. For example, you may want every department to have\na head, but also to allow the creation of a new department without or NULLing\nout that field when somebody quits.\n\nIn those sorts of cases, a reporting tool like `sdic` is likely just what the\ndoctor ordered.\n\n## Install as a cron\n\nIf you want to get an email every night to give you a list of all the soft\nconstraints that have been broken during the last day, just add it to you\ncrontab. We like to have it run daily, so we can fix any bug generating bad\ndata before it becomes a real problem.\n\nExample crontab:\n\n```\nMAILTO=\"dba@acme.com\"\n@daily sdic live\n```\n\n`dba@acme.com` is the email that will get the soft constraints broken every\nday. Make sure your local MTA is well configured on your system. You can test\nit by doing `date | mail -s test dba@acme.com`.\n\n## Databases supported\n\nAny database supported by [SQLAlchemy](http://www.sqlalchemy.org/) should be\nsupported, including [PostgreSQL](https://www.postgresql.org/) and\n[MySQL](https://www.mysql.com/).\n\n## Install\n\n`pip install sdic`\n\n## Configuration\n\nAn example configuration is given in the `example-environment` folder.\n\nThe script reads from a designated folder, whose path you pass as an argument.\nThis folder should consist of the following:\n\n1. A `servers.ini` file, which contains the Database URLs (see the\n    `example-environment` folder)\n1. A sub-folder, which contains the actual queries in a `.sql` file format\n\n## Usage\n\nA `directory` argument is mandatory:\n\n`sdic path/to/your/folder`\n\nIf you have e.g more than one server in a folder, but you want to\nonly run one of them, an optional `server` argument can be passed as well:\n\n`sdic path/to/your/folder server1`\n\nIf a query produces an output, it will look something like this:\n\n```\n-----===== /!\\ INCOMING BAD DATA /!\\ =====-----\n\nServer: big-database\nFile: test_query.sql\n\nSQL Query:\n-- This is a query that returns current time.\nSelect now();\n\n+---------------------+\n|        now()        |\n+---------------------+\n| 2016-06-03 19:27:14 |\n+---------------------+\n```\n\n## FAQ\n\n### Pretty sure that's what `CHECK` constraints are for\n\nRight but there are 2 main differences:\n\n- `CHECK` constraints are checked on every write, for expensive checks (e.g.\n    one that require scanning a whole big table) it's not an option. The point\n    here is to choose how often you run the checks, and to not have to run them\n    on writes. For us it's once, nightly: That's when our servers are doing\n    less.\n- MySQL simply ignores the `CHECK` statement.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpercolate%2Fsdic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpercolate%2Fsdic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpercolate%2Fsdic/lists"}