{"id":20843862,"url":"https://github.com/tkc/sql-dog","last_synced_at":"2026-03-03T21:02:53.419Z","repository":{"id":55382495,"uuid":"300876818","full_name":"tkc/sql-dog","owner":"tkc","description":"Analyzes SQL query logs and triggers a warning if a specified condition is missing from the query.","archived":false,"fork":false,"pushed_at":"2025-03-23T11:29:56.000Z","size":89,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-09T01:48:25.180Z","etag":null,"topics":["analysis","analyzer","database","go","sql","sql-parser"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tkc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-10-03T12:36:55.000Z","updated_at":"2025-03-23T11:29:57.000Z","dependencies_parsed_at":"2025-03-23T11:35:51.894Z","dependency_job_id":null,"html_url":"https://github.com/tkc/sql-dog","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/tkc/sql-dog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkc%2Fsql-dog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkc%2Fsql-dog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkc%2Fsql-dog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkc%2Fsql-dog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tkc","download_url":"https://codeload.github.com/tkc/sql-dog/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tkc%2Fsql-dog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30060677,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-03T18:21:05.932Z","status":"ssl_error","status_checked_at":"2026-03-03T18:20:59.341Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis","analyzer","database","go","sql","sql-parser"],"created_at":"2024-11-18T02:07:37.565Z","updated_at":"2026-03-03T21:02:53.401Z","avatar_url":"https://github.com/tkc.png","language":"Go","readme":"![Build Status](https://github.com/tkc/sql-dog/workflows/sql-dog/badge.svg)\n![Reviewdog](https://github.com/tkc/sql-dog/workflows/reviewdog/badge.svg)\n![CodeQL](https://github.com/tkc/sql-dog/workflows/CodeQL/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/tkc/sql-dog)](https://goreportcard.com/report/github.com/tkc/sql-dog)\n\n# SQL Dog\n\nSQL Dog is a tool that analyzes MySQL query logs and triggers warnings when specified conditions (such as WHERE clauses or NOT NULL constraints) are missing from queries. It's designed to enhance database security and performance.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Design Philosophy and Architecture](#design-philosophy-and-architecture)\n- [Installation](#installation)\n- [Configuration](#configuration)\n  - [MySQL Configuration](#mysql-configuration)\n  - [Database Settings](#database-settings)\n  - [Validation Rules](#validation-rules)\n- [Usage](#usage)\n- [Features](#features)\n- [Technical Design](#technical-design)\n- [Developer Information](#developer-information)\n- [Troubleshooting](#troubleshooting)\n- [Roadmap](#roadmap)\n\n## Overview\n\nSQL Dog helps with:\n\n- **Security**: Detecting queries missing required WHERE conditions\n- **Performance**: Identifying unoptimized queries that might cause full table scans\n- **Quality Control**: Ensuring proper use of NOT NULL constraints\n- **Auditing**: Analyzing database access patterns\n\nIt's a powerful tool for preventing bugs and issues in high-load production environments before they occur.\n\n## Design Philosophy and Architecture\n\nSQL Dog isn't just a query log analyzer—it acts as a watchdog for your database access, protecting both security and performance aspects. Below is a detailed explanation of the project's design philosophy.\n\n### Core Problem and Solution Approach\n\n**Problems Being Addressed**:\nIn many development environments, database access patterns degrade over time, leading to issues such as:\n\n1. Queries missing essential WHERE clauses, causing full table scans\n2. Queries retrieving deleted data by forgetting to check logical deletion flags (e.g., deleted_at)\n3. Queries lacking proper permission conditions (tenant ID, user ID), creating security vulnerabilities\n\n### Main Workflow\n\nThe overall program execution flow:\n\n1. **Load Configuration**: Read database connection information and validation rules from YAML files\n2. **Retrieve Query Logs**: Get executed queries from MySQL's general_log table\n3. **Parse Queries**: Convert each query into an abstract syntax tree (AST) and extract structured information\n4. **Validate Rules**: Compare extracted information with validation rules and detect violations\n5. **Generate Reports**: Format and output validation results\n\n```\n+------------------+     +----------------+     +---------------+\n| Load Configuration | --\u003e | Retrieve Logs  | --\u003e | Parse Queries |\n+------------------+     +----------------+     +---------------+\n                                                       |\n                                                       v\n                         +---------------+     +---------------+\n                         | Output Reports | \u003c-- | Validate Rules |\n                         +---------------+     +---------------+\n```\n\n### Technology Choices\n\n1. **PingCAP's SQL Parser**: Selected as a parser capable of accurately analyzing complex MySQL syntax\n2. **GORM**: Chosen as a simple yet powerful ORM to simplify database access\n3. **go-yaml**: Selected for parsing configuration files, considering readability and extensibility\n4. **testify**: Adopted to write more concise and expressive test code\n\n### Practical Usage Example\n\nFor example, in a multi-tenant application, you can set security rules like:\n\n```yaml\ntables:\n  - name: users\n    mustSelectColumns:\n      - tenant_id # Tenant ID condition is required\n    stmtTypePatterns:\n      - select\n      - update\n      - delete\n```\n\nWith this rule, the following query would trigger a warning:\n\n```sql\n-- Warning: no filtering by tenant_id\nSELECT * FROM users WHERE name = 'John';\n```\n\nWhile this query would be allowed:\n\n```sql\n-- OK: filtered by tenant_id\nSELECT * FROM users WHERE tenant_id = 123 AND name = 'John';\n```\n\n### Summary\n\nSQL Dog is designed with the following philosophy:\n\n1. **Defensive Programming**: Detect issues early to prevent production failures\n2. **Configuration-Driven Approach**: Add and modify rules flexibly without changing code\n3. **Domain-Driven Design**: View the technical domain of SQL queries from the business domain perspective of security and performance\n\nBy using this tool, you can maintain database access quality and prevent security and performance issues.\n\n## Installation\n\n### Prerequisites\n\n- Go 1.18 or higher\n- MySQL 5.7 or higher\n\n### Installation Steps\n\n```bash\n# Clone the repository\ngit clone https://github.com/tkc/sql-dog.git\ncd sql-dog\n\n# Install dependencies\ngo mod download\n```\n\n## Configuration\n\n### MySQL Configuration\n\nEnable query logging in MySQL and configure it to record to the general_log table:\n\n```sql\nSET GLOBAL general_log = 'ON';\nSET GLOBAL log_output = 'TABLE';\n\n# Optional: Configure slow query log\nSET GLOBAL slow_query_log = 'ON';\nSET GLOBAL long_query_time = 0;\n```\n\n### Database Settings\n\n1. Copy the sample configuration file:\n\n   ```bash\n   cp config.sample.yaml config.yaml\n   ```\n\n2. Edit `config.yaml` to configure your connection settings:\n   ```yaml\n   username: \"root\"\n   password: \"your_password\"\n   host: \"localhost\"\n   port: 3306\n   rootDatabase: \"mysql\"\n   serviceDatabase: \"your_database_name\"\n   ```\n\n### Validation Rules\n\n1. Copy the sample validation rules file:\n\n   ```bash\n   cp linter.sample.yaml linter.yaml\n   ```\n\n2. Edit `linter.yaml` to set up validation rules:\n\n   ```yaml\n   # Queries to exclude from validation\n   ignores:\n     - DELETE FROM temp_table\n\n   # Tables and rules to validate\n   tables:\n     - name: users\n       # Required column conditions for SELECT queries\n       mustSelectColumns:\n         - deleted_at\n         - tenant_id\n       # SQL statement types to target\n       stmtTypePatterns:\n         - select\n         - update\n         - delete\n       # Columns requiring NOT NULL constraints\n       notNullColumns:\n         - deleted_at\n   ```\n\n## Usage\n\n### Running Query Analysis\n\n```bash\n# Run query analysis and display report\ngo run ./cmd/lint/main.go\n\n# Or use the compiled binary\n./sql-dog-lint\n```\n\n### Clearing Log Table\n\n```bash\n# Clear records from the general_log table\ngo run ./cmd/clean/main.go\n\n# Or use the compiled binary\n./sql-dog-clean\n```\n\n## Features\n\n- **WHERE Clause Checking**: Verifies that queries to specific tables include the required WHERE conditions\n- **NOT NULL Constraint Checking**: Confirms that target tables have necessary NOT NULL constraints set\n- **Multi-Table Support**: Configure different validation rules for multiple tables\n- **Exclusion Rules**: Exempt specific queries from validation\n\n## Developer Information\n\n### Building\n\n```bash\n# Local build (outputs to ./bin/)\nmake build\n\n# Install to $GOPATH/bin\nmake install\n```\n\n### Running Tests\n\n```bash\n# Run all tests\nmake test\n\n# Run tests with race detection\nmake test-race\n\n# Run tests with coverage reporting\nmake test-cover\n```\n\n### Formatting Code\n\n```bash\n# Format all Go code\nmake fmt\n```\n\n### Running Linter\n\n```bash\n# Run Go linter\nmake lint\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **MySQL Connection Error**\n\n   - Verify connection information in `config.yaml`\n   - Check if MySQL server is running\n\n2. **Empty general_log Table**\n\n   - Verify MySQL logging is correctly enabled\n   - Check if target queries have been executed\n\n3. **Validation Rules Not Working**\n   - Verify `linter.yaml` syntax is correct\n   - Check that table and column names are accurately entered\n\n## Roadmap\n\n- [ ] Support for more query log formats (HTTP requests, text logs, etc.)\n- [ ] Dashboard UI\n- [ ] Real-time monitoring\n- [ ] Automatic correction suggestions\n\n## License\n\nThis project is released under the MIT License.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftkc%2Fsql-dog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftkc%2Fsql-dog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftkc%2Fsql-dog/lists"}