{"id":16658322,"url":"https://github.com/vickyjkwan/sqlanalyzer","last_synced_at":"2025-04-09T18:32:36.554Z","repository":{"id":57470525,"uuid":"260313875","full_name":"vickyjkwan/sqlanalyzer","owner":"vickyjkwan","description":"A SQL parser and analyzer for sql flavors including MySQL, PostgreSQL, BigQuery Standard SQL, Presto SQL and Hive SQL. ","archived":false,"fork":false,"pushed_at":"2023-05-29T19:45:07.000Z","size":485,"stargazers_count":10,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-23T20:36:42.353Z","etag":null,"topics":["athena","bigquery","hiveql","metastore","presto","sqlparser","standardsql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vickyjkwan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-30T20:39:52.000Z","updated_at":"2025-03-06T21:47:06.000Z","dependencies_parsed_at":"2025-02-15T12:41:41.810Z","dependency_job_id":null,"html_url":"https://github.com/vickyjkwan/sqlanalyzer","commit_stats":null,"previous_names":["mathilda0902/sqlanalyzer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vickyjkwan%2Fsqlanalyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vickyjkwan%2Fsqlanalyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vickyjkwan%2Fsqlanalyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vickyjkwan%2Fsqlanalyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vickyjkwan","download_url":"https://codeload.github.com/vickyjkwan/sqlanalyzer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248087651,"owners_count":21045558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["athena","bigquery","hiveql","metastore","presto","sqlparser","standardsql"],"created_at":"2024-10-12T10:05:14.810Z","updated_at":"2025-04-09T18:32:36.535Z","avatar_url":"https://github.com/vickyjkwan.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is a Python package that parses a given sql query, matches the column and tables within your given metastore, and analyzes the query to generate a list of referenced columns within the metastore.\n\n\n## Quick Start\n\n`$ pip install sqlanalyzer`\n\n\n## Example Usage\n\n**1. Format a query to follow the [ANSI standards](https://blog.ansi.org/2018/10/sql-standard-iso-iec-9075-2016-ansi-x3-135/) for SQL:**\n\n\n```\n\u003e\u003e\u003e from sqlanalyzer import column_parser\n\u003e\u003e\u003e query = \"\"\"SELECT api.name, acct.customer_tier_c, acct.name FROM api_requests_by_account api\n... LEFT JOIN accounts \n... acct ON api.user_id = acct.customer_api_id\n... \"\"\"\n\u003e\u003e\u003e formatter = column_parser.Parser(query)\n\u003e\u003e\u003e formatted = formatter.format_query(query)\n\u003e\u003e\u003e print(formatted)\nSELECT api.name,\n       acct.customer_tier_c,\n       acct.name\nFROM api_requests_by_account api\nLEFT JOIN accounts acct ON api.user_id = acct.customer_api_id\n```\n\n**2. Separate CTE's and extract alias names and queries:**\n\n```\n\u003e\u003e\u003e query = \"\"\"WITH a AS\n...   (SELECT DISTINCT anonymous_id,\n...                    user_id\n...    FROM customer_data.segment_identifies\n...    WHERE dt \u003e= '2018-07-01'),\n...      b AS\n...   (SELECT id,\n...           email,\n...           created\n...    FROM customer_data.accounts)\n... SELECT a.*,\n...        b.*\n... FROM a\n... LEFT JOIN b ON a.user_id = b.id\n... WHERE context_campaign_name IS NOT NULL\n... \"\"\"\n\u003e\u003e\u003e formatter = column_parser.Parser(query)\n\u003e\u003e\u003e cte_query = formatter.parse_cte(query)\n\u003e\u003e\u003e cte_query\n{'a': \"SELECT DISTINCT anonymous_id,\\n                   user_id\\n   FROM customer_data.segment_identifies\\n   WHERE dt \u003e= '2018-07-01'\",\n'b': 'SELECT id,\\n          email,\\n          created\\n   FROM customer_data.accounts', \n'main_query': 'SELECT a.*,\\n       b.*\\nFROM a\\nLEFT JOIN b ON a.user_id = b.id\\nWHERE context_campaign_name IS NOT NULL\\n'}\n\u003e\u003e\u003e cte_query.keys()\ndict_keys(['a', 'b', 'main_query'])\n```\n\n**3. Match table aliases with the actual database name:**\n\n```\n\u003e\u003e\u003e query = \"\"\"SELECT *\n... FROM api_requests.requests_by_account m\n... INNER JOIN mapbox_customer_data.styles s ON m.metadata_version = s.id\n... LEFT JOIN sfdc.users u ON m.csm = u.id\n... \"\"\"\n\u003e\u003e\u003e formatter = column_parser.Parser(query)\n\u003e\u003e\u003e formatted = formatter.format_query(query)\n\u003e\u003e\u003e table_alias_mapping = formatter.get_table_names(formatted.split('\\n'))\n\u003e\u003e\u003e table_alias_mapping\n{'m': 'api_requests.requests_by_account', \n's': 'mapbox_customer_data.styles', \n'u': 'sfdc.users'}\n```\n\n\n**4. Analyze and parse complex query with subqueries, Common Table Expressions and a mix of the two types.**\n\n*a)* Parse multiple and deeply (3+ levels) nested subqueries:\n\n```\n\u003e\u003e\u003e from sqlanalyzer import query_analyzer\n\u003e\u003e\u003e query = \"\"\"SELECT *\n... FROM\n...   (SELECT a.*,\n...           b.*\n...    FROM\n...      (SELECT DISTINCT anonymous_id,\n...                       user_id\n...       FROM customer_data.segment_identifies\n...       WHERE dt \u003e= '2018-07-01') a\n...    LEFT JOIN\n...      (SELECT id,\n...              email,\n...              created\n...       FROM customer_data.accounts) b ON a.user_id = b.id\n...    WHERE context_campaign_name IS NOT NULL )\n... \"\"\"\n\u003e\u003e\u003e analyzer = query_analyzer.Analyzer(query)\n\u003e\u003e\u003e analyzer.parse_query(query)\n[{'level_1_main': 'SELECT * FROM no alias '}, \n{'level_2_main': 'SELECT a.*,        b.* WHERE context_campaign_name IS NOT NULL FROM a LEFT JOIN b ON a.user_id = b.id '}, \n{'a': \"SELECT DISTINCT anonymous_id, user_id FROM customer_data.segment_identifies WHERE dt \u003e= '2018-07-01'\"}, \n{'b': 'SELECT id, email, created FROM customer_data.accounts'}]\n```\n\n*b)* Parse Common Table Expressions (CTE's):\n\n```\n\u003e\u003e\u003e query = \"\"\"WITH a AS\n...   (SELECT DISTINCT anonymous_id,\n...                    user_id\n...    FROM customer_data.segment_identifies\n...    WHERE dt \u003e= '2018-07-01'),\n...      b AS\n...   (SELECT id,\n...           email,\n...           created\n...    FROM customer_data.accounts)\n... SELECT a.*,\n...        b.*\n... FROM a\n... LEFT JOIN b ON a.user_id = b.id\n... WHERE context_campaign_name IS NOT NULL\n... \"\"\"\n\u003e\u003e\u003e analyzer = query_analyzer.Analyzer(query)\n\u003e\u003e\u003e analyzer.parse_query(query)\n[{'a': \"SELECT DISTINCT anonymous_id,\\n                   user_id\\n   FROM customer_data.segment_identifies\\n   WHERE dt \u003e= '2018-07-01'\"}, \n{'b': 'SELECT id,\\n          email,\\n          created\\n   FROM customer_data.accounts'}, \n{'main_query': 'SELECT a.*,\\n       b.*\\nFROM a\\nLEFT JOIN b ON a.user_id = b.id\\nWHERE context_campaign_name IS NOT NULL'}]\n```\n\n*c)* Parse mixed type of nested queries and CTE's:\n\n```\n\u003e\u003e\u003e query = \"\"\"SELECT email,\n...        COUNT(DISTINCT context_campaign_name)\n... FROM\n...   (WITH a AS\n...      (SELECT DISTINCT anonymous_id,\n...                       user_id\n...       FROM customer_data.segment_identifies\n...       WHERE dt \u003e= '2018-07-01'),\n...         b AS\n...      (SELECT id,\n...              email,\n...              created\n...       FROM customer_data.accounts) SELECT a.*,\n...                                           b.*\n...    FROM a\n...    LEFT JOIN b ON a.user_id = b.id\n...    WHERE context_campaign_name IS NOT NULL )\n... WHERE user_id IN ('123',\n...                   '234',\n...                   '345')\n... GROUP BY 1\n... ORDER BY 2 DESC\n... LIMIT 200\n... \"\"\"\n\u003e\u003e\u003e analyzer = query_analyzer.Analyzer(query)\n\u003e\u003e\u003e analyzer.parse_query(query)\n[{'level_1_main': \"SELECT email,        COUNT(DISTINCT context_campaign_name) WHERE user_id IN ('123',                   '234',                   '345') FROM no alias \"}, \n{'no alias': [{'a': \"SELECT DISTINCT anonymous_id,\\n                   user_id\\n   FROM customer_data.segment_identifies\\n   WHERE dt \u003e= '2018-07-01'\"}, \n{'b': 'SELECT id,\\n          email,\\n          created\\n   FROM customer_data.accounts'}, \n{'main_query': 'SELECT a.*,\\n       b.*\\nFROM a\\nLEFT JOIN b ON a.user_id = b.id\\nWHERE context_campaign_name IS NOT NULL'}]}]\n```\n\n\n\nNotes: \n\n[Upload instructions](https://packaging.python.org/tutorials/packaging-projects/)\n`python3 -m pip install --user --upgrade setuptools wheel twine`\n`python3 setup.py sdist bdist_wheel`\n`twine check dist/*`\n`twine upload dist/*`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvickyjkwan%2Fsqlanalyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvickyjkwan%2Fsqlanalyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvickyjkwan%2Fsqlanalyzer/lists"}