{"id":21605965,"url":"https://github.com/archf/ansible-sql-runner","last_synced_at":"2025-04-11T04:04:45.938Z","repository":{"id":89975060,"uuid":"84344375","full_name":"archf/ansible-sql-runner","owner":"archf","description":"Run a sequence of sql queries or scripts on a target db","archived":false,"fork":false,"pushed_at":"2020-10-01T04:45:17.000Z","size":41,"stargazers_count":4,"open_issues_count":0,"forks_count":6,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-04-11T04:04:39.604Z","etag":null,"topics":["ansible","database","impala","phoenix","postgresql","role","sql"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/archf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-03-08T16:50:49.000Z","updated_at":"2021-05-11T07:09:07.000Z","dependencies_parsed_at":"2023-05-30T18:00:19.032Z","dependency_job_id":null,"html_url":"https://github.com/archf/ansible-sql-runner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archf%2Fansible-sql-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archf%2Fansible-sql-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archf%2Fansible-sql-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/archf%2Fansible-sql-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/archf","download_url":"https://codeload.github.com/archf/ansible-sql-runner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248339286,"owners_count":21087215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","database","impala","phoenix","postgresql","role","sql"],"created_at":"2024-11-24T20:18:23.064Z","updated_at":"2025-04-11T04:04:45.917Z","avatar_url":"https://github.com/archf.png","language":"Python","readme":"# ansible-sql-runner\n\nRun a sequence of sql queries or scripts on a target db.\n\n## Requirements\n\n### Ansible version\n\nMinimum required ansible version is 2.2.\n\n### Other considerations\n\nIt depends on the SQL engine you use. Here's the list of supported SQL engines\nand their requirements.\n\n  * postgresql requires [psycopg2](http://initd.org/psycopg/)\n  * impala requires [impyla](https://github.com/cloudera/impyla)\n  * phoenix requires the [phoenixdb ](http://python-phoenixdb.readthedocs.io/en/latest/)\n\n## Description\n\nThere are two ways you can use this role.\n\n**The filepath mode or fileglob mode**\n\nIn this mode, your iterate over all recursively registered SQL scripts under\na given directory.\n\n  * All SQL scripts under directory will be run against the same `sql_engine`\n  * All scripts will be run in alphabetical order\n\n**The declarative mode or advanced mode**\n\nIn this mode, you iterate over queries as defined in a precise input data\nstructure. This enables more capabilities. See below.\n\nOn complex setups, this allows your to run batch of queries while\nalternating sql engine. For instance, you can\n  1. run scripts on postgres\n  2. switch to impala\n  3. than go back to postgres\n  4. than you switch to phoenix\n  5. ...\n\n  all without ever leaving the role.\n\nRole invoked this way can perform one of both following tasks or loop.\n  1. The first loop can lookup data on databases and can append results to\n    * a variable in the `ansible_facts`\n    * to a key you specify inside a special `dict` named `sql_facts`\n  2. The second loop can perform various queries that can be templated\n    * using entries in the `sql_facts` hash populated in the first loop\n    * using other data available in the ansible runtime namespace\n\nDuring a single role invocation, your credentials (user//password) will\nbe fetched from a dictionary that must be structure on a per `sql_engine` basis.\n\n### Playbook examples\n\nGiven those variables commonly defined no mather the way you use this role.\n\n```yaml\n# putting the user/team at the second level level allows to naturally pass\n# the role a data structure with everything needed to access the different\n# engines for a complex maintenance activity when impersonating a specific\n# a specific individual/group. It also allows to have even less levels.\n\n# 3 levels example\napp_sql_conn_creds:\n  # per user/team then engine creds\n  me:\n    postgres:\n      user: myuser\n      password: 'userpassword'\n\n  thisotherteam:\n    postgres:\n      user: teamname\n      password: \"teampassword\"\n\n# 2 levels example\napp_sql_conn_creds:\n    postgres:\n      user: myuser\n      password: 'userpassword'\n    impala:\n      user: teamname\n      password: \"teampassword\"\n\napp_sql_conn_targets:\n  # contains ansible's postgresql_query module's arguments defining the\n  # target\n  postgres:\n    host: \u003chostname or ip\u003e\n    port: \u003coptional value (defaults to 5432 when using postgresql_query module)\u003e\n    ssl_mode: \u003coptional value, defaults to 'prefer' for postgresql modules\u003e\n    ssl_rootcert: \u003coptional value\u003e\n    unix_socket: \u003coptional value\u003e\n    engine: postgres\n\n  impala:\n    host: \u003chostname or ip\u003e\n    user: \u003cimpala user\u003e\n    db: dbname\n    engine: impala\n\n# `sql_db` is a loose variable and is no tied to the engine because\n#   1. A target might have multiple databases.\n#   2. It may not exist and role objective would be to create it. This is\n#       makes it easier to be consistent with other role coded with that\n#       purpose in mind.\n#   3. Has to be passed directly when using the fileglob mode\n#   4. It is defined has 'omit' in the default vars.\nsql_db: dbname\n```\n\nSee also overridables variables defined in the [Default vars](#Default-vars)\n\nExact input data structures depends of the way you use the role. In any cases\neither you let it reach for ansible globally defined variables or you pass it\nexplicitly what it expects. Refer to examples below for example usage.\n\n**Fileglob mode example**\n\nFor that mode since you can interact with only one target//db, things are\nmuch more specific.\n\n```yaml\n  - hosts: localhost\n\n    vars:\n      # change the path to some place of your choosing.\n      sql_history_logfile: \u003cpath to the sql_history_logfile\u003e\n\n    roles:\n      - { role: sql-runner,\n          sql_conn_targets: \"{{app_sql_conn_targets['postgres']}}\",\n          sql_conn_creds: \"{{app_sql_conn_creds['me']['postgres']}}\",\n          sql_db: 'mydb',\n          sql_scripts_dir: \"{{playbook_dir}}/scripts/postgres_query_test/*\"\n          }\n  ```\n\n**Advanced mode example**\n\n  ```yaml\n  - hosts: localhost\n\n    roles:\n      - { role: sql-runner,\n          sql_conn_targets: \"{{app_sql_conn_targets}}\",\n          sql_conn_creds: \"{{app_sql_conn_creds['me']}}\",\n          sql_queries: \"{{app_sql_advanced_tasks}}\"\n          }\n```\n\nWhere `app_sql_advanced_tasks` is defined as below. See inlined explanations.\n\n```yaml\napp_sql_advanced_tasks:\n\n    # Fist loop that optionaly gather facts that can be used to alter later\n    # scripts that contains named placeholder. Those should be mainly 'read'\n    # queries that do not alter the database schema.\n    sql_var_queries:\n\n        # engine will default the the one in sql_conn_target if unspecified.\n        # In that case\n      - engine: postgres\n        db: mydb\n        # List of queries to run against a given db//engine.\n        queries:\n          - name: my_var\n            query: \"SELECT this_var FROM that_table limit %s\"\n            positional_args:\n              - 1\n\n          # run query from a file where path is relative to playbook dir\n          - name: other_var\n            db: \u003coverride 'mydb' value for this query\u003e\n            query: \"{{playbook_dir}}/scripts/postgres/test_postgres_query_return_facts.sql\"\n            named_args:\n              nb_results: 6\n\n      - engine: impala\n        db: myimpaladb\n        queries:\n        - name: impala_var\n          query: \u003csql query where result will be assigned to impala_var key\u003e\n\n      # This will populate a `sql_facts` dict globally available for the rest of the\n      # ansible runtime. You can pass that dict as a named_args to `postgresql_query`\n      # or `impala_query` module.\n\n      # ex:\n      # \"sql_facts\":\n      #   {\n      #     \"my_var\": \"my_value\",\n      #     \"other_var\": \"other_value\"\n      #     \"impala_var\": impala_value\n      #   }\n      # ```\n\n    # adm is a memotechnic shorthand for (admin|administration) queries\n    sql_adm_queries:\n      - engine: postgres\n        db: mydb\n        # List of queries that each contains a list of script to execute against an\n        # sql engine perform. If scripts suffix is .sql.j2, it will be templated\n        # locally first. All scripts are run in the defined order.\n        queries:\n          - query: \"{{playbook_dir}}/scripts/postgres/test_postgres_query_use_facts.sql\"\n            named_args: \"{{sql_facts}}\"\n\n      - engine: postgres\n        # enable autocommit to allow database creation. Alternatively, enable\n        # this globally but keep in mind your queries will be run even\n        # in check_mode.\n        autocommit: True\n        # List of queries to run\n        queries:\n            query: \"create database myotherdb\"\n            query: \"create database mythirddb\"\n\n      - engine: impala\n        db: myimpaladb\n        queries:\n          - query: a impala query\n  ```\n\n## Role Variables\n\n### Variables conditionally loaded\n\nNone.\n\n### Default vars\n\nDefaults from `defaults/main.yml`.\n\n```yaml\n# Disable role debugging by default.\nsql_runner_debug: False\n\n# Empty fact dictionary to store query results for later use You can\n# pre-populate it you want. This will be passed to sql query modules as\n# named_args.\nsql_facts: {}\n\n# This file logs a list of sql queries that were performed. It allows\n# rerunning the playbook without the executing sql scripts that were already\n# done. Each line is either:\n# - An SQL query\n# - the full path of an sql scripts\nsql_history_logfile: \"{{playbook_dir}}/tmp/sql_history.log\"\n\n# Disable the autocommit behavior by default. Set this to true for instance if\n# you happen to run a query that creates a database in postgres else psycopg2\n# execution will fail. It is recommended however that you use the module\n# postgresql_db module.  Running in check_mode with this will indeed execute\n# all queries.\nsql_autocommit: False\n\n# This allows to fallback on a globally defined db name that exists on different\n# sql_engine//targets when running in advanced mode. This is MUTUALLY EXCLUSIVE\n# with sql_autocommit IF you create databases in advanced mode. Your play may\n# fail. You have been warned.\nsql_db: omit\n\n```\n\n## Todo\n\n  * test ansible timeout behavior when logging long query\n  * add jinja templating support for file queries in sql_advanced_mode?\n  * better tests and document check mode usage\n\n## License\n\nMIT.\n\n## Author Information\n\nFelix Archambault.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchf%2Fansible-sql-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farchf%2Fansible-sql-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farchf%2Fansible-sql-runner/lists"}