{"id":21846951,"url":"https://github.com/duyet/athena-rs","last_synced_at":"2025-07-07T04:07:15.012Z","repository":{"id":56731646,"uuid":"521912808","full_name":"duyet/athena-rs","owner":"duyet","description":"Managing schema for AWS Athena as Code in GitOps-style","archived":false,"fork":false,"pushed_at":"2025-01-07T21:20:04.000Z","size":113,"stargazers_count":8,"open_issues_count":10,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-28T02:39:07.538Z","etag":null,"topics":["athena","aws","gitops","rust","rust-lang","sql","template-engine"],"latest_commit_sha":null,"homepage":"https://duyet.github.io/athena-rs/athena/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/duyet.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-06T10:17:03.000Z","updated_at":"2024-12-15T00:50:22.000Z","dependencies_parsed_at":"2024-01-25T15:46:17.201Z","dependency_job_id":"a3d0396b-7fcb-49bc-9cf5-778ac3449b3d","html_url":"https://github.com/duyet/athena-rs","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fathena-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fathena-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fathena-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duyet%2Fathena-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/duyet","download_url":"https://codeload.github.com/duyet/athena-rs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248888709,"owners_count":21178096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["athena","aws","gitops","rust","rust-lang","sql","template-engine"],"created_at":"2024-11-27T23:16:17.426Z","updated_at":"2025-04-14T13:32:55.000Z","avatar_url":"https://github.com/duyet.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# athena-rs\n\n[![codecov](https://codecov.io/gh/duyet/athena-rs/branch/master/graph/badge.svg?token=FVVxtMwb4q)](https://codecov.io/gh/duyet/athena-rs)\n[![cargo-clippy](https://github.com/duyet/athena-rs/actions/workflows/cargo-clippy.yml/badge.svg)](https://github.com/duyet/athena-rs/actions/workflows/cargo-clippy.yml)\n[![cargo-test](https://github.com/duyet/athena-rs/actions/workflows/cargo-test.yaml/badge.svg)](https://github.com/duyet/athena-rs/actions/workflows/cargo-test.yaml)\n[![Code coverage](https://github.com/duyet/athena-rs/actions/workflows/cov.yaml/badge.svg)](https://github.com/duyet/athena-rs/actions/workflows/cov.yaml)\n[![cargo-doc](https://github.com/duyet/athena-rs/actions/workflows/cargo-doc.yaml/badge.svg)](https://github.com/duyet/athena-rs/actions/workflows/cargo-doc.yaml)\n[![cargo-fmt](https://github.com/duyet/athena-rs/actions/workflows/cargo-fmt.yaml/badge.svg)](https://github.com/duyet/athena-rs/actions/workflows/cargo-fmt.yaml)\n\n\nAthena-rs is a Rust-based tool for managing AWS Athena schemas. This tool provides two commands to build SQL templates and apply them to Athena. \nThe build command renders SQL from the specified template path using the [`Tera`](https://tera.netlify.app) template engine, while the apply command builds and executes the SQL in Athena.\n\n\n# Installation\n\nThe following command can be used to install athena-rs:\n\n\u003c!-- BEGIN INSTALLATION --\u003e\n```bash\n$ cargo install --git https://github.com/duyet/athena-rs\n$ athena --help\n\nManaging AWS Athena Schemas\n\nUsage: athena \u003cCOMMAND\u003e\n\nCommands:\n  build  Build SQL from template path\n  apply  Build and execute SQL to Athena\n  help   Print this message or the help of the given subcommand(s)\n\nOptions:\n  -h, --help     Print help\n  -V, --version  Print version\n```\n\u003c!-- END INSTALLATION --\u003e\n\n# Usages\n\nThis is an example of a project structure:\n\n```bash\n.\n├── base\n│   ├── index.sql\n│   ├── table_1.sql\n│   └── table_2.sql\n├── prd\n│   └── index.sql\n└── stg\n    └── index.sql\n```\n\nThe `prd/index.sql` file is defined as follows:\n\n\n```sql\n{% set s3_bucket = \"s3://prd\" %}\n{% include 'base/index.sql' %}\n```\n\nThe `base/index.sql` file is defined as follows:\n\n```sql\n{% include \"base/table_1.sql\" %}\n{% include \"base/table_2.sql\" %}\n```\n\nThe `base/table_1.sql` file is defined as follows:\n\n```sql\nCREATE EXTERNAL TABLE IF NOT EXISTS `table_1` (\n  id string,\n  name string\n) LOCATION '{{ s3_bucket }}/table_1';\n\n```\n\n### 1. Build SQL from the specified template\n\nUse the following command to render the template located in [`./prd`](examples/prd):\n\n```bash\n$ cd examples \u0026\u0026 cargo run build ./prd\n```\n\n```sql\nCREATE EXTERNAL TABLE IF NOT EXISTS `table_1` (\n  id string,\n  name string\n) LOCATION 's3://prd/table_1';\n\n\nCREATE EXTERNAL TABLE IF NOT EXISTS `table_2` (\n  id string,\n  name string\n) LOCATION 's3://prd/table_2';\n```\n\n### 2. Apply SQL to Athena\n\n```bash\n$ cd examples \u0026\u0026 athena apply --output_location=s3://athena-output/ ./prd\n```\n\nCompatible with AWS Authentication methods:\n\u003chttps://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html\u003e\n\nUsing AWS profile\n\n```bash\n$ cat ~/.aws/credentials\n# [prd]\n# aws_access_key_id=AKIAIOSFODNN7EXAMPLE\n# aws_secret_access_key=wJalrXUtnFEMI/K7MDENGb/PxRfiCYEXAMPLEKEY\n\n$ cd examples \u0026\u0026 athena apply --profile prd --region us-east-1 ./prd\n```\n\nUsing environment variables\n\n```bash\n$ export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n$ export AWS_DEFAULT_REGION=us-west-2\n\n$ cd examples \u0026\u0026 athena apply ./prd\n```\n\n# Example templates\n\n- Create Athena View: [./examples/base/view.sql](./examples/base/view.sql)\n\n  ```sql\n  -- Database: db1\n  CREATE VIEW \"view\" AS SELECT * FROM table_2;\n\n  /* Database: db2 */\n  CREATE VIEW \"view\" AS SELECT * FROM table_1;\n  ```\n\n- Add partitions date range: [./examples/base/table_1_partitions.sql](./examples/base/table_1_partitions.sql)\n\n  ```sql\n  ALTER TABLE table_name ADD IF NOT EXISTS\n    PARTITION (date_key = \"2022-01-01\") LOCATION \"s3://stg/table_1/year=2022/month=01/day=01\",\n    PARTITION (date_key = \"2022-01-02\") LOCATION \"s3://stg/table_1/year=2022/month=01/day=02\",\n    ...\n  ```\n\n# Limitations\n\n- The athena apply command currently does not support handling errors that occur when executing the generated SQL statements. You must manually check the Athena console or logs for any errors that may occur during execution.\n- This tool has only been tested with basic SQL queries and may not work correctly with more complex queries or with specific versions of AWS Athena.\n- Since Athena can run only one query in a session. So `athena apply` break the queries by semicolon `;`.\n  Must includes the semicolon `;` at the end of each SQL statement.\n- `CREATE VIEW`:\n  - As synopsis of Athena do not accept database name. So please add the database name before the query like this example: [view.sql](./examples/view.sql)\n  - Backquoted identifiers are not supported, use double quotes to quote identifiers.\n- Maybe a bugs of [`walkdir`](https://docs.rs/walkdir), cannot named a file as `macros.sql`\n\n# License\n\nMIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduyet%2Fathena-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fduyet%2Fathena-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduyet%2Fathena-rs/lists"}