https://github.com/livebook-dev/req_athena
Conveniences for querying Amazon Athena with Req
https://github.com/livebook-dev/req_athena
Last synced: 3 months ago
JSON representation
Conveniences for querying Amazon Athena with Req
- Host: GitHub
- URL: https://github.com/livebook-dev/req_athena
- Owner: livebook-dev
- Created: 2022-05-25T20:12:57.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2025-11-12T07:17:26.000Z (7 months ago)
- Last Synced: 2026-02-14T18:41:36.552Z (4 months ago)
- Language: Elixir
- Homepage:
- Size: 70.3 KB
- Stars: 8
- Watchers: 4
- Forks: 9
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# ReqAthena
[](https://hexdocs.pm/req_athena)
[](https://hex.pm/packages/req_athena)
[Req](https://github.com/wojtekmach/req) plugin for [AWS Athena](https://docs.aws.amazon.com/athena/latest/APIReference/Welcome.html).
ReqAthena makes it easy to make Athena queries and save the results into S3 buckets.
By default, `ReqAthena` will query results and use the default output format,
which is the decoded API response. To change that, you can use the `:format`
option documented below.
## Usage
```elixir
Mix.install([
{:req, "~> 0.5.8"},
{:req_athena, "~> 0.3.0"}
])
opts = [
access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY"),
region: System.fetch_env!("AWS_REGION"),
database: "default",
# This may need to be a new directory for every query using the `:json` or `:explorer` formats.
# See the docs for details: https://hexdocs.pm/req_athena/ReqAthena.html#new/1
output_location: "s3://my-bucket/my-location"
]
req = ReqAthena.new(opts)
# Create table from Registry of Open Data on AWS
# See: https://registry.opendata.aws/osm/
query = """
CREATE EXTERNAL TABLE IF NOT EXISTS planet (
id BIGINT,
type STRING,
tags MAP,
lat DECIMAL(9,7),
lon DECIMAL(10,7),
nds ARRAY>,
members ARRAY>,
changeset BIGINT,
timestamp TIMESTAMP,
uid BIGINT,
user STRING,
version BIGINT,
visible BOOLEAN
)
STORED AS ORCFILE
LOCATION 's3://osm-pds/planet/';
"""
ReqAthena.query!(req, query).body
# =>
# %{
# "Output" => "",
# "ResultSet" => %{
# "ColumnInfos" => [],
# "ResultRows" => [],
# "ResultSetMetadata" => %{"ColumnInfo" => []},
# "Rows" => []
# }
# }
# With plain string query
query = "SELECT id, type, tags, members, timestamp, visible FROM planet WHERE id = 470454 and type = 'relation'"
ReqAthena.query!(req, query, [], format: :json).body
# =>
# [
# %{
# "id" => 470454,
# "members" => [
# %{"ref" => 670007839, "role" => "", "type" => "node"},
# %{"ref" => 670007840, "role" => "", "type" => "node"}
# ],
# "tags" => %{
# "name" => "Mérignac A",
# "network" => "NTF-5",
# "ref" => "17229A",
# "site" => "geodesic",
# "source" => "©IGN 2010 dans le cadre de la cartographie réglementaire",
# "type" => "site",
# "url" => "http://geodesie.ign.fr/fiches/index.php?module=e&action=fichepdf&source=carte&sit_no=17229A"
# },
# "timestamp" => "2017-01-21 12:51:34",
# "type" => "relation",
# "visible" => true
# }
# ]
# With parameterized query
query = "SELECT id, type FROM planet WHERE id = ? and type = ?"
ReqAthena.query!(req, query, [239_970_142, "node"], format: :json).body
#=> [%{"id" => 239970142, "type" => "node"}]
```
## License
Copyright (C) 2022 Dashbit
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.