{"id":14065558,"url":"https://github.com/soumilshah1995/AthenaUsageExtractor","last_synced_at":"2025-07-29T20:33:19.019Z","repository":{"id":112454903,"uuid":"534320025","full_name":"soumilshah1995/AthenaUsageExtractor","owner":"soumilshah1995","description":"Athena usage is simple python library that allows you to extract all usage information for given date range and for given workgroup","archived":false,"fork":false,"pushed_at":"2022-09-08T18:08:25.000Z","size":57,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-14T07:09:18.681Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/soumilshah1995.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-08T17:24:51.000Z","updated_at":"2022-12-17T18:30:59.000Z","dependencies_parsed_at":"2023-05-15T04:15:11.604Z","dependency_job_id":null,"html_url":"https://github.com/soumilshah1995/AthenaUsageExtractor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soumilshah1995%2FAthenaUsageExtractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soumilshah1995%2FAthenaUsageExtractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soumilshah1995%2FAthenaUsageExtractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/soumilshah1995%2FAthenaUsageExtractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/soumilshah1995","download_url":"https://codeload.github.com/soumilshah1995/AthenaUsageExtractor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228046163,"owners_count":17861101,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-13T07:04:33.600Z","updated_at":"2024-12-04T04:31:18.430Z","avatar_url":"https://github.com/soumilshah1995.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Athena Usage Extractor \n\n[![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)]\n\n* Athena helps you analyze unstructured, semi-structured, and structured data stored in Amazon S3. Examples include CSV, JSON, or columnar data formats such as Apache Parquet and Apache ORC. You can use Athena to run ad-hoc queries using ANSI SQL, without the need to aggregate or load the data into Athena.\n        \n* Athena usage is simple python library that allows you to extract all usage information \n\n![image](https://user-images.githubusercontent.com/39345855/189194783-6d607f5b-11af-45e4-882a-fbbd117165dc.png)\n\n\n## Installation\nac\n```bash\npip install athena-usage-metrics-extractor\n```\n## Usage\n\n```python\nimport sys\nfrom AthenaUsageExtractor import AthenaUsageExtractor\n\n\ndef main():\n    helper = AthenaUsageExtractor(\n        aws_region='us-east-1',\n        aws_access_key='XXXXX',\n        aws_secret_key='XXXXX'\n    )\n    response = helper.get_usage_for_date(date='2022-08-12', workgroup='primary')\n    while True:\n        try:\n            data = next(response)\n            print(data)\n        except StopIteration as e:\n            break\n        except Exception as e:\n            break\n\nmain()\n\n```\n## Json format Returned \n```json\n{\n   \"QueryExecutionId\":\"490024e6-3e89-4ec4-9ffd-b1302a77d33d\",\n   \"Query\":\"\u003cYOU WILL GET THE QUERY USER RAN \u003e\",\n   \"StatementType\":\"DML\",\n   \"WorkGroup\":\"primary\",\n   \"OutputLocation\":\"\u003cAWS S3 Output Path \u003e\",\n   \"Database\":\"default\",\n   \"SelectedEngineVersion\":\"AUTO\",\n   \"EffectiveEngineVersion\":\"Athena engine version 2\",\n   \"EngineExecutionTimeInMillis\":\"14045\",\n   \"DataScannedInBytes\":\"59597591861\",\n   \"TotalExecutionTimeInMillis\":\"14292\",\n   \"QueryQueueTimeInMillis\":\"214\",\n   \"QueryPlanningTimeInMillis\":\"960\",\n   \"ServiceProcessingTimeInMillis\":\"33\",\n   \"State\":\"SUCCEEDED\",\n   \"SubmissionDateTime\":\"2022-08-12 13:56:07.837000-04:00\",\n   \"CompletionDateTime\":\"2022-08-12 13:56:22.129000-04:00\"\n}\n```\n\n## Authors\n\n* **Soumil Nitin Shah** \n\n\n## Soumil Nitin Shah \nBachelor in Electronic Engineering |\nMasters in Electrical Engineering | \nMaster in Computer Engineering |\n\n* Website : https://soumilshah.herokuapp.com\n* Github: https://github.com/soumilshah1995\n* Linkedin: https://www.linkedin.com/in/shah-soumil/\n* Blog: https://soumilshah1995.blogspot.com/\n* Youtube : https://www.youtube.com/channel/UC_eOodxvwS_H7x2uLQa-svw?view_as=subscriber\n* Facebook Page : https://www.facebook.com/soumilshah1995/\n* Email : shahsoumil519@gmail.com\n* projects : https://soumilshah.herokuapp.com/project\n\n\nI earned a Bachelor of Science in Electronic Engineering and a double master’s in Electrical and Computer Engineering. I have extensive expertise in developing scalable and high-performance software applications in Python. I have a YouTube channel where I teach people about Data Science, Machine learning, Elastic search, and AWS. I work as data collection and processing Team Lead at Jobtarget where I spent most of my time developing Ingestion Framework and creating microservices and scalable architecture on AWS. I have worked with a massive amount of data which includes creating data lakes (1.2T) optimizing data lakes query by creating a partition and using the right file format and compression. I have also developed and worked on a streaming application for ingesting real-time streams data via kinesis and firehose to elastic search\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoumilshah1995%2FAthenaUsageExtractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoumilshah1995%2FAthenaUsageExtractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoumilshah1995%2FAthenaUsageExtractor/lists"}