{"id":24087951,"url":"https://github.com/tinybirdco/analyze-aws-usage-localstack","last_synced_at":"2026-06-13T17:02:13.186Z","repository":{"id":68733128,"uuid":"534746773","full_name":"tinybirdco/analyze-aws-usage-localstack","owner":"tinybirdco","description":"Analyze your AWS usage with LocalStack and Tinybird","archived":false,"fork":false,"pushed_at":"2022-09-21T13:10:37.000Z","size":10,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-27T05:24:44.862Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinybirdco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-09T17:43:34.000Z","updated_at":"2023-01-27T14:47:01.000Z","dependencies_parsed_at":"2023-03-13T20:27:41.229Z","dependency_job_id":null,"html_url":"https://github.com/tinybirdco/analyze-aws-usage-localstack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tinybirdco/analyze-aws-usage-localstack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fanalyze-aws-usage-localstack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fanalyze-aws-usage-localstack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fanalyze-aws-usage-localstack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fanalyze-aws-usage-localstack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinybirdco","download_url":"https://codeload.github.com/tinybirdco/analyze-aws-usage-localstack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fanalyze-aws-usage-localstack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34292326,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-13T02:00:06.617Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-10T03:56:26.689Z","updated_at":"2026-06-13T17:02:13.181Z","avatar_url":"https://github.com/tinybirdco.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# analyze-aws-usage-localstack\nAnalyze your AWS usage with LocalStack and Tinybird\n\n\n## Developing the LocalStack extensions\n\n### Getting started with extension developer mode\n\nPrerequisites:\n\n* Install the LocalStack CLI and cookiecutter for templating `pip install --upgrade localstack cookiecutter`\n* Log in to your LocalStack Pro account with `localstack login`\n\nCreate a new extension and follow the template wizard\n\n```console\n % localstack extensions dev new\n\nYou've downloaded /home/thomas/.cookiecutters/localstack-extensions before. Is it okay to delete and re-download it? [yes]: yes\nproject_name [My LocalStack Extension]: LocalStack Tinybird Logger    \nproject_short_description [All the boilerplate you need to create a LocalStack extension.]: LocalStack extension to log AWS API calls into Tinybird\nproject_slug [localstack-tinybird-logger]: \nmodule_name [localstack_tinybird_logger]: \nfull_name [Jane Doe]: Thomas Rausch      \nemail [jane@example.com]: thomas@localstack.cloud\ngithub_username [janedoe]: thrau\nversion [0.1.0]: \n```\n\nEnable dev mode for the new extension\n```console\n % localstack extensions dev enable localstack-tinybird-logger\n/home/thomas/workspace/localstack/analyze-aws-usage-localstack/localstack-tinybird-logger enabled\n```\n\nPackage the extension so it can be loaded by localstack\n```console\n % cd localstack-tinybird-logger \u0026\u0026 make install\n```\nThis will create a virtual environment containing all the dependencies required to load the extension.\nLocalStack comes with a lot of transitive dependencies, including `requests`, which you can simply use out of the box, but it's advisable to add dependencies explicitly as the project matures.\nWhen you extend the `setup.cfg`, run `make install` again.\n\nStart LocalStack with extension dev mode enabled:\n```console\n % LOCALSTACK_API_KEY=********** EXTENSION_DEV_MODE=1 localstack start\n```\nYou should see `MyExtension: localstack is running` in the log. If so, you are ready to develop the extension!\n\n### Basic idea of the extension\n\nWe want to record every AWS API call going through LocalStack to Tinybird.\nTo that end, we can extend the *response handlers* of the LocalStack Gateway, which is the component responsible for parsing and dispatching an AWS API call to the corresponding backend.\nWith a custom response handler, you can intercept all API calls after they have been processed by the backend, and run custom logic.\nYou can [find out more about how the LocalStack Gateway works](https://localstack.notion.site/LocalStack-Core-Concepts-cd342b31882946a0a3dfb4a7b21e8792#ccbea13c299d4b90b5e12d04c7add1b4) in our contributing guide.\n\n\n### Intercepting the request\n\nConceptually, we want to intercept the AWS request and the generated response with a service response handler, create a log record, and post that log record to Tinybird.\nIn LocalStack, every AWS request is encapsulated in a `RequestContext` object that goes through several handlers that are part of the `HandlerChain`.\nThe request context contains which AWS service the request is made to, which API operation, and other metadata.\nIt also contains the parsed AWS request, i.e., the parameters to the operation, which we can add as an optional payload.\n\n```python\ndef logger(chain: HandlerChain, context: RequestContext, response: Response):\n    payload = {\n        \"service\": context.service.service_name, # e.g., \"sqs\" or \"s3\"\n        \"operation\": context.operation.name, # e.g., \"CreateQueue\" or \"DeleteBucket\"\n        \"request\": json.dumps(context.service_request), # the request parameters\n        \"status_code\": response.status_code, # the HTTP status code the backend created\n        \"response\": json.dumps(context.service_response), # the response from the service backend\n        # ... additional metadata can be added\n    }\n\n```\n\n### Send the request to Tinybird\n\nFor our logger, all we need to do is create a JSON document from our payload and make a POST request to the Tinybird events HFI endpoint.\nThis is as easy as:\n\n```python\ndef logger(...):\n    # ...\n    event = json.dumps(payload)\n\n    requests.post(\n        url=\"https://api.tinybird.co/v0/events?name=aws_api_calls\",\n        data=event,\n        headers={\n            \"Authorization\": \"Bearer \u003ctinybird token\u003e\"\n        }\n    )\n```\n\n## Tinybird analytics\n\n... TODO\n\n### Getting started with queries\n\nHere are some example queries to run:\n\nThe number of localstack runs per day:\n\n```sql\nSELECT\n  toDate(`timestamp`) as `date`,\n  countDistinct(session_id) as number_of_sessions\nFROM aws_api_calls\nGROUP BY `date`\n```\n\nThe most used service operation:\n\n```sql\nSELECT\n  service,\n  operation,\n  countDistinct(session_id) as number_of_sessions\nFROM aws_api_calls\nGROUP BY service, operation\n```\n\nDe-normalize specific AWS operations:\n\n```sql\nSELECT\n  timestamp,\n  operation,\n  JSONExtractString(request, 'Bucket') as bucket\nFROM aws_api_calls\nWHERE service == 's3'\n```\n\n### Materialization\n\n... TODO\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fanalyze-aws-usage-localstack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinybirdco%2Fanalyze-aws-usage-localstack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fanalyze-aws-usage-localstack/lists"}