{"id":16652490,"url":"https://github.com/nicor88/dbt-serverless","last_synced_at":"2025-09-19T02:31:59.589Z","repository":{"id":56764562,"uuid":"204902596","full_name":"nicor88/dbt-serverless","owner":"nicor88","description":"Run dbt serverless in the Cloud (AWS)","archived":false,"fork":false,"pushed_at":"2020-01-20T17:52:52.000Z","size":2855,"stargazers_count":40,"open_issues_count":2,"forks_count":13,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-30T01:42:02.545Z","etag":null,"topics":["aws","cloud","dbt","ecs","fargate","serverless","terraform"],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicor88.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-28T10:07:14.000Z","updated_at":"2024-12-12T09:28:42.000Z","dependencies_parsed_at":"2022-08-16T02:10:35.469Z","dependency_job_id":null,"html_url":"https://github.com/nicor88/dbt-serverless","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Fdbt-serverless","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Fdbt-serverless/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Fdbt-serverless/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Fdbt-serverless/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicor88","download_url":"https://codeload.github.com/nicor88/dbt-serverless/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233546485,"owners_count":18692226,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud","dbt","ecs","fargate","serverless","terraform"],"created_at":"2024-10-12T09:28:51.332Z","updated_at":"2025-09-19T02:31:54.355Z","avatar_url":"https://github.com/nicor88.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dbt-serverless\nRun dbt serverless in the Cloud (AWS)\n\n## Requirements\n* aws credentials configured in `~/.aws/credentials`\n* aws cli\n  \u003cpre\u003e\n  pip install awscli\n  \u003c/pre\u003e\n* terraform\n\n\n## Deploy\nThe infrastructure is based on terraform.\nI setup a terraform backend to keep terraform state. The backend is based an S3 bucket that was created manually.\nYou can create an S3 bucket simply running:\n\u003cpre\u003e\naws s3api create-bucket --bucket nicor88-eu-west-1-terraform --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1\n\u003c/pre\u003e\nRemember to change the name of the S3 bucket inside `infrastructure/provider.tf` before running the following commands:\n\u003cpre\u003e\nexport AWS_PROFILE=your_profile\nmake infra-plan\nmake infra-apply\n\u003c/pre\u003e\n\nAfter the infra is created correctly, you can push an new image to the ECR repository running:\n\u003cpre\u003e\nmake push-to-ecr AWS_ACCOUNT_ID=your_account_id\n\u003c/pre\u003e\n\n### Note\nCurrently Aurora Postgres is only accessible inside the VPC.\nI create a Network load balancer, to connect to the DB from everywhere, but you need to get the Private IP of Aurora Endpoint.\nYou can simply run:\n\u003cpre\u003e\nnslookup your_aurora_enpoint\n# returned from the terraform outputs\n\u003c/pre\u003e\nThen you need to replace the 2 variables:\n* autora_postgres_serverless_private_ip_1\n* autora_postgres_serverless_private_ip_2\n\nand apply again the changes with the command `make infra-apply`\n\n## Infrastructure\n\n### AWS Step Function\n\n#### Input example\n\n\u003cpre\u003e\n{\n  \"commands1\": [\n    \"dbt\",\n    \"run\",\n    \"--models\"\n    \"example\"\n  ],\n  {\n  \"commands2\": [\n    \"dbt\",\n    \"run\",\n  \t\"--models\"\n    \"just_another_example\"\n  ]\n}\n}\n\u003c/pre\u003e\n\n## Airflow operator\nIt's possible to invoke ECS Fargate containers to run dbt also from Airflow.\nHere an example of how to call a DbtOperator from Airflow:\n\u003cpre\u003e\ndbt_run_example = DbtOperator(\n    dag=dag,\n    task_id='dbt_example',\n    command='run',\n    target='dev',\n    dbt_models='my_example',\n    subnets=['subnet_id_1', 'subnet_id_2'],\n    security_groups=['sg_1']\n)\n\u003c/pre\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicor88%2Fdbt-serverless","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicor88%2Fdbt-serverless","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicor88%2Fdbt-serverless/lists"}