{"id":17335113,"url":"https://github.com/feroult/serverless-superset","last_synced_at":"2025-04-14T17:31:14.249Z","repository":{"id":52155803,"uuid":"241522465","full_name":"feroult/serverless-superset","owner":"feroult","description":"Serverless Superset on Google Cloud","archived":false,"fork":false,"pushed_at":"2021-05-06T19:54:12.000Z","size":22,"stargazers_count":22,"open_issues_count":3,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-01T05:42:13.483Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/feroult.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-19T03:20:16.000Z","updated_at":"2024-06-04T09:28:34.000Z","dependencies_parsed_at":"2022-08-28T15:41:09.870Z","dependency_job_id":null,"html_url":"https://github.com/feroult/serverless-superset","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feroult%2Fserverless-superset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feroult%2Fserverless-superset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feroult%2Fserverless-superset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feroult%2Fserverless-superset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/feroult","download_url":"https://codeload.github.com/feroult/serverless-superset/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223639403,"owners_count":17177816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T15:08:17.600Z","updated_at":"2024-11-08T06:04:07.602Z","avatar_url":"https://github.com/feroult.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Serverless Superset on Google Cloud\n\nFor a more detailed guide on how to deploy [Apache Superset (Incubating)](https://superset.incubator.apache.org/) on Google Cloud refere tho this [blog post](https://medium.com/@feroult/serverless-superset-on-google-cloud-87d3cf324845).\n\n## TL;DR Instructions\n\n1. You can start by creating the following environment variables:\n\n```Bash\nexport PROJECT_ID=\"\u003cyour project id\u003e\"\nexport VPC_NETWORK=\"superset-network\"\nexport REGION=\"europe-west1\"\nexport SHORT_REGION=\"europe-west\"\nexport ZONE=\"europe-west1-b\"\nexport REDIS_INSTANCE_NAME=\"superset\"\nexport SQL_INSTANCE_NAME=\"webapps-configs\"\nexport SQL_DATABASE=\"superset\"\nexport SQL_USER=\"superset\"\nexport SQL_PASSWORD=`openssl rand -base64 15`\necho \"Your password is $SQL_PASSWORD\";\n```\n\nDon't worry about the password, if you follow all the steps in the same session, it will get written in the `config\\app.yaml` file created in the 7th point.\n\n2. Don't forget to update the project in gcloud tools (maybe create rather a configuration):\n\n```Bash\ngcloud config set project ${PROJECT_ID}\ngcloud auth configure-docker\n```\n\n3. Building Superset.\n\nCreates a one-time cloud build to create the docker image we are going to be working with (version 0.37.2).\nTo build a different version, change the Dockerfile\n\n```Bash\nsource scripts/build.sh\n```\n\n4. Setup VPC (Networking).\n\nCreates a VPC network to allow private communication between AppEngine and Redis and Cloud SQL. Also creates private addresses for VPC_PEERING and allows connection to the redis and Cloud SQL via those addresses. Finally it creates a firewall-rule to allow ssh connection to the machines inside of the network.\n\n```Bash\nsource scripts/setup_network.sh\n```\n\n5. Setup Memorystore (Redis).\n\nCreates a redis instance and saves the IP into REDIS_HOST environment variable.\n\n```Bash\nsource scripts/setup_redis.sh\n```\n\n6. Setup Cloud SQL.\n\nCreates a PostgreSQL (Version 12) instance, then it creates a database inside, and a user to connect to it. Finally it saves the IP into SQL_HOST environment variable.\n\n```Bash\nsource scripts/setup_sql.sh\n```\n\n7. Create the `config/app.yaml` file.\n\nIt uses the `config/app-template.yaml` file and substitutes all the environment variables necessary to create the configuration file to deploy the app engine. Also, initializes the app engine in the project if it have never been configured before. Another side effect of creating the app engine for the first time, is that it also creates the service account `${PROJECT_ID}@appspot.gserviceaccount.com` necessary to setup the bigquery steps.\n\n```Bash\nsource scripts/setup_appengine.sh\n```\n\n8. Setup BigQuery.\n\nGrant roles to the default app engine service account `${PROJECT_ID}@appspot.gserviceaccount.com` to query BigQuery\n\n```Bash\nsource scripts/setup_bigquery.sh\n```\n\n9. Initializes superset.\n\nSpins a Compute Engine with the Docker image created in point **3**. This will initilize the database and load some demo data. It also creates the admin user, so you will be prompted for a username and password.\n\n```Bash\nsource scripts/superset_init.sh\n```\n\n10. Deploys the App Engine application.\n\nThis step can be done multiple times, each time will create a new App Engine version and start serving from it.\n\nSteps that it does:\n\n- Extract the Superset build from our docker image to staging folder\n- Merge into the same staging folder the following configuration files: `app.yaml`, `superset-config.py`, `requirements-db.txt`\n- Deploy the app to the cloud\n\n```Bash\nsource scripts/deploy.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fferoult%2Fserverless-superset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fferoult%2Fserverless-superset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fferoult%2Fserverless-superset/lists"}