{"id":23870089,"url":"https://github.com/yuyatinnefeld/dataproc-api-service","last_synced_at":"2026-04-22T05:33:16.682Z","repository":{"id":45114625,"uuid":"445486317","full_name":"yuyatinnefeld/dataproc-api-service","owner":"yuyatinnefeld","description":"🧪 Test Features 🧪 | GCP Dataproc + FastAPI","archived":false,"fork":false,"pushed_at":"2022-01-07T11:21:40.000Z","size":17,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-02-22T18:42:00.996Z","etag":null,"topics":["dataproc","fastapi","gcp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuyatinnefeld.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-07T10:41:46.000Z","updated_at":"2023-06-01T06:36:13.000Z","dependencies_parsed_at":"2022-09-14T07:22:20.490Z","dependency_job_id":null,"html_url":"https://github.com/yuyatinnefeld/dataproc-api-service","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yuyatinnefeld/dataproc-api-service","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fdataproc-api-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fdataproc-api-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fdataproc-api-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fdataproc-api-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuyatinnefeld","download_url":"https://codeload.github.com/yuyatinnefeld/dataproc-api-service/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fdataproc-api-service/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32122734,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-22T00:31:26.853Z","status":"online","status_checked_at":"2026-04-22T02:00:05.693Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataproc","fastapi","gcp"],"created_at":"2025-01-03T13:52:12.854Z","updated_at":"2026-04-22T05:33:16.667Z","avatar_url":"https://github.com/yuyatinnefeld.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dataproc API Service\nThis Project is built with FastAPI. You can manage the Dataproc Service in Swagger UI.\n\n## Essential Pieces of the Software\nYou need these softwares\n\n- GCP Project (ex. yt-demo-dev)\n- GCP Service Account Json Key (stored in conf/creds.json)\n- Python 3.7 or older version\n- Cloud SDK \n- Gitlab\n- Docker\n\n## Setup\n\n### Setup Service Account\n1. create a service account\n2. create a ./conf directory in the project\n3. download this in the ./conf direcotry.\n### Setup Environment Variables\n```bash\nvi settings.py\n```\n\n### Setup the Python Environment\n```bash\npython3 -m venv venv\nsource venv/bin/activate\npython3 -m pip install -r requirements.txt\nexport GOOGLE_APPLICATION_CREDENTIALS=\"$(pwd)/conf/creds.json\"\n```\n\n### Setup the GCP Environment\n```bash\nPROJECT_ID=\"yt-demo-dev\"\nREGION=\"europe-west1\"\nSERVICE_ACCOUNT_NAME=\"demo-dataproc\"\nGCP_SERVICE_ACCOUNT_KEY_DEV=${GOOGLE_APPLICATION_CREDENTIALS}\n```\n#### Create a Dataproc Service Account\n```bash\ngcloud auth activate-service-account --key-file ${GCP_SERVICE_ACCOUNT_KEY_DEV}\ngcloud config set project ${PROJECT_ID}\n\n# create a service account for dataproc workflow template\ngcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME} \\\n    --display-name=\"dataproc service account\"\n\ngcloud projects add-iam-policy-binding ${PROJECT_ID} \\\n    --member=\"serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com\" \\\n    --role=\"roles/dataproc.admin\"\n\ngcloud projects add-iam-policy-binding ${PROJECT_ID} \\\n    --member=\"serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com\" \\\n    --role=\"roles/storage.admin\"\n\ngcloud projects add-iam-policy-binding ${PROJECT_ID} \\\n    --member=\"serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com\" \\\n    --role=\"roles/dataproc.editor\"\n```\n\n#### Configure the Dataproc Environment\n```bash\n\n# upload a autoscaling cluster\nPOLICY_NAME=\"yt-demo-dev-autoscaling-policy\"\nFILE_NAME=\"resources/autoscaling_policy.yaml\"\n\ngcloud dataproc autoscaling-policies import ${POLICY_NAME} \\\n    --source=${FILE_NAME} \\\n    --region=${REGION}\n\n# create 2 buckets for the dataproc cluster configuration\nDATAPROC_TAMP_BUCKET=\"${PROJECT_ID}-dataproc-temp\"\nDATAPROC_STAGING_BUCKET=\"${PROJECT_ID}-dataproc-staging\"\ngsutil mb -c \"STANDARD\" -l ${REGION} gs://${DATAPROC_TAMP_BUCKET}\ngsutil mb -c \"STANDARD\" -l ${REGION} gs://${DATAPROC_STAGING_BUCKET}\n\n# upload few spark jobs in the temp bucket for the dataproc workflow template\ngsutil cp spark/data_collection_job.py gs://${DATAPROC_TAMP_BUCKET}/src/data_collection_job.py\ngsutil cp spark/data_processing_job.py gs://${DATAPROC_TAMP_BUCKET}/src/data_processing_job.py\ngsutil cp spark/data_validation_job.py gs://${DATAPROC_TAMP_BUCKET}/src/data_validation_job.py\n\n# create a workflow templates\nDATA_COLLECTION_TEMP=\"data_collection_wf_temp\"\nDATA_PROCESSING_TEMP=\"data_processing_wf_temp\"\nDATA_VALIDATION_TEMP=\"data_validation_wf_temp\"\n\ngcloud dataproc workflow-templates import ${DATA_COLLECTION_TEMP} \\\n    --source=resources/${DATA_COLLECTION_TEMP}.yaml \\\n    --region=${REGION}\n\ngcloud dataproc workflow-templates import ${DATA_PROCESSING_TEMP} \\\n    --source=resources/${DATA_PROCESSING_TEMP}.yaml \\\n    --region=${REGION}\n\ngcloud dataproc workflow-templates import ${DATA_VALIDATION_TEMP} \\\n    --source=resources/${DATA_VALIDATION_TEMP}.yaml \\\n    --region=${REGION}\n```\n## Getting started\n\n### Local Run\n```bash\nuvicorn app.main:app --reload\ncurl -i http://127.0.0.1:8000\n# open the URL\nhttp://127.0.0.1:8000/docs\n\n```\n\n### Dockerize Local Run\n\n1. Open the Dockerfile \n2. Activate `COPY ./conf /code/conf` cmd\n\n2. Docker run \n```bash\n# test run\ndocker build -t fastapi-image .\ndocker run -it -p 8080:8080 -d fastapi-image\ndocker ps\ncurl -i http://127.0.0.1:8080\n```\n\n## Deploy to Cloud Run (NoOps)\n1. Open Cloud Code Extenstion\n2. Select Deploy to Cloud Run\n3. Setup the environment\n- service name: dataproc-api-service\n- region: europe-west1\n- url: gcr.io/yt-demo-dev/dataproc-api-service\n\n## Open the Cloud Run and click the service URL\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuyatinnefeld%2Fdataproc-api-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuyatinnefeld%2Fdataproc-api-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuyatinnefeld%2Fdataproc-api-service/lists"}