https://github.com/agrc/geocode-job

The process for comparing vista data with agrc addresses data in the google cloud
https://github.com/agrc/geocode-job

google-cloud government government-tech kubernetes netcore2-webapi python template

Last synced: 15 days ago
JSON representation

The process for comparing vista data with agrc addresses data in the google cloud

• Host: GitHub
• URL: https://github.com/agrc/geocode-job
• Owner: agrc
• Created: 2019-04-10T19:23:45.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2019-04-18T23:23:44.000Z (almost 6 years ago)
• Last Synced: 2024-11-17T11:30:47.251Z (3 months ago)
• Topics: google-cloud, government, government-tech, kubernetes, netcore2-webapi, python, template
• Language: Python
• Homepage:
• Size: 17.6 KB
• Stars: 1
• Watchers: 5
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Geocode-job

Create a containerized python geocoding script as a kubernetes job.

### Steps to run

1. Prepare data

   1. Run [prep_addresses.py](vista/prep_addresses.py)

        - Pulls data from VISTA

        - Partions the data into multiple CSVs

1. Create k8s job yaml specifications

   1. Run [vista_job_template.py](vista/vista_job_template.py)

        - Uploads data to Cloud Storage with [service account credenitals](.secrets/gcs-gecode-writer.json.template)

        - Creates k8s job template files

1. Apply seceret for service worker with cloud storage permissions to k8s cluster

   1. authorize kubectl with geocoding api cluster

   1. run `kubectl apply -f .secrets/gcs-secret.yml`

        - Service account key must first be base64 encoded into [gcs-secret.yml](.secrets/gcs-secret-template.yml.jinja2)

1. Apply job yamls to cluster 

   1. run `kubectl apply -f job.yaml`

1. Download geocoded CSVs from cloud storage

### Steps to build

1.   Build container from docker file

     1. docker build . -t {container name}

     1. docker tag {container name}:latest gcr.io/{project id}/webapi/{container name}:latest

1. Push to registery

   1. docker push gcr.io/{project id}/webapi/{container name}:latest

   1. User needs project permissions to allow push to gcr