https://github.com/francois-lenne/play-bq-gcp

Data pipeline in order to retrieve data from the playstation API to BigQuery
https://github.com/francois-lenne/play-bq-gcp

bigquery cicd data-engineering google-cloud python

Last synced: about 21 hours ago
JSON representation

Data pipeline in order to retrieve data from the playstation API to BigQuery

• Host: GitHub
• URL: https://github.com/francois-lenne/play-bq-gcp
• Owner: Francois-lenne
• Created: 2024-01-17T20:40:19.000Z (10 months ago)
• Default Branch: master
• Last Pushed: 2024-09-10T13:35:17.000Z (2 months ago)
• Last Synced: 2024-09-10T15:05:03.628Z (2 months ago)
• Topics: bigquery, cicd, data-engineering, google-cloud, python
• Language: Python
• Homepage:
• Size: 121 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# what is the upose of this project

This project have to goal to show my Google cloud platform and data engineering skils. In order, i rpopose in this project to retrieve data from the playstation API by the package PSWANAPI in order to follow the trophee and the time play by game for the user. In order to create this project i use a python script in a google cloud function host in this repo and i create a CI/CD with google cloud code. In order ot orchestrate the execution of the google cloud function i use google cloud schedueler.

# The architecture

![Schema play-gcp-bq](https://github.com/Francois-lenne/play-bq-gcp/assets/114836746/fa529724-7d37-465b-801e-f122090d6e19)