https://github.com/thunchanokbow/global-data-deployment-with-cicd-pipeline

The idea of CI/CD pipeline is becoming more and more important part especially for Data Engineer. GitHub Actions and GitLab CI allow us to have workflow to automate that process.
https://github.com/thunchanokbow/global-data-deployment-with-cicd-pipeline

aws-s3 docker-image git github-actions gitlab-ci python3 workflows

Last synced: 4 months ago
JSON representation

The idea of CI/CD pipeline is becoming more and more important part especially for Data Engineer. GitHub Actions and GitLab CI allow us to have workflow to automate that process.

• Host: GitHub
• URL: https://github.com/thunchanokbow/global-data-deployment-with-cicd-pipeline
• Owner: thunchanokbow
• Created: 2024-01-05T07:19:10.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2024-03-06T06:50:30.000Z (over 1 year ago)
• Last Synced: 2025-01-09T09:58:43.705Z (5 months ago)
• Topics: aws-s3, docker-image, git, github-actions, gitlab-ci, python3, workflows
• Language: Python
• Homepage:
• Size: 19.6 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Global Data Deployment with CI/CD pipeline

Being a data engineer is creating different automations and in particular one area that’s really important is CICD which stands for continuous integration and continuous deployment and this is area where you can automate your testing your release strategy through tools like **Github Actions or GitLab CI** and I also understand that this concept can be a little unclear if you haven’t seen it in action so **this simple project will help you use CI/CD tools to automate some basic tasks**.

## Project Overview

![0](/images/09.png)

Extract data and transform data using **Python**, define a **GitHub Actions** workflow that automates tasks, and deploy to an **AWS S3** bucket using **GitLab CI**. This project enabled the generation of comparative data, which can be used to consider which country has the highest vaccination rate or which country has the fastest economic recovery.



## Contents

[GitHub Actions](sections/01-github-actions.md).


- [Create YAML contents](sections/01-github-actions.md#Create-YAML-contents)


- [Viewing workflows results](sections/01-github-actions.md#Viewing-workflows-results)


[GitLab CI](sections/02-gitlab-ci.md#).


- [Upload files to the GitLab Repository](sections/02-gitlab-ci.md#).


- [Create Buckget on AWS S3](sections/02-gitlab-ci.md#Create-Buckget-on-AWS-S3).
 

- [Generate new access key And secret key](sections/02-gitlab-ci.md#Generate-new-access-key-And-secret-key).
 

- [Store keys in CI/CD Varisbles](sections/02-gitlab-ci.md#Store-keys-in-CI/CD-Varisbles).
 

- [Create a YAML file](sections/02-gitlab-ci.md#Create-a-YAML-file).
 

- [Viewing a workflow result](sections/02-gitlab-ci.md#Viewing-a-job-result).