https://github.com/wfamous/fiv_update-data

This project automates the retrieval, processing, and publishing of digital product data for our Shopify store. It integrates Google Cloud Platform (GCP), Amazon Web Service (AWS), Terraform (Tofu), Python, Bash, Ansible and GitHub Actions to manage data pipelines efficiently.
https://github.com/wfamous/fiv_update-data

ansible aws bash data data-analysis data-science devops gcp python pythonpackage shopify terraform tofu

Last synced: 3 months ago
JSON representation

This project automates the retrieval, processing, and publishing of digital product data for our Shopify store. It integrates Google Cloud Platform (GCP), Amazon Web Service (AWS), Terraform (Tofu), Python, Bash, Ansible and GitHub Actions to manage data pipelines efficiently.

• Host: GitHub
• URL: https://github.com/wfamous/fiv_update-data
• Owner: wfamous
• Created: 2024-10-12T16:10:55.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-03-03T15:09:01.000Z (about 1 year ago)
• Last Synced: 2025-04-03T18:03:48.086Z (about 1 year ago)
• Topics: ansible, aws, bash, data, data-analysis, data-science, devops, gcp, python, pythonpackage, shopify, terraform, tofu
• Language: HCL
• Homepage: https://famousinvogue.com
• Size: 71.3 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Automated digital product updates with Panda, GCP, Terraform, and Shopify and more 🚀

## Overview

If you're looking to automate digital product updates efficiently, this project is for you!

It automates the retrieval, processing, and publishing of digital product data for a Shopify store and thus, 

helps reduce manual work, and ensures seamless product updates. It integrates Google Cloud Platform (GCP), Amazon 

Web Service (AWS), Terraform (Tofu), Ansible, Python, Bash and GitHub Actions to manage data pipelines efficiently.

## Key Features

✅ Data Extraction & Cleaning:

- Uses Google Cloud Storage to store raw product datasets.

- Python & Pandas process, clean, and update product information.

✅ Automation & Infrastructure:

- Terraform (Tofu) provisions GCP resources (buckets, IAM, etc.).

- GitHub Actions automates Terraform deployments and Python scripts.

✅ Product Updates & Publishing:

- Data is transformed and published back to GCP for further use.

- Shopify API integration allows automated product updates.

- AWS Lambda triggers automate workflows.

- Tech Stack

  

    🔹 **Cloud**: Google Cloud Storage, IAM, AWS Lambda

  

    🔹 **IaC**: Terraform (Tofu), Ansible 

    🔹 **Automation**: GitHub Actions 

    🔹 **Data Processing**: Python, Pandas

    🔹 **E-commerce**: Shopify API

## Goal

🚀 Automate & streamline digital product updates for an existing online store (it's been upt for 4 years !), reducing 

manual work and improving data accuracy.

## About Us

We are a DevOps Engineer (@aglorhythm) and a Data Analyst (@azezu), entrepreneurs, working together to build efficient, automated solutions for 

our e-commerce. Passionate about cloud infrastructure, data processing, and automation, we created this project to 

streamline digital product updates for Shopify. You probably guessed, we are very busy ! Our goal is to eliminate 

manual work, improve data accuracy, and optimize workflows using modern DevOps and data engineering practices.