Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/raqssoriano/hha504_assignment_storage

This task is part of my assignment focused on familiarizing myself with cloud storage services. This repository contains codes and documentation on how I learned and gained hands-on experience uploading files to Azure Blob Storage and Google Cloud Platform (GCP) Cloud Storage using both Python scripts and the platform's GUI.
https://github.com/raqssoriano/hha504_assignment_storage

Last synced: 5 days ago
JSON representation

Host: GitHub
URL: https://github.com/raqssoriano/hha504_assignment_storage
Owner: raqssoriano
License: apache-2.0
Created: 2024-10-13T17:33:52.000Z (about 1 month ago)
Default Branch: main
Last Pushed: 2024-10-16T20:42:04.000Z (29 days ago)
Last Synced: 2024-10-19T15:34:44.669Z (26 days ago)
Language: Python
Size: 32.9 MB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# **HHA504 || Working with Cloud Storage in Azure and Google Cloud Platform (GCP)**
---

- **🎯** This task is part of my assignment focused on familiarizing myself with cloud storage services. This repository contains codes and documentation on how I learned and gained hands-on experience uploading files to **`Azure Blob Storage`** and **`Google Cloud Platform (GCP) Cloud Storage`** using both **`Python`** scripts and the platform's **`GUI`**.

- **📌** Please refer to the following python scripts for uploading my files/images to GCP storage and Azure Blob Storage.
- [**gcp_storage.py**](https://github.com/raqssoriano/HHA504_assignment_storage/blob/main/gcp_storage.py) - created using **`Python`** in **`Visual Studio Code`**
- [**azure_storage_blob.py**](https://github.com/raqssoriano/HHA504_assignment_storage/blob/main/azure_storage_blob.py) - created using **`Python`** in both **`Visual Studio Code`** and **`Colab`**.

- **💡** Any feedback or suggestions are welcome to improve the content of this repository.

---

# _*Steps Taken to Complete this Assignment*_

---

# _*☞ AZURE*_
---

## Step 1: Setting up Storage Account
- Create a Storage Account
- Navigate to Storage accounts by searching in the top search bar.
- Click + Create to make a new Storage Account.
- Select a Resource Group or create a new one.
- Provide a Storage account name (it must be unique).
- Select a Region and a Performance/Replication type based on requirements.
- Click Review + create and then Create.

.
.
.
.
.
.
.

- Upload a sample file (e.g., a text file or image) to the Blob container using the Azure portal.

## Step 2: Setting Up Access Credentials
- Create a Shared Access Signature (SAS) Token
- In the Azure Portal, go to Storage Account.
- Navigate to Settings > Shared access signature.
- Set the permissions you need (in this case, Read, Write, Create).
- Set the Start and Expiry Date/Time as needed.
- Click Generate SAS and connection string.
- Copy the SAS Token and Connection String—this will be needed in the script.

.
.
.
.

## Step 3: Upload Files Using Python
- Write a Python script using [**Colab**](https://colab.research.google.com/drive/11xPrBx4gD230cUSt_bTNuJTP2iCXatl5) that uploads a file to the Blob container I created. Use the [**azure_storage_blob.py**](https://github.com/raqssoriano/HHA504_assignment_storage/blob/main/azure_storage_blob.py) library to handle the upload.

.

.

.

.

.

## Step 4: Create a fake image using Pillow and Run the Python Script
- The script created in my [**azure_blob_image.py**](https://github.com/raqssoriano/HHA504_assignment_storage/blob/main/azure_blob_image.py) will generate a fake image and upload it to my Azure Blob Storage account.

## Step 5: Explore Storage Features
- Explore and document the options for managing and securing data in Azure Blob Storage (e.g., access policies, tiers).
- _**Tiers:**_ it allows me to choose the frequency of the project depending on how often I need to access it.

.

- _**Access Policy:**_ it allows me to manage permissions on the Shared Access Signature (SAS) in my container/blob. I was able to specify permissions, such as expiration time and then connect it to a SAS token.

.

---
# _*☞ GCP*_
---

## Step 1: Upload Files Using the GUI
- Access the Google Cloud Console and create a new Cloud Storage bucket.
- Upload a similar sample file to the bucket using the GCP Console.

.
.

.

.

.

.

## Step 2: Setting Up Authentication and Download Access Keys
1. Create a Service Account
- Go to the IAM & Admin Dashboard.
- Click + Create Service Account.
- Fill in the details and click Create and Continue.

.
.
.

2. Grant Permissions to the Service Account
- Click on the newly created service account.
- Assign the role Storage Admin to give permissions to manage Google Cloud Storage.
- Click on Add Key and create a new key in JSON format.

.
.

3. Download the Key File
- Download the JSON key file and save it securely on your local machine.
- This key file will be used to authenticate your application to Google Cloud Storage.
- I ensured that I added this file to my `.gitignore` file.

.

.

.

.

## Step 3: Upload Files Using Python
- Write a Python script that uploads a file to the GCP Cloud Storage bucket you created. Use the [**gcp_storage.py**](https://github.com/raqssoriano/HHA504_assignment_storage/blob/main/gcp_storage.py) library to handle the upload.

- Initially I uploaded any files from my local machine. Then, I uploaded some images I downloaded from [**Stanford Dataset**](https://stanfordaimi.azurewebsites.net/datasets/35866158-8196-48d8-87bf-50dca81df965)

.
.

.
.

## Step 4: Write Python Code to Upload Fake Images
- Create a Python script.

.

## Step 5: Run the Python Script
- Run the Python script using [**gcp_storage.py**](https://github.com/raqssoriano/HHA504_assignment_storage/blob/main/gcp_storage.py)
- Check Google Cloud Storage bucket to see if the fake image was uploaded successfully.

## Step 6: Explore Storage Features
- Explore and document the options for managing and securing data in GCP Cloud Storage (e.g., IAM permissions, lifecycle rules).
- _**IAM permissions:**_ I was able to assigned role like "Storage Object Creator"--to create custom role with specific permissions depending on my need.

- _**Lifecycle Rules:**_ It allows me to see previous versions depending on the rules I set.