https://github.com/shikhar97/hybrid-cloud

https://github.com/shikhar97/hybrid-cloud

aws boto3 ceph cephfs dynamodb microceph minikube openfaas

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/shikhar97/hybrid-cloud
• Owner: Shikhar97
• License: apache-2.0
• Created: 2023-11-28T07:56:59.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2023-12-02T22:34:37.000Z (about 1 year ago)
• Last Synced: 2023-12-03T13:44:34.267Z (about 1 year ago)
• Topics: aws, boto3, ceph, cephfs, dynamodb, microceph, minikube, openfaas
• Language: Python
• Homepage:
• Size: 52.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Project-Hybrid Cloud

This project involves the integration of AWS services, OpenFaaS, Ceph, and other tools to perform cloud-based video processing and data analysis. The primary goal is to develop a system that can handle automated video analysis tasks, such as face recognition, using serverless functions and distributed storage.

## Team Members

1. **Shikhar Gupta**: Installed and configured ubuntu 22.04 virtual machine on Oracle virtualbox on a windows machine and configured Ceph on the same.

2. **Maitry Ronakbhai Trivedi**: Worked on building the docker file required for the openfaas function and deployed the same.

3. **Ayushi Agarwal**:  Worked on setting up Openfaas for deployment and configuring Openfaas on Minikube.

## Project Components

#### check.py

This script checks the correctness of the video processing workflow. It verifies the results obtained from the face recognition function by comparing them with predefined mappings. The mappings are specified in the mapping file.

#### face-recognition.yml

This YAML file defines the OpenFaaS function for face recognition. It specifies the Docker image and dependencies required for the serverless function.

#### notification.py

The notification script monitors the input bucket and triggers the face recognition function when new objects are uploaded. It uses a simple HTTP listener to invoke the OpenFaaS function.

#### student_data.json

A JSON file containing sample student data. This data is used for populating a DynamoDB table in the `upload_data.py` script.

#### upload_data.py

This script uploads student data to a DynamoDB table. It reads data from the `student_data.json` file and uses the AWS SDK for Python (Boto3) to interact with DynamoDB.

#### workload.py

The workload script generates a test case by uploading video files to the input bucket. It prepares the system for testing the video processing workflow.

#### Dockerfile

A multi-stage build for an OpenFaaS function with Python runtime. The final runtime image includes the required dependencies, such as ffmpeg, and copies the function code, dependencies, and handler script. The CMD instruction specifies the default command to execute when the container starts, pointing to the handler function for the OpenFaaS function.

#### handler.py

Python script containing the OpenFaas function logic for face recognition.

#### .env

Contain environment variable settings for AWS and Ceph configurations.

## Getting Started

1. **Setup Environment:**

   Ensure that you have the necessary environment variables set, including AWS and Ceph credentials.

2. **Install Dependencies:**

   Install the required Python packages using `pip install -r requirements.txt`.

3. **Run the Workload Script:**

   Execute `workload.py` to simulate a workload by uploading test videos to the input bucket.

4. **Triggers notification:**

   Execute `notification.py` to trigger the openfaas function.