https://github.com/soumilshah1995/Smart-way-to-Capture-Jobs-and-Process-Meta-Data-Using-DynamoDB-Project-Demo-Python-Templates
Smart way to Capture Jobs and Process Meta Data Using DynamoDB | Project Demo | Python Templates
https://github.com/soumilshah1995/Smart-way-to-Capture-Jobs-and-Process-Meta-Data-Using-DynamoDB-Project-Demo-Python-Templates
Last synced: about 2 months ago
JSON representation
Smart way to Capture Jobs and Process Meta Data Using DynamoDB | Project Demo | Python Templates
- Host: GitHub
- URL: https://github.com/soumilshah1995/Smart-way-to-Capture-Jobs-and-Process-Meta-Data-Using-DynamoDB-Project-Demo-Python-Templates
- Owner: soumilshah1995
- License: gpl-3.0
- Created: 2022-10-16T17:09:52.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2022-10-21T19:27:47.000Z (over 2 years ago)
- Last Synced: 2024-11-13T15:49:07.001Z (7 months ago)
- Language: CSS
- Size: 3.07 MB
- Stars: 4
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- jimsghstars - soumilshah1995/Smart-way-to-Capture-Jobs-and-Process-Meta-Data-Using-DynamoDB-Project-Demo-Python-Templates - Smart way to Capture Jobs and Process Meta Data Using DynamoDB | Project Demo | Python Templates (CSS)
README
[]
# Smart way to Capture Jobs and Process Meta Data Using DynamoDB | Project Demo | Python Templates | Alerts
#### Overview:
In this article, I will present a solution that will allow you to easily monitor and capture status for running jobs and tasks. Capturing the details allows us to determine how long a process takes, what the status of the process is, and if necessary, dive into Task level details. When a job runs, it generates a unique process (GGUID), which represents the running or ongoing work. The process will have a start and end time and will display the status of ongoing activities. Each task in the process will have a name, a start and end time, and a status. If a task fails, the process status will be marked as failed. If a user needs more visibility for a function, they can simply log the function with decorator and all details will be captured in dynamodb for that task. I will demonstrate how to design and implement these solutions.
# Architecture
# Frontend
* Shows all process that ran for given Day
* Shows all Task for Given Process for given Day (Query GSI View )
### Alerts
* Sends Alerts for failed process using kinesis streams lambda and SNS
#### Video : https://www.youtube.com/watch?v=OuqhoAZwFYw
#### Read Article : https://www.linkedin.com/pulse/smart-way-capture-monitor-report-status-python-jobs-using-soumil-shah
#### change and add your email where you would like
please email address in severless.yml
Email address will recive alerts for failed process
```
MySubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: your_email.com
Protocol: email
TopicArn: !Ref 'SNSTopic'
```
#### Install and Deploy Stack
```
command 1: npm install -g serverless
command 2: serverless config credentials --provider aws --key XXXX --secret XXXXX -o
command 3: serverless deploy
```
### How to Use
```
class Jobs(object):
def __init__(self):
self.process_instance = Process()
self.__create_process()
def __create_process(self):
self.process_instance.create()
self.process_instance.progress()
def run(self):
response_1 = self.step_1()
response_2 = self.step_2()
self.process_instance.success()
@dynamodb_task()
def step_1(self):
print("some business rules and code goes here ")
print("some more business rules and code goes here ")
@dynamodb_task()
def step_2(self):
raise Exception ("OUCH")
print("some business rules and fucntion calls logs ")
```
## Explanations
* @dynamodb_task()
* Whenever you Decorate the method with dynamodb task this will log the meta data in dynamodb. this logs task start time and end time and status of task
* Status can be Success | Progress | Failed
##### Since the method raised exception it marks process Failed
##### Exceptionj we generated in code can benn seen
#### tasks
## GSI
* GS1 gives you all Task for a given Process
* GSI2 gives you all process for a guiven day and you can use SK to filter by status or any other things if needed
* GSI3 gives you all process for month
* TTL feilds will delete as the records and process get olders
-------------------------------------------------------------------------------------
## Soumil Nitin Shah
Bachelor in Electronic Engineering |
Masters in Electrical Engineering |
Master in Computer Engineering |
* Website : http://soumilshah.com/
* Github: https://github.com/soumilshah1995
* Linkedin: https://www.linkedin.com/in/shah-soumil/
* Blog: https://soumilshah1995.blogspot.com/
* Youtube : https://www.youtube.com/channel/UC_eOodxvwS_H7x2uLQa-svw?view_as=subscriber
* Facebook Page : https://www.facebook.com/soumilshah1995/
* Email : [email protected]
* projects : https://soumilshah.herokuapp.com/project
* I earned a Bachelor of Science in Electronic Engineering and a double master’s in Electrical and Computer Engineering. I have extensive expertise in developing scalable and high-performance software applications in Python. I have a YouTube channel where I teach people about Data Science, Machine learning, Elastic search, and AWS. I work as data collection and processing Team Lead at Jobtarget where I spent most of my time developing Ingestion Framework and creating microservices and scalable architecture on AWS. I have worked with a massive amount of data which includes creating data lakes (1.2T) optimizing data lakes query by creating a partition and using the right file format and compression. I have also developed and worked on a streaming application for ingesting real-time streams data via kinesis and firehose to elastic search