{"id":23437880,"url":"https://github.com/sanam2405/knowledgetransfer","last_synced_at":"2025-04-09T19:22:06.032Z","repository":{"id":175253263,"uuid":"653581434","full_name":"sanam2405/KnowledgeTransfer","owner":"sanam2405","description":"This sums up my learnings at Razorpay 🚀","archived":false,"fork":false,"pushed_at":"2024-02-12T20:56:22.000Z","size":4520,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-15T11:38:48.084Z","etag":null,"topics":["automation","aws","aws-lambda","boto3","python","sql"],"latest_commit_sha":null,"homepage":"https://sanam.live/KnowledgeTransfer","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sanam2405.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-14T10:23:08.000Z","updated_at":"2024-02-12T20:58:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"e10275cb-0fe8-45e4-b463-084dbd76b554","html_url":"https://github.com/sanam2405/KnowledgeTransfer","commit_stats":null,"previous_names":["sanamrp2405/knowledgetransfer","sanam2405/knowledgetransfer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanam2405%2FKnowledgeTransfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanam2405%2FKnowledgeTransfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanam2405%2FKnowledgeTransfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sanam2405%2FKnowledgeTransfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sanam2405","download_url":"https://codeload.github.com/sanam2405/KnowledgeTransfer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248095459,"owners_count":21046851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","aws","aws-lambda","boto3","python","sql"],"created_at":"2024-12-23T14:38:52.187Z","updated_at":"2025-04-09T19:22:06.013Z","avatar_url":"https://github.com/sanam2405.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [KnowledgeTransfer](https://github.com/sanam2405/KnowledgeTransfer)\n\n## [Manas Pratim Biswas](https://github.com/sanam2405) (SDE Intern, DevSecOps@[Razorpay](https://razorpay.com/))\n\nThis sums up my workflows and learnings during my Razorpay Internship.\n\n## Workflow Diagrams\n\n- **Cloud Automation and Remediation**\n\n    \u003cimg src = \"images/lambda_idea.png\"\u003e\n    \n    \u003cimg src = \"images/lambda_workflow.png\"\u003e\n\n\u003c!-- | Lambda Idea                           | Lambda Workflow                           |\n| --------------------------------------|-------------------------------------------|\n| \u003cimg src = \"images/lambda_idea.png\"\u003e  | \u003cimg src = \"images/lambda_workflow.png\"\u003e  | --\u003e\n\n- **Scanning all the Stages of each Pipelines of Spinnaker Applications for Security Checks**\n\n    \u003cimg src = \"images/spinnaker_visual.png\"\u003e\n\n- **Security Dashboard Migration from Looker to Superset**\n\n    \u003cimg src = \"images/looker_migration_visual.png\"\u003e\n\n## Learning\n\n- Continuous Integration / Continuous Delivery \u0026 Deployment (CI / CD)\n\n- GitHub Actions \u0026 GitHub Marketplace\n\n- Dependabot, Semgrep, Trivy, Pingsafe\n\n- Docker, Kubernetes\n\n- Looker, Superset, Querybook, Postman\n\n- Amazon Web Service\n\n    \u003cimg src = \"images/aws_mindmap.png\"\u003e\n\n  - Architecture\n\n    - AWS Cloud\n      - Region 1\n        - Availability Zone 1\n        - Availability Zone 2\n        - Availability Zone 3\n      - Region 2\n        - Availability Zone 1\n        - Availability Zone 2\n        - Availability Zone 3\n      - Region 3\n        - Availability Zone 1\n        - Availability Zone 2\n        - Availability Zone 3\n\n  - IAM\n\n    \u003e _Identity that represents a person or application that interacts with the AWS services and resources._ \u003e _The user gets access to the resources on the basis of the IAM policies which is a JSON based document._ \u003e _Collection of IAM users can be grouped where the policies can be inherited._\n\n     \u003cimg src = \"images/iam_roles.png\"\u003e\n\n    - User Role - IAM User Role defines the access to the resources and services that the user has\n\n    - Service Role - IAM Service Role defines the services that can be accessed by a particular AWS service. Service like EC2, S3, DynamoDB can be accessed by a Lambda if the\n      service role of the Lambda contains the policies to access the EC2, S3, DynamoDB services\n\n    - Configuring a Lambda function (suppose deployed in Account A) to assume a role from another AWS account (suppose in Account B)\n\n      \u003e _ARN of the **role to be assumed** in the Account B should be configured in the **execution role** of the Lambda in Account A as an **in-line policy**_\n      \u003e\n      \u003e _ARN of the **execution role** of the Lambda in Account A should be configured in the **Trust Policy** of the **role to be assumed** in the Account B_\n\n     \u003cimg src = \"images/iam_roles_visual.png\"\u003e\n\n  - AWS Compute\n\n    - Instance\n      - Classification and Specification\n        - General Purpose\n        - Compute Optimized\n        - Memory Optimized\n        - Accelerated Computing\n        - Storage Optimized\n      - Elastic Compute Cloud (EC2)\n        - Amazon Machine Image (AMI)\n    - Container\n      - Amazon Container Service (ECS)\n      - Amazon Elastic Kubernetes Service (EKS)\n    - Serverless\n      - AWS Fargate\n      - AWS Lambda\n\n    \u003cimg src = \"images/container_vs_vm.jpeg\"\u003e\n\n  - AWS Storage\n\n    - Block Storage\n      - Instance Store (Non-persistent fast storage just like RAM)\n      - Elastic Block Store (EBS)\n        - SSD (NVMe used by Amazon)\n        - HDD\n        - Deployed at only one Availability Zone\n    - File Storage\n      - Elastic File System (EFS) (For Linux)\n        - Shared storage between multiple Availability Zone\n      - FSx (For Windows)\n    - Object Storage (Object = Data + Metadata + Key)\n      - Simple Storage Service (S3 Bucket)\n        - Regional Service (Available in a particular region only)\n        - Global Namespace (Name/Link to the resource is globally available hence must be unique for each regions)\n\n    \u003e _EBS = SAN (Storage Area Network) while EFS = NAS (Network Attached Storage)_\n\n  - AWS Databases\n\n      \u003cimg src = \"images/db.png\"\u003e\n\n    - Relational Database Service (RDS) [SQL]\n      - Amazon Aurora\n      - Microsoft SQL\n      - MariaDB\n      - MySQL\n    - Amazon DynamoDB [NoSQL]\n      - Key-Value Database\n      - Serverless in nature\n\n  - AWS Networking\n\n    \u003cimg src = \"images/vpc.png\"\u003e\n\n    - Classless Inter-Domain Routing (CIDR)\n\n      \u003cimg src = \"images/vpc_cidr.png\"\u003e\n\n    - Network Acknowledgement NACL (Stateless)\n    - Security Group (Stateful)\n\n  - Monitoring, Load Balancing and Scaling\n\n    - Amazon CloudWatch (Monitoring)\n    - Elastic Load Balancing\n\n      - Application Load Balancer\n      - Network Load Balancer\n      - Gateway Load Balancer\n      - Classic Load Balancer\n\n      \u003e _Client requests hit an elastic load balancer as a single point of contact and then the requests are rerouted to multiple EC2 instances thereby balancing the traffic_\n\n  - Scaling\n\n    - Vertical Scaling\n      \u003e _Increase or Scale by increasing the metrics of the previous resource_\n    - Horizontal Scaling\n      \u003e _Increase or Scale by increasing the number of resources keeping individual metrics of the resources constant_\n    - EC2 Auto Scaling\n\n  - Misc. Tools\n    - Elastic Beanstalk\n    - Amazon Route 53\n\n- Base64 Encoding\n\n  - Purpose\n\n    \u003e _Base64 encoding is a method used to convert binary data into ASCII text format._ \u003e _It is widely used in computer systems, particularly for tasks such as data transmission,_ \u003e _file attachments in emails, and data storage._\n    \u003e\n    \u003e _The purpose of base64 encoding is to ensure that binary data can be safely transmitted or_ \u003e _stored using systems that only support text-based formats. Since many systems and protocols_ \u003e _can only handle ASCII characters, base64 encoding allows binary data to be represented as a_ \u003e _string of ASCII characters._\n    \u003e\n    \u003e _ASCII representation uses 7-bits or 128 unique characters. However, currently UTF-8 is widely_ \u003e _used and there are a number of other characters that needs to be represented and transmitted_ \u003e _base64 encoding ensures that no matter how many characters are used to represent the original_ \u003e _binary file, the information can be encoded in base64. 64 unique characters from [A-Za-z0-9+/]_ \u003e _are used for encoding_\n\n  - Implementation\n\n    \u003e 1. _The input binary data is divided into groups of three bytes._\n    \u003e 2. _Each group of three bytes is then split into four 6-bit segments._\n    \u003e 3. _Each 6-bit segment is represented as a character using a predefined set of 64 ASCII characters._ \u003e _The specific set of characters used may vary, but it typically includes uppercase letters, lowercase_ \u003e _letters, numbers, and two additional characters, often '+' and '/'._\n    \u003e 4. _If the input binary data is not divisible by three, padding characters (usually '=') are added to the_ \u003e _encoded output to ensure that the length is a multiple of four characters._\n\n    ***\n\n      \u003cimg src=\"images/base64map.png\"\u003e\n       \n      ---\n      \u003cimg src=\"images/base64.png\"\u003e\n\n    ***\n\n      \u003cimg src=\"images/base64demo.png\"\u003e\n\n  - Commands (Linux/macOS)\n    - Encoding\n    ```console\n        base64 \u003cinput_file_name\u003e \u003e \u003coutput_file_name\u003e\n    ```\n    - Decoding\n    ```console\n        base64 -d \u003cinput_file_name\u003e \u003e \u003coutput_file_name\u003e\n    ```\n    - Print the encoded string on the terminal\n    ```console\n        base64 \u003cinput_file_name\u003e\n    ```\n    - Decoding from a encoded string\n    ```console\n        echo \u003cencoded_string\u003e | base64 -d \u003e \u003coutput_file_name\u003e\n    ```\n\n- Hashing and Encryption\n\n  - Secure Hash Algorithm (SHA)\n  - Message Digest Algorithm (MD5)\n\n- Polling, Streaming and WebHooks\n\n  - Polling\n\n    \u003e _Continuously poll the server at regular intervals of time. This might_ \u003e _be inefficient at times since frequent redundant polls might occur_\n\n  - Streaming\n\n    \u003e _Establishing a continuous two-way connection with the server and_ \u003e _continuously listening to the server. Puts huge load on the server_\n\n  - WebHooks\n\n    \u003e _Communicates with the server based on the trigger for an event. Whenever_ \u003e _the event is triggered, the webhook send a response thereby reducing load_ \u003e _on the server_\n\n## Book Suggestions\n\n- Girish\n  - Concrete Mathematics - Knuth, Graham\n  - All of Stats - Wasserman\n  - Statistical Design - Casella\n  - Modern Age Statistical Inference - Efron, Hastie\n  - Elements of Statistical Learning - Hastie, Tibshirani\n  - Probabilistic ML - Kevin Murphy\n  - Deep Learning - Bengio, Goodfellow\n- Murali\n  - Auth N Capture - Aditya Kulkarni\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanam2405%2Fknowledgetransfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsanam2405%2Fknowledgetransfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsanam2405%2Fknowledgetransfer/lists"}