Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/roeeelnekave/crash-api-application-part-2

Deploying a Crash Management API with Ansible and Cloudformation and Automation Using Jenkins ECS with Jenkins Ec2 Agent and monitoring through Grafana, Promethues
https://github.com/roeeelnekave/crash-api-application-part-2

ansible cloudformation flask gunicorn jenkins nginx

Last synced: 15 days ago
JSON representation

Deploying a Crash Management API with Ansible and Cloudformation and Automation Using Jenkins ECS with Jenkins Ec2 Agent and monitoring through Grafana, Promethues

Host: GitHub
URL: https://github.com/roeeelnekave/crash-api-application-part-2
Owner: roeeelnekave
Created: 2024-08-29T15:45:02.000Z (6 months ago)
Default Branch: main
Last Pushed: 2024-10-02T05:30:21.000Z (4 months ago)
Last Synced: 2024-11-28T19:55:04.576Z (2 months ago)
Topics: ansible, cloudformation, flask, gunicorn, jenkins, nginx
Language: Shell
Homepage:
Size: 164 KB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# How to deploy this project using CloudFormation

## Prerequisites

1. AWS Account
2. Basic Understanding of AWS services (VPC, EFS, ECS, IAM)
3. Docker and Docker Compose
4. Basic understanding of the CloudFormation template
5. AWS CLI

## STEPS

# Directory

```bash
mkdir -p ./jenkins
```

- **We need a Dockerfile to customize our Jenkins image. Create a Dockerfile:**
- Create a folder for your project and inside that folder create a file named `./jenkins/Dockerfile` and paste the following into the Dockerfile:

```Dockerfile
FROM amazonlinux:2023
RUN yum install -y \
python3 \
python3-pip \
git \
zip \
unzip \
tar \
gzip \
wget \
jq \
openssh-server \
openssh-clients \
which \
findutils \
python3-pip && \
python3 -m pip install awscli && \
python3 -m pip install boto3 && \
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo && \
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key && \
yum upgrade -y && \
yum install -y fontconfig && \
dnf install java-17-amazon-corretto -y && \
yum install -y jenkins && \
python3 -m pip install ansible && \
yum clean all
EXPOSE 8080
CMD ["java", "-jar", "/usr/share/java/jenkins.war"]
```

- **Create a CloudFormation file for deploying our infrastructure:**
- Create a file named `./jenkins/main.yaml` and paste the following into the YAML template to create a deploy the infrastructure this will create VPC, IGW, ECS, IAM:

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudFormation template for VPC with 3 public and 3 private subnets'
Parameters:
ImageURL:
Type: String
Description: 'Image URL for the ECR repo'

Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
EnableDnsHostnames: true
EnableDnsSupport: true
InstanceTenancy: default
Tags:
- Key: Name
Value: project-vpc

InternetGateway:
Type: AWS::EC2::InternetGateway

InternetGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref VPC
InternetGatewayId: !Ref InternetGateway

PublicSubnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 0, !GetAZs '' ]
CidrBlock: 10.0.1.0/24
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: project-subnet-public1-us-east-1a

PublicSubnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 1, !GetAZs '' ]
CidrBlock: 10.0.2.0/24
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: project-subnet-public2-us-east-1b

PublicSubnet3:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 2, !GetAZs '' ]
CidrBlock: 10.0.3.0/24
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: project-subnet-public3-us-east-1c

PrivateSubnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 0, !GetAZs '' ]
CidrBlock: 10.0.4.0/24
MapPublicIpOnLaunch: false
Tags:
- Key: Name
Value: project-subnet-private1-us-east-1a

PrivateSubnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 1, !GetAZs '' ]
CidrBlock: 10.0.5.0/24
MapPublicIpOnLaunch: false
Tags:
- Key: Name
Value: project-subnet-private2-us-east-1b

PrivateSubnet3:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 2, !GetAZs '' ]
CidrBlock: 10.0.6.0/24
MapPublicIpOnLaunch: false
Tags:
- Key: Name
Value: project-subnet-private3-us-east-1c

PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: project-rtb-public

DefaultPublicRoute:
Type: AWS::EC2::Route
DependsOn: InternetGatewayAttachment
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway

PublicSubnet1RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet1

PublicSubnet2RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet2

PublicSubnet3RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet3

NATGateway1:
Type: AWS::EC2::NatGateway
Properties:
AllocationId: !GetAtt NATGateway1EIP.AllocationId
SubnetId: !Ref PublicSubnet1

NATGateway1EIP:
Type: AWS::EC2::EIP
DependsOn: InternetGatewayAttachment
Properties:
Domain: vpc

PrivateRouteTable1:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: project-rtb-private1-us-east-1a

DefaultPrivateRoute1:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTable1
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NATGateway1

PrivateSubnet1RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PrivateRouteTable1
SubnetId: !Ref PrivateSubnet1

PrivateRouteTable2:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: project-rtb-private2-us-east-1b

DefaultPrivateRoute2:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTable2
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NATGateway1

PrivateSubnet2RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PrivateRouteTable2
SubnetId: !Ref PrivateSubnet2

PrivateRouteTable3:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: project-rtb-private3-us-east-1c

DefaultPrivateRoute3:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PrivateRouteTable3
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref NATGateway1

PrivateSubnet3RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PrivateRouteTable3
SubnetId: !Ref PrivateSubnet3
ECSSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: "ECS Security Group"
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 8080
ToPort: 8080
CidrIp: 0.0.0.0/0
SecurityGroupEgress:
- IpProtocol: "-1"
CidrIp: 0.0.0.0/0
EFSSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: "EFS Security Group"
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 2049
ToPort: 2049
SourceSecurityGroupId: !Ref ECSSecurityGroup
SecurityGroupEgress:
- IpProtocol: "-1"
CidrIp: 0.0.0.0/0

# Policy for ECS Task
ECSTaskPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: !Sub "ECSExecutionTaskPolicy-${AWS::StackName}"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- ecr:GetAuthorizationToken
- ecr:BatchCheckLayerAvailability
- ecr:GetDownloadUrlForLayer
- ecr:BatchGetImage
- logs:CreateLogStream
- logs:PutLogEvents
Resource: "*"
Roles:
- !Ref ECSEFSmountTaskRole

# Policy for EFS Mount
EFSMountPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: !Sub "ECSMountPolicy-${AWS::StackName}"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: AllowDescribe
Effect: Allow
Action:
- elasticfilesystem:DescribeAccessPoints
- elasticfilesystem:DescribeFileSystems
- elasticfilesystem:DescribeMountTargets
- ec2:DescribeAvailabilityZones
Resource: "*"
- Sid: AllowCreateAccessPoint
Effect: Allow
Action:
- elasticfilesystem:*
Resource: "*"
Roles:
- !Ref ECSEFSmountTaskRole

# IAM Role for ECS Tasks
ECSEFSmountTaskRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: !Sub "ECSTaskPolicy-${AWS::StackName}"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- ecr:GetAuthorizationToken
- ecr:BatchCheckLayerAvailability
- ecr:GetDownloadUrlForLayer
- ecr:BatchGetImage
- logs:CreateLogStream
- logs:PutLogEvents
Resource: "*"
EFSSystem:
Type: AWS::EFS::FileSystem
Properties:
Encrypted: true
FileSystemTags:
- Key: Name
Value: JenkinsEFS
JenkinsHomeVolume1:
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref EFSSystem
SubnetId: !Ref PublicSubnet1
SecurityGroups:
- !Ref EFSSecurityGroup
JenkinsHomeVolume2:
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref EFSSystem
SubnetId: !Ref PublicSubnet2
SecurityGroups:
- !Ref EFSSecurityGroup
JenkinsHomeVolume3:
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref EFSSystem
SubnetId: !Ref PublicSubnet3
SecurityGroups:
- !Ref EFSSecurityGroup
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: JenkinsCluster
CapacityProviders:
- FARGATE
- FARGATE_SPOT
DefaultCapacityProviderStrategy:
- CapacityProvider: FARGATE
Weight: 1
- CapacityProvider: FARGATE_SPOT
Weight: 1
Configuration:
ExecuteCommandConfiguration:
Logging: DEFAULT
ECSLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub "/ecs/test-${AWS::StackName}"
RetentionInDays: 7

ECSTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ExecutionRoleArn: !Ref ECSEFSmountTaskRole
TaskRoleArn: !Ref ECSEFSmountTaskRole
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
RuntimePlatform:
OperatingSystemFamily: LINUX
CpuArchitecture: ARM64
Family: my-jenkins-task-00
Cpu: "1024"
Memory: "2048"
ContainerDefinitions:
- Name: jenkins
Image: !Ref ImageURL
Cpu: 1024
Memory: 2048
MemoryReservation: 1024
Essential: true
PortMappings:
- ContainerPort: 8080
Protocol: tcp
LinuxParameters:
InitProcessEnabled: true
MountPoints:
- SourceVolume: efs-volume
ContainerPath: /root/.jenkins
LogConfiguration:
LogDriver: awslogs
Options:
mode: non-blocking
max-buffer-size: 25m
awslogs-group: !Ref ECSLogGroup
awslogs-region: us-east-1
awslogs-create-group: "true"
awslogs-stream-prefix: efs-task
Volumes:
- Name: efs-volume
EFSVolumeConfiguration:
FilesystemId: !Ref EFSSystem
RootDirectory: /
TransitEncryption: ENABLED

ECSService:
Type: AWS::ECS::Service
Properties:
Cluster: !Ref ECSCluster
TaskDefinition: !Ref ECSTaskDefinition
LaunchType: FARGATE
ServiceName: ebs
SchedulingStrategy: REPLICA
DesiredCount: 1
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !Ref ECSSecurityGroup
Subnets:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
- !Ref PublicSubnet3
PlatformVersion: LATEST
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentCircuitBreaker:
Enable: true
Rollback: true
DeploymentController:
Type: ECS
Tags: []
EnableECSManagedTags: true

Outputs:
VPCID:
Description: The ID of the created VPC
Value: !Ref VPC

PublicSubnet1ID:
Description: The ID of Public Subnet 1
Value: !Ref PublicSubnet1

PublicSubnet2ID:
Description: The ID of Public Subnet 2
Value: !Ref PublicSubnet2
```
**Create the bash script and update the Repository Name, aws region and stack name if you desire and `./jenkins/bash-script.sh` required**

```bash
#!/bin/bash

# update the stack name
STACK_NAME="jenkins-efs-ecs-1"
# update to your desired aws region
AWS_REGION="us-east-1"

# Set or update the repository name
REPOSITORY_NAME="jenkins"

# Set the image tag
IMAGE_TAG="latest"

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output --region $AWS_REGION)
# Create the ECR repository
aws ecr describe-repositories --repository-names "${REPOSITORY_NAME}" --region $AWS_REGION > /dev/null 2>&1
if [ $? -ne 0 ]
then
aws ecr create-repository --repository-name "${REPOSITORY_NAME}" --region $AWS_REGION > /dev/null
fi

# Build the Docker image
docker build -t $REPOSITORY_NAME:$IMAGE_TAG .

# Get the ECR login command
LOGIN_COMMAND=$(aws ecr get-login-password | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com)
# Push the image to ECR
docker tag $REPOSITORY_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$REPOSITORY_NAME:$IMAGE_TAG
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$REPOSITORY_NAME:$IMAGE_TAG
IMAGE_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${REPOSITORY_NAME}:${IMAGE_TAG}"

aws cloudformation update-stack \
--stack-name "${STACK_NAME}" \
--template-body file://main.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameters ParameterKey=ImageURL,ParameterValue="${IMAGE_URI}" \
--region "${AWS_REGION}"
```
- Go to the AWS Console Dashboard navigate to Cloudformation and click on `jenkins-ecs-efs` see the creation process after stack creation is complete go to ecs click on service and click on task access jenkins with the public ip:8080
# Deploying a Crash Management API with Ansible and Cloudformation and Automation Using Jenkins ECS with Jenkins Ec2 Agent and monitoring through Grafana, Promethues
### Prerequities

1. Python3 and python-venv
2. Ansible
3. Docker
4. aws-cli

**Create a Github repository make it public clone to pc and open the project and To Create directory structure copy and paste the following in your terminal of your project**
```bash
mkdir -p ./cloudformation
mkdir -p ./templates
mkdir -p ./ansible
mkdir -p ./ansible/roles
cd ./ansible/roles
ansible-galaxy init grafana
ansible-galaxy init prometheus
ansible-galaxy init crashapi
```

#### Setup the infrastructure
**Create an Elastic IP**
1. **Step 1**: Log in to the AWS Management Console
- Open your web browser and go to AWS Management Console.
- Enter your AWS credentials to log in.
2. **Step 2**: Navigate to the EC2 Dashboard
- Once logged in, find the Services menu at the top of the page and click on it.
- In the search bar, type EC2 and select EC2 under Compute from the dropdown list.
- You will be directed to the EC2 Dashboard.
3. **Step 3**: Allocate a New Elastic IP
- In the EC2 Dashboard, look for the Network & Security section in the left-hand sidebar.
- Click on Elastic IPs.
- You will see a page that lists all your current Elastic IPs (if any). Click on the **Allocate -Elastic IP** address button at the top-right corner of the page.
4. **Step 4**: Leave everything to default click on **Allocate**.
5. **Step 5**: Click the ip you have just created copy the ip address and allocation id the allocation starts with `eipalloc-XXXXXXXXXXXX`

#### Repeat the process for crash api server to create elastic ip

- create a `cloudformation` folder in your root directory of your project `mkdir cloudformation` inside cloudformation folder.
- To create a `main.yaml` inside the cloudformation `touch ./cloudformation/main.yaml`
- Inside the main.yaml paste the following make sure to replace elastic ip allocation id in cloudformation template
```yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudFormation template for VPC with 3 public and 3 private subnets'
Parameters:
ImageURL:
Type: String
Description: 'Image URL for the ECR repo'