Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/bu-ist/webrouter-nonprod

Webrouter for routing web services to backends
https://github.com/bu-ist/webrouter-nonprod

aws codepipeline

Last synced: 2 days ago
JSON representation

Webrouter for routing web services to backends

Awesome Lists containing this project

README 

        
# Over view



This repo contains the landscape specific files for the BU web router.  In general updates to this repo are

safe to release to in a change restriction period.  That is because:



- The version of the base image is date tagged to ensure that no software updates accidently sneak through.

- The config files in this repo are limited to map files

- The CodeBuild part of the release process will run nginx -t to ensure that there are no map errors prior

  to release.

- The CodePipeline will back out to the old running version if the updated version does not stabilize.



# Operational Tasks



## Changing a map entry



1. Use `git pull` to ensure that we are operating with the latest version of the configuration.

2. Use `git status` to double-check that you are on the correct landscape.  If not then do `git checkout landscape`

3. Edit landscape/LANDSCAPE/maps/sites.map to reflect our map path.

4. Use `git status` to double-check status of files.

5. Use `git diff` to see what changed.

6. Use `git add` to select files for inclusion.

7. Use `git commit -m "description" ` to commit the change with the description of the change

8. Use `git push` to push these changes out to the GitHub repo.  This will trigger the CodePipeline update.

9. Watch the CodePipeline release through the web console or cli for any issues.



## Updating to a newer base image



During initial builds and configuration we are using the tag latest.



1. Use `git pull` to ensure that we are operating with the latest version of the configuration.

2. Use `git status` to double-check status of files.

3. Edit Dockerfile and change the tag as part of the `FROM dsmk/web-router-base:date`

4. Use `git status` to double-check status of files.

5. Use `git diff` to see what changed.

6. Use `git add` to select files for inclusion.

7. Use `git commit -m "description" ` to commit the change with the description of the change

8. Use `git push` to push these changes out to the GitHub repo.  This will trigger the CodePipeline update.

9. Watch the CodePipeline release through the web console or cli for any issues.



## Adding a new landscape



1. Clone a new directory with the landscape name `git clone [email protected]:dsmk/web-router-nonprod.git test`

2. Cd into the directory and create and checkout the new branch: `cd test; git checkout -b test`

3. Create the landscape specific subdirectory: `mkdir -p landscape/test/maps`

4. Copy initial files from a previous landscape except for the sites.map which comes from landscape/common/sites-common.map

5. Edit files as appropriate.

6. For sites.map, use PHPMAP and the filesystem scan to find things.



This repo has all the data for the BU AWS web front-end proof of concept.  The architecture sets up a fairly 

standard AWS Elastic Container Service (ECS) environment with CloudFront in front of it.  The CloudFormation 

templates are in the aws/ subdirectory but eventually should be in a separate location.



This is still a work in progress and we still need to address the following issues:



- Patching model for ECS EC2 instances: ECS does not seem to handle this by default.  Is the correct approach

  to use EC2 Systems Manager?



- Method of updating route configuration: This version bakes the routing into the Docker image and relies upon

  building a new Docker image when we change the configuration.  The CodePipeline makes that more straightforward

  but a nice to have would be a simple interface to adjust the change as long as it does not add much complexity 

  to the solution.  



- Release mechanism for multiple landscapes: This POC code only handles one landscape (test).  I have seen a

  different approaches to multiple landscapes:  1) Have a single CodePipeline for multiple landscapes with a 

  manual approval step prior to production; 2) Have multiple CodePipelines like the POC based on different 

  repos and/or tags.  



- Approach for handling redirection of both entire subdirectories and singleton URLs: Redirection ideally 

  should be managed by Help Center folks.  I have seen a few references to lambda/API gateway and S3 bucket 

  based approaches.  If we use those solutions then hopefully the routing table could just have a "redirect" 

  backend that points to the solution.



- Internal load balancers for backends: Our existing front-ends handle load balancing for the WordPress and 

  Django backends (mod_proxy_balancer Apache 2.2).  One refinement would be to have internal application load

  balancers so all load balancing is handled (and monitored/tracked) by AWS load balancing services.  In 

  addition, the WordPress ALB could potentially handle selecting between Application and Asset servers which 

  would simplify this NGINX/HTTPD configuration even more. 



The general workflow while we are building the system is to:



1) Update the test/sut/test_app.sh script to test the new functionality with the current Solaris.  This involves 

   running the script like so `./test/sut/test_app.sh -S test www-test.bu.edu`  - the -S disables the check for 

   upstream headers (only configured in NGINX for now).



2) Once that is done then one needs to update the NGINX configuration to implement the same functionality as 

   Solaris.  This mainly will involve the \*.conf.erb files in the main directory but make involve new environment

   entries.  One can test this on a local build box by running `docker-compose -f www-test.bu.edu.yaml up --build` 

   and then `./test/sut/test_app.sh test`  This is testing with the normal www-test backends.



3) Next the automatic testing needs to be configured.  This involves two sets of changes - `autotest-test.yml` 

   docker-compose file and updates to the `test/backend` Docker image.  The docker-compose file will create a

   container to run the tests, an NGINX frontend container, and a test backend container which emulates test

   responses.  The goal of this test framework is to test and minimize regressions in the frontend containers.  It

   is not intended to be a test of whole services.



4) One can pretest the automatic testing by doing `docker-compose -f autotest-test.yaml up --build ` - if 

   everything is OK then the last line will end "exited with code 0."  Control-C to exit from the test and 

   it will stop all the containers set up.  The test output is stored in the `test/results/` directory and you 

   can see the specific test that failed by searching for `exit code: 1` 



5) When you are ready to release changes you need to check the changes into Git and push them to GitHub.  Here 

   are some common commands:

        a) `git status` - see which files have been modified, which are new, what is selected for commiting, and 

           the branch.

        b) `git diff [file]` - see what has changed from the last commit.

        c) `git add file` - add a file to the list of things to check in.

        d) `git commit -m 'msg'` - commit added files with the message msg (include INC* and ENC* where appropriate)

        e) `git push` - push the changes to github



6) Right now one needs to go to hub.docker.com/r/dsmk/bufe-buedu, select "Build Settings", and select the Trigger

   to start the build.  This will be replaced later.  You can see the status of the build by going to "Build Details"



7) Once the new image is done you can replace the existing container by going to the ECS web console, selecting

   the task we want to modify (bufe-bu.edu-bufe:num), select "Create new revision", and then selecting "Create".  

   This will take a while to run because ECS will create a new container, add it to the ELB/ALB, and wait for the 

   old one to drain completely before being done.  This will be replaced later.



URLs for this process:

- https://sanderknape.com/2016/06/getting-ssl-labs-rating-nginx/

- https://sanderknape.com/2017/08/custom-cloudformation-resource-example-codedeploy/?__s=1qw3d4dssvntsxck5tuk

- https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications-using-parameter-store-and-iam-roles-for-tasks/



- http://blog.xebia.com/docker-container-secrets-aws-ecs/