Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/afermon/piggymetrics-kubernetes
Proof of concept Microservice Architecture with Spring Boot, Spring Cloud, Docker and Kubernetes
https://github.com/afermon/piggymetrics-kubernetes
backend-services circleci circuit-breaker codecov docker eureka google-cloud-platform google-container-registry google-kubernetes-engine hystrix kubernetes microservice spring-boot spring-cloud
Last synced: about 7 hours ago
JSON representation
Proof of concept Microservice Architecture with Spring Boot, Spring Cloud, Docker and Kubernetes
- Host: GitHub
- URL: https://github.com/afermon/piggymetrics-kubernetes
- Owner: afermon
- License: mit
- Created: 2018-11-20T03:19:37.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2018-12-01T17:10:15.000Z (almost 6 years ago)
- Last Synced: 2024-03-15T20:05:29.329Z (8 months ago)
- Topics: backend-services, circleci, circuit-breaker, codecov, docker, eureka, google-cloud-platform, google-container-registry, google-kubernetes-engine, hystrix, kubernetes, microservice, spring-boot, spring-cloud
- Homepage:
- Size: 11.8 MB
- Stars: 21
- Watchers: 4
- Forks: 15
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
[](https://github.com/afermon/PiggyMetrics-Kubernetes/blob/master/LICENCE)
# Piggy Metrics (Kubernetes)
**A simple way to deal with personal finances**
This is a proof-of-concept application, which demonstrates [Microservice Architecture Pattern](http://martinfowler.com/microservices/) using Spring Boot, Spring Cloud, Docker and Kubernetes. With a pretty neat user interface, by the way.
## Functional services
PiggyMetrics was decomposed into three core microservices. All of them are independently deployable applications, organized around certain business domains.
#### Account service
[](https://circleci.com/gh/afermon/PiggyMetrics-account-service) [](https://codecov.io/gh/afermon/PiggyMetrics-account-service) [](https://github.com/afermon/PiggyMetrics-account-service/blob/master/LICENCE)
##### Repository [afermon/PiggyMetrics-account-service](https://github.com/afermon/PiggyMetrics-account-service)
Contains general user input logic and validation: incomes/expenses items, savings and account settings.
Method | Path | Description | User authenticated | Available from UI
------------- | ------------------------- | ------------- |:-------------:|:----------------:|
GET | /accounts/{account} | Get specified account data | |
GET | /accounts/current | Get current account data | × | ×
GET | /accounts/demo | Get demo account data (pre-filled incomes/expenses items, etc) | | ×
PUT | /accounts/current | Save current account data | × | ×
POST | /accounts/ | Register new account | | ×
#### Statistics service
[](https://circleci.com/gh/afermon/PiggyMetrics-statistics-service) [](https://codecov.io/gh/afermon/PiggyMetrics-statistics-service) []((https://github.com/afermon/PiggyMetrics-statistics-service/blob/master/LICENCE))
##### Repository [afermon/PiggyMetrics-statistics-service](https://github.com/afermon/PiggyMetrics-statistics-service)
Performs calculations on major statistics parameters and captures time series for each account. Datapoint contains values, normalized to base currency and time period. This data is used to track cash flow dynamics in account lifetime.
Method | Path | Description | User authenticated | Available from UI
------------- | ------------------------- | ------------- |:-------------:|:----------------:|
GET | /statistics/{account} | Get specified account statistics | |
GET | /statistics/current | Get current account statistics | × | ×
GET | /statistics/demo | Get demo account statistics | | ×
PUT | /statistics/{account} | Create or update time series datapoint for specified account | |
#### Notification service
[](https://circleci.com/gh/afermon/PiggyMetrics-notification-service) [](https://codecov.io/gh/afermon/PiggyMetrics-notification-service) []((https://github.com/afermon/PiggyMetrics-notification-service/blob/master/LICENCE))
##### Repository [afermon/PiggyMetrics-notification-service](https://github.com/afermon/PiggyMetrics-notification-service)
Stores users contact information and notification settings (like remind and backup frequency). Scheduled worker collects required information from other services and sends e-mail messages to subscribed customers.
Method | Path | Description | User authenticated | Available from UI
------------- | ------------------------- | ------------- |:-------------:|:----------------:|
GET | /notifications/settings/current | Get current account notification settings | × | ×
PUT | /notifications/settings/current | Save current account notification settings | × | ×
#### Notes
- Each microservice has it's own database, so there is no way to bypass API and access persistance data directly. [](https://circleci.com/gh/afermon/PiggyMetrics-mongodb-service) [](https://github.com/afermon/PiggyMetrics-mongodb-service/blob/master/LICENCE) Repository [afermon/PiggyMetrics-mongodb-service](https://github.com/afermon/PiggyMetrics-mongodb-service)
- In this project, I use MongoDB as a primary database for each service. It might also make sense to have a polyglot persistence architecture (сhoose the type of db that is best suited to service requirements).
- Service-to-service communication is quite simplified: microservices talking using only synchronous REST API. Common practice in a real-world systems is to use combination of interaction styles. For example, perform synchronous GET request to retrieve data and use asynchronous approach via Message broker for create/update operations in order to decouple services and buffer messages. However, this brings us to the [eventual consistency](http://martinfowler.com/articles/microservice-trade-offs.html#consistency) world.
## Infrastructure services
There's a bunch of common patterns in distributed systems, which could help us to make described core services work. [Spring cloud](http://projects.spring.io/spring-cloud/) provides powerful tools that enhance Spring Boot applications behaviour to implement those patterns. I'll cover them briefly.
### Config service
[](https://circleci.com/gh/afermon/PiggyMetrics-config-service) [](https://github.com/afermon/PiggyMetrics-config-service/blob/master/LICENCE)
##### Repository [afermon/PiggyMetrics-config-service](https://github.com/afermon/PiggyMetrics-config-service)
[Spring Cloud Config](http://cloud.spring.io/spring-cloud-config/spring-cloud-config.html) is horizontally scalable centralized configuration service for distributed systems. It uses a pluggable repository layer that currently supports local storage, Git, and Subversion.
In this project, I use `native profile`, which simply loads config files from the local classpath. You can see `shared` directory in [Config service resources](https://github.com/afermon/PiggyMetrics-config-service/tree/master/src/main/resources). Now, when Notification-service requests it's configuration, Config service responses with `shared/notification-service.yml` and `shared/application.yml` (which is shared between all client applications).
##### Client side usage
Just build Spring Boot application with `spring-cloud-starter-config` dependency, autoconfiguration will do the rest.
Now you don't need any embedded properties in your application. Just provide `bootstrap.yml` with application name and Config service url:
```yml
spring:
application:
name: notification-service
cloud:
config:
uri: http://config:8888
fail-fast: true
```
##### With Spring Cloud Config, you can change app configuration dynamically.
For example, [EmailService bean](https://github.com/afermon/PiggyMetrics-notification-service/blob/master/src/main/java/com/piggymetrics/notification/service/EmailServiceImpl.java) was annotated with `@RefreshScope`. That means, you can change e-mail text and subject without rebuild and restart Notification service application.
First, change required properties in Config server. Then, perform refresh request to Notification service:
`curl -H "Authorization: Bearer #token#" -XPOST http://127.0.0.1:8000/notifications/refresh`
Also, you could use Repository [webhooks to automate this process](http://cloud.spring.io/spring-cloud-config/spring-cloud-config.html#_push_notifications_and_spring_cloud_bus)
##### Notes
- There are some limitations for dynamic refresh though. `@RefreshScope` doesn't work with `@Configuration` classes and doesn't affect `@Scheduled` methods
- `fail-fast` property means that Spring Boot application will fail startup immediately, if it cannot connect to the Config Service.
- There are significant [security notes](https://github.com/afermon/PiggyMetrics-Kubernetes#security) below
### Auth service
[](https://circleci.com/gh/afermon/PiggyMetrics-auth-service) [](https://github.com/afermon/PiggyMetrics-auth-service/blob/master/LICENCE)
##### Repository [afermon/PiggyMetrics-auth-service](https://github.com/afermon/PiggyMetrics-auth-service)
Authorization responsibilities are completely extracted to separate server, which grants [OAuth2 tokens](https://tools.ietf.org/html/rfc6749) for the backend resource services. Auth Server is used for user authorization as well as for secure machine-to-machine communication inside a perimeter.
In this project, I use [`Password credentials`](https://tools.ietf.org/html/rfc6749#section-4.3) grant type for users authorization (since it's used only by native PiggyMetrics UI) and [`Client Credentials`](https://tools.ietf.org/html/rfc6749#section-4.4) grant for microservices authorization.
Spring Cloud Security provides convenient annotations and autoconfiguration to make this really easy to implement from both server and client side. You can learn more about it in [documentation](http://cloud.spring.io/spring-cloud-security/spring-cloud-security.html) and check configuration details in [Auth Server code](https://github.com/afermon/PiggyMetrics-auth-service/tree/master/src/main/java/com/piggymetrics/auth).
From the client side, everything works exactly the same as with traditional session-based authorization. You can retrieve `Principal` object from request, check user's roles and other stuff with expression-based access control and `@PreAuthorize` annotation.
Each client in PiggyMetrics (account-service, statistics-service, notification-service and browser) has a scope: `server` for backend services, and `ui` - for the browser. So we can also protect controllers from external access, for example:
``` java
@PreAuthorize("#oauth2.hasScope('server')")
@RequestMapping(value = "accounts/{name}", method = RequestMethod.GET)
public List getStatisticsByAccountName(@PathVariable String name) {
return statisticsService.findByAccountName(name);
}
```
### API Gateway
[](https://circleci.com/gh/afermon/PiggyMetrics-gateway-service) [](https://github.com/afermon/PiggyMetrics-gateway-service/blob/master/LICENCE)
##### Repository [afermon/PiggyMetrics-gateway-service](https://github.com/afermon/PiggyMetrics-gateway-service)
As you can see, there are three core services, which expose external API to client. In a real-world systems, this number can grow very quickly as well as whole system complexity. Actually, hundreds of services might be involved in rendering of one complex webpage.
In theory, a client could make requests to each of the microservices directly. But obviously, there are challenges and limitations with this option, like necessity to know all endpoints addresses, perform http request for each peace of information separately, merge the result on a client side. Another problem is non web-friendly protocols which might be used on the backend.
Usually a much better approach is to use API Gateway. It is a single entry point into the system, used to handle requests by routing them to the appropriate backend service or by invoking multiple backend services and [aggregating the results](http://techblog.netflix.com/2013/01/optimizing-netflix-api.html). Also, it can be used for authentication, insights, stress and canary testing, service migration, static response handling, active traffic management.
Netflix opensourced [such an edge service](http://techblog.netflix.com/2013/06/announcing-zuul-edge-service-in-cloud.html), and now with Spring Cloud we can enable it with one `@EnableZuulProxy` annotation. In this project, I use Zuul to store static content (ui application) and to route requests to appropriate microservices. Here's a simple prefix-based routing configuration for Notification service:
```yml
zuul:
routes:
notification-service:
path: /notifications/**
serviceId: notification-service
stripPrefix: false
```
That means all requests starting with `/notifications` will be routed to Notification service. There is no hardcoded address, as you can see. Zuul uses [Service discovery](https://github.com/afermon/PiggyMetrics-Kubernetes/blob/master/README.md#service-discovery) mechanism to locate Notification service instances and also [Circuit Breaker and Load Balancer](https://github.com/afermon/PiggyMetrics-Kubernetes/blob/master/README.md#http-client-load-balancer-and-circuit-breaker), described below.
### Service discovery
[](https://circleci.com/gh/afermon/PiggyMetrics-registry-service) [](https://github.com/afermon/PiggyMetrics-registry-service/blob/master/LICENCE)
##### Repository [afermon/PiggyMetrics-registry-service](https://github.com/afermon/PiggyMetrics-registry-service)
Another commonly known architecture pattern is Service discovery. It allows automatic detection of network locations for service instances, which could have dynamically assigned addresses because of auto-scaling, failures and upgrades.
The key part of Service discovery is Registry. I use Netflix Eureka in this project. Eureka is a good example of the client-side discovery pattern, when client is responsible for determining locations of available service instances (using Registry server) and load balancing requests across them.
With Spring Boot, you can easily build Eureka Registry with `spring-cloud-starter-eureka-server` dependency, `@EnableEurekaServer` annotation and simple configuration properties.
Client support enabled with `@EnableDiscoveryClient` annotation an `bootstrap.yml` with application name:
``` yml
spring:
application:
name: notification-service
```
Now, on application startup, it will register with Eureka Server and provide meta-data, such as host and port, health indicator URL, home page etc. Eureka receives heartbeat messages from each instance belonging to a service. If the heartbeat fails over a configurable timetable, the instance will be removed from the registry.
Also, Eureka provides a simple interface, where you can track running services and a number of available instances: `http://localhost:8761`
### Load balancer, Circuit breaker and Http client
Netflix OSS provides another great set of tools.
#### Ribbon
Ribbon is a client side load balancer which gives you a lot of control over the behaviour of HTTP and TCP clients. Compared to a traditional load balancer, there is no need in additional hop for every over-the-wire invocation - you can contact desired service directly.
Out of the box, it natively integrates with Spring Cloud and Service Discovery. [Eureka Client](https://github.com/sqshq/PiggyMetrics#service-discovery) provides a dynamic list of available servers so Ribbon could balance between them.
#### Hystrix
Hystrix is the implementation of [Circuit Breaker pattern](http://martinfowler.com/bliki/CircuitBreaker.html), which gives a control over latency and failure from dependencies accessed over the network. The main idea is to stop cascading failures in a distributed environment with a large number of microservices. That helps to fail fast and recover as soon as possible - important aspects of fault-tolerant systems that self-heal.
Besides circuit breaker control, with Hystrix you can add a fallback method that will be called to obtain a default value in case the main command fails.
Moreover, Hystrix generates metrics on execution outcomes and latency for each command, that we can use to [monitor system behavior](https://github.com/sqshq/PiggyMetrics#monitor-dashboard).
#### Feign
Feign is a declarative Http client, which seamlessly integrates with Ribbon and Hystrix. Actually, with one `spring-cloud-starter-feign` dependency and `@EnableFeignClients` annotation you have a full set of Load balancer, Circuit breaker and Http client with sensible ready-to-go default configuration.
Here is an example from Account Service:
``` java
@FeignClient(name = "statistics-service")
public interface StatisticsServiceClient {
@RequestMapping(method = RequestMethod.PUT, value = "/statistics/{accountName}", consumes = MediaType.APPLICATION_JSON_UTF8_VALUE)
void updateStatistics(@PathVariable("accountName") String accountName, Account account);
}
```
- Everything you need is just an interface
- You can share `@RequestMapping` part between Spring MVC controller and Feign methods
- Above example specifies just desired service id - `statistics-service`, thanks to autodiscovery through Eureka (but obviously you can access any resource with a specific url)
### Monitor dashboard
[](https://circleci.com/gh/afermon/PiggyMetrics-monitoring-service) [](https://github.com/afermon/PiggyMetrics-monitoring-service/blob/master/LICENCE)
##### Repository [afermon/PiggyMetrics-registry-service](https://github.com/afermon/PiggyMetrics-registry-service)
[](https://circleci.com/gh/afermon/PiggyMetrics-turbine-stream-service) [](https://github.com/afermon/PiggyMetrics-turbine-stream-service/blob/master/LICENCE)
##### Repository [afermon/PiggyMetrics-turbine-stream-service](https://github.com/afermon/PiggyMetrics-turbine-stream-service)
In this project configuration, each microservice with Hystrix on board pushes metrics to Turbine via Spring Cloud Bus (with AMQP broker). The Monitoring project is just a small Spring boot application with [Turbine](https://github.com/Netflix/Turbine) and [Hystrix Dashboard](https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard).
See below [how to get it up and running](https://github.com/afermon/PiggyMetrics-Kubernetes#how-to-run-all-the-things).
Let's see our system behavior under load: Account service calls Statistics service and it responses with a vary imitation delay. Response timeout threshold is set to 1 second.
 |  |  | 
--- |--- |--- |--- |
| `0 ms delay` | `500 ms delay` | `800 ms delay` | `1100 ms delay`
| Well behaving system. The throughput is about 22 requests/second. Small number of active threads in Statistics service. The median service time is about 50 ms. | The number of active threads is growing. We can see purple number of thread-pool rejections and therefore about 30-40% of errors, but circuit is still closed. | Half-open state: the ratio of failed commands is more than 50%, the circuit breaker kicks in. After sleep window amount of time, the next request is let through. | 100 percent of the requests fail. The circuit is now permanently open. Retry after sleep time won't close circuit again, because the single request is too slow.
### Log analysis
Centralized logging can be very useful when attempting to identify problems in a distributed environment. Elasticsearch, Logstash and Kibana stack lets you search and analyze your logs, utilization and network activity data with ease.
Ready-to-go Docker configuration described [in sqshq other project](http://github.com/sqshq/ELK-docker).
### Distributed tracing
Analyzing problems in distributed systems can be difficult, for example, tracing requests that propagate from one microservice to another. It can be quite a challenge to try to find out how a request travels through the system, especially if you don't have any insight into the implementation of a microservice. Even when there is logging, it is hard to tell which action correlates to a single request.
[Spring Cloud Sleuth](https://cloud.spring.io/spring-cloud-sleuth/) solves this problem by providing support for distributed tracing. It adds two types of IDs to the logging: traceId and spanId. The spanId represents a basic unit of work, for example sending an HTTP request. The traceId contains a set of spans forming a tree-like structure. For example, with a distributed big-data store, a trace might be formed by a PUT request. Using traceId and spanId for each operation we know when and where our application is as it processes a request, making reading our logs much easier.
The logs are as follows, notice the `[appname,traceId,spanId,exportable]` entries from the Slf4J MDC:
```text
2018-07-26 23:13:49.381 WARN [gateway,3216d0de1384bb4f,3216d0de1384bb4f,false] 2999 --- [nio-4000-exec-1] o.s.c.n.z.f.r.s.AbstractRibbonCommand : The Hystrix timeout of 20000ms for the command account-service is set lower than the combination of the Ribbon read and connect timeout, 80000ms.
2018-07-26 23:13:49.562 INFO [account-service,3216d0de1384bb4f,404ff09c5cf91d2e,false] 3079 --- [nio-6000-exec-1] c.p.account.service.AccountServiceImpl : new account has been created: test
```
- *`appname`*: The name of the application that logged the span from the property `spring.application.name`
- *`traceId`*: This is an ID that is assigned to a single request, job, or action
- *`spanId`*: The ID of a specific operation that took place
- *`exportable`*: Whether the log should be exported to [Zipkin](https://zipkin.io/)
## Security
An advanced security configuration is beyond the scope of this proof-of-concept project. For a more realistic simulation of a real system, consider to use https, JCE keystore to encrypt Microservices passwords and Config server properties content (see [documentation](http://cloud.spring.io/spring-cloud-config/spring-cloud-config.html#_security) for details).
## Infrastructure automation
Deploying microservices, with their interdependence, is much more complex process than deploying monolithic application. It is important to have fully automated infrastructure. We can achieve following benefits with Continuous Delivery approach:
- The ability to release software anytime
- Any build could end up being a release
- Build artifacts once - deploy as needed
Here is a simple Continuous Delivery workflow, implemented in this project:
In this [configuration](https://github.com/sqshq/PiggyMetrics/blob/master/.travis.yml), Travis CI builds tagged images for each successful git push. So, there are always `latest` image for each microservice on [Docker Hub](https://hub.docker.com/r/sqshq/) and older images, tagged with git commit hash. It's easy to deploy any of them and quickly rollback, if needed.
## How to run all the things?
### Kubernetes
#### Local
Windows or Mac you can get [Docker for Windows](https://www.docker.com/products/docker-desktop) and enable Kubernetes. This application runnign on your local computer may comsume a lot of resources, please cupdate docker for desktop settings to have at least `6 Gb` of RAM. Also, you can install the [Kubernetes WEB UI](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) and it will be easier to confirm the pod status and check the logs for any failures.
- `cd .\kubernetes`
- Execute `kubectl create -f .\` for each file in order and wait for each individual service to be runing correctly. config, registry and gateway should be available first for other services to start correctly.
- Gateway will be accesible at [http://localhost:30080](http://localhost:30080)
##### Access the other services:
To access any of the other services internal to the kubernetes cluster you can use the proxy `kubectl proxy` and then use the links below
| Service | Comments | URL |
|--------------------------|-----------------------|--------------------------|
| Gateway | |http://localhost:30080|
| Eureka Dashboard | |http://localhost:8001/api/v1/namespaces/default/services/http:registry:8761/proxy/ |
| Hystrix Dashboard | Turbine stream link: http://turbine-stream-service:8080/turbine/turbine.stream | http://localhost:8001/api/v1/namespaces/default/services/http:monitoring:9000/proxy/hystrix/monitor?stream=http%3A%2F%2Fturbine-stream-service%3A8080%2Fturbine%2Fturbine.stream&title=PiggyMetrics |
| RabbitMq management | (default login/password: guest/guest) | http://localhost:8001/api/v1/namespaces/default/services/http:rabbitmq:15672/proxy/ |
If you would like to delete all the resources and start over, you can execute `kubectl delete svc,deploy -l project=piggymetrics`
#### Google Kubernetes Engine
* TODO
### Docker
Keep in mind, that you are going to start 8 Spring Boot applications, 4 MongoDB instances and RabbitMq. Make sure you have `4 Gb` RAM available on your machine. You can always run just vital services though: Gateway, Registry, Config, Auth Service and Account Service.
#### Before you start
- Install Docker and Docker Compose.
- Export environment variables: `CONFIG_SERVICE_PASSWORD`, `NOTIFICATION_SERVICE_PASSWORD`, `STATISTICS_SERVICE_PASSWORD`, `ACCOUNT_SERVICE_PASSWORD`, `MONGODB_PASSWORD` (make sure they were exported: `printenv`)
- Make sure to build the project: `mvn package [-DskipTests]`
#### Production mode
In this mode, all latest images will be pulled from Docker Hub.
Just copy `docker-compose.yml` and hit `docker-compose up`
#### Development mode
If you'd like to build images yourself (with some changes in the code, for example), you have to clone all repository and build artifacts with maven. Then, build each individual image and update the docker conposer file with each individual image name.
#### Important endpoints
- http://localhost:80 - Gateway
- http://localhost:8761 - Eureka Dashboard
- http://localhost:9000/hystrix - Hystrix Dashboard (Turbine stream link: `http://turbine-stream-service:8080/turbine/turbine.stream`)
- http://localhost:15672 - RabbitMq management (default login/password: guest/guest)
#### Notes
All Spring Boot applications require already running [Config Server](https://github.com/afemron/PiggyMetrics-Kubernetes#config-service) for startup. But we can start all containers simultaneously because of `depends_on` docker-compose option.
Also, Service Discovery mechanism needs some time after all applications startup. Any service is not available for discovery by clients until the instance, the Eureka server and the client all have the same metadata in their local cache, so it could take 3 heartbeats. Default heartbeat period is 30 seconds.
## Todo
* Kubernetes Config Maps
* Kubernetes Service Discovery
## Credits
* Forked from [sqshq/PiggyMetrics](https://github.com/sqshq/PiggyMetrics)
## Contributions welcome!
PiggyMetrics is open source, and would greatly appreciate your help. Feel free to suggest and implement improvements.