Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/platbr/kubernetes-health

This gem allows kubernetes monitoring your app while it is running migrates and after it starts.
https://github.com/platbr/kubernetes-health

gem health kubernetes migrates rails

Last synced: about 2 months ago
JSON representation

This gem allows kubernetes monitoring your app while it is running migrates and after it starts.

Awesome Lists containing this project

README 

        
# Kubernetes::Health

This gem opens a dedicated HTTP port to allow Kubernetes to monitor your Rails app while it is running migrations, Rake tasks, Sidekiq, or Puma.



# Features

- Puma and Sidekiq metrics for autoscaling.

- Prometheus and JSON metrics.

- add routes `/_readiness`, `/_liveness` on Rails Stack.

- add routes `/_readiness`, `/_liveness` and `/_metrics` as a puma plugin at another port to avoid problems when your app get busy. (code copied from `puma-metrics` gem).

- add routes `/_readiness` and `/_liveness` while `rake db:migrate` runs. (optional)

- add routes `/_metrics` while `sidekiq` runs. (optional)

- add support to avoid parallel running of `rake db:migrate` while keep kubernetes waiting (PostgreSQL required for default config, read `Customizing locking` section.).

- allow custom checks for `/_readiness` and `/_liveness`.

 

## Installation



Add this line to your application's Gemfile:



```ruby

gem 'kubernetes-health', '~> 3.14'

```



## Enabling puma plugin



add in `config/puma.rb`

```

plugin 'kubernetes'

kubernetes_url 'tcp://0.0.0.0:9393'

```



In Kubernetes you need to configure your deployment `readinessProbe` and `livenessProbe` like this:



```

        livenessProbe:

          httpGet:

            path: /_liveness

            port: 9393

          initialDelaySeconds: 30

          timeoutSeconds: 5

          failureThreshold: 3

          successThreshold: 1

        readinessProbe:

          httpGet:

            path: /_readiness

            port: 9393

          initialDelaySeconds: 30

          timeoutSeconds: 5

          failureThreshold: 3

          successThreshold: 1

```



Setting `failureThreshold` is import to avoid problems when app finish migrates and is starting the web process.



## Enabling liveness/readiness routes while `rake db:migrate` runs



Your Dockerfile's entry script needs to run migrates before start your web app.



Add `KUBERNETES_HEALTH_ENABLE_RACK_ON_MIGRATE=true` environment variable.



or add in your `application.rb`.



```

# default: false

Kubernetes::Health::Config.enable_rack_on_migrate = true

```

The defined port at `config/puma.rb` will be used but can be overrided by `KUBERNETES_HEALTH_METRICS_PORT` env var.



## Enabling liveness/readiness routes while any `rake` tasks runs



If you need to run another rake tasks than `db:migrate`, like `assets:precompile`, you can enable the monitoring routes by this way:



Add a `rake` file enhancing the original task by using `kubernetes_health:rack_on_rake` task. For example:



```

# File: lib/tasks/kubernetes_health_enable_rack_on_assets_precompile.rake

Rake::Task['assets:precompile'].enhance(['kubernetes_health:rack_on_rake'])

```



```

# File: lib/tasks/kubernetes_health_enable_rack_on_assets_clobber.rake

Rake::Task['assets:clobber'].enhance(['kubernetes_health:rack_on_rake'])

```



I do recommend doing some checks to make it only enabled in the K8S environment.

The defined port at `config/puma.rb` will be used but can be overrided by `KUBERNETES_HEALTH_METRICS_PORT` env var.



## Enabling liveness/readiness routes for `sidekiq`



Add `KUBERNETES_HEALTH_ENABLE_RACK_ON_SIDEKIQ=true` environment variable.



or add in your `application.rb`.



```

# default: false

Kubernetes::Health::Config.enable_rack_on_sidekiq = true

```

The defined port at `config/puma.rb` will be used but can be overrided by `KUBERNETES_HEALTH_METRICS_PORT` env var.



### How `rake` and `sidekiq` monitoring works

It will run a rack server for `/_readiness`, `/_liveness` and `/_metrics` for rake and `/_metrics` for Sidekiq.

The liveness route will respond using `200` but readiness `503`.



## Avoiding migrations running in parallel and making kubernetes happy.

Rails already avoid migrations running in parallel, but it raises exceptions. This gem will just wait for other migrations without exit.

If you enable `rack_on_migrate` together with this, kubernetes will just wait, avoiding erros.



Add `KUBERNETES_HEALTH_ENABLE_LOCK_ON_MIGRATE=true` environment variable.



or add in your `application.rb`.



```

# default: false

Kubernetes::Health::Config.enable_lock_on_migrate = true

```



### Customizing locking

By default it is working for PostgreSQL, but you can customize it using a lambda:

```

Kubernetes::Health::Config.lock_or_wait = lambda {

  ActiveRecord::Base.connection.execute "SET lock_timeout TO '3600s'; SELECT pg_advisory_lock(123456789123456789);"

}



Kubernetes::Health::Config.unlock = lambda {

    ActiveRecord::Base.connection.execute 'select pg_advisory_unlock(123456789123456789);'

}

```



## Customizing checks



It only works for routes in rails stack, they are not executed while `rake db:migrate` runs.



I prefer do nothing else on `liveness` to avoid unnecessary `CrashLoopBackOff` status. `params` is optional (request params).



```

Kubernetes::Health::Config.live_if = lambda { |params|

  true

}



```

Ex. Check if PostgreSQL is reachable on `readiness` indicating that credentials are setup right and keeps cache to avoid doing it a lot. `params` is optional (request params).

```

Kubernetes::Health::Config.ready_if = lambda { |params|

  return $kubernetes_health_test_db_connection if $kubernetes_health_test_db_connection

  $kubernetes_health_test_db_connection = ActiveRecord::Base.connection.execute("SELECT 1").cmd_tuples == 1

}

```



## Customizing routes

```

Kubernetes::Health::Config.route_liveness = '/liveness'

Kubernetes::Health::Config.route_readiness = '/readiness'

Kubernetes::Health::Config.route_metrics = '/metrics'

```

or using env

```

KUBERNETES_HEALTH_LIVENESS_ROUTE='/liveness'

KUBERNETES_HEALTH_READINESS_ROUTE='/readiness'

KUBERNETES_HEALTH_RESPONSE_FORMAT='/metrics'

```



## Response format

If you are using `https://github.com/zalando-incubator/kube-metrics-adapter` you will want to use `json` format.



Default is `prometheus`.

```

Kubernetes::Health::Config.response_format = 'json'

```

or using env

```

KUBERNETES_HEALTH_RESPONSE_FORMAT=json

```



## Customizing requests logs



```

Kubernetes::Health::Config.request_log_callback = lambda { |req, http_code, content|

  Rails.logger.debug "Kubernetes Health: Rack on Migrate - Request: Path: #{req.path_info} / Params: #{req.params} /  HTTP Code: #{http_code}\n#{content}"  rescue nil

}

```