https://github.com/lsst-it/foreman_envsync

Sync pupperserver envs with foreman
https://github.com/lsst-it/foreman_envsync
foreman foreman-proxy hammer puppet puppetserver ruby
Last synced: 25 days ago
JSON representation
Sync pupperserver envs with foreman
Host: GitHub
URL: https://github.com/lsst-it/foreman_envsync
Owner: lsst-it
License: apache-2.0
Created: 2021-06-04T17:34:51.000Z (almost 4 years ago)
Default Branch: master
Last Pushed: 2024-04-30T15:28:48.000Z (12 months ago)
Last Synced: 2025-03-15T13:34:40.237Z (about 1 month ago)
Topics: foreman, foreman-proxy, hammer, puppet, puppetserver, ruby
Language: Ruby
Homepage:
Size: 53.7 KB
Stars: 2
Watchers: 5
Forks: 1
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project

README

        # foreman_envsync

A highly opinionated utility to sync foreman puppet environments with the set

of environments known to a `puppetserver` instance *without* also importing the

classes within those puppet environments.

## OCI Image

An OCI image containing this gem is published to docker hub as

[`ghcr.io/lsst-it/foreman_envsync`](ghcr.io/lsst-it/foreman_envsync).

## The Problem

When using [`r10k`](https://github.com/puppetlabs/r10k) with the [Dynamic

Environments](https://github.com/puppetlabs/r10k/blob/main/doc/dynamic-environments.mkd)

pattern in concert with foreman as an ENC, it is highly desirable for new

environments to automatically appear within foreman.  Foreman has a built in

"import environments" feature which triggers a poll of a `puppetserver`'s known

environments.  Foreman's `hammer` CLI has the ability to trigger an import of

all environments (and classes) with the `proxy import-classes` sub-sub-command.

E.g.:

```bash

/bin/hammer proxy import-classes --id=1

```

`r10k` provides a hook to run a shell commands after completing an environment

build, which can be used to invoke `hammer`. E.g.:

```yaml

---

:postrun: ["/bin/hammer", "proxy", "import-classes", "--id=1"]

```

This automation setup is known to work reasonable well on a VM hosting both

`foreman 2.4.0` + `puppetserver 6.15.3` provisioned with 16 x64 cores / 32GiB

RAM / SSD storage under the following conditions:

* The `puppetserver` environment class cache is **enabled**.

* The number of puppet environments is relatively small. I.e. `<= 10`

* [Probably?] The environments are of moderate complexity. I.e. `< ~100` modules

* `r10k` is triggering an import via `hammer` 10s of times per day.

However, it has been observed that all foreman puma workers (regardless of the

number configured) will over time creep up to consume 100% of a core,

interactivity via the www console is abysmal, and environment import may fail

completely when:

* The `puppetserver` environment class cache is **disabled**.

* The puppet environments have `> 100` modules.

* `r10k` is triggering an import via `hammer` 10s of times per day.

* There are `> 10` puppet environments.

Distressingly, when the puppetserver environment cache is **disabled**, even a

small number of puppet agent hosts (~10) will trigger foreman to consume all

available CPU cores.  Reducing the number of puppet environments to `< 10` is

helpful but not a guarantee that an environment/class import will succeed.  It

was observed that above `40` environments, an import is virtually guaranteed to

fail.

##  What Is The Malfunction?

*Disclaimer: This is essentially speculation based on observed behavior and

isn't backed up by careful inspection of the code.*

From external observation of foreman's behavior, it appears that there is a

strong built-in assumption that the `puppetserver` environment class cache is

always enabled and `ETag`s will turn most queries for the classes in an

environment into essentially a no-op.

When a foreman environment import is triggered, foreman not only enumerates

`puppetserver`'s set of known environments via the `/puppet/v3/environments`

API endpoint, it also queries for all the classes within an environment using

`/puppet/v3/environment_classes`.  When the environment class cache is disabled

the time for this query to return seems to be highly variable for the test

environment with response times ranging from a few seconds to over 30s *per

environment*.  This means that an import cycle with many environments can take

10s of minutes.  It appears that something in this process causes foreman's

puma workers to consume 100% of a CPU.  It isn't known if this is busy waiting,

parsing the class list, or something else all together.

Compounding the extremely slow performance caused by bulk environment

importation, foreman also is calling `/puppet/v3/environment_classes` when it

is invoked as an ENC.  This means that every puppet agent run results in a

puppetserver instance compiling an environment to provide a class list to

foreman and then again after ENC/fact data has been supplied to create a

catalog for the agent.

To add insult to injury, in this situation the foreman enc is used solely to

supply parameters and never for class inclusion.  All of the effort to produce

environment class lists and parsing them is completely wasted.

## How Does This Thing Help?

`foreman_envsync` directly obtains the list of puppet environments from

`puppetserver` without requesting class information.  It then removes any

environments which are present in foreman but not within `puppetserver`.  If

there are any unknown-to-foreman environments, then are created as members of

all foreman organizations and locations.  This completely avoids foreman

having to wait on `puppetserver` to provide class lists and parsing them.

## Narrow Use-case

This utility is opinionated and has many hard-coded assumptions. Including:

* It is being installed on Centos 7.

* Foreman and `puppetserver` are installed on the same host.

* Foreman has been installed via `foreman-installer` / puppet code.

* The `puppetserver` TLS related files are in the standard Puppetlabs locations.

* `hammer` CLI has been installed and pre-configure with credentials.

* "new" puppet environments should be visible to all foreman organizations and locations.

## Obvious Improvements

`foreman_envsync` is shelling out to invoke `hammer`. It should probably be

converted to be a `hammer` plugin.

It probably makes sense to use `foreman-proxy` rather than connecting to

`puppetserver` directly as it avoid needing to deal with TLS authentication.

It would be fantastic if foreman's puppet integration was better able to handle

the environment class cache being disabled.  Including:

* Not requesting an environment's class list any time the ENC functionality was invoked.

* A configuration setting to completely disable class handling when it is not needed.

* An option to import environments only (similar to `foreman_envsync`).

* A configuration setting to disable the "import environments" feature which

  also requests class lists.

## Installation

This gem is currently intended to provide `foreman_envsync` CLI utility and not

for use as a library.

It is recommend that it is installed into the foreman `tfm` SCL and that all

dependencies are ignore to avoid disrupting foreman.  It is expected to function

with foremans' gem deps at least as of foreman `2.4.0`.

```bash

scl enable tfm -- gem install --bindir /bin --ignore-dependencies --no-ri --no-rdoc --version 0.1.4 foreman_envsync

```

**If** dependencies need to be installed, the ruby devel package a compiler will be needed. E.g.:

```bash

yum install -y rh-ruby25-ruby-devel devtoolset-7

scl enable devtoolset-7 tfm -- gem install --bindir /bin --ignore-dependencies --no-ri --no-rdoc --version 0.1.4 foreman_envsync

```

### `r10k` Config

It is is highly recommend that the `systemd-cat` utility to be used when configuring the `postrun` hook in `/etc/puppetlabs/r10k/r10k.yaml` so that status output is logged. E.g.:

```yaml

---

:postrun: ["/bin/scl", "enable", "tfm", "--", "systemd-cat", "-t", "foreman_envsync", "/bin/foreman_envsync", "--verbose"]

```

Example of output in the journal when using `systemd-cat`:

```

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: found 13 puppetserver environment(s).

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: ---

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2978_foreman_tuning_ruby

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2953_dds_debugging

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2907_tu_reip

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - production

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - coredev_production

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2978_foreman_tuning

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2373_velero

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2949_poc_encrypt

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2452_pagerduty

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2471_opsgenie

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2879_ipam

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2935_pathfinder_hcu01

Jun 08 23:00:38 foreman.example.org foreman_envsync[10643]: - IT_2987_rpi_puppet

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: found 13 foreman environment(s).

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: ---

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - coredev_production

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2373_velero

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2452_pagerduty

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2471_opsgenie

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2879_ipam

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2907_tu_reip

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2935_pathfinder_hcu01

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2949_poc_encrypt

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2953_dds_debugging

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2978_foreman_tuning

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2978_foreman_tuning_ruby

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - IT_2987_rpi_puppet

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: - production

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: found 0 foreman environment(s) unknown to puppetserver.

Jun 08 23:00:39 foreman.example.org foreman_envsync[10643]: found 0 puppetserver environment(s) unknown to foreman.

```

### `foreman-proxy` Config

If `foreman_envsync` is being considered then it is probably that foreman is also not being used for class inclusion. In that case, there is no reason for foreman to waste wallclock-time waiting for `puppetserver` to return class lists.

The wasted time may be minimized by setting

```yaml

:api_timeout: 1

```

in `/etc/foreman-proxy/settings.d/puppet_proxy_puppet_api.yml` (and restarting

`foreman-proxy`).

## Usage

*Note that `foreman_envsync` requires permissions to read `puppetserver`'s TLS

related files under `/etc/puppetlabs/puppet/ssl`. This may typically be

achieved by membership in the `puppet` group or running as `root`.*

`foreman_envsync` is silent, except for fatal errors, by default:

```bash

scl enable tfm -- foreman_envsync

```

The `--verbose` enables detailed output.

```bash

scl enable tfm -- foreman_envsync --verbose

```

Example of verbose output:

```bash

# scl enable tfm -- foreman_envsync --verbose

found 22 puppetserver environment(s).

---

- master

- IT_2953_dds_debugging

- corecp_production

- IT_2935_pathfinder_hcu01

- coredev_production

- IT_2949_poc_encrypt

- production

- IT_2907_tu_reip

- IT_2373_velero

- IT_2452_pagerduty

- IT_2978_foreman_tuning

- IT_2987_rpi_puppet

- corels_production

- ncsa_production

- disable_IT_2569_tomcat_tls

- tu_production

- disable_IT_2483_letencrypt_renewal

- disable_IT_2417_maddash

- disable_jhoblitt_rspec

- disable_jhoblitt_colina

- IT_2879_ipam

- IT_2471_opsgenie

found 36 foreman environment(s).

---

- corecp_production

- coredev_production

- corels_production

- coretu_production

- IT_2373_velero

- IT_2417_maddash

- IT_2441_dns3_cp

- IT_2452_pagerduty

- IT_2471_opsgenie

- IT_2483_letencrypt_renewal

- IT_2494_nfs

- IT_2569_tomcat_tls

- IT_2613_no_ccs_sal

- IT_2655_net_audit_ls

- IT_2657_dhcp

- IT_2667_poc

- IT_2693_arista

- IT_2694_vlan_change

- IT_2753_comcam_software_v2

- IT_2820_auxtel_ccs

- IT_2854_nfs_export

- IT_2879_ipam

- IT_2907_tu_reip

- IT_2911_ntp

- IT_2935_pathfinder_hcu01

- IT_2949_poc_encrypt

- IT_2953_dds_debugging

- IT_2978_foreman_tuning

- jhoblitt_colina

- jhoblitt_rspec

- master

- ncsa_production

- production

- tickets_DM_25966

- tickets_DM_27839

- tu_production

found 20 foreman environment(s) unknown to puppetserver.

---

- coretu_production

- IT_2417_maddash

- IT_2441_dns3_cp

- IT_2483_letencrypt_renewal

- IT_2494_nfs

- IT_2569_tomcat_tls

- IT_2613_no_ccs_sal

- IT_2655_net_audit_ls

- IT_2657_dhcp

- IT_2667_poc

- IT_2693_arista

- IT_2694_vlan_change

- IT_2753_comcam_software_v2

- IT_2820_auxtel_ccs

- IT_2854_nfs_export

- IT_2911_ntp

- jhoblitt_colina

- jhoblitt_rspec

- tickets_DM_25966

- tickets_DM_27839

deleted 20 foreman environment(s).

---

- message: Environment deleted.

  id: 12

  name: coretu_production

- message: Environment deleted.

  id: 4602

  name: IT_2417_maddash

- message: Environment deleted.

  id: 4664

  name: IT_2441_dns3_cp

- message: Environment deleted.

  id: 4637

  name: IT_2483_letencrypt_renewal

- message: Environment deleted.

  id: 4649

  name: IT_2494_nfs

- message: Environment deleted.

  id: 4658

  name: IT_2569_tomcat_tls

- message: Environment deleted.

  id: 4676

  name: IT_2613_no_ccs_sal

- message: Environment deleted.

  id: 4691

  name: IT_2655_net_audit_ls

- message: Environment deleted.

  id: 4688

  name: IT_2657_dhcp

- message: Environment deleted.

  id: 4736

  name: IT_2667_poc

- message: Environment deleted.

  id: 4709

  name: IT_2693_arista

- message: Environment deleted.

  id: 4710

  name: IT_2694_vlan_change

- message: Environment deleted.

  id: 4771

  name: IT_2753_comcam_software_v2

- message: Environment deleted.

  id: 4746

  name: IT_2820_auxtel_ccs

- message: Environment deleted.

  id: 4792

  name: IT_2854_nfs_export

- message: Environment deleted.

  id: 4762

  name: IT_2911_ntp

- message: Environment deleted.

  id: 4735

  name: jhoblitt_colina

- message: Environment deleted.

  id: 4619

  name: jhoblitt_rspec

- message: Environment deleted.

  id: 4581

  name: tickets_DM_25966

- message: Environment deleted.

  id: 4686

  name: tickets_DM_27839

found 6 puppetserver environment(s) unknown to foreman.

---

- IT_2987_rpi_puppet

- disable_IT_2569_tomcat_tls

- disable_IT_2483_letencrypt_renewal

- disable_IT_2417_maddash

- disable_jhoblitt_rspec

- disable_jhoblitt_colina

found 1 foreman location(s).

---

- 2

found 1 foreman organization(s).

---

- 1

created 6 foreman environment(s).

---

- message: Environment created.

  id: 4800

  name: IT_2987_rpi_puppet

- message: Environment created.

  id: 4801

  name: disable_IT_2569_tomcat_tls

- message: Environment created.

  id: 4802

  name: disable_IT_2483_letencrypt_renewal

- message: Environment created.

  id: 4803

  name: disable_IT_2417_maddash

- message: Environment created.

  id: 4804

  name: disable_jhoblitt_rspec

- message: Environment created.

  id: 4805

  name: disable_jhoblitt_colina

```

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/lsst-it/foreman_envsync.

## License

The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/lsst-it/foreman_envsync

Awesome Lists containing this project

README
