https://github.com/olekli/mrblackjack

Simple framework for testing Kubernetes operators
https://github.com/olekli/mrblackjack

e2e-testing kubernetes operator testing testing-framework

Last synced: about 1 month ago
JSON representation

Simple framework for testing Kubernetes operators

• Host: GitHub
• URL: https://github.com/olekli/mrblackjack
• Owner: olekli
• License: apache-2.0
• Created: 2024-11-05T08:18:06.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-12-15T22:38:43.000Z (5 months ago)
• Last Synced: 2025-04-18T19:02:43.864Z (about 1 month ago)
• Topics: e2e-testing, kubernetes, operator, testing, testing-framework
• Language: Rust
• Homepage:
• Size: 112 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Mr. Blackjack

Simple framework for testing Kubernetes operators

## Overview

`blackjack` provides end-to-end testing by:

- Watching for resources that match certain patterns (based on group, version, kind, and selectors).

- Storing all observed resources in named "buckets" for later checks.

- Applying and deleting Kubernetes manifests.

- Running scripts to modify the cluster state or verify conditions.

- Waiting for defined conditions on watched resources to be met.

Each test is defined in a `test.yaml` according to a specified schema. Tests can be retried, ordered, and categorized by type. Individual test steps can have multiple operations (watch, bucket configuration, apply, delete, script, sleep, wait) to fully automate and validate complex cluster states.

## Basic Usage

1. Create a `test.yaml` file (or multiple files in a directory structure) describing your test using the schema.

2. Run the `blackjack` binary with the directory containing your tests.

```shell

cargo run --bin blackjack TEST-DIR

```

`blackjack` will:

- Discover all `test.yaml` files recursively under the specified directory.

- For each test, override namespaces in applied manifests with a randomly generated namespace, unique to the test run.

- After the test completes (pass or fail), it will clean up all resources it applied.

For reference, see the examples in `test/`.

## Test Specification

A test specification is defined by a top-level object (see `schema/test_spec.yaml` for the full schema):

### Top-Level Test Fields

- **name** (string):

  The name of the test. Defaults to an empty string.

- **attempts** (integer or null):

  On failure, the test can be retried for a specified number of attempts in total. Defaults to `null`, meaning no additional retries.

- **ordering** (string or null):

  A string to determine test ordering via lexicographical comparison. If two tests have the same type and concurrency rules, this string can be used to order them. Defaults to `null`.

- **type** (enum: `cluster` or `user`):

  Specifies the type of test. Defaults to `user`.

  - **`cluster`** tests are run first and not concurrently with `user` tests.

  - **`user`** tests can run concurrently, with concurrency limits defined by command line arguments.

- **steps** (array):

  A list of test steps. Each step describes a phase of the test with various operations (watch, apply, delete, script, sleep, bucket operations, wait). Each step is defined by a `StepSpec`.

### Steps (StepSpec)

Each step is an object with the following fields:

- **name** (string, required):

  A name for the step. Used for identification and logging.

- **watch** (array of WatchSpec):

  A list of watches to start. Starting a watch sets up a "bucket" that reflects the state of resources matching the given criteria. By default, all operations (create, patch, delete) are recorded unless later modified by bucket operations.

  Each `WatchSpec` can specify:

  - **name** (string, required): Bucket name to store observed resources.

  - **group** (string): Resource group to watch. Defaults to `""` (core group).

  - **version** (string): Resource version (e.g., `v1`). Defaults to `""`.

  - **kind** (string): Resource kind (e.g., `Pod`, `Deployment`). Defaults to `""`.

  - **namespace** (string): Namespace to watch. Defaults to `${BLACKJACK_NAMESPACE}`, the unique namespace created for this test run.

  - **labels** (object or null): A map of label key-value pairs to filter watched resources by label selectors. Defaults to `null`.

  - **fields** (object or null): A map of field selectors. Defaults to `null`.

- **bucket** (array of BucketSpec):

  Modify existing watch buckets to reflect only certain events. For example, you may choose not to record resource deletions or patches.

  Each `BucketSpec` includes:

  - **name** (string, required): Name of the bucket (as defined by a `watch`).

  - **operations** (array of BucketOperation, required): Which operations should be reflected.

    Possible values:

    - `create`: Newly created matching resources are recorded.

    - `patch`: Updated resources are recorded upon modifications.

    - `delete`: Deleted resources are removed from the bucket.

  By omitting an operation, the corresponding changes will not be reflected in the bucket.

- **apply** (array of ApplySpec):

  Apply Kubernetes manifests to the cluster.

  Each `ApplySpec` includes:

  - **path** (string, required): Path to a manifest file or directory of manifests.

  - **namespace** (string): Namespace override. Defaults to `${BLACKJACK_NAMESPACE}`.

  - **override-namespace** (boolean): Whether to override namespace specifications in the manifests. Defaults to `true`.

- **delete** (array of ApplySpec):

  Delete Kubernetes manifests from the cluster. The fields are the same as `apply`, but these resources will be removed.

- **script** (array of strings):

  A list of paths to shell scripts to run. These scripts are sourced by `sh`, and all `BLACKJACK_` prefixed environment variables are available in them. Scripts that exit non-zero cause the test to fail.

- **sleep** (integer):

  Sleep unconditionally for the specified number of seconds. Defaults to `0`.

- **wait** (array of WaitSpec):

  Wait until certain conditions are met for the resources in a specific bucket.

  Each `WaitSpec` includes:

  - **condition** (Expr, required): A logical expression describing the condition to check.

  - **target** (string, required): The name of the bucket to check.

  - **timeout** (integer, required): How many seconds to wait for the condition. If the condition is not met in time, the test fails.

### Condition Expressions (Expr)

Conditions control the logic for `wait` steps. Expressions can be combined with logical operators:

- **and**: An array of expressions all of which must be true.

- **or**: An array of expressions at least one of which must be true.

- **not**: A single expression whose truth value is negated.

- **size**: A numeric check that the number of resources in the target bucket matches a certain integer.

- **one**: Checks that at least one resource in the target bucket matches a certain pattern (partial object match).

- **all**: Checks that all resources in the target bucket match a certain pattern (partial object match).

The `one` and `all` checks are represented as boolean fields in the schema. In practice, these would be used in conjunction with additional logic to define the pattern that the resources must match.

### Test Type

As mentioned, tests have a `type` which can be either `cluster` or `user`:

- **cluster** tests run first and exclusively before `user` tests start.

- **user** tests can run concurrently after all `cluster` tests have completed.

This allows for a clear separation of "setup" or "integration" tests from more common "user scenario" tests.

### Retries

The `attempts` field defines how many times a test can be retried if it fails. By default, `null` means it is not retried beyond the initial attempt.

### Ordering

The `ordering` field is used to lexicographically order tests of the same type and within the same concurrency limits. This ensures a deterministic test run order if desired.

### Full Schema for Test Spec

The full schema is located in `schema/test_spec.yaml`. This document is a high-level description of the fields and how they relate.

**Key elements from the schema:**

- **ApplySpec**: How manifests are applied.

- **BucketSpec**: How watch buckets reflect changes (create/patch/delete).

- **WaitSpec**: Conditions and timeouts for waiting on bucket states.

- **WatchSpec**: How resources are watched and recorded.

- **Expr**: Logical condition structures for `wait` steps.

- **TestType**: Distinguish between `cluster` and `user` tests.

Refer to the schema for precise validation rules and defaults.

## Known Issues