Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/slonopotamus/uest

Unreal Engine Suckless Testing
https://github.com/slonopotamus/uest

testing unreal-engine

Last synced: about 1 month ago
JSON representation

Unreal Engine Suckless Testing

Awesome Lists containing this project

README 

        
= UEST (Unreal Engine Suckless Testing)

:icons: font



ifdef::env-github[]

:caution-caption: :fire:

:important-caption: :exclamation:

endif::[]



This project aims to provide a testing framework for Unreal Engine that does not suck.



== Goals



* Simple things should be simple, complex things should be possible

* Extensibility

* Async execution support

* IDE and CI first-class support

* User-visible API ergonomics is more important than internal implementation



== Dependencies



This project relies on C++20 features.

Tested against Unreal Engine 5.4.



== Usage



=== Defining tests



.Simple test

[source,cpp]

----

#include "UEST.h"



TEST(MyFancyTest)

{

    // test body goes here

    ASSERT_THAT(...);

}

----



.Test class with multiple methods

[source,cpp]

----

#include "UEST.h"



TEST_CLASS(MyFancyTestClass)

{

    TEST_METHOD(Method1)

    {

        // test body goes here

        ASSERT_THAT(...);

    }



    TEST_METHOD(Method2)

    {

        // test body goes here

        ASSERT_THAT(...);

    }



    // put helper fields or methods here

}

----



If you want to execute a common piece of logic before and after each test method in a test class, you can do that using `BEFORE_EACH`/`AFTER_EACH` macros:



[source,cpp]

----

#include "UEST.h"



TEST_CLASS(MyFancyTestClass)

{

    BEFORE_EACH()

    {

        // Place code that will be executed before each test method of this class

    }



    AFTER_EACH()

    {

        // Place code that will be executed after each test method of this class

    }



    ...

}

----



=== Assertions



All UEST assertions are done through `ASSERT_THAT(Expression, Matcher)`.

Failed assertion performs `return`, aborting further test execution.



.Available matchers

`ASSERT_THAT(Value, Is::True)`:: Tests that `Value` is `true`.

`ASSERT_THAT(Value, Is::False)`:: Tests that `Value` is `false`.

`ASSERT_THAT(Value, Is::Null)`:: Tests that `Value` is `nullptr`.

`ASSERT_THAT(Value, Is::EqualTo(Expected))`:: Tests that `Value` is equal to `Expected`.

`ASSERT_THAT(Value, Is::LessThan(OtherValue))`:: Tests that `Value` is less than `OtherValue`.

`ASSERT_THAT(Value, Is::LessThanOrEqualTo(OtherValue))` or `ASSERT_THAT(Value, Is::AtMost(OtherValue)`:: Tests that `Value` is less than or equal to `OtherValue`.

`ASSERT_THAT(Value, Is::GreaterThan(OtherValue))`:: Tests that `Value` is greater than `OtherValue`.

`ASSERT_THAT(Value, Is::GreaterThanOrEqualTo(OtherValue))` or `ASSERT_THAT(Value, Is::AtLeast(OtherValue)`:: Tests that `Value` is greater than or equal to `OtherValue`.

`ASSERT_THAT(Value, Is::Zero)`:: Shortcut for `ASSERT_THAT(Value, Is::EqualTo(0))`.

`ASSERT_THAT(Value, Is::Positive)`:: Shortcut for `ASSERT_THAT(Value, Is::GreaterThan(0))`.

`ASSERT_THAT(Value, Is::Negative)`:: Shortcut for `ASSERT_THAT(Value, Is::LessThan(0))`.

`ASSERT_THAT(Value, Is::InRange(From, To))`:: Tests that `Value` is greater than or equal to `From` and is less than or equal to `To`.

`ASSERT_THAT(Value, Is::Empty)`:: Tests that `Value` is empty using its `IsEmpty()` method.

Use this for `FString` or collections (`TArray`, `TMap`, etc).

`ASSERT_THAT(Value, Is::Valid)`:: Tests that `Value` is valid using its `IsValid()` method.

Use this for `TSharedPtr`, `TWeakObjectPtr` or `TWeakPtr`.

`ASSERT_THAT(Value, Is::NaN)`:: Tests that `Value` is floating NaN.

Supports both float and double.



IMPORTANT: Because of the https://github.com/llvm/llvm-project/issues/73093[bug in Clang template type deduction] in versions older than 19.0, matchers with parameters (`LessThan`, `GreaterThan`, `EqualTo` and so on) require explicit template type specification: `ASSERT_THAT(0, Is::LessThan(1))`.



You can also negate assertions using `ASSERT_THAT(Value, Is::Not::)`.



Negated assertion example:

[source,cpp]

----

ASSERT_THAT(Value, Is::Not::Null);

----



#TODO: Document how to write custom matchers#



== Running tests



UEST is seamlessly integrated into Unreal Engine testing infrastructure, so you can run them using standard Session Frontend or IDE integration plugins.



=== Testing game worlds



UEST provides a convenient way to test game worlds, both standalone and multiplayer.



.Basic usage

[source,cpp]

----

TEST(MyGame, SimpleMultiplayerTest)

{

	auto Tester = FScopedGame().Create();



	// You can create a dedicated server

	UGameInstance* Server = Tester.CreateGame(EScopedGameType::Server, TEXT("/Engine/Maps/Entry"));



	// You can connect a client to it

	UGameInstance* Client = Tester.CreateClientFor(Server);

	ASSERT_THAT(Client, Is::Not::Null);



	// Actually, you can connect as many clients as you want!

	for (int32 Index = 0; Index < 10; ++Index)

	{

		Tester.CreateClientFor(Server);

	}



	// You can access game worlds

	UWorld* ServerWorld = Server->GetWorld();

	ASSERT_THAT(ServerWorld, Is::Not::Null);

	UWorld* ClientWorld = Client->GetWorld();

	ASSERT_THAT(ClientWorld, Is::Not::Null);



	// You can access actors in worlds

	APlayerController* ClientPC = ClientWorld->GetFirstPlayerController();

	ASSERT_THAT(ClientPC, Is::Not::Null);



	// You can lookup matching replicated actors in paired worlds

	APlayerController* ServerPC = Tester.FindReplicatedObjectIn(ClientPC, Server->GetWorld());

	ASSERT_THAT(ServerPC, Is::Not::Null);



	// You can advance game time

	Tester.Tick(1);



	// You can shut down individual game instances

	Tester.DestroyGame(Client);



	// You can also create standalone game worlds

	UGameInstance* Standalone = Tester.CreateGame(EScopedGameType::Client, TEXT("/Engine/Maps/Entry"));



	// Tester automatically cleans everything up when goes out of scope

}

----



== Further development plans



* More matchers

* Add `ASSERT_MULTIPLE` that allows performing multiple assertions without interrupting execution between them, also known as "soft assertions".

* Add API to disable tests (with `EAutomationTestFlags::Disabled` under the hood)

* Add API for asynchronous/latent tests



== Analysis of existing Unreal Engine solutions



As of 5.4, Unreal Engine has 4 (FOUR, that's not a typo) APIs for writing tests and all are very far from being good for various reasons.



Let's analyze them one-by-one.



=== Automation Test



[source,cpp]

----

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FMyTest, "MyGame.MyTest", EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::ProductFilter)

bool FMyTest::RunTest(const FString& Parameters)

{

    UTEST_TRUE_EXPR(true);



    return true;

}

----



.The good

* VisualStudio and JetBrains Rider know how to run this.

* `UTEST*` macros interrupt test execution (though these macros are useless for all other test frameworks because of non-void `return false;`)



.The bad

* Assertions do not capture expression that is being tested.

You have to write descriptive messages by hand.

* Overcomplicated way to add multiple tests with common logic.



.The ugly

* You need to write your test name *three times* as if it isn't clear enough what test name actually is.

* Requires lots of typing.

Macro could easily declare `RunTest` signature automatically.

Also, almost nobody wants to use custom flags.

* You must return a `bool` from the test.

If test reports an error, it should be marked as failing.

If there are no errors, it should be marked as successful.

This bool adds a completely useless (and even harmful) way to *fail without a message*.

* Nontrivial assertions (like `UTEST_EQUAL_EXPR`) are unable to print exact values of actual/expected.

* Inadequate support for async tests.

As soon as something becomes async, test body transforms into `ADD_LATENT_AUTOMATION_COMMAND` monster without an easy way of passing data between commands.



=== Automation Spec



[source,cpp]

----

DEFINE_SPEC(MySpec, "MyGame.MySpec", EAutomationTestFlags::ProductFilter | EAutomationTestFlags::ApplicationContextMask)

void MyCustomSpec::Define()

{

    TestTrue(TEXT("True should be true"), true);

}

----



.The good

* Understood by VS and Rider

* `void` return type

* Better async execution support, but not the best.

Programming community developed much better techniques than callback hell.

* May attract people that are familiar with spec-based approach from other areas.



.The ugly

* Declaring test name three times again

* Flags again

* No builtin way to interrupt test execution when assertion fails, so people have to invent their own assrtion macros.



=== Low Level



[source,cpp]

----

TEST_CASE("MyGame.MyTest", "[ApplicationContextMask][ProductFilter]")

{

    REQUIRE(true);

}

----



.The good

* Test name is written only once...

Well, no.

+

--

The caveat is that `TEST_CASE` macro uses a very broken way to generate unique class names.

They collide across compilation units and namespaces, and you end up asking yourself "why my test doesn't register at all".

Instead, Epics tell users to use `TEST_CASE_NAMED`, where you need to write test name _twice_.

That way, you end up with the same test class name collision chances as other approaches.

--



.The bad

* Not understood by Rider (https://youtrack.jetbrains.com/issue/RIDER-110897[RIDER-110897])



.The ugly

* String tags, really?

I am more than sure people will make typos and spend multiple hours trying to figure out why their test doesn't run.

* Assertions are a joke.

+

--

Just look at it:



[source,cpp]

----

#define REQUIRE(Expr) if (!(Expr)) { FAutomationTestFramework::Get().GetCurrentTest()->AddError(TEXT("Required condition failed, interrupting test")); return; }

----



Yep, you guessed it right, all you will get for failed assertion is "Required condition failed, interrupting test"

--



=== CQTest



[source,cpp]

----

TEST(MyTest, "MyGame")

{

    ASSERT_THAT(IsTrue(true));

}

----



.The good

* Test name is written only once

* No more flags

* `AreEqual` assertion is extensible and can print arbitrary types in error messages

* `void` test body

* Nice way to add multiple test methods to a single test class



.The bad

* Not understood by Rider (https://youtrack.jetbrains.com/issue/RSCPP-36039/Support-Unreal-Engine-CQTest-framework)[RSCPP-36039]).

Not sure about VS, would not be surprised if situation is the same.

* Async execution is as bad as in Automation Test style

* `clang-format` is unable to properly indent `TEST_CLASS` with nested `TEST_METHOD`



.The ugly

* Assertions do not capture tested expression.

`Expected condition to be true.`, seriously?

* Inadequate way to add custom assertions.

You need to use custom macros instead of `TEST` and `TEST_CLASS` because they hardcode `FNoDiscardAsserter`.

And this framework claims they are about composition instead of inheritance!

There was absolutely zero reason to tie test class to a _single_ asserter.

Asserter could easily be absolutely external class to the test itself, see NUnit for example.



// TODO: Write about AFunctionalTest, DaedalicTestAutomationPlugin, Gauntlet