The Apollo Runtime Testing Framework
Welcome to RTF!
RTF is a tool for writing, running and debugging tests of the Apollo Runtime under a variety of deployment setups. This documentation site covers how to get set up with the tooling and work with RTF Test Plans. While it is not necessary to read through every page in order, please be aware that the contents of the tutorials and guides may require you to have already worked through previous sections in some places. Where this is the case, links will be provided to the relevant sections of the documentation.
Who are these docs for?
The documentation is split into two main sections: “User Documentation” and “Developer Documentation”.
User Documentation covers both how to make use of the rtf CLI for running existing Test Plans
(such as those found in the rtf-morgue repo) and how to write your own Test Plans.
Developer Documentation covers details on the internal design of RTF and how to work within the runtime-testing-framework repo.
Where do I ask for help if I am stuck?
RTF is written and maintained by the Runtime Readiness team. You can find our Confluence space
here and we can be reached in Slack in our team channel: #team-runtime-readiness where you
can find links to our intake process and other useful information in the channel bookmarks.
For questions, feedback and support with RTF specifically we also have the
#proj-runtime-testing-framework channel.
Where can I find examples of RTF in use?
The rtf-morgue repo contains a number of test plans, helper scripts and GitHub Actions workflows for running tests of the Apollo Router. This repo is maintained by the Runtime Readiness team in order to support Router Core with release validation, regression testing and running investigations into customer issues.
Understanding RTF
This section explains the thinking behind RTF — what it is, how it fits into your testing strategy, and why it’s designed the way it is.
Overview
Throughout the rest of the User Documentation you’ll see us talking about RTF both as the framework itself and as part of the richer tooling and testing capabilities provided by the Runtime Readiness team. We feel that it is important to draw a distinction between these two areas in order to be able to make the best use of each.
So, while the rest of our documentation tends to focus on the “how” of using RTF and its related tooling, this page instead covers some of the “why” around how things are set up and the different options available for making use of it. To kick things off, its probably best to cover what RTF is (and importantly, what it isn’t).
What RTF is
Compositional Glue
The most important thing to understand about RTF is that it truly is all glue code and meta-data. RTF Test plans follow a simple “setup, run, teardown” execution model and providers are all self contained functions of their inputs. Need access to a resource inside of your test scenario? Add the appropriate provider. Need access to the same resource when you’re spinning up your test environment? Add the provider there as well: RTF will handle de-duplicating the resources for you.
One size fits most (not all)
Aiming to provide a working out of the box solution for every use case we come across inevitably results in chasing a long tail infrequently used aspects of the system that bitrot or are insufficiently tested.
No one wants that.
Instead we focus on paving a path for the majority of use cases that we know are actively required by our users (initially the Router core team) while ensuring that the system remains open to extension. As and when new shared use cases arise we work with the users who have an interest in their semantics to pull them in to the managed set of providers and base configurations.
What RTF is not
A magic bullet
While it is certainly possible to rewrite all of your existing test suites to run under RTF we wouldn’t recommend it. For anyone. The value add for using a framework like RTF is that it can handle pulling together resources and test data for you in a standardised, reliable way. If all you need to do is run something that looks like a unit test with no external dependencies, then RTF is almost guaranteed to be overkill.
Purpose built for your exact use case
By design, RTF is a general purpose framework that focuses on giving users the tools to write their own test plans and tooling. Given the nature of the types of tests we typically see run under RTF, you may be inclined to ask why the framework doesn’t offer built in support for things like running Terraform or managing Kubernetes clusters. The answer is relatively simple: RTF is designed to compose with other tools rather than embed them directly. By setting things up this way we make it possible for end users to leverage the tooling they are already familiar with alongside RTF rather than being forced to pick from a limited set of options that we happened to have added support for.
When is it worthwhile using RTF?
So if that’s what RTF is / is not in terms of design, when is it an appropriate tool to make use of? And, arguably more importantly, when is it not an appropriate tool?
As mentioned above, RTF is designed to compose together your existing tooling, scripts and infrastructure in a way that allows you to share common functionality. It shouldn’t be a surprise then that RTF is best suited to running integration style tests that target the behaviour of entire services and their interactions with specific data sets or configuration. If what you are trying to test can be handled as unit tests or smaller functional tests then you should almost certainly not be using RTF for that particular use case.
There are, of course, exceptions rules like this. For example, this test plan runs a Rust test scenario for the Query Planner to check how memory usage varies for different customer graphs and queries. The test itself is modelled on an existing unit test in the Router repo, but we write it as an RTF test plan so we can leverage the file providers that pull schemas and operations from studio as test data. This is something we can’t do in the public test suite within the Router repo due to the use of customer data.
Pay for what you use
We here at Runtime Readiness are big fans of the Unix Philosophy (as stated by Doug McIlroy):
Write programs that do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is a universal interface.
In our case, as with most developer tooling, “text streams” is better replaced with structured data in the form of YAML and JSON but the spirit of the philosophy remains.
We certainly require a variety of machinery to automate spinning up the services under test and pulling together all of the data and configuration required to run representative test scenarios. But we always make sure that each piece of what we write is usable independently (as far as possible) and that if you don’t want or need something, you shouldn’t have to even know that it exists.
To achieve that, our tooling is written as a collection of layers that you are free to pick and chose from as you see fit. Each individual piece will have its own requirements and expectations for you to be able to make use of it, but you should quickly see that many of those are shared so you get quite a lot of bang for your buck!
So, what’s actually on offer?
RTF and the additional tooling surrounding it is organised as a set of three distinct layers that work together to allow you, the user, to determine how and where to focus your time and resources. With each option there are a different set of trade offs involved which mainly concern whether you would prefer for everything to be as hands off as possible or if you would like to be able to configure things just the way you like them. As you might imagine, the further down the layers you go, the more control you have but also fewer guarantees.
Layer 3: Custom user facing functionality built using RTF
At one end of the scale we have custom applications built using RTF as a foundation such as the Router release validation tests. Here RTF is very much an implementation detail from the user’s perspective and the focus is entirely on addressing a particular testing need. If the application in question is one you want or need to make use of, it will have accompanying documentation to guide you through the process. If you are interested in developing your own testing capabilities on top of RTF then these existing applications are a useful place to start in order to get some ideas about what is possible.
Layer 2: Configurable test plans, environments, scenarios & providers
If you would prefer something a little more custom then the next layer down lets you make use of the
same building blocks we use ourselves for writing our own Layer 3 applications. Here you can find
general purpose Scenario and Environment configurations such as those available in the lib
directory of the rtf-morgue repo.
The focus of Layer 2 is to build and re-use higher level abstractions that address common use cases which require coordinating elements that make use of shared semantics or configuration.
As far as possible, anything that is done by one of our Layer 3 applications is also made
available as part of Layer 2 for direct use as well. If you find yourself needing to do something
that isn’t yet supported by our APIs or libraries, please do reach out to us in the
#proj-runtime-testing-framework channel in Slack to let us know! Chances are that other users
would like the same functionality and it might make sense to support it natively from our side.
Layer 1: RTF core
The final layer is RTF itself. Here we provide the rtf CLI which supports a number of
built-in providers that serve as a minimal foundation for writing test plans. At this layer you
have maximum flexibility to implement what you need but at the cost of minimal guarantees around how
you have composed together the different elements that make up your test plan.
We should emphasise that “minimal” here is in comparison to what is available from Layers 2 and 3: RTF provides a variety of debugging commands and rich error output to help you write and debug your test plans. What it doesn’t provide out of the box is a way for ensuring that you’ve set up your providers and test scripts in a self consistent way.
Next steps
Now that you’ve read a little of the “why” its time to dig into the “how”. Unsurprisingly, the Getting started section is where we recommend you start first if your goal is to learn about how RTF works and how you can work with pre-existing test plans. If you want to dig more into the framework itself, then the Test plans and framework sections are more likely what you are after.
Happy testing!
Where RTF fits in your test suite
Before diving in, it’s worth being precise about what RTF actually is — because it’s easy to mistake it for something it isn’t.
RTF is a framework for building and reproducing complex integrated environments. It handles spinning up services, resolving credentials, pulling in parameterised test data, and tearing everything down cleanly afterwards. What it is not is a framework for writing tests. RTF doesn’t provide assertions, test runners, or anything that touches the logic of your test scenarios — that part is entirely up to you and your existing tooling.
This distinction matters. The value RTF provides is reliability and repeatability at the environment level: the confidence that every time you run a Test Plan, the services under test are in the same known-good state, with the same data, in the same configuration. See Understanding RTF for more on the design philosophy behind this.
A further benefit of this approach is consistency. When Environments and Test Plans are defined in RTF, they can be shared, reviewed, and reproduced by any team — whether that’s another engineering team building on the same services, or Apollo’s support team trying to reproduce a customer issue. Everyone is working from the same definition, in the same way.
Because standing up a full integrated environment has a non-trivial cost, RTF is best suited to tests that justify that overhead — tests that can only be run meaningfully against a real, running system. Understanding where that threshold sits in the testing pyramid will help you decide what belongs in an RTF Test Plan and what doesn’t.
| Test type | ✅ Use RTF | ❌ Don’t use RTF |
|---|---|---|
| E2E | • Full-stack tests against real running services • Tests requiring real credentials or external dependencies | • Tests of logic within a single service • Tests that can run meaningfully against a mock or stub |
| Smoke | • Post-deployment health checks against a real environment • Confirming environment setup before a full run | • Exhaustive regression coverage • Checks that need to run in under a second |
| Performance | • Load testing against staging with representative workloads • Comparing throughput across service configurations | • Microbenchmarks of isolated functions • Profiling the internals of a single service |
| Exploratory | • Reproducing a bug against a live or staging environment • Testing a hypothesis under a specific configuration | • Structured regression runs that need a pass/fail record • Investigations that only require reading logs or metrics |
| Integration tests | — | Tests of internal component boundaries within a single service |
| Unit tests | — | Any test that doesn’t require a running service |
The testing pyramid
The testing pyramid is a useful mental model for thinking about how to distribute your automated tests, originally described by Martin Fowler. It has three layers:
- Unit tests (base) — test the smallest piece of logic in isolation. Fast, numerous, highly specific. A failing unit test points directly at the problem.
- Integration tests (middle) — test how two or more real components work together. Slower than unit tests, but they catch problems that only emerge at the boundaries between components.
- End-to-end tests (top) — test the full system from the outside, the way a real user or consumer would. The most expensive layer: they’re slow, require real infrastructure, and are prone to flakiness.
The pyramid shape reflects a practical ratio: many unit tests, fewer integration tests, and a small number of E2E tests covering your most critical paths. The top of the pyramid is valuable but costly, so you reserve it for the things that matter most.
Where RTF lives
RTF operates at the top of the pyramid. Every RTF Test Plan requires one or more real, running services to test against. The services themselves, how they’re configured, and how your test scenarios interact with them are all up to you — RTF is the layer that makes sure they’re all in place, in the right state, before your tests run.
This is intentional. RTF is designed for tests that can’t be meaningfully run in isolation — where the question you’re answering is “does this actually work, end to end, in this configuration?”
Test types RTF is suited for
All of the following test types sit at the top of the pyramid. What differs is their purpose and how often you’d run them.
End-to-end suites
The most common use of RTF is running a full E2E suite — a set of Scenarios that together verify the critical behaviour of a service across a set of configurations. A good E2E suite covers the flows your users depend on most, not every possible code path. Because E2E tests are expensive to run and maintain, selectivity is a virtue.
| Use RTF | Don’t use RTF |
|---|---|
| Verifying end-to-end behaviour across a full set of running services | Tests of logic within a single service that don’t require a running environment |
| Verifying behaviour across multiple service configurations | Tests that can run meaningfully against a mock or stub |
| Tests requiring real credentials, tokens, or external dependencies | Tests that cover every possible code path |
| Reproducing customer-facing bugs against a representative environment | Fast regression checks that should run on every commit |
Smoke and diagnostic runs
A smoke run should be just enough to confirm configuration and connectivity of a deployment of your service is working as expected. You’d typically run smoke tests immediately before and after a deployment.
In RTF, you can run a smoke suite by writing a dedicated Test Plan that references a subset of your
Scenarios, or by passing specific --var values to target a narrower slice of your matrix.
| Use RTF | Don’t use RTF |
|---|---|
| Pre- and Post-deployment health checks against a real environment | Comprehensive regression coverage |
| Verifying the single most critical path is functional | Deep scenario coverage or exhaustive matrix runs |
| Confirming environment setup is correct before a full run | Checks that should run in under a second |
Performance runs
RTF can be used for performance and load testing, where the Scenario exercises the service under representative load rather than verifying correctness alone. Performance tests are typically run on a different cadence to your functional E2E suite — against a staging environment before a release, rather than on every commit.
Performance tests are most valuable when they reflect realistic workloads against the actual services under test — which means they need a real environment and real data. RTF gives you a reproducible way to stand that environment up consistently.
| Use RTF | Don’t use RTF |
|---|---|
| Load testing against a staging environment with representative workloads | Microbenchmarks of isolated functions or algorithms |
| Measuring throughput or latency across different service configurations | Profiling the internals of a single service process |
| Comparing performance before and after a configuration change | Performance tests that don’t require a running service |
Exploratory testing
RTF’s granular execution flags make it well suited to exploratory testing — running a specific Scenario against a live service to investigate unexpected behaviour or test a hypothesis. For a broader treatment of exploratory testing and where it fits alongside automated suites, see The Practical Test Pyramid.
You can run a single Scenario directly:
rtf run --scenario test-plan.yaml
Or pin --var flags to run against a single matrix configuration without executing the full matrix.
This makes RTF useful not just as an automated test runner, but as a tool for debugging and
investigation in live environments.
| Use RTF | Don’t use RTF |
|---|---|
| Reproducing a bug against a live or staging environment | Structured regression runs that need a pass/fail record |
| Testing a hypothesis about service behaviour under a specific configuration | Investigations that only require reading logs or metrics |
| Manually verifying a fix before promoting to the full suite | Debugging issues within a single service’s internal logic |
Everything else
RTF is not the right tool for unit tests or integration tests within a single service. If what you’re testing doesn’t require a real running service — a parsing edge case, an internal function, a validation rule — you should test it with your language’s native test tooling. If you don’t require a running service, don’t use RTF.
This isn’t a limitation of RTF; it’s a deliberate boundary. The framework is designed to compose with your existing tooling, not replace it. A healthy test suite uses RTF alongside unit and integration tests, not instead of them.
If you’re unsure whether something belongs in RTF, ask: does this test require a real service to be running? If the answer is no, it probably doesn’t belong here.
Getting started
To build rtf from source, start by cloning the repo locally:
git clone git@github.com:apollographql/runtime-testing-framework.git
cd runtime-testing-framework
You will need a local Rust toolchain. We recommend using mise to install the Rust toolchain and
other dependencies. After installing mise, trust the mise config file:
mise trust
Install rtf with cargo:
cargo install --path crates/rtf-cli
Verify installation:
rtf
This should print the rtf help output to your terminal. With that done, you’re ready to run some test plans!
Example test plans
This repo contains a “Hello, world!” test plan to introduce you to rtf concepts. Start by following the “Hello, world!” guide.
The rtf-morgue contains the most up-to-date examples of how to write test plans for a variety of different test cases. It also contains guides for running those test plans.
Hello, world!
Overview
The example_test_plans directory in the rtf repository contains a “hello, world!” test plan that you can edit and run to learn about the concepts and terminology involved with using rtf.
We’ll start with templating and running the test plan as it is written. Then, we’ll take a quick look at a couple of simple ways we can make changes to the config files in order to alter its behaviour.
For more information on the structure of RTF test plan config files see the Framework / Test Plans page.
For details on how to get started with writing your own test plans from scratch see the Writing test plans section.
The “hello, world!” test plan contains a brief description, a few variables, and references to scenario and environment config files:
name: hello-world
description: |
The "hello, world!" of test plans. Intended as a minimal introduction to
the concepts and functionality found within RTF.
variables:
message: "hello, "
setup_subject: "world!"
scenario_subject: "darkness my old friend"
scenario:
from:
kind: local
relative_path: scenario.yaml
environment:
from:
kind: local
relative_path: environment.yaml
Pre-Flight Checks
Use the rtf template subcommand to pull in the scenario and environment config files referenced by
the test plan in order to see the fully templated file:
rtf template example-test-plans/hello-world/test-plan.yaml
You should see a larger YAML file containing all the information rtf needs to be able to run the
test plan. So, let’s try running it!
Running a Test Plan
To run the test plan, use the run subcommand. The -v option sets the log level to INFO:
rtf run example-test-plans/hello-world/test-plan.yaml -v
You should see output similar to this:
INFO loading and resolving test plan
INFO checking if templating will work
INFO creating output directory
INFO executing test plan
INFO templating environment setup
INFO checking environment setup
INFO executing environment setup
env-setup :: hello, world!
INFO templating scenario and environment teardown commands
INFO checking scenario and environment teardown commands
INFO executing scenario
scenario :: hello, darkness my old friend
INFO executing environment teardown
---
INFO writing out resolved test plan and variables
INFO done
You should also see that you now have an output directory in the directory where you ran rtf.
Take a look inside:
ls output
Output:
combined-output.txt
providers
resolved-test-plan.yaml
test-plan-variables.json
The providers directory contains the scripts that were copied from the file_providers
specified in our scenario and environment config files. Let’s look at the combined output:
cat output/combined-output.txt
Output:
env-setup :: hello, world!
scenario :: hello, darkness my old friend
---
The scripts used to generate this output are:
echo-message.sh
#!/usr/bin/env sh
echo "${STAGE} :: ${MESSAGE}${SUBJECT}" | tee -a "$OUTDIR/combined-output.txt"
if [ "${STAGE}" = "env-setup" ]; then
echo '{}' > "$RTF_OUTPUT"
fi
teardown.sh
#!/usr/bin/env sh
echo "---" | tee -a "$OUTDIR/combined-output.txt"
The combined-output.txt file is created by the echo-message.sh script and amended by the
teardown.sh script.
If you run the test plan a second time you will encounter an error: the output directory already
exists. This is a safety mechanism to prevent you from accidentally overwriting existing data or
merging the output from multiple runs together. Either remove the existing directory
(rm -rf output) or specify a new one using the --outdir flag:
rtf run example-test-plans/hello-world/test-plan.yaml -v --outdir=more_output
You should see output similar to before. You can verify both output directories exist:
ls | grep output
Output:
more_output
output
Modifying Variables
The test plan defines scalar values for templating variables which are then applied to the scenario and environment config files.
name: hello-world
description: |
The "hello, world!" of test plans. Intended as a minimal introduction to
the concepts and functionality found within RTF.
variables:
message: "hello, "
setup_subject: "world!"
scenario_subject: "darkness my old friend"
scenario:
from:
kind: local
relative_path: scenario.yaml
environment:
from:
kind: local
relative_path: environment.yaml
Run the test plan again using the default log level:
rtf run example-test-plans/hello-world/test-plan.yaml
Output:
env-setup :: hello, world!
scenario :: hello, darkness my old friend
---
Edit the test-plan.yaml to change the variable being used for the setup command:
variables:
message: "hello, "
- setup_subject: "world!"
+ setup_subject: "sailor!"
scenario_subject: "darkness my old friend"
Run the test plan again to see the modified output:
rm -rf output
rtf run example-test-plans/hello-world/test-plan.yaml
Output:
env-setup :: hello, sailor!
scenario :: hello, darkness my old friend
---
Now, edit the value of the message variable to see that it updates the output for both env-setup
and scenario, as they both reference the same shared variable:
variables:
- message: "hello, "
+ message: "say hi to the "
setup_subject: "world!"
scenario_subject: "darkness my old friend"
rm -rf output
rtf run example-test-plans/hello-world/test-plan.yaml
Output:
env-setup :: say hi to the world!
scenario :: say hi to the darkness my old friend
---
Overriding Individual Variables
If you want to override the value of a variable use the --var or --vars flags to specify
overrides on the command line:
rtf run example-test-plans/hello-world/test-plan.yaml \
--var 'message="say hi to the "'
Output:
env-setup :: say hi to the world!
scenario :: say hi to the darkness my old friend
---
We can also provide the flag multiple times to override multiple variables:
rtf run example-test-plans/hello-world/test-plan.yaml \
--var 'message="say hi to the "' \
--var 'setup_subject=sailor!'
Output:
env-setup :: say hi to the sailor!
scenario :: say hi to the darkness my old friend
---
It is also possible to use the --vars flag to provide the location of a JSON file containing the
variables you want to override on top of the ones given in the test plan:
cat example-test-plans/hello-world/variables.json
Output:
{
"message": "say hi to the ",
"setup_subject": "sailor!"
}
rtf run example-test-plans/hello-world/test-plan.yaml \
--vars example-test-plans/hello-world/variables.json
Output:
env-setup :: say hi to the sailor!
scenario :: say hi to the darkness my old friend
---
Each of these options is useful in different ways:
- Using
--varto provide individual variables on the command line allows you to dynamically set things using environment variables and other shell commands - Using
--varsto provide a JSON file containing multiple variables allows you to define variations on a test plan without having to edit or duplicate the test plan. Those variations can be stored in version control.
Matrix Variables
What if we want to define multiple sets of variables and run them all as part of a batch of tests?
For that, rtf provides a matrix feature that functions in a similar way to matrices in
GitHub Actions.
To convert a variable from a single scalar value to an array of values you’d like to use, move
it under the matrix.dimensions section of the test plan:
Remember to also remove it from the
variablessection or your test plan will fail its check!
variables:
message: "hello, "
- setup_subject: "world!"
scenario_subject: "darkness my old friend"
+matrix:
+ dimensions:
+ setup_subject: [ "world!", "sailor!" ]
Running the test plan with two setup_subject variables produces two results:
rm -rf output
rtf run example-test-plans/hello-world/test-plan.yaml
Output:
env-setup :: hello, world!
scenario :: hello, darkness my old friend
---
env-setup :: hello, sailor!
scenario :: hello, darkness my old friend
---
If we also move the scenario_subject into the matrix:
variables:
message: "hello, "
- setup_subject: "world!"
- scenario_subject: "darkness my old friend"
+
+matrix:
+ dimensions:
+ setup_subject: [ "world!", "sailor!" ]
+ scenario_subject: [ "darkness my old friend", "is it me you're looking for?" ]
We’ll get a run for every combination of variables:
rm -rf output
rtf run example-test-plans/hello-world/test-plan.yaml
Output:
env-setup :: hello, world!
scenario :: hello, darkness my old friend
---
env-setup :: hello, sailor!
scenario :: hello, darkness my old friend
---
env-setup :: hello, world!
scenario :: hello, is it me you're looking for?
---
env-setup :: hello, sailor!
scenario :: hello, is it me you're looking for?
---
By default, matrix output directories will be named matrix_variant_$n with n ranging from 1 to
the number of variants present in the matrix. To override this with a custom, more meaningful, name
you can set the matrix.variant_names key to generate variant names using a simple templating
syntax:
# The following template will generate variants named "one_3", "one_4", "two_3" and "two_4"
matrix:
variant_names: "${a}_${b}"
dimensions:
a: [ "one", "two" ]
b: [ 3, 4 ]
The syntax used for template strings involves placing matrix dimension names inside of ${} along
with static string content in order to generate a unique name for each variant. The resulting string
is then slugified to remove whitespace and slashes.
Writing test plans
This section will guide you through writing a test plan for RTF from a blank file. The test plan
will be a simple example that introduces you to the concepts of writing RTF test plans using a
docker compose environment and docker scenario. For examples of test plans that can be used for
specific testing use cases, please refer to the morgue.
Prerequisites
- Recommended: completed the “Hello, World!” guide — ensures you’re familiar with the RTF CLI and have it installed
dockeranddocker composeavailable locally with your docker daemon running
docker info
docker compose --help
Note This tutorial uses
dockeranddocker composeas execution backends. RTF parameterizes these tools from your config — this tutorial does not teach Docker or Docker Compose themselves. Refer to the Docker documentation and the Docker Compose documentation for details on those tools.
Next: Writing a test plan
Writing a test plan
In this guide, we’ll write a docker-based Test Plan from scratch. We’ll cover the four required fields, run the Test Plan, then add variables and a matrix to parameterize runs across multiple configurations.
Prerequisites
- RTF CLI installed and available in your terminal
- Docker and Docker Compose available in your terminal
Create an empty directory and make it your working directory:
mkdir rtf-hello-world
cd rtf-hello-world
Create an empty YAML file for the Test Plan:
touch test-plan.yaml
Add the following content to test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
inline:
name: Inline scenario config
description: An inline scenario config
docker:
image: alpine
tag: latest
command: echo "hello world!"
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
This will all be explained as we progress through the guide, for now all you need to know is this is the most basic docker based Test Plan it is possible to write in RTF.
Adding required fields
An RTF Test Plan has four required fields: name, description, scenario, and environment. The
sections below add each of these required fields and explain them in more detail.
name
Add the name field to the test-plan.yaml file
name: Hello World
name is used to give each Test Plan an identifiable title. It can be any valid string. The value
used for the name field has no impact on the execution of the Test Plan. This makes it easier to
work with the Test Plan programmatically.
description
Add the description field to the test-plan.yaml file
name: Hello World
description: Created as a guide for writing test plans
description is used to give more information about the Test Plan for future users. It can be any
valid string. The value used for the description field has no impact on the execution of the Test
Plan itself. This is a useful place to add links or reference materials and should be preferred over
adding that context to inline comments.
scenario
The scenario is used to define the configuration and command that runs the actual testing logic in
the Test Plan. The scenario can be defined inline within the Test Plan or in its own file. In this
guide, we’ll define the Scenario inline. The guide on writing a new Scenario covers how to
define a Scenario in a separate file.
We’ll use a docker based Scenario — the recommended approach in RTF. If docker isn’t an option,
see script based Scenarios. We’ll cover more advanced Scenario configuration in the guide on
writing a new Scenario.
Note RTF runs the scenario by passing the
image,tag, andcommandconfig fields as arguments todocker run. For details ondocker runand Docker images, refer to the Docker documentation.
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
inline:
name: Inline docker scenario config
description: An inline docker scenario config
docker:
image: alpine
tag: latest
command: echo "hello world!"
- The
inlinefield is used to indicate the Scenario will be defined in the Test Plan file. - The
nameanddescriptionfields are required and used to identify the Scenario and work the same asnameanddescriptionin the Test Plan. - The
dockerfield is used to define the container the Scenario will runimageis the name of the container image. Note that you will need to ensure that wherever you are runningdockerfrom is authenticated to pull the image.tagdefines the image tag that should be pulled.commandoptionally sets the command the container runs.
environment
The environment is used to define the configuration and commands that set up the Environment
for testing and tear it down after the test has completed. The environment can be defined inline
within the Test Plan or in its own file. In this guide, we’ll define the Environment inline. The
guide on writing a new Environment covers how to define an Environment in a separate file.
We’ll use a docker compose based Environment — the recommended approach in RTF. If
docker compose isn’t an option, see script based Environments. We’ll cover more advanced
Environment configuration in the guide on writing a new Environment.
Note RTF manages the environment by running
docker compose upanddocker compose down, passing the resolved compose files as-farguments. For details on Docker Compose files and options, refer to the Docker Compose documentation.
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
inline:
name: Inline docker scenario config
description: An inline docker scenario config
docker:
image: alpine
tag: latest
command: echo "hello world!"
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
- The
inlinefield is used to indicate the Environment will be defined in the Test Plan file. - The
nameanddescriptionfields are required and used to identify the Environment and work the same asnameanddescriptionin the Test Plan. - The
compose_filesarray defines a list ofdocker composefiles the Environment will run.
The service we are running in the docker-compose.yaml test service is a simple web server. We will
show how to connect the docker container that runs in the scenario to the service running in the
environment in the “Writing a scenario” guide.
Checking the Test Plan
Now, let’s check that the Test Plan has been defined correctly:
rtf template test-plan.yaml
The output is similar to this:
name: Hello World
description: A test plan created as a guide for writing test plans
variables: {}
matrix:
variant_names: null
dimensions: {}
compound: {}
custom_providers: []
scenario:
name: Inline scenario config
description: An inline scenario config
variable_definitions: []
custom_providers: []
docker:
image: alpine
tag: latest
command: echo "hello world!"
env_vars: {}
file_providers: []
environment:
name: Inline docker compose environment config
description: An inline docker compose environment config
variable_definitions: []
custom_providers: []
project_name: null
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
file_providers: []
env_vars: {}
The Test Plan templates successfully! This also highlights two optional fields — variables and
matrix — that have not yet been used. These are discussed more below.
Running the Test Plan
Before looking at variables and matrix, let’s run the Test Plan:
rtf run test-plan.yaml
The output is similar to this:
[+] up 2/2
✔ Network inline-docker-compose-environment-config_default Created 0.0s
✔ Container inline-docker-compose-environment-config-hello-world-1 Healthy 0.7s
hello world!
[+] down 2/2
✔ Container inline-docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network inline-docker-compose-environment-config_default Removed 0.1s
Your first Test Plan ran successfully! The hello world! output confirms the scenario container ran
the command we configured.
This also creates an output directory.
The output directory contains a providers directory and two files: resolved-test-plan.yaml and
test-plan-variables.json.
resolved-test-plan.yamlcontains the fully resolved Test Plan config. This should be the same as what was shown in thertf templatecommand. This is a way to verify the Test Plan that ran to give you the output.test-plan-variables.jsoncontains the variables used during the execution of the Test Plan. This is empty since no variables were set.
Remove the output directory before continuing (forgetting to do this will result in an error next
time rtf run is used):
rm -rf output/
Note RTF is deliberately configured to not overwrite an existing output directory. This is so you cannot accidentally overwrite output you intend to keep. The
--outputflag can be used withrtf runto set a different output directory if you want to keep the existing output and run a new test.
Setting variables
The variables field is used to set global variables that can be referenced in your scenario and/or
environment. Any variables set in the Test Plan config can be overridden using the --var and
--vars flags in the RTF CLI (see the modifying variables section of the hello world guide for
more information).
Let’s add some example variables to test-plan.yaml. We are also going to update the Scenario
command to use this variable. The “Writing a Scenario” section will explain how this works, for
now just add the configuration:
name: Hello World
description: A test plan created as a guide for writing test plans
# --- Add variables ---
variables:
example_variable: example variable
# ------------------
scenario:
inline:
name: Inline scenario config
description: An inline scenario config
# --- Update scenario ---
variable_definitions:
- name: example_variable
description: An example variable
env_vars:
EXAMPLE_VARIABLE: "{{ example_variable }}"
docker:
image: alpine
tag: latest
command: echo "$EXAMPLE_VARIABLE"
# -----------------------
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
You can see the value being used by running the Test Plan again:
rtf run test-plan.yaml
Output:
[+] up 2/2
✔ Network inline-docker-compose-environment-config_default Created 0.0s
✔ Container inline-docker-compose-environment-config-hello-world-1 Healthy 0.7s
example variable
[+] down 2/2
✔ Container inline-docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network inline-docker-compose-environment-config_default Removed 0.1s
Now, the test-plan-variables.json file contains the variable that we set in the Test Plan:
cat output/test-plan-variables.json
Output:
{
"example_variable": "example variable"
}
Using a matrix
The matrix field is used to create a matrix of variable dimensions to iterate over (see the
matrix variables section of the hello world guide for more information). A matrix can only be
defined in the Test Plan config.
Let’s add a matrix to and remove the variables from our test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
# --- Replace variables with a matrix ---
matrix:
variant_names: "${example_variable}"
dimensions:
example_variable:
- variable1
- variable2
# ------------------------------------
scenario:
inline:
name: Inline scenario config
description: An inline scenario config
variable_definitions:
- name: example_variable
description: An example variable
env_vars:
EXAMPLE_VARIABLE: "{{ example_variable }}"
docker:
image: alpine
tag: latest
command: echo "$EXAMPLE_VARIABLE"
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
Run the Test Plan again (make sure the output directory has been deleted after previous test
runs):
rtf run test-plan.yaml
The terminal output will look different this time - the environment and scenario commands are
executed twice. The output directory will also have a different structure:
ls output/
Output:
variable1 variable2
Let’s look at each of those matrix directories to see the different variable values used per execution:
cat output/variable1/test-plan-variables.json
Output:
{
"example_variable": "variable1"
}
cat output/variable2/test-plan-variables.json
Output:
{
"example_variable": "variable2"
}
The example_variable’s variable changes per execution. You’ve run your first matrix — the scenario
executed twice, once for each value in the example_variable dimension!
Next steps
In this guide, we covered writing a docker-based Test Plan with inline scenario and environment configs, and used variables and a matrix to parameterize runs. Next, we’ll walk through how to write scenarios in more detail.
Writing a new scenario
In this guide, we’ll take the inline scenario from the “Writing a test plan” tutorial and improve it step by step. We’ll connect it to the docker compose environment, move the docker command into a script using file providers, extract the scenario into its own file, use variables to pass values into the scenario, and use overrides to swap out scenario config without modifying the base file.
Prerequisites
- Completed the “Writing a test plan” tutorial
- An
rtf-hello-worlddirectory containingtest-plan.yamlwith the exact content shown belowWe’re going to remove the variables and matrix added in the final step of the “Writing a test plan” guide before continuing.
Your test-plan.yaml file should contain:
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
inline:
name: Inline scenario config
description: An inline scenario config
docker:
image: alpine
tag: latest
command: echo "hello world"
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
Changing the docker command
Note RTF runs the scenario container by parameterizing
docker runfrom thedockerconfig. For details ondocker run, Docker images, and networking, refer to the Docker documentation.
The scenario container in the test plan does not interact with the docker compose environment, it just echos “hello world”. In a real test scenario, we would want the scenario container to call a service in the docker compose stack.
Let’s update our command to call the hello-world service we start in the docker compose
environment.
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
inline:
name: Inline scenario config
description: An inline scenario config
docker:
image: alpine
tag: latest
# --- Update the docker scenario command ---
command: wget -qO- http://hello-world:80
# ------------------------------------------
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
We are using wget to make the HTTP request because the alpine container already has this
installed.
Note We are using
http://hello-world:80as the endpoint. This works because RTF automatically adds a docker scenario container to the docker compose environment network using the--netflag. This avoids the need to map the docker compose services to localhost.
Now, if we run the test plan:
rtf run test-plan.yaml
Output:
[+] up 2/2
✔ Network inline-docker-compose-environment-config_default Created 0.0s
✔ Container inline-docker-compose-environment-config-hello-world-1 Healthy 0.7s
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[+] down 2/2
✔ Container inline-docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network inline-docker-compose-environment-config_default Removed 0.1s
Your scenario container is now talking to the docker compose service! The wget output confirms the
two containers are on the same network.
Running a script as a docker command
Specifying commands like this works for a simple one line command, but is not useful for anything
more complex. It is possible to execute a script in the command for more complex use cases
instead. Let’s move the wget command into a file instead of specifying it inline and wrap it in
some additional logic so we just get the HTTP status code in the terminal output.
mkdir scripts
touch scripts/check-status.sh
Then add the command to the check-status.sh file
#!/bin/sh
response=$(wget -S -O /dev/null http://hello-world:80 2>&1)
status=$(echo "$response" | grep "HTTP/" | grep -o "[0-9][0-9][0-9]")
echo "HTTP status: $status"
Next, update the test-plan.yaml file so the scenario executes the check-status.sh file:
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
inline:
name: Inline scenario config
description: An inline scenario config
docker:
image: alpine
tag: latest
# --- Update the docker scenario command ---
command: sh "$CHECK_STATUS_SCRIPT"
file_providers:
- name: check-status.sh
env_var: CHECK_STATUS_SCRIPT
kind: relative_path
path: scripts/check-status.sh
# ------------------------------------------
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
Note File providers won’t be executable directly so you need to make sure to run using
sh $CHECK_STATUS_SCRIPT, not./$CHECK_STATUS_SCRIPT.
We have not explained how File Providers work yet. This is covered more in the “Using file providers” guide.
Let’s run the test plan:
rtf run test-plan.yaml
Output:
[+] up 2/2
✔ Network inline-docker-compose-environment-config_default Created 0.0s
✔ Container inline-docker-compose-environment-config-hello-world-1 Healthy 0.7s
HTTP status: 200
[+] down 2/2
✔ Container inline-docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network inline-docker-compose-environment-config_default Removed 0.1s
We now get a much more readable output from the scenario container instead of the verbose nginx
response. You now have a scenario running a script and reporting a clean HTTP 200!
Creating a scenario file
Writing the scenario inline like this works for simple test plans. However, more comprehensive scenarios quickly become harder to read and maintain. It is possible to create a separate file to store your scenario config and update your test plan to refer to that file. As well as maintainability benefits this also means the same scenario can be reused in multiple test plans. It is also possible to update the base scenario (and environment) config using overrides in the test plan. We cover how to use overrides in the Using overrides section below.
Let’s walkthrough how to create a scenario config. First, create a configs directory and an empty
YAML file inside it:
mkdir configs
touch configs/scenario.yaml
We want to move the scenario config so it is no longer defined inline in the test plan config. Copy
the scenario config from the test plan and add to the scenario.yaml file:
name: Docker scenario config
description: A docker scenario config
docker:
image: alpine
tag: latest
command: sh "$CHECK_STATUS_SCRIPT"
file_providers:
- name: check-status.sh
env_var: CHECK_STATUS_SCRIPT
kind: relative_path
path: scripts/check-status.sh
Now, we need to update the test plan so it uses the scenario config in the scenario.yaml file. To
do this, we need to delete the inline key (and the scenario config nested beneath it) and switch
to using the from key:
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
# --- Replace the inline scenario ---
from:
kind: local
relative_path: configs/scenario.yaml
# -----------------------------------
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
Now check the test plan templates:
rtf template test-plan.yaml
This will result in YAML being printed to the terminal, showing the scenario config has been inlined from the external file. This is exactly what we had when the scenario was defined inline - the template command inlines external configs ahead of execution.
When writing configs with a lot of files on relative paths, it is good practice to use the --check
flag when running the template command. This will check that files on relative paths can be found.
Let’s see what happens when we run with that flag:
rtf template test-plan.yaml --check
Output:
ERROR Static analysis checks failed
(scenario.CHECK_STATUS_SCRIPT) The requested file did not exist
provided path was file:///path/to/configs/scripts/check-status.sh
When copying over our inline scenario config we forgot to account for the fact that our
scenario.yaml file is on a different path to our test-plan.yaml file. Relative paths are always
relative to the file they are defined in. Let’s fix our mistake in the scenario.yaml file:
name: Docker scenario config
description: A docker scenario config
docker:
image: alpine
tag: latest
command: sh "$CHECK_STATUS_SCRIPT"
file_providers:
- name: check-status.sh
env_var: CHECK_STATUS_SCRIPT
kind: relative_path
# --- Update the path to the scenario script ---
path: ../scripts/check-status.sh
# ----------------------------------------------
If we run the template command again with the --check flag you should see the config YAML
printed to your terminal:
rtf template test-plan.yaml --check
The output is similar to this:
name: Hello World
description: A test plan created as a guide for writing test plans
...
The from key can have two variables, local or github. In this case, we are using local. This
will import the scenario config from the file on the relative path defined in the relative_path
key.
The github key allows the scenario config to be imported from a GitHub repo. It is not discussed
in detail in this guide, please refer to the framework reference docs for more detail.
You now have a reusable scenario in its own file! See the framework reference docs for the full Scenario config structure.
Using variables
We saw how to set variables in the test plan in the “Writing a test plan” guide. Let’s look at
this in more detail. To use variables in a scenario, we need to define them in the variables
field.
Declaring the variables here declares a contract between the scenario and test plan and lists the
variables that must be specified for the scenario to complete. The value of the variable can be set
using a default in the scenario, in the test plan or provided via the CLI. If the variable is used
by the scenario and not set via any of those methods, it will cause the test plan execution to fail.
If variables are defined for usage in the scenario but not defined in the variables field then the
test plan will fail to template.
Let’s see that in action. We are going to add an environment variable to the scenario and use a
variable to set its value. Update scenario.yaml:
name: Docker scenario config
description: A docker scenario config
# --- Add a variables section ---
variable_definitions:
- name: scenario_variable
description: An example variable that the scenario expects to be defined
# -------------------------------
docker:
image: alpine
tag: latest
command: sh "$CHECK_STATUS_SCRIPT"
# --- Use the scenario_variable in the environment variables ---
env_vars:
SCENARIO_ENV: "{{ scenario_variable }}"
# --------------------------------------------------------------
file_providers:
- name: check-status.sh
env_var: CHECK_STATUS_SCRIPT
kind: relative_path
path: ../scripts/check-status.sh
Let’s explain how this works. The variables each have a name and description. The name is the
variable’s identifier and is used in the template string. The description is there to give more
information about how and why the variable is used. Variables are templated into the config with the
"{{ ... }}" syntax, where ... is replaced by the variable’s name.
Note The double curly braces and space either side of the variable name are important here. If the template string does not match this exactly, then rtf will error and call out there is a malformed template string.
To show this is added to the scenario container’s environment, let’s also add a line to the
check-status.sh script to echo the environment variable’s value.
#!/bin/sh
# New line to echo the SCENARIO_ENV
echo $SCENARIO_ENV
response=$(wget -S -O /dev/null http://hello-world:80 2>&1)
status=$(echo "$response" | grep "HTTP/" | grep -o "[0-9][0-9][0-9]")
echo "HTTP status: $status"
Let’s attempt to template the test plan:
rtf template test-plan.yaml
Output:
ERROR Templating failed
(scenario.env_vars.SCENARIO_ENV) Missing template variables definition. Make sure the variable is defined in the scenario or environment config variable definitions
- scenario_variable: "An example variable that the scenario expects to be defined"
We have successfully defined a variable and where it should be used. However, we have not specified what value it should actually have. If we attempted to run this test plan we would see the same error. Let’s define a default for this variable:
name: Docker scenario config
description: A docker scenario config
variable_definitions:
- name: scenario_variable
description: An example variable that the scenario expects to be defined
# --- Set a default for this variable ---
default: "default scenario env var value"
# ---------------------------------------
docker:
image: alpine
tag: latest
command: sh "$CHECK_STATUS_SCRIPT"
env_vars:
SCENARIO_ENV: "{{ scenario_variable }}"
file_providers:
- name: check-status.sh
env_var: CHECK_STATUS_SCRIPT
kind: relative_path
path: ../scripts/check-status.sh
Let’s run this and see what happens:
rtf run test-plan.yaml
Output:
[+] up 2/2
✔ Network inline-docker-compose-environment-config_default Created 0.0s
✔ Container inline-docker-compose-environment-config-hello-world-1 Healthy 0.7s
default scenario env var value
HTTP status: 200
[+] down 2/2
✔ Container inline-docker-compose-environment-config-hello-world-1 Removed 0.2s
✔ Network inline-docker-compose-environment-config_default Removed 0.1s
The scenario environment variable echo statement uses the default variable. We can override this
variable by setting a different value in the test plan (this will take precedence over a default).
Update test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
# --- Add a new value for scenario_variable ---
variables:
scenario_variable: "scenario env var value from test plan"
# ---------------------------------------------
scenario:
from:
kind: local
relative_path: configs/scenario.yaml
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
Now if we run:
rtf run test-plan.yaml
Output:
[+] up 2/2
✔ Network inline-docker-compose-environment-config_default Created 0.0s
✔ Container inline-docker-compose-environment-config-hello-world-1 Healthy 0.7s
scenario env var value from test plan
HTTP status: 200
[+] down 2/2
✔ Container inline-docker-compose-environment-config-hello-world-1 Removed 0.2s
✔ Network inline-docker-compose-environment-config_default Removed 0.1s
We can see that the variable from the test plan has overridden the default. Similarly, if the variable is specified from the CLI it will override both the test plan variable and default:
rtf run test-plan.yaml --var scenario_variable="scenario env var value from cli variable"
Output:
[+] up 2/2
✔ Network inline-docker-compose-environment-config_default Created 0.0s
✔ Container inline-docker-compose-environment-config-hello-world-1 Healthy 0.7s
scenario env var value from cli variable
HTTP status: 200
[+] down 2/2
✔ Container inline-docker-compose-environment-config-hello-world-1 Removed 0.2s
✔ Network inline-docker-compose-environment-config_default Removed 0.1s
You now know how to pass variables at all three levels: scenario default, test plan, and CLI. Each level overrides the one before it.
Using overrides
One of the main benefits of defining scenario (and environment) config in a separate file is that it
can be reused across multiple test plans. There will be occasions where you want to reuse the
majority of what’s defined in a scenario config but make small edits. Instead of making a new file
with the edits, you can use overrides.
Before we add the override to our test plan, let’s create an updated script for the scenario to run that prints the response as well as the HTTP status code:
touch scripts/check-status-v2.sh
Add the following to check-status-v2.sh:
#!/bin/sh
echo $SCENARIO_ENV
body=$(wget -qO- http://hello-world:80)
echo "Response: $body"
status=$(wget -S -O /dev/null http://hello-world:80 2>&1 | grep "HTTP/" | grep -o "[0-9][0-9][0-9]")
echo "HTTP status: $status"
Overrides can be added to either the environment or scenario in the test plan config. Add this
to test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
variables:
scenario_variable: "scenario env var value from test plan"
scenario:
from:
kind: local
relative_path: configs/scenario.yaml
# --- Add a new overrides section ---
overrides:
file_providers:
- name: check-status.sh
env_var: CHECK_STATUS_SCRIPT
kind: relative_path
path: scripts/check-status-v2.sh
# ------------------------------------
environment:
inline:
name: Inline docker compose environment config
description: An inline docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
Before checking if this works, let’s look at how it works. We only want to change the
CHECK_STATUS_SCRIPT, so only that segment of the config is required. RTF will merge the YAML on
matching keys before checking if it templates. If you run the template command now:
rtf template test-plan.yaml --check
You should see that the scenario CHECK_STATUS_SCRIPT now matches what we defined in the
overrides, while the rest of the scenario config is unchanged. If we run the test plan:
rtf run test-plan.yaml
Output:
[+] up 2/2
✔ Network docker-compose-environment-config_default Created 0.0s
✔ Container docker-compose-environment-config-hello-world-1 Healthy 0.7s
scenario env var value from test plan
Response: <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
HTTP status: 200
[+] down 2/2
✔ Container docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network docker-compose-environment-config_default Removed 0.1s
We can now see the response being printed in the terminal output again. This confirms we’re
successfully using the new check-status-v2.sh script for the scenario without changing any other
config for the scenario.
Next steps
In this guide, we moved a scenario into its own file, used file providers to run a script, wired up variables with defaults that can be overridden at runtime, and used overrides to swap out config without touching the base file. Next, we’ll walk through writing an environment config.
Writing a new environment
In this guide, we’ll take the inline environment from the “Writing a scenario” tutorial and move it into its own file. We’ll then extend it by adding a second compose service, and use RTF environment variables to configure the docker compose stack.
Note RTF manages the environment by parameterizing
docker compose upanddocker compose downfrom thecompose_filesconfig. For details on Docker Compose files and multi-file composition, refer to the Docker Compose documentation.
Prerequisites
- Completed the “Writing a scenario” tutorial
- An
rtf-hello-worlddirectory in the state it was at the end of that guide
Your directory should look like this:
configs scripts test-plan.yaml
./configs:
scenario.yaml
./scripts:
check-status-v2.sh check-status.sh
Creating an environment file
Creating a separate environment file works exactly the same way and has the same benefits as creating a separate scenario file outlined in the “Writing a scenario” guide.
Let’s update our test plan to specify the environment in a separate file:
touch configs/environment.yaml
Make sure the environment.yaml contains a copy of the environment config currently in your
test-plan.yaml file:
name: Docker compose environment config
description: A docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
Finally, update the test-plan.yaml file to use the new environment.yaml file:
name: Hello World
description: A test plan created as a guide for writing test plans
variables:
scenario_variable: "scenario executed with test plan variable"
scenario:
from:
kind: local
relative_path: configs/scenario.yaml
environment:
from:
kind: local
relative_path: configs/environment.yaml
Let’s verify this has made no material difference to the templated test plan:
rtf template test-plan.yaml --check
The output is similar to this:
name: Hello World
description: A test plan created as a guide for writing test plans
variables:
scenario_variable: scenario env var value from test plan
matrix:
variant_names: null
dimensions: {}
compound: {}
custom_providers: []
scenario:
...
environment:
...
You now have a reusable environment in its own file! See the framework reference docs for the full Environment config structure.
Adding additional compose files
RTF supports multiple compose files via the compose_files key — each file is passed to
docker compose up using the -f flag.
Let’s create a new compose file.
mkdir data
touch data/compose-app.yaml
We are going to create a python web server that returns Hello, World! when called. Add the
following content to the compose-app.yaml file:
services:
app:
image: python:alpine
environment:
HELLO_MESSAGE: Hello, World!
ports:
- 8000:8000
command:
- python
- -c
- |
from http.server import HTTPServer, BaseHTTPRequestHandler
import os
class H(BaseHTTPRequestHandler):
def do_GET(self):
msg = os.environ.get('HELLO_MESSAGE').encode()
self.send_response(200)
self.send_header('Content-Type', 'text/plain')
self.end_headers()
self.wfile.write(msg)
def log_message(self, *a):
pass
HTTPServer(('', 8000), H).serve_forever()
Let’s add this to our compose_files in environment.yaml:
name: Docker compose environment config
description: A docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
# --- Add the compose-app.yaml file ---
- name: compose-app.yaml
kind: relative_path
path: ../data/compose-app.yaml
# -------------------------------------
Note The path to the
compose-app.yamlfile is relative to theenvironment.yamlfile its path is defined in.
We have not explained File Providers yet and will cover them in detail in the “Using file providers” guide.
Let’s run the test plan
rtf run test-plan.yaml
Output:
[+] up 3/3
✔ Network docker-compose-environment-config_default Created 0.0s
✔ Container docker-compose-environment-config-app-1 Healthy 0.6s
✔ Container docker-compose-environment-config-hello-world-1 Healthy 0.6s
scenario env var value from test plan
Response: <removed for brevity>
HTTP status: 200
[+] down 3/3
✔ Container docker-compose-environment-config-app-1 Removed 10.1s
✔ Container docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network docker-compose-environment-config_default Removed 0.1s
In addition to the docker-compose-environment-config-hello-world-1 container that was running
before, we now can also see the docker-compose-environment-config-app-1 starting up. You now have
an additional service running from a new compose file! Let’s update our test to actually use it!
Using environment variables
We have a running nginx server and our new python web server. We are going to update the nginx
config so that when we call nginx, it forwards our request to our new service. We are going to do
this by updating the docker compose configuration in our environment.yaml file:
name: Docker compose environment config
description: A docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
# -------- Add nginx config ---------
configs:
- source: nginx_conf
target: /etc/nginx/conf.d/default.conf
configs:
nginx_conf:
content: |
server {
listen 80;
location / {
proxy_pass http://app:8000;
}
}
# -----------------------------------
- name: compose-app.yaml
kind: relative_path
path: ../data/compose-app.yaml
Let’s run the test plan:
rtf run test-plan.yaml
Output:
[+] up 3/3
✔ Network docker-compose-environment-config_default Created 0.0s
✔ Container docker-compose-environment-config-app-1 Healthy 0.6s
✔ Container docker-compose-environment-config-hello-world-1 Healthy 0.6s
scenario env var value from test plan
Response: Hello, World!
HTTP status: 200
[+] down 3/3
✔ Container docker-compose-environment-config-app-1 Removed 10.1s
✔ Container docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network docker-compose-environment-config_default Removed 0.1s
Note Inlining configuration to docker compose like this is an antipattern in RTF, as it requires you to supply a completely new docker compose YAML file if you want to update any part of its configuration. There is a better way to do this, using File Providers. We cover how to do this in the “Using file providers” guide.
Notice that we now also get Response: Hello, World!. Now, everything in our test is connecting as
expected! Let’s verify this by changing the response message using an environment variable.
First, let’s update our environment.yaml to define an environment variable for the docker compose
stack to use. We are not going to set this environment variable using RTF variables, instead we are
just going to hardcode it for ease. The “Writing a scenario” guide contains an example of
setting environment variables using RTF variables. Update environment.yaml:
name: Docker compose environment config
description: A docker compose environment config
# ------ Add an environment variable ------
env_vars:
HELLO_MESSAGE: "Goodbye, World!"
# -----------------------------------------
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
configs:
- source: nginx_conf
target: /etc/nginx/conf.d/default.conf
configs:
nginx_conf:
content: |
server {
listen 80;
location / {
proxy_pass http://app:8000;
}
}
- name: compose-app.yaml
kind: relative_path
path: ../data/compose-app.yaml
We also need to update compose-app.yaml to use this environment variable, instead of the hardcoded
value:
services:
app:
image: python:alpine
environment:
# ---- Replace hardcoded message with one from the env var ----
HELLO_MESSAGE: ${HELLO_MESSAGE}
# ------------------------------------------------------------
ports:
- 8000:8000
command:
- python
- -c
- |
from http.server import HTTPServer, BaseHTTPRequestHandler
import os
class H(BaseHTTPRequestHandler):
def do_GET(self):
msg = os.environ.get('HELLO_MESSAGE').encode()
self.send_response(200)
self.send_header('Content-Type', 'text/plain')
self.end_headers()
self.wfile.write(msg)
def log_message(self, *a):
pass
HTTPServer(('', 8000), H).serve_forever()
Note This is making use of docker compose variable interpolation.
Now, run the test plan:
rtf run test-plan.yaml
Output:
[+] up 3/3
✔ Network docker-compose-environment-config_default Created 0.0s
✔ Container docker-compose-environment-config-app-1 Healthy 0.6s
✔ Container docker-compose-environment-config-hello-world-1 Healthy 0.6s
scenario env var value from test plan
Response: Goodbye, World!
HTTP status: 200
[+] down 3/3
✔ Container docker-compose-environment-config-app-1 Removed 10.1s
✔ Container docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network docker-compose-environment-config_default Removed 0.1s
Our test plan has successfully used an environment variable from RTF to set configuration in our docker compose environment!
Next steps
In this guide, we moved environment config into its own file, added an additional compose service, and used RTF environment variables to configure the docker compose stack. Next, we’ll guide you through how to use file providers.
Using file providers
In this guide, we’ll use File Providers to manage files that our environment and scenario configs
depend on. We’ll move the nginx config from being inlined in docker compose to being managed by
RTF, then learn about the inline, relative_path, and required provider types.
Prerequisites
- Completed the “Writing an environment” tutorial
- An
rtf-hello-worlddirectory in the state it was at the end of that guide
Your directory should look like this:
ls -R
Output:
configs data scripts test-plan.yaml
./configs:
environment.yaml scenario.yaml
./data:
compose-app.yaml
./scripts:
check-status-v2.sh check-status.sh
What are file providers?
We’ve already been using File Providers in the scenario configuration throughout the previous
guides. File Providers are RTF’s way of referring to files and data that are required for the
environment and scenario to run successfully. The most commonly used File Providers are inline and
relative_path. They provide the file content either from the local filesystem or from within the
test plan configuration itself.
There are additional File Providers that, amongst other things, can pull data from APIs (such as GraphOS specific providers). These will not be discussed in this guide but can be seen in the framework reference.
All File Providers follow the same basic principle — they create one or more files and place them on a path for RTF to make use of. If you need to know that path (for your scenario docker command, for example), RTF will assign an environment variable that contains the file’s path. This is a deliberate design choice that allows RTF to make changes to how it stores files obtained from providers without breaking assumptions made about paths in user-written scripts.
Adding an inline file
Let’s see how File Providers can be used in our test plan config. In the
“Writing an environment” guide we created an nginx config file by writing it inline in the
compose-app.yaml file. This is an antipattern and forces us to write a completely new
compose-app.yaml file if we just want to amend the nginx config. Let’s fix that by using a file
provider to supply the config instead.
Let’s update our environment.yaml file to make use of the file_providers key:
name: Docker compose environment config
description: A docker compose environment config
env_vars:
HELLO_MESSAGE: "Goodbye, World!"
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
# ---- Replace the inline config with a volume ----
volumes:
- ${NGINX_CONFIG}:/etc/nginx/conf.d/default.conf
# -------------------------------------------------
- name: compose-app.yaml
kind: relative_path
path: ../data/compose-app.yaml
# ---- Add an inline file provider ----
file_providers:
- name: nginx.conf
env_var: NGINX_CONFIG
kind: inline
content: |
server {
listen 80;
location / {
proxy_pass http://app:8000;
}
}
# -------------------------------------
Note
compose_filesalso use a subset of File Providers. These do not set an environment variable that RTF can refer to since they are run using thedocker compose-fflag.
Now, run the test plan:
rtf run test-plan.yaml
Output:
[+] up 3/3
✔ Network docker-compose-environment-config_default Created 0.0s
✔ Container docker-compose-environment-config-app-1 Healthy 0.6s
✔ Container docker-compose-environment-config-hello-world-1 Healthy 0.6s
scenario env var value from test plan
Response: Goodbye, World!
HTTP status: 200
[+] down 3/3
✔ Container docker-compose-environment-config-app-1 Removed 10.1s
✔ Container docker-compose-environment-config-hello-world-1 Removed 0.1s
✔ Network docker-compose-environment-config_default Removed 0.1s
This is the exact same output as we got before making this change. All we have done is refactored where the config file is defined.
This works because RTF sets the path to the nginx.conf file it creates as an environment variable.
The docker compose file (which is also written inline to the test plan) uses this path to volume
mount the config file to the container.
Let’s examine the providers output:
ls output/providers/setup_providers
Output:
compose-app.yaml docker-compose.yaml nginx.conf
cat output/providers/setup_providers/nginx.conf
Output:
server {
listen 80;
location / {
proxy_pass http://app:8000;
}
}
We’ve successfully updated our test plan to write the nginx.conf file to the output with the
content we specified inline!
File provider config structure
Now that we’ve seen a File Provider being defined, let’s discuss how the config is structured. There are three required fields:
nameis the name the file will be saved with in theprovidersdirectory of the output.env_varis the environment variable the file’s path will be stored in. This is used by subsequent commands to refer to the file.kindis used to set which kind of File Provider is being used. See the framework reference for details on all the providers available.
Each File Provider will have other fields that need to be defined, like content for inline.
These are specific to each provider type, and the template command will highlight any missing or
incorrectly defined keys.
Adding a file from a relative path
Our nginx config is no longer defined inline to our docker compose file, but it is still inline
in our environment config. Let’s further reduce our inline dependencies by changing from using an
inline File Provider to using a relative_path File Provider. This allows us to define the
nginx.conf file in the local filesystem and direct RTF to use that file.
First, let’s create the nginx.conf file:
touch data/nginx.conf
Then, add the config to that file:
server {
listen 80;
location / {
proxy_pass http://app:8000;
}
}
Let’s update environment.yaml to refer to this file, this time using the relative_path file
provider:
name: Docker compose environment config
description: A docker compose environment config
env_vars:
HELLO_MESSAGE: "Goodbye, World!"
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
volumes:
- ${NGINX_CONFIG}:/etc/nginx/conf.d/default.conf
- name: compose-app.yaml
kind: relative_path
path: ../data/compose-app.yaml
file_providers:
- name: nginx.conf
env_var: NGINX_CONFIG
# --- Replace inline provider with relative_path ---
kind: relative_path
path: ../data/nginx.conf
# --------------------------------------------------
Note The path is relative to the
environment.yamlfile!
Now, let’s run the test plan:
rtf run test-plan.yaml
The output should be exactly as it was when we defined this file inline. Likewise, if we examine the output directory:
ls output/providers/setup_providers
Output:
compose-app.yaml docker-compose.yaml nginx.conf
cat output/providers/setup_providers/nginx.conf
Output:
server {
listen 80;
location / {
proxy_pass http://app:8000;
}
}
As expected, we also see the nginx.conf file with the same content as before.
Required files
As discussed in previous sections of this guide, the environment and scenario configs are designed
to be reused. For config designed to be reused often, you might want to force the user of the test
plan to define a file, but not give them a default file to work with. For this use case, the
required File Provider is the perfect solution. Any test plan that tries to execute with a
required File Provider will fail and tell the user to specify the file themselves (normally this
is done using overrides). Let’s walk through how required can be used:
Let’s update our environment to require the user supply the nginx.conf file. We don’t want to give
a base example because users might end up using that without thinking about what configuration they
actually need for their specific test. In other words, we don’t want it to “just work” by design.
Let’s add a required file to our environment.yaml:
name: Docker compose environment config
description: A docker compose environment config
env_vars:
HELLO_MESSAGE: "Goodbye, World!"
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
volumes:
- ${NGINX_CONFIG}:/etc/nginx/conf.d/default.conf
- name: compose-app.yaml
kind: relative_path
path: ../data/compose-app.yaml
file_providers:
- name: nginx.conf
env_var: NGINX_CONFIG
# --- Replace relative_path provider with required ---
kind: required
message: Please specify an nginx config file
# --------------------------------------------------
Now, let’s see what happens when we try to template this test plan:
rtf template test-plan.yaml --check
Output:
ERROR Static analysis checks failed
(environment.file_providers.NGINX_CONFIG) A required file has not been defined
Please specify an nginx config file
We get an error saying we haven’t defined a required file, along with the message we put in the
message field. We’d get the same error if we tried rtf run.
To make this work, the test plan user should make use of overrides. Let’s update test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
variables:
scenario_variable: "scenario env var value from test plan"
scenario:
from:
kind: local
relative_path: configs/scenario.yaml
overrides:
file_providers:
- name: check-status.sh
env_var: CHECK_STATUS_SCRIPT
kind: relative_path
path: scripts/check-status-v2.sh
environment:
from:
kind: local
relative_path: configs/environment.yaml
# --- Override the nginx config ---
overrides:
file_providers:
- name: nginx.conf
env_var: NGINX_CONFIG
kind: relative_path
path: data/nginx.conf
# --------------------------------------
The override will match based on the name key. If we template now:
rtf template test-plan.yaml --check
You should see the full templated output. If you look at environment.file_providers, you can see
that nginx.conf now uses the config from the overrides. Our test plan no longer contains a
required file, so we no longer get that error. The rtf run command can also complete
successfully now.
Next steps
You’ve now completed the test plan tutorials! You’ve worked through writing a test plan, scenario, and environment from scratch, and learned how to use File Providers to manage external files.
To go further, explore these topics:
- Custom providers — learn how to write your own providers to pull data from external sources
- Script-based test plans — an alternative to docker-based scenarios and environments, for cases where docker isn’t available
- Framework reference — the full reference for all config fields and provider types
Running test plans with the RTF Orchestrator Service
This section guides you through running RTF test plans using the RTF Orchestrator Service: a
managed, remote execution service for running RTF Test Plans at Apollo. Instead of running your
Test Plan locally with rtf run, you submit it to the Orchestrator which provisions an
isolated Kubernetes namespace for deploying your Environment before then executing your
Scenario and storing the results in GCS for you to retrieve.
Why use the Orchestrator?
When you run a Test Plan with rtf run, RTF handles spinning up your test environment and running
your scenario on your local machine. When using scripted environments and scenarios this is
incredibly flexible but also highly susceptible to being affected by how that local machine is
configured. Running locally using a docker compose based environment and docker based scenario
helps with making things more reproducible, but you are still subject to the constraints of the
local machine you are running on.
In contrast, the Orchestrator provides a dedicated execution environment that handles running your test plan in an isolated Kubernetes namespace. This provides several advantages:
- Minimal local requirements: the machine triggering the test run only needs to be able to submit the Test Plan to the Orchestrator, not run the services under test.
- Output storage: logs and output artifacts are uploaded to GCS and can be retrieved by anyone with access to the Orchestrator after the run completes.
- Consistent environments: every execution runs in an isolated namespace, giving far more consistent performance.
- Parallel matrix execution: matrix dimensions run as independent executions managed by the Orchestrator concurrently, leading to a significant speed up in wall-clock execution time.
Accessing the Orchestrator
The Orchestrator is deployed at https://api.rtf.apollographql.com and is protected by Google Cloud
IAP. Access is managed in GCP by the Runtime Readiness team.
Once access is granted, you can authenticate with GCP using the following command:
gcloud auth application-default login
You can check whether or not you have access by attempting to hit the healthcheck endpoint on the
Orchestrator using the rtf CLI like so:
rtf remote request health
If you see {"ok":true} in your terminal then you are correctly authenticated. If you are unable to
reach the Orchestrator, reach out to the Runtime Readiness team in Slack and we will set up access
for you and your team.
Prerequisites
- you have completed the “Writing test plans” tutorial series
- the
rtfCLI installed- the
gcloudCLI installed- you have authenticated with
gcloud auth application-default login- you have validated your access to the Orchestrator using
rtf remote request health
Next: Making your test plan Orchestrator-ready
Making your test plan Orchestrator-ready
In this guide, we’ll take the Test Plan from the “Writing test plans” tutorial series and
update it so it can be run under the RTF Orchestrator. To do this we will be adding rtf.io docker
compose labels that tell the Orchestrator how to handle deploying and interacting with your
services.
Prerequisites
- Completed the “Writing test plans” tutorial series
- An
rtf-hello-worlddirectory in the state it was at the end of that series
Restrictions on Orchestrator Test Plans
The Orchestrator enforces that Test Plans submitted to it are compatible for running in a Kubernetes cluster.
Compatible Environments are:
docker composebased. The Orchestrator converts yourdocker composebased environments into Kubernetes resources using kompose which are then patched with kustomize according to the labels detailed below.null(if theskip = truefield is set in the Environment config). This skips environment provisioning in the Orchestrator and just runs the Scenario.
The only compatible Scenario is a docker based one.
The Test Plan you built in the “Writing test plans” series already uses a docker compose
based Environment and docker based Scenario, so all we need to add is the appropriate labels and
it will be ready to run!
The rtf.io labels
The Orchestrator uses docker compose service labels in the rtf.io namespace to determine how it
should handle your services during deployment and execution. These labels have no effect when you
run a Test Plan locally with rtf run: they exist solely to allow the Orchestrator to replicate the
execution behaviour of rtf run in Kubernetes where we can’t rely on shared local filesystem.
Currently there are three labels available:
| Label | Value | Effect |
|---|---|---|
rtf.io/file-providers | true | Mounts file provider output into the container. Required for services that read provider files at runtime. |
rtf.io/log-collection | true | Container logs are uploaded to GCS after the run. Absent by default, logs are not collected unless opted in. |
rtf.io/otel | true | Injects RTF collector endpoints into the container as environment variables. The following variables are set automatically: RTF_OTEL_COLLECTOR_GRPC (gRPC endpoint, port 4317) and RTF_OTEL_COLLECTOR_HTTP (HTTP/protobuf endpoint, port 4318). Use these in your service’s config instead of the standard OTEL_* variables. |
Adding log collection
When set to "true" on a service, the rtf.io/log-collection label will instruct the orchestrator
to collect that service’s container logs after the Scenario completes and includes them in the
output zip file that gets pushed to GCS under output/logs/.
Let’s add it to the hello-world service in our environment.yaml:
name: Docker compose environment config
description: A docker compose environment config
compose_files:
- name: docker-compose.yaml
kind: inline
content: |
services:
hello-world:
image: nginx:alpine
ports:
- "8080:80"
# -------- Add rtf.io labels --------
labels:
rtf.io/log-collection: "true"
# -----------------------------------
Note Docker compose label values must be strings. Unquoted
trueis interpreted as a boolean and will not match the expected string value"true". Always quote the value.
Remember: there is no automatic log collection. You must add the rtf.io/log-collection label to
each service you want to collect logs from.
Accessing file provider output
Any service that needs to access the output from RTF file providers needs to be annotated with
the rtf.io/file-providers: "true" label. This instructs the Orchestrator to add an init container
that will resolve and mount the required file provider output into the container via a shared
volume. As with rtf run the environment variables you specify in your test plan will contain the
correct absolute path for the requested resources, regardless of whether you run locally or under
the Orchestrator.
services:
router:
image: "ghcr.io/apollographql/router:v2.14.0"
labels:
rtf.io/log-collection: "true"
rtf.io/file-providers: "true"
# The SUPERGRAPH_SCHEMA and ROUTER_CONFIG environment variables here
# are set according to the file providers that have been added using
# the label.
command: -s ${SUPERGRAPH_SCHEMA} -c ${ROUTER_CONFIG}
Note A service using
rtf.io/file-providersmust not attempt to define explicit volume mounts for individual provider paths. Doing so will cause the Orchestrator to reject your Test Plan as invalid.
Verifying the test plan
As with local execution, we run rtf template to confirm the updated test plan still parses
correctly:
rtf template test-plan.yaml --check
The output should be similar to:
name: Hello World
description: A test plan created as a guide for writing test plans
variables:
scenario_variable: scenario env var value from test plan
matrix:
variant_names: null
dimensions: {}
compound: {}
custom_providers: []
scenario:
...
environment:
...
Your test plan is now ready to submit to the Orchestrator.
Next steps
In this guide, we added rtf.io labels to our docker compose environment so the Orchestrator knows
how to handle our services. Next, we’ll submit the Test Plan to the Orchestrator for execution and
look at how we monitor the status of the execution before fetching the results.
Running and fetching results
In this guide, we’ll submit our Test Plan to the RTF Orchestrator Service using
rtf remote ci-run, watch it progress through the execution lifecycle, and then fetch the results
using rtf remote run-output.
Prerequisites
- Completed the “Making your test plan Orchestrator-ready” guide
gcloudCLI installed and authenticated withgcloud auth application-default login- Access to the Orchestrator at
https://api.rtf.apollographql.comgranted by the Runtime Readiness team
Doing things manually
For actual execution we recommend using the rtf remote ci-run subcommand which handles all of the
following steps for you automatically. But before we take a look at how to use that subcommand,
we’ll first explain what that is doing under the hood so you have an understanding of what is
happening when the Orchestrator executes a Test Plan.
Preparing the Trigger Payload
Before a Test Plan can be submitted to the Orchestrator, it must be converted into a
Trigger Payload. This is simply a JSON representation of the Test Plan in the inline form
generated by rtf template, along with any local files that are referenced by file providers
contained in the Test Plan.
To build the payload manually we use the rtf remote prepare command:
rtf remote prepare test-plan.yaml
rtf remote prepare writes its output as a single JSON object to stdout, allowing you to pipe it
directly to other tools or inspect it with jq. You should not edit this payload by hand before
submitting to the Orchestrator as this is likely to result in broken test runs.
Triggering a run with rtf remote request
To manually trigger a Test Run we can use the rtf remote request subcommand that allows for
making individual requests to the Orchestrator. The trigger request is made using the payload we
covered in the previous section like so:
rtf remote request -X POST --body "$(rtf remote prepare test-plan.yaml)" test-run/trigger
Provided the Test Plan is accepted, this will return a Test Run Summary that details the
current status of our newly created test run. We won’t go into the details of everything that is
contained in the summary just now, as all we need for the next step is the ID of our run which we
can extract from the summary by piping it through jq and using the following filter:
| jq -r '.id'
Polling for status updates
Armed with our test run ID we can use the test-run/$id/status endpoint to see what is happening as
our run is processed by the Orchestrator:
rtf remote request "test-run/$RUN_ID/status" | jq
This endpoint returns the same Test Run Summary data as the trigger endpoint, but with the
details being pulled fresh each time we make the request. Using this we can check the value of the
current_status field to see how our run is progressing (see “The execution lifecycle” below).
Once our run enters a terminal state we are able to try pulling the results for analysis. But before
we take a look at that, let’s introduce the rtf remote ci-run subcommand that automates the
process we’ve just implemented by hand.
Triggering a run with rtf remote ci-run
rtf remote ci-run is the primary command for running a Test Plan through the Orchestrator. It
automates all of the steps we just covered, preparing the Trigger Payload, submitting it for
execution, and polling for status updates until the run completes.
rtf remote ci-run test-plan.yaml
The command prints a status table that updates at each poll interval (default: 10 seconds) showing the most recent activity associated with your run:
status latest_message running successful failed unrunnable
INITIALISING run created 1 0 0 0
RESOLVING resolving environment config 1 0 0 0
PROVISIONING deploying environment 1 0 0 0
RUNNING running scenario 1 0 0 0
SUCCESSFUL scenario completed successfully 0 1 0 0
Test run complete. Final status: successful
Run 'rtf remote request test-run/<run-id>/status' to view the summary for this run
Run the following to fetch the status, log or output.zip for an execution:
rtf remote request test-execution/$ID/status
rtf remote request test-execution/$ID/log.txt
rtf remote request test-execution/$ID/output.zip > output.zip
The execution lifecycle
The status column from that table shows the current execution status of the run as a whole. Each
test run (and test execution inside of a run) progresses through the following statuses as it
executes:
| Status | Description |
|---|---|
INITIALISING | The run has been accepted and executions are being queued |
RESOLVING | The Orchestrator is resolving the environment configuration |
PROVISIONING | The environment namespace is being created and services are being deployed |
ENVIRONMENT_READY | The environment is healthy and the Scenario is starting |
RUNNING | The Scenario job is running |
SUCCESSFUL | The Scenario exited cleanly |
FAILED | The Scenario exited with a non-zero exit code |
UNRUNNABLE | The execution could not be started (for example, due to an environment failure) |
The running column in the table counts executions that are not yet in a terminal state. For matrix
Test Plans, the table shows the aggregate count across all executions.
rtf remote ci-run exits with:
0if all executions wereSUCCESSFUL1if any executionFAILED2if any execution wasUNRUNNABLE
This makes it suitable for use in CI pipelines, allowing a non-zero exit code to fail the pipeline when a test run does not succeed.
Adjusting the poll interval
You can use --poll-interval-seconds to change how frequently the command polls for status updates:
rtf remote ci-run test-plan.yaml --poll-interval-seconds 30
Fetching output for a single execution
Once our run is complete, we can use rtf remote execution-output to fetch the log and output zip
for a specific execution. We can find the execution ID in the output of rtf remote ci-run, or by
querying the run status with rtf remote request test-run/<run-id>/status.
rtf remote execution-output <execution-id>
This writes the following to a newly created output/ directory by default:
output/
execution-summary.json # JSON summary of the execution status and history
log.txt # Orchestrator log for the execution
output.zip # Zipped output directory from the Scenario and service logs
The output.zip contains:
output/— files written to$OUTPUT_PATHby the Scenario containeroutput/logs/<pod>/<container>.txt— container logs for services withrtf.io/log-collection: "true"
If we want to write output to a different directory, we can use --outdir:
rtf remote execution-output <execution-id> --outdir my-results
If the output directory already exists, the command exits with an error. We can use --force to
remove it and start fresh:
rtf remote execution-output <execution-id> --force
Fetching output for all executions in a run
For matrix Test Plans, we can use rtf remote run-output to fetch the output for every execution in
our run in parallel; it groups each execution’s output into a named subdirectory:
rtf remote run-output <run-id>
This produces:
output/
run-summary.json # JSON summary of the full run status
<execution-name>/
log.txt
output.zip
<execution-name>/
log.txt
output.zip
...
We can use the same --outdir and --force flags as with rtf remote execution-output.
Making raw API requests
As covered above, rtf remote request lets us make arbitrary authenticated requests to the
Orchestrator directly, for situations where higher level commands such as rtf remote ci-run don’t
provide the functionality we need:
rtf remote request test-run/<run-id>/status
rtf remote request test-execution/<execution-id>/log.txt
The response body for requests we make this way is always written to stdout. We can use -X to
specify a method and -b to supply a request body. See the CLI reference for the full set of
options.
Next steps
You’ve now completed the “Running with the Orchestrator” tutorial series: you can submit a Test Plan to the Orchestrator, monitor it through to completion, and fetch its results. To go further:
- CLI reference — full reference for
rtf remoteand its subcommands - Troubleshooting — common issues and how to resolve them
Writing custom providers
This section will guide you through writing a custom provider for rtf from a blank file. The custom provider will be a simple example that introduces you to the concepts of writing RTF custom providers and why they are helpful. For examples of custom providers that can be used for specific use cases, please refer to the morgue.
Prerequisites
We recommend completing the “Writing test plans” guide first to ensure you’re familiar with rtf test plans. Many of the concepts in that guide will be reused here.
Next: Writing a custom provider definition
Writing a custom provider definition
Create an empty directory and make it your working directory:
mkdir rtf-custom-provider
cd rtf-custom-provider
Create an empty YAML file for the custom provider definition:
touch my-provider.yaml
We will now step through creating a simple custom provider definition.
Adding required fields
A custom provider definition has three required fields: name, description, and command. The
sections below add each of these required fields and explain them in more detail.
name
Add the name field to the my-provider.yaml file:
name: env-generator
name is used to give each custom provider an identifiable title. It can be any valid string. The
value used for the name field has no impact on the execution of the custom provider. This makes it
easier to work with the custom provider programmatically.
description
Add the description field to the my-provider.yaml file:
name: env-generator
description: |
Generates an ENV_VARS.txt file with configuration for downstream scripts.
Output directory contains:
- ENV_VARS.txt (environment variables for setup scripts)
- base-config.txt (a base configuration file)
description is used to give more information about the custom provider for future users. It can be
any valid string. The value used for the description field has no impact on the execution of the
custom provider itself. This is a useful place to document what files the provider generates and how
they should be used.
command
The command is used to define what will be executed when the custom provider runs. The command is
responsible for generating output files in the $RTF_OUTPUT directory.
Add the command field to the my-provider.yaml file:
name: env-generator
description: |
Generates an ENV_VARS.txt file with configuration for downstream scripts.
Output directory contains:
- ENV_VARS.txt (environment variables for setup scripts)
- base-config.txt (a base configuration file)
command:
name: generate.sh
kind: relative_path
path: scripts/generate.sh
This references a script file that we will create in the custom provider commands section. Before we can run the custom provider, we need to create this script.
Custom provider commands
Before we can run the custom provider, we need to create the script referenced in the command
section. Create a scripts directory and the generate.sh file:
mkdir scripts
touch scripts/generate.sh
Add the following content to scripts/generate.sh:
#!/usr/bin/env bash
set -e
mkdir -p "$RTF_OUTPUT"
cat > "$RTF_OUTPUT/ENV_VARS.txt" << EOF
# Environment variables for project setup
# Generated by env-generator provider
export PROJECT_NAME="default-project"
export LOG_LEVEL="info"
EOF
echo "Generated environment config in $RTF_OUTPUT"
This script does the following:
- Creates the output directory using the
$RTF_OUTPUTenvironment variable, which is automatically set by rtf to the directory where output files should be written. All custom provider command scripts must write to this location. - Writes an
ENV_VARS.txtfile with environment variable exports that can be sourced by downstream scripts. This is not a required pattern for a custom provider but can be useful in many use cases which is why it is included in this guide. - Prints a message confirming the generation was successful.
The key difference between a command in a custom provider and one in a config file is that the command in a custom provider MUST write one or more files to RTF_OUTPUT. Commands in config files perform an action, for example, setting up an environment, whereas commands in a custom provider write files for config files to make use of.
Make the script executable:
chmod +x scripts/generate.sh
Your directory structure should now look like this:
ls -R
my-provider.yaml scripts
./scripts:
generate.sh
Checking the custom provider
Now, let’s check that the custom provider has been defined correctly:
rtf custom-provider template my-provider.yaml
This should result in the fully templated custom provider being printed to the terminal:
name: env-generator
description: |
Generates an ENV_VARS.txt file with configuration for downstream scripts.
Output directory contains:
- ENV_VARS.txt (environment variables for setup scripts)
- base-config.txt (a base configuration file)
variable_definitions: []
command:
name: generate.sh
kind: relative_path
path: scripts/generate.sh
args: []
env_vars: {}
file_providers: []
This highlights the optional fields for custom providers that have not yet been used:
variable_definitions, env_vars, and file_providers. These are discussed in the following
sections.
Running the custom provider
Before looking at the optional fields, let’s run the custom provider:
rtf custom-provider run my-provider.yaml
You should see output similar to this:
Generated environment config in /path/to/rtf-custom-provider/output/RTF_OUTPUT
The output directory contains the generated files. Let’s examine what was created:
cat output/RTF_OUTPUT/ENV_VARS.txt
Output:
# Environment variables for project setup
# Generated by env-generator provider
export PROJECT_NAME="default-project"
export LOG_LEVEL="info"
Remove the output directory before continuing:
rm -rf output/
Note rtf is deliberately configured to not overwrite an existing output directory. This is so you cannot accidentally overwrite output you intend to keep. The
--outputflag can be used withrtf custom-provider runto set a different output directory if you want to keep the existing output and run a new test.
Defining variables
The variable_definitions field is used to define parameters that the custom provider accepts. When
the custom provider is used in a test plan, these variables are passed as arguments. The
using a custom provider guide covers how to pass arguments to a custom
provider in detail.
For now, let’s add a variable to make the project name configurable. We are also going to update the
script to use this variable. The env_vars field is used to pass variable values to the command as
environment variables.
Update my-provider.yaml:
name: env-generator
description: |
Generates an ENV_VARS.txt file with configuration for downstream scripts.
Output directory contains:
- ENV_VARS.txt (environment variables for setup scripts)
- base-config.txt (a base configuration file)
# --- Add variable definitions ---
variable_definitions:
- name: project_name
description: "The name of the project"
# --------------------------------
command:
name: generate.sh
kind: relative_path
path: scripts/generate.sh
# --- Add env_vars to pass variables to the command ---
env_vars:
PROJECT_NAME: "{{ project_name }}"
# -----------------------------------------------------
Update scripts/generate.sh to use the environment variable:
#!/usr/bin/env bash
set -e
mkdir -p "$RTF_OUTPUT"
cat > "$RTF_OUTPUT/ENV_VARS.txt" << EOF
# Environment variables for project setup
# Generated by env-generator provider
export PROJECT_NAME="$PROJECT_NAME"
EOF
echo "Generated environment config in $RTF_OUTPUT"
Now run the custom provider with the --var flag to set the variable:
rtf custom-provider run my-provider.yaml --var project_name=my-app
Output:
Generated environment config in /path/to/rtf-custom-provider/output/RTF_OUTPUT
cat output/RTF_OUTPUT/ENV_VARS.txt
Output:
# Environment variables for project setup
# Generated by env-generator provider
export PROJECT_NAME="my-app"
The project_name variable is now configurable. Remove the output directory before continuing:
rm -rf output/
Adding a default value
Variables can have default values, like in config files. Let’s add a log_level variable with a
default value of "info".
Update my-provider.yaml:
name: env-generator
description: |
Generates an ENV_VARS.txt file with configuration for downstream scripts.
Output directory contains:
- ENV_VARS.txt (environment variables for setup scripts)
- base-config.txt (a base configuration file)
variable_definitions:
- name: project_name
description: "The name of the project"
# --- Add a variable with a default ---
- name: log_level
description: "Logging verbosity level"
default: "info"
# -------------------------------------
command:
name: generate.sh
kind: relative_path
path: scripts/generate.sh
env_vars:
PROJECT_NAME: "{{ project_name }}"
# --- Add LOG_LEVEL ---
LOG_LEVEL: "{{ log_level }}"
# ---------------------
Update scripts/generate.sh:
#!/usr/bin/env bash
set -e
mkdir -p "$RTF_OUTPUT"
cat > "$RTF_OUTPUT/ENV_VARS.txt" << EOF
# Environment variables for project setup
# Generated by env-generator provider
export PROJECT_NAME="$PROJECT_NAME"
export LOG_LEVEL="$LOG_LEVEL"
EOF
echo "Generated environment config in $RTF_OUTPUT"
Now run the custom provider without specifying log_level:
rtf custom-provider run my-provider.yaml --var project_name=my-app
Output:
Generated environment config in /path/to/rtf-custom-provider/output/RTF_OUTPUT
cat output/RTF_OUTPUT/ENV_VARS.txt
Output:
# Environment variables for project setup
# Generated by env-generator provider
export PROJECT_NAME="my-app"
export LOG_LEVEL="info"
The default value "info" is used. You can override it by passing --var log_level=debug.
Remove the output directory before continuing:
rm -rf output/
Using file providers
The file_providers field is used to make files available to the command script. Each file provider
creates a file and sets an environment variable containing the path to that file. This allows the
command script to read input files or copy template files to the output directory.
Let’s add a base configuration file that the command will copy to the output. Update
my-provider.yaml:
name: env-generator
description: |
Generates an ENV_VARS.txt file with configuration for downstream scripts.
Output directory contains:
- ENV_VARS.txt (environment variables for setup scripts)
- base-config.txt (a base configuration file)
variable_definitions:
- name: project_name
description: "The name of the project"
- name: log_level
description: "Logging verbosity level"
default: "info"
command:
name: generate.sh
kind: relative_path
path: scripts/generate.sh
# --- Add file providers ---
file_providers:
- name: base-config.txt
env_var: BASE_CONFIG
kind: inline
content: |
# Base configuration
# Project-specific values set via ENV_VARS.txt
# -------------------------
env_vars:
PROJECT_NAME: "{{ project_name }}"
LOG_LEVEL: "{{ log_level }}"
The file provider defines:
name: The filename used when the file is written to the providers directory.env_var: The environment variable that will contain the path to the file.kind: The type of file provider. Here we useinlineto define the content directly.content: The file content (forinlineproviders).
Update scripts/generate.sh to copy the base config to the output:
#!/usr/bin/env bash
set -e
mkdir -p "$RTF_OUTPUT"
cat > "$RTF_OUTPUT/ENV_VARS.txt" << EOF
# Environment variables for project setup
# Generated by env-generator provider
export PROJECT_NAME="$PROJECT_NAME"
export LOG_LEVEL="$LOG_LEVEL"
export BASE_CONFIG="$RTF_OUTPUT/base-config.txt"
EOF
# Copy the base config to output
cp "$BASE_CONFIG" "$RTF_OUTPUT/base-config.txt"
echo "Generated environment config in $RTF_OUTPUT"
Now run the custom provider:
rtf custom-provider run my-provider.yaml --var project_name=my-app
Output:
Generated environment config in /path/to/rtf-custom-provider/output/RTF_OUTPUT
cat output/RTF_OUTPUT/base-config.txt
Output:
# Base configuration
# Project-specific values set via ENV_VARS.txt
The base configuration file is now included in the output. Remove the output directory before continuing:
rm -rf output/
This completes the custom provider definition.
In this guide we have covered writing a simple custom provider definition. Next, we will guide you through how to use it in a test plan.
Next: Using a custom provider
Using a custom provider
This guide assumes you’ve completed the
“Writing a custom provider definition” guide. You should
already have the files in a directory named rtf-custom-provider. Your directory should be in the
state it was at the end of that guide.
ls -R
my-provider.yaml scripts
./scripts:
generate.sh
You should also have no output directory. If you do, remove it before continuing:
rm -rf output/
Creating a test plan
Now that we have a custom provider, let’s use it in a test plan. Create an empty test plan file:
touch test-plan.yaml
Add the following content to test-plan.yaml:
name: Custom Provider Test
description: A test plan that uses the env-generator custom provider
custom_providers:
- kind: local
relative_path: .
using:
env_generator: my-provider.yaml
scenario:
inline:
name: Empty scenario
description: A scenario that does nothing
command:
name: scenario.sh
kind: inline
content: |
#!/usr/bin/env sh
environment:
inline:
name: Custom provider environment
description: An environment that uses the env-generator custom provider
setup:
command:
name: setup.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "=== ENV_VARS.txt contents ==="
source "$ENV_GENERATOR/ENV_VARS.txt"
echo "PROJECT_NAME=$PROJECT_NAME"
echo "LOG_LEVEL=$LOG_LEVEL"
echo ""
echo "=== base-config.txt contents ==="
cat "$ENV_GENERATOR/base-config.txt"
file_providers:
- name: env-generator
env_var: ENV_GENERATOR
kind: custom_provider
type: env_generator
teardown:
command:
name: teardown.sh
kind: inline
content: |
#!/usr/bin/env sh
This test plan does the following:
- The
custom_providerssection registers the custom provider definition. Each entry specifies:kind: local: indicates the provider definition is in the local filesystem.relative_path: the directory containing the provider definition file.using: a map of names to provider definition files. The name (env_generator) is used to reference the provider in file providers.
- The
scenariois an empty command that does nothing. This is acceptable for testing the custom provider. - The
environment.setupuses the custom provider as a file provider. Thefile_providersentry specifies:name: a name for this file provider instance.env_var: the environment variable that will contain the path to the provider’s output directory.kind: custom_provider: indicates this is a custom provider.type: the name registered incustom_providers(in this case,env_generator).
- The setup command sources the
ENV_VARS.txtfile using$ENV_GENERATOR/ENV_VARS.txtand prints the values it sets, then prints the contents ofbase-config.txt. - The
teardownis an empty command that does nothing.
First template attempt
Let’s check if the test plan templates correctly:
rtf template test-plan.yaml
This fails with an error:
ERROR Templating failed
(environment.setup.file_providers.ENV_GENERATOR.env_vars.PROJECT_NAME) Unknown templating variable
project_name
This error occurs because the env-generator custom provider has a project_name variable with no
default value. Notice that log_level is not in the error because it has a default value of
"info".
Adding arguments to the custom provider
To fix this, we need to provide the required project_name argument. Arguments are added as
additional fields on the file provider. Update the file_providers section in test-plan.yaml:
name: Custom Provider Test
description: A test plan that uses the env-generator custom provider
custom_providers:
- kind: local
relative_path: .
using:
env_generator: my-provider.yaml
scenario:
inline:
name: Empty scenario
description: A scenario that does nothing
command:
name: scenario.sh
kind: inline
content: |
#!/usr/bin/env sh
environment:
inline:
name: Custom provider environment
description: An environment that uses the env-generator custom provider
setup:
command:
name: setup.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "=== ENV_VARS.txt contents ==="
source "$ENV_GENERATOR/ENV_VARS.txt"
echo "PROJECT_NAME=$PROJECT_NAME"
echo "LOG_LEVEL=$LOG_LEVEL"
echo ""
echo "=== base-config.txt contents ==="
cat "$ENV_GENERATOR/base-config.txt"
file_providers:
- name: env-generator
env_var: ENV_GENERATOR
kind: custom_provider
type: env_generator
# --- Add the required argument ---
project_name: my-test-project
# ---------------------------------
teardown:
command:
name: teardown.sh
kind: inline
content: |
#!/usr/bin/env sh
Now run the template command again:
rtf template test-plan.yaml
This time the command succeeds and outputs the fully templated test plan. The custom provider’s arguments are resolved and used to template the provider definition.
Running the test plan
Let’s run the test plan to see the custom provider in action:
rtf run test-plan.yaml
You should see output similar to this:
Generated environment config in /path/to/output/providers/setup_providers/env-generator
=== ENV_VARS.txt contents ===
PROJECT_NAME=my-test-project
LOG_LEVEL=info
=== base-config.txt contents ===
# Base configuration
# Project-specific values set via ENV_VARS.txt
This confirms:
- The custom provider ran and generated its output files.
- The
PROJECT_NAMEis set to the value we provided as an argument. - The
LOG_LEVELuses the default value of"info". - The
base-config.txtfile is available to the setup command.
Remove the output directory before continuing:
rm -rf output/
Overriding the default value
You can override the default log_level value by adding it as an argument. Update the
file_providers section:
file_providers:
- name: env-generator
env_var: ENV_GENERATOR
kind: custom_provider
type: env_generator
project_name: my-test-project
# --- Override the default ---
log_level: debug
# ----------------------------
Run the test plan again:
rtf run test-plan.yaml
Output:
Generated environment config in /path/to/output/providers/setup_providers/env-generator
=== ENV_VARS.txt contents ===
PROJECT_NAME=my-test-project
LOG_LEVEL=debug
=== base-config.txt contents ===
# Base configuration
# Project-specific values set via ENV_VARS.txt
The LOG_LEVEL is now "debug" instead of the default "info".
Remove the output directory before continuing:
rm -rf output/
Using test plan variables for arguments
Instead of hardcoding values in the arguments, you can use test plan variables. This allows the same
test plan to be run with different configurations. Update test-plan.yaml:
name: Custom Provider Test
description: A test plan that uses the env-generator custom provider
# --- Add variables ---
variables:
project_name: my-variable-project
# ---------------------
custom_providers:
- kind: local
relative_path: .
using:
env_generator: my-provider.yaml
scenario:
inline:
name: Empty scenario
description: A scenario that does nothing
command:
name: scenario.sh
kind: inline
content: |
#!/usr/bin/env sh
environment:
inline:
name: Custom provider environment
description: An environment that uses the env-generator custom provider
# --- Add variable_definitions to allow templating ---
variable_definitions:
- name: project_name
description: The project name passed to the custom provider
# ----------------------------------------------------
setup:
command:
name: setup.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "=== ENV_VARS.txt contents ==="
source "$ENV_GENERATOR/ENV_VARS.txt"
echo "PROJECT_NAME=$PROJECT_NAME"
echo "LOG_LEVEL=$LOG_LEVEL"
echo ""
echo "=== base-config.txt contents ==="
cat "$ENV_GENERATOR/base-config.txt"
file_providers:
- name: env-generator
env_var: ENV_GENERATOR
kind: custom_provider
type: env_generator
# --- Use templating for the argument value ---
project_name: "{{ project_name }}"
log_level: debug
# ---------------------------------------------
teardown:
command:
name: teardown.sh
kind: inline
content: |
#!/usr/bin/env sh
Run the test plan:
rtf run test-plan.yaml
Output:
Generated environment config in /path/to/output/providers/setup_providers/env-generator
=== ENV_VARS.txt contents ===
PROJECT_NAME=my-variable-project
LOG_LEVEL=debug
=== base-config.txt contents ===
# Base configuration
# Project-specific values set via ENV_VARS.txt
The PROJECT_NAME is now set from the test plan variable. You can also override this at runtime
using the --var flag:
rtf run test-plan.yaml --var project_name=runtime-override
Output:
Generated environment config in /path/to/output/providers/setup_providers/env-generator
=== ENV_VARS.txt contents ===
PROJECT_NAME=runtime-override
LOG_LEVEL=debug
=== base-config.txt contents ===
# Base configuration
# Project-specific values set via ENV_VARS.txt
In this guide, we have covered how to use a custom provider in a test plan, pass arguments to it, and use test plan variables for dynamic configuration.
Writing script-based test plans
This section guides you through writing a script-based Test Plan from a blank file. Script
environments and scenarios run shell commands directly on the host instead of using docker compose
and containers.
When to use script-based test plans
The docker compose tutorial is the recommended starting point. Script-based test plans are for cases where docker isn’t available or isn’t suitable — for example, when testing infrastructure that already manages its own lifecycle, or in environments where docker is not permitted.
Prerequisites
- Completed the “Writing test plans” tutorial
- RTF CLI installed and available in your terminal
Next: Writing a test plan
Writing a test plan
In this guide, we’ll write a script-based Test Plan from scratch. The Test Plan structure is
identical to the docker tutorial — the differences are in the Scenario and Environment configs,
which will use commands instead of docker.
Create a working directory:
mkdir rtf-hello-world
cd rtf-hello-world
mkdir configs
Create the three config files:
touch test-plan.yaml configs/scenario.yaml configs/environment.yaml
Add the following to test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
scenario:
from:
kind: local
relative_path: configs/scenario.yaml
environment:
from:
kind: local
relative_path: configs/environment.yaml
Add the following to configs/scenario.yaml:
name: Script scenario
description: A script scenario
command:
name: scenario.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "scenario executed"
Add the following to configs/environment.yaml:
name: Script environment
description: A script environment
setup:
command:
name: setup.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "environment setup"
teardown:
command:
name: teardown.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "environment teardown"
Scenario config
A script Scenario uses command: instead of docker:. The command: defines a shell script
that runs directly on the host. We cover command: in detail in the Writing a command guide.
Environment config
A script Environment uses setup: and teardown: instead of compose_files:. Each phase
defines a command: that runs a shell script on the host, bracketing the Scenario’s execution. See
the framework reference for the full Environment config structure.
Checking the Test Plan
The rtf template command works the same way as in the docker tutorial. Run it to confirm the Test
Plan templates correctly:
rtf template test-plan.yaml --check
Running the Test Plan
rtf run test-plan.yaml
Output:
environment setup
scenario executed
environment teardown
RTF runs the Environment setup command, then the Scenario, then the Environment teardown — in that order. Remove the output directory before the next run:
rm -rf output/
Variables and matrix
The variables and matrix fields work identically in script-based Test Plans. See the
Setting variables and Using a matrix sections of the docker tutorial for full
walkthroughs.
Next steps
In this guide, we wrote a script-based Test Plan and ran it successfully. Next, we’ll look at the
command configuration in more detail.
Writing a command
In this guide, we’ll look at how the command configuration works. We’ll add environment variables
to the scenario command, then move the command script to its own file.
Prerequisites
- Completed the “Writing a test plan” tutorial
- An
rtf-hello-worlddirectory in the state it was at the end of that guide
Your directory should look like this:
rtf-hello-world/
├── configs/
│ ├── environment.yaml
│ └── scenario.yaml
└── test-plan.yaml
Environment variables
Any place a command can be specified, env_vars can also be specified. Environment variables
defined here are set before the command runs. Let’s add one to configs/scenario.yaml, updating the
script to reference it:
name: Script scenario
description: A script scenario
command:
name: scenario.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "$SCENARIO_ENV"
# --- Add an environment variable ---
env_vars:
SCENARIO_ENV: scenario executed
# ------------------------------------
Run the Test Plan to confirm this works:
rtf run test-plan.yaml
Output:
environment setup
scenario executed
environment teardown
Remove the output before continuing:
rm -rf output/
Inline command structure
Let’s look at the three fields in the command config:
nameis the name of the file RTF will write the script to. You can see it in the output directory after a run.kindspecifies how the file content is sourced. Usinginlinemeans the content is written directly in the config.contentis required whenkindisinlineand contains the script. Scripts must include a shebang line so the OS knows how to execute them.
Local command files
Defining command scripts inline works for short scripts but becomes hard to maintain as scripts
grow. You can save command scripts in separate files and refer to them using kind: relative_path.
Create a scripts directory and a scenario script:
mkdir scripts
touch scripts/scenario.sh
Add the following to scripts/scenario.sh:
#!/usr/bin/env sh
echo "Running scenario from an external file"
echo "$SCENARIO_ENV"
Now update configs/scenario.yaml to point to this file:
name: Script scenario
description: A script scenario
command:
name: scenario.sh
# --- Switch from inline to relative_path ---
kind: relative_path
path: ../scripts/scenario.sh
# -------------------------------------------
env_vars:
SCENARIO_ENV: scenario executed
Two things changed: kind is now relative_path, and content is replaced by path. The name
field still controls what the file is called in the output.
Note Paths are always relative to the file that defines them — here
configs/scenario.yaml— not relative to where the CLI is run from.
Verify with the --check flag:
rtf template test-plan.yaml --check
The output is similar to this:
name: Hello World
description: A test plan created as a guide for writing test plans
...
Now run the Test Plan:
rtf run test-plan.yaml
Output:
environment setup
Running scenario from an external file
scenario executed
environment teardown
The extra echo confirms we’re running the external script. RTF also copies the script to the
output — examine it to confirm:
cat output/providers/scenario_providers/scenario.sh
Output:
#!/usr/bin/env sh
echo "Running scenario from an external file"
echo "$SCENARIO_ENV"
RTF copies relative_path files to the providers output directory and executes from there. This
guarantees a stable path regardless of where the CLI is invoked from.
Note The
commandconfig inenvironment.setupandenvironment.teardownworks in exactly the same way as shown here.
Remove the output before continuing:
rm -rf output/
Next steps
In this guide, we covered how command config works — inline scripts, environment variables, and
pointing to external script files. Next, we’ll use variables and overrides to customize the Scenario
at runtime.
Using variables and overrides
In this guide, we’ll add a variable to the Scenario and use an override to swap the scenario command without modifying the base config. Both features work identically to the docker tutorial — this guide shows them applied to a script-based Test Plan.
Prerequisites
- Completed the “Writing a command” tutorial
- An
rtf-hello-worlddirectory in the state it was at the end of that guide
Your directory should look like this:
rtf-hello-world/
├── configs/
│ ├── environment.yaml
│ └── scenario.yaml
├── scripts/
│ └── scenario.sh
└── test-plan.yaml
Using variables
Variables allow a Test Plan to pass values into Scenario and Environment configs at runtime. See the Using variables section of the docker tutorial for a full explanation.
Let’s add a variable to configs/scenario.yaml:
name: Script scenario
description: A script scenario
# --- Add a variable definition ---
variable_definitions:
- name: scenario_variable
description: An example variable that the scenario expects to be defined
default: "scenario executed with default value"
# ---------------------------------
command:
name: scenario.sh
kind: relative_path
path: ../scripts/scenario.sh
# --- Use the variable ---
env_vars:
SCENARIO_ENV: "{{ scenario_variable }}"
# ------------------------
Set a value for it in test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
# --- Add a variable value ---
variables:
scenario_variable: "scenario executed with test plan variable"
# ----------------------------
scenario:
from:
kind: local
relative_path: configs/scenario.yaml
environment:
from:
kind: local
relative_path: configs/environment.yaml
Run the Test Plan:
rtf run test-plan.yaml
Output:
environment setup
Running scenario from an external file
scenario executed with test plan variable
environment teardown
The value from test-plan.yaml overrides the default in scenario.yaml. You can also pass it at
the CLI:
rtf run test-plan.yaml --var scenario_variable="scenario executed with CLI variable"
Output:
environment setup
Running scenario from an external file
scenario executed with CLI variable
environment teardown
Using overrides
Overrides let you replace specific parts of a Scenario or Environment config in the Test Plan without editing the base file. See the Using overrides section of the docker tutorial for a full explanation.
Let’s create a second scenario script to override with:
touch scripts/scenario-v2.sh
Add the following to scripts/scenario-v2.sh:
#!/usr/bin/env sh
echo "Running scenario from override script"
echo "$SCENARIO_ENV"
Add an override for the scenario command in test-plan.yaml:
name: Hello World
description: A test plan created as a guide for writing test plans
variables:
scenario_variable: "scenario executed with test plan variable"
scenario:
from:
kind: local
relative_path: configs/scenario.yaml
# --- Add an override for the command ---
overrides:
command:
name: scenario.sh
kind: relative_path
path: scripts/scenario-v2.sh
# ---------------------------------------
environment:
from:
kind: local
relative_path: configs/environment.yaml
Run the Test Plan:
rtf run test-plan.yaml
Output:
environment setup
Running scenario from override script
scenario executed with test plan variable
environment teardown
The override replaces only the command — the variable_definitions and env_vars from
scenario.yaml are unchanged, so SCENARIO_ENV still receives its value from scenario_variable.
Next steps
You’ve now completed the script-based test plan tutorials. To go further:
- Framework reference — full reference for all config fields
- Using file providers — manage files that your Environment and Scenario depend on
How-To Guides
This section contains how-to guides and runbooks for working with RTF.
Working with common RTF patterns
This page is a collection of short configuration snippets and strategies for working with RTF Test Plans to achieve specific goals. For an introductory overview of how to work with RTF please refer to the getting started guide. For a technical reference on RTF as a whole please refer to the framework section of the docs.
Only running a stage from a Test Plan
Problem: You only want to run a single stage of a Test Plan rather than the entire thing.
Solution: RTF’s run command supports limiting execution to one of the three stages:
- Environment setup
- Scenario
- Environment teardown
To do so, simply add the appropriate flag to your use of rtf run:
# Only the environment setup
rtf run --environment-up <test-plan>
# Only the scenario
rtf run --scenario <test-plan>
# Only the environment teardown
rtf run --environment-down <test-plan>
Discussion: All other rtf run flags behave normally in combination with these flags, but it is
an error to specify multiple at the same time. If your Test Plan includes a matrix this will still
result in an execution per matrix variant. To limit execution to a single variant you should make
use of the --var flag to pin variables to a single value (see below).
Limiting a matrix based Test Plan to run a single dimension
Problem: Your Test Plan contains a matrix but you would like to run it for a single dimension.
Solution: You can use the command line --var flag to replace individual matrix dimensions with
scalar variables:
rtf run --var 'router_version=v2.5.0' --var 'graph_ref=foo@prod' test-plan.yaml
Alternatively, if you are happy to edit the Test Plan itself, you can always comment out the matrix definition and replace it with variable definitions like so:
variables:
router_cpu_limit: "4"
# New variables to replace the matrix dimensions
router_version: "v2.5.0"
graph_ref: "foo@prod"
# Commented out matrix dimensions
#
# matrix:
# dimensions:
# router_version:
# - "v2.5.0"
# - "v2.6.1"
#
# graph_ref:
# - "foo@prod"
# - "bar@production"
Discussion: The use of command line arguments is recommended over editing the test plan file directly as it prevents the common issue of accidentally committing a modified test plan that now no longer runs the originally intended set of dimensions.
Running multiple iterations of a Test Plan
Problem: You have a test plan that you would like to run multiple times in order to collect results that can be analyzed statistically.
Solution: RTF’s matrix feature can be used with a placeholder dimension that will be
expanded over to produce n copies of a given test plan (or dimensions within an existing matrix)
by providing a series of unique values for the dimension:
matrix:
variant_names: "iteration_${n}"
dimensions:
n: [0, 1, 2, 3, 4]
This also works if you have an existing matrix:
matrix:
variant_names: "${my_dimension}_${n}"
dimensions:
n: [0, 1, 2, 3, 4]
my_dimension: ["foo", "bar"]
Discussion: You must include your iteration variable within your variant_names template in
order to produce unique variant names, as each original variant you are wanting to repeat will have
the same values other than this.
Using this approach is encouraged over simply running the Test Plan multiple times manually as it allows RTF to cache provider data internally as it runs in order to reduce network calls and work required to generate the output of each provider.
Determining how many variants of Test Plan will be run by a given matrix
Problem: You have written a Test Plan that defines a non-trivial matrix and you want to work out something like the expected running time or other properties related to the number of variants being run.
Solution: The rtf expand-matrix subcommand can be used to output the fully expanded variables
for each variant in the order they will be executed by rtf run:
# Example matrix setup
matrix:
variant_names: "${color}_${fruit}_${name}_${count}"
dimensions:
name: ["foo", "bar"]
count: [1, 2, 3]
compound:
fruits:
- fruit: apple
color: red
- fruit: pear
color: green
rtf expand-matrix test-plan.yaml
Output:
{
"variants": [
{
"name": "red_apple_foo_1",
"variables": {
"color": "red",
"count": 1,
"fruit": "apple",
"name": "foo"
}
},
...
]
}
jq can be used to count the variants like so:
rtf expand-matrix test-plan.yaml | jq '.variants | length'
Output:
12
Discussion: This will work even for Test Plans without a matrix by returning the single “variant” representing the top level variables for the Test Plan.
The variants returned by rtf expand-matrix are computed using the Test Plan as it is written.
Specifying additional matrix dimensions via the --vars flag on rtf run will alter the number of
variants.
Previewing matrix variant names
Problem: You are specifying a custom matrix variant name template using matrix.variant_names
and you want to check that it will produce the expected directory names.
Solution: The rtf expand-matrix command can be used alongside jq to output a list of
matrix variant names as a shell one-liner:
# Example matrix setup
matrix:
variant_names: "${color}_${fruit}_${name}_${count}"
dimensions:
name: ["foo", "bar"]
count: [1, 2, 3]
compound:
fruits:
- fruit: apple
color: red
- fruit: pear
color: green
rtf expand-matrix test-plan.yaml |
jq -r '.variants | map(.name) | join("\n")'
Output:
red_apple_foo_1
green_pear_foo_1
red_apple_bar_1
green_pear_bar_1
red_apple_foo_2
green_pear_foo_2
red_apple_bar_2
green_pear_bar_2
red_apple_foo_3
green_pear_foo_3
red_apple_bar_3
green_pear_bar_3
Discussion: The variants returned by rtf expand-matrix are computed using the Test Plan as it
is written. Specifying additional matrix dimensions via the --vars flag on rtf run will alter
the number of variants which in turn may result in a previously valid variant_names template
becoming invalid if it now produces non-unique names.
Conditional YAML merging
Problem: You want to use a variable to conditionally decide which YAML snippets get merged into your YAML files.
Solution: Use a conditional file provider to apply the correct YAML snippets based on the
variable’s value. Here, we use the example of optionally merging DataDog telemetry configuration
into router configuration if the telemetry_backend variable is set to "datadog":
file_providers:
- name: router-config.yaml
env_var: ROUTER_CONFIG
kind: conditional
cases:
# DataDog: include telemetry exporter config
- where: { var: telemetry_backend, eq: datadog }
kind: merge_yaml
base:
kind: relative_path
path: data/router-config.yaml
overrides:
- kind: relative_path
path: data/datadog-telemetry-overlay.yaml
- kind: graphos_subgraph_router_url_overrides
graph_ref: "{{ graph_ref }}"
url_format: "docker"
# Local: no exporter config needed
- where: { var: telemetry_backend, eq: local }
kind: merge_yaml
base:
kind: relative_path
path: data/router-config.yaml
overrides:
kind: graphos_subgraph_router_url_overrides
graph_ref: "{{ graph_ref }}"
url_format: "docker"
This allows you to run the same test plan with different telemetry backends:
# Run with local telemetry
rtf run test-plan.yaml -v "telemetry_backend=local"
# Run with DataDog telemetry
rtf run test-plan.yaml -v "telemetry_backend=datadog"
Discussion: This pattern combines kind: conditional with kind: merge_yaml to dynamically
compose configuration files based on variable values. Each case applies different overlays while
sharing the same base configuration.
Key considerations:
- The first matching
whereclause is used; order cases from most to least specific - All cases must produce valid output—RTF validates during static analysis
- Base configuration should omit sections that overlays will provide to avoid merge conflicts
- Multiple overlays can be chained as an array, merging in sequence
This pattern is useful when:
- Different deployment targets require different configuration snippets
- Feature flags should toggle configuration sections
- Environment-specific settings need conditional inclusion
The telemetry example above demonstrates this by conditionally including DataDog exporter
configuration only when telemetry_backend=datadog, while both cases share the same base router
config and subgraph URL overrides.
Troubleshooting
This page covers common issues encountered when using RTF and how to resolve them.
Output directory already exists
Symptom:
ERROR /path/to/output already exists and is non-empty
Cause: RTF refuses to overwrite existing output directories to prevent accidental data loss.
Solution: Either remove the existing directory or specify a different output directory:
rm -rf output
rtf run test-plan.yaml
Or:
rtf run test-plan.yaml --outdir=new_output
Missing template variable
Symptom:
ERROR Templating failed
(scenario.env_vars.MY_VAR) Missing template variables definition. Make sure the variable is defined in the scenario or environment config variable definitions
- my_variable: "Description of the variable"
Cause: A config file references a variable that isn’t defined in the Test Plan’s
variables section and has no default value.
Solution: Either define the variable in the Test Plan:
variables:
my_variable: "some value"
Or add a default value in the scenario/environment’s variable_definitions:
variable_definitions:
- name: my_variable
description: "Description of the variable"
default: "default value"
Static analysis check failed - required file
Symptom:
ERROR Static analysis checks failed
(MY_FILE) A required file has not been defined
You must provide a file for MY_FILE
Cause: A required File Provider exists that must be replaced with an actual provider in
the Test Plan’s overrides.
Solution: Add an override in the Test Plan that replaces the required provider:
environment:
from:
kind: local
relative_path: ./environment.yaml
overrides:
file_providers:
- name: my-file
env_var: MY_FILE
kind: inline
content: "actual content"
Static analysis check failed - file not found
Symptom:
ERROR Static analysis checks failed
(scenario.command.command_provider) The requested file did not exist
provided path was file:///path/to/missing-script.sh
Cause: A relative_path Command Provider or File Provider references a file that doesn’t
exist.
Solution:
- Verify the file exists at the specified path
- Remember that paths are relative to the config file containing them, not the Test Plan
- If using overrides, paths in the override are relative to the Test Plan file
Command execution failed
Symptom:
ERROR Unable to execute the my-script.sh command: "/path/to/my-script.sh" failed to terminate successfully
Cause: The command exited with a non-zero status code.
Solution: Check the command output above the error for clues. Common causes include:
- Script logic errors - The script encountered an error during execution
- Missing dependencies - A command used within the script isn’t available
Interpreter not found
Symptom:
ERROR Unable to execute the my-script.sh command: No such file or directory (os error 2)
Cause: The script’s shebang references an interpreter that doesn’t exist.
Solution: Use a portable shebang:
#!/usr/bin/env bash
Avoid hardcoded paths like #!/bin/bash which may not exist on all systems.
GitHub authentication failed
Symptom:
ERROR malformed scenario config section
ERROR Unable to load and resolve test plan: HTTP status client error (401 Unauthorized) for url (https://api.github.com/repos/org/repo/contents/path/to/file.yaml)
Cause: The GITHUB_TOKEN environment variable is not set or the token lacks required
permissions.
Solution: Export a valid GitHub personal access token:
export GITHUB_TOKEN="ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
The token needs read access to the repositories referenced in the Test Plan.
GraphOS authentication failed
Symptom:
ERROR errors returned when running graphql operation
ERROR Unable to resolve and write SUPERGRAPH file: unable to fetch details for graph@variant: graphql errors returned when running operation: ["HTTP fetch failed from 'kotlin': 406: Not Acceptable", "Invalid credentials provided"]
Cause: The APOLLO_KEY environment variable is not set or is invalid.
Solution: Export a valid Apollo API key:
export APOLLO_KEY="service:my-graph:xxxxxxxxxxxxxxxxxxxx"
Matrix variant names are not unique
Symptom:
ERROR The provided variant_names template produced duplicate names: ["duplicate_name"]
Cause: The matrix.variant_names template produces duplicate names for different variants.
Solution: Ensure the template includes enough dimensions to produce unique names:
matrix:
variant_names: "${dim1}_${dim2}" # Include all varying dimensions
dimensions:
dim1: ["a", "b"]
dim2: [1, 2]
Use rtf expand-matrix to check how the matrix dimensions will be named.
Debugging tips
Use verbose logging
Add -v flags to increase log verbosity:
rtf run test-plan.yaml -v # INFO level
rtf run test-plan.yaml -vv # DEBUG level
rtf run test-plan.yaml -vvv # TRACE level
Preview templated config
Use rtf template to see the fully resolved config before running:
rtf template test-plan.yaml
Add --check to also run static analysis:
rtf template test-plan.yaml --check
Inspect matrix expansion
Use rtf expand-matrix to see all variants that will be run:
rtf expand-matrix test-plan.yaml
Check File Provider resolution
The output directory contains a providers/ subdirectory with all resolved File Provider outputs.
Inspect these files to verify providers produced the expected content.
Generating CLI shell completions
The rtf completion -s <shell> command can be used to generate shell completion scripts for your
shell of choice.
To set up completions manually, follow the instructions below. The exact config file locations might vary based on your system. Make sure to restart your shell before testing whether completions are working.
bash
First, ensure that you install bash-completion using your package manager.
After, add this to your ~/.bash_profile:
eval "$(rtf completion -s bash)"
zsh
Generate an _rtf completion script and put it somewhere in your $fpath:
rtf completion -s zsh > /usr/local/share/zsh/site-functions/_rtf
Ensure that the following is present in your ~/.zshrc:
autoload -U compinit
compinit -i
fish
Generate an rtf.fish completion script:
rtf completion -s fish > ~/.config/fish/completions/rtf.fish
Generating JSON schemas for config files
The rtf json-schemas <config> command can be used to generate the JSON schema for the test plan,
environment and scenario config file formats.
To save the JSON schema files, navigate to the directory they should be stored in and run:
rtf json-schemas test-plan > test-plan-schema.json
rtf json-schemas environment > environment-schema.json
rtf json-schemas scenario > scenario-schema.json
To reference the JSON schemas in your config files, add the following annotation to the start of your file
# yaml-language-server: $schema=relative/path/to/test-plan-schema.json
name: Test Plan
description: A test plan config validated against its JSON schema
...
Reference
This section contains reference documentation for RTF.
- Framework - detailed reference for test plans, environments, scenarios, and providers
- CLI reference - command line help
- Glossary - definitions of RTF terms
The framework
The following pages provide reference information for various aspects of the runtime testing framework for end users who are interested in writing their own test plans.
New users should start with the tutorials section.
Test Plans
The Test Plan is the top-level entry point for RTF. It defines variables, matrix dimensions, and references to Scenario and Environment configurations.
An example of a valid
test-plan.yamlis provided in the Full example section below.
Top level keys
name: The name for this Test Plan.- Uniqueness is not enforced by the
rtfCLI but test plans should have unique names that can be used to distinguish them.
- Uniqueness is not enforced by the
description: A brief, human readable description of the behaviour of the Test Plan.- If there are any pre-requisites to running this Test Plan it is best to call them out here rather than in comments or other files (such as a README).
variables: Key value pairs for templating the Test Plan where the variables are all scalar.- Scalar here is defined to be a number, string or boolean.
matrix: Dimensions specified as key value pairs for templating the Test Plan where the variables are arrays of scalars.- Each matrix entry must have a consistent type for the variables array. Mixing different scalar variables will result in an error when the Test Plan is run.
- An optional
variant_nameskey can be provided to customise the names of the output directories used by each variant.
custom_providers: Declarations for loading Custom Provider Definitions.- For full details on the structure of Custom Provider Declarations and Definitions see the Custom Providers page of the Framework documentation.
scenario: A Config Spec for the scenario to be run.- For full details on the structure of a Scenario see the Scenario page of the Framework documentation.
environment: A Config Spec for the test environment to provision.- For full details on the structure of an Environment see the Environment page of the Framework documentation.
Config Specs
Both the scenario and environment keys map to a structure known as a Config Spec, which tells
RTF how to find the appropriate configuration for that aspect of the Test Plan. Config Specs support
three source types:
inline: Embed configuration directly in the Test Planlocal: Reference a local file by relative pathgithub: Fetch from a GitHub repository
The local and github types support optional overrides that merge on top of the base
configuration before the Test Plan is templated and checked.
Inline configuration
To provide configuration inline, add an inline key under the relevant top level scenario or
environment key and provide the config file contents underneath.
scenario:
inline:
# your scenario.yaml configuration goes here
environment:
inline:
# your environment.yaml configuration goes here
From a local file
To use a local file as a base, add a from key under the relevant top level scenario or
environment key, specifying the kind as local and the relative path from the test-plan.yaml
file under the relative_path key.
To define overrides, add the overrides key at the same indentation level as from and then add
your override configuration under that key. The structure here is not required to parse as a full
config file so you are free to only specify the keys that you need.
For details on how overrides are applied, see Applying Overrides below.
scenario:
from:
kind: local
relative_path: ../my-scenario.yaml
overrides:
# overrides go here
environment:
from:
kind: local
relative_path: ../my-environment.yaml
overrides:
# overrides go here
From a remote file in GitHub
To fetch from a GitHub repository, add a from key under the relevant top level scenario or
environment key, specifying the kind as github along with details for the org, repo and
path to the file relative to the root of the repository. It is also possible to optionally provide
a git_ref to pull the file from. If this is not specified then RTF will default to pulling from
the mainline branch.
You must have a valid GitHub access token with permissions to interact with your chosen repository exported as
GITHUB_TOKENin your shell environment for this method to work. See here for GitHub’s documentation on how to create and manage access tokens.
As with using a local file, overrides can be defined by adding the
overrides key at the same indentation level as from and then adding your override configuration
under that key. The structure here is not required to parse as a full config file so you are free
to only specify the keys that you need.
For details on how overrides are applied, see Applying Overrides below.
scenario:
from:
kind: github
org: my-org
repo: my-repo
# git_ref: testing-branch
path: test-plans/example/my-scenario.yaml
overrides:
# overrides go here
environment:
from:
kind: github
org: my-org
repo: my-repo
# git_ref: testing-branch
path: test-plans/example/my-environment.yaml
overrides:
# overrides go here
Applying overrides
The overrides section of a Config Spec is merged with the configuration file provided under the
from key as raw YAML before the configuration is parsed by RTF.
The merging strategy used is as follows:
- For maps, keys from overrides merge on top of matching keys in the base. If a key exists in both, RTF recursively merges the values; otherwise, RTF inserts the override key directly.
- For arrays, override values are appended to base values.
- For differing types or scalars, RTF replaces the base value with the override.
Once the overrides have been applied and the resulting config file is successfully parsed, all arrays are then sorted and deduplicated based on an appropriate key in order to support replacing array elements:
- For File providers the key used is
env_var. - For variable declarations the key used is
name.
A note on relative paths
There are several places within RTF config files that you be required to specify relative paths to other files on disk. In order to help with reasoning about how to provide these paths, RTF has strict semantics about how such relative paths are resolved.
Namely, they are always resolved relative to the file the path is written in. At first glance this
may sound obvious but the important aspect to remember is around writing overrides in your
test-plan.yaml.
When a base config file is loaded by RTF, all pre-existing relative paths will be resolved relative
to the path that the config file was loaded from (this is also true when pulling remote files from
GitHub). When overrides are then applied from the Test Plan, any new relative paths found there
are resolved relative to the location of the test-plan.yaml file, not the location of the base
config file.
Working with matrices
The matrix key expands to multiple test plan variants via the cartesian product of its
dimensions.
For example, the following matrix:
matrix:
dimensions:
a: ["foo", "bar"]
b: [1, 2, 3]
Expands to six test plans (known as “variants”) covering each of the possible combinations of values
for a and b.
Each variant is run individually and writes its output to its own subdirectory, named
matrix_variant_$n by default. The order in which variants are run is deterministic: dimensions are
ordered alphanumerically and the cartesian product is formed from the user provided ordering of
values for each dimension (as shown below).
- a=foo b=1 (matrix_variant_1)
- a=foo b=2 (matrix_variant_2)
- a=foo b=3 (matrix_variant_3)
- a=bar b=1 (matrix_variant_4)
- a=bar b=2 (matrix_variant_5)
- a=bar b=3 (matrix_variant_6)
Naming variants
To customize output subdirectory names, specify the matrix.variant_names key in your test plan
with a template string for generating the variant names:
matrix:
variant_names: "${a}_${b}"
dimensions:
a: ["foo", "bar"]
b: [1, 2, 3]
When doing so, the ordering for variants remains the same but the output directory names are generated using the template provided:
- a=foo b=1 (foo_1)
- a=foo b=2 (foo_2)
- a=foo b=3 (foo_3)
- a=bar b=1 (bar_1)
- a=bar b=2 (bar_2)
- a=bar b=3 (bar_3)
The syntax used for template strings involves placing matrix dimension names inside of ${} along
with static string content in order to generate a unique name for each variant. The resulting string
is then slugified to remove whitespace and slashes.
Combining related variables
The following initial matrix expands out to four variants covering different crate revisions for inclusion in a Rust build as shown below:
matrix:
dimensions:
federation_rev: [ "v2.6.2", "v2.7.0" ]
compiler_rev: [ "apollo-compiler@1.28.0", "apollo-compiler@1.30.0" ]
# Produces the following variants:
# - federation_rev: v2.6.2
# compiler_rev: apollo-compiler@1.28.0
#
# - federation_rev: v2.6.2
# compiler_rev: apollo-compiler@1.30.0
#
# - federation_rev: v2.7.0
# compiler_rev: apollo-compiler@1.28.0
#
# - federation_rev: v2.7.0
# compiler_rev: apollo-compiler@1.30.0
If each value of federation_rev only has a single matching compiler_rev, two of the resulting
four variants are invalid. Adding further matrix dimensions makes the problem worse, by adding more
undesirable variants.
To fix this, use matrix.compound to define a named compound dimension that groups several
variables together. Each entry in the compound dimension must contain the same set of variables, and
only those explicit groups of values will be used to construct the resulting matrix variants:
matrix:
# The dimensions key must always be present, even if it is an empty map
dimensions: {}
compound:
crate_revisions:
- federation_rev: v2.6.2
compiler_rev: apollo-compiler@1.28.0
- federation_rev: v2.7.0
compiler_rev: apollo-compiler@1.30.0
# Produces the following variants:
# - federation_rev: v2.6.2
# compiler_rev: apollo-compiler@1.28.0
#
# - federation_rev: v2.7.0
# compiler_rev: apollo-compiler@1.30.0
Now only the valid revision pairs are produced. The group name (crate_revisions here) exists only
to identify the group within the compound map and allow for runtime overriding of the dimension;
it is not templated into the test plan itself. You can add further dimensions to the matrix as
normal while preserving the correct compound combinations:
matrix:
dimensions:
graph_ref: [ "graph_1@prod", "graph_2@dev" ]
compound:
crate_revisions:
- federation_rev: v2.6.2
compiler_rev: apollo-compiler@1.28.0
- federation_rev: v2.7.0
compiler_rev: apollo-compiler@1.30.0
# Produces the following variants:
# - graph_ref: "graph_1@prod"
# federation_rev: v2.6.2
# compiler_rev: apollo-compiler@1.28.0
#
# - graph_ref: "graph_1@prod"
# federation_rev: v2.7.0
# compiler_rev: apollo-compiler@1.30.0
#
# - graph_ref: "graph_2@dev"
# federation_rev: v2.6.2
# compiler_rev: apollo-compiler@1.28.0
#
# - graph_ref: "graph_2@dev"
# federation_rev: v2.7.0
# compiler_rev: apollo-compiler@1.30.0
When working with older test plans you may encounter the
matrix.includekey, which is a deprecated alias for a single compound group namedinclude.matrix.include: [...]behaves exactly likematrix.compound: { include: [...] }and is still supported for backwards compatibility purposes, but if you see it in a test plan you’re working with, you should migrate it to usematrix.compoundinstead.
Combining multiple compound dimensions
Compound dimensions interact with one another in the way you would expect: with each compound dimension contributing blocks of values to the expanded set of variants rather than individual ones (as with a normal matrix dimension). If in the above example we found that we needed to work with the graph name and variant as individual variables, we could express that using a second compound dimension like so:
matrix:
dimensions: {}
compound:
graph_ref:
- graph_name: graph_1
variant: prod
- graph_name: graph_2
variant: dev
crate_revisions:
- federation_rev: v2.6.2
compiler_rev: apollo-compiler@1.28.0
- federation_rev: v2.7.0
compiler_rev: apollo-compiler@1.30.0
# Produces the following variants:
# - graph_name: "graph_1"
# variant: "prod"
# federation_rev: v2.6.2
# compiler_rev: apollo-compiler@1.28.0
#
# - graph_name: "graph_1"
# variant: "prod"
# federation_rev: v2.7.0
# compiler_rev: apollo-compiler@1.30.0
#
# - graph_name: "graph_2"
# variant: "dev"
# federation_rev: v2.6.2
# compiler_rev: apollo-compiler@1.28.0
#
# - graph_name: "graph_2"
# variant: "dev"
# federation_rev: v2.7.0
# compiler_rev: apollo-compiler@1.30.0
Producing the same number of variants as before, but now with the ability to reference graph name and variant directly.
Full example
The following is a minimal “kitchen sink” example of the structure of a valid test-plan.yaml.
name: example
description: An example description
variables:
foo: "A value for foo"
matrix:
variant_names: "${bar}_${baz}_${a}_${b}"
dimensions:
bar: [1, 2, 3]
baz: [true, false]
compound:
extra:
- a: 4
b: 5
- a: 6
b: 7
custom_providers:
- kind: local
relative_path: ./providers
using:
my_provider: my_provider.yaml
scenario:
inline:
name: An inline scenario
description: A description for the inline scenario
variable_definitions:
- name: foo
description: "A description for foo"
command:
name: my-test.sh
kind: relative_path
path: scripts/my-test.sh
env_vars:
FOO: "{{ foo }}"
environment:
from:
kind: local
relative_path: environment.yaml
overrides:
file_providers:
- name: my-additional-file.txt
env_var: ADDITIONAL_FILE
kind: inline
content: |
An additional file that wasn't present in the original environment.yaml
Environments
An Environment defines how RTF sets up and tears down the infrastructure required for a test. RTF currently supports three environment kinds: docker compose, kubernetes, and script. Depending on your use case you may find that you prefer (or need) to use one kind over another.
If you are unsure of where to start, we recommend the docker compose based environment as the
reasonable default choice that is suitable for most use cases; providing a flexible setup that can
be run both locally via the rtf CLI and remotely via the Orchestrator. The kubernetes environment
is intended for cases where you need to author native Kubernetes resources directly, while the
script-based environment is provided as a local-only fallback for situations where you need to
interact with the host machine executing the test plan.
For reference, the following table summarises the compatibility of each environment kind with different RTF operations:
| Environment | Template & check via CLI | Resolve via CLI | Run via CLI | Run via Orchestrator |
|---|---|---|---|---|
| Docker compose | ✅ | ✅ | ✅ | ✅ (via kompose) |
| Kubernetes | ✅ | ✅ | ❌ | ✅ |
| Script | ✅ | ✅ | ✅ | ❌ |
In terms of the trade offs being made between the different environment kinds: docker compose is the only kind that is fully supported everywhere. It allows for a faster local development loop and provides sensible defaults for deploying resources to a k8s cluster when needed, at the expense of offering more limited control over how those deployments look. A kubernetes Environment can only be executed under the RTF Orchestrator but allows for full control over what resources get applied to the cluster. The script Environment is a “last resort” escape hatch for running tests on a local machine when the environment itself can only be configured via locally executed commands outside of RTF’s control.
Regardless of which environment kind you use, the RTF configuration file you write will broadly have the same structure. Below we cover both the shared config details utilised by all environments as well as the configuration details unique to each kind.
Skipping the environment step entirely
If your Test Plan doesn’t require any environment to execute (e.g. you are using RTF’s matrix and file provider features to parameterise a stand alone test suite) then you can instruct RTF to skip the environment stages of its execution flow by adding
skip: trueto your environment config file. This will take priority over any other configuration within the file:name: skipped description: No environment needed skip: true
Shared keys
The following keys apply to all three execution models.
name: The name for this Environment configuration.- Uniqueness is not enforced by the
rtfCLI but environments should have unique names that can be used to distinguish them.
- Uniqueness is not enforced by the
description: A brief, human-readable description of the behavior of the Environment.- Describe any prerequisites required to run this Environment, rather than placing them in comments or other files (such as a README).
variable_definitions: Declarations of the templating variables supported by this Environment.- Variable declarations require specifying both the variable name and a short description of how the variable is used.
- Variable declarations also support an optional
defaultfield where you can specify a default scalar value to use if none is provided within the Test Plan. - If the same variable name is defined in both the Environment and Scenario used by a given Test Plan but with different defaults, each config file will fall back to its own default.
custom_providers: Declarations for loading Custom Provider Definitions.- For full details on the structure of Custom Provider Declarations and Definitions see the Custom Providers page of the Framework documentation.
Manifest file providers
The docker compose and kubernetes environments each accept a list of files that make up their
underlying infrastructure manifests (compose_files and resources respectively). These are
not arbitrary File Providers, rather, a restricted subset of file provider kinds meant for
describing manifest content:
- GitHub file (
github_file) - Inline file (
inline) - Inline directory (
inline_dir) - Relative dir (
relative_dir) - Relative path (
relative_path) - Required file (
required) - Templated file (
templated)
Note that there is no env_var field needed for manifest provider definitions: these resources are
used by RTF itself to bring up your test environment so it already has all of the information it
needs.
Declaring manifest providers is done at the top level of the config file like so:
# docker compose
compose_files:
- name: my-docker-compose.yaml
kind: relative_path
path: data/docker-compose.yaml
# kubernetes
resources:
- name: my-manifest.yaml
kind: relative_path
path: data/my-manifest.yaml
Referencing file providers within manifests
It is possible to reference the output of your file providers within your manifest files. The syntax
for this is the same as used by docker-compose for variable interpolation, namely
${MY_ENV_VAR} and $MY_ENV_VAR. You must use the same environment variable strings as in your
file provider and environment variable declarations elsewhere within your Test Plan. Provided the
environment variable you reference is known to RTF, it will substitute the appropriate value before
deploying your environment.
Example
# docker compose
services:
nginx:
image: nginx:alpine
labels:
rtf.io/file-providers: true
command: sh -c "nginx -c ${NGINX_CONF_FILE};'"
pull_policy: always
# kubernetes
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
annotations:
rtf.io/file-providers: "true"
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:alpine
args: ["sh", "-c", "nginx -c ${NGINX_CONF_FILE};'"]
imagePullPolicy: Always
Docker compose environment
A docker compose Environment brings the test infrastructure up and down using docker compose. RTF
writes the declared compose files and any additional file providers to a temporary directory before
invoking docker compose up, and runs docker compose down during teardown.
A docker compose environment is identified by the presence of the compose_files key.
Docker compose keys
compose_files: A list of named manifest file providers describing the compose files to start. This field is required.project_name: The docker compose project name. Optional; defaults to the environmentnameif not set.env_vars: Environment variables to pass todocker compose up.file_providers: Additional files the compose stack depends on, exposed to the stack as environment variables. Each entry is a File Provider with a requirednameandenv_varfield.
Service labels
Services in the compose files can carry RTF-specific labels that control behaviour when the test
runs under the RTF Orchestrator. These labels have no effect when running locally with the
rtf CLI.
| Label | Value | Effect |
|---|---|---|
rtf.io/file-providers | true | Mounts file provider output into the container. Required for services that read provider files at runtime. |
rtf.io/log-collection | true | Container logs are uploaded to GCS after the run. Absent by default — logs are not collected unless opted in. |
rtf.io/otel | true | Injects RTF collector endpoints into the container as environment variables. The following variables are set automatically: RTF_OTEL_COLLECTOR_GRPC (gRPC endpoint, port 4317) and RTF_OTEL_COLLECTOR_HTTP (HTTP/protobuf endpoint, port 4318). Use these in your service’s config instead of the standard OTEL_* variables. |
services:
router:
image: my-router:latest
labels:
rtf.io/file-providers: true
rtf.io/log-collection: true
rtf.io/otel: true
Full example
name: docker-compose-environment
description: An example docker compose environment
variable_definitions:
- name: message
description: "A message to echo out"
project_name: docker-compose-env
env_vars:
ECHO_MESSAGE: "{{ message }}"
compose_files:
- name: compose.yaml
kind: relative_path
path: providers/compose.yaml
file_providers:
- name: echo-server.py
env_var: ECHO_SERVER_SCRIPT
kind: relative_path
path: providers/echo-server.py
Kubernetes environment
A kubernetes Environment applies a set of native Kubernetes manifests directly, rather than relying on RTF’s docker compose to Kubernetes conversion (via kompose).
This is intended as a more advanced operating model for users who are already familiar with authoring and debugging Kubernetes manifests. If you do not have experience with working directly with Kubernetes manifests then we advise that you do not make use of this environment kind.
A kubernetes Environment can be templated, checked, and resolved locally like any other Environment,
but it can only be executed under the RTF Orchestrator as shows in the
capability table above. While it is certainly possible to run Kubernetes workloads
locally under a tool like kind – or apply the resulting manifests to a remote cluster that you
have access to – this is not an execution model supported by RTF directly.
A kubernetes environment is identified by the presence of the resources key.
Kubernetes keys
resources: A list of named manifest file providers describing the Kubernetes resource manifests to apply. This field is required.env_vars: Environment variables made available to the environment’s file providers.file_providers: Additional files the environment depends on, exposed to the environment’s file providers as environment variables. Each entry is a File Provider with a requirednameandenv_varfield.
Resource annotations
The same RTF-specific docker compose service labels can be applied to kubernetes manifests, but they are set as annotations rather than labels:
At present, only
Deploymentresources are supported for these annotation. If you think that you need this behaviour on other resource types, please reach out to the Runtime Readiness team in Slack to discuss your use case.
| Annotation | Value | Effect |
|---|---|---|
rtf.io/file-providers | true | Mounts file provider output into the container. Required for containers that read provider files at runtime. |
rtf.io/log-collection | true | Container logs are uploaded to GCS after the run. Absent by default — logs are not collected unless opted in. |
rtf.io/otel | true | Injects RTF collector endpoints into the container as environment variables. The following variables are set automatically: RTF_OTEL_COLLECTOR_GRPC (gRPC endpoint, port 4317) and RTF_OTEL_COLLECTOR_HTTP (HTTP/protobuf endpoint, port 4318). Use these in your service’s config instead of the standard OTEL_* variables. |
apiVersion: apps/v1
kind: Deployment
metadata:
name: router
annotations:
rtf.io/file-providers: "true"
rtf.io/log-collection: "true"
rtf.io/otel: "true"
spec:
replicas: 1
selector:
matchLabels:
app: router
template:
metadata:
labels:
app: router
spec:
containers:
- name: router
image: my-router:latest
Full example
name: kubernetes-environment
description: An example Kubernetes environment
variable_definitions:
- name: message
description: "A message to echo out"
env_vars:
ECHO_MESSAGE: "{{ message }}"
resources:
- name: manifest.yaml
kind: relative_path
path: providers/manifest.yaml
file_providers:
- name: echo-server.py
env_var: ECHO_SERVER_SCRIPT
kind: relative_path
path: providers/echo-server.py
Script environment
A script Environment defines setup and teardown commands that bracket the Scenario’s execution. Each
phase is a Command Provider. This model is intended as a fallback for cases that cannot be
achieved using a docker compose or kubernetes Environment, and only runs locally via the rtf CLI.
Script keys
setup: A Command Provider that defines how the environment should be set up before the scenario is run.teardown: A Command Provider that defines how the environment should be torn down after the scenario is run.
Full example
name: example
description: An example description
variable_definitions:
- name: my_variable
description: "A description for my variable"
default: "foo"
custom_providers:
- kind: local
relative_path: ./providers
using:
my_provider: my_provider.yaml
setup:
command:
name: my-setup-command.sh
kind: relative_path
path: scripts/setup.sh
env_vars:
MY_VARIABLE: "{{ my_variable }}"
teardown:
command:
name: my-teardown-command.sh
kind: relative_path
path: scripts/teardown.sh
file_providers:
- name: my-file.txt
env_var: MY_FILE
kind: inline
content: My inline content
Scenarios
A Scenario defines the test command to execute within a Test Plan. It is a Command Provider that runs between the Environment’s setup and teardown phases.
Top level keys
name: The name for this Scenario configuration.- Uniqueness is not enforced by the
rtfCLI but scenarios should have unique names that can be used to distinguish them.
- Uniqueness is not enforced by the
description: A brief, human readable description of the behaviour of the Scenario.- If there are any pre-requisites to running this Scenario it is best to call them out here rather than in comments or other files (such as a README).
variable_definitions: Declarations of the templating variables supported by this Scenario.- Variable declarations require specifying both the variable name and a short description of how the variable is used.
- Variable declarations also support an optional
defaultfield where you can specify a default scalar value to use if none is provided within the Test Plan. - If the same variable name is defined in both the Environment and Scenario used by a given Test Plan but with different defaults, each config file will fall back to its own default.
custom_providers: Declarations for loading Custom Provider Definitions.- For full details on the structure of Custom Provider Declarations and Definitions see the Custom Providers page of the Framework documentation.
The remaining keys depend on the command variant:
Docker command
Runs the scenario inside a Docker container. The container has access to the output of any
file_providers, with their paths remapped to /output/… inside the container.
docker: Configuration for the Docker image and command to run.image: The Docker image to use (e.g.alpine,ghcr.io/my-org/my-image). Supports templating.tag: The image tag to pull. Defaults tolatestif omitted. Supports templating.command: The command executed undersh -cinside the container. Supports templating.
env_vars: Environment variables passed into the container. Values support templating.file_providers: See File Providers. It is possible to run a script from a file provider as your command (see the full example below). File providers won’t be executable directly so you need to make sure to run usingsh $MY_SCRIPT, not./$MY_SCRIPT.
Full Example
name: example
description: An example description
variable_definitions:
- name: my_variable
description: "A description for my variable"
default: "foo"
custom_providers:
- kind: local
relative_path: ./providers
using:
my_provider: my_provider.yaml
docker:
image: alpine
tag: "3.18"
command: "sh $MY_SCRIPT"
env_vars:
MY_VARIABLE: "{{ my_variable }}"
file_providers:
- name: my-script.sh
env_var: MY_SCRIPT
kind: relative_path
path: scripts/scenario.sh
Script command
Runs the scenario as a script or binary executed directly on the host. See Command Provider for the full set of supported command kinds.
command: See Command Provider.env_vars: Environment variables passed to the command. Values support templating.file_providers: See Command Provider.
Full example
name: example
description: An example description
variable_definitions:
- name: my_variable
description: "A description for my variable"
default: "foo"
custom_providers:
- kind: local
relative_path: ./providers
using:
my_provider: my_provider.yaml
command:
name: my-scenario-command.sh
kind: relative_path
path: scripts/scenario.sh
args: ["a", "b"]
env_vars:
MY_VARIABLE: "{{ my_variable }}"
file_providers:
- name: my-file.txt
env_var: MY_FILE
kind: inline
content: My inline content
Command providers
Command Providers are the core executable element of RTF Test Plans. They specify how a command runs and what resources it needs. Environment and Scenario configurations are Command Providers with defined execution semantics.
The configuration for a Command Provider consists of three top level sections:
- The command itself.
- Environment variables that should be set before the command is run.
- A set of File Providers that should be run and made available before the command is run.
The details of each section are outlined below.
Full examples appear in the Environment and Scenario pages.
The command section
Commands can be defined in two ways:
- As an inline script that will be written to disk and made executable.
- As a relative path to an existing executable script.
Both strategies result in the appropriate UTF-8 encoded text file being written to disk and made executable before being executed as a subprocess by RTF. Scripts must include an appropriate shebang line at the top in order to run correctly.
⚠️ At this time, RTF does not support directly executing binaries via command providers
If the command you wish to execute is simply a pre-existing binary, you should provide an inline wrapper script that ensures that the binary in question is available on the PATH before calling the binary with the appropriate arguments:
#!/usr/bin/env sh if ! which "$YOUR_BINARY" > /dev/null 2>&1; then echo "ERROR: $YOUR_BINARY is not available on the path" exit 1 fi "$YOUR_BINARY" # arguments to the binary
It is also possible to mark that the command is required as an override specified in the Test Plan. This is primarily used as part of a Scenario or Environment configuration to set up supporting resources and data around an arbitrary user-specified command.
In each of the three options, the name of the command must be provided along with a kind that
specifies which strategy is used to define the command.
Inline scripts
To provide a command as an inline script, specify the kind as inline and provide the script
contents under the content key.
For multiline script content, see YAML multiline string syntax.
command:
name: my-shell-script.sh
kind: inline
content: |
#!/usr/bin/env sh
echo "Hello from RTF"
Relative paths
To use a pre-existing script, specify the kind as relative_path and provide the relative path to
the script under the path key. (See here for details on how relative paths are handled by
RTF).
command:
name: my-shell-script.sh
kind: relative_path
path: ../scripts/my-shell-script.sh
Required commands
To mark a command as required but not specified by default, use the required kind with an
accompanying message to inform users how to define their own command. If an override for the
command is not provided in the Test Plan, RTF will error at the templating stage of execution
and print the error message as the reason for the failure.
command:
name: my-command
kind: required
message: "This command must be provided in the test plan explicitly"
The env vars section
Environment variables are defined simply as key value pairs under the env_vars key. Variables may
be templated using the "{{ my_variable }}" syntax using any scalar value (not just strings). The
environment variables explicitly defined under this key will be merged with the environment
available to RTF itself before your command is executed.
# variables:
# my_string_env_var: "bar"
# my_integer_env_var: 42
env_vars:
FOO: "foo"
BAR: "{{ my_string_env_var }}"
BAZ: "{{ my_integer_env_var }}"
The file providers section
The file_providers key accepts any number of File Providers as resources made available before
command execution. See File Providers for provider-specific details. This section covers shared
structure and semantics.
When defined under a Command Provider the following shared keys are added to the variant specific keys defined by each file provider:
name: the name for this specific provider that will be used to report any errors encountered during execution.env_var: the environment variable to set containing the absolute path to the resources created by the provider.- Depending on the provider this may either be a single file or a directory of files.
- See the relevant documentation for each provider to learn more about the structure of their outputs.
kind: the variant “kind” which then sets the expected keys required for the rest of the provider block.
file_providers:
- name: my-inline-file.txt
env_var: MY_INLINE_FILE
content: "some file content"
- name: supergraph.graphql
env_var: SUPERGRAPH_SCHEMA
kind: graphos_supergraph
graph_ref: "foo@bar"
with_subgraph_overrides: docker
Note: Every resource needed by a Command Provider must be specified via a File Provider. RTF only guarantees paths set in provider environment variables. Do not construct relative paths between resources or from command scripts.
RTF internally caches and reuses file providers that share identical keys, so you are free to duplicate providers between different Command Providers. This allows sharing resources without providers running multiple times.
File Providers
Available file providers:
- Build Router from source
- Conditional
- Custom provider
- From command
- GitHub file
- GraphOS canned operations
- GraphOS canned operations by ID
- GraphOS supergraph Router URL overrides
- GraphOS subgraph SDL
- GraphOS subgraph names
- GraphOS supergraph SDL
- Inline file
- Inline directory
- Merge YAML
- GraphOS offline license
- Relative dir
- Relative path
- Required file
- Router download script
- Templated file
Build Router from source
A file provider used for building the Router from source at a specific git commit or reference. A profile and list of features can optionally be provided.
- name: "router-build.sh"
env_var: ROUTER_BUILD_SCRIPT
kind: build_router_from_source
git_ref: "some-ref"
rust_version: "1.89.0"
profile: "release"
features: "default"
Fields
git_ref
A git reference that can be passed to git checkout. This may be a full or partial commit hash,
branch name, or tag.
rust_version
A Rust version string that can be passed to rustup run {rust_version}, such as "1.78.0",
"beta", or "nightly".
Defaults to "stable" if unset.
profile
The profile to build the Router with.
Defaults to "release" if unset.
features
Comma separated list of features to build the Router with.
Defaults to "default" if unset.
Conditional
Conditionally run a file provider from an ordered list based on simple “where” clauses that make use of the provided templating variables. The first case with a “where” clause that holds will be run as the output of this provider.
Writing where clauses
The “where” clause on each case is a simple comparison against a single templating variable. You
must include the var key which accepts a string variable name that is required to be defined
within the test plan containing this provider. You may then assert that the variable is equal (eq)
or not equal (ne) to a given scalar value.
If none of the provider where clauses match, this provider will error during static analysis checks.
- name: conditional_config.json
env_var: CONDITIONAL_CONFIG
kind: conditional
cases:
- where: { var: test_type, eq: load }
kind: relative_path
path: data/config-load.json
- where: { var: test_type, eq: ramp }
kind: relative_path
path: data/config-ramp.json
Fields
cases
The ordered list of cases to be checked against the variables used for templating the test plan.
Custom provider
Use a custom provider to execute a command and produce a set of files.
- name: "router-docker-compose"
env_var: ROUTER_DOCKER_COMPOSE
kind: custom_provider
type: "router-docker-compose"
graph_ref: "graph@variant"
router_version: "v2.x.y"
build_router_from_source: "false"
Fields
ty
The type of custom provider to use. This is the name of the custom provider to use.
arguments
The arguments to pass to the custom provider.
From command
Run a command provider and use its output as a file provider resource.
As with all other command providers, you can provide both environment variables and other file
providers as inputs to the command being executed. RTF will use the contents of the $RTF_OUTPUT
path as the output of this provider, supporting both writing a single file to that path and creating
a directory at that path containing multiple files.
- name: vegeta-ops.json
env_var: VEGETA_OPS
kind: from_command
command:
name: format-for-vegeta.sh
kind: relative_path
path: scripts/format-for-vegeta.sh
env_vars:
ROUTER_URL: "http://127.0.0.1:4000/"
file_providers:
- name: canned_ops.json
env_var: CANNED_OPS_FILE
kind: graphos_canned_ops
graph_ref: "my@graph"
top_n: 20
skip_mutations: true
format-for-vegeta.sh
#!/usr/bin/env sh
while read -r req; do
if [[ "$OSTYPE" == "darwin"* ]]; then
encoded=$(echo "$req" | base64 -b 0)
else
encoded=$(echo "$req" | base64 -w 0)
fi
jq -nc \
--arg body "$encoded" \
--arg url "$ROUTER_URL" \
'{
"body": $body,
"header": { "Content-type": ["application/json"] },
"method": "POST",
"url": $url
}' >> "$RTF_OUTPUT"
done <"$CANNED_OPS_FILE"
Fields
command
The command to be run
env_vars
Environment variables to set
file_providers
File providers to run and make available prior to execution
GitHub file
The user specifies a path to a file within a GitHub repository, optionally providing a specific ref of the repository to pull the file from. If no ref is providing then the provider will pull the version of the file found on the default branch.
- name: "my-file.txt"
env_var: MY_FILE
kind: github_file
org: "my-org"
repo: "my-repo"
path: "resources/test-data/my-file.txt"
git_ref: "some-ref"
Fields
org
The GitHub org for the repository containing the target file
repo
The GitHub repository containing the target file
path
The absolute path from the root of the repository to the target file
git_ref
An optional git reference to pull the file from. This may be a full or partial commit hash, branch name, or tag.
Defaults to the mainline branch as specified in GitHub if unset.
GraphOS canned operations
The user specifies the graph ref and parameters that should be used to generate canned GraphQL requests based on operations data obtained from the GraphOS API.
- name: canned_ops.json
env_var: CANNED_OPS_FILE
kind: graphos_canned_ops
graph_ref: graph@variant
top_n: 10
skip_mutations: true
time_range: 7d
Fields
graph_ref
The Apollo graph ref to pull operations for.
top_n
The number of operations to attempt to fetch.
Defaults to 20 if unset.
skip_mutations
Whether or not to include mutations in the returned operations.
Defaults to false if unset.
time_range
How far back to query for operations.
Accepts duration strings like “30d”, “7d”, “12h”. Defaults to “30d” if unset.
GraphOS canned operations by ID
The user specifies the graph ref and a set of operation IDs to be fetched from the GraphOS API for generating canned GraphQL requests.
- name: canned_ops.json
env_var: CANNED_OPS_FILE
kind: graphos_canned_ops_by_id
graph_ref: graph@variant
operation:
kind: inline
content: |
5b1f8a2a1bd4be697559013a23fcbcb9186afe77
3f56aa92aad650bbfc7ba481cbe029aba2f6c5f4
50b77d7351052abd84dcd2c2ccb63eff2fa2f94c
Fields
graph_ref
The Apollo graph ref to pull operations for.
operations
A text file provider containing the operation IDs from the Apollo studio API for the operations you
want to work with as queried from an OperationInsightsListItem in the Studio graphQL API.
Operations must be specified one per-line
GraphOS supergraph Router URL overrides
The user specifies the graph ref that should be used to fetch subgraph SDL files from the GraphOS API and generates a the override_subgraph_urls YAML snippet that can be merged into a router config file.
This should be used whenever subgraph requests need to be mapped to a mock server instead of hitting the real subgraph as defined in the supergraph, which is typically desirable behavior when working with real graphs.
- name: subgraph-url-overrides.yaml
env_var: SUBGRAPH_URL_OVERRIDES
kind: graphos_subgraph_router_url_overrides
graph_ref: graph@variant
url_format: localhost
Fields
graph_ref
The Apollo graph ref to pull the subgraphs for.
url_format
The format of the overrides url.
Variants
localhost: Overrides tohttp://localhost:<port>for each subgraph. Port is defined as4001 + nwherenis the nth subgraph, starting at 0.docker: Overrides tohttp://loadbalancer:8080.
custom
Accepts a custom formatting configuration that will define the URLs.
Fields
base_url
The base URL to route subgraph requests to.
Defaults to the [UrlFormat::Docker] format if not set.
base_port
The base port that the subgraph requests should use
Defaults to the [UrlFormat::Docker] port if not set.
increment_port
Whether or not to increment the port number from the base for each subgraph.
Defaults to false if unset.
add_subgraph_route
Whether or not to include a /{subgraph_name} route for each subgraph.
Defaults to false if unset.
custom_subgraph_urls
Custom subgraph URL overrides for routes that do not fit the structure built by the above parameters.
GraphOS subgraph SDL
The user specifies the graph ref that should be used to fetch a subgraph SDL files from the GraphOS API.
Note that this file provider will output a directory of SDL schema files, one for each subgraph.
- name: "subgraphs"
env_var: SUBGRAPHS
kind: graphos_subgraphs
graph_ref: graph@variant
GraphOS subgraph names
The user specifies the graph ref that should be used to fetch the names of subgraphs in the supergraph from the GraphOS API.
This file provider will output a newline-delimited file of the subgraph names.
- name: "subgraph_names"
env_var: SUBGRAPH_NAMES
kind: graphos_subgraph_names
graph_ref: graph@variant
GraphOS supergraph SDL
The user specifies the ref that should be used to fetch a supergraph SDL file from the GraphOS API.
- name: "supergraph.graphql"
env_var: SUPERGRAPH
kind: graphos_supergraph
graph_ref: graph@variant
with_subgraph_overrides: docker
Fields
graph_ref
The Apollo graph ref to pull supergraph SDL for.
with_subgraph_overrides
Replace the supergraph’s subgraph urls with overridden values for testing.
Defaults to null if unset.
with_connector_overrides
Replace the supergraph’s connector urls with overridden values for testing.
Defaults to null if unset.
Inline file
The simplest form of file provider: the user specifies the contents of the file inline within their config file.
- name: "my-file.txt"
env_var: MY_FILE
kind: inline
content: |
my raw file content.
specified inline within an RTF config file.
Inline directory
An inline representation of a directory of files. The environment variable will be set to the path
of the directory itself. All files within that directory will need to be referenced using a
combination of this environment variable and its path.
This file provider primarily exists so that other file providers that produce a directory of files can be converted into their inline representations.
If, as a user of RTF, you need to specify multiple inline files, we strongly advise you use an
inline file provider for each file and that you DO NOT use this file provider.
- name: "my-directory"
env_var: MY_DIRECTORY
kind: inline_dir
files:
- path: file1.txt
content: |
content for file1
- path: nested/file2.txt
content: |
content for file2
Merge YAML
Merge the YAML output of text based file providers into a single YAML file.
Matching keys in the overrides file will replace scalar values, concatenate arrays and merge keys for maps.
When merging a single overrides file the overrides provider can be specified directly under the
overrides key:
- name: router-config.yaml
env_var: ROUTER_CONFIG
kind: merge_yaml
base:
kind: relative_path
path: "data/base-router-config.yaml"
overrides:
kind: relative_path
path: "../my-overrides.yaml"
When merging multiple overrides files, specify the providers in the order you want to merge them as an array:
- name: router-config.yaml
env_var: ROUTER_CONFIG
kind: merge_yaml
base:
kind: relative_path
path: "data/base-router-config.yaml"
overrides:
- kind: relative_path
path: "../my-overrides.yaml"
- kind: relative_path
path: "../my-other-overrides.yaml"
Fields
base
A base YAML file to start with.
Variants
overrides
One or more YAML files to merge on top of the base file in sequence.
GraphOS offline license
The user specifies the graph id that should be used to fetch an offline license from the GraphOS API.
- name: license.jwt
env_var: LICENSE
kind: offline_graphos_license
graph_id: graph
Relative dir
A relative path from the containing config file to a target directory and a list of files that should be made available as part of the test run. This provider works both with local directories and directories within GitHub if the containing config file was pulled from a repository.
The specified environment variable for this provider will point to the location of the directory itself. Relative paths under the specified directory will be maintained and in order to provided deterministic locations for each included file.
Only the specified files will be included and there is no way to wildcard multiple files.
- name: "my-data"
env_var: MY_DATA
kind: relative_dir
path: "../../resources/test-data"
files:
- "my-file.txt"
- "nested/my-nested-file.json"
Fields
path
The relative path from the containing config file to the target directory.
files
The file paths under this directory that should be included.
Relative path
A relative path from the containing config file to a target file that should be made available as part of the test run. This provider works both with local files and files within GitHub if the containing config file was pulled from a repository.
- name: "my-file.txt"
env_var: MY_FILE
kind: relative_path
path: "../../resources/test-data/my-file.txt"
Required file
The only purpose of this file provider is to throw an error if it still exists when the file providers are being checked. All definitions of a required file are expected to be replaced by user defined file providers.
- name: "router-config.yaml"
env_var: ROUTER_CONFIG
kind: required
message: "you must specify a router config file to use"
Router download script
Produces a POSIX shell script that can be run in order to download a target version of the Apollo Router.
- name: "router-download.sh"
env_var: ROUTER_DOWNLOAD
kind: router_download_script
version: "v2.6.0"
Templated file
Write a file whose content is an inline string with ${variable} patterns interpolated from the RTF
template variables defined for the current run.
- name: config.json
env_var: CONFIG_FILE
kind: templated
content: |
{ "endpoint": "${router_url}" }
Custom providers
Custom Providers are reusable file providers that execute commands to produce files. They encapsulate file generation logic for sharing across test plans, environments, and scenarios.
An example of a valid Custom Provider Definition is provided in the Full example section below.
Custom provider definitions
A Custom Provider Definition is a YAML file that defines a reusable custom provider. They execute commands to produce one or more files for other RTF config files to use, whereas Scenarios execute test actions. Custom provider definitions cannot reference other custom providers. They are templated using the arguments provided to the custom provider in a configuration file.
Top level keys
name: The name for this Custom Provider Definition.- Uniqueness is not enforced by the
rtfCLI but custom providers should have unique names that can be used to distinguish them.
- Uniqueness is not enforced by the
description: A brief, human readable description of the behaviour of the Custom Provider.- If there are any pre-requisites to using this Custom Provider it is best to call them out here rather than in comments or other files (such as a README).
variable_definitions: Declarations of the templating variables supported by this Custom Provider.- Variable declarations require specifying both the variable name and a short description of how the variable is used.
- Variable declarations also support an optional
defaultfield where you can specify a default scalar value to use if none is provided when the Custom Provider is invoked.
command: See Command Provider.env_vars: See Command Provider.file_providers: See Command Provider.
Full example
The following is a minimal “kitchen sink” example of the structure of a valid Custom Provider Definition file.
name: example
description: An example Custom Provider that generates a configuration file
variable_definitions:
- name: config_name
description: "The name to use in the generated configuration"
- name: config_value
description: "The value to include in the configuration"
default: "default_value"
command:
name: generate-config.sh
kind: relative_path
path: scripts/generate-config.sh
env_vars:
CONFIG_NAME: "{{ config_name }}"
CONFIG_VALUE: "{{ config_value }}"
file_providers:
- name: template.txt
env_var: TEMPLATE_FILE
kind: inline
content: |
some inline file content
Custom provider declarations
Custom provider definitions are loaded into config files through Custom Provider Declarations. These declarations specify a source directory (either a local relative path or a GitHub repository) and a mapping of provider names to definition files within that directory. The provider names are the names referenced when using the custom providers.
Custom provider declarations can be specified at three levels:
- Test Plan level: Custom providers declared in the test plan are only available to any overrides defined within the test plan itself. Unlike variables, custom providers DO NOT become available globally. If referencing a custom provider directly in an environment or scenario config, the declaration for that provider MUST be in that config file.
- Environment level: Custom providers declared in an environment config are only available within that environment’s setup and teardown command sections.
- Scenario level: Custom providers declared in a scenario config are only available within that scenario’s command section.
Structure
kind: The source directory containing the custom provider definition files. This can be specified as either:- A local relative path using
kind: localandrelative_path - A GitHub repository using
kind: githubalong withorg,repo,path, and optionallygit_ref
- A local relative path using
using: A map of provider names to definition file paths. The provider name is what you will use in thetypefield when referencing the custom provider in file provider sections.
Local path example
custom_providers:
- kind: local
relative_path: ./providers
using:
my_provider: my_provider.yaml
another_provider: another_provider.yaml
GitHub repository example
The git_ref field is optional and should be used if you want to target a branch that is not the
repository default.
You must have a valid GitHub access token with permissions to interact with your chosen repository exported as
GITHUB_TOKENin your shell environment for this method to work. See here for GitHub’s documentation on how to create and manage access tokens.
custom_providers:
- kind: github
org: my-org
repo: my-repo
path: path/to/providers
# git_ref: main
using:
my_provider: my_provider.yaml
another_provider: another_provider.yaml
Using custom providers
Custom providers appear in the file_providers list with kind: custom_provider.
Structure
name: The name for the output file produced by this custom provider. This is common to all file providers.env_var: The environment variable that will be set to the path of the output file. This is common to all file providers.kind: Must becustom_provider.type: The name of the custom provider to use. This must match a name from theusingmap in a Custom Provider Declaration.- Additional keys are passed as arguments to the custom provider and should match the
variable_definitionsin the Custom Provider Definition. Arguments can be literal values or template strings referencing variables from the config file.
Example
The following example uses a custom provider named my_provider (which must be declared in the
config file’s custom_providers section) to generate a configuration file:
file_providers:
- name: generated-config.yaml
env_var: GENERATED_CONFIG
kind: custom_provider
type: my_provider
config_name: "my-service"
config_value: "{{ service_value }}"
In this example, config_name and config_value are the arguments to my_provider.
Command-Line Help for rtf
This document contains the help content for the rtf command-line program.
Command Overview:
rtf↴rtf run↴rtf docs↴rtf expand-matrix↴rtf template↴rtf custom-provider↴rtf custom-provider template↴rtf custom-provider run↴rtf inline↴rtf inline all↴rtf inline relative-files↴rtf resolve↴rtf resolve scenario↴rtf resolve environment↴rtf completion↴rtf json-schemas↴rtf remote↴rtf remote prepare↴rtf remote request↴rtf remote run↴rtf remote run-known↴rtf remote ci-run↴rtf remote ci-run-known↴rtf remote execution-log↴rtf remote execution-output↴rtf remote execution-status↴rtf remote run-output↴rtf remote run-status↴rtf version↴
rtf
A swiss army knife for testing the Apollo Runtime
Usage: rtf [OPTIONS] <COMMAND>
Subcommands:
run— Check and run a test plandocs— Open the RTF documentation in your browserexpand-matrix— Expand a test plan matrix into JSONtemplate— Template a test plan using provided variables, outputting the resulting config to stdoutcustom-provider— Work directly with custom file provider definitionsinline— Inline file providers in a test plan. Outputs the resulting test plan to the given directoryresolve— Resolve file providers for a config file without executing itcompletion— Write a shell completion file to STDOUT for the given shelljson-schemas— Output json schemas for environment configurationremote— Interactions with the RTF Orchestrator Serviceversion— Display CLI version and exit
Options:
--var <VAR>— A single additional templating variable in the form “key=value”--vars <VARS>— Path to a JSON file containing additional template variables-v,--verbose— Flag to control logging verbosity. Default level iswarn.-vsets logging level toinfo,-vvtodebugand-vvvtotrace
rtf run
Check and run a test plan
Usage: rtf run [OPTIONS] <TEST_PLAN_PATH>
Arguments:
<TEST_PLAN_PATH>— Relative path to the test plan file that should be executed. When using –github this must be in the format ORG/REPO/PATH
Options:
-
--environment-up— Only run the environment setup -
--environment-down— Only run the environment teardown -
--scenario— Only run the environment scenario -
--github— Execute a test plan file in GitHub instead of from a local pathDefault value:
false -
--ref <GIT_REF>— Optional git ref to pull files from when using –github -
--outdir <OUTDIR>— Output directory for providers when they runDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false
rtf docs
Open the RTF documentation in your browser
Usage: rtf docs [SEARCH_TERM]...
Arguments:
<SEARCH_TERM>— An optional search term to search for within the docs
rtf expand-matrix
Expand a test plan matrix into JSON
Usage: rtf expand-matrix [OPTIONS] <TEST_PLAN_PATH>
Arguments:
<TEST_PLAN_PATH>— Relative path to the test-plan.yaml file that should have its matrix expanded
Options:
-c,--compact— Return the expanded matrix JSON in compact form
rtf template
Template a test plan using provided variables, outputting the resulting config to stdout
Usage: rtf template [OPTIONS] <TEST_PLAN_PATH>
Arguments:
<TEST_PLAN_PATH>— Relative path to the test plan file that should be templated. When using –github this must be in the format ORG/REPO/PATH
Options:
-
--check— Run a static check of the resulting test plan after templating -
--github— Template a test plan file from GitHub instead of from a local pathDefault value:
false -
--ref <GIT_REF>— Optional git ref to pull files from when using –github
rtf custom-provider
Work directly with custom file provider definitions
Usage: rtf custom-provider <COMMAND>
Subcommands:
template— Template a custom provider definition, outputting the resulting config to stdoutrun— Execute a custom provider definition
rtf custom-provider template
Template a custom provider definition, outputting the resulting config to stdout
Usage: rtf custom-provider template [OPTIONS] <DEFINITION_PATH>
Arguments:
<DEFINITION_PATH>— Relative path to the custom provider definition file
Options:
--check— Run a static check of the resulting test plan after templating
rtf custom-provider run
Execute a custom provider definition
Usage: rtf custom-provider run [OPTIONS] <DEFINITION_PATH>
Arguments:
<DEFINITION_PATH>— Relative path to the custom provider definition file
Options:
-
--outdir <OUTDIR>— Output directory for provider executionDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false
rtf inline
Inline file providers in a test plan. Outputs the resulting test plan to the given directory
Usage: rtf inline <COMMAND>
Subcommands:
all— Inline all file providersrelative-files— Inline only relative file providers
rtf inline all
Inline all file providers
Usage: rtf inline all [OPTIONS] <TEST_PLAN_PATH>
Arguments:
<TEST_PLAN_PATH>— Relative path to the test-plan.yaml file that should be inlined. When using –github this must be in the format ORG/REPO/PATH
Options:
-
--outdir <OUTDIR>— Output directory for inlined test planDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false -
--github— Inline a test plan file from GitHub instead of from a local pathDefault value:
false -
--ref <GIT_REF>— Optional git ref to pull files from when using –github
rtf inline relative-files
Inline only relative file providers
Usage: rtf inline relative-files [OPTIONS] <TEST_PLAN_PATH>
Arguments:
<TEST_PLAN_PATH>— Relative path to the test-plan.yaml file that should be inlined. When using –github this must be in the format ORG/REPO/PATH
Options:
-
--outdir <OUTDIR>— Output directory for inlined test planDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false -
--github— Inline a test plan file from GitHub instead of from a local pathDefault value:
false -
--ref <GIT_REF>— Optional git ref to pull files from when using –github
rtf resolve
Resolve file providers for a config file without executing it
Usage: rtf resolve <COMMAND>
Subcommands:
scenario— Resolve file providers for a standalone scenario configenvironment— Resolve file providers for a standalone environment config
rtf resolve scenario
Resolve file providers for a standalone scenario config
Usage: rtf resolve scenario [OPTIONS] <SCENARIO_PATH>
Arguments:
<SCENARIO_PATH>— Relative path to the scenario.yaml file
Options:
-
--outdir <OUTDIR>— Output directory for resolved providers and scenario.envDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false
rtf resolve environment
Resolve file providers for a standalone environment config
Usage: rtf resolve environment [OPTIONS] <ENVIRONMENT_PATH>
Arguments:
<ENVIRONMENT_PATH>— Relative path to the environment.yaml file
Options:
-
--outdir <OUTDIR>— Output directory for resolved providers and env filesDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false
rtf completion
Write a shell completion file to STDOUT for the given shell
Usage: rtf completion [OPTIONS]
Options:
-
-s,--shell <SHELL>— The shell to generate completions for (defaults to identifying from the environment)Possible values:
bash,elvish,fish,powershell,zsh
rtf json-schemas
Output json schemas for environment configuration
Usage: rtf json-schemas <CONFIG>
Arguments:
-
<CONFIG>Possible values:
test-plan,environment,scenario
rtf remote
Interactions with the RTF Orchestrator Service.
Set the RTF_ORCHESTRATOR_URL environment variable to override the Orchestrator base URL used by these subcommands; it defaults to the production Orchestrator when unset.
Usage: rtf remote <COMMAND>
Subcommands:
prepare— Prepare a test plan for remote execution by the Orchestrator. Outputs an Orchestrator-compatible JSON payload with inlined relative files and custom providersrequest— Send an IAP-authenticated HTTP request to the Orchestratorrun— Trigger a test run using the Orchestratorrun-known— Trigger a test run of a known test plan using the Orchestratorci-run— Trigger a test run using the Orchestrator and poll for the resultci-run-known— Trigger a test run of a known test plan using the Orchestrator and poll for the resultexecution-log— View the scenario log for a single test executionexecution-output— Pull all output for a single test execution (log, output.zip & status)execution-status— View the status summary for a single test executionrun-output— Pull output for all executions within a given test runrun-status— View the status summary for a test run
rtf remote prepare
Prepare a test plan for remote execution by the Orchestrator. Outputs an Orchestrator-compatible JSON payload with inlined relative files and custom providers
Usage: rtf remote prepare [OPTIONS] <TEST_PLAN_PATH>
Arguments:
<TEST_PLAN_PATH>— Relative path to the test plan file. When using –github this must be in the format ORG/REPO/PATH
Options:
-
--github— Prepare a test plan file from GitHub instead of from a local pathDefault value:
false -
--ref <GIT_REF>— Optional git ref to pull files from when using –github
rtf remote request
Send an IAP-authenticated HTTP request to the Orchestrator.
The response body is written to stdout on success.
Usage: rtf remote request [OPTIONS] <PATH>
Arguments:
<PATH>— Path on the Orchestrator to request (e.g./health)
Options:
-
-X,--method <METHOD>— HTTP methodDefault value:
GET -
-d,--body <BODY>— Request body as a literal string -
--plain-text— Skip setting a content type on POST/PUT requestsDefault value:
false
rtf remote run
Trigger a test run using the Orchestrator.
The output of this command will be the test run id and a link to the RTF UI to view the status
Usage: rtf remote run [OPTIONS] <TEST_PLAN_PATH>
Arguments:
<TEST_PLAN_PATH>— Relative path to the test plan file. When using –github this must be in the format ORG/REPO/PATH
Options:
-
--github— Prepare a test plan file from GitHub instead of from a local pathDefault value:
false -
--ref <GIT_REF>— Optional git ref to pull files from when using –github
rtf remote run-known
Trigger a test run of a known test plan using the Orchestrator.
The output of this command will be the test run id and a link to the RTF UI to view the status
Usage: rtf remote run-known [OPTIONS] <TEST_PLAN_ID>
Arguments:
<TEST_PLAN_ID>— Relative path to the test plan file. When using –github this must be in the format ORG/REPO/PATH
Options:
--ref <GIT_REF>— Optional git ref to pull files from when using –github
rtf remote ci-run
Trigger a test run using the Orchestrator and poll for the result.
The output of this command is aimed at being usable in CI runs and is non-interactive.
Usage: rtf remote ci-run [OPTIONS] <TEST_PLAN_PATH>
Arguments:
-
<TEST_PLAN_PATH>— Known Test Plan ID from the Orchestrator.You can find this on the UI page providing your known Test Plan’s details.
Options:
-
--github— Prepare a test plan file from GitHub instead of from a local pathDefault value:
false -
--ref <GIT_REF>— Optional git ref to pull files from when using –github -
--poll-interval-seconds <POLL_INTERVAL_SECONDS>Default value:
10
rtf remote ci-run-known
Trigger a test run of a known test plan using the Orchestrator and poll for the result.
The output of this command is aimed at being usable in CI runs and is non-interactive.
Usage: rtf remote ci-run-known [OPTIONS] <TEST_PLAN_ID>
Arguments:
-
<TEST_PLAN_ID>— Known Test Plan ID from the Orchestrator.You can find this on the UI page providing your known Test Plan’s details.
Options:
-
--ref <GIT_REF>— Optional git ref to pull the test plan from -
--poll-interval-seconds <POLL_INTERVAL_SECONDS>Default value:
10
rtf remote execution-log
View the scenario log for a single test execution
Usage: rtf remote execution-log <ID>
Arguments:
<ID>— ID of the Orchestrator test execution you wish to view the log of
rtf remote execution-output
Pull all output for a single test execution (log, output.zip & status)
Usage: rtf remote execution-output [OPTIONS] <ID>
Arguments:
<ID>— ID of the Orchestrator test execution you wish to pull output for
Options:
-
--outdir <OUTDIR>— Directory to place output inDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false
rtf remote execution-status
View the status summary for a single test execution
Usage: rtf remote execution-status <ID>
Arguments:
<ID>— ID of the Orchestrator test execution you wish to view the status of
rtf remote run-output
Pull output for all executions within a given test run
Usage: rtf remote run-output [OPTIONS] <ID>
Arguments:
<ID>— ID of the Orchestrator test run you wish to pull output for
Options:
-
--outdir <OUTDIR>— Directory to place output inDefault value:
output -
--force— Force removal of an existing output directory before runningDefault value:
false
rtf remote run-status
View the status summary for a test run
Usage: rtf remote run-status [OPTIONS] <ID>
Arguments:
<ID>— ID of the Orchestrator test run you wish to view the status of
Options:
-
--with-executions— Whether or not details of the underlying test executions should be includedDefault value:
false
rtf version
Display CLI version and exit
Usage: rtf version
Glossary
Custom Provider Declaration
A YAML snippet within an RTF configuration file (Test Plan, Environment or Scenario) that specifies where to load Custom Provider Definitions from. This maps Custom Provider Definitions to the name used to reference them in a Custom Provider. An RTF configuration file can only use Custom Providers defined in its Custom Provider Declaration.
Custom Provider Definition
An RTF YAML file for specifying a reusable Custom Provider. A Custom Provider Definition specifies the Variables, File Providers and Command required to produce one or more files from the arguments supplied to a Custom Provider. Custom Provider Definitions support Templating using arguments from the invoking Custom Provider.
Environment
An RTF configuration file for specifying how to set up and tear down the services and infrastructure under test using Command and File providers. Like all RTF configuration files, Environment configurations support Templating using Variables from the Test Plan.
Provider
A self contained piece of functionality within RTF that can provide file content for use in executing commands found in a Scenario or Environment configuration. Providers are declared as part of RTF configuration files by specifying their kind and the parameters needed to run them.
Command Provider
A Provider that defines an executable command along with environment variables and a set of attached File Providers whose content will be made available to the command when it is run. The purpose of each of the Scenario and Environment configuration files is to parameterise, resolve and run one or more Command Providers.
Custom Provider
A Provider that uses a Custom Provider Definition to execute a Command to produce one or more files. Custom Providers generate file(s) using the arguments provided.
File Provider
A Provider that will write out one or more files into a Test Plan’s output directory when resolved. File Providers range from being completely general purpose (e.g. pulling an arbitrary file from GitHub) to generating data specific to the Test Plan being resolved (e.g. generating valid GraphQL operations to run against the supergraph under test).
RTF
May refer to either the Runtime Testing Framework as a whole or the rtf CLI which is used to
resolve and run Test Plans.
RTF Orchestrator Service
A server-side system that receives Test Plans over HTTP, manages their execution in a provisioned Kubernetes cluster asynchronously, and reports results back to callers via status endpoints. The RTF Orchestrator Service is complementary to the RTF CLI rather than a replacement for it: the CLI runs Test Plans locally, while the Orchestrator runs them remotely and at scale. Often shortened to “the Orchestrator” after first use.
Scenario
An RTF configuration file for specifying how to run a test against services spun up by an Environment configuration using Command and File providers. Like all RTF configuration files, Scenario configurations support Templating using Variables from the Test Plan.
Templating
The use of template strings within RTF configuration files for declaring how users of that
configuration file may specify how to set Provider parameters and Command Provider environment
variables. Template strings are denoted with opening and closing double braces surrounding the name
of the Variable to inject with a single space at either side: "{{ my_variable }}"
Test Execution
A single Environment and Scenario pair executed by the RTF Orchestrator Service as part of a Test Run. A Test Run for a matrix Test Plan comprises one Test Execution per matrix dimension, each provisioned and run independently. A Test Execution progresses through the same lifecycle statuses as its parent Test Run.
Test Plan
The main RTF configuration file that defines an runnable test by combining a Scenario configuration with the Environment configuration it should be executed against. If either the Scenario or Environment supports Templating Variables then they can be specified statically as part of the Test Plan itself or dynamically through command line arguments to the RTF CLI.
Test Run
The RTF Orchestrator Service’s record of a single submission of a Test Plan for remote execution,
created from a Trigger Payload. A Test Run comprises one or more Test Executions — one per matrix
dimension for matrix Test Plans — and progresses through a lifecycle of statuses: INITIALISING,
RESOLVING, PROVISIONING, ENVIRONMENT_READY, RUNNING, and a terminal status of SUCCESSFUL,
FAILED, or UNRUNNABLE.
Test Run Summary
The JSON object returned by the RTF Orchestrator Service’s trigger and status endpoints, describing the current status of a Test Run and each of its Test Executions.
Trigger Payload
A JSON representation of a fully resolved Test Plan, produced by rtf remote prepare. It bundles
the inline Test Plan (as generated by rtf template) together with the contents of any local files
referenced by its File Providers, so the RTF Orchestrator Service — which has no filesystem access
to the machine the Test Plan was prepared on — has everything it needs to run it. Submitted to the
Orchestrator via POST /test-run/trigger.
Variables
Templating Variables are declared within the Scenario and Environment configuration files and have their values defined within the Test Plan referencing those configuration files.
Understanding RTF and the Orchestrator
This section explains the design decisions and architectural thinking behind RTF and the RTF Orchestrator Service.
- Concepts and Architecture - high-level architecture overview
- Logging Philosophy - why RTF logs the way it does
- Use of IO in Providers - the ResolutionContext pattern
- CLI Design - principles behind CLI subcommand design
Concepts and architecture
This page provides a high-level overview of RTF’s architecture and the key concepts that inform its design.
Crates
RTF is organized as a Cargo workspace with crates stored in the crates directory. Each crate
has its own README file explaining its purpose at the crate’s root.
The rtf-config crate is the heart of RTF. It handles:
- Parsing - YAML config files (Test Plans, Environments, Scenarios) are parsed into strongly typed Rust structs
- Templating - Variable substitution using the
{{ variable }}syntax - Validation - Static analysis checks before execution
- Providers - Both File Providers and Command Providers live here
The crate exposes a ResolutionContext trait that abstracts all IO operations, enabling testability and CLI control over execution.
The rtf-orchestrator crate is a server-side orchestration layer for RTF. It manages the lifecycle
of test runs and individual test executions across Kubernetes clusters: a management cluster (Argo
workflows for environment provisioning) and workload clusters (scenario jobs). It uses the
rtf-orchestrator-shared crate for types shared between it and the rtf-orchestrator-cli which
submits updates to the clusters.
The rtf-integrations crate provides the rtf-config crate with clients to make various HTTP
requests.
Data flow
When a user runs rtf run test-plan.yaml, the following flow occurs:
┌─────────────────────────────────────────────────────────────────────────┐
│ rtf run │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 1. Load Test Plan │
│ - Parse test-plan.yaml │
│ - Load custom provider definitions │
│ - Resolve scenario/environment references (local or GitHub) │
│ - Apply any overrides │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 2. Template Test Plan │
│ - Substitute variables into test plan. │
│ - Run static analysis checks │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 3. Execute Environment Setup │
│ - Resolve file providers │
│ - Run setup command │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 4. Execute Scenario │
│ - Resolve file providers │
│ - Run scenario command │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 5. Execute Environment Teardown │
│ - Resolve file providers │
│ - Run teardown command │
└─────────────────────────────────────────────────────────────────────────┘
RTF Orchestrator Service
RTF and the Orchestrator are related but distinct systems that serve different execution contexts:
- RTF CLI (
rtf) is a local command-line tool. A developer runs it directly to perform actions against test plans on the same system the CLI is hosted on. - RTF Orchestrator Service is a server-side system. It receives Test Plans over HTTP, manages their execution in a provisioned cluster asynchronously, and reports results back to callers via status endpoints.
The handoff point between the two systems is the Trigger Payload — a resolved Test Plan
produced by rtf remote prepare and submitted to the Orchestrator via POST /test-run/trigger. The
Orchestrator does not replace the RTF CLI; they are complementary tools for different execution
contexts.
Orchestrator data flow
When a caller triggers a test run via the Orchestrator:
┌─────────────────────────────────────────────────────────────────────────┐
│ POST /test-run/trigger (Trigger Payload) │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 1. Create Test Run │
│ - Test run record created in DB (status: Initialising) │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 2. Resolve │
│ - Resolver task picks up the run │
│ - Test plan resolved into individual executions (status: Resolving) │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 3. Provision │
│ - Event loop provisions the environment (status: Provisioning) │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 4. Run │
│ - Scenario job dispatched (status: Running) │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ 5. Complete │
│ - Event loop monitors for terminal status │
│ (Successful / Failed) │
│ - Environment teardown run │
│ - Test run marked complete │
└─────────────────────────────────────────────────────────────────────────┘
Key abstractions
Providers
Providers are the primary extension point in RTF. They come in two forms:
-
File Providers - Generate files that are made available to commands via environment variables. Examples include
relative_path(reading a file from a specific relative path),graphos_supergraph(fetch from GraphOS), andmerge_yaml(a utility to merge YAML from multiple sources). Custom Providers allow users to define their own file providers using YAML definitions that execute commands to produce files. -
Command Providers - Define executable commands with their environment variables and file provider dependencies. Environments and Scenarios are both command providers with defined execution semantics.
The Template trait
The Template trait enables recursive traversal of config structs to find and resolve templatable
fields. It is typically derived using #[derive(Template)] from rtf-derive.
The Field type
The Field<T> enum is the mechanism that enables templating within config structs. It wraps scalar
types (strings, numbers, booleans) and can exist in one of two states:
#![allow(unused)]
fn main() {
pub enum Field<T> {
Pending(String), // Contains a variable name to be resolved
Resolved(T), // Contains the final value
}
}When RTF parses a YAML config file, any value matching the "{{ variable_name }}" pattern is
deserialized as Field::Pending("variable_name"). Values without this pattern become
Field::Resolved(value) immediately.
For example, given this YAML:
graph_ref: "{{ graph }}"
top_n: 20
The graph_ref field parses as Field::Pending("graph") while top_n parses as
Field::Resolved(20).
During templating, the Template trait’s try_template method walks the config struct and resolves
each pending field by looking up its variable name in the TemplateContext. Once resolved, the
field transitions from Pending to Resolved and can be used during execution.
This design provides several benefits:
- Type safety - The generic parameter
Tensures variables resolve to the correct type - Validation - Pending fields are detected before execution, enabling early error reporting
- Traceability - The path to each field is tracked, producing clear error messages like
ERROR (environment.setup.file_providers[0].graph_ref) unknown templating variable: graph
ResolutionContext
All IO in providers must go through the ResolutionContext trait. This abstraction:
- Enables mocking in tests
- Gives the CLI control over execution
- Provides a consistent interface for file operations, HTTP requests, and command execution
See Use of IO in Providers for more details.
Design principles
RTF follows several key design principles:
-
Composition over embedding - RTF composes with external tools rather than embedding them. See the Overview for more on this philosophy.
-
Plumbing and porcelain - Commands are split into low-level “plumbing” (like
template) and high-level “porcelain” (likerun). See Plumbing vs Porcelain. -
No built-in magic - Commands don’t have special inline logic. See No Built-in Magic.
-
Fail fast with good errors - RTF validates early and reports all known errors in batch rather than failing on the first error. See Error Handling.
Logging Philosophy
RTF takes the “no news is good news” approach - quiet by default. Logging exists to help debug issues and provide feedback when things go wrong.
Design goals
When adding log statements to RTF, consider whether the information is helpful and avoid overwhelming users with unnecessary data.
Performance considerations
Logging performance is not a primary concern in RTF. The framework is expected to become I/O bound (waiting for network requests, file operations, etc.) before logging becomes a bottleneck.
However, keep these guidelines in mind:
- Avoid expensive computations solely for log messages
- Use structured fields instead of string formatting when possible
- Don’t worry about the overhead of log statements that won’t be displayed
Testing approach
Do not test logging at the low level using a crate like tracing_test. Instead, make sure to test
the output the user sees in the CLI tests. Tests should ensure the user sees the logging statement
in situations where it is expected and required to give helpful feedback. Tests should not cover
debug and trace level logs.
Use of IO in providers
All IO that is run as part of provider logic must be run using the context argument that is passed to methods. This allows the caller to control how providers are run as well as allowing us to swap out real IO for mock implementations within tests.
If you are writing a new provider and need to perform IO that is not currently possible via the existing ResolutionContext methods, you will need to first expose the functionality through that trait and provide a default “live” implementation for the concrete Context struct that is used by the CLI.
Concrete implementations
Context (RTF CLI)
Context is the concrete implementation used by the RTF CLI. It performs real IO: reading
files from the filesystem, executing commands, making HTTP requests to GraphOS and GitHub. When the
CLI runs rtf run, it constructs a Context and passes it through to all provider logic.
OrchestratorContext (RTF Orchestrator Service)
OrchestratorContext is the concrete implementation used by the RTF Orchestrator Service
(rtf-orchestrator). It wraps an inner Context but overrides the file IO behaviour: instead of
reading from the filesystem, it reads from an in-memory map of file contents that was pre-bundled by
rtf remote prepare into the Trigger Payload.
This is a deliberate design constraint. The Orchestrator is a server — it has no access to the
filesystem paths that existed on the developer’s machine when the test plan was prepared. The
Trigger Payload carries everything the server needs, and OrchestratorContext enforces that only
that content is accessible.
As a consequence, any ResolutionContext methods that would touch the filesystem directly are
implemented as panics:
#![allow(unused)]
fn main() {
fn read_path_to_string(&self, _path: impl AsRef<Path>) -> io::Result<String> {
panic!("attempt to read path to string")
}
}This is intentional. We should not be attempting to use the filesystem in the ways these methods expose during execution in the Orchestrator. Attempting to use the filesystem via those methods is a bug in the architecture, not something to handle gracefully.
HTTP and API client operations (platform_client, github_client, http_client) are delegated to
the inner Context.
CLI subcommand design
The following pages provide an overview of how the CLI subcommands and flags are designed.
Plumbing vs porcelain
The CLI subcommands are split into two different concepts; porcelain and plumbing.
The porcelain commands expose full end-to-end functionality to the users. They can be thought of as pipelines running the functionality in the plumbing commands sequentially to achieve the user’s desired outcome.
The plumbing commands expose ways for a user to run a subset of the porcelain commands. These commands are primarily used for debugging when the porcelain command has errored or so the user can test a portion of their configuration ahead of running a porcelain command.
No built-in magic
CLI subcommands should reference functions that handle the logic of the command. There should be no inline logic at the CLI struct level.
Orchestrator Helm chart versioning policy
This document records the decisions behind how we version and deploy the rtf-orchestrator Helm
chart for the RTF Orchestrator Service, the rationale for the phased approach, and the options
that were considered and rejected.
When a phase is completed, update the Current phase section to reflect actual state. The phase descriptions below are a permanent historical record of what was done and why — they should not be edited after the fact.
Current phase
Phase 1 + 2 — chart/image link fixed; production uses
edgefor fast iterationThe chart is published with version
0.0.0+<git-sha>andappVersionset to the git SHA. A pinned chart version guarantees a pinned image via theappVersionfallback in the Deployment template. Production overridesimage.tag: edgeandpullPolicy: Alwaysin the ArgoCDvaluesObject, so pods pick up the latest image on every rollout without a chart bump PR. Production is still updated by manually opening a PR onkanaveralto bumptargetRevision.
Design intent
The chart and image are published together on every code merge by design. The intent is that updating the Helm chart is the canonical way to update the Orchestrator — a chart version change implies a software change, and a software change produces a new chart version. This keeps deployment straightforward: there is one thing to bump (the chart), and it brings everything with it.
Options considered
Before settling on the phased approach, three alternatives were evaluated:
Option A — rolling tag (prod / latest)
Publish the chart under a mutable tag and point ArgoCD at it so it auto-tracks.
Rejected. ArgoCD’s OCI Helm source requires an exact targetRevision string; it cannot track a
mutable tag or a semver range. It would pull the tag once at install time and never update. Making
this work would require ArgoCD Image Updater, an additional operational dependency. More
fundamentally, with no staging environment between main and production, a mutable tag means any
bad merge silently redeploys with no human checkpoint — unacceptable as we approach live consumers.
Option B — immediate move to semver
Replace 0.0.0+sha with incrementing semver (0.1.0, 0.1.1, …) from the outset.
Rejected for now. ArgoCD OCI still requires an exact version; semver ranges do not work, so we gain no auto-tracking benefit. During early active development (several merges per day) we would accumulate hundreds of patch versions with no semantic distinction between them.
Option C — SHA pinning with phased improvement (chosen)
Keep SHA-based pinning but fix its problems in layers, making the process progressively more automated and semantically correct as the service matures.
Phase history
Phase 0 — Baseline
State:
- Chart published as
0.0.0+<git-sha>on every merge tomainthat touchescrates/**. - Image published with tags: full git SHA,
main-<short-sha>, and mutableedge. values.yamlhardcodesimage.tag: edge— every chart version deploys whateveredgepoints to at rollout time, regardless of the chart’s own SHA.- Production updated by manually opening a PR on
kanaveralto bumptargetRevision.
Problems that motivated moving to Phase 1:
- Chart and image are decoupled. Pinning a chart version does not guarantee a pinned image. The SHA in the chart version identifies the templates, not the running binary, contradicting the lockstep intent.
0.0.0+shais semantically unordered. Semver build metadata (the+segment) has no defined precedence —0.0.0+abcand0.0.0+defare equal. This rules out any tooling that relies on version ordering (including Renovate).- Manual PRs at high velocity. With several merges per day, repeatedly bumping
targetRevisionby hand is a significant source of toil.
Phase 1 — fix the chart/image version link
What changed:
- The
helm packagestep passes--app-version ${{ github.sha }}, embedding the git SHA intoChart.appVersion. - The Helm
Deploymenttemplate usesappVersionas the fallback image tag:image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" values.yamlclearsimage.tag(previously hardcoded toedge) and setspullPolicy: IfNotPresentas the chart default. SHA tags are immutable;Alwaysis wasteful when the image cannot have changed.
Result: A pinned chart version guarantees a pinned image. image.tag remains overridable via
values, which is used in Phase 2.
Phase 2 — production uses edge for fast iteration
What changed:
- The ArgoCD
valuesObjectinkanaveraladds:image: tag: edge pullPolicy: Always
Result: Production picks up the latest image on every pod restart or rollout without requiring a chart bump PR. With several merges per day and no live consumers at this stage, this removes the manual PR toil during the highest-velocity period of development.
Trade-off: The edge override is a known, intentional deviation from the lockstep intent. The
deployed image is not pinned. This is acceptable while there are no consumers; it must be removed
before the service takes on production traffic (Phase 3).
Note: Phases 1 and 2 are a single implementation effort, listed separately because Phase 1 is a change in this repo and Phase 2 is a values change in
kanaveral.
Phase 3 — remove edge override (enforce lockstep in production)
Trigger: The service has external consumers and a bad deploy has real consequences.
What changed:
- The
image.tag: edgeandpullPolicy: Alwaysoverrides are removed from the productionvaluesObjectinkanaveral. - The chart’s
appVersion(the git SHA baked in at publish time) becomes the live image tag. - Deploying a new image now requires a chart bump PR to
kanaveral.
Result: The lockstep intent is fully enforced in production. Every code change produces a new
chart version; bumping to that version in kanaveral is the single operation that updates the
running Orchestrator.
Summary
| Phase | Image tag in production | Lockstep enforced? | Automation |
|---|---|---|---|
| 0 | edge (hardcoded in values) | No | None — manual PRs to kanaveral |
| 1 + 2 | edge (values override) | No — intentional deviation | None — manual PRs to kanaveral |
| 3 | Git SHA (appVersion) | Yes | None — manual PRs to kanaveral |
Developer how-to guides
Task-oriented guides for common development workflows.
- Run PR checks - running CI checks locally
- Run rtf-orchestrator tests - running unit, DB, and integration tests for the Orchestrator
- Run a test coverage report - generating and inspecting a full workspace coverage report
- Update trybuild .stderr files - refreshing expected compiler output after a Rust version change
Running PR checks
This page documents all the checks that run when raising a PR, how to run the corresponding checks locally and how to resolve them.
All the tasks required to run these checks locally are defined using mise tasks. To run all the PR checks locally:
mise run pr-all
test
The GitHub action runs cargo test against the stable rust toolchain.
To run locally
mise run pr-test
This will show details for all failing test cases.
rust-fmt
The GitHub action runs the cargo pr-format alias defined in the .cargo/config.toml file. It
checks the code complies with rust fmt.
To run locally
mise run pr-format
This will show details for all formatting issues.
clippy
The GitHub action runs the cargo pr-clippy alias defined in the .cargo/config.toml file. It
checks the code complies with rust clippy.
To run locally
mise run pr-clippy
This will show details for all clippy issues.
rustdoc-links
The GitHub action runs cargo doc with additional flags to check that doc links work correctly.
To run locally
mise run pr-rustdoc
This will show details for all doclink issues.
Note: Most of the
cargo docsoutput will be warnings. This might be quite noisey but won’t necessarily fail your PR checks.
spell-check
This uses the typos-cli crate to check all the docstrings, YAML files and READMEs for typos.
To run locally
mise run pr-spell-check
This will print all the typos in the repo to screen.
To fix all these typos automatically
mise run fix-spelling
This will automatically fix all the detected spelling errors locally.
Before committing to main, manually review the diff to check all spelling errors are actually
errors and have been fixed correctly.
If there are any detected spelling errors that should be valid, or files that should be ignored,
then update the .typos.toml file at the root of the repo.
lint-markdown
This uses the dprint crate lint the markdown files and ensure consistent style and formatting.
To run locally
mise run pr-markdown
This will highlight any issues with the markdown files.
To fix all these issues automatically
mise run format-markdown
Before committing to main, manually review the diff to check all formatting fixes are desired.
Files can be excluded using the dprint.json file at the root of the repo.
doc-types
All documentation pages must have a valid Diataxis type annotation as the first line of the file:
<!-- diataxis-type: <type> -->
The valid types are explanation, howto, reference, and tutorial. This check ensures every
page is intentionally categorised.
To run locally
mise run pr-doc-types
This will print the path of any file that is missing or has an invalid annotation.
How to run rtf-orchestrator tests
The rtf-orchestrator crate partitions its tests into three categories, each requiring a different
level of infrastructure. This guide explains how to run each category.
Prerequisites
Unit tests
Unit tests require no external infrastructure and run via the standard workspace test command:
cargo test -p rtf-orchestrator --lib
Stack-dependent tests are skipped automatically and reported as ignored in the output.
DB tests
DB tests require a running PostgreSQL instance.
Start the lightweight database stack:
cd crates/rtf-orchestrator
make db-up
In a second terminal, run the DB tests:
make db-tests
Stop the stack when done:
make db-down
The DB tests can also be run against the tilt stack (documented in the section below).
Integration tests
Integration tests exercise the full HTTP API and require the complete tilt stack (Orchestrator server + database).
Set up the cluster:
cd crates/rtf-orchestrator
make cluster-setup
Then start the tilt stack (and view the status of the resources via the link provided in the output):
make cluster-up
In a second terminal, run the integration tests:
make integration-tests
Stop the stack when done:
make cluster-teardown
Running all tests as CI does
To replicate the full CI flow — start stack, wait for readiness, run all tests, tear down:
cd crates/rtf-orchestrator
make ci-tests
Feature flags reference
| Flag | What it gates |
|---|---|
db_tests | Unit tests requiring a live PostgreSQL database |
k8s_tests | Full integration tests in tests/suite.rs (implies db_tests) |
To test a feature flag directly, use the make targets as these also set required environment
variables that are necessary for the tests to work:
make db-tests
make integration-tests
See the HTTP API integration test reference for details on test organization and the
TestHelper infrastructure.
How to run a test coverage report
Prerequisites
cargo-llvm-covinstalled (included if you usemise)- Valid API credentials for any ignored tests you want to include
Note: There are no coverage targets for RTF. Reports are used to identify unexpected gaps, not to hit a percentage. High coverage does not mean the code is fully tested.
Clear previous coverage data:
cargo llvm-cov clean --workspace
Run all unignored tests:
cargo llvm-cov --no-report
Optionally, run the ignored tests to add their coverage. Supply only the credentials you have available:
GITHUB_TOKEN="$GITHUB_TOKEN" cargo llvm-cov -p rtf-cli --no-report -- --ignored github::
APOLLO_KEY="$APOLLO_KEY" APOLLO_SUDO="true" cargo llvm-cov -p rtf-cli --no-report -- --ignored graphos::
Generate and open the report:
cargo llvm-cov report --open
Inspect the report manually for unexpected gaps. Two things to keep in mind:
- 100% coverage for an area doesn’t mean it’s fully tested — it means the code ran at least once. A single execution rarely covers all scenarios.
- Low coverage in some areas may be justified. Where there’s a good reason, leaving lines uncovered is acceptable.
How to update trybuild .stderr files for rtf-derive
Rust may change compiler error message formatting between versions. When this causes trybuild
fail tests to break in the rtf-derive crate, update the .stderr files to match the new output.
Prerequisites
- The current Rust toolchain installed via
rustup
Delete the .stderr files for the failing tests:
rm crates/rtf-derive/tests/compile/fail/*.stderr
Run cargo test. trybuild writes new .stderr files to a wip/ directory at the repository
root:
cargo test -p rtf-derive
Copy the generated files into tests/compile/fail/:
cp wip/*.stderr crates/rtf-derive/tests/compile/fail/
Confirm the tests pass:
cargo test -p rtf-derive
Developer Reference
Technical reference documentation for RTF contributors.
- Style Guide - documentation and code style standards
- Error Handling - the ErrorBuilder pattern
- Logging Reference - log levels and structured logging
- Parsing Config Files - config resolution flow
- Config Traits - Template, Check, and provider traits
- Global Flags - CLI global flag reference
- Testing - test organization and conventions
- Data Structures - internal data structure documentation
Code style and idioms
This page contains a summary of the common code style elements and idioms we follow within this codebase and notes on why we feel they are important. While we aim to keep this list up to date, it should not be assumed to be exhaustive and should be treated as a living document to be updated as we identify missing items or establish new ones.
As a staring point, we follow the official Rust style guide but there are a few places where we deviate from the default advice for practical reasons.
As a general rule, we aim to optimise for being able to quickly and accurately read / navigate the code we write without relying on IDE support or external tooling. This is so we are able to work with the source code itself in contexts where the use of such tooling is not possible or would otherwise require additional setup (e.g. diffs, GitHub web UI, GitHub issue). This may sound pedantic, but it is based on years of experience of needing to be able to work with a codebase in this way in time sensitive situations such as production incidents and release windows. When additional tooling is available (which is most of the time), none of the style and idiom requirements here make their use more difficult. So we view it as being “worth it” to plan for the times when we need the code to be clear and easy to understand without them.
Imports
Rust allows organising use statements in a number of different ways, with rustfmt and IDE
tools such as rust-analyzer making “best effort” attempts to match the existing style within any
given file. Use of nightly rustfmt can provide more control over this but complicates the local
developer setup and is an easy thing to miss / forget, leading to failed CI checks and churn between
different setups.
As such, we use the following convention which all developer setups should work with automatically without explicit configuration:
- All use statements for a given file are placed at the top of the file in a single block
- We always use nested imports rather than “line per import / module”
use super::is only permitted within test modules to bring in the code under test
Exceptions
- Imports specific to a single function that would otherwise pollute the file-scope namespace may be
placed at the start of that function body
- e.g. use statements for enum variants, external API structs for building request payloads
Reasoning
- Placing use statements in a single block allows formatting tools to consistently organise imports. When using multiple blocks, the user defined blocks will typically be preserved even if they are not grouped according to the official style guide.
- Using nested imports reduces line noise at the top of the file and makes it easier to read the import list.
Examples
#![allow(unused)]
fn main() {
// Correct
use axum::{Router, routing::{get, post}};
use tokio::{net::TcpListener, sync::mpsc::unbounded_channel};
// Correct - bringing in enum variants within a function to reduce line noise
fn validate(payload: &SetStatusPayload) -> Result<()> {
use SharedStatus::*;
match (payload.status, payload.exit_code) {
(Failed, None) => return Err(Error::MissingExitCode),
(Failed, Some(0)) => return Err(Error::InvalidFailedExitCode),
(Failed, Some(_)) => (),
(Successful, Some(0)) => (),
(_, Some(code)) => {
return Err(Error::InvalidExitCode {
status: payload.status,
code,
});
}
_ => (),
};
Ok(())
}
// Incorrect - multiple blocks
use axum::{Router, routing::{get, post}};
use tokio::{net::TcpListener, sync::mpsc::unbounded_channel};
// Incorrect - multiple use statements for the same crate
use axum::Router;
use axum::routing::{get, post};
use tokio::net::TcpListener;
use tokio::sync::mpsc::unbounded_channel;
// Incorrect - top level use super;
use super::Result;
}Function returns
There must always be a blank line before implicit function returns.
Exceptions
- Single line functions do not require a blank line before the return as there is nothing to isolate the return value from.
Reasoning
- Having a blank line before function returns allows the reader’s eye to quickly jump to value being returned and isolates it from the rest of the function so it can be easily read.
Examples
#![allow(unused)]
fn main() {
// Correct
fn prepare_resolution(cfg: &Config, payload: TriggerPayload) -> Result<(OrchestratorContext, OrchestratorTestPlan)> {
let TriggerPayload {
mut test_plan,
relative_files,
custom_providers,
} = payload;
let ctx = OrchestratorContext::new(cfg, relative_files, custom_providers);
test_plan
.check_templating_will_work(&HashMap::new(), &ctx)
.map_err(ResolverError::TemplatingCheck)?;
Ok((ctx, test_plan))
}
// Incorrect
fn prepare_resolution(cfg: &Config, payload: TriggerPayload) -> Result<(OrchestratorContext, OrchestratorTestPlan)> {
let TriggerPayload {
mut test_plan,
relative_files,
custom_providers,
} = payload;
let ctx = OrchestratorContext::new(cfg, relative_files, custom_providers);
test_plan
.check_templating_will_work(&HashMap::new(), &ctx)
.map_err(ResolverError::TemplatingCheck)?;
Ok((ctx, test_plan))
}
}Fully qualified paths
use statements should bring in the leaf item they are targeting rather than modules. Inline fully
qualified paths for types (e.g. std::collections::HashMap) should be avoided wherever possible.
NOTE: Both Claude and Rust Analyzer have an annoying habit of using fully qualified paths. Claude does this somewhat randomly in the code it writes, while Rust Analyzer will tend to add imports when tab completing individual items, but not when inserting snippets (e.g. for trait methods).
When namespacing is required in order to resolving name collisions there are two accepted solutions:
- Alias the import:
- Import the module name for use as a single element path prefix:
Exceptions
- Free functions from the
io,fmt,fsStandard library modules should always be used via their module name. Types may be imported directly where naming is unambiguous. - When working with
ResultandErrortypes, the primary types for the file should be imported and used directly, with results and errors from other modules being qualified by their module name. thiserror::Errorandanyhow::Errormust always be fully qualified
Reasoning
- Fully qualified paths add significant line noise and result in larger diffs when code is refactored.
- Inline fully qualified paths also obscure the external dependencies of the file they are used in,
as they can often end up removing an entire module or crate from the list of
usestatements at the top of the file.
Examples
#![allow(unused)]
fn main() {
// Correct
use rtf_orchestrator_shared::status::Status as SharedStatus;
use std::io
fn example_1() -> SharedStatus { ... }
fn example_2() -> io::Result<()> { ... }
// Incorrect - ambiguous qualifier
use rtf_orchestrator_shared::status;
// Incorrect - inline fully qualified paths
fn example_2() -> std::io::Result<()> {
Err(std::io::Error::new(std::io::ErrorKind::NotFound, "foo"))
}
}IO handles in function arguments
A lot of the code we write in this repo makes use of API clients and / or context structs for controlling interaction with IO related logic. Outside of top level code such as request handlers and CLI commands, these structs must always be obtained through function parameters rather than being constructed within the body of the function where they are used.
To aid in reading function signatures we always place these “IO handles” at the end of the function arguments list.
Exceptions
- None
Reasoning
- Having a consistent position for these arguments at the end of the argument list allows the reader to quickly identify the non-IO handle related arguments that affect the logic of the function.
- Injecting IO logic via handles allows us to mock out IO behaviour in tests cleanly and without resorting to macros or external mocking crates.
Examples
#![allow(unused)]
fn main() {
// Correct
async fn init_test_run(name: &str, conn: &mut PgConnection) -> Result<TestRun> { ... }
impl Check for NamedFileProvider {
fn try_check(
&self,
path: &mut Vec<String>,
ctx: &impl ResolutionContext,
) -> checks::Result<()> { .. }
}
// Incorrect
async fn init_test_execution(
name: &str,
conn: &mut PgConnection,
test_run_id: i32
) -> Result<TestExecution> { ... }
}RTF Documentation Style Guide
This style guide defines writing standards for RTF documentation. It applies to all contributors
(internal and external) writing or editing docs in docs/src/.
Base style guide
RTF adopts the Microsoft Writing Style Guide as its foundation. This document covers RTF-specific decisions and deviations only. For topics not covered here, defer to Microsoft.
Key Microsoft principles we follow:
- Write like you speak
- Use second person (“you”)
- Use active voice and present tense
- Use contractions (it’s, you’ll, we’re)
- Get to the point fast
- Use sentence-case capitalization
Diataxis framework
RTF documentation follows the Diataxis framework, which defines four documentation types. Each page declares its type in an HTML comment on the first line.
Type declaration format
Every documentation page must include a type declaration comment:
<!-- diataxis-type: tutorial -->
# Page title
Content starts here...
Valid values: tutorial, howto, reference, explanation
The HTML comment format ensures the type declaration doesn’t render in the built documentation while remaining easy for contributors and tooling to identify.
The four types
| Type | Purpose | Reader State | Style |
|---|---|---|---|
| Tutorial | Teach through hands-on experience | Learning | Guiding, step-by-step |
| How-to | Solve a specific problem | Working | Direct, action-focused |
| Reference | Describe the machinery | Looking up | Neutral, comprehensive |
| Explanation | Provide context and background | Studying | Conversational, reflective |
Type-specific guidelines
Tutorials
- Guide the reader step-by-step to a working result
- Every step should produce visible output
- Minimize explanation (link to Explanation docs instead)
- Use “you” throughout
- Celebrate milestones (“You now have a working test plan!”)
How-to Guides
- Assume competence; don’t teach fundamentals
- Focus on the task, not the concepts
- Use imperative mood (“Add the provider”, “Run the command”)
- No digressions or background information
- Title format: “How to [verb] [noun]” or “[Verb]-ing [noun]”
Reference
- Be neutral and factual
- Use consistent structure across similar pages
- Include all parameters, options, and fields
- Provide examples without explanation
- Use third person or passive voice where appropriate
Explanation
- Provide context, rationale, and background
- “We” voice is permitted (“We designed RTF to…”)
- Connect concepts to each other
- Include trade-offs and design decisions
- May express opinions and preferences
Voice and tone
Person
| Doc Type | Person | Example |
|---|---|---|
| Tutorial | First plural (“we”) + Second (“you”) | “We’ll start by…”, “You run…” |
| How-to | Second (“you”) / Imperative | “Configure the environment…” |
| Reference | Third / Neutral | “The environment defines…” |
| Explanation | First plural (“we”) + Second (“you”) | “We designed this because…” |
“We” Voice
The “we” voice is permitted in Tutorials, Explanation docs, and Developer Reference docs:
- Tutorials: Use “we” to create a collaborative journey between writer and reader (“We’ll start by…”, “Now we can…”). This affirms the tutor-learner relationship recommended by Diataxis.
- Explanation: Use “we” for team perspective and design rationale (“We designed RTF to…”).
- Developer Reference: Use “we” for internal team perspective when documenting implementation details (“We use a set of four traits…”, “We check for any errors…”).
How-to guides and User Reference docs should use “you”, imperative, or neutral third-person voice.
Allowed (Tutorial):
We’ll start with templating and running the test plan. Then, we’ll make some changes.
Allowed (Explanation):
We here at Runtime Readiness are big fans of the Unix Philosophy.
Not allowed (How-to/Reference):
We recommend using the→ Use the--dry-runflag.--dry-runflag.
Formality
- Tutorials and How-to guides: Friendly, contractions encouraged
- Reference: More formal, fewer contractions
- Explanation: Conversational, personality allowed
Formatting
Code and Commands
Inline code - Use backticks for:
- Commands:
rtf run - Config keys:
environment.setup - File paths:
test-plans/example.yaml - Values:
true,false - Flags:
--dry-run
Code blocks - Use triple backticks with language identifier:
```yaml
environment:
name: production
```
Command examples - Do NOT include shell prompts:
<!-- Good -->
rtf run my-plan.yaml --environment staging.yaml
<!-- Bad -->
$ rtf run my-plan.yaml --environment staging.yaml
Commands with output - Use separate code blocks:
Only show output when it adds value (reader needs to copy or verify something). Use an introductory phrase to separate the command from its output:
Verify the test plan templates correctly:
rtf template test-plan.yaml --check
You should see output similar to this::
name: Hello World
description: A test plan created as a guide
...
Guidelines for output:
- Use “The output is similar to this:” or “Output:” as the intro phrase
- Use
...on its own line to indicate omitted output - Keep output concise; trim to the relevant lines
Placeholders
Use angle brackets for user-supplied values:
rtf run <test-plan> --environment <env-file>
Explain placeholders if not self-evident:
Where
<test-plan>is the path to your test plan YAML file.
Headings
- Use sentence case (capitalize first word only)
- No trailing punctuation (question marks are allowed for rhetorical headers)
- Use H2 (
##) for main sections, H3 (###) for subsections - Avoid H1 (
#) except for page title
| Good | Bad |
|---|---|
| Configure the environment | Configure The Environment |
| Running test plans | Running Test Plans. |
| When is it worthwhile using RTF | When Is It Worthwhile? |
Lists
- Use numbered lists for sequential steps
- Use bullet lists for non-sequential items
- Use the Oxford comma in inline lists (“setup, run, and teardown”)
Links
Use reference-style links with definitions at the bottom of the file:
See the [Test Plan][0] reference and [Command Provider][1] docs.
<!-- at bottom of file -->
[0]: ./test-plans.md
[1]: ./command-providers.md
Guidelines:
- Use numbered references (
[0],[1], etc.) for simplicity - Order references by first appearance in the document (
[0]appears before[1], etc.) - Place all link definitions at the bottom of the file
- Use relative paths for internal links
- Use descriptive link text, not “click here” or bare URLs
| Good | Bad |
|---|---|
| See the Test Plans reference. | See here. |
| Configure environments. | https://example.com/environments.md |
| The glossary defines this term. | Click this link. |
Terminology
Glossary usage
RTF maintains a central glossary. When using RTF-specific terms:
- Always ensure the term is defined in the glossary
- Link to the glossary on first use in a page
- Inline definitions are encouraged in Tutorials where clicking away disrupts flow
Example (Tutorial):
A Test Plan is the top-level entry point for RTF that defines variables and references to Scenarios and Environments. See the glossary for the full definition.
Example (How-to/Reference):
Configure the Test Plan with your variables.
RTF concepts
Use these exact capitalizations:
| Term | Usage |
|---|---|
| Test Plan | UpperCamelCase as concept; test-plan.yaml for files |
| Environment | UpperCamelCase as concept |
| Scenario | UpperCamelCase as concept |
| Provider | Generic term; specific types are File Provider, Command Provider |
| RTF | Always uppercase, no periods |
Config keys
Use backticks and exact casing from the YAML schema:
variables,matrix,environment.setup- NOT: “Variables”, “the matrix key”, “
VARIABLES”
Inclusive language
Follow Microsoft’s inclusive language guidelines. Key points:
Pronouns
- Use singular “they” for gender-neutral reference
- Prefer “you” to avoid pronouns entirely
- Never use “he” as generic
Terms to avoid
| Avoid | Use Instead |
|---|---|
| blacklist/whitelist | denylist/allowlist |
| master/slave | primary/replica, leader/follower |
| sanity check | confidence check, quick check |
| dummy | placeholder, sample |
| simple/easy | (use sparingly; subjective) |
Accessibility
- Don’t use color alone to convey meaning
- Use descriptive link text
Images
- Include images only when necessary (diagrams of complex flows, UI screenshots)
- Prefer text and code examples over images where possible
- Always provide meaningful alt text that conveys the image’s purpose
Document structure
Page length
No fixed limit. Pages should cover one focused topic. If a page requires more than 2 heading levels or you find yourself scrolling extensively, consider splitting into subpages.
Guidelines:
- Define scope clearly at the start (what the page covers and what it doesn’t)
- Front-load key information; readers scan rather than read linearly
- Cut everything unnecessary; prefer a short, accurate page over a comprehensive stale one
- Structure for skimmability: short paragraphs, bullet points, tables
Standard sections
Tutorials should include:
- Overview (what you’ll learn/build)
- Prerequisites
- Step-by-step instructions
- Next steps
How-to guides should include:
- Brief intro (1-2 sentences)
- Prerequisites (if any)
- Steps
- (Optional) Troubleshooting
Reference pages should include:
- Brief description
- All fields/options (consistent format)
- Examples
- Related pages
Explanation pages have flexible structure based on content.
Prerequisites
List prerequisites in a blockquote or admonition:
> **Prerequisites**
>
> - RTF installed (`cargo install rtf-cli`)
> - A GitHub access token exported as `GITHUB_TOKEN`
Checklist for contributors
Before submitting documentation:
- First line includes
<!-- diataxis-type: <type> -->comment - Voice matches doc type (no “we” in how-to/reference)
- No shell prompts in command examples
- Placeholders use angle brackets
- RTF terminology capitalized correctly
- New terms added to glossary and linked on first use
- Links use reference-style with definitions at bottom
- Inclusive language guidelines followed
Error handling
Wherever possible we aim to provide users with as much debugging information as possible when rtf
encounters an problem that prevents continuing execution (as opposed to early exiting with the first
error encountered). To support this, the error module in the rtf-config crate provides a
generic API for gathering and reporting multiple errors to the user.
The general idea is to create an ErrorBuilder whenever you are running batch logic such as templating, validation and static analysis checks. So long as there are no side effects to the logic being run, you should try to make use of an error builder to collect related errors where possible.
See the implementation of try_template for the CommandSection struct for an example of what
this looks like in practice.
Logging Reference
This page covers when and how to use different log levels in the Runtime Testing Framework. The
project uses the tracing crate for structured logging.
For the design rationale behind logging choices, see Logging Philosophy.
Log levels overview
The Runtime Testing Framework uses five log levels, from most to least verbose:
| Level | Verbosity Flag | Purpose | Audience |
|---|---|---|---|
TRACE | -vvv | Extremely detailed execution flow | Framework developers debugging |
DEBUG | -vv | Detailed diagnostic information | Developers and end users troubleshooting |
INFO | -v | High-level progress indicators | End users |
WARN | (default) | Potentially problematic situations | End users |
ERROR | (always shown) | Error conditions that prevent normal operation | End users |
When to use each level
ERROR level
Use error! for conditions that prevent the application from continuing normal operation or cause
significant functionality to fail.
Examples:
- Failed to initialize logging system
- Missing required configuration
- Network requests that fail completely
- File I/O errors that prevent core functionality
Avoid using ERROR for:
- Temporary failures that will be retried
- Optional operations that fail
- Expected validation failures
WARN level
Use warn! for situations that are unusual or potentially problematic but don’t prevent the
operation from continuing. WARN is the default log level shown to end users, so these messages
should provide useful context about what users should look for if errors occur later in the process.
Examples:
- Deprecated features being used
- Non-critical parsing failures
- Missing optional data
- Retrying failed operations
INFO level
Use info! for high-level progress indicators that help users understand what the application is
doing.
Examples:
- Major phase transitions (loading, executing, completing)
- Processing of user-provided inputs
- Successful completion of significant operations
- Progress indicators for long-running operations
DEBUG level
Use debug! for detailed diagnostic information that helps developers understand the internal
workings and troubleshoot issues.
Examples:
- File system operations (creating directories, writing files)
- Detailed processing steps
- Configuration values being used
- Internal state changes
TRACE level
Use trace! for extremely detailed execution flow information, typically for debugging complex
logic or data flow issues.
Warning: Do not log potentially sensitive data in the
tracelogs. Assume all data supplied by the user or from sources specified by the user could contain sensitive data. For this reason, we do not log raw API responses or the contents of a file.
Examples:
- The URL of an API being called. Do not log the response or parameters used to call the API since these could be sensitive.
- Fine-grained execution flow
- Performance-sensitive debugging information
Logging best practices
Use structured fields
Take advantage of tracing’s structured logging capabilities by including relevant context as fields:
#![allow(unused)]
fn main() {
// Good: structured fields for easy filtering and analysis
info!(%graph_id, %variant, "pulling supergraph details");
warn!(error = %e, "failed to parse configuration");
// Avoid: embedding everything in the message
info!("pulling supergraph details for graph_id={} variant={}", graph_id, variant);
}Common field naming conventions
Use these prefixes to control how values are formatted in log output:
- Use
%prefix for Display formatting (human-readable):%graph_id,%error - Use
?prefix for Debug formatting (developer-oriented):?config_object,?response_data - Use
error = %efor error context - Use descriptive field names:
operation_count,file_path,duration_ms
Examples:
#![allow(unused)]
fn main() {
// Good field names
debug!(file_path = %path, size_bytes = file_size, "reading configuration file");
warn!(retry_count = attempts, max_retries = MAX_ATTEMPTS, "operation failed, retrying");
// Avoid generic or unclear names
debug!(thing = %path, num = file_size, "reading file");
}Error Context
Always include relevant context when logging errors:
#![allow(unused)]
fn main() {
// Good: includes context about what failed
error!(path = %config_path, "failed to read configuration file: {e}");
// Avoid: generic error without context
error!("file operation failed: {e}");
}Configuration
The logging level can be controlled in several ways:
Command-line verbosity flags
Use these flags to control the overall log level:
- No flags: WARN level (default - only warnings and errors)
-v: INFO level (includes progress indicators)-vv: DEBUG level (includes detailed diagnostic information)-vvv: TRACE level (includes extremely detailed execution flow)
Environment Variable
Set APOLLO_RTF_LOG for fine-grained control over specific modules:
APOLLO_RTF_LOG=rtf_integrations=debug,rtf_cli=info cargo run
Per-module Filtering
You can set different log levels for different parts of the codebase:
APOLLO_RTF_LOG=warn,rtf_integrations::graphos=debug cargo run
Note: When both command-line flags and environment variables are used, the environment variable takes precedence for the modules it specifies, while the command-line flag sets the default level for other modules.
Quick reference
When to use each level
- ERROR: Operation cannot continue, user needs to take action
- WARN: Something unusual happened, but operation continues (default visibility)
- INFO: High-level progress updates, what RTF is currently doing
- DEBUG: Detailed diagnostic information for troubleshooting
- TRACE: Extremely detailed execution flow for debugging
Common patterns
#![allow(unused)]
fn main() {
// Error with context
error!(path = %config_path, "failed to read configuration file: {e}");
// Progress indication
info!(test_count = tests.len(), "executing test suite");
// Debug with structured data
debug!(endpoint = %url, method = "POST", "making API request");
// Warning about potential issues
warn!(feature = "deprecated_option", "using deprecated configuration option");
}Verbosity flags
- Default:
WARNandERRORonly -v: AddINFOmessages-vv: AddDEBUGmessages-vvv: AddTRACEmessages
Parsing config files
The rtf-config crate provides parsers for the three config files used by RTF, along with parsers for the various provider fragments that are used to expose the rest of the framework to users through those config files.
The parsers for TestPlanConfig, EnvironmentConfig and ScenarioConfig live in the formats
module. Each exposes a similar API for how they are parsed from YAML data files, templated,
validated and executed. The shared behaviour used to do this is defined in a set of traits that
provide tree-walk based methods for traversing the nested data structures obtained from parsing user
written config files.
Config resolution & execution
The full resolution and execution of a test plan has the following flow:
- Load and parse the user specified
TestPlanfile. - Locate and load any required
ScenarioandEnvironmentfiles defined infromdirectives as raw YAML. If the test plan defines overrides for either section then deep merge before parsing into concrete structs. - Check that the test plan contains all of the required variables for templating to be possible. If it doesn’t then early exit reporting the missing variables.
- Template the environment setup section before running static analysis checks.
- If all checks pass, run the setup command and use the provided output to finish templating the scenario and environment teardown sections.
- Run static analysis checks for both sections. If any checks fail for either section then early exit.
- Run the test scenario.
- Run the environment teardown.
For each of these stages we check for any errors or inconsistencies and report all known errors to the user as a batch operation. Each of the config file structs provides an API for running partial checks and templating so that end users are able to efficiently debug and iterate on their config files.
Traits for working with config structs
We have a set of five traits that are used to provide the shared behaviour needed to parse, validate and execute rtf config files:
- Template: used to locate and resolve templatable fields within larger config data structures.
- Check: used to run side-effect free static analysis checks on config files before they are executed.
- AsUtf8FileContent: used by providers that return a single string data file to define how their content is generated.
- ResolveFileContent: used by providers that return multiple files to define how their content is generated.
- ResolveAndWrite: used by providers to define how they generate their content and write it out to the user specified output directory.
All File Providers are required to implement the Template and Check traits and either the
AsUtf8FileContent trait, the ResolveFileContent trait, or the ResolveAndWrite trait. You
should prefer implementing AsUtf8FileContent where possible as it will handle some of the
boilerplate logic for you.
This is possible whenever your provider is writing out a single utf8 encoded file as its output.
Global flags
The CLI exposes the following global flags available to all subcommands:
--var
Provides a single additional templating variable.
Syntax:
--var <KEY>=<VALUE>
Example:
rtf run test-plan.yaml --var 'message="hello, world"'
Multiple --var flags can be specified to override multiple variables:
rtf run test-plan.yaml \
--var 'message="hello"' \
--var 'subject="world"'
--vars
Provides multiple templating variables from a JSON file.
Syntax:
--vars <PATH>
Where <PATH> is a path to a JSON file containing key-value pairs.
Example:
Given a file variables.json:
{
"message": "hello",
"subject": "world"
}
rtf run test-plan.yaml --vars variables.json
Precedence
Variables are resolved in the following order (later values override earlier):
- Variables defined in the test plan’s
variablessection - Variables provided via
--varsJSON file - Variables provided via
--varflags
Related
- Hello, world! tutorial - demonstrates variable overrides
- Test Plans reference - documents the
variablessection
Testing
RTF maintains comprehensive, well-structured, automated tests to enable safe, rapid iteration and continuous delivery. This document covers testing conventions across the codebase.
Testing guidance is split into two areas:
- Styles of test — patterns that apply across multiple crates, documenting the tools and conventions for each broad category of test
- Crate-specific pages — pages for each crate that describe any additional conventions on top of the relevant style
Styles of test
- Unit tests — tests inside
#[cfg(test)] mod testsblocks - CLI integration tests — testing compiled binaries via
assert_cmd - HTTP API integration tests — full-stack tests against a running server
- Proc macro tests — compile-time testing with
trybuild
Crate-specific pages
Organizing tests
Each style page and crate-specific page describes a naming hierarchy for its test category. The key aim is consistent naming at each level of the hierarchy, making it easy to find related tests. The hierarchies use a combination of module structure and test case naming. For module structure, the primary concern is ensuring code has the right scope and privacy level. Test structure is a secondary concern to module structure.
Test case naming
Test cases should be uniquely named within their hierarchy. The test cases should be given meaningful names that describe what is being tested.
Examples of good test case names:
all_fields_specified_and_validoptional_fields_not_definedrequired_field_missing
Examples of bad test case names:
works- does not specify what worksfails- does not specify what causes the failuretest_case- does not identify what is happening in this testoptional_field_specified1&optional_field_specified2- incrementing test cases by integer does not differentiate the test cases
Test coverage reports
RTF uses cargo-llvm-cov for coverage reporting. There are no coverage targets — reports are
used to identify unexpected gaps. See How to run a test coverage report for the full procedure.
Unit tests
Unit tests in RTF are defined inside #[cfg(test)] mod tests blocks within source files. This keeps
tests close to the code they test and ensures test helper code remains private to the module.
Crate-specific pages describe any additional conventions that apply within that crate.
Organization
Unit tests follow this naming hierarchy:
- Module — A logical grouping of functionality. Maps directly to the Rust module structure;
sub-modules may appear in the path (e.g.
providers::file). The primary concern is the Rust code’s scope and privacy; test structure is secondary. - Function — The function or logic under test. Maps to a function or trait method. If not fully specified in the module path, it becomes the first prefix in the test case name.
- Test class — A logical grouping of test cases. Defined when using
simple_test_caseto create multiple parameterized tests. Named after the test function when usingdir_cases. - Test case — The specific test case. Should be uniquely and meaningfully named. See test case naming.
Following the hierarchy above leads to the following generic test case paths:
#![allow(unused)]
fn main() {
// Without simple_test_case
module::path::tests::function_test_case
// With simple_test_case
module::path::tests::function_test_class::test_case
}Testing infrastructure
Unit tests commonly use the following tools:
simple_test_case— Parameterized testing with#[test_case]for multiple input variations sharing the same test logicindoc— Clean multi-line string literals for inline test data (YAML, JSON, etc.)anyhow— Ergonomic error handling via-> anyhow::Result<()>return type and.context()for test failure messages
Parameterized tests
Use #[test_case] when multiple inputs should exercise the same logic:
#![allow(unused)]
fn main() {
use simple_test_case::test_case;
#[test_case("{{ valid }}"; "standard")]
#[test_case("{{ with_underscore }}"; "with underscore")]
#[test]
fn field_parse_valid(raw: &str) {
// Same assertion logic for each case
}
}The test class is named after the function (field_parse_valid). Each test case gets a unique label
in the semicolon position ("standard", "with underscore").
Test data
Inline: Use indoc! for small, focused YAML or text:
#![allow(unused)]
fn main() {
let yaml = indoc! {"
command:
script: echo hello
"};
}Resource files: Place larger configs and any non-UTF-8 content in a resources/ directory at
the crate root. Reference them with include_str! or by path via assert_fs.
CLI integration tests
CLI tests verify the compiled binary’s behavior end-to-end: argument parsing, output formatting,
exit codes, and filesystem interactions. They live in a tests/ directory at the crate root and
compile the binary rather than calling functions directly.
Organization
CLI tests are not organized by source module. Modules within the tests/ directory create the
naming hierarchy:
- Environment dependency — Tests that require API tokens are grouped by their credential
requirement. Files named after the command they test have no external dependency. Examples:
github(requiresGITHUB_TOKEN),graphos(requiresAPOLLO_KEY). - Command — The CLI subcommand under test. Examples:
template,run,remote. Where no command is supplied, useno_command. - Flag(s) — Flags that cause significant branching in functionality, added as a prefix to the test case name. Omit if the flag is already implied by the environment dependency.
- Test class — A grouping of parameterized cases defined with
simple_test_case. - Test case — The specific scenario, uniquely and meaningfully named. See test case naming.
Following the hierarchy above leads to the following generic test case paths:
#![allow(unused)]
fn main() {
// Without simple_test_case
env_dependency::command::flags_test_case
// With simple_test_case
env_dependency::command::flags_test_class::test_case
}Testing infrastructure
CLI tests use the following tools:
assert_cmd— Spawns the compiled binary and asserts on stdout, stderr, and exit statusassert_fs— Temporary filesystem utilities for managing test directoriespredicates— Composable assertion predicates for output matchingindoc— Clean multi-line expected output stringssimple_test_case— Parameterized testing across multiple input variationscargo_bin_cmd!— Macro that builds a command for the target binary
Ignoring credential-dependent tests
Tests that require API tokens must be ignored by default. The #[ignore] attribute takes a reason
string explaining which credential is needed:
#![allow(unused)]
fn main() {
#[test]
#[ignore = "requires a valid GitHub API Token"]
fn github_flag_completes() {
// ...
}
}This keeps cargo test fast and dependency-free. To run these tests, supply the credential as an
environment variable and pass -- --ignored or -- --include-ignored to cargo.
Temporary directory management
CmdWithTmpDir combines a command with a temporary directory, ensuring filesystem state is cleaned
up after each test:
#![allow(unused)]
fn main() {
pub struct CmdWithTmpDir {
cmd: Command,
tmp: TempDir,
}
}Use this when a test must write output files and the test needs to inspect them.
Test resources
Test data lives under resources/ at the crate root, organized by resource type and validity:
resources/
└── <resource_type>/
├── valid/ — complete, valid test plans for success scenarios
└── invalid/ — organized by failure category (load-and-resolve, templating, checks, run)
Each test plan directory is self-contained with all necessary files.
HTTP API integration tests
HTTP API integration tests verify HTTP servers end-to-end. They require a running stack (on either Docker Compose or Kubernetes) and are distinct from unit tests that test business logic in isolation.
Test scope
Integration tests are expensive to run — starting a full stack adds significant overhead. Keep them focused: verify that the stack wires together correctly and that the HTTP API behaves as expected for the happy path and key error boundaries.
Don’t use integration tests to cover every error scenario exhaustively. Detailed error-case coverage belongs in unit tests, where it’s fast and deterministic. Integration tests should give just enough confidence that the end-to-end plumbing works.
Feature flags
Stack-dependent tests are gated behind feature flags so that cargo test passes without any
infrastructure:
| Flag | What it enables |
|---|---|
db_tests | Unit tests that require a live PostgreSQL database |
k8s_tests | Full integration tests requiring the Docker Compose stack (implies db_tests) |
Tests gated by these flags are annotated as follows:
#![allow(unused)]
fn main() {
#[cfg_attr(not(feature = "db_tests"), ignore)]
fn some_db_test() { ... }
#[cfg_attr(not(feature = "k8s_tests"), ignore)]
fn some_k8s_test() { ... }
}Do not remove these annotations — they prevent the standard workspace test run from failing when no stack is available.
Organization
Integration tests test the HTTP API as a whole and are not organized by source module. Instead, they are named by endpoint and scenario:
- Endpoint — The HTTP endpoint under test. Examples:
trigger(POST /test-run/trigger),run_status(GET /test-run/{id}/status). - Scenario — The specific behaviour being verified. Examples:
valid_rep_test_plan_returns_200,returns_404_for_unknown_run.
Following the hierarchy above leads to the following generic test case path:
#![allow(unused)]
fn main() {
endpoint_scenario
}For example: trigger_valid_rep_test_plan_returns_200.
Testing infrastructure
Integration tests use the following tools:
tokio—#[tokio::test]for async test functionsserial_test— Forces tests to run serially; required because tests share a single running stack and would interfere with each other if run in parallelreqwest— HTTP client for making requests to the server under testassert_fs— Temporary filesystem utilitiessimple_test_case— Parameterized testing with#[test_case]for multiple input variations
Define shared test setup and request helpers in a tests/common/ module. See the relevant
crate-specific page for implementation details.
Proc macro tests
Proc macro testing requires a different approach from standard unit testing because the code under test runs at compile time.
Test structure
Tests live in a tests/ directory at the crate root, split into two modules:
tests/
├── compile.rs # trybuild pass/fail compilation tests
├── compile/
│ ├── pass/ # .rs files that must compile successfully
│ └── fail/ # .rs files that must fail to compile, with .stderr files
└── template.rs # runtime tests for the generated trait implementations
Compile tests
There are two types of compile test:
Pass tests — .rs files imported into the compile module. They are not executed but will
cause a test failure if they do not produce valid Rust under cargo check. These verify that the
macro generates correct code for supported input types.
Fail tests — .rs files that should not compile. The trybuild crate captures the
compiler’s error output and compares it against a .stderr file:
#![allow(unused)]
fn main() {
#[test]
fn ui() {
let t = trybuild::TestCases::new();
t.compile_fail("tests/compile/fail/*.rs");
}
}This ensures users receive helpful, stable error messages when the macro is applied incorrectly.
Updating .stderr files after a Rust version change
Rust may change compiler error message formatting between versions, causing fail tests to break. See How to update trybuild .stderr files for the procedure.
Template tests
The template module tests that the generated Template trait implementations work correctly at
runtime. simple_test_case is used to run the same assertions against multiple Rust container
types (structs, tuple structs, enums, etc.):
#![allow(unused)]
fn main() {
#[test_case("Struct"; "basic_struct")]
#[test_case("TupleStruct"; "tuple_struct")]
#[test]
fn try_template_method_works(container_type: &str) {
// ...
}
}Tests are organized using the following hierarchy:
- Trait method — The
Templatemethod under test, including whether the error path is being tested. - Data structure — The Rust container type, defined as the test case name.
- Test case — Additional context to uniquely identify the test.
Following the hierarchy above leads to the following generic test case path:
#![allow(unused)]
fn main() {
template::tests::trait_method::data_structure_test_case
}rtf-config
This page documents how tests in the rtf-config crate are organized and implemented. The
crate follows the unit test style. This page describes the additional conventions that apply
within rtf-config.
Organization
Tests in rtf-config extend the standard unit test hierarchy with an extra level between Module and
Function to account for the data-centric structure of the crate:
- Module — A logical grouping of config data structures (e.g.
providers::file). - Data structure — The specific type under test (e.g.
RelativePath,GithubFile). If not fully specified in the module path, this becomes the first prefix in the test case name. - Functionality — The trait or function being tested (e.g.
parse,template,resolve). This becomes the second prefix in the test case name. - Test class — A logical grouping of parameterized cases created with
simple_test_case. - Test case — The specific scenario, uniquely and meaningfully named. See test case naming.
Following the hierarchy above leads to the following generic test case paths:
#![allow(unused)]
fn main() {
// Without simple_test_case
module::path::tests::data_structure_functionality_test_case
// With simple_test_case
module::path::tests::data_structure_functionality_test_class::test_case
}Mock system
The crate implements a MockContext<T> that allows dependency injection for HTTP and GitHub clients
during testing:
#![allow(unused)]
fn main() {
// For HTTP client testing
let mock_ctx = MockContext::with_http_client(&[
("https://example.com/api", "200", "response body")
]);
// For GitHub client testing
let mock_ctx = MockContext::with_github_client("file content");
}This enables testing without external dependencies while using the same interfaces as production code.
For GraphOS interactions, the crate tests against mock SupergraphDetails rather than mocking the
GraphOS API directly:
#![allow(unused)]
fn main() {
#[test]
fn supergraph_resolve_success() {
let details = Arc::new(SupergraphDetails {
graph_id: "test-graph".to_string(),
variant: "test-variant".to_string(),
supergraph_sdl: "schema { query: Query }".to_string(),
subgraphs: vec![/* test subgraphs */],
});
let supergraph = GraphosSupergraph {
graph_ref: Field::Resolved("graph@variant".to_string()),
with_subgraph_overrides: None,
};
let result = supergraph.content_from_details(details);
assert_eq!(result, "schema { query: Query }");
}
}This tests the core content transformation logic while keeping API interaction complexity in the
production context method (with_supergraph_details).
Configuration testing strategy
Config handling is tested in four phases:
- Parsing — YAML deserialization works correctly for valid inputs and fails for invalid ones
- Template resolution — The
{{ variable }}templating system works across all supported types. Derived implementations ofTemplategenerally don’t need separate tests; only custom implementations require them. - Validation — Config validation catches common errors and edge cases
- Resolve — Providers and configuration resolve and execute correctly
rtf-core
The rtf-core crate follows the standard unit test style. Tests live alongside source
code in #[cfg(test)] mod tests blocks.
The crate’s tests focus on execution logic and diff/comparison output. Test data is typically
generated inline or loaded via include_str! from resource files.
rtf-integrations
This page documents how tests in the rtf-integrations crate are organized and implemented.
The crate follows the unit test style. This page describes the additional conventions that
apply within rtf-integrations.
Organization
Tests follow the standard unit test hierarchy. The test class level uses the dir_cases macro
naming convention: when #[dir_cases] is used, the test function name becomes the test class (e.g.
try_parse_ok, try_parse_err), and test case names are derived from the test data file names.
Following the hierarchy leads to the following generic test case paths:
#![allow(unused)]
fn main() {
// With dir_cases
module::path::tests::function_test_class::file_name
// Without simple_test_case
module::path::tests::function_test_case
}Trait-based testing with PlatformQuery
The crate’s primary testing pattern leverages the PlatformQuery trait to test GraphQL response
parsing without requiring HTTP mocking:
#![allow(unused)]
fn main() {
pub trait PlatformQuery: GraphQLQuery {
type Output;
type Error;
fn try_parse(
data: Self::ResponseData,
variables: Self::Variables,
) -> Result<Self::Output, Self::Error>;
}
}Tests deserialize JSON directly into the GraphQL response type and call try_parse in isolation:
#![allow(unused)]
fn main() {
fn try_parse_ok(path: &str, contents: &str) -> anyhow::Result<()> {
let raw: <OfflineLicense as GraphQLQuery>::ResponseData = serde_json::from_str(contents)?;
let result = OfflineLicense::try_parse(raw, variables)?;
assert_eq!(result, "expected-value");
Ok(())
}
}Directory-driven parameterized tests
The dir_cases macro generates test cases from files in a directory:
#![allow(unused)]
fn main() {
#[dir_cases("crates/rtf-integrations/resources/test_data/offline_license/valid")]
#[test]
fn try_parse_ok(path: &str, contents: &str) -> anyhow::Result<()> {
// Test logic runs once per file in the directory
}
}Each file becomes a separate test case. path provides context for error messages; contents is
the file text. Adding new test cases is as simple as adding a file to the directory.
Test data organization
Test data lives in resources/test_data/, organized by operation and validity:
resources/test_data/
└── <operation>/
├── valid/ — complete GraphQL response JSON
└── invalid/ — response JSON wrapped with an expected_error field
Valid test case — the raw GraphQL response:
{
"graph": {
"account": {
"offlineLicense": { "jwt": "test-jwt-token" }
}
}
}
Invalid test case — wrapped with expected_error so tests can assert the correct error variant
is returned:
{
"expected_error": "UnknownSupergraph",
"data": { "graph": null }
}
Each error variant in the crate’s error types should have at least one corresponding file in the
invalid/ directory.
rtf-derive
The rtf-derive crate is a procedural macro crate. Its tests follow the
proc macro test style.
rtf-cli
This page documents how tests in the rtf-cli crate are organized and implemented. The crate
follows the CLI integration test style. This page describes the additional conventions that
apply within rtf-cli.
Organization
Tests in rtf-cli use the environment dependency → command → flags → test class → test case
hierarchy described in the CLI integration test style.
The key convention specific to this crate: when a flag identifier duplicates the environment
dependency identifier (e.g. a --github flag inside the github test module), omit the flag from
the test case name. The environment dependency already encodes it.
#![allow(unused)]
fn main() {
// Environment dependency is "github", command is "run"
// Flag "--github" is omitted — already implied by the module
test github::run::github_flag_completes // good
test github::run::github_github_flag_completes // bad — duplicated identifier
}Ignored tests
Tests with external API dependencies are #[ignore] by default. The reason string must name the
specific credential required:
#![allow(unused)]
fn main() {
#[test]
#[ignore = "requires a valid GitHub API Token"]
fn github_flag_completes() { ... }
}Tests are grouped by their credential requirement into separate files (e.g. tests/github/,
tests/graphos/), making it easy to run only the tests relevant to a given credential.
rtf-orchestrator
This page documents how tests in the rtf-orchestrator crate are organized and implemented.
The crate uses two test styles:
- Unit tests follow the unit test style
- HTTP API integration tests follow the HTTP API integration test style
See Run rtf-orchestrator tests for a step-by-step guide to running each category locally.
Categories
| Category | Location | Requires |
|---|---|---|
| Unit tests | src/ (#[cfg(test)] mod tests blocks) | Nothing |
| DB tests | src/ (gated by db_tests feature) | PostgreSQL via make db-up |
| Integration tests | tests/suite.rs (gated by k8s_tests feature) | Full tilt stack |
Running cargo test from the workspace root skips all stack-dependent tests. They must be run
explicitly via make.
Unit test organization
Unit tests follow the Module → Function → Test class → Test case hierarchy from the unit test style:
#![allow(unused)]
fn main() {
// Without simple_test_case
module::path::tests::function_test_case
// With simple_test_case
module::path::tests::function_test_class::test_case
}Integration test organization
Integration tests follow the Endpoint → Scenario naming described in the HTTP API integration test style:
#![allow(unused)]
fn main() {
endpoint_scenario
}For example: trigger_valid_test_plan_returns_200.
Feature flag annotation
Stack-dependent tests must always carry this annotation to prevent the standard test run from failing without a stack:
#![allow(unused)]
fn main() {
#[cfg_attr(not(feature = "db_tests"), ignore)] // for DB tests
#[cfg_attr(not(feature = "k8s_tests"), ignore)] // for integration tests
}Do not remove these annotations from existing tests.
Testing infrastructure
Integration tests use the following tools:
TestHelper— Defined intests/common/mod.rs. Wrapsreqwest::Clientand provides convenience methods for calling server endpointstokio—#[tokio::test]for async test functionsreqwest— HTTP client used insideTestHelperassert_fs— Temporary filesystem utilities; usesCARGO_TARGET_TMPDIRrather than/tmpto avoid macOS symlink issues with Docker volume mountssimple_test_case— Parameterized testing with#[test_case]for multiple input variations
TestHelper API
#![allow(unused)]
fn main() {
pub struct TestHelper {
client: Client,
}
}| Method | Purpose |
|---|---|
prepare_orchestrator_payload | Prepares a TriggerPayload from a test plan directory |
json_get / json_post | Typed helpers that deserialize JSON responses into the expected type |
get / post | Raw helpers returning Response for status-code assertions |
Shared logic between tests should be added as further methods on TestHelper.
Make targets
| Target | Description |
|---|---|
make db-up / make db-down | Start/stop the lightweight DB-only test stack |
make db-tests | Run DB-gated tests against the running DB stack |
make cluster-setup && make cluster-up / make cluster-teardown | Start/stop the full stack |
make integration-tests | Run tests/suite.rs against the running full stack |
make ci-tests | Full CI flow: stack up → wait → test → stack down |
rtf-orchestrator-cli
The rtf-orchestrator-cli crate follows the standard unit test style for any unit tests.
CLI behavior tests, if added, should follow the CLI integration test style.
Data structures
The rtf-config crate provides a number of data structures for working with user written YAML config files, forming an overall tree structure that is used to execute commands in the CLI.
The following pages provide a brief overview of the various data data structures we use and how they relate to one another, starting from the leaves of the tree and working up to the TestPlanConfig struct that acts as the root.
Templating fields
The leaves of the config data structures are either concrete scalar values (such as numbers, strings and booleans) or Fields which is how we support a limited form of type-checked templating within config files.
While hard coded scalar values are parsed directly using serde, Fields are allowed to be in
one of two states:
Pending, where they hold the name of a templating variable that the user must specify as part of their test plan.Resolved, where they hold a concrete scalar value, either because a value was provided directly within the config file or following successful templating.
Pending fields are indicated within a config file using "{{ field_name }}" syntax and are parsed
into the Field::Pending enum variant directly using serde. When writing new providers you should
make use of templating fields where it makes sense for users to be able to dynamically set values
when executing a test plan and avoid using them where such flexibility is not required.
For example, we do not allow the
inlinefile provider’s content argument to be templated as the intention is for this to always be provided within the test plan itself.
File providers
File providers are the primary way that rtf exposes useful functionality to users. Each provider is defined as a struct that can be parsed from a YAML snippet within a larger config file, with fields that serve as inputs to business logic from the rtf-integrations crate. Running a file provider will generate one or more resources within a user designated output directory which can then be referenced by the commands being run as part of a test plan.
Within config files, file providers are always wrapped in a NamedFileProvider that adds filename and environment variable fields along with logic for running the provider and outputting the new resources in the correct location on the filesystem.
Command providers
Command providers support running a subset of file providers in order to obtain an executable file that can be used as one of the environment setup, environment teardown or scenario commands. Within config files they are wrapped in a CommandSpec which allows the user to provide command line arguments.
Each command section supports specifying environment variables and file providers that will be made available when the command executes.
Config file formats
The rtf-config crate provides parsers for the three config files used by RTF: TestPlanConfig,
EnvironmentConfig and ScenarioConfig.
Scenario config
The scenario config pairs templating variables with a command to execute. The command can be one of two variants: a docker command, which runs the scenario inside a Docker container using a specified image, tag, and command string; or a script command, which executes a script or binary directly on the host.
Environment config
The environment config file defines how RTF sets up and tears down the test environment. There are two execution models: docker compose and script.
Docker compose environment
A docker compose environment uses a list of docker compose files to bring the environment up and
down using docker compose. Additional environment variables can be passed to docker compose up,
and any extra files the compose stack depends on can be declared in a list of File Providers, which
exposes them to the stack as environment variables.
Script environment
A script environment defines a pair of command sections: one for setting up the environment before the test is run and another for tearing it down after the test is complete.
Test plan config
The test plan config file is where the user defines the variables they wish to use for templating along with the scenario and environment configurations, either provided inline or as references to external files.