Running and fetching results
In this guide, we’ll submit our Test Plan to the RTF Orchestrator Service using
rtf remote ci-run, follow it through the execution lifecycle in the Orchestrator UI, and then
fetch the results using rtf remote run-output.
Prerequisites
- Completed the “Making your test plan Orchestrator-ready” guide
- Completed the Getting set up with the Orchestrator guide
What happens when we trigger a run
Before a Test Plan can be submitted to the Orchestrator, it must be converted into a
Trigger Payload. This is simply a JSON representation of the Test Plan in the inline form
generated by rtf template, along with any local files that are referenced by file providers
contained in the Test Plan.
The rtf remote commands that trigger runs build this payload for us, submit it to the
Orchestrator, and receive a Test Run Summary in return that contains the ID of our newly
created Test Run. If we want to see what is being submitted, we can build the payload
ourselves using rtf remote prepare:
rtf remote prepare test-plan.yaml | jq
rtf remote prepare writes its output as a single JSON object to stdout. You should never need to
edit this payload by hand, and doing so is likely to result in broken test runs.
Triggering a run with rtf remote ci-run
rtf remote ci-run prepares the Trigger Payload, submits it for execution, and then polls for
status updates until the run completes:
rtf remote ci-run test-plan.yaml
The command prints a status table that updates at each poll interval (default: 10 seconds) showing the most recent activity associated with your run:
status latest_message running successful failed unrunnable cancelled
INITIALISING run created 1 0 0 0 0
RESOLVING resolving environment config 1 0 0 0 0
PROVISIONING deploying environment 1 0 0 0 0
RUNNING running scenario 1 0 0 0 0
SUCCESSFUL scenario completed successfully 0 1 0 0 0
Test run complete. Final status: successful
Run 'rtf remote run-status <run-id> --with-executions' to view the summary for this run
Run the following to fetch the status, log or output.zip for an individual execution:
rtf remote execution-status $ID
rtf remote execution-log $ID
rtf remote execution-output $ID
If you’d rather not wait for the run to complete, rtf remote run triggers the run in the same way
but exits straight away, printing the ID of the run along with a link to its page in the
Orchestrator UI.
The execution lifecycle
The status column from that table shows the current execution status of the run as a whole. Each
test run (and test execution inside of a run) progresses through the following statuses as it
executes:
| Status | Description |
|---|---|
INITIALISING | The run has been accepted and executions are being queued |
RESOLVING | The Orchestrator is resolving the environment configuration |
PROVISIONING | The environment namespace is being created and services are being deployed |
ENVIRONMENT_READY | The environment is healthy and the Scenario is starting |
RUNNING | The Scenario job is running |
SUCCESSFUL | The Scenario exited cleanly |
FAILED | The Scenario exited with a non-zero exit code |
UNRUNNABLE | The execution could not be started (for example, due to an environment failure) |
CANCELLED | The run was cancelled by a user |
The running column in the table counts executions that are not yet in a terminal state. For matrix
Test Plans, the table shows the aggregate count across all executions.
rtf remote ci-run exits with:
0if all executions wereSUCCESSFUL1if any executionFAILED2if the run wasUNRUNNABLEorCANCELLED
This makes it suitable for scripting, allowing a non-zero exit code to signal that a test run did not succeed. To run Test Plans from CI, see Running from CI.
Adjusting the poll interval
You can use --poll-interval-seconds to change how frequently the command polls for status updates:
rtf remote ci-run test-plan.yaml --poll-interval-seconds 30
Following the run in the Orchestrator UI
While our run is in progress we can also follow it in the Orchestrator UI. Enter the test run
ID printed by rtf remote ci-run into the box in the header and select View status to open the
run’s page.
The run page refreshes automatically while the run is in progress and shows the status of each of
our Test Executions. Each execution links to its logs in GCP, its metrics in Grafana, and its
log.txt and output.zip. If we need to stop our run early, we can select Cancel run.
See Working with the Orchestrator UI for everything else the UI can do.
Fetching output for a single execution
Once our run is complete, we can use rtf remote execution-output to fetch the log and output zip
for a specific execution.
Note
Output is only retained for a limited time. Fetch anything you want to keep promptly. We can find the execution ID on the run’s page in the Orchestrator UI, or by running
rtf remote run-status <run-id> --with-executions.
rtf remote execution-output <execution-id>
This writes the following to a newly created output/ directory by default:
output/
execution-summary.json # JSON summary of the execution status and history
log.txt # Orchestrator log for the execution
output.zip # Zipped output directory from the Scenario and service logs
The output.zip contains:
output/— files written to$OUTPUT_PATHby the Scenario containeroutput/logs/<pod>/<container>.txt— container logs for services withrtf.io/log-collection: "true"
If we want to write output to a different directory, we can use --outdir:
rtf remote execution-output <execution-id> --outdir my-results
If the output directory already exists, the command exits with an error. We can use --force to
remove it and start fresh:
rtf remote execution-output <execution-id> --force
Fetching output for all executions in a run
For matrix Test Plans, we can use rtf remote run-output to fetch the output for every execution in
our run in parallel; it groups each execution’s output into a named subdirectory:
rtf remote run-output <run-id>
This produces:
output/
run-summary.json # JSON summary of the full run status
<execution-name>/
log.txt
output.zip
<execution-name>/
log.txt
output.zip
...
We can use the same --outdir and --force flags as with rtf remote execution-output.
Next steps
You’ve now completed the “Running with the Orchestrator” tutorial series: you can submit a Test Plan to the Orchestrator, follow it through to completion, and fetch its results. To go further:
- Registering and managing Known Test Plans — run your Test Plan from the Orchestrator UI and CI
- Working with the Orchestrator UI — monitor, trigger, and cancel runs from the UI
- Using
rtf remote— reference for thertf remotesubcommands - Troubleshooting — common issues and how to resolve them