Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Running and fetching results

In this guide, we’ll submit our Test Plan to the RTF Orchestrator Service using rtf remote ci-run, follow it through the execution lifecycle in the Orchestrator UI, and then fetch the results using rtf remote run-output.

Prerequisites

What happens when we trigger a run

Before a Test Plan can be submitted to the Orchestrator, it must be converted into a Trigger Payload. This is simply a JSON representation of the Test Plan in the inline form generated by rtf template, along with any local files that are referenced by file providers contained in the Test Plan.

The rtf remote commands that trigger runs build this payload for us, submit it to the Orchestrator, and receive a Test Run Summary in return that contains the ID of our newly created Test Run. If we want to see what is being submitted, we can build the payload ourselves using rtf remote prepare:

rtf remote prepare test-plan.yaml | jq

rtf remote prepare writes its output as a single JSON object to stdout. You should never need to edit this payload by hand, and doing so is likely to result in broken test runs.

Triggering a run with rtf remote ci-run

rtf remote ci-run prepares the Trigger Payload, submits it for execution, and then polls for status updates until the run completes:

rtf remote ci-run test-plan.yaml

The command prints a status table that updates at each poll interval (default: 10 seconds) showing the most recent activity associated with your run:

 status        latest_message                                                running  successful  failed  unrunnable  cancelled
 INITIALISING  run created                                                   1        0           0       0           0
 RESOLVING     resolving environment config                                  1        0           0       0           0
 PROVISIONING  deploying environment                                         1        0           0       0           0
 RUNNING       running scenario                                              1        0           0       0           0
 SUCCESSFUL    scenario completed successfully                               0        1           0       0           0


Test run complete. Final status: successful

Run 'rtf remote run-status <run-id> --with-executions' to view the summary for this run

Run the following to fetch the status, log or output.zip for an individual execution:
   rtf remote execution-status $ID
   rtf remote execution-log $ID
   rtf remote execution-output $ID

If you’d rather not wait for the run to complete, rtf remote run triggers the run in the same way but exits straight away, printing the ID of the run along with a link to its page in the Orchestrator UI.

The execution lifecycle

The status column from that table shows the current execution status of the run as a whole. Each test run (and test execution inside of a run) progresses through the following statuses as it executes:

StatusDescription
INITIALISINGThe run has been accepted and executions are being queued
RESOLVINGThe Orchestrator is resolving the environment configuration
PROVISIONINGThe environment namespace is being created and services are being deployed
ENVIRONMENT_READYThe environment is healthy and the Scenario is starting
RUNNINGThe Scenario job is running
SUCCESSFULThe Scenario exited cleanly
FAILEDThe Scenario exited with a non-zero exit code
UNRUNNABLEThe execution could not be started (for example, due to an environment failure)
CANCELLEDThe run was cancelled by a user

The running column in the table counts executions that are not yet in a terminal state. For matrix Test Plans, the table shows the aggregate count across all executions.

rtf remote ci-run exits with:

  • 0 if all executions were SUCCESSFUL
  • 1 if any execution FAILED
  • 2 if the run was UNRUNNABLE or CANCELLED

This makes it suitable for scripting, allowing a non-zero exit code to signal that a test run did not succeed. To run Test Plans from CI, see Running from CI.

Adjusting the poll interval

You can use --poll-interval-seconds to change how frequently the command polls for status updates:

rtf remote ci-run test-plan.yaml --poll-interval-seconds 30

Following the run in the Orchestrator UI

While our run is in progress we can also follow it in the Orchestrator UI. Enter the test run ID printed by rtf remote ci-run into the box in the header and select View status to open the run’s page.

The run page refreshes automatically while the run is in progress and shows the status of each of our Test Executions. Each execution links to its logs in GCP, its metrics in Grafana, and its log.txt and output.zip. If we need to stop our run early, we can select Cancel run.

See Working with the Orchestrator UI for everything else the UI can do.

Fetching output for a single execution

Once our run is complete, we can use rtf remote execution-output to fetch the log and output zip for a specific execution.

Note

Output is only retained for a limited time. Fetch anything you want to keep promptly. We can find the execution ID on the run’s page in the Orchestrator UI, or by running rtf remote run-status <run-id> --with-executions.

rtf remote execution-output <execution-id>

This writes the following to a newly created output/ directory by default:

output/
  execution-summary.json   # JSON summary of the execution status and history
  log.txt                  # Orchestrator log for the execution
  output.zip               # Zipped output directory from the Scenario and service logs

The output.zip contains:

  • output/ — files written to $OUTPUT_PATH by the Scenario container
  • output/logs/<pod>/<container>.txt — container logs for services with rtf.io/log-collection: "true"

If we want to write output to a different directory, we can use --outdir:

rtf remote execution-output <execution-id> --outdir my-results

If the output directory already exists, the command exits with an error. We can use --force to remove it and start fresh:

rtf remote execution-output <execution-id> --force

Fetching output for all executions in a run

For matrix Test Plans, we can use rtf remote run-output to fetch the output for every execution in our run in parallel; it groups each execution’s output into a named subdirectory:

rtf remote run-output <run-id>

This produces:

output/
  run-summary.json         # JSON summary of the full run status
  <execution-name>/
    log.txt
    output.zip
  <execution-name>/
    log.txt
    output.zip
  ...

We can use the same --outdir and --force flags as with rtf remote execution-output.

Next steps

You’ve now completed the “Running with the Orchestrator” tutorial series: you can submit a Test Plan to the Orchestrator, follow it through to completion, and fetch its results. To go further: