Running and fetching results

In this guide, we’ll submit our Test Plan to the RTF Orchestrator Service using rtf remote ci-run, watch it progress through the execution lifecycle, and then fetch the results using rtf remote run-output.

Prerequisites

• Completed the “Making your test plan Orchestrator-ready” guide
• gcloud CLI installed and authenticated with gcloud auth application-default login
• Access to the Orchestrator at https://api.rtf.apollographql.com granted by the Runtime Readiness team

Doing things manually

For actual execution we recommend using the rtf remote ci-run subcommand which handles all of the following steps for you automatically. But before we take a look at how to use that subcommand, we’ll first explain what that is doing under the hood so you have an understanding of what is happening when the Orchestrator executes a Test Plan.

Preparing the Trigger Payload

Before a Test Plan can be submitted to the Orchestrator, it must be converted into a Trigger Payload. This is simply a JSON representation of the Test Plan in the inline form generated by rtf template, along with any local files that are referenced by file providers contained in the Test Plan.

To build the payload manually we use the rtf remote prepare command:

rtf remote prepare test-plan.yaml

rtf remote prepare writes its output as a single JSON object to stdout, allowing you to pipe it directly to other tools or inspect it with jq. You should not edit this payload by hand before submitting to the Orchestrator as this is likely to result in broken test runs.

Triggering a run with rtf remote request

To manually trigger a Test Run we can use the rtf remote request subcommand that allows for making individual requests to the Orchestrator. The trigger request is made using the payload we covered in the previous section like so:

rtf remote request -X POST --body "$(rtf remote prepare test-plan.yaml)" test-run/trigger

Provided the Test Plan is accepted, this will return a Test Run Summary that details the current status of our newly created test run. We won’t go into the details of everything that is contained in the summary just now, as all we need for the next step is the ID of our run which we can extract from the summary by piping it through jq and using the following filter:

| jq -r '.id'

Polling for status updates

Armed with our test run ID we can use the test-run/$id/status endpoint to see what is happening as our run is processed by the Orchestrator:

rtf remote request "test-run/$RUN_ID/status" | jq

This endpoint returns the same Test Run Summary data as the trigger endpoint, but with the details being pulled fresh each time we make the request. Using this we can check the value of the current_status field to see how our run is progressing (see “The execution lifecycle” below).

Once our run enters a terminal state we are able to try pulling the results for analysis. But before we take a look at that, let’s introduce the rtf remote ci-run subcommand that automates the process we’ve just implemented by hand.

Triggering a run with rtf remote ci-run

rtf remote ci-run is the primary command for running a Test Plan through the Orchestrator. It automates all of the steps we just covered, preparing the Trigger Payload, submitting it for execution, and polling for status updates until the run completes.

rtf remote ci-run test-plan.yaml

The command prints a status table that updates at each poll interval (default: 10 seconds) showing the most recent activity associated with your run:

 status        latest_message                                                running  successful  failed  unrunnable
 INITIALISING  run created                                                   1        0           0       0
 RESOLVING     resolving environment config                                  1        0           0       0
 PROVISIONING  deploying environment                                         1        0           0       0
 RUNNING       running scenario                                              1        0           0       0
 SUCCESSFUL    scenario completed successfully                               0        1           0       0


Test run complete. Final status: successful

Run 'rtf remote request test-run/<run-id>/status' to view the summary for this run

Run the following to fetch the status, log or output.zip for an execution:
   rtf remote request test-execution/$ID/status
   rtf remote request test-execution/$ID/log.txt
   rtf remote request test-execution/$ID/output.zip > output.zip

The execution lifecycle

The status column from that table shows the current execution status of the run as a whole. Each test run (and test execution inside of a run) progresses through the following statuses as it executes:

The running column in the table counts executions that are not yet in a terminal state. For matrix Test Plans, the table shows the aggregate count across all executions.

rtf remote ci-run exits with:

• 0 if all executions were SUCCESSFUL
• 1 if any execution FAILED
• 2 if any execution was UNRUNNABLE

This makes it suitable for use in CI pipelines, allowing a non-zero exit code to fail the pipeline when a test run does not succeed.

Adjusting the poll interval

You can use --poll-interval-seconds to change how frequently the command polls for status updates:

rtf remote ci-run test-plan.yaml --poll-interval-seconds 30

Fetching output for a single execution

Once our run is complete, we can use rtf remote execution-output to fetch the log and output zip for a specific execution. We can find the execution ID in the output of rtf remote ci-run, or by querying the run status with rtf remote request test-run/<run-id>/status.

rtf remote execution-output <execution-id>

This writes the following to a newly created output/ directory by default:

output/
  execution-summary.json   # JSON summary of the execution status and history
  log.txt                  # Orchestrator log for the execution
  output.zip               # Zipped output directory from the Scenario and service logs

The output.zip contains:

• output/ — files written to $OUTPUT_PATH by the Scenario container
• output/logs/<pod>/<container>.txt — container logs for services with rtf.io/log-collection: "true"

If we want to write output to a different directory, we can use --outdir:

rtf remote execution-output <execution-id> --outdir my-results

If the output directory already exists, the command exits with an error. We can use --force to remove it and start fresh:

rtf remote execution-output <execution-id> --force

Fetching output for all executions in a run

For matrix Test Plans, we can use rtf remote run-output to fetch the output for every execution in our run in parallel; it groups each execution’s output into a named subdirectory:

rtf remote run-output <run-id>

This produces:

output/
  run-summary.json         # JSON summary of the full run status
  <execution-name>/
    log.txt
    output.zip
  <execution-name>/
    log.txt
    output.zip
  ...

We can use the same --outdir and --force flags as with rtf remote execution-output.

Making raw API requests

As covered above, rtf remote request lets us make arbitrary authenticated requests to the Orchestrator directly, for situations where higher level commands such as rtf remote ci-run don’t provide the functionality we need:

rtf remote request test-run/<run-id>/status
rtf remote request test-execution/<execution-id>/log.txt

The response body for requests we make this way is always written to stdout. We can use -X to specify a method and -b to supply a request body. See the CLI reference for the full set of options.

Next steps

You’ve now completed the “Running with the Orchestrator” tutorial series: you can submit a Test Plan to the Orchestrator, monitor it through to completion, and fetch its results. To go further:

• CLI reference — full reference for rtf remote and its subcommands
• Troubleshooting — common issues and how to resolve them