CLI reference

Kubectl Commands

Common kubectl commands for managing site recovery resources, checking status, and inspecting events

Description

This page is a reference for the kubectl commands you use day-to-day to manage Trilio Site Recovery resources across your OpenShift clusters. Because Site Recovery is a Kubernetes operator, all operations are driven by Custom Resources (CRDs); kubectl is the primary interface for creating, inspecting, and deleting those resources, monitoring controller reconciliation, and diagnosing problems. Commands are grouped by workflow stage—protection, failover, test failover, replication health, and diagnostics—so you can find the right command quickly during normal operations or an incident.

Synopsis

kubectl [--kubeconfig <path>] [-n <namespace>] <verb> <resource> [<name>] [flags]

Site Recovery spans three OpenShift clusters. Always specify --kubeconfig or set KUBECONFIG to target the correct cluster. The conventions used throughout this page are:

Variable	Cluster
`$KUBECONFIG_QUORUM`	Quorum cluster — runs the control plane controllers
`$KUBECONFIG_PRIMARY`	Primary cluster — runs production VMs
`$KUBECONFIG_DR`	DR cluster — runs standby VMs

Export these before running any command:

export KUBECONFIG_QUORUM=~/.kube/config-quorum
export KUBECONFIG_PRIMARY=~/.kube/config-primary
export KUBECONFIG_DR=~/.kube/config-dr

Flags

The flags below are standard kubectl flags that appear frequently in Site Recovery workflows. Site Recovery does not add custom kubectl plug-ins or flags.

Flag	Short	Default	Description
`--kubeconfig <path>`	—	`$KUBECONFIG` or `~/.kube/config`	Path to the kubeconfig file for the target cluster. Always set this explicitly when working across multiple clusters to avoid targeting the wrong cluster.
`--namespace <name>`	`-n`	`default`	Namespace for the resource. Quorum-side CRDs (FailoverRequest, TestFailover, ProtectionRequest, RPOEvent, ReplicationGroupStatus) live in the DR deployment namespace (e.g., `dr-prod`). Workload-side CRDs (ProtectionGroup, DRBDReplicationPolicy) live in the VM namespace or the site-recovery system namespace.
`--all-namespaces`	`-A`	false	List resources across all namespaces. Useful for getting a fleet-wide view of ProtectionGroups or FailoverRequests.
`--output <format>`	`-o`	table	Output format. Use `yaml` to inspect full resource spec and status, `json` for scripting, `wide` for extra columns, or `jsonpath` for extracting a single field.
`--watch`	`-w`	false	Stream updates as the controller reconciles. Use this to monitor failover or protection phases in real time.
`--selector <label>`	`-l`	—	Filter by label selector. Useful for listing all resources belonging to a specific Protection Group or deployment.
`--field-selector <expr>`	—	—	Filter by field value, e.g., `status.phase=Failed`.
`--show-labels`	—	false	Print all labels in the output. Helpful for identifying which deployment or Protection Group a resource belongs to.
`--dry-run=client`	—	—	Validate a manifest locally without sending it to the API server. Use before applying FailoverRequests or ProtectionRequests in production.
`--context <name>`	—	Current context	Named kubeconfig context. An alternative to `--kubeconfig` when all clusters are in a single kubeconfig file.

Subcommands

The kubectl verbs used most often with Site Recovery CRDs are listed below. There are no Site Recovery-specific subcommands; these are standard Kubernetes verbs.

Verb	Description
`apply -f <file>`	Create or update a resource from a manifest. The primary way to create ProtectionRequests, FailoverRequests, TestFailovers, and DRBDReplicationPolicies.
`get <resource> [<name>]`	List or inspect resources. Add `-o yaml` for full spec and status detail, `-w` to stream phase changes.
`describe <resource> <name>`	Show human-readable detail including events emitted by the controller. Use this first when a resource is stuck or in a Failed phase.
`delete <resource> <name>`	Delete a resource. Deleting a TestFailover triggers the CleaningUp phase. Deleting a ProtectionRequest removes protection.
`patch <resource> <name>`	Apply a partial update using JSON merge patch or strategic merge patch. Useful for updating a single field such as `spec.failoverType`.
`edit <resource> <name>`	Open the resource in your editor for interactive modification.
`wait <resource> <name> --for=condition=<cond>`	Block until a specific condition is true. Useful in automation scripts waiting for failover to complete.
`logs <pod> [-n <ns>] [-c <container>]`	Stream controller logs. Target the appropriate controller pod on the quorum cluster.
`events --for <resource>/<name>`	Show events scoped to a single resource. Faster than `describe` when you only need the event stream.

Environment variables

Variable	Default	Description
`KUBECONFIG`	`~/.kube/config`	Path to the active kubeconfig file. Override with `--kubeconfig` per-command or export separate variables per cluster as shown in the synopsis section.
`KUBECONFIG_QUORUM`	(none)	Convention used in this documentation and in the `scripts/` runbooks. Set to the kubeconfig path for your quorum cluster.
`KUBECONFIG_PRIMARY`	(none)	Convention used in this documentation and in the `scripts/` runbooks. Set to the kubeconfig path for your primary (active VM) cluster.
`KUBECONFIG_DR`	(none)	Convention used in this documentation and in the `scripts/` runbooks. Set to the kubeconfig path for your DR (standby VM) cluster.

Exit codes

These are standard kubectl exit codes. Site Recovery does not modify them.

Code	Meaning
`0`	Command succeeded.
`1`	Command failed — server-side error, resource not found, or validation rejected the manifest. Check stderr for details.
`130`	Process interrupted by Ctrl-C (SIGINT). Common when watching (`-w`) a long-running operation.

Examples

Examples are grouped by workflow. Each example includes the target cluster in the caption.

Checking installed CRDs and component health

List all Site Recovery CRDs (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get crds | grep siterecovery.trilio.io

Expected output (subset):

drbd replicationpolicies.siterecovery.trilio.io   2024-01-15T10:00:00Z
drbdresources.siterecovery.trilio.io              2024-01-15T10:00:00Z
failoverrequests.siterecovery.trilio.io           2024-01-15T10:00:00Z
protectiongroups.siterecovery.trilio.io           2024-01-15T10:00:00Z
protectionrequests.siterecovery.trilio.io         2024-01-15T10:00:00Z
protectionzones.siterecovery.trilio.io            2024-01-15T10:00:00Z
replicationgroupstatuses.siterecovery.trilio.io   2024-01-15T10:00:00Z
rpoevents.siterecovery.trilio.io                  2024-01-15T10:00:00Z
testfailovers.siterecovery.trilio.io              2024-01-15T10:00:00Z

Verify quorum control plane pods are running (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod

Expected output:

NAME                                              READY   STATUS    RESTARTS   AGE
site-recovery-quorum-control-plane-7d9f6c-xvp2k  1/1     Running   0          2d
site-recovery-protectionzone-controller-5bcd-abc  1/1     Running   0          2d

Verify DRBD node agent DaemonSet on the primary cluster

kubectl --kubeconfig $KUBECONFIG_PRIMARY get daemonset -n trilio-site-recovery-system

Expected output:

NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
drbd-node-agent  3         3         3       3            3

ProtectionZone — viewing your DR deployment definition

List all ProtectionZones (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionzone -A

Expected output:

NAMEPACE   NAME       DISPLAY NAME   STORAGE BACKEND   AGE
dr-prod    zone-prod  Production DR  drbd-operator     5d

Inspect a ProtectionZone in full (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionzone zone-prod -n dr-prod -o yaml

Key fields to review in the output:

spec.clusters — the primary and DR cluster references
spec.storageBackendMode — drbd-operator for DRBD Operator deployments
status.conditions — whether the zone is ready

ProtectionRequest — protecting a VM (DRBD Operator model)

Apply a ProtectionRequest (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-database
  namespace: dr-prod
spec:
  sourceCluster: primary-cluster
  virtualMachine:
    name: vm-database
    namespace: production
EOF

Watch protection phases (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-vm-database \
  -n dr-prod -w

Expected phase progression:

NAME                 PHASE       AGE
protect-vm-database  Pending     0s
protect-vm-database  Validating  5s
protect-vm-database  Protecting  12s
protect-vm-database  Protected   45s

Inspect full ProtectionRequest status (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-vm-database \
  -n dr-prod -o yaml

Look at status.phase and status.conditions to confirm the protection-controller has created the DRBDResource pair and switched the VM to DRBD-backed frontend PVCs.

List all ProtectionRequests across all namespaces (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest -A

DRBDReplicationPolicy — inspecting replication configuration

List replication policies (primary cluster)

kubectl --kubeconfig $KUBECONFIG_PRIMARY get drbdreplicationpolicy -A

Show full policy spec including protocol and storage class mappings

kubectl --kubeconfig $KUBECONFIG_PRIMARY get drbdreplicationpolicy cross-cluster-policy \
  -n dr-prod -o yaml

Key fields:

spec.drbdProtocol — C (synchronous, RPO=0) or A (asynchronous)
spec.clusters — primary and DR endpoint definitions
spec.replicationMode — replication direction
spec.rpo — RPO objective configuration

Check whether a policy is the default

kubectl --kubeconfig $KUBECONFIG_PRIMARY get drbdreplicationpolicy cross-cluster-policy \
  -n dr-prod -o jsonpath='{.spec.isDefault}'

DRBDResource — inspecting per-VM replication state

List all DRBDResources on the primary cluster

kubectl --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A

Expected output:

NAMESPACE    NAME                  ROLE     SIDE      AGE
production   drbd-vm-database      Primary  primary   3d
production   drbd-vm-web-server    Primary  primary   3d

Inspect replication state for a single VM's DRBD resource

kubectl --kubeconfig $KUBECONFIG_PRIMARY get drbdresource drbd-vm-database \
  -n production -o yaml

Review:

spec.role — Primary or Secondary
spec.side — primary or dr
status.volumes — per-disk sync progress and connection state
spec.paused — whether replication is paused

Compare the same DRBDResource from the DR side

kubectl --kubeconfig $KUBECONFIG_DR get drbdresource drbd-vm-database \
  -n production -o yaml

ReplicationGroupStatus — monitoring replication health

List all ReplicationGroupStatuses (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -A

Expected output:

NAMESPACE  NAME           HEALTH    PROTOCOL  AGE
dr-prod    rgs-prod-pg    Healthy   C         2d

Get aggregated health for a specific Protection Group

kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus rgs-prod-pg \
  -n dr-prod -o yaml

Key fields:

status.health — Healthy, Degraded, or Critical
status.perVolumeDetail — per-disk sync state
status.lastSyncTimestamp — time of last successful sync

Watch replication health in real time (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -w

RPOEvent — auditing replication lag violations

List recent RPO events (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod --sort-by=.spec.timestamp

Inspect a specific RPO violation

kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent <event-name> -n dr-prod -o yaml

Key fields:

spec.eventType — violation category
spec.rpoAtEvent — observed lag in seconds at the time of the event
spec.violationReason — cause of the lag (e.g., network saturation)
spec.newRPOStatus — severity after the event
spec.affectedVolumes — which volumes were out of sync

Filter only critical RPO violations

kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod \
  -o jsonpath='{range .items[?(@.spec.newRPOStatus=="critical")]}{.metadata.name}\t{.spec.rpoAtEvent}s\n{end}'

FailoverRequest — executing failover

Create a planned failover (quorum cluster)

A planned failover gracefully shuts down VMs on the primary cluster before promoting DRBD volumes on the DR cluster.

kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-prod-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-production-pg
  targetCluster: dr-cluster
  failoverType: planned
  drainTimeoutSeconds: 120
  batchBootTimeoutSeconds: 300
EOF

Create an unplanned failover (quorum cluster)

Use this only when the primary cluster is unavailable. DRBD volumes are force-promoted on the DR cluster without waiting for the primary to shut down.

kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: emergency-failover-prod-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-production-pg
  targetCluster: dr-cluster
  failoverType: unplanned
EOF

Watch failover progress (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest failover-prod-pg \
  -n dr-prod -w

Expected phase progression:

NAME                 PHASE       AGE
failover-prod-pg     Pending     0s
failover-prod-pg     InProgress  8s
failover-prod-pg     Completed   4m22s

Inspect the full FailoverRequest after completion

kubectl --kubeconfig $KUBECONFIG_QUORUM describe failoverrequest failover-prod-pg \
  -n dr-prod

The Events section shows each step (VM shutdown, volume promotion, VM startup) with timestamps, which is critical for RTO auditing.

List all FailoverRequests across all deployments (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest -A

TestFailover — non-disruptive DR validation

Create a TestFailover (quorum cluster)

The test-failover-controller creates snapshot-based copies of VMs in an isolated namespace on whichever cluster is currently standby, runs verification checks, then cleans up. Production workloads are not affected.

kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-prod-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-production-pg
  cleanupPolicy: Manual
  retentionTime: 2h
  batchBootTimeoutSeconds: 300
EOF

Watch TestFailover phases (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM get testfailover test-prod-pg -n dr-prod -w

Expected phase progression:

NAME            PHASE               TARGET CLUSTER   AGE
test-prod-pg    Pending             —                0s
test-prod-pg    CreatingSnapshots   dr-cluster       5s
test-prod-pg    CreatingVolumes     dr-cluster       30s
test-prod-pg    CreatingVMs         dr-cluster       55s
test-prod-pg    VerifyingData       dr-cluster       2m10s
test-prod-pg    Succeeded           dr-cluster       3m45s

Inspect which cluster was selected as the test target

kubectl --kubeconfig $KUBECONFIG_QUORUM get testfailover test-prod-pg -n dr-prod \
  -o jsonpath='{.status.targetCluster}  reason: {.status.targetClusterReason}'

Trigger cleanup after reviewing test results (quorum cluster)

When cleanupPolicy: Manual, the controller waits for you to delete the TestFailover before removing test VMs and PVCs:

kubectl --kubeconfig $KUBECONFIG_QUORUM delete testfailover test-prod-pg -n dr-prod

Watch the CleaningUp phase complete:

kubectl --kubeconfig $KUBECONFIG_QUORUM get testfailover test-prod-pg -n dr-prod -w

List all TestFailovers

kubectl --kubeconfig $KUBECONFIG_QUORUM get testfailover -A

Diagnostics — controller logs

All quorum-side controllers (failover-controller, protection-controller, pg-sync-controller, test-failover-controller) run inside the site-recovery-quorum-control-plane deployment. Use the pod logs to diagnose stuck or failed resources.

Get the quorum control plane pod name

kubectl --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod \
  -l app.kubernetes.io/component=quorum-control-plane

Stream all controller logs (quorum cluster)

kubectl --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \
  deployment/site-recovery-quorum-control-plane -f

Filter logs for a specific FailoverRequest

kubectl --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \
  deployment/site-recovery-quorum-control-plane \
  | grep failover-prod-pg

Stream DRBD node agent logs from all nodes on the primary cluster

kubectl --kubeconfig $KUBECONFIG_PRIMARY logs daemonset/drbd-node-agent \
  -n trilio-site-recovery-system -f

Stream DRBD node agent logs from a specific node

kubectl --kubeconfig $KUBECONFIG_PRIMARY logs \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=worker-node-01 -f

Diagnostics — events and describe

Show controller events for a stuck ProtectionRequest

kubectl --kubeconfig $KUBECONFIG_QUORUM events \
  --for protectionrequest/protect-vm-database -n dr-prod

Describe a failed FailoverRequest for root-cause analysis

kubectl --kubeconfig $KUBECONFIG_QUORUM describe failoverrequest failover-prod-pg \
  -n dr-prod

The Conditions and Events blocks show exactly which step the failover-controller reported failure on.

Describe a TestFailover stuck in VerifyingData

kubectl --kubeconfig $KUBECONFIG_QUORUM describe testfailover test-prod-pg -n dr-prod

Diagnostics — waiting for conditions in automation scripts

Wait for a ProtectionRequest to reach the Protected phase

kubectl wait uses condition fields from the resource's status.conditions. The exact condition name is set by the protection-controller; adapt the condition value to match what the controller emits.

kubectl --kubeconfig $KUBECONFIG_QUORUM wait \
  protectionrequest/protect-vm-database \
  -n dr-prod \
  --for=condition=Protected \
  --timeout=5m

Wait for a FailoverRequest to complete

kubectl --kubeconfig $KUBECONFIG_QUORUM wait \
  failoverrequest/failover-prod-pg \
  -n dr-prod \
  --for=condition=Completed \
  --timeout=10m

Fleet-wide status summary

Run these commands to get a quick health overview across all DR namespaces on the quorum cluster.

# All ProtectionRequests and their phases
kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest -A

# All FailoverRequests (detect any stuck or failed operations)
kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest -A

# All TestFailovers
kubectl --kubeconfig $KUBECONFIG_QUORUM get testfailover -A

# Replication health across all Protection Groups
kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -A

# RPO violations in the last reconcile window
kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent -A --sort-by=.spec.timestamp

# DRBDResources on the primary cluster (per-VM replication roles)
kubectl --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A

# DRBDResources on the DR cluster
kubectl --kubeconfig $KUBECONFIG_DR get drbdresource -A

Collecting a support bundle with tsr-gather

tsr-gather is an oc adm must-gather compatible tool. Run it against the quorum cluster; it automatically collects logs and CRD state from the quorum cluster and all associated workload clusters into a single read-only bundle. Secret data, kubeconfigs, and DRBD shared secrets are never collected.

# Collect the full estate (default: 48h of logs)
oc adm must-gather --image=<tsr-gather-image>:<tag>

# Collect only the last 2 hours of logs
oc adm must-gather --image=<tsr-gather-image>:<tag> -- /usr/bin/gather --since 2h

# Collect only a specific ProtectionZone
oc adm must-gather --image=<tsr-gather-image>:<tag> -- /usr/bin/gather --zone dr-prod

# Collect only from the quorum cluster (no cross-cluster queries)
oc adm must-gather --image=<tsr-gather-image>:<tag> -- /usr/bin/gather --local-only

After collection, read SUMMARY.md for a triage overview, then manifest.yaml for the full inventory of what was gathered.