Automation and GitOps

This page explains how to integrate Trilio Site Recovery with GitOps tools—ArgoCD and Flux—and CI/CD pipelines to automate disaster recovery workflows on OpenShift. Because every DR operation in Site Recovery is expressed as a Kubernetes Custom Resource (ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, and DRBDReplicationPolicy), the entire DR lifecycle maps naturally onto GitOps principles: desired state lives in Git, controllers continuously reconcile toward that state, and automation pipelines can trigger or verify failover without any out-of-band scripts. This approach gives your team an auditable, repeatable, and policy-driven DR practice that integrates with existing OpenShift delivery toolchains.

Before you begin, ensure the following are in place:

• OpenShift ≥ 4.14 on all clusters (primary, DR, and quorum)
• OpenShift Virtualization (CNV) ≥ 1.0 installed on the primary and DR clusters
• DRBD kernel module ≥ 9.x on all worker nodes
• Helm ≥ 3.0 available in your pipeline environment
• kubectl or oc CLI available in pipeline agents/runners
• Kubeconfig files for the quorum cluster, primary cluster, and DR cluster, with appropriate RBAC permissions to create and patch Site Recovery CRDs
• Site Recovery components deployed: site-recovery-protectionzone-controller and site-recovery-quorum-control-plane on the quorum cluster; site-recovery-workload-control-plane on the primary and DR clusters
• DRBD Operator installed on the primary and DR clusters (DRBD Operator deployment model)
• ArgoCD ≥ 2.6 or Flux ≥ 2.0 installed on the quorum cluster or a dedicated management cluster with access to all three kubeconfigs
• TCP ports 7000–7999 open between primary and DR worker nodes (DRBD replication)
• TCP port 6443 open from the quorum cluster to primary and DR API servers
• A Git repository structured with per-deployment directories for DR manifests (see Configuration)

The following steps configure Site Recovery manifests for GitOps management and wire them into your CI/CD pipeline. All commands assume you have set the appropriate --kubeconfig context for each cluster.

Step 1: Structure your Git repository

Organize your DR manifests so that each deployment namespace maps to a directory. This lets ArgoCD or Flux target a specific deployment without overlapping resources from other tenants.

site-recovery/
├── base/
│   ├── drbd-replication-policy.yaml
│   └── protection-zone.yaml
├── deployments/
│   ├── prod-east/
│   │   ├── protection-group.yaml
│   │   ├── protection-requests/
│   │   │   ├── vm-web-01.yaml
│   │   │   └── vm-db-01.yaml
│   │   └── replication-group-status.yaml
│   └── prod-west/
│       └── ...
└── tests/
    └── test-failover.yaml

Step 2: Commit base CRs to Git

Create and commit your DRBDReplicationPolicy and ProtectionZone manifests. These define the replication topology and must exist before protection resources are applied.

# base/drbd-replication-policy.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: prod-replication-policy
  namespace: dr-prod-east
spec:
  clusters:
    - name: primary
      kubeConfigSecretRef:
        name: primary-kubeconfig
    - name: dr
      kubeConfigSecretRef:
        name: dr-kubeconfig
  drbdProtocol: C
  replicationMode: synchronous
  rpo:
    objectiveSeconds: 0
  networkConfig:
    port: 7000

git add base/drbd-replication-policy.yaml
git commit -m "feat: add DRBD replication policy for prod-east"
git push

Step 3: Configure ArgoCD to watch the quorum cluster namespace

Create an ArgoCD Application CR that targets your DR deployment namespace on the quorum cluster. Apply this to your ArgoCD instance.

# argocd-application-dr-prod-east.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dr-prod-east
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/site-recovery.git
    targetRevision: main
    path: deployments/prod-east
  destination:
    server: https://quorum-cluster.example.com:6443
    namespace: dr-prod-east
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

kubectl apply -f argocd-application-dr-prod-east.yaml \
  --kubeconfig ~/.kube/config-argocd

Step 4: Configure Flux (alternative to ArgoCD)

If you use Flux, create a Kustomization resource that targets the same path.

# flux-kustomization-dr-prod-east.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dr-prod-east
  namespace: flux-system
spec:
  interval: 2m
  path: ./deployments/prod-east
  prune: true
  sourceRef:
    kind: GitRepository
    name: site-recovery
  targetNamespace: dr-prod-east
  kubeConfig:
    secretRef:
      name: quorum-cluster-kubeconfig

kubectl apply -f flux-kustomization-dr-prod-east.yaml \
  --kubeconfig ~/.kube/config-quorum

Step 5: Add ProtectionRequest manifests and push

For each VM you want to protect, add a ProtectionRequest manifest to the deployment directory and push it. The protection-controller on the quorum cluster will reconcile the request automatically.

# deployments/prod-east/protection-requests/vm-web-01.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-web-01
  namespace: dr-prod-east
spec:
  sourceCluster: primary
  virtualMachine:
    name: vm-web-01
    namespace: production
  replicationConfig:
    policyRef:
      name: prod-replication-policy

git add deployments/prod-east/protection-requests/vm-web-01.yaml
git commit -m "feat: protect vm-web-01 in prod-east deployment"
git push

ArgoCD or Flux will detect the new commit and apply the manifest. The protection-controller will transition the ProtectionRequest through Pending → Provisioning → Protected.

Step 6: Add a CI/CD job to run scheduled test failovers

Create a pipeline job (OpenShift Pipelines / Tekton, GitHub Actions, GitLab CI, or Jenkins) that applies a TestFailover CR, waits for it to reach Succeeded, and then validates the result. Store the TestFailover manifest in the tests/ directory.

# tests/test-failover.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: weekly-dr-test-$(date +%Y%m%d)
  namespace: dr-prod-east
spec:
  protectionGroupRef:
    name: prod-east-pg
    namespace: dr-prod-east
  cleanupPolicy: OnSuccess
  timeout: 30m
  batchBootTimeoutSeconds: 300
  verification:
    enabled: true

A minimal pipeline step using oc:

#!/usr/bin/env bash
set -euo pipefail

TF_NAME="weekly-dr-test-$(date +%Y%m%d%H%M%S)"
NAMESPACE="dr-prod-east"

# Apply the TestFailover CR
oc apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: ${TF_NAME}
  namespace: ${NAMESPACE}
spec:
  protectionGroupRef:
    name: prod-east-pg
    namespace: ${NAMESPACE}
  cleanupPolicy: OnSuccess
  timeout: 30m
  batchBootTimeoutSeconds: 300
  verification:
    enabled: true
EOF

# Wait for the TestFailover to complete
oc wait testfailover/${TF_NAME} \
  --namespace=${NAMESPACE} \
  --for=jsonpath='{.status.phase}'=Succeeded \
  --timeout=35m

echo "Test failover ${TF_NAME} succeeded."

Site Recovery's GitOps behavior is governed by the spec fields of each Custom Resource. The table below covers the fields most relevant to automation and pipeline use.

DRBDReplicationPolicy

This CR defines the replication contract between clusters and is the foundation for all protection resources in a deployment.

ProtectionGroup

Groups VMs that must fail over together. The desiredState field is the primary handle for automation.

ProtectionRequest

Requests protection for a single VM in the DRBD Operator model.

FailoverRequest

Triggers a failover. In GitOps workflows, committing this manifest to Git initiates a failover; deleting or reverting the manifest does not undo the failover—use a separate failback manifest.

TestFailover

Used in scheduled pipeline jobs to validate DR readiness non-disruptively.

ReplicationGroupStatus

Drives continuous health monitoring and can be read by pipeline health gates.

Protecting VMs through GitOps

To bring a VM under DR protection using GitOps, add a ProtectionRequest manifest to your Git repository under the appropriate deployment directory and push. Your GitOps tool (ArgoCD or Flux) detects the change and applies it to the quorum cluster. The protection-controller picks up the new CR, validates the VM, provisions a DRBDResource pair on the primary and DR clusters, and transitions the request to Protected.

You can verify protection status at any time:

oc get protectionrequest protect-vm-web-01 \
  --namespace dr-prod-east \
  --kubeconfig ~/.kube/config-quorum \
  -o jsonpath='{.status.phase}'

Expected output when protection is fully established:

Protected

Monitoring replication health in a pipeline

Before allowing a deployment to proceed to production, your pipeline can check that all ProtectionGroup resources are in a Consistent replication state by reading the associated ReplicationGroupStatus:

STATUS=$(oc get replicationgroupstatus prod-east-rgs \
  --namespace dr-prod-east \
  --kubeconfig ~/.kube/config-quorum \
  -o jsonpath='{.status.overallHealth}')

if [[ "$STATUS" != "Healthy" ]]; then
  echo "Replication health is ${STATUS}. Blocking deployment."
  exit 1
fi

Triggering a planned failover via GitOps

For scheduled maintenance failovers managed through Git, commit a FailoverRequest manifest and push. The failover-controller reconciles it immediately. Track progress by watching the CR's status:

oc get failoverrequest prod-east-failover-2025q3 \
  --namespace dr-prod-east \
  --kubeconfig ~/.kube/config-quorum \
  -w

When the failover completes, the CR's status transitions to Completed. After confirming, remove the manifest from Git. Note that removing a FailoverRequest from Git does not reverse the failover—failback requires a separate FailoverRequest targeting the original primary cluster.

Running automated test failovers on a schedule

The recommended pattern for recurring DR validation is a scheduled CI/CD pipeline job (for example, a weekly OpenShift Pipeline or a cron-triggered GitHub Actions workflow) that:

1. Generates a unique TestFailover CR name (using a timestamp)
2. Applies the CR to the quorum cluster
3. Waits for the status.phase to reach Succeeded
4. Reports the result and cleans up

This pattern does not commit the TestFailover manifest to Git—test CRs are ephemeral and should be created directly by the pipeline. Only structural configuration (protection policies, groups, and replication policies) lives in Git.

Using spec.desiredState for maintenance automation

The ProtectionGroup spec.desiredState field lets automation pause all VMs in a group without triggering a failover. This is useful for maintenance windows managed via GitOps:

# deployments/prod-east/protection-group.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: prod-east-pg
  namespace: dr-prod-east
spec:
  desiredState: stopped   # Change to 'running' to resume
  virtualMachines:
    - name: vm-web-01
      namespace: production
    - name: vm-db-01
      namespace: production

Commit desiredState: stopped, push, and the protection-group-controller stops all VMs in the group on the local cluster. Revert to running and push to restart them.

Example 1: ArgoCD Application managing a DR deployment namespace

This manifest tells ArgoCD to continuously synchronize the deployments/prod-east directory from Git into the dr-prod-east namespace on the quorum cluster. Any ProtectionRequest, ProtectionGroup, or DRBDReplicationPolicy manifest you push is applied automatically.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dr-prod-east
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/site-recovery.git
    targetRevision: main
    path: deployments/prod-east
  destination:
    server: https://quorum-cluster.example.com:6443
    namespace: dr-prod-east
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

After applying this, ArgoCD polls Git every few minutes (or on webhook push). When a new ProtectionRequest is merged, ArgoCD applies it and the protection-controller begins provisioning.

Expected ArgoCD status after sync:

Name:               dr-prod-east
Project:            default
Server:             https://quorum-cluster.example.com:6443
Namespace:          dr-prod-east
URL:                https://argocd.example.com/applications/dr-prod-east
Repo:               https://git.example.com/platform/site-recovery.git
Target:             main
Path:               deployments/prod-east
SyncPolicy:         Automated
Sync Status:        Synced
Health Status:      Healthy

Example 2: Flux Kustomization for a DR deployment

Flux's Kustomization resource applies the same directory to the quorum cluster, reconciling every 2 minutes.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dr-prod-east
  namespace: flux-system
spec:
  interval: 2m
  path: ./deployments/prod-east
  prune: true
  sourceRef:
    kind: GitRepository
    name: site-recovery
  targetNamespace: dr-prod-east
  kubeConfig:
    secretRef:
      name: quorum-cluster-kubeconfig

Expected Flux status:

flux get kustomizations dr-prod-east
# NAME          READY  MESSAGE                        REVISION        SUSPENDED
# dr-prod-east  True   Applied revision: main/a1b2c3d  main/a1b2c3d   False

Example 3: Scheduled TestFailover pipeline job (Tekton Task)

This Tekton Task runs a non-disruptive DR test and fails the pipeline if the test does not succeed within the timeout.

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: run-dr-test-failover
  namespace: dr-pipelines
spec:
  params:
    - name: protection-group
      type: string
    - name: dr-namespace
      type: string
  steps:
    - name: apply-test-failover
      image: registry.redhat.io/openshift4/ose-cli:latest
      script: |
        #!/usr/bin/env bash
        set -euo pipefail

        TF_NAME="auto-test-$(date +%Y%m%d%H%M%S)"
        NS="$(params.dr-namespace)"
        PG="$(params.protection-group)"

        oc apply -f - <<EOF
        apiVersion: siterecovery.trilio.io/v1alpha1
        kind: TestFailover
        metadata:
          name: ${TF_NAME}
          namespace: ${NS}
        spec:
          protectionGroupRef:
            name: ${PG}
            namespace: ${NS}
          cleanupPolicy: OnSuccess
          timeout: 30m
          batchBootTimeoutSeconds: 300
          verification:
            enabled: true
        EOF

        echo "Waiting for TestFailover ${TF_NAME} to succeed..."
        oc wait testfailover/${TF_NAME} \
          --namespace=${NS} \
          --for=jsonpath='{.status.phase}'=Succeeded \
          --timeout=35m

        echo "DR test passed: ${TF_NAME}"

Expected output when the test passes:

testfailover.siterecovery.trilio.io/auto-test-20250901120000 created
Waiting for TestFailover auto-test-20250901120000 to succeed...
testfailover.siterecovery.trilio.io/auto-test-20250901120000 condition met
DR test passed: auto-test-20250901120000

Example 4: Replication health gate in a CI pipeline

This shell snippet checks ReplicationGroupStatus before promoting a release. If replication is not Healthy, the pipeline blocks.

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="dr-prod-east"
RGS_NAME="prod-east-rgs"

STATUS=$(oc get replicationgroupstatus "${RGS_NAME}" \
  --namespace="${NAMESPACE}" \
  -o jsonpath='{.status.overallHealth}')

echo "Replication health: ${STATUS}"

if [[ "${STATUS}" != "Healthy" ]]; then
  echo "ERROR: Replication is ${STATUS}. Release blocked until DR health is restored."
  exit 1
fi

echo "Replication healthy. Proceeding with release."

Expected output on a healthy deployment:

Replication health: Healthy
Replication healthy. Proceeding with release.

Expected output when degraded:

Replication health: Degraded
ERROR: Replication is Degraded. Release blocked until DR health is restored.

Example 5: GitOps-driven planned failover

This manifest, when committed to Git and synced by ArgoCD or Flux, triggers a planned failover of the prod-east-pg ProtectionGroup to the DR cluster.

# deployments/prod-east/failover-2025-q3-maintenance.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-2025-q3-maintenance
  namespace: dr-prod-east
spec:
  protectionGroupRef:
    name: prod-east-pg
    namespace: dr-prod-east
  targetCluster: dr
  failoverType: planned
  drainTimeoutSeconds: 600
  batchBootTimeoutSeconds: 300

Monitor the failover:

oc get failoverrequest failover-2025-q3-maintenance \
  --namespace dr-prod-east \
  -w

Expected status progression:

NAME                           STATUS      PHASE
failover-2025-q3-maintenance   Pending     Initializing
failover-2025-q3-maintenance   InProgress  StoppingVMs
failover-2025-q3-maintenance   InProgress  PromotingVolumes
failover-2025-q3-maintenance   InProgress  StartingVMs
failover-2025-q3-maintenance   Completed   Completed

Issue 1: ArgoCD reports OutOfSync but resources appear correct

Symptom: ArgoCD shows the DR namespace as OutOfSync even though oc get commands return the expected resources.

Likely cause: Site Recovery CRDs include status subresources that ArgoCD compares against the Git manifest. ArgoCD may also flag resources managed by the operator (such as auto-created DRBDResource CRs) as unexpected.

Fix:

1. Add the auto-generated resource kinds to ArgoCD's ignoreDifferences list in the Application spec:
   Copy

   spec:
     ignoreDifferences:
       - group: siterecovery.trilio.io
         kind: DRBDResource
         jsonPointers:
           - /status
       - group: siterecovery.trilio.io
         kind: ProtectionRequest
         jsonPointers:
           - /status
   

2. For resources ArgoCD should not manage (operator-owned CRs), add the annotation argocd.argoproj.io/managed-by only to the resources you own.

Issue 2: TestFailover CR stuck in CreatingSnapshots phase

Symptom: A TestFailover CR applied by a pipeline does not progress past CreatingSnapshots and eventually times out.

Likely cause: The test-failover-controller cannot create volume snapshots because no VolumeSnapshotClass is configured on the DR cluster, or the ProtectionGroup's VMs have no associated DRBDResource (protection was never fully established).

Fix:

1. Verify that a VolumeSnapshotClass exists on the DR cluster:
   Copy

   oc get volumesnapshotclass --kubeconfig ~/.kube/config-dr
   

2. Confirm that ProtectionRequest resources for all VMs in the group show phase: Protected:
   Copy

   oc get protectionrequest --namespace dr-prod-east \
     -o jsonpath='{range .items[*]}{.metadata.name}: {.status.phase}\n{end}'
   

3. Check test-failover-controller logs on the quorum cluster for snapshot errors:
   Copy

   oc logs -l app=site-recovery-quorum-control-plane \
     --namespace dr-prod-east \
     --kubeconfig ~/.kube/config-quorum \
     | grep -i "testfailover"
   

Issue 3: FailoverRequest remains in Pending state after GitOps sync

Symptom: A FailoverRequest manifest is synced to the quorum cluster by ArgoCD or Flux, but the CR stays in Pending and the failover-controller does not act on it.

Likely cause: The failover-controller pod is not running, or the protectionGroupRef does not match an existing ProtectionGroup in the same namespace.

Fix:

1. Check that the site-recovery-quorum-control-plane deployment is healthy:
   Copy

   oc get deployment site-recovery-quorum-control-plane \
     --namespace dr-prod-east \
     --kubeconfig ~/.kube/config-quorum
   

2. Verify the ProtectionGroup name and namespace exactly match the protectionGroupRef in the FailoverRequest:
   Copy

   oc get protectiongroup --namespace dr-prod-east \
     --kubeconfig ~/.kube/config-quorum
   

3. Inspect failover-controller logs for reconciliation errors:
   Copy

   oc logs -l app=site-recovery-quorum-control-plane \
     --namespace dr-prod-east \
     --kubeconfig ~/.kube/config-quorum \
     | grep -i "failoverrequest"
   

Issue 4: Pipeline replication health gate fails intermittently

Symptom: The ReplicationGroupStatus check returns Degraded occasionally, blocking releases even when the infrastructure appears healthy.

Likely cause: Transient DRBD resync events (after a brief network interruption or a node restart) briefly set the status to Degraded. The pollingIntervalSeconds default of 30 seconds may catch the status mid-resync.

Fix:

1. Increase the polling interval in the ReplicationGroupStatus spec to reduce noise:
   Copy

   spec:
     pollingIntervalSeconds: 60
   

2. Add a short retry loop to the pipeline gate rather than failing immediately:
   Copy

   for i in $(seq 1 5); do
     STATUS=$(oc get replicationgroupstatus prod-east-rgs \
       --namespace dr-prod-east \
       -o jsonpath='{.status.overallHealth}')
     [[ "$STATUS" == "Healthy" ]] && break
     echo "Attempt $i: status is ${STATUS}, retrying in 30s..."
     sleep 30
   done
   [[ "$STATUS" == "Healthy" ]] || { echo "Replication not healthy after retries."; exit 1; }
   

3. Review RPOEvent resources for recurring violations that indicate a persistent problem:
   Copy

   oc get rpoevent --namespace dr-prod-east \
     --kubeconfig ~/.kube/config-quorum \
     --sort-by='.spec.timestamp'
   

Issue 5: Flux Kustomization fails with resource already exists

Symptom: Flux reports a conflict error when applying DR manifests because a resource (such as a DRBDReplicationPolicy) was previously created manually outside of Git.

Likely cause: The resource was created imperatively and Flux's server-side apply detects a field manager conflict.

Fix:

1. Annotate the existing resource to transfer ownership to Flux:
   Copy

   oc annotate drbdreplicationpolicy prod-replication-policy \
     --namespace dr-prod-east \
     app.kubernetes.io/managed-by=flux \
     --overwrite \
     --kubeconfig ~/.kube/config-quorum
   

2. Alternatively, delete the manually created resource and allow Flux to recreate it from Git. Ensure the manifest in Git matches the intended configuration before deleting, to avoid a protection gap.
3. Enable Flux's force flag in the Kustomization as a last resort (use with caution in production):
   Copy

   spec:
     force: true