Logging

Site Recovery components emit structured logs in newline-delimited JSON format to standard output, giving you machine-readable records that integrate directly with log aggregation stacks such as the Elastic Stack, Loki, or any OpenTelemetry-compatible collector. This page covers the log levels available across Site Recovery controllers and the Site Manager API, the structured fields present on every log line, and recommended configuration for collecting and querying logs in production. Understanding the logging schema helps you correlate events across the quorum, primary, and DR clusters when diagnosing replication issues, failover sequences, or controller reconciliation errors.

Before configuring log aggregation for Site Recovery, ensure you have:

• A running quorum cluster with Site Recovery controllers deployed (failover-controller, protection-controller, pg-sync-controller, protection-group-controller, test-failover-controller, and replication-monitor)
• kubectl configured with access to all clusters (quorum, primary, and DR)
• Access to the namespace for each DR deployment (dr-<name> convention)
• A log aggregation solution capable of ingesting JSON-formatted log streams from Kubernetes pod stdout (for example, Fluentd, Fluent Bit, Promtail, or the OpenTelemetry Collector)
• Familiarity with Kubernetes label selectors to target specific controller pods

Log output is enabled by default for all Site Recovery components — no additional installation steps are required to start receiving logs. However, you should verify that your log aggregation agent is collecting stdout from the relevant namespaces.

Step 1: Confirm controllers are running and emitting logs

# Check controllers on the quorum cluster
kubectl get pods -n dr-<deployment-name> -l app.kubernetes.io/part-of=site-recovery

Step 2: Tail live logs from a specific controller

# failover-controller
kubectl logs -n dr-<deployment-name> -l app=failover-controller -f

# protection-controller
kubectl logs -n dr-<deployment-name> -l app=protection-controller -f

# pg-sync-controller
kubectl logs -n dr-<deployment-name> -l app=pg-sync-controller -f

# replication-monitor (deployed automatically by the Ansible playbooks)
kubectl logs -n dr-<deployment-name> -l app=replication-monitor -f

Step 3: Tail logs from the Site Manager API

The Site Manager API (Python Flask backend) runs on the quorum cluster and emits the same structured JSON format:

kubectl logs -n dr-<deployment-name> -l app=site-manager-api -f

Step 4: Configure your log aggregation agent

Point your log collector at all namespaces matching dr-* on the quorum cluster and any Site Recovery namespaces on the primary and DR clusters. An example Fluent Bit filter to parse the JSON payload:

[FILTER]
    Name       parser
    Match      kube.dr-*.*
    Key_Name   log
    Parser     json
    Reserve_Data On

With this configuration every structured field (timestamp, level, msg, logger, request_id) becomes a queryable attribute in your aggregation backend.

Log Level

All Site Recovery controllers and the Site Manager API respect a configurable log level. The level controls the minimum severity of messages emitted.

Site Manager API (Python Flask backend)

The log level for the Site Manager API is set via the LOG_LEVEL environment variable on the site-manager-api deployment. The value is case-insensitive.

env:
  - name: LOG_LEVEL
    value: "info"

Valid values: debug, info, warning, error (case-insensitive).

Default: info.

JSON vs. console output (Site Manager API)

By default the Site Manager API emits single-line JSON. Set FLASK_ENV=development to switch to a human-readable console renderer — this is intended for local development only and should not be used in production because it produces output that log aggregators cannot parse reliably.

env:
  - name: FLASK_ENV
    value: "production"   # default; set to "development" for console output

Go-based controllers

The failover-controller, protection-controller, pg-sync-controller, protection-group-controller, test-failover-controller, and replication-monitor are Go-based operators. Log verbosity follows the standard controller-runtime / zap convention and is configured via the --zap-log-level flag in the controller's deployment arguments:

args:
  - --zap-log-level=info

Valid values: debug, info, error.

Reading a structured log line

Every log line — regardless of which controller emitted it — is a single-line JSON object. The fields you will see on every line are:

Example log line from the Site Manager API during a failover trigger:

{"timestamp":"2024-11-15T08:32:01.452Z","level":"info","msg":"FailoverRequest accepted","logger":"site_manager.operations","request_id":"a3f2c8d1"}

Example log line from the replication-monitor recording an RPO violation:

{"timestamp":"2024-11-15T08:33:10.001Z","level":"warning","msg":"RPO threshold exceeded","logger":"replication-monitor","protection_group":"pg-web-tier","observed_lag_seconds":42,"threshold_seconds":30}

Filtering by controller

Use the logger field to filter events to a specific controller in your log aggregation tool. For example, in a Loki query:

{namespace=~"dr-.*"} | json | logger="failover-controller"

Correlating a failover sequence

When a FailoverRequest is created, the failover-controller logs each stage (VM shutdown, volume promotion, VM startup) as a sequence of info-level messages. Filter on the protection_group and level fields to reconstruct the timeline:

{namespace="dr-production"} | json | protection_group="pg-web-tier" | level != "debug"

Surfacing RPO violations

The replication-monitor writes an RPOEvent CRD and emits a warning-level log line whenever lag exceeds the configured threshold. You can alert on these lines directly:

{namespace=~"dr-.*"} | json | level="warning" | msg=~"RPO.*exceeded"

Example 1: Inspect live Site Manager API logs during a failover

Trigger a failover and immediately stream the Site Manager API logs to watch request-scoped events:

kubectl logs -n dr-production -l app=site-manager-api -f | jq 'select(.level != "debug")'

Expected output (abbreviated):

{"timestamp":"2024-11-15T09:01:00.123Z","level":"info","msg":"POST /api/v1/operations/failover received","logger":"site_manager.views","request_id":"b7e1a042"}
{"timestamp":"2024-11-15T09:01:00.145Z","level":"info","msg":"FailoverRequest created","logger":"site_manager.operations","request_id":"b7e1a042","protection_group":"pg-web-tier"}

Example 2: Extract all error events from a deployment namespace

kubectl logs -n dr-production --selector app.kubernetes.io/part-of=site-recovery --prefix \
  | grep '"level":"error"' \
  | jq '{ts: .timestamp, ctrl: .logger, msg: .msg}'

Expected output:

{"ts":"2024-11-15T09:05:33.987Z","ctrl":"failover-controller","msg":"VM shutdown timed out after 120s"}

Example 3: Query RPO violation events in Loki

In Grafana connected to a Loki data source:

sum by (protection_group) (
  count_over_time(
    {namespace=~"dr-.*"} | json | msg=~"RPO.*exceeded" [1h]
  )
)

This query returns the count of RPO violations per Protection Group over the last hour, which you can visualize as a bar chart or use as an alerting rule.

Example 4: View replication-monitor logs for a specific Protection Group

kubectl logs -n dr-production -l app=replication-monitor -f \
  | jq 'select(.protection_group == "pg-database")'

Expected output:

{"timestamp":"2024-11-15T09:10:05.001Z","level":"info","msg":"Replication healthy","logger":"replication-monitor","protection_group":"pg-database","sync_state":"Consistent"}
{"timestamp":"2024-11-15T09:11:42.300Z","level":"warning","msg":"RPO threshold exceeded","logger":"replication-monitor","protection_group":"pg-database","observed_lag_seconds":55,"threshold_seconds":30}

Example 5: Retrieve logs across all DR deployment namespaces

for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep '^dr-'); do
  echo "=== $ns ==="
  kubectl logs -n "$ns" --selector app.kubernetes.io/part-of=site-recovery \
    --since=1h 2>/dev/null | grep '"level":"error"'
done

This script iterates every DR deployment namespace and surfaces errors from the last hour, useful for a quick health sweep across a multi-tenant quorum.

Issue: Log lines appear as raw text instead of JSON

Symptom: Your log aggregation pipeline receives unstructured text lines, or jq parsing fails with a parse error.

Likely cause: The FLASK_ENV environment variable on the site-manager-api pod is set to development, which switches the renderer to the human-readable console format.

Fix: Ensure FLASK_ENV is set to production (or unset — the default is production):

kubectl set env deployment/site-manager-api -n dr-<deployment-name> FLASK_ENV=production

Issue: No logs visible from a controller pod

Symptom: kubectl logs returns an empty stream or the pod is not found.

Likely cause: The controller pod may not be running, or you are querying the wrong namespace. Each DR deployment runs in its own dr-<name> namespace.

Fix:

1. List all Site Recovery namespaces:
   Copy

   kubectl get namespaces | grep '^dr-'
   

2. Check pod status in the correct namespace:
   Copy

   kubectl get pods -n dr-<deployment-name>
   

3. Describe any pod in a non-Running state:
   Copy

   kubectl describe pod <pod-name> -n dr-<deployment-name>
   

Issue: stacktrace field is missing on error log lines

Symptom: Error-level log lines from the Site Manager API do not include a stacktrace field, making it difficult to diagnose the root cause.

Likely cause: The error was logged without an active exception context. The stacktrace field is only present when an exception is caught and re-raised through the structlog pipeline — ordinary log.error("message") calls without an active exception will not include it.

Fix: This is expected behavior. To capture a traceback, the calling code must log from within an except block. If you are extending the Site Manager API and need stack information, ensure your handler catches the exception before logging and passes exc_info=True.

Issue: request_id field is absent from Site Manager API log lines

Symptom: Log lines from the Site Manager API do not include a request_id field, making it impossible to correlate all entries for a single HTTP request.

Likely cause: The log line was emitted outside a request context — for example, during application startup, background tasks, or initialization code — where no HTTP request is active.

Fix: This is expected for startup and background log entries. Request-scoped events (any code path triggered by an incoming HTTP request) will always carry request_id. Filter your aggregation query to lines that include the field:

{namespace=~"dr-.*"} | json | request_id != ""

Issue: Log volume is too high in production

Symptom: Log aggregation costs or storage consumption are unexpectedly high.

Likely cause: One or more controllers are running at debug level, which emits detailed reconciliation steps and internal state for every controller loop iteration.

Fix: Set all controllers to info level. For the Site Manager API:

kubectl set env deployment/site-manager-api -n dr-<deployment-name> LOG_LEVEL=info

For Go-based controllers, update the deployment arguments to --zap-log-level=info and roll the deployment:

kubectl rollout restart deployment/failover-controller -n dr-<deployment-name>
kubectl rollout restart deployment/protection-controller -n dr-<deployment-name>

Issue: Third-party library logs (werkzeug, urllib3, gunicorn) are not in JSON format

Symptom: Most log lines are valid JSON but occasional lines from the HTTP server or Kubernetes client library appear as plain text.

Likely cause: A logging configuration change has reset the stdlib root logger or replaced its handler, bypassing the structlog ProcessorFormatter bridge.

Fix: The Site Manager API's configure_logging() function clears all root logger handlers and replaces them with the structlog-based handler. If you have added custom logging configuration after application startup (for example, via a third-party extension that calls logging.basicConfig()), remove it. The stdlib bridge is intentional and must remain the sole handler on the root logger.