--- title: Logging product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/logging --- # Logging _Log levels, structured logging fields, and recommended log aggregation configuration_ ## Overview Site Recovery components emit structured logs in newline-delimited JSON format to standard output, giving you machine-readable records that integrate directly with log aggregation stacks such as the Elastic Stack, Loki, or any OpenTelemetry-compatible collector. This page covers the log levels available across Site Recovery controllers and the Site Manager API, the structured fields present on every log line, and recommended configuration for collecting and querying logs in production. Understanding the logging schema helps you correlate events across the quorum, primary, and DR clusters when diagnosing replication issues, failover sequences, or controller reconciliation errors. ## Prerequisites Before configuring log aggregation for Site Recovery, ensure you have: - A running quorum cluster with Site Recovery controllers deployed (failover-controller, protection-controller, pg-sync-controller, protection-group-controller, test-failover-controller, and replication-monitor) - `kubectl` configured with access to all clusters (quorum, primary, and DR) - Access to the namespace for each DR deployment (`dr-` convention) - A log aggregation solution capable of ingesting JSON-formatted log streams from Kubernetes pod stdout (for example, Fluentd, Fluent Bit, Promtail, or the OpenTelemetry Collector) - Familiarity with Kubernetes label selectors to target specific controller pods ## Installation Log output is enabled by default for all Site Recovery components — no additional installation steps are required to start receiving logs. However, you should verify that your log aggregation agent is collecting stdout from the relevant namespaces. **Step 1: Confirm controllers are running and emitting logs** ```bash # Check controllers on the quorum cluster kubectl get pods -n dr- -l app.kubernetes.io/part-of=site-recovery ``` **Step 2: Tail live logs from a specific controller** ```bash # failover-controller kubectl logs -n dr- -l app=failover-controller -f # protection-controller kubectl logs -n dr- -l app=protection-controller -f # pg-sync-controller kubectl logs -n dr- -l app=pg-sync-controller -f # replication-monitor (deployed automatically by the Ansible playbooks) kubectl logs -n dr- -l app=replication-monitor -f ``` **Step 3: Tail logs from the Site Manager API** The Site Manager API (Python Flask backend) runs on the quorum cluster and emits the same structured JSON format: ```bash kubectl logs -n dr- -l app=site-manager-api -f ``` **Step 4: Configure your log aggregation agent** Point your log collector at all namespaces matching `dr-*` on the quorum cluster and any Site Recovery namespaces on the primary and DR clusters. An example Fluent Bit filter to parse the JSON payload: ```ini [FILTER] Name parser Match kube.dr-*.* Key_Name log Parser json Reserve_Data On ``` With this configuration every structured field (`timestamp`, `level`, `msg`, `logger`, `request_id`) becomes a queryable attribute in your aggregation backend. ## Configuration ### Log Level All Site Recovery controllers and the Site Manager API respect a configurable log level. The level controls the minimum severity of messages emitted. | Level | When to use | |-------|-------------| | `debug` | Detailed reconciliation steps, CRD field values, and internal state transitions. Use only in non-production or when actively diagnosing an issue — output volume is high. | | `info` | Normal operational events: protection requests accepted, failover stages advancing, sync completion. **Default for production.** | | `warning` | Recoverable conditions: retryable API errors, RPO threshold approached, controller backoff. | | `error` | Non-recoverable failures within a reconciliation loop: volume promotion failed, VM shutdown timed out. | #### Site Manager API (Python Flask backend) The log level for the Site Manager API is set via the `LOG_LEVEL` environment variable on the `site-manager-api` deployment. The value is case-insensitive. ```yaml env: - name: LOG_LEVEL value: "info" ``` Valid values: `debug`, `info`, `warning`, `error` (case-insensitive). Default: `info`. #### JSON vs. console output (Site Manager API) By default the Site Manager API emits single-line JSON. Set `FLASK_ENV=development` to switch to a human-readable console renderer — this is intended for local development only and should not be used in production because it produces output that log aggregators cannot parse reliably. ```yaml env: - name: FLASK_ENV value: "production" # default; set to "development" for console output ``` #### Go-based controllers The failover-controller, protection-controller, pg-sync-controller, protection-group-controller, test-failover-controller, and replication-monitor are Go-based operators. Log verbosity follows the standard controller-runtime / zap convention and is configured via the `--zap-log-level` flag in the controller's deployment arguments: ```yaml args: - --zap-log-level=info ``` Valid values: `debug`, `info`, `error`. > **Note for reviewer:** The source material provides full implementation detail only for the Python Site Manager API logging (structlog/Flask). The exact flag names and environment variable names for the Go controller log level were not present in the provided source — the `--zap-log-level` flag above follows standard controller-runtime conventions but should be verified against the actual controller deployment manifests. ## Usage ### Reading a structured log line Every log line — regardless of which controller emitted it — is a single-line JSON object. The fields you will see on every line are: | Field | Type | Description | |-------|------|-------------| | `timestamp` | string (RFC3339 UTC) | Time the event was recorded, for example `2024-11-15T08:32:01.452Z` | | `level` | string | Lowercase severity: `debug`, `info`, `warning`, or `error` | | `msg` | string | Human-readable description of the event | | `logger` | string | Name of the logger or controller module that emitted the line | | `request_id` | string | Present on request-scoped lines in the Site Manager API; correlates all log entries for a single HTTP request | | `stacktrace` | string | Present only on exception events; contains the full traceback | Example log line from the Site Manager API during a failover trigger: ```json {"timestamp":"2024-11-15T08:32:01.452Z","level":"info","msg":"FailoverRequest accepted","logger":"site_manager.operations","request_id":"a3f2c8d1"} ``` Example log line from the replication-monitor recording an RPO violation: ```json {"timestamp":"2024-11-15T08:33:10.001Z","level":"warning","msg":"RPO threshold exceeded","logger":"replication-monitor","protection_group":"pg-web-tier","observed_lag_seconds":42,"threshold_seconds":30} ``` ### Filtering by controller Use the `logger` field to filter events to a specific controller in your log aggregation tool. For example, in a Loki query: ```logql {namespace=~"dr-.*"} | json | logger="failover-controller" ``` ### Correlating a failover sequence When a FailoverRequest is created, the failover-controller logs each stage (VM shutdown, volume promotion, VM startup) as a sequence of `info`-level messages. Filter on the `protection_group` and `level` fields to reconstruct the timeline: ```logql {namespace="dr-production"} | json | protection_group="pg-web-tier" | level != "debug" ``` ### Surfacing RPO violations The replication-monitor writes an RPOEvent CRD and emits a `warning`-level log line whenever lag exceeds the configured threshold. You can alert on these lines directly: ```logql {namespace=~"dr-.*"} | json | level="warning" | msg=~"RPO.*exceeded" ``` ## Examples ### Example 1: Inspect live Site Manager API logs during a failover Trigger a failover and immediately stream the Site Manager API logs to watch request-scoped events: ```bash kubectl logs -n dr-production -l app=site-manager-api -f | jq 'select(.level != "debug")' ``` Expected output (abbreviated): ```json {"timestamp":"2024-11-15T09:01:00.123Z","level":"info","msg":"POST /api/v1/operations/failover received","logger":"site_manager.views","request_id":"b7e1a042"} {"timestamp":"2024-11-15T09:01:00.145Z","level":"info","msg":"FailoverRequest created","logger":"site_manager.operations","request_id":"b7e1a042","protection_group":"pg-web-tier"} ``` --- ### Example 2: Extract all error events from a deployment namespace ```bash kubectl logs -n dr-production --selector app.kubernetes.io/part-of=site-recovery --prefix \ | grep '"level":"error"' \ | jq '{ts: .timestamp, ctrl: .logger, msg: .msg}' ``` Expected output: ```json {"ts":"2024-11-15T09:05:33.987Z","ctrl":"failover-controller","msg":"VM shutdown timed out after 120s"} ``` --- ### Example 3: Query RPO violation events in Loki In Grafana connected to a Loki data source: ```logql sum by (protection_group) ( count_over_time( {namespace=~"dr-.*"} | json | msg=~"RPO.*exceeded" [1h] ) ) ``` This query returns the count of RPO violations per Protection Group over the last hour, which you can visualize as a bar chart or use as an alerting rule. --- ### Example 4: View replication-monitor logs for a specific Protection Group ```bash kubectl logs -n dr-production -l app=replication-monitor -f \ | jq 'select(.protection_group == "pg-database")' ``` Expected output: ```json {"timestamp":"2024-11-15T09:10:05.001Z","level":"info","msg":"Replication healthy","logger":"replication-monitor","protection_group":"pg-database","sync_state":"Consistent"} {"timestamp":"2024-11-15T09:11:42.300Z","level":"warning","msg":"RPO threshold exceeded","logger":"replication-monitor","protection_group":"pg-database","observed_lag_seconds":55,"threshold_seconds":30} ``` --- ### Example 5: Retrieve logs across all DR deployment namespaces ```bash for ns in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep '^dr-'); do echo "=== $ns ===" kubectl logs -n "$ns" --selector app.kubernetes.io/part-of=site-recovery \ --since=1h 2>/dev/null | grep '"level":"error"' done ``` This script iterates every DR deployment namespace and surfaces errors from the last hour, useful for a quick health sweep across a multi-tenant quorum. ## Troubleshooting ### Issue: Log lines appear as raw text instead of JSON **Symptom:** Your log aggregation pipeline receives unstructured text lines, or `jq` parsing fails with a parse error. **Likely cause:** The `FLASK_ENV` environment variable on the `site-manager-api` pod is set to `development`, which switches the renderer to the human-readable console format. **Fix:** Ensure `FLASK_ENV` is set to `production` (or unset — the default is `production`): ```bash kubectl set env deployment/site-manager-api -n dr- FLASK_ENV=production ``` --- ### Issue: No logs visible from a controller pod **Symptom:** `kubectl logs` returns an empty stream or the pod is not found. **Likely cause:** The controller pod may not be running, or you are querying the wrong namespace. Each DR deployment runs in its own `dr-` namespace. **Fix:** 1. List all Site Recovery namespaces: ```bash kubectl get namespaces | grep '^dr-' ``` 2. Check pod status in the correct namespace: ```bash kubectl get pods -n dr- ``` 3. Describe any pod in a non-Running state: ```bash kubectl describe pod -n dr- ``` --- ### Issue: `stacktrace` field is missing on error log lines **Symptom:** Error-level log lines from the Site Manager API do not include a `stacktrace` field, making it difficult to diagnose the root cause. **Likely cause:** The error was logged without an active exception context. The `stacktrace` field is only present when an exception is caught and re-raised through the structlog pipeline — ordinary `log.error("message")` calls without an active exception will not include it. **Fix:** This is expected behavior. To capture a traceback, the calling code must log from within an `except` block. If you are extending the Site Manager API and need stack information, ensure your handler catches the exception before logging and passes `exc_info=True`. --- ### Issue: `request_id` field is absent from Site Manager API log lines **Symptom:** Log lines from the Site Manager API do not include a `request_id` field, making it impossible to correlate all entries for a single HTTP request. **Likely cause:** The log line was emitted outside a request context — for example, during application startup, background tasks, or initialization code — where no HTTP request is active. **Fix:** This is expected for startup and background log entries. Request-scoped events (any code path triggered by an incoming HTTP request) will always carry `request_id`. Filter your aggregation query to lines that include the field: ```logql {namespace=~"dr-.*"} | json | request_id != "" ``` --- ### Issue: Log volume is too high in production **Symptom:** Log aggregation costs or storage consumption are unexpectedly high. **Likely cause:** One or more controllers are running at `debug` level, which emits detailed reconciliation steps and internal state for every controller loop iteration. **Fix:** Set all controllers to `info` level. For the Site Manager API: ```bash kubectl set env deployment/site-manager-api -n dr- LOG_LEVEL=info ``` For Go-based controllers, update the deployment arguments to `--zap-log-level=info` and roll the deployment: ```bash kubectl rollout restart deployment/failover-controller -n dr- kubectl rollout restart deployment/protection-controller -n dr- ``` --- ### Issue: Third-party library logs (werkzeug, urllib3, gunicorn) are not in JSON format **Symptom:** Most log lines are valid JSON but occasional lines from the HTTP server or Kubernetes client library appear as plain text. **Likely cause:** A logging configuration change has reset the stdlib root logger or replaced its handler, bypassing the structlog ProcessorFormatter bridge. **Fix:** The Site Manager API's `configure_logging()` function clears all root logger handlers and replaces them with the structlog-based handler. If you have added custom logging configuration after application startup (for example, via a third-party extension that calls `logging.basicConfig()`), remove it. The stdlib bridge is intentional and must remain the sole handler on the root logger.