Observability

This guide explains how to monitor and observe your tlsproxy instance using its built-in logging, metrics, tracing, and health check capabilities. Understanding what your proxy is doing at runtime is essential for diagnosing connection failures, measuring throughput, and ensuring your TLS termination layer is healthy. The sections below walk you through enabling and querying each observability feature via tlsproxy's HTTP API. Whether you are debugging a single connection or operating tlsproxy at scale, these tools give you the visibility you need.

Before working through this guide, make sure you have the following in place:

• tlsproxy installed and running (see the Install and Configure guides)
• Python 3.8+ and Twisted installed in your environment
• A running tlsproxy instance with its management HTTP API accessible (default: http://127.0.0.1:8080)
• curl or a similar HTTP client for querying the API endpoints
• Sufficient permissions to read log files and access the management port on the host
• (Optional) A Prometheus-compatible scraper if you intend to collect metrics
• (Optional) An OpenTelemetry-compatible collector if you intend to use distributed tracing

The observability endpoints are built into tlsproxy and require no separate installation. However, you may need to install optional dependencies depending on which features you enable.

Step 1 — Verify the management API is reachable

curl -s http://127.0.0.1:8080/healthz

You should receive an HTTP 200 OK response. If the connection is refused, confirm that tlsproxy is running and that the management port is correctly configured (see Configuration below).

Step 2 — Install Prometheus client library (optional, for metrics)

If you want to expose Prometheus-format metrics, install the Python client:

pip install prometheus_client

Step 3 — Install OpenTelemetry SDK (optional, for tracing)

If you want distributed tracing, install the OpenTelemetry packages:

pip install opentelemetry-sdk opentelemetry-exporter-otlp

Step 4 — Restart tlsproxy to apply any newly installed libraries

# If running via a process manager such as systemd:
sudo systemctl restart tlsproxy

# Or, if running directly:
kill -HUP $(pgrep -f tlsproxy)

Observability behaviour is controlled by settings in your tlsproxy configuration file (typically tlsproxy.yaml or passed as environment variables). The table below describes the relevant options.

Management API

Logging

Metrics

Tracing

Once tlsproxy is running with observability enabled, you interact with it primarily through HTTP GET requests to the management API. No authentication is required by default, so restrict network access to the management port using firewall rules or a reverse proxy.

Checking proxy health

Send a GET request to /healthz to confirm the proxy is alive and accepting connections:

curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8080/healthz

A 200 response means the proxy is healthy. Automate this check in your load balancer or container orchestrator to route traffic away from unhealthy instances.

Querying live connection status

Send a GET request to /status to retrieve a snapshot of active proxied connections:

curl -s http://127.0.0.1:8080/status | python3 -m json.tool

This is useful when you want to understand current load or verify that a specific backend is receiving traffic.

Scraping metrics

If metrics.enabled is true, point your Prometheus scrape configuration at the /metrics endpoint:

# prometheus.yml excerpt
scrape_configs:
  - job_name: tlsproxy
    static_configs:
      - targets: ['127.0.0.1:8080']
    metrics_path: /metrics
    scrape_interval: 15s

You can also query metrics manually during development:

curl -s http://127.0.0.1:8080/metrics

Tailing logs

If logging.destination is set to stdout, capture logs with your process manager or pipe them to a stream processor:

journalctl -u tlsproxy -f

If you configured JSON log format, pipe through jq for readable output:

journalctl -u tlsproxy -f | jq .

The following examples are self-contained and assume tlsproxy is running on 127.0.0.1:8080.

Example 1 — Health check returning 200 OK

curl -i http://127.0.0.1:8080/healthz

Expected output:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 15

{"status":"ok"}

Example 2 — Retrieving proxy status as formatted JSON

curl -s http://127.0.0.1:8080/status | python3 -m json.tool

Expected output:

{
    "active_connections": 4,
    "backends": [
        {
            "address": "127.0.0.1:3000",
            "connections": 4,
            "healthy": true
        }
    ],
    "uptime_seconds": 3721
}

Example 3 — Scraping Prometheus metrics manually

With metrics.enabled: true:

curl -s http://127.0.0.1:8080/metrics | grep tlsproxy

Expected output (excerpt):

# HELP tlsproxy_connections_total Total number of proxied connections accepted
# TYPE tlsproxy_connections_total counter
tlsproxy_connections_total 1024.0
# HELP tlsproxy_active_connections Current number of open proxied connections
# TYPE tlsproxy_active_connections gauge
tlsproxy_active_connections 4.0
# HELP tlsproxy_bytes_transferred_total Total bytes forwarded through the proxy
# TYPE tlsproxy_bytes_transferred_total counter
tlsproxy_bytes_transferred_total{direction="upstream"} 5.24288e+06
tlsproxy_bytes_transferred_total{direction="downstream"} 3.14572e+06

Example 4 — Emitting traces to the console for local debugging

Set tracing.enabled: true and tracing.exporter: console, then send a request through the proxy:

curl -sk https://127.0.0.1:8443/

Expected output in the tlsproxy process log:

{
    "name": "proxy.connection",
    "context": {"trace_id": "0x4bf92f3577b34da6", "span_id": "0x00f067aa0ba902b7"},
    "start_time": "2024-06-01T12:00:00.000000Z",
    "end_time": "2024-06-01T12:00:00.012000Z",
    "attributes": {
        "net.peer.ip": "192.168.1.10",
        "net.peer.port": 54321,
        "backend.address": "127.0.0.1:3000"
    },
    "status": {"status_code": "OK"}
}

Use the patterns below to diagnose the most common observability problems.

Issue: /healthz returns Connection refused

• Symptom: curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused
• Likely cause: The management API is disabled (management.enabled: false) or tlsproxy has crashed.
• Fix:
  1. Confirm tlsproxy is running: pgrep -a tlsproxy
  2. Check that management.enabled is true in your configuration.
  3. Verify the port in your config matches the port you are querying.
  4. Inspect tlsproxy logs for startup errors: journalctl -u tlsproxy --since "5 minutes ago"

Issue: /metrics returns 404 Not Found

• Symptom: The endpoint exists but responds with a 404 or an HTML error page.
• Likely cause: metrics.enabled is false (the default), or metrics.path has been changed from /metrics.
• Fix:
  1. Set metrics.enabled: true in your configuration and restart tlsproxy.
  2. Confirm prometheus_client is installed in the same Python environment as tlsproxy: pip show prometheus_client
  3. Check the metrics.path setting — if it was changed, query the custom path instead.

Issue: Logs are emitted but contain no DEBUG-level detail

• Symptom: Log output is sparse; connection-level events are missing.
• Likely cause: logging.level is set to INFO or higher, suppressing DEBUG messages.
• Fix: Set logging.level: DEBUG in your configuration and restart tlsproxy. Revert to INFO or WARNING once debugging is complete to avoid excessive I/O.

Issue: No traces appear in the OpenTelemetry collector

• Symptom: tlsproxy starts without errors, but the collector receives no spans.
• Likely cause: The tracing.endpoint is incorrect, the collector is not reachable from the tlsproxy host, or opentelemetry-sdk is not installed.
• Fix:
  1. Install the SDK: pip install opentelemetry-sdk opentelemetry-exporter-otlp
  2. Verify the collector is reachable: curl -s http://<collector-host>:4318/ (HTTP) or grpc_health_probe -addr=<collector-host>:4317 (gRPC).
  3. Temporarily set tracing.exporter: console and restart tlsproxy. If spans appear in the process log, the SDK is working and the problem is network or collector configuration.
  4. Check for TLS requirements on your collector — if it requires mTLS, you will need to configure client certificates in the tracing exporter settings.

Issue: JSON log lines are malformed or split across multiple lines

• Symptom: Your log aggregator cannot parse JSON; lines appear truncated or concatenated.
• Likely cause: A logging buffer flush issue or a log rotation tool is splitting long lines.
• Fix:
  1. Ensure logging.format: json is set and tlsproxy has been restarted.
  2. Disable line-buffering workarounds in your process manager that may be inserting newlines mid-record.
  3. Increase the log aggregator's maximum line length to accommodate large JSON objects (especially those containing TLS handshake details).