---
title: Development Guide
product: trilio-site-recovery-for-kubernetes-openshift-virtualization
doc_type: guide
version: feature-tsr-24
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/development-guide
---

# Development Guide

_How to build, test, and contribute to the site-recovery operator including local development with envtest_

## Overview

This guide explains how to build, test, and contribute to the site-recovery operator. It covers the repository layout, the Make-based build system, how to run controllers locally against a real Kubernetes API server using envtest, and the conventions you must follow when adding new APIs, controllers, or webhooks. Whether you are fixing a bug, adding a new CRD field, or onboarding as a first-time contributor, this page gives you the full development loop from source change to verified test run.

## Prerequisites

Before you begin local development, ensure you have the following installed and available on your `PATH`:

- **Go 1.26+** — the module is `github.com/trilioData/site-recovery` and requires Go 1.26
- **Docker** (or a compatible `CONTAINER_TOOL`) — used by `make docker-build` and cross-platform buildx targets
- **kubectl** — for inspecting cluster state during local runs
- **Helm ≥ 3.0** — the only supported deployment path for the operator
- **kubebuilder CLI** — required to scaffold new APIs, controllers, and webhooks; do not create scaffolded files manually
- **controller-gen v0.19.0** — installed automatically by the Makefile into `bin/`; used by `make manifests` and `make generate`
- **golangci-lint** — installed automatically into `bin/golangci-lint-custom` by the Makefile
- **setup-envtest** — installed automatically by the Makefile; provides the `kube-apiserver` and `etcd` binaries used by `make test`
- **Git** — version metadata (`GIT_COMMIT`, `BUILD_DATE`) is injected at build time via `-ldflags`
- A local kubeconfig context if you want to run controllers against a live cluster with `make run`

> **Note:** E2E tests require a dedicated [Kind](https://kind.sigs.k8s.io/) cluster. Never run them against a development or production cluster.

## Installation

### 1. Clone the repository

```bash
git clone https://github.com/trilioData/site-recovery.git
cd site-recovery
```

### 2. Install tool dependencies

The Makefile installs all Go toolchain dependencies (controller-gen, envtest, golangci-lint) into `bin/` on first use. No manual installation of these tools is required.

```bash
make help   # verify the Makefile is accessible and list all targets
```

### 3. Generate CRDs and DeepCopy code

After cloning, or any time you edit a `api/v1alpha1/*_types.go` file, regenerate the manifests and DeepCopy implementations:

```bash
make manifests generate
```

- `make manifests` runs controller-gen to produce CRD YAMLs into `deploy/crds/` (the single canonical location) and then calls `make sync-crds` to copy the per-chart subsets into each Helm chart's `crds/` directory.
- `make generate` regenerates the `zz_generated.deepcopy.go` files.

> **Do not edit** `deploy/crds/*.yaml`, `helm/*/crds/*.yaml`, `**/zz_generated.*.go`, or the `PROJECT` file. These are all auto-generated.

### 4. Build the control-plane binaries

```bash
make build
```

This produces two binaries in `bin/`:
- `bin/quorum-manager` — built from `cmd/quorum-control-plane/main.go`
- `bin/workload-manager` — built from `cmd/workload-control-plane/main.go`

Both binaries are shipped inside the single control-plane container image. Each Helm chart selects its own binary via the container command.

### 5. Build container images

Build the control-plane image (controllers and webhooks):

```bash
export IMG=<registry>/site-recovery-control-plane:<tag>
make docker-build IMG=$IMG
make docker-push IMG=$IMG
```

Additional image targets:

```bash
# DRBD node agent
make docker-build-node-agent NODE_AGENT_IMG=<registry>/site-recovery-drbd-node-agent:<tag>

# UI frontend (Next.js)
make docker-build-frontend FRONTEND_IMG=<registry>/site-recovery-frontend:<tag>

# UI backend (Flask)
make docker-build-backend BACKEND_IMG=<registry>/site-recovery-backend:<tag>

# must-gather diagnostics collector
make docker-build-must-gather MUST_GATHER_IMG=<registry>/site-recovery-must-gather:<tag>
```

The UI frontend and backend images build from the `ui/` directory. The Go image build contexts live under `docker-images/`.

### 6. Deploy via Helm

Helm is the only supported deployment path. The `make deploy` target runs `helm upgrade --install` using the chart selected by `HELM_CHART`:

```bash
export IMG=<registry>/site-recovery-control-plane:<tag>
make deploy IMG=$IMG
```

To load a locally built image into a Kind cluster instead of pushing to a registry:

```bash
kind load docker-image $IMG --name <cluster-name>
```

### 7. Run locally against your kubeconfig context

To run the quorum control plane process on your host against the cluster pointed to by `~/.kube/config`:

```bash
make run
```

To run the workload control plane (admission webhooks, no reconcilers):

```bash
make run-workload
```

## Configuration

### Build-time variables

The Makefile resolves the following variables automatically. Override them on the command line when needed.

| Variable | Default | Description |
|---|---|---|
| `IMG` | `$(DOCKER_REGISTRY)/site-recovery-control-plane:$(VERSION)` | Control-plane image reference used by `docker-build`, `docker-push`, and `deploy` |
| `DOCKER_REGISTRY` | Branch-dependent (see below) | Container image registry |
| `VERSION` | Resolved from git tag, Prow env, or branch name | Image tag applied to all built images |
| `GIT_COMMIT` | `git rev-parse --short HEAD` | Injected into binaries via `-ldflags -X` |
| `BUILD_DATE` | Current UTC timestamp | Injected into binaries via `-ldflags -X` |
| `CONTAINER_TOOL` | `docker` | Set to `podman` or another compatible tool if needed |
| `TEST_PKG_PARALLELISM` | `4` | Maximum number of envtest packages running concurrently; reduce if you see control-plane start timeouts |
| `KUBEBUILDER_CONTROLPLANE_START_TIMEOUT` | `2m` | Time allowed for each envtest kube-apiserver to start |
| `KUBEBUILDER_CONTROLPLANE_STOP_TIMEOUT` | `2m` | Time allowed for each envtest kube-apiserver to stop |
| `ENVTEST_K8S_VERSION` | Set in Makefile | Kubernetes API version used by setup-envtest |

**Registry selection logic:**
- Presubmit CI builds always publish to `eu.gcr.io/amazing-chalice-243510`.
- `release*` branches publish to `quay.io/triliodata`.
- `tech-preview*` branches publish to `quay.io/triliovault`.
- All other branches (including local dev) publish to `eu.gcr.io/amazing-chalice-243510`.

### Helm chart values

The three Helm charts under `helm/` accept values that control logging and image references. The most commonly tuned values are:

| Value path | Default | Description |
|---|---|---|
| `logging.level` | `info` | Log verbosity: `debug`, `info`, or `error`, or an integer V-level |
| `logging.encoder` | `json` | Log format: `json` (production) or `console` (human-readable dev mode) |
| `agent.logging.level` | `info` | Log verbosity for the DRBD node agent |
| `agent.logging.encoder` | `json` | Log format for the DRBD node agent |

> **Important:** Never hardcode `Development: true` in any Go logging initialization. Production logging is always JSON-encoded at the `info` level unless overridden by Helm values or a `--zap-log-level` flag at startup. The log level can be changed at runtime without restarting the process.

### CRD API group

All CRDs use the API group `siterecovery.trilio.io/v1alpha1`. The single-group layout places all types in `api/v1alpha1/`. Do not move files or change this group without a kubebuilder multi-group migration (see the project layout section below).

### Kubebuilder markers

Key markers for `api/v1alpha1/*_types.go`:

```go
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespaced
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=".status.conditions[?(@.type=='Ready')].status"

// On fields:
// +kubebuilder:validation:Required
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:MaxLength=100
// +kubebuilder:validation:Pattern="^[a-z]+$"
// +kubebuilder:default="value"
```

Always use `metav1.Condition` for status conditions, `metav1.Time` for timestamps, and standard Kubernetes field names (`spec`, `status`, `metadata`).

### Controller RBAC markers

Place RBAC markers in `pkg/controllers/*_controller.go`. They are processed by `make manifests` into the Helm chart templates — do not hand-edit the generated RBAC manifests.

```go
// +kubebuilder:rbac:groups=siterecovery.trilio.io,resources=protectiongroups,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=siterecovery.trilio.io,resources=protectiongroups/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=siterecovery.trilio.io,resources=protectiongroups/finalizers,verbs=update
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
```

## Usage

### Typical development loop

The standard flow for making a code change is:

1. **Edit** `api/v1alpha1/*_types.go` or controller code in `pkg/controllers/`.
2. **Regenerate** if you changed types:
   ```bash
   make manifests generate
   ```
3. **Format and vet:**
   ```bash
   make fmt
   make vet
   ```
4. **Run tests:**
   ```bash
   make test-unit
   ```
5. **Auto-fix lint issues:**
   ```bash
   make lint-fix
   ```
6. **Build and verify locally:**
   ```bash
   make build
   ```

### Scaffolding new APIs

Always use the kubebuilder CLI to scaffold new types. Never create scaffolded files manually.

```bash
kubebuilder create api --group siterecovery --version v1alpha1 --kind MyKind
```

After scaffolding, run:

```bash
make manifests generate
```

### Scaffolding webhooks

```bash
kubebuilder create webhook --group siterecovery --version v1alpha1 --kind MyKind \
  --defaulting --programmatic-validation
```

> **Note:** Always create validation and defaulting webhooks together. If you need to re-scaffold, back up any custom logic in `pkg/webhook/` first, re-run with `--force`, then restore your customizations.

### Running tests selectively

```bash
# All envtest unit tests (CI merge gate)
make test-unit

# Full suite including codegen and format checks
make test

# Controller tests only
make test-controllers

# Webhook tests only
make test-webhooks
```

Tests use **Ginkgo + Gomega** in BDD style. Each test package boots its own `kube-apiserver` and `etcd` via envtest. Check `suite_test.go` in each package for setup details. Parallelism is capped at `TEST_PKG_PARALLELISM=4` by default to prevent CPU starvation on CI pods; increase it on well-resourced workstations.

### Running a controller locally

To run the quorum control plane directly on your host against the cluster in your current kubeconfig context:

```bash
make run
```

The process registers all controllers and webhooks defined in `cmd/quorum-control-plane/main.go`. It uses structured JSON logging at `info` level by default. Pass `--zap-log-level=debug` to increase verbosity:

```bash
go run ./cmd/quorum-control-plane/main.go --zap-log-level=debug
```

### Writing controller reconcilers

Follow these rules in all reconciler implementations:

- **Idempotent reconciliation:** Every reconcile call must be safe to run multiple times with the same outcome.
- **Re-fetch before updates:** Always call `r.Get(ctx, req.NamespacedName, obj)` immediately before `r.Update` or `r.Patch` to avoid stale-object conflicts.
- **Structured logging:** Acquire the logger from context — `log := logf.FromContext(ctx)` — never use a package-level logger. This inherits `controller`, `namespace`, `name`, and `reconcileID` automatically.
- **Owner references:** Use `SetControllerReference` to enable automatic garbage collection of secondary resources.
- **Watch secondary resources:** Use `.Owns()` or `.Watches()` on the controller builder rather than polling with `RequeueAfter`.
- **Finalizers:** Register a finalizer for any controller that creates external resources (VMs, volumes, DNS entries) that must be cleaned up on deletion.

### Logging conventions

All logging uses `logr` backed by controller-runtime's zap integration. Use the named verbosity constants from `internal/logging`:

```go
log := logf.FromContext(ctx)

// State changes and decisions (always visible)
log.Info("Starting reconciliation")
log.Info("Created ProtectionGroup", "protectionGroup", pg.Name)

// Actionable failures (always pass the real error object)
log.Error(err, "Failed to create DRBDVolume", "pvc", pvcName)

// Per-phase debug detail (diagnoses stuck failovers)
log.V(logging.Debug).Info("Waiting for volume sync", "drbdResource", res.Name, "phase", phase)

// High-frequency trace
log.V(logging.Trace).Info("Polling replication state", "cluster", clusterName)
```

Use the canonical key vocabulary — `protectionGroup`, `vm`, `cluster`, `phase`, `error`, `node`, `pvc`, `pv`, `drbdResource`, `result` — consistently. Never log whole CR objects, Secret data, kubeconfigs, or tokens.

### UI development

The UI lives in the `ui/` directory. It is a split stack with a Next.js frontend and a Flask backend.

```bash
cd ui

# Frontend
npm install
npm run dev          # Dev server at localhost:3000
npm run build        # Production build
npm run lint         # ESLint
npm run type-check   # TypeScript check

# Backend
pip install -r requirements.txt
python app.py        # Flask API server
```

The UI is not Helm-packaged; it ships in the OLM bundle under `deploy/olm-catalog/`.

## Examples

### Example 1: Full code-change loop — add a field to ProtectionGroup

This example adds a new optional field to the `ProtectionGroup` spec and regenerates all derived artifacts.

```bash
# 1. Edit the type
$EDITOR api/v1alpha1/protectiongroup_types.go

# 2. Regenerate CRDs and DeepCopy
make manifests generate

# 3. Verify CRDs are assigned to their charts and in sync
make verify-crds

# 4. Format, vet, and test
make fmt vet
make test-unit

# 5. Build the binary
make build
```

Expected output from `make manifests`:
```
/path/to/repo/bin/controller-gen crd paths="./..." output:crd:artifacts:config=deploy/crds
# CRDs written to deploy/crds/
# make sync-crds called automatically
```

Expected output from `make verify-crds` (no drift):
```
# exits 0 with no output
```

---

### Example 2: Running envtest controller tests

```bash
make test-controllers
```

Expected output (abbreviated):
```
KUBEBUILDER_ASSETS="/path/to/repo/bin/k8s/..." go test -p 4 \
  $(go list -f '...' ./pkg/controllers/...) -coverprofile cover-controllers.out
ok  	github.com/trilioData/site-recovery/pkg/controllers/failoverrequest	12.345s
ok  	github.com/trilioData/site-recovery/pkg/controllers/protectionrequest	9.876s
ok  	github.com/trilioData/site-recovery/pkg/controllers/pgsync	8.123s
ok  	github.com/trilioData/site-recovery/pkg/controllers/testfailover	11.234s
coverage: 74.3% of statements
```

> If you see `context deadline exceeded` during API server startup, increase the start timeout: `make test-controllers KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=5m`

---

### Example 3: Building and loading the control-plane image into a Kind cluster

```bash
export IMG=eu.gcr.io/amazing-chalice-243510/site-recovery-control-plane:dev-local

# Build the image
make docker-build IMG=$IMG

# Load into Kind without needing a registry push
kind load docker-image $IMG --name my-dev-cluster

# Deploy via Helm
make deploy IMG=$IMG
```

---

### Example 4: Scaffolding a new controller with kubebuilder

```bash
# Scaffold the API and controller
kubebuilder create api \
  --group siterecovery \
  --version v1alpha1 \
  --kind MyNewKind

# Regenerate all derived artifacts
make manifests generate

# Verify CRD is assigned to a Helm chart (it will not be yet — add it to the Makefile)
make verify-crds
# Output: "CRD kind(s) generated in deploy/crds but not assigned to any Helm chart: mynewkinds"
# Fix: add 'mynewkinds' to QUORUM_CRDS or WORKLOAD_CRDS in the Makefile, then re-run make manifests
```

---

### Example 5: Running the quorum control plane locally with debug logging

```bash
# Ensure CRDs are installed in your current kubeconfig cluster
kubectl apply -f deploy/crds/

# Run with debug verbosity
go run ./cmd/quorum-control-plane/main.go \
  --zap-log-level=debug \
  --zap-encoder=console
```

Expected log output (console encoder, debug level):
```
2024-01-15T10:23:45Z	INFO	Starting manager
2024-01-15T10:23:45Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
2024-01-15T10:23:45Z	DEBUG	Starting reconciliation	{"controller": "failoverrequest", "namespace": "dr-prod", "name": "fr-sample"}
```

---

### Example 6: Checking for generated-file drift in CI

```bash
make verify-generated
```

If types were edited without regenerating:
```
Generated files are out of sync with api/v1alpha1. Run 'make manifests generate' and commit the result:
 api/v1alpha1/zz_generated.deepcopy.go | 12 ++++++------
 deploy/crds/siterecovery.trilio.io_protectiongroups.yaml | 8 ++++++++
```

## Troubleshooting

### CRD drift detected by `make verify-crds`

**Symptom:** `make verify-crds` exits non-zero with output similar to:
```
CRDs are out of sync with api/v1alpha1. Run 'make manifests' and commit the result:
 deploy/crds/siterecovery.trilio.io_protectiongroups.yaml | 4 ++--
 helm/site-recovery-quorum-control-plane/crds/siterecovery.trilio.io_protectiongroups.yaml | 4 ++--
```

**Cause:** `api/v1alpha1/*_types.go` was edited but `make manifests` was not run, or the regenerated files were not committed.

**Fix:**
```bash
make manifests generate
git add deploy/crds/ helm/
git commit -m "Regenerate CRDs after type change"
```

---

### New CRD kind not assigned to a Helm chart

**Symptom:** `make verify-crds` exits with:
```
CRD kind(s) generated in deploy/crds but not assigned to any Helm chart: mynewkinds
```

**Cause:** A new kind was scaffolded but not added to `QUORUM_CRDS`, `WORKLOAD_CRDS`, or `PROTECTIONZONE_CRDS` in the Makefile.

**Fix:** Add the plural lowercase kind name to the correct variable in the Makefile, then re-run `make manifests`:
```makefile
QUORUM_CRDS := drbdreplicationpolicies failoverrequests protectiongroups protectionrequests replicationgroupstatuses rpoevents testfailovers mynewkinds
```

---

### Envtest kube-apiserver fails to start within the timeout

**Symptom:** `make test-unit` or `make test-controllers` fails with:
```
timeout waiting for process to be ready
```
or individual test packages report `context deadline exceeded`.

**Cause:** Running too many envtest instances in parallel on a resource-constrained machine. Each package boots its own kube-apiserver and etcd. The default parallelism of 4 can overwhelm a CI pod that sees many CPUs but has a limited CPU quota.

**Fix:** Reduce parallelism or increase the start timeout:
```bash
make test-unit TEST_PKG_PARALLELISM=2 KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=5m
```

---

### `make run` panics or fails to connect to the cluster

**Symptom:** Running `make run` produces:
```
failed to get API group resources: unable to retrieve the complete list of server APIs
```
or a kubeconfig error.

**Cause:** The CRDs defined in `api/v1alpha1/` are not installed in the cluster pointed to by `~/.kube/config`, or the current context does not have cluster-admin access.

**Fix:** Install the CRDs first:
```bash
kubectl apply -f deploy/crds/
make run
```

---

### Lint fails with "key must be a string literal"

**Symptom:** `make lint` reports an error from the `logcheck` linter:
```
pkg/controllers/failoverrequest/failoverrequest_controller.go:42: key must be a string literal
```

**Cause:** A logging call uses a Go constant as a key instead of an inline string literal. The `logcheck` `keyCheck` rule requires literal strings.

**Fix:** Replace the constant with the corresponding string literal inline:
```go
// Wrong
const pgKey = "protectionGroup"
log.Info("Starting reconciliation", pgKey, pg.Name)

// Correct
log.Info("Starting reconciliation", "protectionGroup", pg.Name)
```

---

### `make build` fails with ldflags version errors

**Symptom:** `make build` fails because `VERSION` resolves to an empty string.

**Cause:** The repository has no git tags and the build is not running under a recognized Prow job type, so `VERSION` is empty.

**Fix:** Set `VERSION` explicitly on the command line:
```bash
make build VERSION=0.0.0-dev
```

---

### Controller logs show `Failed to get object` after a reconcile

**Symptom:** The controller log repeatedly shows errors fetching a CR that you know exists.

**Cause:** The controller is using a stale in-memory copy of the object. This typically happens when an `Update` is called without first re-fetching the latest version, causing a resource version conflict on the next reconcile loop.

**Fix:** Always re-fetch the object immediately before calling `Update` or `Patch`:
```go
if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
    return ctrl.Result{}, client.IgnoreNotFound(err)
}
// Now safe to mutate and update
```