Development Guide

This guide explains how to build, test, and contribute to the site-recovery operator. It covers the repository layout, the Make-based build system, how to run controllers locally against a real Kubernetes API server using envtest, and the conventions you must follow when adding new APIs, controllers, or webhooks. Whether you are fixing a bug, adding a new CRD field, or onboarding as a first-time contributor, this page gives you the full development loop from source change to verified test run.

Before you begin local development, ensure you have the following installed and available on your PATH:

• Go 1.26+ — the module is github.com/trilioData/site-recovery and requires Go 1.26
• Docker (or a compatible CONTAINER_TOOL) — used by make docker-build and cross-platform buildx targets
• kubectl — for inspecting cluster state during local runs
• Helm ≥ 3.0 — the only supported deployment path for the operator
• kubebuilder CLI — required to scaffold new APIs, controllers, and webhooks; do not create scaffolded files manually
• controller-gen v0.19.0 — installed automatically by the Makefile into bin/; used by make manifests and make generate
• golangci-lint — installed automatically into bin/golangci-lint-custom by the Makefile
• setup-envtest — installed automatically by the Makefile; provides the kube-apiserver and etcd binaries used by make test
• Git — version metadata (GIT_COMMIT, BUILD_DATE) is injected at build time via -ldflags
• A local kubeconfig context if you want to run controllers against a live cluster with make run

Note: E2E tests require a dedicated Kind cluster. Never run them against a development or production cluster.

1. Clone the repository

git clone https://github.com/trilioData/site-recovery.git
cd site-recovery

2. Install tool dependencies

The Makefile installs all Go toolchain dependencies (controller-gen, envtest, golangci-lint) into bin/ on first use. No manual installation of these tools is required.

make help   # verify the Makefile is accessible and list all targets

3. Generate CRDs and DeepCopy code

After cloning, or any time you edit a api/v1alpha1/*_types.go file, regenerate the manifests and DeepCopy implementations:

make manifests generate

• make manifests runs controller-gen to produce CRD YAMLs into deploy/crds/ (the single canonical location) and then calls make sync-crds to copy the per-chart subsets into each Helm chart's crds/ directory.
• make generate regenerates the zz_generated.deepcopy.go files.

Do not edit deploy/crds/*.yaml, helm/*/crds/*.yaml, **/zz_generated.*.go, or the PROJECT file. These are all auto-generated.

4. Build the control-plane binaries

make build

This produces two binaries in bin/:

• bin/quorum-manager — built from cmd/quorum-control-plane/main.go
• bin/workload-manager — built from cmd/workload-control-plane/main.go

Both binaries are shipped inside the single control-plane container image. Each Helm chart selects its own binary via the container command.

5. Build container images

Build the control-plane image (controllers and webhooks):

export IMG=<registry>/site-recovery-control-plane:<tag>
make docker-build IMG=$IMG
make docker-push IMG=$IMG

Additional image targets:

# DRBD node agent
make docker-build-node-agent NODE_AGENT_IMG=<registry>/site-recovery-drbd-node-agent:<tag>

# UI frontend (Next.js)
make docker-build-frontend FRONTEND_IMG=<registry>/site-recovery-frontend:<tag>

# UI backend (Flask)
make docker-build-backend BACKEND_IMG=<registry>/site-recovery-backend:<tag>

# must-gather diagnostics collector
make docker-build-must-gather MUST_GATHER_IMG=<registry>/site-recovery-must-gather:<tag>

The UI frontend and backend images build from the ui/ directory. The Go image build contexts live under docker-images/.

6. Deploy via Helm

Helm is the only supported deployment path. The make deploy target runs helm upgrade --install using the chart selected by HELM_CHART:

export IMG=<registry>/site-recovery-control-plane:<tag>
make deploy IMG=$IMG

To load a locally built image into a Kind cluster instead of pushing to a registry:

kind load docker-image $IMG --name <cluster-name>

7. Run locally against your kubeconfig context

To run the quorum control plane process on your host against the cluster pointed to by ~/.kube/config:

make run

To run the workload control plane (admission webhooks, no reconcilers):

make run-workload

Build-time variables

The Makefile resolves the following variables automatically. Override them on the command line when needed.

Registry selection logic:

• Presubmit CI builds always publish to eu.gcr.io/amazing-chalice-243510.
• release* branches publish to quay.io/triliodata.
• tech-preview* branches publish to quay.io/triliovault.
• All other branches (including local dev) publish to eu.gcr.io/amazing-chalice-243510.

Helm chart values

The three Helm charts under helm/ accept values that control logging and image references. The most commonly tuned values are:

Important: Never hardcode Development: true in any Go logging initialization. Production logging is always JSON-encoded at the info level unless overridden by Helm values or a --zap-log-level flag at startup. The log level can be changed at runtime without restarting the process.

CRD API group

All CRDs use the API group siterecovery.trilio.io/v1alpha1. The single-group layout places all types in api/v1alpha1/. Do not move files or change this group without a kubebuilder multi-group migration (see the project layout section below).

Kubebuilder markers

Key markers for api/v1alpha1/*_types.go:

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespaced
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=".status.conditions[?(@.type=='Ready')].status"

// On fields:
// +kubebuilder:validation:Required
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:MaxLength=100
// +kubebuilder:validation:Pattern="^[a-z]+$"
// +kubebuilder:default="value"

Always use metav1.Condition for status conditions, metav1.Time for timestamps, and standard Kubernetes field names (spec, status, metadata).

Controller RBAC markers

Place RBAC markers in pkg/controllers/*_controller.go. They are processed by make manifests into the Helm chart templates — do not hand-edit the generated RBAC manifests.

// +kubebuilder:rbac:groups=siterecovery.trilio.io,resources=protectiongroups,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=siterecovery.trilio.io,resources=protectiongroups/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=siterecovery.trilio.io,resources=protectiongroups/finalizers,verbs=update
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch

Typical development loop

The standard flow for making a code change is:

1. Edit api/v1alpha1/*_types.go or controller code in pkg/controllers/.
2. Regenerate if you changed types:
   Copy

   make manifests generate
   

3. Format and vet:
   Copy

   make fmt
   make vet
   

4. Run tests:
   Copy

   make test-unit
   

5. Auto-fix lint issues:
   Copy

   make lint-fix
   

6. Build and verify locally:
   Copy

   make build
   

Scaffolding new APIs

Always use the kubebuilder CLI to scaffold new types. Never create scaffolded files manually.

kubebuilder create api --group siterecovery --version v1alpha1 --kind MyKind

After scaffolding, run:

make manifests generate

Scaffolding webhooks

kubebuilder create webhook --group siterecovery --version v1alpha1 --kind MyKind \
  --defaulting --programmatic-validation

Note: Always create validation and defaulting webhooks together. If you need to re-scaffold, back up any custom logic in pkg/webhook/ first, re-run with --force, then restore your customizations.

Running tests selectively

# All envtest unit tests (CI merge gate)
make test-unit

# Full suite including codegen and format checks
make test

# Controller tests only
make test-controllers

# Webhook tests only
make test-webhooks

Tests use Ginkgo + Gomega in BDD style. Each test package boots its own kube-apiserver and etcd via envtest. Check suite_test.go in each package for setup details. Parallelism is capped at TEST_PKG_PARALLELISM=4 by default to prevent CPU starvation on CI pods; increase it on well-resourced workstations.

Running a controller locally

To run the quorum control plane directly on your host against the cluster in your current kubeconfig context:

make run

The process registers all controllers and webhooks defined in cmd/quorum-control-plane/main.go. It uses structured JSON logging at info level by default. Pass --zap-log-level=debug to increase verbosity:

go run ./cmd/quorum-control-plane/main.go --zap-log-level=debug

Writing controller reconcilers

Follow these rules in all reconciler implementations:

• Idempotent reconciliation: Every reconcile call must be safe to run multiple times with the same outcome.
• Re-fetch before updates: Always call r.Get(ctx, req.NamespacedName, obj) immediately before r.Update or r.Patch to avoid stale-object conflicts.
• Structured logging: Acquire the logger from context — log := logf.FromContext(ctx) — never use a package-level logger. This inherits controller, namespace, name, and reconcileID automatically.
• Owner references: Use SetControllerReference to enable automatic garbage collection of secondary resources.
• Watch secondary resources: Use .Owns() or .Watches() on the controller builder rather than polling with RequeueAfter.
• Finalizers: Register a finalizer for any controller that creates external resources (VMs, volumes, DNS entries) that must be cleaned up on deletion.

Logging conventions

All logging uses logr backed by controller-runtime's zap integration. Use the named verbosity constants from internal/logging:

log := logf.FromContext(ctx)

// State changes and decisions (always visible)
log.Info("Starting reconciliation")
log.Info("Created ProtectionGroup", "protectionGroup", pg.Name)

// Actionable failures (always pass the real error object)
log.Error(err, "Failed to create DRBDVolume", "pvc", pvcName)

// Per-phase debug detail (diagnoses stuck failovers)
log.V(logging.Debug).Info("Waiting for volume sync", "drbdResource", res.Name, "phase", phase)

// High-frequency trace
log.V(logging.Trace).Info("Polling replication state", "cluster", clusterName)

Use the canonical key vocabulary — protectionGroup, vm, cluster, phase, error, node, pvc, pv, drbdResource, result — consistently. Never log whole CR objects, Secret data, kubeconfigs, or tokens.

UI development

The UI lives in the ui/ directory. It is a split stack with a Next.js frontend and a Flask backend.

cd ui

# Frontend
npm install
npm run dev          # Dev server at localhost:3000
npm run build        # Production build
npm run lint         # ESLint
npm run type-check   # TypeScript check

# Backend
pip install -r requirements.txt
python app.py        # Flask API server

The UI is not Helm-packaged; it ships in the OLM bundle under deploy/olm-catalog/.

Example 1: Full code-change loop — add a field to ProtectionGroup

This example adds a new optional field to the ProtectionGroup spec and regenerates all derived artifacts.

# 1. Edit the type
$EDITOR api/v1alpha1/protectiongroup_types.go

# 2. Regenerate CRDs and DeepCopy
make manifests generate

# 3. Verify CRDs are assigned to their charts and in sync
make verify-crds

# 4. Format, vet, and test
make fmt vet
make test-unit

# 5. Build the binary
make build

Expected output from make manifests:

/path/to/repo/bin/controller-gen crd paths="./..." output:crd:artifacts:config=deploy/crds
# CRDs written to deploy/crds/
# make sync-crds called automatically

Expected output from make verify-crds (no drift):

# exits 0 with no output

Example 2: Running envtest controller tests

make test-controllers

Expected output (abbreviated):

KUBEBUILDER_ASSETS="/path/to/repo/bin/k8s/..." go test -p 4 \
  $(go list -f '...' ./pkg/controllers/...) -coverprofile cover-controllers.out
ok  	github.com/trilioData/site-recovery/pkg/controllers/failoverrequest	12.345s
ok  	github.com/trilioData/site-recovery/pkg/controllers/protectionrequest	9.876s
ok  	github.com/trilioData/site-recovery/pkg/controllers/pgsync	8.123s
ok  	github.com/trilioData/site-recovery/pkg/controllers/testfailover	11.234s
coverage: 74.3% of statements

If you see context deadline exceeded during API server startup, increase the start timeout: make test-controllers KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=5m

Example 3: Building and loading the control-plane image into a Kind cluster

export IMG=eu.gcr.io/amazing-chalice-243510/site-recovery-control-plane:dev-local

# Build the image
make docker-build IMG=$IMG

# Load into Kind without needing a registry push
kind load docker-image $IMG --name my-dev-cluster

# Deploy via Helm
make deploy IMG=$IMG

Example 4: Scaffolding a new controller with kubebuilder

# Scaffold the API and controller
kubebuilder create api \
  --group siterecovery \
  --version v1alpha1 \
  --kind MyNewKind

# Regenerate all derived artifacts
make manifests generate

# Verify CRD is assigned to a Helm chart (it will not be yet — add it to the Makefile)
make verify-crds
# Output: "CRD kind(s) generated in deploy/crds but not assigned to any Helm chart: mynewkinds"
# Fix: add 'mynewkinds' to QUORUM_CRDS or WORKLOAD_CRDS in the Makefile, then re-run make manifests

Example 5: Running the quorum control plane locally with debug logging

# Ensure CRDs are installed in your current kubeconfig cluster
kubectl apply -f deploy/crds/

# Run with debug verbosity
go run ./cmd/quorum-control-plane/main.go \
  --zap-log-level=debug \
  --zap-encoder=console

Expected log output (console encoder, debug level):

2024-01-15T10:23:45Z	INFO	Starting manager
2024-01-15T10:23:45Z	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
2024-01-15T10:23:45Z	DEBUG	Starting reconciliation	{"controller": "failoverrequest", "namespace": "dr-prod", "name": "fr-sample"}

Example 6: Checking for generated-file drift in CI

make verify-generated

If types were edited without regenerating:

Generated files are out of sync with api/v1alpha1. Run 'make manifests generate' and commit the result:
 api/v1alpha1/zz_generated.deepcopy.go | 12 ++++++------
 deploy/crds/siterecovery.trilio.io_protectiongroups.yaml | 8 ++++++++

CRD drift detected by make verify-crds

Symptom: make verify-crds exits non-zero with output similar to:

CRDs are out of sync with api/v1alpha1. Run 'make manifests' and commit the result:
 deploy/crds/siterecovery.trilio.io_protectiongroups.yaml | 4 ++--
 helm/site-recovery-quorum-control-plane/crds/siterecovery.trilio.io_protectiongroups.yaml | 4 ++--

Cause: api/v1alpha1/*_types.go was edited but make manifests was not run, or the regenerated files were not committed.

Fix:

make manifests generate
git add deploy/crds/ helm/
git commit -m "Regenerate CRDs after type change"

New CRD kind not assigned to a Helm chart

Symptom: make verify-crds exits with:

CRD kind(s) generated in deploy/crds but not assigned to any Helm chart: mynewkinds

Cause: A new kind was scaffolded but not added to QUORUM_CRDS, WORKLOAD_CRDS, or PROTECTIONZONE_CRDS in the Makefile.

Fix: Add the plural lowercase kind name to the correct variable in the Makefile, then re-run make manifests:

QUORUM_CRDS := drbdreplicationpolicies failoverrequests protectiongroups protectionrequests replicationgroupstatuses rpoevents testfailovers mynewkinds

Envtest kube-apiserver fails to start within the timeout

Symptom: make test-unit or make test-controllers fails with:

timeout waiting for process to be ready

or individual test packages report context deadline exceeded.

Cause: Running too many envtest instances in parallel on a resource-constrained machine. Each package boots its own kube-apiserver and etcd. The default parallelism of 4 can overwhelm a CI pod that sees many CPUs but has a limited CPU quota.

Fix: Reduce parallelism or increase the start timeout:

make test-unit TEST_PKG_PARALLELISM=2 KUBEBUILDER_CONTROLPLANE_START_TIMEOUT=5m

make run panics or fails to connect to the cluster

Symptom: Running make run produces:

failed to get API group resources: unable to retrieve the complete list of server APIs

or a kubeconfig error.

Cause: The CRDs defined in api/v1alpha1/ are not installed in the cluster pointed to by ~/.kube/config, or the current context does not have cluster-admin access.

Fix: Install the CRDs first:

kubectl apply -f deploy/crds/
make run

Lint fails with "key must be a string literal"

Symptom: make lint reports an error from the logcheck linter:

pkg/controllers/failoverrequest/failoverrequest_controller.go:42: key must be a string literal

Cause: A logging call uses a Go constant as a key instead of an inline string literal. The logcheck keyCheck rule requires literal strings.

Fix: Replace the constant with the corresponding string literal inline:

// Wrong
const pgKey = "protectionGroup"
log.Info("Starting reconciliation", pgKey, pg.Name)

// Correct
log.Info("Starting reconciliation", "protectionGroup", pg.Name)

make build fails with ldflags version errors

Symptom: make build fails because VERSION resolves to an empty string.

Cause: The repository has no git tags and the build is not running under a recognized Prow job type, so VERSION is empty.

Fix: Set VERSION explicitly on the command line:

make build VERSION=0.0.0-dev

Controller logs show Failed to get object after a reconcile

Symptom: The controller log repeatedly shows errors fetching a CR that you know exists.

Cause: The controller is using a stale in-memory copy of the object. This typically happens when an Update is called without first re-fetching the latest version, causing a resource version conflict on the next reconcile loop.

Fix: Always re-fetch the object immediately before calling Update or Patch:

if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
    return ctrl.Result{}, client.IgnoreNotFound(err)
}
// Now safe to mutate and update