Architecting Custom Kubernetes Operators in Go with controller-runtime & Kubebuilder

Kubernetes native primitives like Deployments and StatefulSets excel at managing stateless web workloads, but fail when tasked with automating stateful distributed clusters (e.g., PostgreSQL HA, Redis Sentinel, Apache Kafka partitions, or custom microservice topologies). The Kubernetes Operator pattern bridges this gap by embedding domain-specific operational logic directly into the Kubernetes control plane using Custom Resource Definitions (CRDs) and custom controller reconcile loops. Here is how we build production-grade operators in Go with controller-runtime.

1. The Operator Mental Model: Level-Triggered Reconciliation

At its core, a Kubernetes controller operates on a continuous feedback loop: it observes the actual state of the cluster via an API informers cache, compares it against the desired state declared in a Custom Resource spec, and executes mutations until both converge.

Level-Triggered vs Edge-Triggered

Unlike edge-triggered message queues where losing an event causes missed state changes, Kubernetes controllers are level-triggered. A single reconcile execution does not care how many events triggered it; it simply reads the latest snapshot of the world and acts idempotently.

2. Custom Resource Definition (CRD) Architecture

Using Kubebuilder annotations, we define our Go struct schema. The Spec represents user intent, while the Status holds observed cluster state and phase indicators:

api/v1alpha1/redisfallback_types.go

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// RedisClusterSpec defines the desired state of RedisCluster
type RedisClusterSpec struct {
	// Replicas is the desired number of Redis read-replicas
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=10
	Replicas int32 `json:"replicas"`

	// MemoryLimit is the maximum container memory request
	MemoryLimit string `json:"memoryLimit"`

	// EnablePersistence configures RDB/AOF volume attachments
	EnablePersistence bool `json:"enablePersistence"`
}

// RedisClusterStatus defines the observed state of RedisCluster
type RedisClusterStatus struct {
	// Phase captures the current lifecycle state (Pending, Provisioning, Ready, Failed)
	Phase string `json:"phase"`

	// ActiveReplicas is the count of operational running pods
	ActiveReplicas int32 `json:"activeReplicas"`

	// LastSyncTimestamp records the latest successful reconcile run
	LastSyncTimestamp *metav1.Time `json:"lastSyncTimestamp,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`

// RedisCluster is the Schema for the redisclusters API
type RedisCluster struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   RedisClusterSpec   `json:"spec,omitempty"`
	Status RedisClusterStatus `json:"status,omitempty"`
}

3. Writing an Idempotent Reconciler in Go

The reconciler must be completely idempotent. If invoked ten times in a row without state changes, it must perform zero destructive API operations.

controllers/rediscluster_controller.go

package controllers

import (
	"context"
	"fmt"
	"time"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"

	cachev1alpha1 "seang.org/operator/api/v1alpha1"
)

const redisFinalizer = "cache.seang.org/finalizer"

type RedisClusterReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

func (r *RedisClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx).WithValues("rediscluster", req.NamespacedName)

	// 1. Fetch the Custom Resource from the cached client
	var cluster cachev1alpha1.RedisCluster
	if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
		if errors.IsNotFound(err) {
			logger.Info("Resource deleted, skipping reconcile loop")
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	// 2. Handle Graceful Deletion via Finalizers
	if !cluster.ObjectMeta.DeletionTimestamp.IsZero() {
		if controllerutil.ContainsFinalizer(&cluster, redisFinalizer) {
			logger.Info("Executing external cleanup before resource deletion")
			if err := r.cleanupExternalResources(ctx, &cluster); err != nil {
				return ctrl.Result{}, err
			}
			controllerutil.RemoveFinalizer(&cluster, redisFinalizer)
			return ctrl.Result{}, r.Update(ctx, &cluster)
		}
		return ctrl.Result{}, nil
	}

	// Register finalizer if absent
	if !controllerutil.ContainsFinalizer(&cluster, redisFinalizer) {
		controllerutil.AddFinalizer(&cluster, redisFinalizer)
		if err := r.Update(ctx, &cluster); err != nil {
			return ctrl.Result{}, err
		}
	}

	// 3. Ensure Child Resources (ConfigMaps, Pods, Services)
	if err := r.reconcileConfigMap(ctx, &cluster); err != nil {
		return ctrl.Result{}, err
	}

	if err := r.reconcileStatefulSet(ctx, &cluster); err != nil {
		return ctrl.Result{}, err
	}

	// 4. Update Status Subresource
	cluster.Status.Phase = "Ready"
	cluster.Status.ActiveReplicas = cluster.Spec.Replicas
	now := metav1.Now()
	cluster.Status.LastSyncTimestamp = &now

	if err := r.Status().Update(ctx, &cluster); err != nil {
		if errors.IsConflict(err) {
			// Optimistic concurrency conflict: retry without error logging
			return ctrl.Result{Requeue: true}, nil
		}
		return ctrl.Result{}, err
	}

	logger.Info("Successfully synchronized RedisCluster state")
	return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

4. Event Filtering with Predicates: Preventing Cache Thrashing

By default, controller-runtime triggers a Reconcile on any mutation event to watched objects, including periodic status updates or resync ticks. In high-traffic clusters, this causes CPU starvation and reconcile storms.

We attach custom predicate.Funcs to only trigger reconciliation when the metadata.Generation increments (signifying an actual change to the Spec), discarding non-essential metadata changes:

controllers/setup.go

import (
	"sigs.k8s.io/controller-runtime/pkg/event"
	"sigs.k8s.io/controller-runtime/pkg/predicate"
)

func (r *RedisClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.RedisCluster{}).
		WithEventFilter(predicate.Funcs{
			UpdateFunc: func(e event.UpdateEvent) bool {
				// Ignore updates if the generation has not changed (e.g., status-only updates)
				return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration()
			},
			DeleteFunc: func(e event.DeleteEvent) bool {
				// Evaluates deletion tombstone
				return !e.DeleteStateUnknown
			},
		}).
		Complete(r)
}

5. High Availability with Leader Election

To run multiple replicas of the Operator pod across availability zones without risking split-brain resource collisions, we activate Kubernetes API Lease-based leader election:

main.go

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
	Scheme:                 scheme,
	MetricsBindAddress:     ":8080",
	Port:                   9443,
	HealthProbeBindAddress: ":8081",
	LeaderElection:         true,
	LeaderElectionID:       "redis-operator-lease.seang.org",
	LeaderElectionNamespace: "kube-system",
})

6. Production Operator Engineering Checklist

Always Use Status Subresources: Isolating /status avoids triggering unnecessary spec increments and simplifies RBAC security boundaries.
Adopt Server-Side Apply: For complex child resources, use Server-Side Apply (SSA) to avoid full object overwrites and race conditions with other cluster controllers.
Graceful Finalizers: Never leave orphaned cloud load balancers, DNS records, or external stateful disks when a Custom Resource is deleted.
Bounded Workqueues: Ensure rate-limited requeuing with exponential backoff (workqueue.NewItemExponentialFailureRateLimiter) so downstream errors do not overwhelm the Kubernetes API server.