“

Series: S6 Platform Engineering · Article 19 | Level: Advanced | Reading Time: 18 min

Kubernetes Operator: Declarative Platform Lifecycle Management

#TL;DR

The coomia-dip Kubernetes Operator implements declarative platform lifecycle management through Custom Resource Definitions (CRDs) that describe the platform topology. The Operator controller automatically handles service deployment, configuration management, rolling upgrades, elastic scaling, and fault recovery. This article covers Operator architecture, CRD design, controller implementation, upgrade strategies, and operational capabilities.

#1. Why an Operator

#1.1 Docker Compose Limitations in Production

No self-healing: Crashed services do not auto-restart or reschedule
No elastic scaling: Cannot auto-adjust instances based on load
Rolling upgrades difficult: Manual orchestration of upgrade ordering
Lack of declarative management: Config changes require manual application

#1.2 Operator Pattern

The Operator pattern encodes operational knowledge as software for automated Day-2 operations:

Code

User → Declare desired state (CRD) → Operator → Reconcile actual state → Kubernetes

#2. CRD Design

#2.1 OntoPlatform CRD

YAML

apiVersion: onto.paas/v1alpha1
kind: OntoPlatform
metadata:
  name: onto-production
  namespace: onto-system
spec:
  version: "1.5.0"

  global:
    imagePullPolicy: IfNotPresent
    imageRegistry: registry.example.com/coomia-dip
    storageClass: fast-ssd
    tlsEnabled: true

  infrastructure:
    postgres:
      replicas: 3
      storage: 100Gi
      resources:
        requests: { cpu: "2", memory: "4Gi" }
        limits: { cpu: "4", memory: "8Gi" }
    redis:
      replicas: 3
      mode: sentinel
    minio:
      replicas: 4
      storage: 500Gi
    kafka:
      replicas: 3
      storage: 100Gi

  controlPlane:
    ontologyService:
      replicas: 3
      autoscaling:
        enabled: true
        minReplicas: 2
        maxReplicas: 10
        targetCPUUtilization: 70
    schemaRegistry:
      replicas: 2
    authService:
      replicas: 2

  dataPlane:
    dataService:
      replicas: 3
      autoscaling:
        enabled: true
        minReplicas: 2
        maxReplicas: 20
    nessie:
      replicas: 2

  intelligencePlane:
    reasoningService:
      replicas: 2
      gpu:
        enabled: true
        count: 1
    agentRuntime:
      replicas: 2

  gateway:
    replicas: 3
    ingress:
      enabled: true
      host: platform.example.com
      tls: true

  observability:
    metrics:
      enabled: true
      serviceMonitor: true
    tracing:
      enabled: true
    logging:
      level: INFO
      format: json

#3. Operator Controller

#3.1 Reconciliation Loop

Python

class OntoPlatformReconciler:
    async def reconcile(self, request: ReconcileRequest) -> ReconcileResult:
        platform = await self._get_platform(request)
        if platform is None:
            return ReconcileResult(requeue=False)

        try:
            await self._reconcile_infrastructure(platform)
            await self._reconcile_control_plane(platform)
            await self._reconcile_data_plane(platform)
            await self._reconcile_intelligence_plane(platform)
            await self._reconcile_gateway(platform)
            await self._reconcile_observability(platform)
            await self._update_status(platform, phase="Running")

            return ReconcileResult(requeue_after=timedelta(minutes=5))
        except Exception as e:
            await self._update_status(platform, phase="Error", message=str(e))
            return ReconcileResult(requeue_after=timedelta(seconds=30))

#3.2 Rolling Upgrades

Python

class RollingUpgradeController:
    UPGRADE_ORDER = [
        "infrastructure",
        "control-Layer",
        "data-Layer",
        "intelligence",
        "gateway",
    ]

    async def upgrade(self, platform, target_version: str) -> UpgradeResult:
        current_version = platform.status.version

        for component_group in self.UPGRADE_ORDER:
            components = self._get_components(platform, component_group)
            for component in components:
                await self._canary_deploy(component, target_version)
                healthy = await self._wait_for_healthy(component, timeout=300)
                if not healthy:
                    await self._rollback(component, current_version)
                    return UpgradeResult(success=False, failed_component=component.name)
                await self._rolling_replace(component, target_version)

        return UpgradeResult(success=True, version=target_version)

#4. Elastic Scaling

YAML

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ontology-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ontology-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Pods
      pods:
        metric:
          name: grpc_requests_per_second
        target:
          type: AverageValue
          averageValue: "1000"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300

#5. Fault Recovery

Python

class SelfHealingController:
    async def on_pod_failure(self, event: PodEvent) -> None:
        if event.reason == "OOMKilled":
            await self._increase_memory_limit(event.pod, factor=1.5)
        elif event.reason == "CrashLoopBackOff":
            await self._diagnose_crash_loop(event.pod)
        elif event.reason == "Evicted":
            await self._check_node_resources(event.node)

    async def on_node_failure(self, event: NodeEvent) -> None:
        affected_pods = await self._get_pods_on_node(event.node)
        for pod in affected_pods:
            await self._reschedule_pod(pod)

#6. Multi-Tenancy

YAML

apiVersion: onto.paas/v1alpha1
kind: OntoPlatform
metadata:
  name: tenant-alpha
  namespace: tenant-alpha
spec:
  tenancy:
    mode: namespace
    resourceQuota:
      requests.cpu: "10"
      requests.memory: "20Gi"
    networkPolicy:
      isolate: true
      allowedNamespaces: ["onto-system"]

#7. Testing

Python

class TestK8sOperator:
    def test_crd_validation(self):
        with pytest.raises(ValidationError):
            OntoPlatform(spec={"version": ""})

    async def test_reconciliation(self):
        platform = make_test_platform()
        reconciler = OntoPlatformReconciler(k8s_client)
        result = await reconciler.reconcile(make_request(platform))
        assert result.requeue_after is not None

    async def test_rolling_upgrade(self):
        controller = RollingUpgradeController(k8s_client)
        result = await controller.upgrade(platform, "1.6.0")
        assert result.success

#8. Production Best Practices

Use dedicated node pools to isolate platform and business workloads
Set pod anti-affinity for critical services ensuring cross-node distribution
Use PDB (Pod Disruption Budget) to protect high availability
Regularly test failure recovery procedures
Monitor the Operator itself for health and performance
Record audit logs for all reconciliation operations

#9. Summary

The coomia-dip Kubernetes Operator achieves production-grade platform lifecycle management through declarative CRDs and automated controllers. Key highlights:

Declarative management: CRDs describe topology, Operator reconciles automatically
Rolling upgrades: Component-ordered upgrades with canary validation and auto-rollback
Elastic scaling: HPA + custom metrics for intelligent scaling
Self-healing: Automatic detection and recovery of common failures
Multi-tenancy: Namespace isolation with resource quotas

The next article will explore coomia-dip OpenTelemetry observability.