Error Handling: How Three Processes Handle Failures Gracefully

“

Series: S2 Architecture Overview · Article 11 | Level: Intermediate | Reading Time: 18 min

#TL;DR

The coomia-dip platform adopts a unified gRPC error code system, mapping all inter-service errors to standard gRPC Status Codes. The API Gateway converts these to HTTP status codes for frontend consumption, ensuring error semantic consistency.
Three processes implement layered retry strategies: onto-control uses Spring Retry with exponential backoff, onto-data uses Quarkus Fault Tolerance circuit breakers, and onto-intelligence uses Temporal workflow-level retries — ensuring failures don't cascade.
Dead Letter Queues (DLQ) + manual approval fallback + graceful degradation provide triple assurance, enabling the platform to deliver degraded but available service during partial failures.

#Introduction: Failure Is the Norm, Not the Exception

In distributed systems, network partitions, process crashes, and resource exhaustion are everyday events. The coomia-dip platform consists of three independent processes — onto-control (Control Layer), onto-data (Data Layer), and onto-intelligence (Reasoning & Decision Layer + Agent Runtime Layer) — communicating via gRPC. Any cross-process call can fail; any process can crash.

Code

Failure Probability Model (Simplified):

Single gRPC call failure rate:     ~0.1% (network jitter)
Single process restart/hour:       ~0.5% (OOM, GC pause)
Calls per business operation:      3-7
Business operation failure rate:   ~0.3%-0.7%

At 100K daily requests:
  Expected daily failed requests:  300-700
  User-visible errors:             Target < 10 (absorbed by retry + degradation)

This article systematically explains how the coomia-dip platform minimizes user impact through five mechanisms: gRPC error code standards, multi-layer retry strategies, circuit breaker patterns, graceful degradation, and dead letter queues.

#1. gRPC Error Code System

#1.1 Standard Error Code Mapping

The platform mandates gRPC for all inter-service communication. gRPC defines 16 standard status codes, and the platform specifies usage scenarios for each:

Code

gRPC Status Code Usage Specification:

Code               Value  Usage Scenario                    Retryable?
──────────────────────────────────────────────────────────────────────
OK                 0      Success                           -
CANCELLED          1      Client-initiated cancellation     No
UNKNOWN            2      Unclassified error (fallback)     Depends
INVALID_ARGUMENT   3      Validation failure                No (fix first)
DEADLINE_EXCEEDED  4      Timeout                           Yes
NOT_FOUND          5      Resource doesn't exist            No
ALREADY_EXISTS     6      Resource exists (idempotency)     No
PERMISSION_DENIED  7      Insufficient permissions          No
RESOURCE_EXHAUSTED 8      Quota or rate limiting            Yes (with backoff)
FAILED_PRECONDITION 9     Precondition not met              No (fix first)
ABORTED            10     Transaction/optimistic lock conflict  Yes
OUT_OF_RANGE       11     Pagination out of bounds          No
UNIMPLEMENTED      12     Method not implemented            No
INTERNAL           13     Internal service error            Yes (limited)
UNAVAILABLE        14     Service unavailable               Yes (with backoff)
DATA_LOSS          15     Unrecoverable data loss           No
UNAUTHENTICATED    16     Not authenticated                 No (re-auth first)

#1.2 Error Details

Bare status codes are insufficient for debugging. The platform requires all gRPC errors to carry structured error details:

PROTOBUF

// Platform unified error detail
message PlatformErrorDetail {
  string error_id = 1;          // Unique error ID (for log correlation)
  string service = 2;           // Service where error occurred
  string Layer = 3;             // Layer (B/C/D/E)
  int64 timestamp = 4;          // Error timestamp
  string trace_id = 5;          // Distributed trace ID
  string world_id = 6;          // Associated World ID
  repeated string context = 7;  // Context information chain
  RetryAdvice retry_advice = 8; // Retry recommendation
}

message RetryAdvice {
  bool retryable = 1;           // Whether retryable
  int32 max_retries = 2;        // Suggested max retry count
  int64 retry_after_ms = 3;     // Suggested wait time (milliseconds)
  string strategy = 4;          // "exponential" | "linear" | "fixed"
}

#1.3 Error Propagation Chain

When a request spans multiple processes, errors must propagate along the call chain while preserving context at each layer:

Code

Error Propagation Example (Action Creation Failure):

User Request -> API Gateway -> onto-control -> onto-intelligence
                                                   |
                                              Rule evaluation timeout
                                              gRPC DEADLINE_EXCEEDED
                                                   |
                                onto-control catches, appends context
                                "Action pre-check failed: rule evaluation timeout"
                                gRPC DEADLINE_EXCEEDED + PlatformErrorDetail
                                                   |
                                API Gateway converts
                                HTTP 504 Gateway Timeout
                                {
                                  "error": "ACTION_PRECHECK_TIMEOUT",
                                  "message": "Action pre-check timed out",
                                  "trace_id": "abc-123-def",
                                  "retry_after": 5000
                                }

#1.4 gRPC Error Interceptors in Each Process

Each process implements a gRPC interceptor for unified error capture, wrapping, and reporting:

Code

onto-control (Java / Spring Boot):
  GlobalGrpcExceptionInterceptor
    +-- Catch Spring exceptions -> map to gRPC error codes
    +-- Inject trace_id and world_id
    +-- Log to structured log (JSON format)
    +-- Report to Prometheus metrics (grpc_server_errors_total)

onto-data (Java / Quarkus):
  QuarkusGrpcErrorInterceptor
    +-- Catch Iceberg/Nessie exceptions -> map to gRPC error codes
    +-- NessieConflictException -> ABORTED (retryable)
    +-- IcebergCommitFailedException -> ABORTED (retryable)
    +-- Report to Micrometer metrics

onto-intelligence (Python / FastAPI):
  grpc_error_interceptor (async interceptor)
    +-- Catch Python exceptions -> map to gRPC error codes
    +-- TimeoutError -> DEADLINE_EXCEEDED
    +-- RuleEvaluationError -> FAILED_PRECONDITION
    +-- Report to Prometheus metrics (prometheus_client)

#2. Retry Strategies

#2.1 Retry Decision Tree

Not all errors should be retried. The platform defines a clear retry decision tree:

Code

Received gRPC Error
  |
  +-- Check RetryAdvice.retryable
  |   +-- false -> Return error immediately
  |   +-- true -> Continue
  |
  +-- Check error code
  |   +-- UNAVAILABLE -> Retry (service may be restarting)
  |   +-- DEADLINE_EXCEEDED -> Retry (may be transient timeout)
  |   +-- ABORTED -> Retry (optimistic lock conflict)
  |   +-- RESOURCE_EXHAUSTED -> Retry (rate limited, needs backoff)
  |   +-- INTERNAL -> Limited retry (could be bug or transient)
  |   +-- Other -> Don't retry
  |
  +-- Check retry count
  |   +-- < max_retries -> Execute retry
  |   +-- >= max_retries -> Give up, trigger degradation logic
  |
  +-- Calculate backoff time
      +-- exponential: base * 2^attempt + jitter
      +-- linear: base * attempt + jitter
      +-- fixed: base + jitter

#2.2 onto-control: Spring Retry

onto-control uses the Spring Retry framework for declarative retry:

Java

// gRPC client retry configuration in onto-control
@Configuration
public class GrpcRetryConfig {

    @Bean
    public RetryTemplate grpcRetryTemplate() {
        return RetryTemplate.builder()
            .maxAttempts(3)
            .exponentialBackoff(
                100,    // initialInterval: 100ms
                2.0,    // multiplier
                2000,   // maxInterval: 2s
                true    // withJitter
            )
            .retryOn(StatusRuntimeException.class)
            .traversingCauses()
            .build();
    }
}

// Usage: calling onto-intelligence for rule evaluation
@GrpcService
public class ActionServiceImpl {

    @Retryable(
        retryFor = {StatusRuntimeException.class},
        maxAttempts = 3,
        backoff = @Backoff(delay = 200, multiplier = 2)
    )
    public ActionResult executeAction(ActionRequest request) {
        // Call onto-intelligence
        var result = intelligenceClient.evaluateRules(request);
        return result;
    }

    @Recover
    public ActionResult executeActionFallback(
            StatusRuntimeException e, ActionRequest request) {
        // Degradation: enqueue to pending queue
        pendingActionQueue.enqueue(request, e);
        return ActionResult.pending("Rule evaluation unavailable, Action queued");
    }
}

#2.3 onto-data: Quarkus Fault Tolerance

onto-data uses MicroProfile Fault Tolerance annotations:

Java

// Nessie operation retry in onto-data
@ApplicationScoped
public class NessieCommitService {

    @Retry(
        maxRetries = 5,
        delay = 100,
        maxDuration = 10000,  // max 10 seconds
        jitter = 50,
        retryOn = {NessieConflictException.class}
    )
    @Fallback(fallbackMethod = "commitWithMerge")
    public CommitResult commitChanges(Branch branch, List<Operation> ops) {
        return nessieApi.commitMultipleOperations()
            .branch(branch)
            .operations(ops)
            .commit();
    }

    // Fallback: try merge then commit
    public CommitResult commitWithMerge(Branch branch, List<Operation> ops) {
        var latestBranch = nessieApi.getReference()
            .refName(branch.getName())
            .get();
        // Three-way merge + retry commit
        return mergeAndCommit(latestBranch, branch, ops);
    }
}

#2.4 onto-intelligence: Temporal Workflow Retry

Long-running tasks in onto-intelligence use Temporal workflows with built-in retry:

Python

# Temporal Activity retry configuration in onto-intelligence
from temporalio import activity, workflow
from temporalio.common import RetryPolicy
from datetime import timedelta

STANDARD_RETRY_POLICY = RetryPolicy(
    initial_interval=timedelta(seconds=1),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(minutes=5),
    maximum_attempts=10,
    non_retryable_error_types=[
        "InvalidArgumentError",
        "PermissionDeniedError",
        "DataValidationError",
    ],
)

@workflow.defn
class RuleEvaluationWorkflow:
    @workflow.run
    async def run(self, request: RuleEvalRequest) -> RuleEvalResult:
        # Activity calls auto-retry
        result = await workflow.execute_activity(
            evaluate_rules,
            request,
            start_to_close_timeout=timedelta(minutes=2),
            retry_policy=STANDARD_RETRY_POLICY,
        )
        return result

@activity.defn
async def evaluate_rules(request: RuleEvalRequest) -> RuleEvalResult:
    """Evaluate rules - Temporal auto-retries on failure per policy"""
    engine = get_rule_engine()
    return await engine.evaluate(request.rules, request.context)

#3. Circuit Breaker Pattern

#3.1 Why Circuit Breakers Are Needed

Retries handle transient failures, but if a downstream service is persistently unavailable, unlimited retries only make things worse. A Circuit Breaker "opens" after detecting consecutive failures, returning degraded results directly.

Code

Circuit Breaker State Machine:

    +---------+     Consecutive    +--------+
    | CLOSED  | ---- failures --> |  OPEN   |
    | (Normal)|   > threshold     | (Tripped)|
    +---------+                   +--------+
         ^                            |
         |    Successful              | After timeout,
         |    requests                | allow probe
         |    > threshold             | requests
         |                            v
    +-------------+
    |  HALF-OPEN   |
    |  (Probing)   |
    +-------------+

Parameters:
  Failure threshold: 5 consecutive failures -> open breaker
  Open duration: 30 seconds before entering half-open
  Probe count: 3 requests allowed in half-open state
  Recovery threshold: All 3 succeed -> close breaker

#3.2 Circuit Breaker Implementation by Process

Code

onto-control (Spring Boot):
  Uses Resilience4j CircuitBreaker
  Config: application.yml
  ---
  resilience4j:
    circuitbreaker:
      instances:
        intelligence-service:
          slidingWindowSize: 10
          failureRateThreshold: 50
          waitDurationInOpenState: 30s
          permittedNumberOfCallsInHalfOpenState: 3
        data-service:
          slidingWindowSize: 20
          failureRateThreshold: 60
          waitDurationInOpenState: 20s

onto-data (Quarkus):
  Uses MicroProfile Fault Tolerance @CircuitBreaker
  ---
  @CircuitBreaker(
      requestVolumeThreshold = 10,
      failureRatio = 0.5,
      delay = 30000,     // 30s
      successThreshold = 3
  )
  public QueryResult queryObjects(QueryRequest request) { ... }

onto-intelligence (Python):
  Uses pybreaker library
  ---
  control_breaker = CircuitBreaker(
      fail_max=5,
      reset_timeout=30,
      state_storage=RedisCircuitBreakerStorage(
          state_key="cb:control-service",
          redis_client=redis_client,
      ),
  )

  @control_breaker
  async def call_control_service(request):
      async with grpc.aio.insecure_channel(CONTROL_ADDR) as channel:
          stub = ControlServiceStub(channel)
          return await stub.GetOntologySchema(request)

#3.3 Circuit Breaker Monitoring

All circuit breaker states are exposed to Prometheus for real-time Grafana dashboards:

Code

Prometheus Metrics:

# onto-control
resilience4j_circuitbreaker_state{name="intelligence-service"} 0|1|2
resilience4j_circuitbreaker_failure_rate{name="intelligence-service"} 23.5
resilience4j_circuitbreaker_calls_total{name="intelligence-service",kind="successful"} 9823
resilience4j_circuitbreaker_calls_total{name="intelligence-service",kind="failed"} 42

# onto-intelligence
circuit_breaker_state{service="control-service"} closed|open|half-open
circuit_breaker_failure_count{service="control-service"} 3

#4. Graceful Degradation

#4.1 Degradation Levels

The platform defines four degradation levels with different service capabilities:

Code

Degradation Level Matrix:

Level  Name            Trigger                          Impact
────────────────────────────────────────────────────────────────
L0     Fully Normal    All services healthy              None
L1     Partial         One downstream breaker open       Limited features
L2     Severe          2+ downstream services down       Read-only mode
L3     Minimal         Core process abnormal             Cache-only

Feature Availability by Level:
                        L0    L1    L2    L3
Ontology queries        Y     Y     Y     ~(cache)
Object instance CRUD    Y     Y     N     N
Action execution        Y     ~     N     N
Rule evaluation         Y     ~     N     N
Derived properties      Y     N     N     N
Dashboard display       Y     Y     ~     ~(snapshot)

#4.2 Degradation Strategy Implementation

Python

# onto-intelligence degraded service
class DegradedReasoningService:
    """Degraded implementation when onto-control is unavailable"""

    def __init__(self, cache: OntologyCache):
        self.cache = cache
        self.degradation_level = DegradationLevel.L0

    async def evaluate_rule(self, rule_id: str, context: dict) -> RuleResult:
        if self.degradation_level >= DegradationLevel.L2:
            # L2: Return cached last evaluation result
            cached = await self.cache.get_last_result(rule_id)
            if cached:
                return RuleResult(
                    value=cached.value,
                    confidence=0.5,  # Reduced confidence
                    source="cache",
                    stale=True,
                    cached_at=cached.timestamp,
                )
            raise ServiceDegradedError("Rule evaluation unavailable, no cache")

        if self.degradation_level == DegradationLevel.L1:
            # L1: Try evaluation with cached schema
            try:
                schema = await self.cache.get_schema(rule_id)
                return await self._evaluate_with_cached_schema(
                    rule_id, schema, context
                )
            except Exception:
                cached = await self.cache.get_last_result(rule_id)
                if cached:
                    return RuleResult(
                        value=cached.value, confidence=0.3, source="cache"
                    )
                raise

        # L0: Normal evaluation
        return await self._evaluate_normal(rule_id, context)

#4.3 Automatic Degradation and Recovery

Code

Automatic Degradation Controller (runs in each process):

Every 5 seconds ->
  +-- Check all circuit breaker states
  |   +-- All CLOSED -> L0
  |   +-- 1 OPEN -> L1
  |   +-- 2+ OPEN -> L2
  |   +-- Self health check failed -> L3
  |
  +-- Send degradation event to Kafka
  |   topic: platform.degradation.events
  |   {
  |     "service": "onto-intelligence",
  |     "level": "L1",
  |     "reason": "control-service circuit breaker open",
  |     "timestamp": "2026-03-24T10:30:00Z"
  |   }
  |
  +-- When all breakers recover to CLOSED ->
      Wait 60 seconds to confirm stability -> Restore L0
      Send recovery event

#5. Dead Letter Queues (DLQ)

#5.1 Message Consumption Failure Handling

When Kafka message consumption fails, messages cannot simply be discarded. The platform uses dead letter queues to prevent message loss:

Code

Kafka Message Processing Flow:

Normal Topic                Retry Topic                 Dead Letter Topic
platform.actions    ->   platform.actions.retry  ->  platform.actions.dlq
    |                        |                           |
    +-- Consume OK -> ACK    +-- Retry OK -> ACK         +-- Manual review
    +-- Consume fail ->      +-- 3 retries still fail -> |
    |   Send to retry topic  |   Send to dlq topic       +-- Ops alert
    +-- Deserialization  ->  +--
        fail: straight to dlq

Retry Strategy:
  Attempt 1: delay 1 second
  Attempt 2: delay 5 seconds
  Attempt 3: delay 30 seconds
  Still failing -> Enter DLQ

#5.2 DLQ Message Structure

JSON

{
  "original_topic": "platform.actions",
  "original_partition": 3,
  "original_offset": 12847,
  "original_key": "world-001:action-execute",
  "original_value": "<base64 encoded original message>",
  "error_history": [
    {
      "attempt": 1,
      "timestamp": "2026-03-24T10:30:01Z",
      "error": "NessieConflictException: commit conflict on branch main",
      "service": "onto-data",
      "trace_id": "abc-123"
    },
    {
      "attempt": 2,
      "timestamp": "2026-03-24T10:30:06Z",
      "error": "NessieConflictException: commit conflict on branch main",
      "service": "onto-data",
      "trace_id": "abc-124"
    },
    {
      "attempt": 3,
      "timestamp": "2026-03-24T10:30:36Z",
      "error": "NessieConflictException: commit conflict on branch main",
      "service": "onto-data",
      "trace_id": "abc-125"
    }
  ],
  "dlq_timestamp": "2026-03-24T10:30:36Z",
  "status": "PENDING_REVIEW"
}

#5.3 DLQ Management and Replay

Code

DLQ Management API (provided by onto-control):

GET    /api/v1/dlq/messages              -- List DLQ messages
GET    /api/v1/dlq/messages/{id}         -- View message details
POST   /api/v1/dlq/messages/{id}/replay  -- Replay single message
POST   /api/v1/dlq/messages/replay-all   -- Replay all messages
DELETE /api/v1/dlq/messages/{id}         -- Acknowledge and delete
GET    /api/v1/dlq/stats                 -- DLQ statistics

DLQ Alert Rules (Prometheus AlertManager):
  - DLQ count > 10 -> Warning alert
  - DLQ count > 100 -> Critical alert
  - DLQ backlog > 1 hour -> Warning
  - DLQ backlog > 24 hours -> Critical

#6. Temporal Workflow Error Handling

#6.1 Workflow-Level Fault Tolerance

Complex business logic in onto-intelligence (decision execution, batch rule evaluation) uses Temporal workflow orchestration. Temporal provides stronger fault tolerance than simple retries:

Code

Temporal Fault Tolerance Features:

1. Activity Auto-Retry
   +-- Configurable retry policy (attempts, interval, backoff)
   +-- Specify non-retryable exception types
   +-- Retries transparent to workflow code

2. Workflow Timeout Control
   +-- WorkflowExecutionTimeout: max duration for entire workflow
   +-- WorkflowRunTimeout: max time for single run
   +-- ActivityStartToCloseTimeout: max time per Activity

3. Heartbeat Detection
   +-- Long-running Activities must report heartbeats periodically
   +-- Heartbeat timeout -> Temporal considers Activity failed -> retry
   +-- Heartbeats can carry progress info, resume from checkpoint on retry

4. Compensation (Saga Pattern)
   +-- Each Activity can define a corresponding compensation Activity
   +-- On workflow failure, compensations execute in reverse order
   +-- Guarantees eventual consistency

#6.2 Saga Compensation Pattern

For operations spanning multiple services (like Action execution), the platform uses the Saga pattern for eventual consistency:

Python

@workflow.defn
class ExecuteActionWorkflow:
    """Action execution workflow - Saga pattern"""

    @workflow.run
    async def run(self, request: ActionExecuteRequest) -> ActionResult:
        compensations: list[Callable] = []

        try:
            # Step 1: Precondition rule evaluation
            pre_check = await workflow.execute_activity(
                evaluate_preconditions,
                request,
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=STANDARD_RETRY_POLICY,
            )
            if not pre_check.passed:
                return ActionResult.rejected(pre_check.reason)

            # Step 2: Lock related objects (optimistic locking)
            lock = await workflow.execute_activity(
                acquire_object_locks,
                request.object_ids,
                start_to_close_timeout=timedelta(seconds=10),
            )
            compensations.append(lambda: release_object_locks(lock))

            # Step 3: Apply data mutations
            mutation = await workflow.execute_activity(
                apply_mutations,
                request.mutations,
                start_to_close_timeout=timedelta(minutes=1),
                retry_policy=STANDARD_RETRY_POLICY,
            )
            compensations.append(lambda: rollback_mutations(mutation))

            # Step 4: Postcondition rule evaluation
            post_check = await workflow.execute_activity(
                evaluate_postconditions,
                request,
                start_to_close_timeout=timedelta(seconds=30),
            )

            if not post_check.passed:
                # Postcondition failed -> trigger compensation rollback
                raise PostconditionFailedError(post_check.reason)

            # Step 5: Commit and publish events
            await workflow.execute_activity(
                commit_and_publish,
                mutation,
                start_to_close_timeout=timedelta(seconds=15),
            )

            return ActionResult.success(mutation.summary)

        except Exception as e:
            # Execute compensations in reverse order
            for compensate in reversed(compensations):
                try:
                    await workflow.execute_activity(
                        compensate,
                        start_to_close_timeout=timedelta(seconds=30),
                    )
                except Exception as comp_error:
                    workflow.logger.error(
                        f"Compensation failed: {comp_error}, manual intervention needed"
                    )
            raise

#6.3 Temporal Workflow Monitoring

Code

Temporal Dashboard Key Metrics:

Workflow Statistics:
  +-- Active workflow count
  +-- Workflow completion rate (success / total)
  +-- Average workflow execution time
  +-- Workflow failure reason distribution

Activity Statistics:
  +-- Activity execution count (including retries)
  +-- Activity average latency
  +-- Activity failure rate
  +-- Retry count distribution

Alert Rules:
  +-- Workflow failure rate > 5% -> Warning
  +-- Workflow failure rate > 15% -> Critical
  +-- Active workflow count > 1000 -> Warning (possible backlog)
  +-- Activity avg latency > 30s -> Warning

#7. Error Recovery and Self-Healing

#7.1 Health Check Mechanism

Each process implements multi-layer health checks:

Code

Health Check Layers:

Layer 1: Liveness
  Checks: Process running, can respond to requests
  Frequency: Every 5 seconds
  Failure action: Docker/K8s restarts container
  Endpoints:
    onto-control:      GET /actuator/health/liveness
    onto-data:         GET /q/health/live
    onto-intelligence: GET /health/live

Layer 2: Readiness
  Checks: Dependencies available (database, Kafka, downstream services)
  Frequency: Every 10 seconds
  Failure action: Remove from load balancer, stop receiving new requests
  Endpoints:
    onto-control:      GET /actuator/health/readiness
    onto-data:         GET /q/health/ready
    onto-intelligence: GET /health/ready

Layer 3: Startup
  Checks: Initialization complete (schema loaded, caches warmed)
  Frequency: Every 2 seconds, max wait 120 seconds
  Failure action: Mark as startup failure, trigger alert

#7.2 Automatic Recovery Strategies

Code

Automatic Recovery Matrix:

Failure Type            Detection                Recovery Strategy
──────────────────────────────────────────────────────────────────
Process OOM             Docker health check      Auto-restart (max 3/hour)
DB connection lost      Connection pool heartbeat Auto-reconnect + backoff
Kafka consumer offline  Consumer group rebalance  Auto-rejoin
gRPC connection broken  Channel state monitoring  Auto-rebuild connection
Nessie conflict         ABORTED error code        Auto-merge retry
Schema cache expired    TTL + version number      Auto-refresh
Temporal Worker crash   Temporal Server detection  Auto-reassign tasks

#7.3 Manual Recovery Tools

When automatic recovery cannot resolve issues, operators can use these tools:

Code

Operations Commands (via onto-control Admin API):

# Force refresh all schema caches
POST /admin/cache/schema/refresh

# Reinitialize gRPC connections
POST /admin/grpc/reconnect?target=onto-intelligence

# Manually force circuit breaker state transition
POST /admin/circuit-breaker/intelligence-service/force-close

# Replay DLQ messages
POST /admin/dlq/replay?topic=platform.actions&count=10

# Export error report
GET /admin/errors/report?from=2026-03-24T00:00:00Z&to=2026-03-24T23:59:59Z

#8. Case Study: Handling a Cascading Failure

#8.1 Failure Scenario

Code

Timeline:

T+0s:   onto-data's Doris connection pool exhausted (100/100 connections)
T+1s:   New queries start queuing, timeout after 10s
T+10s:  onto-control's gRPC calls to onto-data start timing out
        Error: DEADLINE_EXCEEDED
T+15s:  onto-control data-service breaker: 3/10 failures
T+30s:  onto-control data-service breaker: 5/10 failures -> OPEN
T+30s:  Degradation controller detects breaker open -> Switch to L1
T+30s:  Dashboard queries serve from cache, Actions paused

T+45s:  Doris connection pool starts recovering (slow queries killed)
T+60s:  onto-data health check returns healthy
T+90s:  onto-control breaker enters HALF-OPEN, allows 3 probe requests
T+91s:  All 3 probes succeed
T+91s:  Breaker closes -> L0 recovery countdown 60s
T+151s: Degradation controller confirms stability -> Restore L0
T+151s: Paused Actions resume from pending queue

#8.2 Post-Mortem Analysis

Code

Root Cause:
  A user submitted a full-table scan query consuming 30 connections x 30 seconds
  Connection pool config max=100, normal load already using 75
  Remaining 25 connections exhausted within 3 seconds by new requests

Fixes Applied:
  1. Query timeout: max 10s per query (was 60s)
  2. Connection pool expansion: max=200, min-idle=50
  3. Query complexity check: queries estimating > 1M rows require approval
  4. Connection pool utilization alert: > 80% triggers Warning

Impact Statistics:
  Failure duration: 151 seconds
  Affected requests: ~240
  User-visible errors: 12 (rest absorbed by degradation and retry)
  Data loss: 0 (DLQ guarantees no message loss)
  Delayed Action executions: 37 (all completed within 5 min of recovery)

#Key Takeaways

Standardized Error Codes: Unified gRPC error codes + structured ErrorDetail ensure errors don't lose semantics when propagating across services.
Layered Retry: Different tech stacks use different retry implementations (Spring Retry / MicroProfile FT / Temporal), but follow the same retry decision tree.
Circuit Breaker Protection: Prevents cascading failures; each process independently manages circuit breaker state for its downstream dependencies.
Graceful Degradation: Four-level degradation model ensures the platform provides limited service during partial failures rather than complete unavailability.
Dead Letter Queue Safety Net: Failed message consumption doesn't discard messages; they enter DLQ for manual processing or replay.
Saga Compensation: Cross-service long transactions use Temporal workflows + Saga pattern for eventual consistency.
Self-Healing First: Most failures resolve through automatic restart, reconnection, and retry; manual intervention is the last resort.

#Next Article

The next article, S2-12 Configuration Management: YAML to Runtime Config Chain, will explain how three processes manage configuration — from local YAML files to Docker Compose environment variable injection, runtime config hot-reloading, and Feature Flag control.

tags: gRPC, error-handling, circuit-breaker, retry, dead-letter-queue, Temporal, saga, fault-tolerance, graceful-degradation