Event-Driven Architecture: Kafka's 7 Roles in the Platform

“

Series: S2 Architecture Overview · Article 6 | Level: Intermediate | Reading Time: 18 min

TL;DR

Kafka in coomia-dip is not just a message queue -- it simultaneously plays 7 distinct roles: CDC change capture, 3-topic audit logging, subscription routing, pipeline triggers, derived property cascade, reasoning triggers, and cross-Layer coordination. Each role has independent Topic naming conventions, partition strategies, and consumer group designs.
Events are uniformly encoded in Protobuf (not JSON), consistent with the platform's gRPC-first strategy. A single event's serialized size is approximately 1/3 to 1/5 of JSON, significantly reducing network and storage overhead in high-throughput scenarios.
Through the Topic naming convention {project_id}.{role}.{sub_type} and Consumer Group naming convention {Layer}-{service}-{role}, the platform achieves event stream discoverability, traceability, and zero-coordination across teams.

#1. Introduction: Message Queue or Event Backbone?

Many platforms treat Kafka as a "message queue" during technology selection -- an asynchronous communication pipe. This limited view restricts Kafka's use cases, typically employing it only for decoupling service-to-service calls.

In coomia-dip, Kafka's positioning is entirely different: it is the platform's Event Backbone. Every data change, every permission check, every computation request is connected through Kafka event streams.

Code

Traditional usage: Service A --message--> Kafka --message--> Service B
                   (async RPC substitute)

coomia-dip usage:
  ┌───────────┐    ┌─────────────────────────┐
  │ Control Layer   │    │        Kafka            │
  │ Control   │──> │ ┌─────────────────────┐ │
  │           │    │ │ Role 1: CDC         │ │──> Data Layer (Data)
  │           │    │ │ Role 2: Audit (x3)  │ │──> Audit Store
  │           │    │ │ Role 3: Subscription │ │──> External Apps
  │           │    │ │ Role 4: Pipeline     │ │──> Data Layer (Pipeline)
  │           │    │ │ Role 5: Derived Prop │ │──> Reasoning & Decision Layer (Reasoning)
  │           │    │ │ Role 6: Reasoning    │ │──> Reasoning & Decision Layer (Reasoning)
  │           │    │ │ Role 7: Cross-Layer  │ │──> All Layers
  │           │    │ └─────────────────────┘ │
  └───────────┘    └─────────────────────────┘

Let us dive into each role in detail.

#2. Role 1: CDC -- Change Data Capture

#2.1 What Is CDC

CDC (Change Data Capture) means capturing every create, update, and delete operation on data, converting them into an event stream. In coomia-dip, all data changes through OntologyRuntimeService generate CDC events.

#2.2 Topic Design

Code

Topic naming: {project_id}.cdc.data-change
Partition strategy: Hash by object_type
Partition count: 12 (default, adjustable per Project size)
Retention: 7 days
Compaction: delete (expire after retention)

#2.3 Event Schema

PROTOBUF

message DataChangeEvent {
  string event_id = 1;           // UUID, globally unique
  string project_id = 2;         // Owning Project
  string world_id = 3;           // Owning World
  string object_type = 4;        // Object type
  string object_id = 5;          // Object ID

  ChangeType change_type = 6;    // CREATE / UPDATE / DELETE
  string principal_id = 7;       // Operator

  // Change content
  map<string, Value> before = 8; // Before change (present for UPDATE/DELETE)
  map<string, Value> after = 9;  // After change (present for CREATE/UPDATE)
  repeated string changed_properties = 10; // List of changed properties

  google.protobuf.Timestamp timestamp = 11;
  string correlation_id = 12;    // Request correlation ID (for tracing)
}

enum ChangeType {
  CREATE = 0;
  UPDATE = 1;
  DELETE = 2;
}

#2.4 CDC Downstream Consumers

Code

CDC Topic consumers:

Consumer Group 1: data-Layer-materializer
  └── Data Layer: Sync changes to Iceberg tables (data lake persistence)

Consumer Group 2: data-Layer-search-indexer
  └── Data Layer: Update Doris inverted indexes and vector indexes

Consumer Group 3: intelligence-derived-property
  └── Reasoning & Decision Layer: Detect whether derived property recalculation is needed

Consumer Group 4: control-subscription-router
  └── Control Layer: Check for matching subscription rules

Consumer Group 5: intelligence-reasoning-trigger
  └── Reasoning & Decision Layer: Check whether rule engine should be triggered

A single CDC event is independently consumed by 5 different Consumer Groups -- this is the power of Kafka's publish-subscribe model. Each consumer processes at its own pace without interference.

#3. Role 2: Audit Logging (3 Topics)

#3.1 Three-Topic Design

We do not use a single audit Topic -- instead, we split into 3 topics based on audit event type and retention requirements:

Code

Topic 1: {project_id}.audit.data-access
  Content: Data read operations (GET, SEARCH, AGGREGATE)
  Retention: 90 days
  Partitions: 6
  Partition strategy: Hash by principal_id
  Typical QPS: 500-2000 (reads far outnumber writes)

Topic 2: {project_id}.audit.data-mutation
  Content: Data write operations (CREATE, UPDATE, DELETE)
  Retention: 365 days
  Partitions: 6
  Partition strategy: Hash by object_type
  Typical QPS: 50-200

Topic 3: {project_id}.audit.admin-operation
  Content: Admin operations (schema changes, permission changes, config changes)
  Retention: Permanent (compact mode)
  Partitions: 3
  Partition strategy: Hash by operation_type
  Typical QPS: 1-10

#3.2 Why Not Use a Single Topic

Code

Comparison:

Single Topic:
  ├── Pros: Simple management
  ├── Cons:
  │   ├── Read audit older than 90 days consumes storage (no longer valuable)
  │   ├── Compliance audits only need mutation + admin but must scan all events
  │   ├── Retention can only be set to longest (permanent), storage cost explodes
  │   └── Consumers must filter client-side, wasting network bandwidth
  └── Verdict: ❌

Three Topics:
  ├── Pros:
  │   ├── Each Topic has independent retention, optimal storage cost
  │   ├── Compliance audits directly read mutation + admin Topics, no filtering
  │   ├── Real-time monitoring reads only data-access Topic
  │   └── Consumers subscribe to exactly what they need, zero waste
  ├── Cons: Managing 3 Topics (but Auto-Provisioning creates them automatically)
  └── Verdict: ✅

#3.3 Audit Event Schema

PROTOBUF

message AuditEvent {
  string event_id = 1;
  string project_id = 2;
  string world_id = 3;

  // Subject info
  string principal_id = 4;
  string principal_type = 5;     // USER / SERVICE / SYSTEM
  repeated string principal_roles = 6;
  string source_ip = 7;

  // Operation info
  string operation = 8;          // GET_OBJECT / SEARCH / CREATE / UPDATE / DELETE / ...
  string object_type = 9;
  string object_id = 10;
  repeated string accessed_properties = 11;

  // Result
  AuditResult result = 12;       // SUCCESS / DENIED / ERROR
  string denial_reason = 13;     // If DENIED, explains why
  int32 result_count = 14;       // Number of results returned

  // Context
  google.protobuf.Timestamp timestamp = 15;
  string correlation_id = 16;
  int64 duration_ms = 17;        // Operation duration
}

#4. Role 3: Subscription Routing

#4.1 What Is Subscription Routing

Subscription routing allows external applications and internal services to subscribe to specific data changes. For example:

"When Equipment A's temperature exceeds 80 degrees C, notify the O&M system"
"When order status changes to SHIPPED, trigger logistics tracking"
"When inventory falls below the safety line, notify the procurement system"

#4.2 Topic Design

Code

Topic: {project_id}.subscription.routed
Partitions: 6
Partition strategy: Hash by subscription_id
Retention: 24 hours (consumed immediately after routing)

#4.3 Subscription Rule Engine

Code

Subscription rule definition:

{
  "subscription_id": "sub-temp-alert-001",
  "name": "Equipment Temperature Alert",
  "object_type": "Equipment",
  "filter": {
    "property": "temperature",
    "operator": "GREATER_THAN",
    "value": 80.0
  },
  "trigger_on": ["UPDATE"],
  "notify": {
    "type": "WEBHOOK",
    "url": "https://ops-system.internal/api/alerts",
    "headers": { "Authorization": "Bearer {{token}}" }
  },
  "rate_limit": {
    "max_per_minute": 10,
    "cooldown_seconds": 300
  }
}

#4.4 Routing Flow

Code

CDC event arrives → SubscriptionRouter (Consumer Group: control-subscription-router)
  │
  ├── Step 1: Load all matching subscription rules
  │   └── Filter by object_type and change_type
  │
  ├── Step 2: Evaluate each rule's filter condition
  │   └── Check if changed_properties includes the subscribed property
  │   └── Evaluate condition expression (temperature > 80.0)
  │
  ├── Step 3: Rate limit check
  │   └── Check rate_limit (times triggered in last minute for this subscription)
  │   └── Check cooldown (whether last trigger is within cooldown period)
  │
  ├── Step 4: Generate routed event
  │   └── Write to {project_id}.subscription.routed Topic
  │
  └── Step 5: Execute notification
      └── Webhook / gRPC / internal event
      └── Async execution, retry on failure (exponential backoff, max 3 attempts)

#5. Role 4: Pipeline Triggers

#5.1 Scenario

Data pipelines are typically triggered on schedule by DolphinScheduler. But some scenarios require event-driven pipeline execution -- automatically triggering downstream ETL pipelines when upstream data changes.

#5.2 Topic Design

Code

Topic: {project_id}.pipeline.trigger
Partitions: 6
Partition strategy: Hash by pipeline_id
Retention: 48 hours

#5.3 Trigger Rules

PROTOBUF

message PipelineTriggerEvent {
  string event_id = 1;
  string project_id = 2;
  string pipeline_id = 3;        // Pipeline to trigger
  string trigger_type = 4;       // CDC / SCHEDULE / MANUAL / DEPENDENCY

  // Info when triggered by CDC
  string source_object_type = 5;
  int32 change_count = 6;        // Accumulated change count
  google.protobuf.Timestamp window_start = 7;
  google.protobuf.Timestamp window_end = 8;

  // Trigger parameters
  map<string, string> parameters = 9;

  google.protobuf.Timestamp timestamp = 10;
}

#5.4 Micro-Batch Trigger Strategy

To avoid triggering a pipeline for every CDC event (too frequent), we use a micro-batch strategy:

Code

Micro-batch trigger strategy:

PipelineTriggerAggregator:
  ├── Receives CDC events
  ├── Groups by object_type + pipeline_id
  ├── Accumulates changes until trigger condition is met:
  │   ├── Condition 1: Accumulated changes >= threshold (default 100)
  │   ├── Condition 2: Time window >= interval (default 5 minutes)
  │   └── Whichever condition is met first
  ├── Generates PipelineTriggerEvent
  └── Writes to pipeline.trigger Topic

Example:
  Pipeline: "equipment-data-etl"
  Trigger config: threshold=50, interval=300s

  09:00:00 - Received 20 Equipment changes → accumulate
  09:02:00 - Received 15 more → accumulated = 35
  09:03:30 - Received 18 more → accumulated = 53 > 50 → TRIGGER!
  09:03:30 - Send PipelineTriggerEvent(change_count=53)
  09:03:31 - Data Layer PipelineExecutor consumes event, submits DolphinScheduler task

#6. Role 5: Derived Property Cascade

#6.1 Scenario

Derived properties depend on other properties for their values. When a dependency changes, the derived property must be recalculated. If derived property A depends on derived property B, and B depends on property C, then a change to C must cascade to trigger recalculation of both B and A.

#6.2 Topic Design

Code

Topic: {project_id}.derived-property.cascade
Partitions: 6
Partition strategy: Hash by object_id (ensures cascade events for the same object are processed in order)
Retention: 24 hours

#6.3 Cascade Event Schema

PROTOBUF

message DerivedPropertyCascadeEvent {
  string event_id = 1;
  string project_id = 2;
  string world_id = 3;
  string object_type = 4;
  string object_id = 5;

  // Trigger source
  string trigger_property = 6;       // The changed source property
  Value trigger_old_value = 7;
  Value trigger_new_value = 8;

  // Derived properties needing recalculation
  repeated DerivedPropertyTarget targets = 9;

  // Cascade depth (prevents infinite loops)
  int32 cascade_depth = 10;          // Current cascade depth
  int32 max_cascade_depth = 11;      // Maximum allowed depth (default 10)

  google.protobuf.Timestamp timestamp = 12;
  string correlation_id = 13;
}

message DerivedPropertyTarget {
  string property_name = 1;
  string computation_strategy = 2;   // SQL / EXPRESSION / FUNCTION / ...
  repeated string dependency_properties = 3;
}

#6.4 Cascade Processing Flow

Code

CDC event: Equipment.temperature changed from 75 to 85

DerivedPropertyCascadeDetector:
  │
  ├── Step 1: Query dependency graph (DAG)
  │   temperature ← health_score (derived)
  │   health_score ← risk_level (derived)
  │   risk_level ← (no downstream dependencies)
  │
  ├── Step 2: Topological sort
  │   Computation order: health_score → risk_level
  │
  ├── Step 3: Send cascade event (depth=1)
  │   Event 1: Recalculate health_score
  │     trigger: temperature
  │     cascade_depth: 1
  │
  └── Step 4: After health_score computation completes
      │
      └── Send cascade event (depth=2)
          Event 2: Recalculate risk_level
            trigger: health_score
            cascade_depth: 2

Safety mechanisms:
  - cascade_depth > max_cascade_depth → Stop cascade, log alert
  - Circular dependency detection: Detected during DAG construction,
    rejects creation of circular derived properties
  - Each cascade event contains correlation_id for full chain tracing

#7. Role 6: Reasoning Triggers

#7.1 Scenario

Reasoning & Decision Layer's reasoning engine (rule engine + AI reasoning) needs to trigger automatically under specific conditions. For example:

When a device's risk_level becomes HIGH, trigger fault diagnosis reasoning
When inventory drops below safety line, trigger replenishment recommendation
When a financial transaction amount is abnormal, trigger anti-fraud rule chain

#7.2 Topic Design

Code

Topic: {project_id}.reasoning.trigger
Partitions: 6
Partition strategy: Hash by reasoning_type
Retention: 48 hours

#7.3 Reasoning Trigger Event

PROTOBUF

message ReasoningTriggerEvent {
  string event_id = 1;
  string project_id = 2;
  string world_id = 3;

  // Trigger object
  string object_type = 4;
  string object_id = 5;

  // Reasoning configuration
  string reasoning_type = 6;         // RULE_CHAIN / AI_INFERENCE / HYBRID
  string reasoning_config_id = 7;    // Reasoning config ID

  // Trigger context
  map<string, Value> trigger_context = 8;  // Context passed to reasoning engine
  string trigger_source = 9;         // CDC / SCHEDULE / MANUAL / CASCADE

  // Priority
  Priority priority = 10;           // LOW / NORMAL / HIGH / CRITICAL

  google.protobuf.Timestamp timestamp = 11;
  string correlation_id = 12;
}

#7.4 Relationship Between Reasoning Triggers and CDC

Code

CDC event → DerivedPropertyCascade → Derived property updated
                                         │
                                         ▼
                                   ReasoningTrigger
                                   (based on updated derived property value)

Example:
  1. temperature changes from 75 to 85           (CDC)
  2. health_score recalculated from 0.9 to 0.6   (derived property cascade)
  3. risk_level recalculated from LOW to HIGH     (derived property cascade)
  4. Trigger fault diagnosis reasoning            (reasoning trigger)
     → Rule engine evaluates 15 fault rules
     → Output: "Suspected bearing overheating, recommend checking lubrication system"
     → Creates Action: MaintenanceWorkOrder

#8. Role 7: Cross-Layer Coordination

#8.1 Scenario

coomia-dip has 8 Layers (5 independently deployed units). Their primary communication uses gRPC (request-response), but certain scenarios require asynchronous coordination:

Schema changes need to notify all Layers to refresh caches
New World creation requires multiple Layers to initialize resources
System maintenance needs to notify all Layers to enter read-only mode

#8.2 Topic Design

Code

Topic: platform.coordination.{event_type}
Partitions: 3 (global Topic, not partitioned by Project)
Retention: 24 hours

#8.3 Coordination Event Types

Code

Event types:

1. schema.changed
   Trigger: Schema changes (ObjectType added/modified/deleted)
   Consumers: All Layers' Schema caches
   Effect: Invalidate local caches, reload

2. world.created / world.deleted
   Trigger: World lifecycle changes
   Consumers: Data Layer (Doris connection pool), Reasoning & Decision Layer (compute context)
   Effect: Initialize/cleanup World-related resources

3. system.maintenance.enter / system.maintenance.exit
   Trigger: System maintenance window
   Consumers: All Layers
   Effect: Enter/exit read-only mode

4. config.updated
   Trigger: Platform configuration changes (rate limits, feature flags)
   Consumers: All Layers
   Effect: Hot-update configuration

5. permission.policy.changed
   Trigger: RBAC/ABAC policy changes
   Consumers: Control Layer (PolicyEngine), all services with caches
   Effect: Recompile permission decision trees

#8.4 Idempotency of Coordination Events

Cross-Layer coordination events must be idempotent -- even if the same event is consumed multiple times, the effect occurs only once:

Code

Idempotency guarantee:

Each coordination event contains:
  - event_id (UUID): Globally unique identifier
  - event_version (int64): Monotonically increasing version number

Consumer side:
  - Maintains processed_events set (Redis SET, TTL = 24h)
  - Before consuming: if event_id in processed_events → skip
  - After consuming: processed_events.add(event_id)

Version check:
  - Maintains latest_version (Redis KV)
  - If event_version <= latest_version → skip (stale event)
  - If event_version > latest_version + 1 → missing intermediate events, trigger full sync

#9. Topic Naming Conventions

#9.1 Complete Naming Convention

Code

Naming pattern: {scope}.{role}.{sub_type}

scope:
  - {project_id}: Project-level events (most events)
  - platform: Platform-level global events

role:
  - cdc: Change data capture
  - audit: Audit logging
  - subscription: Subscription routing
  - pipeline: Pipeline triggers
  - derived-property: Derived property cascade
  - reasoning: Reasoning triggers
  - coordination: Cross-Layer coordination

sub_type:
  - Specific event subtype

Complete examples:
  proj_001.cdc.data-change
  proj_001.audit.data-access
  proj_001.audit.data-mutation
  proj_001.audit.admin-operation
  proj_001.subscription.routed
  proj_001.pipeline.trigger
  proj_001.derived-property.cascade
  proj_001.reasoning.trigger
  platform.coordination.schema.changed
  platform.coordination.world.created
  platform.coordination.system.maintenance.enter

#9.2 Consumer Group Naming Convention

Code

Naming pattern: {Layer}-{service}-{role}

Examples:
  control-ontology-runtime-cdc          (Control Layer consuming CDC)
  control-subscription-router-cdc       (Control Layer subscription router)
  data-materializer-cdc                 (Data Layer materializer consuming CDC)
  data-search-indexer-cdc               (Data Layer search indexer consuming CDC)
  data-pipeline-executor-trigger        (Data Layer pipeline executor)
  intelligence-derived-property-cascade  (Reasoning & Decision Layer derived property cascade)
  intelligence-reasoning-engine-trigger  (Reasoning & Decision Layer reasoning engine)
  intelligence-rule-engine-trigger       (Reasoning & Decision Layer rule engine)

#10. Event Schema Management

#10.1 Why Protobuf Instead of JSON

Code

JSON vs Protobuf comparison:

Serialization size:
  JSON DataChangeEvent:     ~800 bytes
  Protobuf DataChangeEvent: ~200 bytes
  Compression ratio: 4:1

Serialization/Deserialization performance:
  JSON (Jackson):    ~5us / ~8us
  Protobuf:          ~1us / ~1.5us
  Improvement: 5x

Schema evolution:
  JSON: No enforced schema, backward compatibility by convention
  Protobuf: Strongly typed schema, compile-time checks, guaranteed backward compatibility
    - Add field: ✅ (old consumers ignore)
    - Remove field: ✅ (mark as reserved)
    - Change type: ❌ (compile error)

Cross-language:
  JSON: Each language needs hand-written or generated DTOs
  Protobuf: One .proto file generates Java + Python + TypeScript

#10.2 Protobuf Schema Management Strategy

Code

Directory structure:

proto/
├── events/
│   ├── cdc.proto                    # CDC events
│   ├── audit.proto                  # Audit events
│   ├── subscription.proto           # Subscription routing events
│   ├── pipeline.proto               # Pipeline trigger events
│   ├── derived_property.proto       # Derived property cascade events
│   ├── reasoning.proto              # Reasoning trigger events
│   └── coordination.proto           # Cross-Layer coordination events
├── common/
│   ├── value.proto                  # Common value types
│   └── context.proto                # WorldContext definition
└── buf.yaml                         # Buf Schema Registry config

Version management:
  - Each .proto file has a package version: onto.events.v1
  - Backward-compatible changes: Add fields within same package
  - Breaking changes: Create new package: onto.events.v2
  - Consumers support both v1 and v2, routing via event header version identifier

#10.3 Event Envelope

All Kafka events use a unified envelope format:

PROTOBUF

message EventEnvelope {
  // Envelope metadata (in Kafka Headers)
  string event_type = 1;       // "cdc.data-change" / "audit.data-access" / ...
  string schema_version = 2;   // "v1" / "v2"
  string content_type = 3;     // "application/x-protobuf"
  string correlation_id = 4;   // Request trace ID
  string source_plane = 5;     // "control" / "data" / "intelligence"

  // Event body (in Kafka Value, Protobuf encoded)
  bytes payload = 6;           // Actual event content (deserialize by event_type)
}

The envelope's event_type and schema_version are stored in Kafka Headers, allowing consumers to determine routing strategy without deserializing the payload.

#11. Failure Handling and Reliability

#11.1 Dead Letter Topic

When event processing fails beyond the retry limit, events are routed to a Dead Letter Topic:

Code

Dead Letter Topic: {original_topic}.dlq

Processing flow:
  Event → Consumer → Processing fails
    → Retry 1 (after 1s)
    → Retry 2 (after 5s)
    → Retry 3 (after 30s)
    → Route to DLQ Topic
    → Alert notification

DLQ event format:
  Original event + error info + retry history + original Topic + original Partition + Offset

DLQ handling:
  1. Automatic: Scheduled task checks DLQ hourly, attempts reprocessing
  2. Manual: Ops team reviews DLQ, fixes issues, replays events

#11.2 Backpressure Handling

Code

Backpressure strategy:

When consumer processing speed < producer send speed:

Level 1: Consumer Lag Monitoring
  if consumer_lag > threshold_warning (1000 events):
    → Alert notification
    → Scale up Consumer instance count

Level 2: Dynamic Partition Rebalancing
  if consumer_lag > threshold_critical (10000 events):
    → Trigger Consumer Group rebalance
    → Assign more Partitions to idle Consumers

Level 3: Producer Throttling
  if consumer_lag > threshold_emergency (100000 events):
    → OntologyRuntimeService reduces write rate
    → Returns 429 Too Many Requests
    → Wait for consumers to catch up

#11.3 Event Ordering Guarantees

Code

Ordering guarantee levels:

Level 1: Strict order within the same Partition
  Guarantee: ✅ Native Kafka guarantee
  Application: CDC events for the same object in the same Partition (hash by object_id)

Level 2: Causal ordering
  Guarantee: ✅ Via correlation_id and cascade_depth
  Application: Derived property cascades processed in depth order

Level 3: Global ordering
  Guarantee: ❌ Not guaranteed, not needed
  Reason: Changes to different objects are inherently parallel;
          enforcing global order would severely limit throughput

#12. Performance Benchmarks and Tuning

#12.1 Throughput Benchmarks

Code

Test environment: 3 Brokers, 4 CPU / 16GB RAM per Broker

CDC Topic (12 partitions):
  Producer throughput: 50,000 events/s
  Consumer throughput: 30,000 events/s (single Consumer Group)
  End-to-end latency P99: 12ms

Audit Topic (6 partitions):
  Producer throughput: 10,000 events/s
  Consumer throughput: 20,000 events/s (audit store writes are fast)
  End-to-end latency P99: 5ms

Derived Property Cascade (6 partitions):
  Producer throughput: 5,000 events/s
  Consumer throughput: 2,000 events/s (computation is the bottleneck)
  End-to-end latency P99: 50ms (includes computation time)

#12.2 Key Tuning Parameters

Code

Producer configuration:
  acks=1                    # Use acks=all for audit
  batch.size=32768           # 32KB batch
  linger.ms=5                # 5ms batch wait
  compression.type=snappy    # Snappy compression
  max.in.flight.per.connection=5

Consumer configuration:
  max.poll.records=500       # Max 500 records per poll
  max.poll.interval.ms=300000 # 5 minute timeout
  session.timeout.ms=30000   # 30 second session timeout
  auto.offset.reset=latest   # latest for CDC, earliest for Audit
  enable.auto.commit=false   # Manual offset commit

#13. Comparison with Palantir Foundry

Dimension	Palantir Foundry	coomia-dip
Event system	Foundry internal event bus (closed source)	Kafka (open source)
Event format	JSON / Avro	Protobuf
CDC	Dataset Transaction Log	CDC Topic per Project
Audit	Audit Service (centralized)	3-Topic tiered audit
Derived properties	TypeScript OSDK triggered	Kafka cascade events
Subscriptions	Webhook + Object Set Subscription	Rule engine + routing Topic
Cross-service coordination	Internal RPC	Kafka coordination Topic

#Key Takeaways

Kafka is not just a message queue -- it is the platform's event backbone. Through the division into 7 roles, coomia-dip elevates Kafka from a simple async communication pipe to a platform-level event orchestration system. Each role has independent Topics, partition strategies, and consumer groups that scale independently without interference.
Protobuf event schemas are the foundation for cross-language collaboration. Java (Control Layer + Data Layer) and Python (Reasoning & Decision Layer + Agent Runtime Layer) share the same .proto definitions, with compile-time type checking ensuring event producers and consumers always remain compatible. Compared to JSON, Protobuf has a 4-5x advantage in serialization size and performance -- critical in high-throughput scenarios.
Cascade event depth control is the safety valve. Both derived property cascades and reasoning triggers can form long chains (A → B → C → ...). The cascade_depth field and max_cascade_depth limit ensure cascades do not spiral out of control. Circular dependencies are detected and rejected during DAG construction, not discovered as infinite loops at runtime.

“

Next Article Preview: [S2-07] Computation Strategy Routing: 7 Compute Mode Priority Scheduling -- a deep dive into how ComputationCoordinator makes optimal choices among SQL, Expression, Reducer, Function, Cached, Materialized, and Real-time computation modes.

Tags: #event-driven #kafka #cdc #audit #subscription #pipeline #derived-property #reasoning #protobuf #coomia-dip