Source Code Reading: DerivedPropertyService — Dependency DAG and Cascade

“

Series: S9 Source Code Reading · Article 15 | Level: Advanced | Reading Time: 25 min

#TL;DR

DerivedPropertyService is the derived property engine within coomia-dip's Reasoning & Decision layer (Reasoning & Decision Layer), implementing virtual computed properties on ontology instances. It supports four computation modes -- FunctionRuntime functions, SQL queries, arithmetic expressions, and Reducer aggregations -- along with three storage strategies (VIRTUAL/CACHED/MATERIALIZED). Through the DerivedPropertyEngine core engine, it manages property definitions, dependency DAGs, cache invalidation, cascade recomputation, and the three-axis version model (data version x schema version x compute logic version). This article dissects the four computation mode priority rules in the Proto contract, the Servicer layer's Pydantic-Proto bidirectional conversion, cache statistics observability, and the FEAT-012 Reducer aggregation's multi-hop link traversal mechanism.

#Table of Contents

1. Overall Architecture: Engine + Servicer Layering
2. Four Computation Modes and Priority Rules
3. Three Storage Strategies: VIRTUAL / CACHED / MATERIALIZED
4. Three-Axis Version Model: Data x Schema x Compute Logic
5. DerivedPropertyServicer: Request Model Design
6. Property Definition: Four Computation Mode Build Flow
7. Property Computation: Cache Strategy and WorldContext
8. Reducer Aggregation: Multi-Hop Link Traversal
9. Cache Management: Invalidation and Statistics
10. BranchBindingPolicy: Version Strategy on Branches
11. Key Takeaways

#1. Overall Architecture: Engine + Servicer Layering

DerivedPropertyService code is distributed across three layers:

intelligence-Layer/src/reasoning_decision_plane/
├── derived_property/
│   ├── engine.py                 # DerivedPropertyEngine -- core engine
│   └── models.py                 # Domain models (Pydantic v2)
├── api/grpc/
│   └── derived_property_servicer.py  # gRPC Servicer adapter
proto/plane_d/
└── derived_property.proto        # 463-line core contract

The Servicer follows the same "injectable with defaults" pattern as FunctionRuntime, accepting an optional DerivedPropertyEngine through constructor injection.

#2. Four Computation Modes and Priority Rules

DerivedPropertyDefinition supports four mutually exclusive computation modes carried by different fields:

message DerivedPropertyDefinition {
  string function_id = 6;              // Mode 1: FunctionRuntime
  SqlComputation sql_computation = 15;       // Mode 2: SQL query
  ExpressionComputation expression_computation = 16;  // Mode 3: Arithmetic expression
  ReducerDefinition reducer = 17;            // Mode 4: Reducer aggregation
}

Priority rule explicitly defined in Proto comments: sql_computation > expression_computation > reducer > function_id. This means if both sql_computation and function_id are set, the engine uses SQL computation. This design enables incremental migration -- implement quickly with function_id, then add a more efficient SQL or expression computation later without removing the old field.

#3. Three Storage Strategies: VIRTUAL / CACHED / MATERIALIZED

enum StorageMode {
  STORAGE_MODE_VIRTUAL = 1;           // Computed on every query
  STORAGE_MODE_CACHED = 2;            // Cached in Redis
  STORAGE_MODE_MATERIALIZED = 3;      // Stored in Iceberg
}

CACHED mode uses CacheConfig with invalidation_events -- event patterns that trigger cache invalidation, e.g., "entity.Employee.updated" clears this property's cache when any Employee instance is updated.

MATERIALIZED mode uses MaterializationConfig with a debounce field -- a key optimization that delays materialization execution when many change events arrive in quick succession, avoiding redundant computation.

#4. Three-Axis Version Model: Data x Schema x Compute Logic

DerivedProperty introduces a "three-axis version model" for deterministic, reproducible computation:

message DerivedPropertyDefinition {
  optional string compute_version = 13;
  BranchBindingPolicy branch_binding_policy = 14;
}

The three axes are:

1. Data version: Determined by WorldContext.commit_hash (Nessie commit)
2. Schema version: Version of the ontology type definition
3. Compute logic version: The compute_version field -- when set to a specific version (e.g., "1.2.0"), function_id resolves to that specific function version

When compute_version is empty or "latest", the latest function version is used -- suitable for development. In production, locking to a specific version ensures identical inputs always produce identical outputs.

#5. DerivedPropertyServicer: Request Model Design

The Servicer defines dedicated request models for each computation mode. The DefineDerivedPropertyRequest merges all modes into a single request, using None values to distinguish which computation mode is active:

class DefineDerivedPropertyRequest(BaseModel):
    property_id: str = ""
    property_name: str = ""
    ontology_type: str = ""
    return_type: str = "string"
    storage_mode: str = "VIRTUAL"
    function_id: str = ""
    dependencies: list[str] = Field(default_factory=list)
    sql_computation: SqlComputationRequest | None = None
    expression_computation: ExpressionComputationRequest | None = None
    reducer: ReducerDefinitionRequest | None = None

Each computation mode's request model has its own specific fields. For SQL computation: connection_id, sql_template, params, result_column. For expressions: just expression and return_type. For reducers: the full link traversal configuration.

#6. Property Definition: Four Computation Mode Build Flow

The define_derived_property method demonstrates each mode's build pipeline. Each has an "existence check" -- not just whether the request object exists, but whether its key fields are non-empty. Empty connection_id or expression values are treated as "this computation mode is not configured."

# SQL computation: check connection_id is non-empty
if request.sql_computation and request.sql_computation.connection_id:
    sql_comp = SqlComputation(
        connection_id=request.sql_computation.connection_id,
        sql_template=request.sql_computation.sql_template,
        params=[SqlTemplateParam(...) for p in request.sql_computation.params],
        result_column=request.sql_computation.result_column,
    )

# Expression computation: check expression is non-empty
if request.expression_computation and request.expression_computation.expression:
    expr_comp = ExpressionComputation(
        expression=request.expression_computation.expression,
        return_type=request.expression_computation.return_type,
    )

Reducer building is more complex, involving enum parsing with fallback and link path step conversion with auto-generated reducer_id when not provided.

#7. Property Computation: Cache Strategy and WorldContext

The compute_derived_property method processes computation requests with three cache control switches: force_recompute (ignore cache), skip_cache (don't write results to cache), and include_metadata (return computation metadata).

The response's PropertyValue includes origin information through PropertyOrigin (COMPUTED/CACHED/MATERIALIZED) combined with ttl_remaining_seconds, allowing clients to judge cached value freshness.

Computation metrics provide comprehensive observability:

message ComputeMetrics {
  int64 total_objects = 1;
  double computation_time_ms = 2;
  int32 cache_hits = 3;
  int32 cache_misses = 4;
  int32 errors = 5;
}

#8. Reducer Aggregation: Multi-Hop Link Traversal

FEAT-012's Reducer is the most complex of the four computation modes:

message ReducerDefinition {
  string source_object_type = 3;
  repeated LinkPathStep link_path = 4;     // Max 3 hops
  string target_property = 5;
  AggregationFunction aggregation = 6;
  string filter_expression = 7;
  ReducerExecutionStrategy strategy = 8;
}

Nine aggregation functions cover common statistical needs: SUM, COUNT, AVG, MIN, MAX, COLLECT (gather into list), FIRST, LAST, COUNT_DISTINCT. FIRST/LAST require the order_by_property field to specify sort criteria.

Three execution strategies balance freshness and performance:

• VIRTUAL: Computed on every query (default)
• MATERIALIZED: Pre-computed via Pipeline
• HYBRID: Cached with TTL

The 3-hop limit on link_path prevents explosive graph traversal growth while still supporting common business scenarios like "department -> team -> employee".

#9. Cache Management: Invalidation and Statistics

Cache invalidation supports three granularity levels: when both property_id and object_ids are empty, all caches are cleared; when only property_id is provided, all object caches for that property are cleared; when both are provided, specific property caches for specific objects are precisely invalidated.

Proto-level cache statistics include per-property breakdowns with evictions reporting -- if this value grows continuously, cache capacity needs expansion.

#10. BranchBindingPolicy: Version Strategy on Branches

When Nessie branches are created, how should derived property compute logic versioning behave?

enum BranchBindingPolicy {
  BRANCH_BINDING_POLICY_PIN_AT_CREATION = 1;  // Default: freeze compute version
  BRANCH_BINDING_POLICY_FOLLOW_LATEST = 2;    // Dev mode: always use latest
}

PIN_AT_CREATION (default) is critical for A/B testing -- two branches use different data but the same compute logic, ensuring fair comparison. FOLLOW_LATEST suits development/sandbox environments where developers want to see the effects of latest function changes.

#11. Key Takeaways

1. Four computation modes cover the full computation spectrum from simple expressions to complex functions, with priority rules enabling incremental migration.
2. Three storage strategies balance real-time accuracy and performance: VIRTUAL for low-frequency high-precision, CACHED for high-frequency moderate-latency, MATERIALIZED for large-scale batch.
3. Three-axis version model (data x schema x compute logic) ensures computation determinism and reproducibility.
4. Reducer's multi-hop links with a 3-hop limit balance functionality and safety.
5. BranchBindingPolicy provides flexible choice between A/B testing rigor and development convenience.
6. Cache observability through hit_rate and evictions metrics helps ops teams optimize cache configuration.

#Next Article

S9-16: PipelineService -- DSL to DAG Compilation, where we dive into Pipeline & Orchestration Layer's pipeline engine to understand how declarative Pipeline DSL compiles into executable DAGs.

Tags: #coomia-dip #source-code-reading #derived-property #dependency-dag #cascade #grpc #Layer-d