API Design Philosophy: Governing 63 Protos and 59 REST Endpoints

“

Series: S2 Architecture Overview · Article 8 | Level: Intermediate | Reading Time: 18 min

#TL;DR

The coomia-dip platform adopts a Contract-First development model: all internal communication is defined through 63 Protobuf files, while external clients access the system through 59 REST endpoints, with an API Gateway handling automatic conversion between the two layers.
The 63 proto files are organized by 8 Layers, each Layer maintaining its own .proto files, unified through a consistent error code system and versioning strategy across all Layers.
The Python SDK auto-generates 38 gRPC clients from proto files, achieving a "change proto once, SDK/Server/docs sync automatically" development experience.

#Introduction: APIs Are the Nervous System of Distributed Systems

In any distributed platform, APIs are not merely "interfaces" — they are the system's contract layer, collaboration protocol, and evolution boundary. A well-designed API system enables teams to develop, deploy, and evolve independently; a chaotic API system turns every change into a nightmare.

The coomia-dip platform faces particularly challenging API governance requirements:

Code

8 Layers x multiple tech stacks x 3 languages (Java/Python/TypeScript)
= massive cross-boundary calls requiring precise definition

Our choice: Protobuf First, REST Second. All core logic communicates via gRPC; REST is merely a "translation layer" for external clients. This article details how this API system is designed, organized, and governed.

#1. Contract-First Development: Proto Files Are the Single Source of Truth

#1.1 What Is Contract-First?

Traditional development follows Code-First: write code first, then generate API documentation (like Swagger) from code. While seemingly efficient, this approach has a fatal flaw — interface definitions are scattered across code with no single source of truth.

Contract-First reverses this: write the interface definition (proto files) first, then generate code from the definition.

Code

Contract-First Flow:

  ┌──────────────────┐
  │   .proto files    │  <-- Single Source of Truth
  └────────┬─────────┘
           │
    ┌──────┴────┬────────────┬────────────┐
    v           v            v            v
  Java       Python     TypeScript      Docs
  Server     Client     Client        (auto-gen)
  Stubs      Stubs      Stubs

#1.2 Why Protobuf Over OpenAPI?

Dimension	Protobuf/gRPC	OpenAPI/REST
Serialization	Binary, 5-10x faster than JSON	JSON text
Type safety	Strong typing, compile-time checks	Runtime validation
Streaming	Native bidirectional streaming	SSE/WebSocket requires extra work
Code generation	First-class citizen, official support	Fragmented toolchain
Backward compat	Field numbering inherently supports it	Manual management
Browser friendly	Not friendly, needs proxy	Native support

The conclusion is clear: gRPC internally (performance + type safety), REST externally (compatibility + browser friendliness).

#1.3 Proto File Development Workflow

Code

1. Design Phase:
   +-- Define interface semantics in design docs
   +-- Write .proto files
   +-- Team review (check backward compatibility)

2. Code Generation Phase:
   +-- protoc generates Java/Python stubs
   +-- grpc-gateway generates REST reverse proxy
   +-- Auto-update SDK clients

3. Implementation Phase:
   +-- Server-side implements Service interfaces
   +-- Write unit tests
   +-- Integration tests verify cross-Layer calls

4. Release Phase:
   +-- Increment version number
   +-- Update CHANGELOG
   +-- Publish new SDK version

#2. Organizing 63 Proto Files

#2.1 Package-per-Layer Structure

Our proto files are organized along the boundaries of 8 Layers, each Layer maintaining its own proto package:

Code

protos/
+-- plane_b/                          # Control Layer (Control Layer)
|   +-- ontology/
|   |   +-- object_type.proto         # ObjectType CRUD
|   |   +-- link_type.proto           # LinkType CRUD
|   |   +-- property_type.proto       # PropertyType management
|   |   +-- action_type.proto         # ActionType definitions
|   |   +-- ontology_version.proto    # Ontology versioning
|   +-- world/
|   |   +-- world.proto               # World lifecycle
|   |   +-- world_branch.proto        # World branching
|   |   +-- world_merge.proto         # World merge operations
|   +-- auth/
|   |   +-- authentication.proto      # Authentication service
|   |   +-- authorization.proto       # Authorization checks
|   |   +-- rbac.proto                # Role-based access control
|   +-- governance/
|   |   +-- audit.proto               # Audit logging
|   |   +-- data_lineage.proto        # Data lineage
|   |   +-- compliance.proto          # Compliance checks
|   +-- registry/
|       +-- service_registry.proto    # Service registry
|       +-- config_registry.proto     # Configuration registry
|
+-- plane_c/                          # Data Layer (Data Layer)
|   +-- storage/
|   |   +-- object_storage.proto      # Object instance storage
|   |   +-- link_storage.proto        # Relationship storage
|   |   +-- property_storage.proto    # Property storage
|   |   +-- time_series.proto         # Time series data
|   +-- query/
|   |   +-- object_query.proto        # Object queries
|   |   +-- aggregation.proto         # Aggregation computation
|   |   +-- search.proto              # Full-text search
|   +-- pipeline/
|   |   +-- sync_task.proto           # Data sync tasks
|   |   +-- transform.proto           # Data transformation
|   |   +-- schedule.proto            # Schedule management
|   +-- materialization/
|       +-- view.proto                # Materialized views
|       +-- subscription.proto        # Change subscriptions
|
+-- plane_d/                          # Reasoning & Decision (Reasoning & Decision Layer)
|   +-- reasoning/
|   |   +-- rule.proto                # Rule definitions
|   |   +-- rule_evaluation.proto     # Rule evaluation
|   |   +-- constraint.proto          # Constraints
|   +-- decision/
|   |   +-- decision.proto            # Decision definitions
|   |   +-- decision_tree.proto       # Decision trees
|   |   +-- scoring.proto             # Scoring models
|   +-- derived/
|   |   +-- derived_property.proto    # Derived properties
|   |   +-- dependency_dag.proto      # Dependency DAG
|   +-- simulation/
|       +-- scenario.proto            # Scenario simulation
|       +-- what_if.proto             # What-If analysis
|
+-- plane_e/                          # Agent Runtime (Agent Runtime Layer)
|   +-- action/
|   |   +-- action.proto              # Action execution
|   |   +-- action_template.proto     # Action templates
|   |   +-- approval.proto            # Approval workflows
|   +-- workflow/
|   |   +-- workflow.proto            # Workflow definitions
|   |   +-- workflow_execution.proto  # Workflow execution
|   +-- agent/
|       +-- agent.proto               # Agent definitions
|       +-- agent_execution.proto     # Agent execution
|
+-- plane_h/                          # SDK & Developer Experience
|   +-- sdk/
|   |   +-- client_config.proto       # SDK configuration
|   |   +-- batch_operation.proto     # Batch operations
|   +-- developer/
|       +-- function.proto            # Custom functions
|       +-- webhook.proto             # Webhook management
|
+-- common/                           # Cross-Layer shared types
    +-- types.proto                   # Base types
    +-- pagination.proto              # Pagination
    +-- error.proto                   # Error codes
    +-- metadata.proto                # Metadata
    +-- health.proto                  # Health checks

#2.2 Proto File Naming and Numbering Conventions

Each proto file follows strict conventions:

PROTOBUF

// file: protos/plane_b/ontology/object_type.proto
syntax = "proto3";

package onto.plane_b.ontology;

option java_package = "com.onto.control.ontology.grpc";
option java_outer_classname = "ObjectTypeProto";
option java_multiple_files = true;

import "common/types.proto";
import "common/pagination.proto";
import "common/error.proto";

// ObjectType defines an object type in the ontology model
service ObjectTypeService {
  // Create a new object type
  rpc CreateObjectType(CreateObjectTypeRequest)
      returns (CreateObjectTypeResponse);

  // Get object type by ID
  rpc GetObjectType(GetObjectTypeRequest)
      returns (GetObjectTypeResponse);

  // List object types with pagination
  rpc ListObjectTypes(ListObjectTypesRequest)
      returns (ListObjectTypesResponse);

  // Update an existing object type
  rpc UpdateObjectType(UpdateObjectTypeRequest)
      returns (UpdateObjectTypeResponse);

  // Delete an object type
  rpc DeleteObjectType(DeleteObjectTypeRequest)
      returns (DeleteObjectTypeResponse);
}

message CreateObjectTypeRequest {
  string world_id = 1;           // Owning World
  string api_name = 2;           // API name (camelCase)
  string display_name = 3;       // Display name
  string description = 4;        // Description
  string icon = 5;               // Icon identifier
  repeated PropertyDefinition properties = 6;
  string primary_key_property = 7;
}

message ObjectType {
  string id = 1;
  string api_name = 2;
  string display_name = 3;
  string description = 4;
  string icon = 5;
  repeated PropertyDefinition properties = 6;
  string primary_key_property = 7;
  int64 created_at = 8;
  int64 updated_at = 9;
  string created_by = 10;
  ObjectTypeStatus status = 11;
}

enum ObjectTypeStatus {
  OBJECT_TYPE_STATUS_UNSPECIFIED = 0;
  OBJECT_TYPE_STATUS_ACTIVE = 1;
  OBJECT_TYPE_STATUS_DEPRECATED = 2;
  OBJECT_TYPE_STATUS_ARCHIVED = 3;
}

#2.3 Field Numbering Strategy

We use segmented numbering to reserve expansion space:

Code

Field Number Allocation Strategy:
1-15    : High-frequency fields (1-byte varint, optimal performance)
16-99   : Regular fields
100-199 : Extension fields (reserved for future versions)
200-299 : Internal fields (not exposed externally)
900-999 : Debug/diagnostic fields

#3. The 59 REST Endpoints and Gateway Pattern

#3.1 REST Endpoint Distribution

While all internal communication uses gRPC, external clients (browsers, third-party systems, mobile apps) need REST APIs. We achieve REST-to-gRPC automatic conversion through an API Gateway:

Code

59 REST Endpoints by Layer:

Control Layer (Control):
  /api/v1/ontology/object-types     (CRUD = 5)
  /api/v1/ontology/link-types       (CRUD = 5)
  /api/v1/ontology/action-types     (CRUD = 5)
  /api/v1/worlds                    (CRUD + branch/merge = 8)
  /api/v1/auth/*                    (login/logout/token = 4)
  Subtotal: 27 endpoints

Data Layer (Data):
  /api/v1/objects                   (CRUD + search = 6)
  /api/v1/links                     (CRUD = 4)
  /api/v1/queries                   (execute/save/list = 3)
  /api/v1/pipelines                 (CRUD + run = 5)
  Subtotal: 18 endpoints

Reasoning & Decision Layer (Intelligence):
  /api/v1/rules                     (CRUD + evaluate = 5)
  /api/v1/decisions                 (CRUD + execute = 3)
  Subtotal: 8 endpoints

Agent Runtime Layer (Agent):
  /api/v1/actions                   (execute/list/status = 3)
  /api/v1/workflows                 (CRUD = 3)
  Subtotal: 6 endpoints

Total: 59 endpoints

#3.2 Gateway Conversion Architecture

Code

                    +----------------------------------+
                    |          API Gateway              |
                    |   (Spring Cloud Gateway / Envoy)  |
                    |                                   |
  REST Client ---->|  1. Auth (JWT validation)          |
  (Browser,         |  2. Rate limiting (Token Bucket)  |
   Mobile,          |  3. REST -> gRPC conversion       |
   3rd Party)       |  4. Response gRPC -> JSON         |
                    |  5. Error code mapping             |
                    +--------+---------+--------+------+
                             |         |        |
                    +--------v--+ +----v----+ +-v----------+
                    |onto-control| |onto-data| |onto-intelli|
                    |  (gRPC)    | | (gRPC)  | |  (gRPC)    |
                    | Control Layer    | | Data Layer | | Reasoning & Decision Layer + Agent Runtime Layer  |
                    +------------+ +---------+ +------------+

#3.3 REST-to-gRPC Mapping Rules

The conversion follows deterministic mapping rules:

Code

REST Method -> gRPC Method Mapping:

POST   /api/v1/object-types          -> ObjectTypeService.CreateObjectType
GET    /api/v1/object-types/{id}     -> ObjectTypeService.GetObjectType
GET    /api/v1/object-types          -> ObjectTypeService.ListObjectTypes
PUT    /api/v1/object-types/{id}     -> ObjectTypeService.UpdateObjectType
DELETE /api/v1/object-types/{id}     -> ObjectTypeService.DeleteObjectType

URL Path Parameters -> Protobuf Field Mapping:
  {id} -> request.id
  ?page_size=20 -> request.pagination.page_size
  ?page_token=xxx -> request.pagination.page_token

HTTP Header -> gRPC Metadata Mapping:
  Authorization: Bearer <token> -> metadata["authorization"]
  X-World-Id: <world_id>       -> metadata["x-world-id"]
  X-Request-Id: <uuid>         -> metadata["x-request-id"]

#3.4 JSON and Protobuf Field Name Conversion

Code

Protobuf (snake_case)  ->  JSON (camelCase)

object_type_id         ->  objectTypeId
display_name           ->  displayName
created_at             ->  createdAt
page_token             ->  pageToken

This conversion is handled automatically by protobuf-java-util's JsonFormat — no manual mapping required.

#4. API Versioning Strategy

#4.1 Version Number Design

Code

Versioning Strategy:

URI Version: /api/v1/...  /api/v2/...
Proto Package Version: onto.plane_b.ontology.v1 -> onto.plane_b.ontology.v2

Current version: v1 (all 59 endpoints)
Planned version: v2 (when v1 has breaking changes)

Version Lifecycle:
  v1 released -> v2 released -> v1 deprecated -> 6-month transition -> v1 removed

#4.2 Backward Compatibility Rules

We define an explicit "compatibility contract":

Code

ALLOWED - Backward-compatible changes (no version bump needed):
  - Adding new RPC methods
  - Adding new message fields (with new field numbers)
  - Adding new enum values
  - Adding new REST endpoints
  - Relaxing validation rules (e.g., required -> optional)

FORBIDDEN - Breaking changes (must bump version):
  - Removing or renaming RPC methods
  - Removing or renaming message fields
  - Changing field types or numbers
  - Changing RPC method semantics
  - Tightening validation rules

#4.3 Proto Compatibility Checks in CI

We use the buf tool for automated compatibility checking in CI/CD:

YAML

# .gitlab-ci.yml proto check stage
proto-lint:
  stage: validate
  script:
    - buf lint protos/
    - buf breaking protos/ --against .git#branch=main
  rules:
    - changes:
        - protos/**/*.proto

YAML

# buf.yaml configuration
version: v1
breaking:
  use:
    - WIRE_JSON      # Check wire format and JSON compatibility
    - PACKAGE        # Check package-level changes
lint:
  use:
    - DEFAULT
    - COMMENTS       # Require comments on all public elements
  except:
    - PACKAGE_VERSION_SUFFIX

#5. Authentication and Rate Limiting

#5.1 Authentication Flow

Code

Authentication Architecture:

  Client                Gateway              Auth Service (Control Layer)
    |                      |                        |
    |  1. Login            |                        |
    |  POST /auth/login    |                        |
    |  {user, password}    |                        |
    |--------------------->|                        |
    |                      |   2. gRPC Authenticate |
    |                      |----------------------->|
    |                      |                        | 3. Validate creds
    |                      |   4. JWT + Refresh     |    Query RBAC
    |                      |<-----------------------|
    |  5. {access_token,   |                        |
    |      refresh_token}  |                        |
    |<---------------------|                        |
    |                      |                        |
    |  6. API request      |                        |
    |  Authorization:      |                        |
    |  Bearer <jwt>        |                        |
    |--------------------->|                        |
    |                      |  7. JWT local validation|
    |                      |  (public key cached,    |
    |                      |   no Auth Service call) |
    |                      |                        |
    |                      |  8. Forward gRPC req    |
    |                      |  (metadata injected:    |
    |                      |   user_id, roles)       |
    |                      |----------------------->|

#5.2 JWT Token Structure

JSON

{
  "sub": "user-001",
  "iss": "onto-platform",
  "iat": 1711234567,
  "exp": 1711238167,
  "roles": ["admin", "data-engineer"],
  "worlds": ["world-prod", "world-staging"],
  "permissions": [
    "ontology:read",
    "ontology:write",
    "objects:read",
    "objects:write",
    "actions:execute"
  ]
}

#5.3 Rate Limiting Design

Code

Rate Limiting Strategy (Three-Layer Defense):

Layer 1: Global Rate Limit
  +-- All clients share: 10,000 req/s
  +-- Exceeding returns HTTP 429

Layer 2: Tenant Rate Limit
  +-- Per tenant: 1,000 req/s
  +-- Critical APIs (writes): 100 req/s
  +-- Exceeding returns HTTP 429 + Retry-After header

Layer 3: User Rate Limit
  +-- Per user: 100 req/s
  +-- Special APIs (export/batch): 10 req/s
  +-- Exceeding returns HTTP 429

Algorithm: Token Bucket
Implementation: Gateway in-memory + Redis shared counters

Java

// Gateway rate limit configuration example
@Configuration
public class RateLimitConfig {

    @Bean
    public KeyResolver userKeyResolver() {
        return exchange -> Mono.just(
            exchange.getRequest()
                .getHeaders()
                .getFirst("X-User-Id")
        );
    }

    @Bean
    public RateLimiter rateLimiter() {
        return new RedisRateLimiter(100, 200); // 100 req/s, burst 200
    }
}

#6. Python SDK: Auto-Generating 38 gRPC Clients

#6.1 Generation Pipeline

This is one of our core development efficiency mechanisms: when proto files change, Python SDK gRPC client code is automatically regenerated.

Code

Proto File Change Triggers:

protos/*.proto
    |
    v
protoc + grpc_python_plugin
    |
    +-- *_pb2.py          (message classes)
    +-- *_pb2_grpc.py     (stub classes)
    +-- *_pb2.pyi         (type hints)
    |
    v
SDK Wrapper Layer (hand-written)
    |
    +-- grpc_object_type_client.py    (ObjectType client)
    +-- grpc_world_client.py          (World client)
    +-- grpc_action_client.py         (Action client)
    +-- ...38 clients total

#6.2 SDK Client Wrapper Pattern

Each gRPC client follows a uniform wrapper pattern:

Python

"""ObjectType gRPC client wrapper"""
from typing import Optional, List
import grpc
from ontology_sdk.grpc_client.base_client import BaseGrpcClient
from ontology_sdk.models.object_type import (
    ObjectTypeModel,
    CreateObjectTypeRequest,
    ObjectTypeListResponse,
)
from plane_b.ontology import object_type_pb2, object_type_pb2_grpc


class GrpcObjectTypeClient(BaseGrpcClient):
    """ObjectType gRPC Client

    Provides CRUD operations for ObjectTypes with automatic:
    - gRPC connection management (connection pool + reconnect)
    - Auth token injection
    - Error code conversion (gRPC Status -> SDK Exception)
    - Protobuf <-> Pydantic model conversion
    """

    def __init__(self, channel: grpc.Channel, metadata_provider):
        super().__init__(channel, metadata_provider)
        self._stub = object_type_pb2_grpc.ObjectTypeServiceStub(channel)

    def create(
        self,
        world_id: str,
        api_name: str,
        display_name: str,
        description: str = "",
        properties: Optional[List[dict]] = None,
    ) -> ObjectTypeModel:
        """Create a new ObjectType"""
        request = object_type_pb2.CreateObjectTypeRequest(
            world_id=world_id,
            api_name=api_name,
            display_name=display_name,
            description=description,
        )

        response = self._call_with_retry(
            self._stub.CreateObjectType,
            request,
        )
        return ObjectTypeModel.from_proto(response.object_type)

    def get(self, object_type_id: str) -> ObjectTypeModel:
        """Get ObjectType details"""
        request = object_type_pb2.GetObjectTypeRequest(id=object_type_id)
        response = self._call_with_retry(
            self._stub.GetObjectType, request
        )
        return ObjectTypeModel.from_proto(response.object_type)

#6.3 BaseGrpcClient Common Capabilities

Python

class BaseGrpcClient:
    """Base class for all gRPC clients

    Provides:
    - Retry mechanism (exponential backoff)
    - Error code conversion
    - Metadata injection
    - Logging
    """

    RETRYABLE_CODES = {
        grpc.StatusCode.UNAVAILABLE,
        grpc.StatusCode.DEADLINE_EXCEEDED,
        grpc.StatusCode.RESOURCE_EXHAUSTED,
    }

    def _call_with_retry(self, method, request, max_retries=3):
        """gRPC call with retry"""
        last_error = None

        for attempt in range(max_retries + 1):
            try:
                metadata = self._metadata_provider.get_metadata()
                return method(request, metadata=metadata)
            except grpc.RpcError as e:
                last_error = e
                if e.code() not in self.RETRYABLE_CODES:
                    raise self._convert_error(e)
                if attempt < max_retries:
                    delay = (2 ** attempt) * 0.1
                    time.sleep(delay)

        raise self._convert_error(last_error)

    def _convert_error(self, rpc_error: grpc.RpcError):
        """gRPC error code -> SDK exception"""
        code = rpc_error.code()
        detail = rpc_error.details()
        mapping = {
            grpc.StatusCode.NOT_FOUND: NotFoundError,
            grpc.StatusCode.ALREADY_EXISTS: AlreadyExistsError,
            grpc.StatusCode.INVALID_ARGUMENT: InvalidArgumentError,
            grpc.StatusCode.PERMISSION_DENIED: PermissionDeniedError,
            grpc.StatusCode.UNAUTHENTICATED: UnauthenticatedError,
            grpc.StatusCode.RESOURCE_EXHAUSTED: RateLimitError,
            grpc.StatusCode.INTERNAL: InternalError,
        }
        exc_class = mapping.get(code, PlatformError)
        return exc_class(detail, grpc_code=code)

#6.4 Full List of 38 Clients

Code

38 gRPC Clients by Layer:

Control Layer (Control) -- 16 clients:
  grpc_object_type_client.py
  grpc_link_type_client.py
  grpc_property_type_client.py
  grpc_action_type_client.py
  grpc_ontology_version_client.py
  grpc_world_client.py
  grpc_world_branch_client.py
  grpc_world_merge_client.py
  grpc_auth_client.py
  grpc_rbac_client.py
  grpc_audit_client.py
  grpc_lineage_client.py
  grpc_compliance_client.py
  grpc_service_registry_client.py
  grpc_config_registry_client.py
  grpc_metric_client.py

Data Layer (Data) -- 10 clients:
  grpc_object_storage_client.py
  grpc_link_storage_client.py
  grpc_property_storage_client.py
  grpc_time_series_client.py
  grpc_query_client.py
  grpc_aggregation_client.py
  grpc_search_client.py
  grpc_sync_task_client.py
  grpc_view_client.py
  grpc_subscription_client.py

Reasoning & Decision Layer (Intelligence) -- 7 clients:
  grpc_rule_client.py
  grpc_rule_evaluation_client.py
  grpc_decision_client.py
  grpc_derived_property_client.py
  grpc_dependency_dag_client.py
  grpc_scenario_client.py
  grpc_reasoning_client.py

Agent Runtime Layer (Agent) -- 5 clients:
  grpc_action_client.py
  grpc_action_template_client.py
  grpc_approval_client.py
  grpc_workflow_client.py
  grpc_agent_client.py

#7. Unified Error Code System

#7.1 Error Code Structure

A unified error code format across all Layers:

Code

Error Code Format: ONTO-{Layer}-{CATEGORY}-{NUMBER}

Layer:
  B = Control Data Layer = Data Reasoning & Decision Layer = Intelligence Agent Runtime Layer = Agent Runtime
  H = SDK

CATEGORY:
  VAL = Validation error
  AUTH = Authentication/Authorization
  RES = Resource error
  SYS = System error
  BIZ = Business logic error

Examples:
  ONTO-B-VAL-001  = Control Layer parameter validation failed
  ONTO-C-RES-003  = Data Layer object not found
  ONTO-D-BIZ-007  = Intelligence Layer rule conflict
  ONTO-E-SYS-002  = Agent Runtime workflow timeout

#7.2 gRPC Status to HTTP Status Mapping

Code

gRPC -> HTTP Status Code Mapping:

gRPC Code               HTTP Status    Meaning
---------------------------------------------------
OK                       200           Success
INVALID_ARGUMENT         400           Bad request
UNAUTHENTICATED          401           Not authenticated
PERMISSION_DENIED        403           Forbidden
NOT_FOUND                404           Not found
ALREADY_EXISTS           409           Conflict
FAILED_PRECONDITION      412           Precondition failed
RESOURCE_EXHAUSTED       429           Rate limited
CANCELLED                499           Client cancelled
INTERNAL                 500           Internal error
UNAVAILABLE              503           Service unavailable
DEADLINE_EXCEEDED        504           Timeout

#7.3 Error Response Format

JSON

{
  "error": {
    "code": "ONTO-B-RES-003",
    "message": "ObjectType 'CustomerOrder' not found in World 'world-prod'",
    "grpc_code": "NOT_FOUND",
    "http_status": 404,
    "details": {
      "resource_type": "ObjectType",
      "resource_id": "CustomerOrder",
      "world_id": "world-prod"
    },
    "request_id": "req-abc-123",
    "timestamp": "2026-03-24T10:30:00Z"
  }
}

#8. Cross-Layer Request Tracing

#8.1 Request Tracing Design

Code

Complete call chain for a single API request:

Client -> Gateway -> onto-control -> onto-data -> onto-intelligence
                                                       |
                                                       v
                                                  Kafka Event
                                                       |
                                                       v
                                              onto-data (Consumer)

Every hop carries:
  X-Request-Id:   Globally unique request ID
  X-Trace-Id:     Distributed trace ID (OpenTelemetry)
  X-Span-Id:      Current span ID
  X-Parent-Span:  Parent span ID
  X-World-Id:     Current World context

#8.2 gRPC Interceptor Implementation

Python

class TracingInterceptor(grpc.UnaryUnaryClientInterceptor):
    """Client-side tracing interceptor"""

    def intercept_unary_unary(self, continuation,
                               client_call_details, request):
        metadata = list(client_call_details.metadata or [])
        trace_context = get_current_trace_context()
        metadata.extend([
            ("x-request-id", trace_context.request_id),
            ("x-trace-id", trace_context.trace_id),
            ("x-span-id", generate_span_id()),
            ("x-parent-span", trace_context.span_id),
        ])
        new_details = client_call_details._replace(metadata=metadata)
        return continuation(new_details, request)

Java

// Java server-side interceptor
@Component
public class TracingServerInterceptor implements ServerInterceptor {

    @Override
    public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(
        ServerCall<ReqT, RespT> call,
        Metadata headers,
        ServerCallHandler<ReqT, RespT> next) {

        String requestId = headers.get(
            Metadata.Key.of("x-request-id",
                Metadata.ASCII_STRING_MARSHALLER));
        String traceId = headers.get(
            Metadata.Key.of("x-trace-id",
                Metadata.ASCII_STRING_MARSHALLER));

        MDC.put("requestId", requestId);
        MDC.put("traceId", traceId);

        try {
            return next.startCall(call, headers);
        } finally {
            MDC.clear();
        }
    }
}

#9. Comparison: Palantir vs coomia-dip API Design

Dimension	Palantir Foundry	coomia-dip
External API	REST (OSDK)	REST (59 endpoints)
Internal comms	Undisclosed (likely gRPC/custom)	gRPC (63 protos)
API definition	Partially public Swagger	Protobuf Contract-First
SDK generation	OSDK CLI	protoc + manual wrappers
Authentication	OAuth 2.0	JWT + OAuth 2.0
API versioning	URI versioning (/v1, /v2)	URI + Proto package version
Rate limiting	Yes (details undisclosed)	Token Bucket 3-layer
Error codes	HTTP Status + JSON	Unified ONTO-X-XXX-NNN
API gateway	Proprietary	Spring Cloud Gateway

#10. Practical Lessons and Pitfalls

#10.1 Five Lessons from Proto Design

Code

Lesson 1: Field naming must be strictly unified from Day 1
  Wrong: mixing object_type_id, objectTypeId, type_id
  Fixed: all snake_case, strict {entity}_{field} naming

Lesson 2: Don't expose domain models directly as proto messages
  Wrong: proto messages mirror database table structure 1:1
  Fixed: proto is an API contract, can differ from internal models

Lesson 3: First enum value must be UNSPECIFIED
  Wrong: enum Status { ACTIVE = 0; }
  Fixed: enum Status { STATUS_UNSPECIFIED = 0; ACTIVE = 1; }

Lesson 4: Pagination must be a standard component from Day 1
  Wrong: each List method defines its own offset/limit
  Fixed: unified PaginationRequest/PaginationResponse

Lesson 5: Reserve field number space
  Wrong: sequential numbering 1,2,3,4,5...
  Fixed: segmented 1-15(hot), 16-99(regular), 100+(extension)

#10.2 Three Gateway Pitfalls

Code

Pitfall 1: REST mapping for streaming APIs
  Problem: gRPC ServerStream can't directly map to REST
  Solution: Use SSE (Server-Sent Events) or paginated polling

Pitfall 2: Large file uploads
  Problem: gRPC default message size is 4MB
  Solution: Large files use dedicated REST endpoints with chunked upload

Pitfall 3: WebSocket requirements
  Problem: Real-time push scenarios where REST is insufficient
  Solution: Gateway provides dedicated WebSocket endpoints,
            internally converted to gRPC bidirectional streams

#Key Takeaways

Contract-First is mandatory for large distributed platforms: 63 proto files serve as the system's single source of truth — generating code from protos rather than the reverse ensures cross-language, cross-team interface consistency.
REST and gRPC are not either/or: Internal gRPC ensures performance and type safety, external REST ensures compatibility and usability, and the Gateway layer handles conversion — best of both worlds.
A unified error code and tracing system is the foundation of observability: The ONTO-X-XXX-NNN encoding enables quick identification of the specific Layer and category for any error, combined with X-Request-Id for end-to-end tracing.

#Next Article Preview

S2-09 Data Flow Panorama: A Data Point's Complete Journey from Ingestion to Decision — We will trace a single data change through the entire system: external database change -> Flink CDC capture -> Kafka transport -> storage layer write -> Ontology Runtime triggers subscription -> derived property recalculation -> rule evaluation -> decision execution -> audit recording. A complete panorama of data flow through the coomia-dip platform.

tags: API-Design, Protobuf, gRPC, REST, Gateway, Contract-First, SDK, Error-Handling, Rate-Limiting, coomia-dip