Why We Chose gRPC Over REST: Internal Communication Design Decisions
TL;DR
Why We Chose gRPC Over REST: Internal Communication Design Decisions
“Series: S2 Architecture Overview · Article 2 | Level: Intermediate | Reading Time: 18 min
TL;DR
- In internal communication benchmarks, gRPC delivers 3-10x throughput improvement and 60-70% latency reduction over REST/JSON, with Protobuf serialization producing payloads only 30-50% the size of JSON equivalents.
- The platform's 64 proto files are organized by Layer namespace with a unified code generation pipeline producing type-safe stubs for both Java (gRPC-Java) and Python (grpcio), enabling a contract-driven development workflow.
- REST exists solely as an external facade (59 endpoints) via Spring Cloud Gateway performing REST-to-gRPC protocol translation -- internal REST/JSON usage is a strict architectural red line.
#1. Decision Context: The Birth of a Technical Red Line
Among coomia-dip's technical red lines, one stands out prominently:
“Internal services MUST NOT use REST/JSON (gRPC is mandatory)
This is not a casual preference but a rational decision backed by rigorous benchmarking and architectural analysis. This article presents the complete reasoning.
#2. gRPC vs REST Benchmarks
#2.1 Test Environment
Hardware: 4C8G Linux VM (Intel Xeon E5-2680)
Network: Same datacenter, latency < 0.1ms
Scenario: Ontology instance CRUD operations
Data: Typical Ontology instance (15 attributes, 3 relations)
Framework comparison:
REST: Spring Boot 3.x + Jackson + HTTP/1.1
gRPC: gRPC-Java 1.60 + Protobuf 3 + HTTP/2
#2.2 Latency Comparison
Operation REST (P50/P99) gRPC (P50/P99) Improvement
----------- ---------------- ---------------- -----------
GetInstance 2.1ms / 8.3ms 0.6ms / 2.1ms 71% / 75%
CreateInstance 3.8ms / 15.2ms 1.2ms / 4.8ms 68% / 68%
BatchGet(100) 18ms / 52ms 5ms / 14ms 72% / 73%
QueryFederation 45ms / 180ms 12ms / 48ms 73% / 73%
StreamSubscribe N/A 0.3ms/msg N/A
#2.3 Throughput Comparison
Concurrent Conns REST (req/s) gRPC (req/s) Multiplier
---------------- ------------ ------------ ----------
10 4,200 14,500 3.5x
50 8,100 38,000 4.7x
100 10,200 62,000 6.1x
500 11,500 105,000 9.1x
1000 10,800 118,000 10.9x
#2.4 Payload Size Comparison
Data Object JSON (bytes) Protobuf (bytes) Ratio
----------------------- ------------ ---------------- -----
OntologyInstance (small) 512 185 36%
OntologyInstance (medium) 2,048 680 33%
OntologyInstance (large) 8,192 2,870 35%
BatchResponse (100 items) 204,800 68,000 33%
WorldContext 256 48 19%
#2.5 Why Such a Large Gap?
HTTP/1.1 + JSON (REST):
+------+------------------+------+------------------+------+
| Conn | Headers (500B) | Body | Headers (500B) | Body |
| Est | Content-Type | JSON | Content-Type | JSON |
| | Accept | | Accept | |
| | Authorization | | Authorization | |
| | X-Request-Id | | X-Request-Id | |
+------+------------------+------+------------------+------+
Each request uses independent TCP connection or serialized Keep-Alive
Text serialization: human-readable but redundant
HTTP/2 + Protobuf (gRPC):
+------+--------+--------+--------+--------+--------+
| Conn | Frame1 | Frame2 | Frame3 | Frame4 | Frame5 |
| Est | (Req) | (Req) | (Resp) | (Req) | (Resp) |
| | 48B | 185B | 680B | 48B | 185B |
+------+--------+--------+--------+--------+--------+
Single connection with multiplexed parallel streams
Binary serialization: compact and efficient
Header compression (HPACK)
Key Differentiating Factors:
| Factor | REST/JSON | gRPC/Protobuf |
|---|---|---|
| Transport protocol | HTTP/1.1 | HTTP/2 multiplexing |
| Serialization | JSON (text) | Protobuf (binary) |
| Headers | Full headers per request | HPACK compression |
| Connection management | Keep-Alive serialized | Single connection parallel streams |
| Streaming | Not native | Native bidirectional streaming |
| Type safety | Runtime validation | Compile-time checking |
#3. Protobuf Contract-Driven Development Workflow
#3.1 Proto-First Development Process
In coomia-dip, proto files are the starting point for everything:
Step 1: Define Proto
architect writes .proto file
|
v
Step 2: Team Review
proto PR review (breaking change check)
|
v
Step 3: Code Generation
buf generate (Java stubs + Python stubs)
|
+---> Java: control-Layer/src/gen/
| com.onto.{Layer}.api.v1.*
|
+---> Python: intelligence-Layer/gen/
| onto_{Layer}_v1_pb2.py
| onto_{Layer}_v1_pb2_grpc.py
|
+---> SDK: python-sdk/ontology_sdk/gen/
(gRPC client wrappers)
|
v
Step 4: Implement Server
Java: implements XxxServiceGrpc.XxxServiceImplBase
Python: class XxxServicer(xxx_pb2_grpc.XxxServicer)
|
v
Step 5: SDK Wrapping
Python SDK wraps gRPC stubs into Pythonic API
|
v
Step 6: REST Facade Mapping
Spring Cloud Gateway maps REST endpoints to gRPC calls
#3.2 Actual Proto Example
Consider OntologyRuntimeService, the platform's most critical service:
// proto/plane_c/ontology_runtime.proto
syntax = "proto3";
package com.onto.data.v1;
option java_package = "com.onto.data.api.v1";
option java_outer_classname = "OntologyRuntimeProto";
option java_multiple_files = true;
import "common/common.proto";
import "common/errors.proto";
import "google/protobuf/struct.proto";
import "google/protobuf/timestamp.proto";
service OntologyRuntimeService {
// Instance CRUD
rpc CreateInstance(CreateInstanceRequest)
returns (CreateInstanceResponse);
rpc GetInstance(GetInstanceRequest)
returns (GetInstanceResponse);
rpc UpdateInstance(UpdateInstanceRequest)
returns (UpdateInstanceResponse);
rpc PatchInstance(PatchInstanceRequest)
returns (PatchInstanceResponse);
rpc DeleteInstance(DeleteInstanceRequest)
returns (DeleteInstanceResponse);
// Batch Operations
rpc BatchCreateInstances(BatchCreateInstancesRequest)
returns (BatchCreateInstancesResponse);
rpc BatchGetInstances(BatchGetInstancesRequest)
returns (BatchGetInstancesResponse);
// Relation Management
rpc CreateRelation(CreateRelationRequest)
returns (CreateRelationResponse);
rpc GetRelations(GetRelationsRequest)
returns (GetRelationsResponse);
rpc BatchCreateRelations(BatchCreateRelationsRequest)
returns (BatchCreateRelationsResponse);
// Time Travel
rpc GetInstanceAtVersion(GetInstanceAtVersionRequest)
returns (GetInstanceResponse);
}
message CreateInstanceRequest {
com.onto.common.v1.RequestContext context = 1;
string object_type_id = 2;
google.protobuf.Struct attributes = 3;
repeated RelationInput relations = 4;
}
message GetInstanceRequest {
com.onto.common.v1.RequestContext context = 1;
string instance_id = 2;
repeated string select_properties = 3; // field projection
bool include_relations = 4;
bool include_derived = 5; // include derived properties
}
#3.3 Shared Types: WorldContext and RequestContext
// proto/common/common.proto
message RequestContext {
string request_id = 1;
string user_id = 2;
repeated string roles = 3;
com.onto.common.v1.WorldContext world_context = 4;
string trace_id = 5;
string span_id = 6;
map<string, string> headers = 7;
}
Every gRPC request takes RequestContext as its first parameter, which embeds WorldContext. This guarantees:
- Tenant isolation:
tenant_id+org_idprevent data leakage - Branch isolation:
world_id+branch_nameensure operations target the correct world branch - Traceability:
trace_id+span_idsupport OpenTelemetry distributed tracing - Authorization:
user_id+rolesfeed into Policy Engine for access control
#4. Organization of 64 Proto Files by Namespace
#4.1 Proto Distribution by Layer
proto/
├── common/ [3 files]
│ ├── common.proto # Base shared types
│ ├── common_b.proto # Control Layer extension types
│ └── errors.proto # Unified error definitions
│
├── plane_b/ [16 files]
│ ├── authentication_service.proto # Authentication
│ ├── classification_service.proto # Data classification
│ ├── connection_registry.proto # Connection registry
│ ├── dashboard_service.proto # Dashboard
│ ├── interface_registry.proto # Interface type registry
│ ├── ontology_proposal.proto # Ontology change proposals
│ ├── policy_engine.proto # Policy engine
│ ├── property_access.proto # Property access control
│ ├── scenario_service.proto # Scenario management
│ ├── schema_registry.proto # Schema registry
│ ├── struct_type.proto # Struct types
│ ├── user_management.proto # User management
│ └── world_manager.proto # World management
│
├── plane_c/ [15 files]
│ ├── ontology_runtime.proto # Ontology runtime (CORE)
│ ├── query_federation.proto # Federated queries
│ ├── object_set.proto # Object sets
│ ├── data_ingestion.proto # Data ingestion
│ └── ... (11 more)
│
├── plane_d/ [14 files]
│ ├── reasoning_engine.proto # Reasoning engine
│ ├── decision_engine.proto # Decision engine
│ ├── derived_property.proto # Derived properties
│ └── ... (11 more)
│
├── plane_e/ [5 files]
├── plane_f/ [8 files]
└── plane_g/ [5 files]
Total: 3 + 16 + 15 + 14 + 5 + 8 + 5 = 66 files
#4.2 Naming Conventions
Package naming: com.onto.{Layer}.v1
Java package: com.onto.{Layer}.api.v1
File naming: {service_name}.proto (lowercase_underscore)
Service naming: {ServiceName}Service (PascalCase)
Method naming: {VerbNoun} (PascalCase, e.g., CreateInstance)
Message naming: {MethodName}Request / {MethodName}Response
#5. Code Generation Pipeline
#5.1 Buf Build Configuration
# buf.yaml
version: v1
name: buf.build/onto/platform
breaking:
use:
- FILE
lint:
use:
- DEFAULT
# buf.gen.yaml
version: v1
plugins:
# Java gRPC stubs
- plugin: buf.build/protocolbuffers/java
out: control-Layer/src/main/java
- plugin: buf.build/grpc/java
out: control-Layer/src/main/java
# Python gRPC stubs
- plugin: buf.build/protocolbuffers/python
out: intelligence-Layer/gen
- plugin: buf.build/grpc/python
out: intelligence-Layer/gen
# Python type stubs (mypy)
- plugin: buf.build/community/nipunn1313-mypy
out: intelligence-Layer/gen
#5.2 Generated Artifact Examples
Java side (control-Layer/data-Layer):
// Auto-generated gRPC Service base class
public abstract class OntologyRuntimeServiceGrpc
.OntologyRuntimeServiceImplBase {
public void createInstance(
CreateInstanceRequest request,
StreamObserver<CreateInstanceResponse> observer) {
// To be implemented
}
}
// Implementation class
@GrpcService
public class OntologyRuntimeServiceImpl
extends OntologyRuntimeServiceGrpc
.OntologyRuntimeServiceImplBase {
@Override
public void createInstance(
CreateInstanceRequest request,
StreamObserver<CreateInstanceResponse> response) {
WorldContext world = request.getContext()
.getWorldContext();
// 1. Validate WorldContext
worldValidator.validate(world);
// 2. Schema validation
schemaRegistry.validateAttributes(
world, request.getObjectTypeId(),
request.getAttributes());
// 3. Write to Doris
OntologyInstance instance = ontologyStore
.create(world, request);
// 4. Publish change event
eventPublisher.publish(
OntologyChangeEvent.of(instance));
response.onNext(CreateInstanceResponse.newBuilder()
.setInstance(instance.toProto())
.build());
response.onCompleted();
}
}
Python side (intelligence-Layer / python-sdk):
# Auto-generated gRPC Client Stub
class OntologyRuntimeServiceStub:
def __init__(self, channel: grpc.Channel):
self.CreateInstance = channel.unary_unary(
'/com.onto.data.v1.OntologyRuntimeService/'
'CreateInstance',
request_serializer=CreateInstanceRequest
.SerializeToString,
response_deserializer=CreateInstanceResponse
.FromString,
)
# SDK wrapping layer
class GrpcOntologyClient:
"""Ontology client in the Python SDK"""
def __init__(self, channel: grpc.Channel):
self._stub = OntologyRuntimeServiceStub(channel)
def get_instance(
self,
instance_id: str,
*,
select: list[str] | None = None,
include_relations: bool = False,
include_derived: bool = False,
) -> OntologyInstance:
request = GetInstanceRequest(
context=self._build_context(),
instance_id=instance_id,
select_properties=select or [],
include_relations=include_relations,
include_derived=include_derived,
)
response = self._stub.GetInstance(request)
return OntologyInstance.from_proto(
response.instance)
#6. Versioning and Backward Compatibility
#6.1 Proto Versioning Strategy
v1 = Stable version (currently used by all services)
v2 = Next-generation (planned)
Compatibility rules:
OK: Add new fields (field numbers must not conflict)
OK: Add new RPC methods
OK: Add new enum values
BAD: Remove or rename fields
BAD: Change field types
BAD: Change field numbers
BAD: Remove RPC methods
#6.2 Buf Breaking Change Detection
# Automated breaking change detection in CI
$ buf breaking --against proto-baseline.binpb
# If passing, update baseline
$ buf build -o proto-baseline.binpb
#6.3 Field Evolution Example
// v1.0 - Initial version
message WorldContext {
string world_id = 1;
string branch_name = 2;
WorldType type = 5;
}
// v1.1 - Multi-tenancy support added (compatible)
message WorldContext {
string world_id = 1;
string branch_name = 2;
optional string commit_hash = 3; // added
optional Timestamp as_of_timestamp = 4; // added
WorldType type = 5;
map<string, string> metadata = 6; // added
string tenant_id = 7; // added
string org_id = 8; // added
string project_id = 9; // added
WorldBranchType world_branch_type = 10; // added
string manifest_id = 11; // added
optional string compute_version = 12; // added
}
#7. gRPC Error Handling
#7.1 Unified Error Code System
// proto/common/errors.proto
enum ErrorCode {
ERROR_CODE_UNSPECIFIED = 0;
// General errors (1xxx)
ERROR_CODE_INTERNAL = 1000;
ERROR_CODE_INVALID_ARGUMENT = 1001;
ERROR_CODE_NOT_FOUND = 1002;
ERROR_CODE_ALREADY_EXISTS = 1003;
ERROR_CODE_PERMISSION_DENIED = 1004;
ERROR_CODE_UNAUTHENTICATED = 1005;
// World errors (2xxx)
ERROR_CODE_WORLD_NOT_FOUND = 2001;
ERROR_CODE_BRANCH_NOT_FOUND = 2002;
ERROR_CODE_BRANCH_CONFLICT = 2003;
// Ontology errors (3xxx)
ERROR_CODE_SCHEMA_VIOLATION = 3001;
ERROR_CODE_INSTANCE_NOT_FOUND = 3002;
ERROR_CODE_RELATION_INVALID = 3003;
ERROR_CODE_DERIVED_COMPUTATION_FAILED = 3004;
// Pipeline errors (4xxx)
ERROR_CODE_PIPELINE_FAILED = 4001;
ERROR_CODE_TRANSFORM_ERROR = 4002;
}
#7.2 Error Propagation Mechanism
Java (Server) -> gRPC Status -> Python (Client)
Java:
throw new StatusRuntimeException(
Status.NOT_FOUND
.withDescription("Instance not found")
.augmentDescription("instance_id=" + id));
Python:
try:
response = stub.GetInstance(request)
except grpc.RpcError as e:
if e.code() == grpc.StatusCode.NOT_FOUND:
raise InstanceNotFoundError(
e.details()) from e
elif e.code() == grpc.StatusCode.PERMISSION_DENIED:
raise PermissionDeniedError(
e.details()) from e
#8. Streaming RPCs: Subscriptions and Real-Time Push
#8.1 Server Streaming
Used for subscribing to ontology change events:
// proto/plane_d/reasoning_subscription.proto
service ReasoningSubscriptionService {
// Subscribe to reasoning result changes
rpc SubscribeReasoningResults(SubscribeRequest)
returns (stream ReasoningEvent);
// Subscribe to derived property changes
rpc SubscribeDerivedPropertyChanges(
DerivedPropertySubscribeRequest)
returns (stream DerivedPropertyChangeEvent);
}
message SubscribeRequest {
RequestContext context = 1;
repeated string object_type_ids = 2;
repeated string instance_ids = 3;
EventFilter filter = 4;
}
#8.2 Bidirectional Streaming
Used for interactive reasoning conversations:
service RagService {
// Interactive RAG conversation
rpc Chat(stream ChatRequest)
returns (stream ChatResponse);
}
#8.3 Relationship with WebSocket
External Client (Browser/SDK)
|
| WebSocket
v
+------------------+
| REST Facade |
| (Spring Gateway) |
+------------------+
|
| gRPC Server Streaming
v
+------------------+
| Reasoning & Decision Layer + Agent Runtime Layer |
| (Intelligence) |
+------------------+
WebSocket: push protocol for external clients
gRPC Streaming: push protocol for inter-Layer communication
REST Facade: handles protocol translation
#9. REST Gateway: External Facade Design
#9.1 Why Keep REST?
gRPC is not browser-friendly (requires gRPC-Web proxy), and REST remains the de facto standard for external APIs. We retain REST as a pure external facade.
#9.2 REST Endpoint Distribution
REST Facade endpoints (59 total):
Ontology operations : 15 endpoints
POST /api/v1/ontology/instances
GET /api/v1/ontology/instances/{id}
PUT /api/v1/ontology/instances/{id}
PATCH /api/v1/ontology/instances/{id}
DELETE /api/v1/ontology/instances/{id}
POST /api/v1/ontology/instances/batch
GET /api/v1/ontology/instances/{id}/relations
...
Schema management : 8 endpoints
World management : 10 endpoints
Reasoning/Decision : 8 endpoints
Query/Analytics : 6 endpoints
Pipeline : 5 endpoints
User/Permissions : 4 endpoints
Audit/Governance : 3 endpoints
#9.3 REST-to-gRPC Translation Implementation
@RestController
@RequestMapping("/api/v1/ontology")
public class OntologyRestController {
private final OntologyRuntimeServiceGrpc
.OntologyRuntimeServiceBlockingStub stub;
@GetMapping("/instances/{id}")
public ResponseEntity<JsonNode> getInstance(
@PathVariable String id,
@RequestHeader("X-World-Id") String worldId,
@RequestHeader("X-Branch") String branch) {
// 1. Build gRPC Request
GetInstanceRequest request = GetInstanceRequest
.newBuilder()
.setContext(RequestContext.newBuilder()
.setWorldContext(WorldContext.newBuilder()
.setWorldId(worldId)
.setBranchName(branch)
.build())
.build())
.setInstanceId(id)
.build();
// 2. Call gRPC
GetInstanceResponse response =
stub.getInstance(request);
// 3. Proto -> JSON
return ResponseEntity.ok(
ProtoJsonConverter.toJson(
response.getInstance()));
}
}
#10. Cross-Language Call Paths
#10.1 Python SDK -> Java Service Complete Call Chain
Python SDK (user code)
|
| ontology.get_instance("obj_123")
v
GrpcOntologyClient (Python SDK wrapper)
|
| Build GetInstanceRequest Protobuf
v
grpcio Channel (Python gRPC client)
|
| HTTP/2 + Protobuf binary
| Target: onto-data:6668
v
Quarkus gRPC Server (onto-data.jar)
|
| Deserialize Protobuf
v
OntologyRuntimeServiceImpl (Java implementation)
|
| Business logic + Doris query
v
GetInstanceResponse (Protobuf response)
|
| HTTP/2 + Protobuf binary
v
Python SDK (deserialize to Pydantic model)
|
| OntologyInstance returned to user
v
User code
#10.2 Java -> Python Call (Reasoning Trigger)
onto-data.jar (Ontology change trigger)
|
| DerivedPropertyService.compute()
v
grpc-java Channel
|
| HTTP/2 + Protobuf
| Target: onto-intelligence:6670
v
FastAPI gRPC Server (onto-intelligence)
|
| Python reasoning engine executes computation
v
DerivedPropertyResponse
|
| HTTP/2 + Protobuf
v
onto-data.jar (write results back to Doris)
#11. Performance Optimization Practices
#11.1 Connection Pool Management
// Java-side gRPC channel configuration
@Configuration
public class GrpcChannelConfig {
@Bean
public ManagedChannel intelligenceChannel() {
return ManagedChannelBuilder
.forAddress("onto-intelligence", 6670)
.usePlaintext() // internal network
.maxInboundMessageSize(16 * 1024 * 1024)
.keepAliveTime(30, TimeUnit.SECONDS)
.keepAliveTimeout(10, TimeUnit.SECONDS)
.build();
}
}
# Python-side gRPC channel configuration
import grpc
channel = grpc.insecure_channel(
'onto-data:6668',
options=[
('grpc.max_receive_message_length',
16 * 1024 * 1024),
('grpc.keepalive_time_ms', 30000),
('grpc.keepalive_timeout_ms', 10000),
('grpc.http2.max_pings_without_data', 0),
]
)
#11.2 Batch Operation Optimization
Single vs batch operation performance (100 records):
Mode Duration Network Round Trips
--- -------- -------------------
100 x GetInstance 180ms 100
1 x BatchGetInstances 5ms 1
Conclusion: Batch RPCs reduce latency by 97%
#11.3 Deadline Propagation
// Set 3-second timeout
stub.withDeadlineAfter(3, TimeUnit.SECONDS)
.getInstance(request);
// On timeout, gRPC automatically cancels all downstream calls
// Prevents cascading failures
#12. Comparison with Other Internal Communication Solutions
| Solution | Latency | Throughput | Type Safety | Streaming | Cross-lang | Ecosystem |
|---|---|---|---|---|---|---|
| REST/JSON | High | Low | Weak | None | Excellent | Widest |
| gRPC/Protobuf | Low | High | Strong | Native | Good | Wide |
| Thrift | Low | High | Strong | Limited | Good | Narrowing |
| GraphQL | Medium | Medium | Strong | Subscriptions | Good | Web-oriented |
| NATS/MQ | Medium | High | Weak | Native | Good | Async-focused |
Why gRPC over Thrift?
- Google backing with far superior community activity
- Native HTTP/2 integrates perfectly with cloud-native ecosystem
- Mature buf toolchain (lint, breaking change detection, code generation)
- Kubernetes service mesh natively supports gRPC load balancing
#Key Takeaways
-
gRPC is an architecture choice, not just a performance choice. Protobuf contract files serve as the "interface constitution" between Layers -- they define data structures, service methods, error codes, and version evolution rules. The 64 proto files cover 100% of the platform's internal interactions.
-
Proto-First workflow eliminates implicit cross-language conventions. Define proto first, generate code second, implement third -- this workflow ensures Java servers and Python clients have perfectly aligned understanding of every interface, with incompatibilities caught at compile time.
-
REST is the facade, gRPC is the skeleton. The 59 REST endpoints carry no business logic -- they are merely HTTP wrappers around gRPC calls. This layering gives internal communication maximum performance while maintaining REST's usability for external consumers.
“Next Article Preview: [S2-03] Storage Architecture Evolution: Unifying from 9 Components to 5 -- a deep dive into how Apache Doris simultaneously handles OLAP, vector search, and full-text search, and how Nessie+Iceberg enables complete versioning.
Tags: #grpc #protobuf #rest #api-design #contract-driven #code-generation #coomia-dip