“

Series: S14 Engineering Stories · Article 3 | Level: Intermediate | Reading Time: 15 min

Merging 8 Layers into 3 Processes

#TL;DR

coomia-dip's initial architecture divided the system into 8 Layers, each independently deployed and evolved. This design perfectly followed "separation of concerns" in theory, but for a 4-person team in practice, 8 independent processes meant 8 sets of configuration, 8 startup scripts, and 28 potential gRPC connections — complexity far beyond the team's capacity. This article documents how we merged 8 Layers into 3 deployment processes: Control (B+G), Compute (C+F), and Intelligence (D+E), while maintaining logical architecture clarity, along with the technical challenges and lessons learned.

#1. The Problem: The Cost of 8 Layers

#1.1 The Gap Between Theory and Practice

The 8-Layer architecture looked elegant on the whiteboard:

A (Deployment) → B (Control) → C (Data)
                     ↓            ↓
                 G (Metadata)  F (Pipeline)
                     ↓
                 D (Reasoning) → E (Agent)
                     ↓
                 H (SDK)

Each Layer had clear responsibility boundaries, technology stacks could be chosen independently, and teams could develop in parallel. A textbook microservices case.

But reality was different:

Problem 1: Starting a complete dev environment took 12 minutes

The first thing a developer did every morning was docker compose up, then wait 12 minutes. This included:

• Image pulling and startup for 8 services
• Initialization of 3 infrastructure components (Doris, MinIO, Redis)
• Health checks and connection establishment between services

If any single service failed to start (which happened roughly 2-3 times per week in WSL2), troubleshooting, clearing state, and restarting could take another 20 minutes.

Problem 2: Debugging cross-Layer issues was extremely difficult

A typical Action execution flow passed through 4 Layers:

SDK (H) → Control Layer (B) → Data Layer (C) → Intelligence Layer (D)

When an Action failed, developers had to jump between logs from 4 services to piece together the complete call chain. Although we integrated distributed tracing (Jaeger), running an additional Jaeger service in dev consumed another 500MB of memory.

Problem 3: gRPC connection management complexity

Between 8 Layers, there could be up to C(8,2) = 28 bidirectional connections. In practice, we had 14 active gRPC connections. Each required:

• Configuring target address and port
• Handling connection timeouts and retries
• Managing connection pools
• Handling graceful shutdown

The combined configuration and management code for 14 connections exceeded 2,000 lines.

Problem 4: Test environment resource consumption

In the CI/CD pipeline, running a complete integration test required starting all 8 services + 3 infrastructure = 11 containers. On our CI server (32GB RAM), this consumed ~20GB, leaving almost no headroom.

#1.2 Quantifying the Pain

We spent a week collecting the following data:

These numbers told us: architectural elegance cannot come at the cost of development efficiency.

#2. Design Constraints: What Cannot Change

Before discussing the merge plan, we defined non-negotiable constraints:

#2.1 Logical Boundaries Must Be Preserved

We were merging deployment units (processes), not code organization. Each Layer's code would remain in independent packages/modules with clear API boundaries. This meant if the team grew in the future, we could split back to independent deployment at any time.

#2.2 gRPC Interfaces Remain Unchanged

All inter-Layer communication continues via gRPC. After merging, gRPC calls between Layers within the same process use in-process channels, avoiding network overhead, but interface definitions remain unchanged.

#2.3 Technology Stack Constraints

• B (Control) and G (Metadata) are both Java/Spring Boot — mergeable
• C (Data) and F (Pipeline) are both Java/Quarkus — mergeable
• D (Reasoning) and E (Agent) are both Python/FastAPI — mergeable
• A (Deployment) stays independent (it's a deployment tool, not a runtime service)
• H (SDK) stays independent (it's a client library, not a service)

#2.4 Reversibility

The merge must be reversible. If needed in the future, any merged Layer should be splittable back to independent deployment within one week.

#3. The Merge Plan

#3.1 Final 3-Process Architecture

After evaluation, we settled on the following merge plan:

Process 1: Control Service (Java/Spring Boot)
├── Control Layer: Control Layer (Ontology Runtime, Schema Registry, Action Engine)
└── Metadata & Governance Layer: Metadata & Governance (metadata management, data lineage, audit)

Process 2: Compute Service (Java/Quarkus)
├── Data Layer: Data Layer (storage engine, query engine, OQL)
└── Pipeline & Orchestration Layer: Pipeline & Orchestration (data pipelines, ETL, scheduling)

Process 3: Intelligence Service (Python/FastAPI)
├── Reasoning & Decision Layer: Reasoning & Decision (rule engine, reasoning engine)
└── Agent Runtime Layer: Agent Runtime (agent framework, Temporal workflows)

Deployment & Operations Layer (Deployment) and SDK & Developer Experience Layer (SDK) don't participate in the merge — they were never runtime services.

#3.2 Why These Combinations

B + G → Control Service

B (Control) handles the core Ontology runtime; G (Metadata) handles metadata governance. They share extensive domain models (ObjectType, LinkType, PropertyType). After merging, massive amounts of cross-process data transfer were eliminated.

Before the merge, G needed to pull ObjectType definitions from B via gRPC to build data lineage. After merging, G can call B's internal API directly — latency dropped from 5-10ms to < 0.1ms.

C + F → Compute Service

C (Data) handles data storage and queries; F (Pipeline) handles data pipelines. They share the storage layer — pipeline output writes directly to Data Layer storage.

Before the merge, F needed to push data to C via gRPC after completing ETL. After merging, F can call C's storage API directly — bulk import performance improved 3-5x.

D + E → Intelligence Service

D (Reasoning) handles rules and inference; E (Agent) handles the Agent runtime. They share the Python runtime and AI model loader.

Before the merge, E's Agents needed to call D's reasoning interface via gRPC for decisions. After merging, Agents can call reasoning engine Python functions directly, eliminating serialization/deserialization overhead.

#4. Technical Implementation

#4.1 In-Process gRPC Channels

The core technical challenge was: how to preserve gRPC communication between two Layers within the same process while avoiding network overhead.

Java Side (Control Service / Compute Service)

We used gRPC's InProcessServer and InProcessChannel:

// Create in-process server
Server inProcessServer = InProcessServerBuilder
    .forName("control-internal")
    .addService(new MetadataGrpcService(metadataService))
    .build()
    .start();

// Create in-process channel
ManagedChannel channel = InProcessChannelBuilder
    .forName("control-internal")
    .directExecutor()
    .build();

// Create stub using channel (identical to remote calls)
MetadataServiceGrpc.MetadataServiceBlockingStub stub =
    MetadataServiceGrpc.newBlockingStub(channel);

Benefits:

• Stub usage is identical to remote calls
• To split back to independent deployment, just change InProcessChannel to ManagedChannel
• In-process call latency dropped from 5-10ms to < 0.1ms

Python Side (Intelligence Service)

Python's grpcio library also supports in-process channels, but we adopted a simpler approach — direct function calls + interface adapters:

class ReasoningClient:
    """Unified interface supporting both remote and local modes"""

    def __init__(self, mode: str = "local"):
        if mode == "remote":
            self._client = GrpcReasoningClient(address="reasoning:50051")
        else:
            self._client = LocalReasoningClient(reasoning_engine)

    def evaluate_rules(self, context: RuleContext) -> RuleResult:
        return self._client.evaluate_rules(context)

#4.2 Configuration Unification

Before the merge, 8 services each had their own config files. After merging, we used layered configuration:

# control-service.yml
server:
  port: 8080
  grpc-port: 50051

# Control Layer (B) config
ontology:
  runtime:
    cache-size: 10000
    schema-version-limit: 100

# Metadata Layer (G) config
metadata:
  lineage:
    enabled: true
    storage: doris
  audit:
    retention-days: 90

Each Layer's configuration has a different prefix, preventing interference.

#4.3 Health Check Consolidation

Before the merge, each service had its own /health endpoint. After merging, health checks aggregate multiple Layer statuses:

@GetMapping("/health")
public HealthResponse health() {
    Map<String, PlaneHealth> Layers = new LinkedHashMap<>();
    Layers.put("control-Layer", controlPlaneHealth.check());
    Layers.put("metadata-Layer", metadataPlaneHealth.check());

    boolean allHealthy = Layers.values().stream()
        .allMatch(h -> h.getStatus() == Status.UP);

    return new HealthResponse(
        allHealthy ? Status.UP : Status.DEGRADED,
        Layers
    );
}

#4.4 Log Isolation

A key concern after merging was mixed logs. We used MDC (Mapped Diagnostic Context) to add prefixes for each Layer:

[2024-08-15 10:23:45] [CONTROL-B] Creating ObjectType: Device
[2024-08-15 10:23:46] [METADATA-G] Recording lineage for ObjectType: Device
[2024-08-15 10:23:46] [CONTROL-B] ObjectType created: Device (v1)

#5. The Merge Process

#5.1 Timeline

The entire merge took 3 weeks:

Week 1: Preparation

• Cataloged all cross-Layer gRPC interfaces
• Identified which could become in-process calls
• Wrote in-process channel adapters
• Updated Docker Compose configuration

Week 2: Merge Execution

• Day 1-2: Merge B + G → Control Service
• Day 3-4: Merge C + F → Compute Service
• Day 5: Merge D + E → Intelligence Service

Week 3: Verification

• Ran all 3,500+ tests
• Performance benchmark comparisons
• Fixed merge-introduced bugs (7 total)
• Updated documentation

#5.2 Problems Encountered During the Merge

Problem 1: Spring Boot Bean Conflicts

Both B and G were Spring Boot applications. After merging, Bean name conflicts appeared — both Layers defined ObjectTypeRepository Beans.

Solution: Use @Qualifier annotations with a {Layer}{BeanName} naming convention.

Problem 2: Quarkus Class Loading Issues

Both C and F were Quarkus applications. After merging, Quarkus CDI showed ambiguity when handling identical interfaces from both modules.

Solution: Use @Alternative and @Priority annotations to establish explicit priority.

Problem 3: Python Module Name Conflicts

Both D and E had models packages. After merging, import models was ambiguous.

Solution: All imports switched to absolute path imports:

# Before
from models import RuleContext

# After
from intelligence_plane.reasoning.models import RuleContext
from intelligence_plane.agent.models import AgentContext

Problem 4: Port Conflicts

Before the merge, each service listened on different ports (8080-8087). After merging, only 3 HTTP ports + 3 gRPC ports were needed.

Problem 5: Database Connection Pool Tuning

Before: each service maintained its own pool (10 connections x 8 = 80 total). After: 3 services needed fewer connections (20 connections x 3 = 60 total).

#6. Results After Merging

#6.1 Development Efficiency Improvement

#6.2 Performance Improvement

In-process gRPC calls eliminated network overhead:

Note: cross-process calls (e.g., B → C) latency was unchanged since they still use network gRPC.

#7. Reversibility Verification

To verify the "reversibility" constraint, we spent half a day after the merge doing a "split drill":

1. Extracted governance-metadata from control-Layer
2. Created an independent metadata-service entry point
3. Changed in-process channels to network gRPC channels
4. Ran all Metadata & Governance Layer tests

Result: split completed in ~3 hours; all tests passed. This verified our merge approach was indeed reversible.

#8. Lessons Learned

#8.1 Architecture Should Match Team Size

Conway's Law says "system architecture reflects the organization's communication structure." Our experience adds a corollary: the number of deployment units should not exceed 2x the team size.

4-person team maintaining 8 deployment units = 2 per person. Considering configuration, monitoring, and troubleshooting overhead per unit, this exceeded reasonable bounds. Merging to 3 units (< 1 per person) significantly improved development efficiency.

#8.2 Logical and Deployment Architecture Can Be Decoupled

This is the most important insight from the merge. Logically having 8 Layers and deploying as 3 processes can coexist perfectly. Code organization follows logical architecture (clear boundaries, independent modules); deployment strategy follows practical needs (merge during development, split during scaling).

#8.3 In-Process gRPC Is a Great Pattern

In-process gRPC channels let us merge deployment units while preserving interface contracts. This means:

• Interface tests don't need modification
• Splitting only requires changing channel configuration
• Performance improvement is orders of magnitude

#8.4 Merge Early

If we had done this merge in month three instead of month nine, we could have saved ~6 months of "operations tax." The delay was caused by "fear of breaking existing functionality" — but with a comprehensive test suite, the merge was safe.

#9. When to Split Back

We preset three "split trigger conditions":

1. Team grows beyond 8 people: At least 2 people per process, sufficient staffing for independent deployment
2. Single process resource consumption hits bottleneck: CPU or memory can't be met via vertical scaling
3. Different Layers need different scaling strategies: e.g., Intelligence Layer needs GPUs while Control Layer doesn't

As of Day 420, none of these conditions have been triggered.

#10. Conclusion

Merging 8 Layers into 3 processes was an important exercise in "pragmatic engineering" for coomia-dip. It taught us: good architecture isn't about pursuing theoretically perfect separation, but finding the balance between clarity and practicality.

Logically, we still have 8 Layers. In code, each Layer is still an independent module. But in deployment, we chose the most efficient approach for a 4-person team — 3 processes.

If the team grows and business scales in the future, we can split back at any time. And because we preserved gRPC interfaces and module boundaries, the cost of splitting is manageable.

#Key Takeaways

1. Deployment units <= team size x 2: Exceeding this ratio causes operational overhead to consume development efficiency
2. Logical and deployment architecture can be decoupled: Code follows logical boundaries; deployment follows practical needs
3. In-process gRPC channels are an excellent pattern for merging deployment while preserving interface contracts
4. Merge early: With a comprehensive test suite, merge risk is far lower than expected
5. Preset split conditions: Clearly define when to split back, avoiding post-merge inertia

#Next Article

Next: S14-04 Storage Unification Journey — The complete story of three storage architecture transitions, from PostgreSQL to ClickHouse to Doris.

Tags: #coomia-dip #ArchitectureMerge #Microservices #DeploymentArchitecture #gRPC #ConwaysLaw #EngineeringEfficiency