Designing a Palantir from Scratch: The 8-Layer Architecture

“

Series: S2 Architecture Overview · Article 1 | Level: Intermediate | Reading Time: 18 min

TL;DR

coomia-dip decomposes the Palantir Foundry problem domain into 8 logical Layers covering deployment, control, data, reasoning, agent runtime, pipeline orchestration, governance, and SDK experience -- each owned by a single architect.
These 8 logical Layers are physically consolidated into 3 processes (onto-control.jar, onto-data.jar, onto-intelligence) via ARCH-001/002 decisions, achieving "logical decoupling with physical cohesion."
All inter-Layer communication uses gRPC + Protobuf exclusively (64 proto files). REST is reserved solely for external SDK facades -- internal REST is a hard architectural red line.

#1. Introduction: Why Build a Palantir Alternative?

Palantir Foundry is the benchmark in enterprise data platforms. It unifies data integration, ontology modeling, data governance, reasoning, and application building into a single platform. However, Foundry is closed-source, expensive, and raises data sovereignty concerns for organizations operating under strict regulatory environments.

coomia-dip (literally "Ontology PaaS" or "Intelligent Decision PaaS") aims to build an open-source, privately deployable Palantir alternative. But this is not about copying Foundry feature-by-feature -- it requires architectural decisions tailored to open-source ecosystems and enterprise deployment realities.

The first and most critical question: How do we decompose this massive problem domain?

Our answer: the 8-Layer Architecture.

#2. Problem Domain Decomposition: Why 8 Layers?

#2.1 Starting from Foundry's Capability Matrix

Foundry's core capabilities can be categorized into the following domains:

Code

+------------------------------------------------------------------+
|                   Palantir Foundry Capability Matrix               |
+------------------------------------------------------------------+
| 1. Data Integration & Pipelines | Pipeline Builder, Contour       |
| 2. Ontology Modeling            | Ontology Manager, Object Types  |
| 3. Data Governance              | Lineage, Audit, Classification  |
| 4. Reasoning & Decision         | Workshop, Logic Rules, AIP      |
| 5. Application Building         | Slate, Carbon, Actions          |
| 6. User Management & Authz      | RBAC, ABAC, Multipass           |
| 7. Deployment & Operations       | Apollo, Resource Management     |
| 8. Developer Experience          | OSDK, API, Developer Console    |
+------------------------------------------------------------------+

#2.2 What is a Layer?

We use the term Layer rather than Service or Module, inspired by networking architecture's Control Layer / Data Layer separation. Each Layer represents an Engineering Responsibility Domain with the following characteristics:

Independent technology selection: Free to choose the best-fit language and framework
Clear responsibility boundaries: Interactions with other Layers defined via gRPC contracts
Logical deployment independence: Can be developed and tested independently
Single architect ownership: One architect has full authority over each Layer

#2.3 The 8-Layer Overview

Code

+------------------------------------------------------------------------+
|                      Application Layer                                  |
|           Business Apps / Agents / Dashboards / Operations Tools        |
+------------------------------------------------------------------------+
         ^                                    |
         | REST/gRPC-Web                      | REST/gRPC-Web
         v                                    v
+------------------------------------------------------------------------+
| H. SDK & Developer Experience Layer                                     |
|    Python SDK / TypeScript OSDK / REST Facade / Developer Portal        |
+------------------------------------------------------------------------+
         ^
         | gRPC
         v
+------------------------------------------------------------------------+
| B. Control Layer              | G. Metadata & Governance Layer         |
|  - API Gateway                |  - Lineage Service                     |
|  - Auth (OAuth2+Policy)       |  - Audit Service                       |
|  - Schema Registry            |  - Replay Service                      |
|  - World Manager              |  - Metadata Service                    |
|  - User & Org Management      |  - Workflow Lineage                    |
+------------------------------------------------------------------------+
         ^
         | gRPC Internal Bus
         v
+--------------------+--------------------+------------------------------+
| C. Data Layer      | D. Reasoning &     | F. Pipeline &                |
|                    |    Decision Layer   |    Orchestration Layer       |
| - Ontology Runtime | - Reasoning Engine  | - Pipeline Engine            |
| - Query Federation | - Decision Engine   | - World Transform            |
| - Data Ingestion   | - Rule Engine       | - Scheduler                  |
| - Object Set       | - Derived Property  | - Trigger Rules              |
| - Vector Service   | - Function Runtime  | - Connection Management      |
| - Materialized View|                     |                              |
+--------------------+--------------------+------------------------------+
         |           | E. Agent Runtime    |
         |           |    Layer            |
         |           | - Action Engine     |
         |           | - AIP Logic         |
         |           | - Approval Workflow |
         |           | - Notification      |
         |           +---------------------+
         v
+------------------------------------------------------------------------+
| A. Platform Deployment & Ops Layer                                      |
|    Docker Compose / K8s Operator / Observability / Backup & Recovery    |
+------------------------------------------------------------------------+
         ^
         |
         v
+------------------------------------------------------------------------+
|                      Unified Storage Layer                               |
|  Apache Doris | Iceberg+Nessie | Kafka | Redis | PostgreSQL | MinIO    |
+------------------------------------------------------------------------+

#3. Each Layer's Responsibility Boundary

#3.1 Deployment & Operations Layer -- Platform Deployment & Ops

Tech Stack: Docker Compose / Python / YAML / Kubernetes Operator

Core Responsibilities:

Platform component orchestration and deployment (Docker Compose local / K8s production)
Observability integration (OpenTelemetry: Traces, Metrics, Logs)
Backup and recovery (Doris incremental backup, Nessie snapshots, MinIO cold archival)
Resource management and elastic scaling
Multi-tenant infrastructure isolation

Does NOT own: Business logic, data processing, authentication/authorization

#3.2 Control Layer

Tech Stack: Spring Boot 3.x / Java 21 / gRPC

Core Responsibilities:

API Gateway (routing, rate limiting, circuit breaking)
Authentication & Authorization (OAuth2 + mTLS + Policy Engine)
Schema Registry (ontology type definition registration and versioning)
World Manager (world branch creation, switching, merging -- backed by Nessie)
User & Organization Management (Tenant -> Org -> Space -> Project -> World hierarchy)
Configuration Management
Data Classification Service (7-level sensitivity auto-classification)

Key Proto Files: authentication_service.proto, schema_registry.proto, world_manager.proto, policy_engine.proto, user_management.proto, classification_service.proto -- 16 files total.

#3.3 Data Layer

Tech Stack: Quarkus 3.x / Java 21 / gRPC

Core Responsibilities:

OntologyRuntimeService: CRUD, batch operations, relation management for ontology instances -- the data kernel of the entire platform
Query Federation (federated queries across data sources)
Data Ingestion (Flink CDC -> Kafka -> Doris pipeline)
Object Set (object set operations: filter, aggregate, group)
Vector Service (vector storage and HNSW search)
Materialized View management
Time Series Store (time-series data storage and query)
Temporal Query (time-travel queries via Nessie commits)

Key Proto Files: ontology_runtime.proto, query_federation.proto, object_set.proto, data_ingestion.proto, vector_service.proto -- 15 files total.

#3.4 Reasoning & Decision Layer -- Reasoning & Decision

Tech Stack: Python 3.x / FastAPI / gRPC

Core Responsibilities:

Reasoning Engine (rule-based and AI-driven reasoning)
Decision Engine (multi-objective optimization, constraint solving)
Derived Property (derived property computation with 7 compute strategies)
Function Runtime (sandboxed user-defined function execution)
Rule Engine (low-code rule engine)
RAG Service (retrieval-augmented generation)
ML Model Service (model management and inference)
Approval Service

Key Proto Files: reasoning_engine.proto, decision_engine.proto, derived_property.proto, function_runtime.proto, rule_script.proto -- 14 files total.

#3.5 Agent Runtime Layer -- Agent Runtime

Tech Stack: Python 3.x / FastAPI / Temporal

Core Responsibilities:

Action Engine (side-effect execution engine)
AIP Logic Workflow (AI-powered logic workflows)
Approval Workflow (approval orchestration via Temporal)
Mutation Rules (pre-write validation and transformation)
Notification (multi-channel message delivery)

Key Proto Files: action_engine.proto, aip_logic_workflow.proto, approval_workflow.proto, mutation_rules.proto, notification.proto

#3.6 Pipeline & Orchestration Layer -- Pipeline & Orchestration

Tech Stack: Quarkus 3.x / DolphinScheduler / Java 21

Core Responsibilities:

Pipeline Engine (data pipeline definition and execution)
Transform Executor (data transformation execution)
Scheduler (schedule management, integrated with DolphinScheduler)
Connection Management (external data source connection management)
Trigger Rules (trigger rule definition)
Lineage (data lineage tracking)

Key Proto Files: pipeline_engine.proto, scheduler.proto, transform_executor.proto, connection_management.proto, trigger_rule.proto -- 8 files total.

#3.7 Metadata & Governance Layer -- Metadata & Governance

Tech Stack: Java / Spring Boot 3.x / gRPC

Core Responsibilities:

Lineage Service (end-to-end data lineage)
Audit Service (audit logging at attribute level)
Replay Service (operation replay for incident investigation)
Metadata Service (unified metadata management)
Workflow Lineage (workflow lineage tracking)

Key Proto Files: audit_service.proto, lineage_service.proto, metadata_service.proto, replay_service.proto, workflow_lineage.proto

#3.8 SDK & Developer Experience Layer -- SDK & Developer Experience

Tech Stack: Python SDK / TypeScript OSDK / REST

Core Responsibilities:

Python SDK (ontology_sdk package wrapping all gRPC calls into Pythonic API)
TypeScript OSDK (type-safe frontend SDK, mirroring Palantir's OSDK)
REST Facade (exposing gRPC services as REST API, 59 endpoints)
Developer Portal (documentation and API Explorer)

#4. Logical vs Physical: The 8 -> 3 Consolidation (ARCH-001/002)

#4.1 Why Consolidate?

Deploying 8 independent Layers sounds elegant, but creates serious operational problems:

Problem	Impact
8 independent processes = 8x operational cost	Small teams cannot sustain this
Cross-process gRPC call latency	Performance degradation on hot paths
Distributed transaction complexity	Data consistency becomes very hard
Resource fragmentation	Each process has fixed memory/CPU overhead
Deployment orchestration complexity	Startup ordering, health checks, dependency management

#4.2 ARCH-001: 2-JAR Consolidation (Java Side)

Code

Before consolidation (4 Java processes):
  control-Layer.jar       (Spring Boot)
  data-Layer.jar          (Quarkus)
  pipeline-Layer.jar      (Quarkus)
  governance-Layer.jar    (Spring Boot)

After consolidation (2 Java processes):
  onto-control.jar        (Spring Boot) = Control Layer + Metadata & Governance Layer
     Ports: 6666 (gRPC) / 6667 (REST)
  onto-data.jar           (Quarkus)     = Data Layer + Pipeline & Orchestration Layer
     Ports: 6668 (gRPC) / 6669 (REST/Flight SQL)

Consolidation Strategy:

Affinity-based merging: Combine Layers with the highest mutual call frequency
Tech stack consistency: Spring Boot Layers merge together, Quarkus Layers merge together
Logical isolation preserved: Code organization still follows Layer-based package paths

#4.3 ARCH-002: D+E Consolidation (Python Side)

Code

Before consolidation (2 Python processes):
  reasoning-Layer/       (FastAPI + gRPC)
  agent-runtime-Layer/   (FastAPI + Temporal)

After consolidation (1 Python process):
  onto-intelligence      (FastAPI + gRPC + Temporal)
     Ports: 6670 (gRPC) / 6671 (REST)

#4.4 Final Physical Topology

Code

+---------------------------------------------------------------+
|               Physical Deployment (3 Processes)                |
+---------------------------------------------------------------+
|                                                                 |
|  +---------------------------+  gRPC  +---------------------+  |
|  | onto-control.jar          |<------>| onto-data.jar       |  |
|  | (Spring Boot)             |        | (Quarkus)           |  |
|  |                           |        |                     |  |
|  | Control Layer: Control          |        | Data Layer: Data       |  |
|  |  - Schema Registry       |        |  - Ontology Runtime |  |
|  |  - World Manager         |        |  - Query Federation |  |
|  |  - Auth/Policy           |        |  - Data Ingestion   |  |
|  |  - User Management       |        |  - Object Set       |  |
|  |                           |        |                     |  |
|  | Metadata & Governance Layer: Governance       |        | Pipeline & Orchestration Layer: Pipeline   |  |
|  |  - Audit Service         |        |  - Pipeline Engine  |  |
|  |  - Lineage Service       |        |  - Scheduler        |  |
|  |  - Metadata Service      |        |  - Transform        |  |
|  |                           |        |                     |  |
|  | Ports: 6666/6667         |        | Ports: 6668/6669    |  |
|  +---------------------------+        +---------------------+  |
|              ^                                ^                 |
|              | gRPC                           | gRPC            |
|              v                                v                 |
|  +----------------------------------------------------------+  |
|  | onto-intelligence (FastAPI + gRPC)                        |  |
|  |                                                            |  |
|  | Reasoning & Decision Layer: Reasoning & Decision                              |  |
|  |  - Reasoning Engine, Decision Engine                       |  |
|  |  - Derived Property, Function Runtime                      |  |
|  |                                                            |  |
|  | Agent Runtime Layer: Agent Runtime                                     |  |
|  |  - Action Engine, AIP Logic, Approval Workflow             |  |
|  |                                                            |  |
|  | Ports: 6670/6671                                           |  |
|  +----------------------------------------------------------+  |
|                              |                                  |
|                              v                                  |
|  +----------------------------------------------------------+  |
|  |                   Storage Layer                            |  |
|  | Doris | Nessie+Iceberg | Kafka | Redis | PostgreSQL | MinIO|  |
|  +----------------------------------------------------------+  |
+---------------------------------------------------------------+

#4.5 Consolidation Trade-offs

Dimension	Before (8 processes)	After (3 processes)
JVM instances	4 JVMs	2 JVMs
Python processes	2	1
Base memory consumption	~4 GB	~1.5 GB
Deployment YAML complexity	High	Medium
Internal call latency	Cross-process gRPC (~1ms)	In-process method call (~0.01ms)
Code isolation	Physical isolation	Logical isolation (package paths)
Independent scaling	Fully independent	Co-located Layers scale together

#5. Inter-Layer Communication: gRPC Contract-Driven

#5.1 Communication Matrix

Code

              onto-control    onto-data    onto-intelligence
              (B+G)           (C+F)        (D+E)
            +---------------+------------+-----------------+
onto-control|  In-process   |   gRPC     |     gRPC        |
(B+G)       |               |            |                 |
            +---------------+------------+-----------------+
onto-data   |   gRPC        | In-process |     gRPC        |
(C+F)       |               |            |                 |
            +---------------+------------+-----------------+
onto-intel  |   gRPC        |   gRPC     |  In-process     |
(D+E)       |               |            |                 |
            +---------------+------------+-----------------+

#5.2 Proto File Organization

Code

proto/
├── common/
│   ├── common.proto          # WorldContext, RequestContext
│   ├── common_b.proto        # Control Layer shared types
│   └── errors.proto          # Unified error codes
├── plane_b/                  # 16 files
│   ├── authentication_service.proto
│   ├── schema_registry.proto
│   ├── world_manager.proto
│   └── ...
├── plane_c/                  # 15 files
│   ├── ontology_runtime.proto
│   ├── query_federation.proto
│   └── ...
├── plane_d/                  # 14 files
│   ├── reasoning_engine.proto
│   ├── decision_engine.proto
│   ├── derived_property.proto
│   └── ...
├── plane_e/                  # 5 files
│   ├── action_engine.proto
│   └── ...
├── plane_f/                  # 8 files
│   ├── pipeline_engine.proto
│   └── ...
└── plane_g/                  # 5 files
    ├── audit_service.proto
    └── ...

#5.3 WorldContext Permeates Everything

Every gRPC request must include a WorldContext. This is the platform's first-class concept:

PROTOBUF

message WorldContext {
  string world_id = 1;
  string branch_name = 2;
  optional string commit_hash = 3;
  optional Timestamp as_of_timestamp = 4;
  WorldType type = 5;
  map<string, string> metadata = 6;
  string tenant_id = 7;
  string org_id = 8;
  string project_id = 9;
  WorldBranchType world_branch_type = 10;
  string manifest_id = 11;
  optional string compute_version = 12;
}

There is no such thing as a "global operation" in coomia-dip -- every operation occurs within a specific world branch context.

#6. End-to-End Data Flow: From Ingestion to Decision

Let us trace a complete data flow to see how Layers collaborate:

Code

[External Data Source]
     |
     | (1) Flink CDC captures changes
     v
+----------+     (2) Message delivery    +----------+
|  Kafka   | --------------------------> |  Data Layer |
|          |                             |  Data    |
+----------+                             |  Ingestion|
                                          +----------+
                                               |
                      (3) Schema validation    | (4) Write to Doris
                      +----------+             |     entity_common
                      |  Control Layer |<------------+     entity_edge
                      |  Schema  |             |     entity_event
                      |  Registry|             |
                      +----------+             v
                                          +----------+
                                          |  Doris   |
                                          | (unified)|
                                          +----------+
                                               |
                      (5) Change events        |
                      +----------+       Kafka CDC
                      |  Metadata & Governance Layer |<------------+
                      |  Audit   |
                      +----------+
                                               |
                      (6) Trigger derived      |
                      +----------+       compute request
                      |  Reasoning & Decision Layer |<------------+
                      |  Derived |
                      |  Property|
                      +----------+
                           |
                      (7) Write back results
                           |
                           v
                      +----------+
                      |  Data Layer |
                      |  Ontology|
                      |  Runtime |
                      +----------+
                           |
                      (8) Notify subscribers
                           v
                      +----------+
                      |  Agent Runtime Layer |
                      |  Action  |
                      |  Engine  |
                      +----------+

Step-by-Step:

External data source changes captured via Flink CDC
Changes delivered to Kafka as Protobuf messages
Data Ingestion calls Schema Registry to validate data format
Validated data written to Doris three-table model
Change events pushed to Audit Service via Kafka for audit logging
Change events trigger Derived Property cascade recomputation
Computation results written back through Ontology Runtime
Final changes trigger Action Engine for business side effects (notifications, approvals, etc.)

#7. Comparison with Mainstream Architecture Patterns

#7.1 Comparison Table

Dimension	Microservices	Monolith	Modular Monolith	8-Layer
Deployment unit	Per-service	Single process	Single process	3 processes
Code isolation	Repository-level	Package-level	Module-level	Layer-level
Communication	HTTP/gRPC	Method calls	Method calls	gRPC + in-process
Tech heterogeneity	Fully free	Single stack	Single stack	Per-Layer selection
Team autonomy	High	Low	Medium	High
Operational complexity	High	Low	Low	Medium
Data consistency	Eventual	Strong	Strong	Hybrid
Independent scaling	Full	Impossible	Impossible	Layer-group level

#7.2 Our Position

The 8-Layer architecture is essentially a "tech-heterogeneous modular polyglot" pattern:

It has microservices' tech heterogeneity advantage (Java / Python / TypeScript)
It has modular monolith's deployment simplicity (3 processes instead of 20+)
It has a unique gRPC contract boundary (64 proto files defining all interactions)
It supports on-demand splitting (if Reasoning & Decision Layer becomes a bottleneck, deploy it independently)

#7.3 Why Not Pure Microservices?

Code

Pure microservices (hypothetical):

  schema-registry-service     (independent process)
  auth-service                (independent process)
  world-manager-service       (independent process)
  ontology-runtime-service    (independent process)
  query-federation-service    (independent process)
  data-ingestion-service      (independent process)
  reasoning-engine-service    (independent process)
  decision-engine-service     (independent process)
  action-engine-service       (independent process)
  pipeline-engine-service     (independent process)
  audit-service               (independent process)
  lineage-service             (independent process)
  ... (20+ total)

Problems:
  - Average request traverses 4-5 services = 4-5 network round trips
  - Distributed transactions everywhere
  - Small teams (< 10 people) cannot sustain operational cost
  - Local dev requires starting 20+ containers

#7.4 Why Not a Monolith?

The fatal problem with monolithic architecture is technology lock-in:

Reasoning & Decision Layer + Agent Runtime Layer's reasoning and AI capabilities require the Python ecosystem (PyTorch, LangChain, scikit-learn)
Control Layer + Data Layer's enterprise services require the Java ecosystem (Spring Security, Quarkus)
SDK & Developer Experience Layer's OSDK requires TypeScript

You simply cannot cover all scenarios with a single language.

#8. Architecture Decision Records (ADR) Summary

Key architecture decisions related to the 8-Layer architecture:

ADR	Decision	Status
ADR-001	Adopt Apache Doris as unified storage engine	Accepted
ADR-002	Project-World hierarchy management	Accepted
ADR-009	Python Layer consolidation (D+E -> onto-intelligence)	Accepted
ARCH-001	Java service consolidation (B+G -> onto-control, C+F -> onto-data)	Accepted
ARCH-002	Physical deployment simplified from 8 to 3 processes	Accepted

#9. Inter-Layer Call Frequency Analysis

By analyzing dependencies across 64 proto files, we derive the inter-Layer call heatmap:

Code

Call Frequency Matrix (estimated requests per second):

            B       C       D       E       F       G
      +-------+-------+-------+-------+-------+-------+
  B   |   -   |  500  |   50  |   30  |   20  |  100  |
      +-------+-------+-------+-------+-------+-------+
  C   |  200  |   -   |  300  |  100  |  150  |   50  |
      +-------+-------+-------+-------+-------+-------+
  D   |   30  |  400  |   -   |  200  |   10  |   20  |
      +-------+-------+-------+-------+-------+-------+
  E   |   50  |  200  |  150  |   -   |   30  |   40  |
      +-------+-------+-------+-------+-------+-------+
  F   |   20  |  300  |   10  |    5  |   -   |   80  |
      +-------+-------+-------+-------+-------+-------+
  G   |   30  |   50  |   10  |   10  |   30  |   -   |
      +-------+-------+-------+-------+-------+-------+

Hottest paths:
  B <-> C : 700 req/s  (Schema validation + Ontology queries)
  C <-> D : 700 req/s  (Derived property computation + data reads)
  D <-> E : 350 req/s  (Reasoning triggers Actions)

This call frequency analysis is the data foundation for the ARCH-001 consolidation decision -- merging the most frequently communicating Layers into the same process converts cross-process gRPC calls into in-process method invocations.

#10. Future Evolution Path

#10.1 Elastic Splitting

As the platform scales, individual Layers can be selectively split out:

Code

Phase 1 (current): 3 processes
  onto-control.jar (B+G)
  onto-data.jar    (C+F)
  onto-intelligence (D+E)

Phase 2 (medium-term): 4 processes
  onto-control.jar  (B+G)
  onto-data.jar     (C)
  onto-pipeline.jar (F)        <-- Pipeline split out
  onto-intelligence (D+E)

Phase 3 (long-term): 5 processes
  onto-control.jar  (B+G)
  onto-data.jar     (C)
  onto-pipeline.jar (F)
  onto-reasoning    (D)        <-- Reasoning split out
  onto-agent        (E)        <-- Agent split out

Because inter-Layer communication always uses gRPC contracts, splitting requires zero code changes -- only deployment configuration and service discovery addresses need updating.

#10.2 Cloud-Native Evolution

Code

Docker Compose (Dev)
     |
     v
Kubernetes (Staging/Prod)
     |
     v
Kubernetes + Karpenter (Auto-scaling)
     |
     v
Serverless Reasoning & Decision Layer + Agent Runtime Layer (Function-as-a-Service)

#Key Takeaways

Layers are engineering responsibility domains, not microservices. The 8-Layer decomposition is based on problem domain analysis, not technical decomposition. Each Layer has independent technology selection rights and single-architect ownership, but physically consolidates into 3 processes to reduce operational cost.
"Logical decoupling, physical cohesion" is the core strategy. gRPC contracts maintain logical independence between Layers while process consolidation delivers performance and operational benefits. The 64 proto files are the architecture's "constitution" -- they define all interaction boundaries.
WorldContext permeates every operation. This is not an optional parameter but the platform's fundamental assumption: no operation exists outside a world branch context. This design decision makes branch isolation, time travel, and scenario simulation first-class platform capabilities.

“

Next Article Preview: [S2-02] Why We Chose gRPC Over REST -- a deep dive into how 64 proto files are organized, the Protobuf contract-driven development workflow, cross-language code generation pipelines, and how the REST Gateway serves as an external facade.

Tags: #ontology #architecture #palantir #Layer #grpc #modular-monolith #coomia-dip