“

Series: S1 Palantir Decoded · Article 20 | Level: Beginner | Reading Time: 15 min

Why We're Building an Open-Source Palantir — and Our Roadmap

#TL;DR

Palantir cannot serve the majority of the world's enterprises due to geopolitics, data sovereignty regulations, and pricing that starts at $1M+/year -- yet the core problem it solves (turning data into decisions) is universal and urgent
coomia-dip takes an open-source, first-principles, Ontology-driven approach, delivering Palantir Foundry-equivalent capabilities through an 8-Layer architecture -- with 6,000+ tests, 95% backend readiness, and 109 distinct capabilities already built
Our vision: make Palantir-grade intelligent decision infrastructure accessible to every enterprise on earth, not just the Fortune 500

#Table of Contents

Why Palantir Doesn't Serve Most of the World
The Universal Problem
Our Approach: Open-Source, First-Principles, Ontology-Driven
The 8-Layer Architecture
Three Processes, One Platform
The Storage Layer
Current Status: Where We Stand
Technology Choices and Why
What's Done, What's Next
How to Get Involved
Vision: Palantir-Grade Capabilities for Every Enterprise

#1. Why Palantir Doesn't Serve Most of the World

Over the previous 19 articles in this series, we have dissected Palantir's products, architecture, business model, and market performance in detail. One inescapable conclusion emerges: Palantir does not serve -- and structurally cannot serve -- the majority of the world's enterprises. This is not a temporary gap; it is a permanent architectural constraint rooted in geopolitics, regulation, and business strategy.

#Three Layers of Exclusion

Code

+-------------------------------------------------------------------+
| Layer 1: Geopolitical Alignment                                    |
|                                                                    |
| - Palantir's core customers are the US DoD, CIA, NSA, and         |
|   Five Eyes intelligence agencies                                  |
| - CEO Alex Karp openly advocates "Western values" alignment        |
| - Corporate charter explicitly excludes operations in China        |
|   and Russia                                                       |
| - US export controls (EAR/ITAR) restrict technology transfer       |
| - Operating in certain markets would jeopardize government         |
|   contracts worth billions                                         |
+-------------------------------------------------------------------+
                    |
                    v
+-------------------------------------------------------------------+
| Layer 2: Data Sovereignty Regulations                              |
|                                                                    |
| - China's Data Security Law (2021) mandates data localization      |
| - EU GDPR imposes strict cross-border transfer requirements        |
| - India's DPDP Act requires local data storage for certain data    |
| - Brazil's LGPD, Indonesia's PDP Law -- each nation is building   |
|   data sovereignty walls                                           |
| - Palantir's cloud-centric architecture does not support fully     |
|   air-gapped, on-premises deployment                               |
+-------------------------------------------------------------------+
                    |
                    v
+-------------------------------------------------------------------+
| Layer 3: Market Strategy and Pricing                               |
|                                                                    |
| - Palantir focuses on North America + Europe + Five Eyes           |
| - Annual contracts start at $1M, enterprise deals reach $100M+    |
| - Requires Palantir Forward Deployed Engineers (FDEs) on-site     |
| - Many markets need completely different sales/deployment models   |
| - Local competitors reduce entry motivation                        |
+-------------------------------------------------------------------+

#This Is Not Just About One Country

Palantir's exclusion zone is far broader than any single nation:

Code

Markets Palantir Does Not or Cannot Fully Serve:
+----------------------+---------------------------------------------+
| Market               | Reason                                      |
+----------------------+---------------------------------------------+
| China                | Geopolitics + active exclusion               |
| Russia               | Sanctions + active exclusion                 |
| Middle East (parts)  | Export controls                              |
| Southeast Asia (most)| Market strategy + deployment cost            |
| Africa               | Market size + deployment complexity          |
| South America (most) | Limited coverage                             |
| India (partially)    | Data localization + price sensitivity        |
| SMEs globally        | Price floor ($1M+/year)                      |
+----------------------+---------------------------------------------+

= Roughly 70% of global enterprises cannot use Palantir

The result is a massive gap. The technology exists. The problems it solves are universal. But access is restricted to a narrow slice of the world's organizations.

#2. The Universal Problem

Palantir does not solve an American problem or a defense problem. It solves a universal enterprise problem: how to turn "having data" into "making decisions."

#The Dilemma Every Organization Faces

Code

Every enterprise has data, but...
                         |
          +--------------+--------------+
          |              |              |
     Data in silos   Data doesn't    Data doesn't
          |           connect          drive action
          |              |              |
    +----------+   +-----------+   +----------+
    | ERP      |   | Can't     |   | Reports  |
    | CRM      |   | answer    |   | are read  |
    | WMS      |   | cross-    |   | then      |
    | MES      |   | system    |   | forgotten |
    | Custom   |   | business  |   | No action |
    | systems  |   | questions |   | taken     |
    +----------+   +-----------+   +----------+

    "I have 100        "Which           "The CEO says
     systems but       customers        data-driven but
     they don't        are affected     we only achieved
     talk to each      by the supply    data-displayed"
     other"            chain issue?"

#Palantir's Solution (Abstracted)

Code

Palantir's core solution boils down to four steps:

Step 1: Data Integration
  100 siloed systems --> Unified data layer
  "All data in one place, kept fresh"

Step 2: Semantic Modeling (Ontology)
  Raw data --> Business objects + relationships
  "Not tables and rows, but Customers, Orders, Factories, Suppliers"

Step 3: Intelligent Analysis (Reasoning + AI)
  Business objects --> Rules + reasoning + AI
  "Automatically detect anomalies, calculate impact, recommend decisions"

Step 4: Operational Closure (Actions)
  Decisions --> Automated execution
  "Not just dashboards, but triggering real-world operations"

+----------+     +---------+     +----------+     +--------+
| Data     | --> | Ontology | --> | Reasoning| --> | Action |
| Integrate|     | Model   |     | + AI     |     | Execute|
+----------+     +---------+     +----------+     +--------+

These four steps have nothing to do with geography or culture.
Every enterprise needs them.

#Specific Needs That Palantir Cannot Address

Beyond the universal problem, enterprises in regulated or sovereignty-sensitive markets have additional requirements that Palantir's architecture fundamentally cannot satisfy:

Code

+-------------------------------+----------------------------------------+
| Requirement                   | Why Palantir Falls Short               |
+-------------------------------+----------------------------------------+
| Data must stay on-premises    | Cloud-centric architecture             |
| Support local databases       | Tied to proprietary storage layer      |
| Air-gapped deployment         | Requires internet connectivity         |
| Local language NLP            | English-centric AI models              |
| Regulatory compliance (local) | Designed for US/EU compliance only     |
| Adaptable to local cloud      | AWS/Azure-centric                     |
| Controllable tech stack       | Proprietary, cannot be audited         |
| Affordable pricing            | $1M+ annual minimum                   |
+-------------------------------+----------------------------------------+

#3. Our Approach: Open-Source, First-Principles, Ontology-Driven

coomia-dip is not a copy of Palantir. We took a first-principles approach, asking: "If we were building an Ontology-driven intelligent decision platform from scratch today, what would it look like?"

#Three Founding Principles

Code

Principle 1: Open-Source First
+-------------------------------------------------------------------+
| "If Ontology is the infrastructure of enterprise intelligence,     |
|  then it should be open -- just as Linux is to operating systems." |
|                                                                    |
| - Core platform is fully open-source                               |
| - Community-driven development                                     |
| - Transparent development process                                  |
| - Enterprise edition provides commercial support and advanced      |
|   features for organizations that need them                        |
+-------------------------------------------------------------------+

Principle 2: First-Principles Thinking
+-------------------------------------------------------------------+
| "Don't copy Palantir's product. Solve the problem Palantir solves."|
|                                                                    |
| - Start from the problem, not the existing solution                |
| - Every technology choice has a documented reason                  |
| - Neither blindly follow nor blindly oppose                        |
| - In some areas, go deeper than Palantir                           |
|   (e.g., native reasoning engine integration)                      |
+-------------------------------------------------------------------+

Principle 3: Ontology-Driven
+-------------------------------------------------------------------+
| "The Ontology is not an optional add-on. It is the kernel of the   |
|  entire platform."                                                 |
|                                                                    |
| - All operations execute within a World Context                    |
| - All data is semantically modeled through the Ontology            |
| - All decisions trace back to rules and relationships in the       |
|   Ontology                                                         |
| - AI/LLMs understand the business through the Ontology             |
+-------------------------------------------------------------------+

#How coomia-dip Compares to Palantir

Code

+------------------+-------------------------------+-------------------+
| Dimension        | Palantir Foundry              | coomia-dip         |
+------------------+-------------------------------+-------------------+
| Source code       | Closed                        | Open-source       |
| Deployment        | SaaS + limited on-prem        | Self-hosted first |
| Target customers  | Fortune 500 / governments     | All enterprises   |
| Annual cost       | $1M - $100M+                  | Free (core)       |
| Data sovereignty  | Data on Palantir's cloud      | Full data control |
| Tech controllable | Proprietary technology        | Fully auditable   |
| Community         | No open community             | Developer community|
| AI integration    | AIP (added 2023)              | Native (Day 1)    |
| Reasoning engine  | Limited (mostly in Workshop)  | Native engine     |
| Scenario sim.     | World (restricted)            | World Fork (full) |
+------------------+-------------------------------+-------------------+

#4. The 8-Layer Architecture

coomia-dip uses an 8-Layer architecture. Each Layer is an independent engineering responsibility domain that can be developed, tested, and deployed independently. The Layers communicate exclusively via gRPC -- never REST -- ensuring type safety, contract-first design, and high-performance binary serialization across language boundaries.

#Architecture Overview

Code

+====================================================================+
|                          coomia-dip                                  |
|              Ontology-driven Intelligent Decision PaaS              |
+====================================================================+
|                                                                     |
|  +-------------------------------+  +----------------------------+  |
|  |  SDK & Developer Experience Layer: SDK & Dev Exp.     |  |  Deployment & Operations Layer: Deployment & Ops |  |
|  |  Python SDK (OntoPlatform)    |  |  Docker Compose / K8s      |  |
|  |  TypeScript OSDK Generator    |  |  Monitoring & Alerting     |  |
|  |  28 sub-modules, 39 gRPC     |  |  Platform Console          |  |
|  +-------------------------------+  +----------------------------+  |
|                  |                              |                   |
|                  v                              v                   |
|  +==============================================================+  |
|  |                    gRPC Gateway Layer                         |  |
|  +==============================================================+  |
|                  |                                                  |
|      +-----------+------------+--------------+                      |
|      |           |            |              |                      |
|      v           v            v              v                      |
|  +--------+  +--------+  +--------+  +----------+                  |
|  |Control Layer |  |Data Layer |  |Reasoning & Decision Layer |  | Pipeline & Orchestration Layer  |                  |
|  |Control |  | Data   |  | +E Int.|  | Pipeline |                  |
|  | Layer  |  | Layer  |  | Runtime|  |  & Orch  |                  |
|  |        |  |        |  |        |  |          |                  |
|  |Spring  |  |Quarkus |  |FastAPI |  | Quarkus  |                  |
|  |Boot 3  |  | 3.x    |  |Python  |  | Dolphin  |                  |
|  |Java 21 |  |Java 21 |  | 3.x   |  | Scheduler|                  |
|  |        |  |        |  |        |  |          |                  |
|  |Schema  |  |Query   |  |Reason  |  |Pipeline  |                  |
|  |Action  |  |Storage |  |Decision|  |Schedule  |                  |
|  |Policy  |  |Sync    |  |Agent   |  |Transform |                  |
|  |Object  |  |Search  |  |Function|  |Connection|                  |
|  |Metric  |  |Export  |  |Rule    |  |          |                  |
|  |Notif.  |  |Analyti.|  |Approval|  |          |                  |
|  +--------+  +--------+  +--------+  +----------+                  |
|      |           |            |                                     |
|      +-----+-----+-----+-----+                                     |
|            |           |                                            |
|            v           v                                            |
|  +==============================================================+  |
|  |  Metadata & Governance Layer: Metadata & Governance                              |  |
|  |  Audit Service | Classification | Lineage | Compliance       |  |
|  +==============================================================+  |
|                         |                                           |
|                         v                                           |
|  +==============================================================+  |
|  |                  Unified Storage Layer                        |  |
|  |  +--------+  +--------+  +-------+  +-------+  +---------+  |  |
|  |  | Doris  |  | Iceberg|  | Nessie|  | Redis |  | Kafka   |  |  |
|  |  | (OLAP+ |  | (Versnd|  | (Git- |  | (Cache|  | (Event  |  |  |
|  |  |  Vector|  |  Tables)|  |  like |  |  +Pub/ |  |  Stream)|  |  |
|  |  |  +Full |  |        |  |  Branc|  |  Sub)  |  |         |  |  |
|  |  |  Text) |  |        |  |  hing)|  |       |  |         |  |  |
|  |  +--------+  +--------+  +-------+  +-------+  +---------+  |  |
|  +==============================================================+  |
+====================================================================+

#What Each Layer Does

Code

+-------+----------------------------+----------------------------------+
| Layer | Name                       | Core Responsibilities            |
+-------+----------------------------+----------------------------------+
| A     | Platform Deployment & Ops  | Deployment orchestration,        |
|       |                            | monitoring, alerting,            |
|       |                            | K8s platform console             |
+-------+----------------------------+----------------------------------+
| B     | Control Layer              | Ontology Schema management,      |
|       |                            | Action/Policy/Object management, |
|       |                            | auth, notifications              |
+-------+----------------------------+----------------------------------+
| C     | Data Layer                 | Query engine, storage sync,      |
|       |                            | full-text search, analytics,     |
|       |                            | data export, real-time subscr.   |
+-------+----------------------------+----------------------------------+
| D+E   | Intelligence Runtime       | Rule reasoning, decision trees,  |
|       | (Reasoning + Agent)        | Agent runtime, Functions,        |
|       |                            | approval workflows, simulation   |
+-------+----------------------------+----------------------------------+
| F     | Pipeline & Orchestration   | Data pipelines, scheduling,      |
|       |                            | transformations, connectors      |
+-------+----------------------------+----------------------------------+
| G     | Metadata & Governance      | Audit logs, data classification, |
|       |                            | data lineage, compliance mgmt    |
+-------+----------------------------+----------------------------------+
| H     | SDK & Developer Experience | Python SDK, TypeScript OSDK,     |
|       |                            | code generators, dev docs        |
+-------+----------------------------+----------------------------------+

#Architectural Design Principles

Code

1. World Context Is a First-Class Concept
   All operations execute within a Project + World context.
   Worlds can be Forked (like Git branches), enabling scenario simulation.

2. gRPC First (Internal Communication)
   All inter-Layer communication uses gRPC. Never REST.
   gRPC provides: type safety + streaming + high performance + codegen.

3. Contract-Driven
   Layers define contracts via .proto files.
   Any Layer can be developed and tested independently.

4. Version Everything
   Data, Schema, Pipelines, Functions, Rules, Code -- all versioned.
   Forking a World snapshots all artifact versions, ensuring
   100% replayable simulations.

5. SDK Is the Capability Exposure Layer
   The SDK carries no business logic; it wraps gRPC calls.
   All business logic lives in the backend Layers.

#5. Three Processes, One Platform

Despite having 8 logical Layers, coomia-dip deploys as just three processes in production. This keeps operational complexity low while preserving clean architectural boundaries in code.

Code

+====================================================================+
|                    Deployment Topology                              |
+====================================================================+
|                                                                     |
|  Process 1: onto-control.jar                                       |
|  +--------------------------------------------------------------+  |
|  |  Framework: Spring Boot 3.x  |  Language: Java 21            |  |
|  |                                                               |  |
|  |  Contains:                                                    |  |
|  |  +------------------+  +--------------------+                 |  |
|  |  | Control Layer          |  | Metadata & Governance Layer            |                 |  |
|  |  | Control Layer    |  | Metadata &         |                 |  |
|  |  |                  |  | Governance         |                 |  |
|  |  | - Schema mgmt    |  | - Audit service    |                 |  |
|  |  | - Action mgmt    |  | - Classification   |                 |  |
|  |  | - Policy mgmt    |  | - Data lineage     |                 |  |
|  |  | - Object mgmt    |  | - Compliance       |                 |  |
|  |  | - Auth & RBAC    |  |                    |                 |  |
|  |  | - Notifications  |  |                    |                 |  |
|  |  +------------------+  +--------------------+                 |  |
|  +--------------------------------------------------------------+  |
|                                                                     |
|  Process 2: onto-data.jar                                          |
|  +--------------------------------------------------------------+  |
|  |  Framework: Quarkus 3.x  |  Language: Java 21                |  |
|  |                                                               |  |
|  |  Contains:                                                    |  |
|  |  +------------------+  +--------------------+                 |  |
|  |  | Data Layer          |  | Pipeline & Orchestration Layer            |                 |  |
|  |  | Data Layer       |  | Pipeline &         |                 |  |
|  |  |                  |  | Orchestration      |                 |  |
|  |  | - Query engine   |  | - Pipeline CRUD    |                 |  |
|  |  | - Storage sync   |  | - Scheduling       |                 |  |
|  |  | - Full-text srch |  | - Transformations  |                 |  |
|  |  | - Analytics      |  | - Connectors       |                 |  |
|  |  | - Data export    |  |                    |                 |  |
|  |  | - Subscriptions  |  |                    |                 |  |
|  |  +------------------+  +--------------------+                 |  |
|  +--------------------------------------------------------------+  |
|                                                                     |
|  Process 3: onto-intelligence                                      |
|  +--------------------------------------------------------------+  |
|  |  Framework: FastAPI  |  Language: Python 3.x                  |  |
|  |                                                               |  |
|  |  Contains:                                                    |  |
|  |  +------------------+  +--------------------+                 |  |
|  |  | Reasoning & Decision Layer          |  | Agent Runtime Layer            |                 |  |
|  |  | Reasoning &      |  | Agent Runtime      |                 |  |
|  |  | Decision         |  |                    |                 |  |
|  |  | - Rule engine    |  | - Agent lifecycle  |                 |  |
|  |  | - Decision trees |  | - Temporal wkflows |                 |  |
|  |  | - Streaming infer|  | - Function exec    |                 |  |
|  |  | - Approval flows |  | - LLM integration  |                 |  |
|  |  | - Simulation     |  |                    |                 |  |
|  |  +------------------+  +--------------------+                 |  |
|  +--------------------------------------------------------------+  |
|                                                                     |
|  Communication: All three processes talk via gRPC (binary,          |
|  type-safe, streaming). Never REST between services.               |
+====================================================================+

This three-process model means you can run the entire platform on a single machine for development (via Docker Compose) or scale each process independently on Kubernetes for production.

#6. The Storage Layer

coomia-dip uses a carefully chosen set of storage technologies, each selected to solve a specific problem while minimizing operational complexity.

Code

+====================================================================+
|                     Storage Architecture                            |
+====================================================================+
|                                                                     |
|  +------------------------------------------------------------+    |
|  | Apache Doris (Primary OLAP + Vector + Full-Text)           |    |
|  |                                                             |    |
|  | Role: Unified analytical storage engine                     |    |
|  | Why:  One engine for OLAP queries, vector similarity        |    |
|  |       search, and full-text search. Eliminates the need     |    |
|  |       to sync data between ClickHouse + Qdrant + Elastic.  |    |
|  | Used by: Data Layer (queries, search, analytics)               |    |
|  +------------------------------------------------------------+    |
|                                                                     |
|  +------------------------------------------------------------+    |
|  | Apache Iceberg + Project Nessie (Versioned Data Tables)    |    |
|  |                                                             |    |
|  | Role: Git-like versioning for data tables                   |    |
|  | Why:  World Fork requires data snapshots. Nessie provides   |    |
|  |       branch/merge semantics. Iceberg provides snapshot     |    |
|  |       isolation and time travel. Zero-copy fork is O(1).    |    |
|  | Used by: Data Layer (versioned storage), Reasoning & Decision Layer (simulation)  |    |
|  +------------------------------------------------------------+    |
|                                                                     |
|  +------------------------------------------------------------+    |
|  | Apache Kafka (Event Streaming)                              |    |
|  |                                                             |    |
|  | Role: Event bus for async communication                     |    |
|  | Why:  Audit events, data change notifications, pipeline     |    |
|  |       triggers, real-time subscriptions.                    |    |
|  | Used by: Metadata & Governance Layer (audit), Data Layer (subscriptions)           |    |
|  +------------------------------------------------------------+    |
|                                                                     |
|  +------------------------------------------------------------+    |
|  | Redis (Cache + Pub/Sub)                                     |    |
|  |                                                             |    |
|  | Role: Caching layer and lightweight pub/sub                 |    |
|  | Why:  Schema caching, session management, real-time         |    |
|  |       notifications, rate limiting.                         |    |
|  | Used by: Control Layer (cache), SDK & Developer Experience Layer (SDK cache)               |    |
|  +------------------------------------------------------------+    |
|                                                                     |
|  +------------------------------------------------------------+    |
|  | PostgreSQL (Metadata Store)                                 |    |
|  |                                                             |    |
|  | Role: Relational metadata storage                           |    |
|  | Why:  Ontology Schema definitions, Action definitions,      |    |
|  |       user/role management -- strong ACID transactions.     |    |
|  | Used by: Control Layer (schema), Metadata & Governance Layer (governance metadata)    |    |
|  +------------------------------------------------------------+    |
|                                                                     |
|  +------------------------------------------------------------+    |
|  | MinIO (Object Storage)                                      |    |
|  |                                                             |    |
|  | Role: S3-compatible object storage                          |    |
|  | Why:  Pipeline artifacts, data exports, large file storage, |    |
|  |       Iceberg data files in on-prem deployments.            |    |
|  | Used by: Data Layer (exports), Pipeline & Orchestration Layer (pipeline artifacts)    |    |
|  +------------------------------------------------------------+    |
+====================================================================+

The key insight is Doris as the unified analytical engine. Instead of operating three separate systems (ClickHouse for OLAP, Qdrant for vectors, Elasticsearch for full-text), we use one engine that handles all three workloads. This eliminates data synchronization headaches, reduces operational burden, and guarantees real-time consistency across query types.

#7. Current Status: Where We Stand

As of March 2026, coomia-dip has completed the vast majority of its core backend development.

#Overall Progress

Code

+-------+---------------------------+-------+------------+-----------+
| Layer | Name                      | Done  | Code Files | Tests     |
+-------+---------------------------+-------+------------+-----------+
| A     | Deployment & Ops          | 20%   | Docker     | -         |
| B     | Control Layer             | 90%   | 211+ Java  | 1,238     |
| C     | Data Layer                | 98%   | 520+ Java  | 1,961     |
| D+E   | Intelligence Runtime      | 98%   | 200+ Py    | 2,519     |
| F     | Pipeline & Orchestration  | 92%   | 100+ Java  | 20+       |
| G     | Metadata & Governance     | 87%   | 4 Services | 16+       |
| H     | SDK & Developer Exp.      | 99%   | 125+ Py+TS | 475       |
+-------+---------------------------+-------+------------+-----------+
| Total |                           | ~95%  | 1,156+     | 6,229+    |
+-------+---------------------------+-------+------------+-----------+

#109 Capabilities Across 10 Domains

coomia-dip currently implements 109 independent capabilities that cover the core feature set of Palantir Foundry:

Code

Ontology Management (16 capabilities):
  Object Type CRUD | Link Type CRUD | Property management |
  Schema versioning | Constraint definitions | Derived properties | ...

Data Operations (18 capabilities):
  Object instance CRUD | Batch operations | Conditional queries |
  Sorting & pagination | Relationship traversal | Aggregation |
  Time series | TopN | Distribution analysis | ...

Action & Function (12 capabilities):
  Action Type definitions | Action execution | Parameter validation |
  Function registration | Function execution | Code binding generation | ...

Reasoning & Decision (15 capabilities):
  RuleSet CRUD | Rule reasoning | Streaming inference |
  Decision tree CRUD | Decision execution | Simulation |
  Approval workflows | Trigger rules | Change rules | ...

Scenario Simulation (6 capabilities):
  World CRUD | World Fork | Branch merge |
  Snapshot comparison | Simulation replay | ...

Search & Analytics (12 capabilities):
  Full-text search (6 modes) | Faceted statistics | Spell suggestions |
  Saved searches | Analytical aggregation | Time bucketing | ...

Data Pipeline (8 capabilities):
  Pipeline CRUD | Schedule management | Connectors |
  Transformation definitions | Execution monitoring | ...

Governance & Security (10 capabilities):
  Audit logs | Data classification | Data lineage |
  Policy management | Permission control | ...

Platform Operations (6 capabilities):
  Metric management | Notifications | Dashboards |
  Data export | Real-time subscriptions | ...

SDK & Developer Tools (6 capabilities):
  Python SDK | TypeScript OSDK | Async client |
  gRPC clients (39) | Code generation | ...

#Design Documentation

Code

Completed detailed design documents: 116+ specs
+------------------+---------+
| Category         | Count   |
+------------------+---------+
| Control Layer specs    | 40      |
| Data Layer specs    | 25      |
| Reasoning & Decision Layer + Agent Runtime Layer specs  | 17      |
| Pipeline & Orchestration Layer specs    | 23      |
| Metadata & Governance Layer specs    | 5       |
| SDK & Developer Experience Layer specs    | 6       |
+------------------+---------+

#8. Technology Choices and Why

Every technology choice in coomia-dip has a documented rationale. We do not adopt technologies because they are trendy. We adopt them because they solve specific problems better than the alternatives.

#Why gRPC, Not REST?

Code

+------------------+------------------+------------------------+
| Dimension        | REST/JSON        | gRPC/Protobuf          |
+------------------+------------------+------------------------+
| Type safety      | None (JSON is    | Yes (.proto defines    |
|                  | untyped)         | contracts)             |
+------------------+------------------+------------------------+
| Code generation  | Requires OpenAPI | Native support         |
+------------------+------------------+------------------------+
| Streaming        | Not natively     | Native bidirectional   |
|                  | supported        | streaming              |
+------------------+------------------+------------------------+
| Performance      | Text serializa-  | Binary serialization   |
|                  | tion             | (10x faster)           |
+------------------+------------------+------------------------+
| Contract-first   | Documentation-   | Schema-first           |
|                  | first            |                        |
+------------------+------------------+------------------------+
| Cross-language   | Needs SDK        | Native cross-language  |
|                  | wrappers         | code generation        |
+------------------+------------------+------------------------+

Decision: All internal service communication uses gRPC.
         SDK also exposes gRPC interfaces (higher performance).
         HTTP REST is available only as an optional gateway layer.

#Why Apache Doris, Not ClickHouse + Qdrant + Elasticsearch?

Code

Traditional approach: Multiple engines stitched together
+------------+  +----------+  +---------+
| ClickHouse |  | Qdrant   |  | Elastic |
| (OLAP)     |  | (Vector) |  | (Search)|
+------------+  +----------+  +---------+
      |              |             |
  Need to sync   Need to sync  Need to sync
      |              |             |
  +--------------------------------------+
  | Data consistency? Sync lag? OpEx?     |
  +--------------------------------------+

coomia-dip approach: Unified engine
+--------------------------------------------+
|              Apache Doris                   |
|                                             |
|  OLAP Analysis + Vector Search + Full-Text  |
|  (Columnar)     (ANN Index)    (Inverted)   |
|                                             |
|  One copy of data, three query capabilities |
|  Zero sync lag, zero consistency issues     |
+--------------------------------------------+

Why this matters:
- Operational complexity: 1 component vs. 3 components
- Data consistency: Real-time vs. eventually consistent
- Development speed: Single SQL dialect vs. three APIs
- Community: Doris has strong ecosystem and commercial support

#Why Nessie + Iceberg for Versioning?

Code

The versioning requirement:
- World Fork needs data snapshots
- Simulation/rollback needs historical versions
- Schema changes need to be reversible
- Parallel branches need isolation

Nessie + Iceberg solution:
+-------------------------------------------+
|  Nessie (Git-like version control)        |
|  +-------+  +-------+  +-------+         |
|  | main  |  | fork1 |  | fork2 |  ...    |
|  | branch|  | branch|  | branch|         |
|  +-------+  +-------+  +-------+         |
|      |          |          |              |
|      v          v          v              |
|  Iceberg (versioned table format)         |
|  - Snapshot isolation: each branch sees   |
|    its own version of the data            |
|  - Time travel: go back to any point      |
|  - Schema evolution: seamlessly add/drop  |
|  - Zero-copy fork: branch creation is O(1)|
+-------------------------------------------+

This aligns closely with Palantir's World/Branch mechanism,
but uses open-source components instead of proprietary tech.

#Why Python + FastAPI for the Intelligence Layer?

Code

+-------------------+-------------------------------------------+
| Choice            | Reason                                    |
+-------------------+-------------------------------------------+
| Python 3.x        | Richest AI/ML ecosystem (PyTorch,         |
|                   | Transformers, LangChain)                  |
+-------------------+-------------------------------------------+
| FastAPI           | Async-native, auto OpenAPI docs,          |
|                   | Pydantic type validation                  |
+-------------------+-------------------------------------------+
| gRPC (grpcio)     | Unified communication protocol with       |
|                   | Java Layers                               |
+-------------------+-------------------------------------------+
| Pydantic v2       | High-performance data validation,         |
|                   | JSON Schema generation                    |
+-------------------+-------------------------------------------+

Why not Spring Boot for intelligence?
- AI/ML libraries are immature in Java
- Python development is significantly faster for reasoning tasks
- Data scientists are fluent in Python, not Java
- Rule engine DSLs are more naturally expressed in Python

#Why Spring Boot for the Control Layer?

Code

+-------------------+-------------------------------------------+
| Choice            | Reason                                    |
+-------------------+-------------------------------------------+
| Spring Boot 3.x   | Highest enterprise maturity and ecosystem |
| Java 21           | Virtual Threads for concurrent performance|
| Spring Data JPA   | Best fit for Schema CRUD operations       |
| Spring Security   | Mature auth framework                     |
| gRPC-Spring       | Seamless gRPC + Spring integration        |
+-------------------+-------------------------------------------+

Why not Python for control?
- Schema management needs strong transactional support
- Java's type system better suits metadata management
- Spring ecosystem excels at enterprise middleware integration
- Shares Java ecosystem with Data Layer (Quarkus)

#Complete Technology Stack

Code

+------------------+--------------------+---------------------------+
| Layer            | Technology         | Purpose                   |
+------------------+--------------------+---------------------------+
| SDK              | Python 3.x         | Primary SDK language      |
|                  | TypeScript         | Frontend OSDK             |
|                  | Protobuf           | API contract definitions  |
+------------------+--------------------+---------------------------+
| Control Layer    | Spring Boot 3.x    | Metadata management       |
|                  | Java 21            | Core runtime              |
|                  | Spring Data JPA    | ORM                       |
|                  | gRPC-Spring        | Service communication     |
+------------------+--------------------+---------------------------+
| Data Layer       | Quarkus 3.x        | Data services             |
|                  | Java 21            | Core runtime              |
|                  | Apache Doris       | Unified storage engine    |
|                  | Iceberg + Nessie   | Versioned data tables     |
+------------------+--------------------+---------------------------+
| Intelligence     | FastAPI            | API framework             |
|                  | Python 3.x         | Reasoning/Decision/Agent  |
|                  | Pydantic v2        | Data models               |
|                  | Temporal           | Workflow orchestration    |
+------------------+--------------------+---------------------------+
| Pipeline         | Quarkus 3.x        | Pipeline services         |
|                  | DolphinScheduler   | Scheduling engine         |
+------------------+--------------------+---------------------------+
| Governance       | Java + gRPC        | Audit/Classification/     |
|                  | Kafka              | Lineage, event consuming  |
+------------------+--------------------+---------------------------+
| Infrastructure   | Docker Compose     | Development environment   |
|                  | Kubernetes         | Production environment    |
|                  | Redis              | Cache + pub/sub           |
|                  | Kafka              | Event streaming           |
|                  | PostgreSQL         | Metadata store            |
|                  | MinIO              | Object storage            |
+------------------+--------------------+---------------------------+

#9. What's Done, What's Next

#Completed Milestones

Code

Phase 1: Foundation (2025 Q3-Q4) -- COMPLETE
  - 8-Layer architecture design
  - Multi-Agent collaboration specification
  - Development toolchain setup
  - Sprint/Release workflow established

Phase 2: Core Capabilities (2025 Q4 - 2026 Q1) -- COMPLETE
  - Ontology Schema CRUD (Control Layer)
  - Object instance management (Control Layer + Data Layer)
  - Data query engine (Data Layer)
  - Rule reasoning engine (Reasoning & Decision Layer)
  - Decision tree engine (Reasoning & Decision Layer)
  - Python SDK v1.0 (SDK & Developer Experience Layer)

Phase 3: Advanced Features (2026 Q1 - Q2) -- IN PROGRESS
  - Action execution chain (Control Layer)
  - Full-text search (Data Layer)
  - Analytical aggregation queries (Data Layer)
  - Real-time subscriptions (Data Layer)
  - Data export (Data Layer)
  - Approval workflows (Reasoning & Decision Layer)
  - Audit service (Metadata & Governance Layer)
  - SDK: all 39 gRPC clients wired (SDK & Developer Experience Layer)

#Planned Milestones

Code

Phase 4: Productization (2026 Q2-Q3) -- PLANNED
  +---+------------------------------------------+----------+
  | # | Task                                     | Priority |
  +---+------------------------------------------+----------+
  | 1 | Frontend UI (React + TypeScript)         | P0       |
  | 2 | TypeScript OSDK code generator           | P0       |
  | 3 | Derived property dependency DAG          | P1       |
  | 4 | K8s Operator and deployment orchestration | P1       |
  | 5 | Monitoring & alerting (Prometheus+Grafana)| P1       |
  | 6 | Pipeline visualization                   | P2       |
  | 7 | Ontology visual browser                  | P2       |
  +---+------------------------------------------+----------+

Phase 5: Enterprise-Grade (2026 Q3-Q4) -- PLANNED
  +---+------------------------------------------+----------+
  | # | Task                                     | Priority |
  +---+------------------------------------------+----------+
  | 1 | Multi-tenancy isolation                  | P0       |
  | 2 | Full RBAC + ABAC implementation          | P0       |
  | 3 | Data masking engine                      | P1       |
  | 4 | Compliance frameworks (SOC2, local regs) | P1       |
  | 5 | Additional database adapters             | P1       |
  | 6 | LLM integration (AIP-equivalent layer)   | P0       |
  | 7 | Documentation and tutorials              | P1       |
  +---+------------------------------------------+----------+

Phase 6: Ecosystem (2027 Q1+) -- LONG-TERM
  +---+------------------------------------------+----------+
  | # | Task                                     | Priority |
  +---+------------------------------------------+----------+
  | 1 | Ontology template marketplace            | P1       |
  | 2 | Industry solution templates              | P1       |
  | 3 | Third-party connector ecosystem          | P2       |
  | 4 | Developer certification program          | P2       |
  | 5 | Community contribution guide and process  | P1       |
  +---+------------------------------------------+----------+

#10. How to Get Involved

coomia-dip is an open-source project, and we welcome contributors of all kinds.

#Ways to Contribute

Code

+---+---------------------------+----------------------------------+
| # | Method                    | Who It's For                     |
+---+---------------------------+----------------------------------+
| 1 | Star & Fork               | Everyone                         |
| 2 | File Issues               | Users who find problems           |
| 3 | Submit PRs                | Developers                       |
| 4 | Write documentation       | Technical writers                |
| 5 | Design industry solutions | Domain experts                   |
| 6 | Translate                 | Multilingual contributors        |
| 7 | Test and give feedback    | Early adopters                   |
| 8 | Spread the word           | Tech bloggers and speakers       |
+---+---------------------------+----------------------------------+

#Contribute to a Layer

The 8-Layer architecture naturally supports distributed collaboration -- each Layer can be owned by an independent team or individual:

Code

If you're skilled in...          You can contribute to...
+-----------------------------+--------------------------------+
| Java + Spring Boot          | Control Layer (Control Layer)        |
| Java + Quarkus              | Data Layer (Data) / F (Pipeline)  |
| Python + FastAPI            | Reasoning & Decision Layer + Agent Runtime Layer (Intelligence)       |
| Python SDK development      | SDK & Developer Experience Layer (SDK)                  |
| TypeScript + React          | Frontend UI / TypeScript OSDK  |
| DevOps + Kubernetes         | Deployment & Operations Layer (Deployment)           |
| Data governance / compliance| Metadata & Governance Layer (Governance)           |
| Technical writing           | Documentation and tutorials    |
+-----------------------------+--------------------------------+

#Setting Up the Development Environment

Bash

# 1. Clone the repository
git clone https://github.com/your-org/coomia-dip.git
cd coomia-dip

# 2. Read the essential docs
cat CLAUDE.md       # Project conventions
cat AGENTS.md       # Collaboration rules
cat PROGRESS.md     # Current progress

# 3. Start the dev environment
docker-compose up -d  # Start dependency services

# 4. Pick your Layer
# Control Layer (Java)
cd control-Layer && ./gradlew build

# Data Layer (Java)
cd data-Layer && ./gradlew build

# Intelligence Layer (Python)
cd intelligence-Layer && pip install -e . && pytest

# SDK (Python)
cd python-sdk && pip install -e . && pytest

#11. Vision: Palantir-Grade Capabilities for Every Enterprise

#The Current Inequity

Code

Today's reality:

  Fortune 500 enterprises:
  +--------------------------------------------+
  | Palantir Foundry                           |
  | Annual fee: $1M-$100M+                     |
  | Dedicated Forward Deployed Engineers       |
  | Full Ontology + AI + Decision capabilities |
  | --> Competitive edge: 10x decision speed   |
  +--------------------------------------------+

  The other 99% of enterprises:
  +--------------------------------------------+
  | Excel + BI tools + disconnected systems    |
  | Data trapped in silos                      |
  | Decisions made by gut feeling              |
  | AI is "something other companies have"     |
  | --> Competitive disadvantage: 10x slower   |
  +--------------------------------------------+

#The Future We Want

Code

Our vision:

  Every enterprise:
  +--------------------------------------------+
  | coomia-dip (open-source)                    |
  | Free to use (core platform)               |
  | Self-hosted, full data control             |
  | Full Ontology + AI + Decision capabilities |
  | --> Democratized: Palantir-grade for all   |
  +--------------------------------------------+

  The analogy:
  - Linux made server-grade OS accessible to everyone
  - Kubernetes made container orchestration accessible to every team
  - PostgreSQL made enterprise-grade databases free
  - coomia-dip makes Ontology-driven intelligent decision
    infrastructure accessible to every enterprise

#Why This Is Worth Building

Code

1. The Last Mile of Digital Transformation

   Most enterprises have already:
   [done] Moved to the cloud
   [done] Built data lakes / warehouses
   [done] Deployed BI tools

   But they are stuck on:
   [stuck] Data --> Decision conversion
   [stuck] Landing AI safely in production
   [stuck] Cross-system business object modeling

   coomia-dip solves this last mile.

2. Open-Source Is the Right Model

   Ontology-driven decision platforms should be infrastructure.
   And infrastructure should be open.

   Just as:
   - OS: Windows (closed) --> Linux (open)
   - Database: Oracle (closed) --> PostgreSQL (open)
   - Container orchestration: proprietary --> Kubernetes (open)
   - Decision platform: Palantir (closed) --> coomia-dip (open)

3. Every Market Deserves Access

   Data sovereignty regulations are proliferating globally.
   No single vendor can serve every jurisdiction.
   Open-source, self-hosted platforms are the only architecture
   that works everywhere:
   - Compliant with any local data regulation
   - Adaptable to any cloud or on-prem environment
   - Auditable by any security team
   - Affordable for any size organization

#Key Takeaways

Palantir structurally cannot serve roughly 70% of the world's enterprises due to geopolitics, data sovereignty, and pricing -- yet the "data to decisions" problem it solves is universal and urgent. coomia-dip uses an open-source model to make this capability accessible to organizations that Palantir will never reach.
coomia-dip's 8-Layer architecture and technology choices are not imitation but first-principles engineering. gRPC over REST, Doris over ClickHouse+Qdrant, Nessie+Iceberg for versioning, Python+FastAPI for intelligence -- each choice has a clear, documented engineering rationale. The platform deploys as just three processes while maintaining clean architectural boundaries.
With 6,000+ tests, 95% backend readiness, 109 capabilities, and 116+ design specs, coomia-dip is not a concept or a plan -- it is a rapidly maturing product. The foundation is built. What comes next is productization (UI, K8s Operator), enterprise hardening (multi-tenancy, compliance), and ecosystem growth (industry templates, community).

#Series Wrap-Up: From S1 to S2

This article marks the final entry in the S1 series (Palantir Decoded). Over 20 articles, we have explored Palantir from every angle -- its history, products (Gotham, Foundry), architecture, components (Pipeline Builder, Contour, Workshop, Apollo), data lineage, search and discovery, OSDK developer experience, stock performance, and competitive landscape.

Code

S1 Series Recap (20 articles):
+------+-------------------------------------------+
| #    | Topic                                     |
+------+-------------------------------------------+
| 01   | What is Palantir?                         |
| 02   | Gotham vs. Foundry                        |
| ...  | ...                                       |
| 17   | Competitive landscape analysis            |
| 18   | Stock: from $6 to $80                     |
| 19   | OSDK developer experience                 |
| 20   | Why we're building open-source Palantir   |
|      | (this article)                            |
+------+-------------------------------------------+

The question S1 answered was: "What is Palantir, and why does it matter?"

The question S2 will answer is: "How do you actually build it?"

Code

S2 Preview: coomia-dip Architecture Deep-Dive Series
+------+-------------------------------------------+
| #    | Topic                                     |
+------+-------------------------------------------+
| 01   | 8-Layer Architecture: Why This Design?    |
| 02   | gRPC Over REST: The Engineering Case      |
| 03   | Storage Architecture: Doris + Iceberg     |
| 04   | Control Layer Deep-Dive                   |
| 05   | Data Layer: The Unified Query Engine      |
| ...  | More to come                              |
+------+-------------------------------------------+

S2 is for developers who want to understand, use, or contribute to coomia-dip. If S1 was "why," S2 is "how."

“

S2-01: 8-Layer Architecture: Why This Design?

A deep dive into coomia-dip's architectural decision process. Why 8 Layers instead of microservices? How are Layer boundaries determined? How are gRPC contracts designed? How do multiple teams develop in parallel? From ADRs (Architecture Decision Records) to running code -- the complete chain.

tags: coomia-dip, open-source, Palantir, Ontology, 8-Layer architecture, roadmap, gRPC, Apache Doris, Nessie, Iceberg, FastAPI, Spring Boot, Quarkus, data sovereignty, enterprise intelligence, manifesto