Palantir's Branching: Managing Data Worlds Like Git
Traditional databases are fundamentally "single-world" systems -- there is one global copy of data, shared by all users as the single source of truth:
Palantir's Branching: Managing Data Worlds Like Git
“Series: S1 Palantir Decoded · Article 6 | Level: Beginner | Reading Time: 15 min
#TL;DR
- Traditional databases have only one timeline -- once you change data, it's changed, no undo. Palantir's Branching lets you manage data like Git manages code: fork a branch, run what-if analysis on the branch, then merge back to main if you're satisfied. This isn't snapshots or backups -- it's true parallel worlds.
- Branching unlocks three critical capabilities for enterprise data management: what-if scenario simulation (What happens if the supply chain breaks?), safe data change workflows (modify on a branch first, merge after approval), and time-travel queries (go back to last Tuesday and see what the data looked like). It is the "fourth dimension" of Ontology -- time.
- coomia-dip implements Palantir-equivalent data branching through Nessie + Apache Iceberg, using a three-level concept of World / Branch / Release to manage data versions, with zero-copy branching powered by Iceberg's snapshot isolation at the storage layer.
#1. Why Traditional Databases Can't Do What-If
#1.1 The Single-World Trap of Databases
Traditional databases are fundamentally "single-world" systems -- there is one global copy of data, shared by all users as the single source of truth:
The Traditional Database Worldview
====================================
Time T1: inventory = 1000
|
| UPDATE inventory SET qty = 800
v
Time T2: inventory = 800 (T1's state is gone forever)
|
| UPDATE inventory SET qty = 600
v
Time T3: inventory = 600 (T2's state is also gone)
Problems:
1. Want to see T1's data? Sorry, it's gone
2. Want to simulate "what if inventory drops to 200" without
affecting production? Can't do it
3. Want two teams to test different data change plans
simultaneously? Impossible
4. Want to safely merge an "experimental" data change
into production? No mechanism for that
#1.2 Limitations of Existing Approaches
Enterprises typically use the following workarounds, each with fatal flaws:
| Approach | Method | Flaw |
|---|---|---|
| DB snapshots | Periodic pg_dump / mysqldump | Full copy, storage explosion; no incremental merge |
| Read replicas | Primary-replica replication | Still one copy of data, can't fork |
| Temp tables | CREATE TABLE tmp_xxx AS SELECT... | Ad hoc, manual management, easily forgotten |
| Temporal tables | Temporal Table (SQL:2011) | Can only look back at history, can't fork branches |
| Test environments | Duplicate entire environment | Expensive, data sync is hard, merging is manual |
Capability Matrix Across Approaches
=====================================
Snapshot Replica TmpTbl Temporal Git-Branch
View history ~ x x Y Y
Create parallel x x x x Y
branches
Independent x x ~ x Y
modifications
Compare two x x x x Y
branches
Merge branch x x x x Y
back to main
Conflict detection x x x x Y
& resolution
Zero-copy (no x x x x Y
data duplication)
Y = fully supported ~ = partial x = not supported
#2. Git-Style Data Branching: Core Concepts
#2.1 From Code Version Control to Data Version Control
Git solved the core problem of code collaboration: multiple people modifying the same codebase without overwriting each other. Palantir's Branching applies the same idea to data:
Git Code Branches vs. Palantir Data Branches
==============================================
Git (code):
main ----o----o----o----o----o----o----> time
| ^
| git branch | git merge
v |
feature --------o----o----o---------+
Palantir (data):
main ----[D1]--[D2]--[D3]--[D4]--[D5]--[D6]----> time
| ^
| branch | merge
v |
what-if --------[D3']--[D3'']--[D3''']-----+
D1, D2... = different versions of data (dataset snapshots)
D3' = modified version of D3 on the branch
#2.2 Six Core Operations
Six Core Operations
====================
1. BRANCH (Create Branch)
Create an independent data copy from a point in time on main.
Key: this is NOT a physical copy but a logical reference (zero-copy).
main: [...1000 rows of data...]
|
| BRANCH "supply-chain-scenario-A"
v
branch-A: [points to main's snapshot, 0 bytes extra storage]
2. MODIFY (Modify on Branch)
Modify data on the branch independently, without affecting main.
main: [inventory=1000, price=50, supplier=A] (unchanged)
branch-A: [inventory=200, price=50, supplier=A] (only qty changed)
Actual storage: only "inventory: 1000->200" delta
3. COMPARE (Diff Branches)
View differences between a branch and main.
diff(main, branch-A):
inventory.qty: 1000 -> 200 (changed)
inventory.price: 50 -> 50 (unchanged)
inventory.supplier: A -> A (unchanged)
+ new_orders: [3 rows added on branch-A]
- deleted_items: [1 row removed on branch-A]
4. MERGE (Merge Branch)
Merge changes from a branch back into main.
main: [inventory=1000] ----merge----> [inventory=200]
branch-A: [inventory=200] (branch can be kept or deleted)
5. CONFLICT RESOLUTION
When both main and branch modify the same record,
conflicts must be resolved.
main: [inventory=1000] ----> [inventory=900] (someone removed 100)
branch-A: [inventory=1000] ----> [inventory=200] (branch set to 200)
Three-way merge:
base = 1000 (value when branch was created)
main = 900 (current main value)
branch = 200 (current branch value)
Strategy: automatic / latest wins / manual decision
6. TIME TRAVEL
Query data state at any historical point in time.
SELECT * FROM inventory AT TIMESTAMP '2025-03-15 10:30:00'
-- returns the inventory snapshot at 2025-03-15 10:30
#3. Real-World Scenarios: The Power of Data Branching
#3.1 Scenario 1: Supply Chain What-If Analysis
An automotive manufacturer needs to assess the impact of "What if our core chip supplier is disrupted for 3 months?":
Supply Chain What-If Scenario
==============================
1. Create a branch from main:
main ---> branch "chip-shortage-scenario"
2. Simulate supplier disruption on the branch:
UPDATE suppliers SET status='DISRUPTED',
delivery_capacity=0
WHERE supplier_id = 'CHIP_VENDOR_A'
ON BRANCH "chip-shortage-scenario"
3. Let downstream derived properties auto-recompute:
Main (Normal World): Branch (Simulated World):
+---------------------------+ +---------------------------+
| Chip inventory: 50,000 | | Chip inventory: 50,000 |
| Daily consumption: 2,000 | | Daily consumption: 2,000 |
| Supplier delivery: 3,000 | | Supplier delivery: 0 |
| Days of stock: Restocking | | Days of stock: STOCKOUT |
| | | IN 25 DAYS!! |
| Affected models: None | | Affected models: X, Y, Z |
| Affected orders: None | | Affected orders: 3,847 |
| Est. loss: $0 | | Est. loss: $284M |
+---------------------------+ +---------------------------+
4. Test response plans on sub-branches:
branch "chip-shortage-scenario"
|
+-- sub-branch "plan-A-switch-supplier"
| UPDATE suppliers SET ... (switch to backup supplier B)
| Result: stock days extended to 45, loss reduced to $120M
|
+-- sub-branch "plan-B-redesign-board"
| UPDATE bom SET chip_id = 'CHIP_ALT_002' (use alt chip)
| Result: 60 days for recertification, loss $95M
|
+-- sub-branch "plan-C-combined"
Combine plan-A + plan-B
Result: loss reduced to $45M (optimal plan)
5. Decision: adopt plan-C-combined
Merge contingency plan config from branch to main
main <--- merge "plan-C-combined" (merge config only, not sim data)
#3.2 Scenario 2: Financial Risk Scenario Modeling
A bank needs to run stress tests -- "What if interest rates rise 300 basis points?":
Financial Stress Test Scenario
================================
main (current market data)
|
+-- branch "rate-hike-300bp"
| Modify: benchmark rate 3.5% -> 6.5%
| Auto-recompute:
| - All loan repayment amounts
| - All bond market values
| - All customer debt-service scores
| - Overall NPL ratio forecast
|
| Results:
| +------------------------------+
| | NPL ratio: 1.2% -> 4.8% |
| | Capital adequacy: 14% -> 9.2%|
| | Affected customers: 23,000 |
| | Potential loss: $1.8B |
| +------------------------------+
|
+-- branch "rate-hike-200bp"
| Modify: benchmark rate 3.5% -> 5.5%
| Result: NPL 3.1%, capital adequacy 11.5%
|
+-- branch "rate-hike-500bp"
Modify: benchmark rate 3.5% -> 8.5%
Result: NPL 8.7%, capital adequacy 6.1%
(below regulatory threshold!)
Three scenarios coexist, can be compared at any time,
evolve independently.
Delete branches when analysis is complete. Zero cost.
#3.3 Scenario 3: Safe Data Change Workflows
Large enterprise data changes should never be made directly in production -- just like code should never be modified directly on main:
Data Change Git-Flow
=====================
1. Data engineer creates a branch:
main ---> branch "data-fix-customer-dedup"
2. Execute data changes on the branch:
-- Merge duplicate customer records
MERGE customers c1, c2
WHERE c1.email = c2.email AND c1.id != c2.id
ON BRANCH "data-fix-customer-dedup"
3. Automated validation:
+--------------------------------------+
| Change Impact Report |
|--------------------------------------|
| Affected records: 2,847 customers |
| Post-merge records: 1,423 customers |
| Linked orders: 12,384 updated refs |
| Linked contracts: 891 updated refs |
| Data quality score: 72% -> 94% |
+--------------------------------------+
4. Submit Merge Request (like a Pull Request):
Reviewers: Data owner + business owner
Before approval: view diff, validate on branch, run test queries
5. Merge after approval:
main <--- merge "data-fix-customer-dedup"
Audit log auto-created
#4. Technical Architecture: Storage Layer Internals
#4.1 Zero-Copy Branching: Why Storage Doesn't Explode
The most common question: "Doesn't branching require copying all data? Won't storage explode?"
The answer: No copying needed. Data branching uses Copy-on-Write (CoW):
Zero-Copy Branching Storage Internals
=======================================
Initial state: main branch has 1TB of data
File composition:
main/
data-file-001.parquet (100MB)
data-file-002.parquet (100MB)
...
data-file-100.parquet (100MB)
Total: ~10GB (after Parquet compression)
Create branch:
branch-A/
metadata.json -> points to all of main's files
(no data files copied)
Extra storage: ~1KB (metadata pointers only)
Modify 100 rows on the branch:
branch-A/
metadata.json -> points to 99 of main's files
-> points to 1 new file of its own
data-file-003-branch.parquet (100MB, replaces original file-003)
Extra storage: ~100MB (only the modified file)
Storage comparison:
+------------------------------------------+
| Approach | Cost of branching 1TB |
|------------------------------------------|
| Full copy | +1TB (100% overhead) |
| DB snapshot | +200GB-1TB (varies) |
| Zero-copy branch | +1KB-100MB (delta only)|
+------------------------------------------+
#4.2 Three-Way Merge
Merging data is more complex than merging code because data has "semantics":
Three-Way Merge Algorithm
===========================
Scenario: both main and branch modified the same record
Base (at branch creation):
customer_001: {name: "Alice", credit: 750, city: "NYC"}
Main (current):
customer_001: {name: "Alice", credit: 780, city: "NYC"}
(credit score changed from 750 to 780)
Branch (current):
customer_001: {name: "Alice Chen", credit: 750, city: "LA"}
(name changed, city changed)
Three-way merge result:
customer_001: {name: "Alice Chen", credit: 780, city: "LA"}
(field-level merge: each field compared independently)
Merge rules:
+--------+---------+---------+--------+----------+
| Field | Base | Main | Branch | Result |
+--------+---------+---------+--------+----------+
| name | Alice | Alice | Alice | Alice |
| | | | Chen | Chen |
+--------+---------+---------+--------+----------+
| credit | 750 | 780 | 750 | 780 |
| | |(changed)| | (main) |
+--------+---------+---------+--------+----------+
| city | NYC | NYC | LA | LA |
| | | |(changed)| (branch)|
+--------+---------+---------+--------+----------+
Conflict scenario (same field changed by both):
Base: credit = 750
Main: credit = 780
Branch: credit = 800
Conflict resolution strategies:
1. Auto: take the latest timestamp value
2. Auto: take the higher value (conservative)
3. Manual: flag conflict, let user decide
4. Custom: business rules decide (e.g., average of both)
#4.3 Time-Travel Queries
Every data change creates an immutable snapshot. Any historical point in time can be queried:
Time-Travel Queries
====================
Data timeline:
T1 (Mar 1) T2 (Mar 5) T3 (Mar 10) T4 (Mar 15) NOW
[snap-001] [snap-002] [snap-003] [snap-004] [latest]
| | | | |
qty=1000 qty=800 qty=600 qty=900 qty=850
Query examples:
-- View inventory on March 5
SELECT * FROM inventory
AT SNAPSHOT 'snap-002'
-- Result: qty=800
-- View inventory trend from March 1 to March 15
SELECT snapshot_time, qty
FROM inventory
BETWEEN SNAPSHOT 'snap-001' AND 'snap-004'
-- Result: [(3/1, 1000), (3/5, 800), (3/10, 600), (3/15, 900)]
-- Compare two points in time
SELECT * FROM inventory
DIFF BETWEEN 'snap-001' AND 'snap-004'
-- Result: qty: 1000 -> 900 (changed)
Snapshot retention policy:
+------------------------------------------+
| Time Range | Retention Granularity |
|------------------------------------------|
| Last 7 days | Every commit retained |
| 7-30 days | One snapshot per day |
| 30-365 days | One snapshot per week |
| Over 1 year | One snapshot per month|
+------------------------------------------+
#5. coomia-dip Data Branching: Nessie + Iceberg
#5.1 Technology Selection
coomia-dip uses two key open-source components to implement data branching:
coomia-dip Data Branching Tech Stack
=====================================
+--------------------------------------------------+
| Application Layer (Python SDK / API) |
| client.branch("what-if-scenario") |
| client.switch_branch("what-if-scenario") |
| client.query("SELECT * FROM inventory") |
| client.merge("what-if-scenario", into="main") |
+--------------------------------------------------+
|
| gRPC
v
+--------------------------------------------------+
| Control Layer (World Manager) |
| |
| World: A complete business data universe |
| Branch: A data timeline within a World |
| Release: An immutable snapshot on a Branch |
| |
| WorldManager.create_branch(world, name, from) |
| WorldManager.merge_branch(source, target) |
| WorldManager.create_release(branch, tag) |
+--------------------------------------------------+
|
| gRPC
v
+--------------------------------------------------+
| Data Layer (Nessie + Iceberg) |
| |
| Nessie: Git-like version control server |
| - Manages branches / tags / commit history |
| - RESTful API for branch operations |
| - Atomic multi-table commits |
| |
| Iceberg: Table format |
| - Snapshot isolation (one snapshot per commit) |
| - Zero-copy branching (shared data files) |
| - Time-travel queries |
| - Schema evolution |
+--------------------------------------------------+
|
v
+--------------------------------------------------+
| Storage Layer (MinIO / S3 / HDFS) |
| |
| /warehouse/ |
| /inventory/ |
| /data/ |
| file-001.parquet |
| file-002.parquet |
| file-003-branch-a.parquet |
| /metadata/ |
| snap-001.avro |
| snap-002.avro |
+--------------------------------------------------+
#5.2 World / Branch / Release: Three-Level Concepts
coomia-dip builds a business semantic layer on top of Nessie:
Three-Level Concept Mapping
=============================
coomia-dip Concept Nessie Concept Git Analogy
================================================================
World Repository Repository
Branch Branch Branch
Release Tag Tag / Release
Typical usage:
+----------------------------------------------------------+
| |
| World: "supply-chain" |
| |
| main ----[R1.0]----[R1.1]----[R2.0]--------> (prod) |
| | ^ |
| | branch | merge |
| v | |
| dev ---------o----o----o----o--------+ (dev branch) |
| | |
| | branch |
| v |
| scenario-A -----------o----o (sim, delete when done) |
| |
| [R1.0] = Release 1.0 (immutable snapshot for audit) |
+----------------------------------------------------------+
#5.3 Code Example: Data Branching with the Python SDK
from ontology_sdk import OntoPlatform
# Connect to platform
platform = OntoPlatform("http://localhost:8080")
# 1. Create a branch
platform.worlds.get("supply-chain").create_branch(
name="chip-shortage-sim",
from_branch="main",
description="Simulate chip supplier disruption scenario"
)
# 2. Modify data on the branch
with platform.branch("chip-shortage-sim") as ctx:
# Update supplier status
supplier = ctx.objects.get("Supplier", "CHIP_VENDOR_A")
ctx.actions.execute("UpdateSupplierStatus", {
"supplier": supplier,
"status": "DISRUPTED",
"delivery_capacity": 0,
"disruption_reason": "Geopolitical risk simulation"
})
# View auto-recomputed derived properties
affected = ctx.objects.filter("Product",
chip_supply_status="CRITICAL"
)
print(f"Affected products: {len(affected)}")
for product in affected:
print(f" {product.name}: "
f"inventory_days={product.derived.inventory_days}, "
f"affected_orders={product.derived.affected_orders}")
# 3. Compare branch vs. main
diff = platform.worlds.get("supply-chain").compare(
source="chip-shortage-sim",
target="main"
)
print(f"Total changes: {diff.total_changes}")
print(f"Affected object types: {diff.affected_object_types}")
# 4. Create a Release (immutable snapshot)
platform.worlds.get("supply-chain").create_release(
branch="chip-shortage-sim",
tag="sim-v1.0",
description="Chip disruption simulation - initial scenario"
)
# 5. Time-travel query
historical = platform.worlds.get("supply-chain").at_release("R1.0")
old_inventory = historical.objects.filter("Inventory",
product_category="semiconductor"
)
print(f"Chip inventory at R1.0: {sum(i.quantity for i in old_inventory)}")
# 6. Merge branch (if deciding to adopt simulation config)
platform.worlds.get("supply-chain").merge(
source="chip-shortage-sim",
target="main",
strategy="FIELD_LEVEL", # field-level three-way merge
conflict_resolution="MANUAL" # manual conflict resolution
)
#6. Comprehensive Comparison with Traditional Approaches
| Dimension | DB Snapshots | Temporal Tables (SQL:2011) | Data Lake Time Travel | Palantir Branching | coomia-dip (Nessie+Iceberg) |
|---|---|---|---|---|---|
| Create branches | Full copy | Not supported | Not supported | Zero-copy | Zero-copy |
| Parallel branches | Storage-limited | N/A | N/A | Unlimited | Unlimited |
| Modify on branch | Separate instance | N/A | N/A | In-place | In-place |
| Three-way merge | Not supported | Not supported | Not supported | Supported | Supported |
| Time travel | Not supported | Supported (row-level) | Supported (snapshot) | Supported (snapshot) | Supported (snapshot) |
| Cross-table atomicity | Depends | Not supported | Not supported | Supported | Supported (Nessie) |
| Ontology integration | None | None | None | Deep | Deep |
| Open source | Depends on DB | Depends on DB | Partial (Delta/Iceberg) | No | Yes |
#7. Data Branching Best Practices
#7.1 Branch Naming Conventions
Branch Naming Convention
=========================
Format: {type}/{description}-{date-or-number}
Types:
- sim/ Simulation scenario
- fix/ Data fix
- etl/ ETL pipeline
- test/ Testing
- exp/ Experiment
Examples:
- sim/chip-shortage-2025Q2
- fix/customer-dedup-batch-003
- etl/daily-load-20250315
- test/new-pricing-model
- exp/ml-feature-engineering-v2
#7.2 Branch Lifecycle Management
Branch Lifecycle
=================
Create --> Use --> Review --> Merge or Discard --> Cleanup
+--------------------+------------------+------------------+
| Branch Type | Suggested TTL | Cleanup Strategy |
+--------------------+------------------+------------------+
| Simulation (sim/) | 1-4 weeks | Create Release, |
| | | then delete |
+--------------------+------------------+------------------+
| Data fix (fix/) | 1-3 days | Delete after |
| | | merge |
+--------------------+------------------+------------------+
| ETL (etl/) | Auto daily | Auto-delete on |
| | create/delete | success |
+--------------------+------------------+------------------+
| Testing (test/) | 1-2 weeks | Delete after |
| | | test completion |
+--------------------+------------------+------------------+
| Experiment (exp/) | 1-3 months | Periodic review |
| | | then decide |
+--------------------+------------------+------------------+
#8. Data Branching and Ontology Synergy
Data branching is not an isolated feature -- it deeply integrates with Ontology:
Ontology Behavior on Branches
===============================
main branch:
+------------------------------------------+
| ObjectType: Supplier |
| CHIP_VENDOR_A: |
| status = "ACTIVE" |
| delivery_capacity = 3000 |
| risk_level [derived] = "LOW" |
+------------------------------------------+
sim/chip-shortage branch:
+------------------------------------------+
| ObjectType: Supplier |
| CHIP_VENDOR_A: |
| status = "DISRUPTED" |
| delivery_capacity = 0 |
| risk_level [derived] = "CRITICAL" |
| <-- auto-recomputed! |
+------------------------------------------+
| |
| Cascade effects (auto-propagated via |
| LinkType): |
| |
| Supplier --SUPPLIES--> Product |
| Product.chip_supply_status = "CRITICAL"|
| Product.inventory_days = 25 |
| |
| Product --FULFILLS--> Order |
| Order.at_risk = true |
| Order.delay_estimate = "30-45 days" |
| |
| Order --PLACED_BY--> Customer |
| Customer.affected_orders = 12 |
| Customer.satisfaction_risk = "HIGH" |
+------------------------------------------+
Modify one supplier's status on a branch,
and derived properties across the entire Ontology
graph auto-cascade and recompute.
This is the power of Branching + Ontology.
#Key Takeaways
-
Data branching is the "Git moment" for enterprise data management. Just as Git revolutionized code collaboration, data branching lets enterprises safely run what-if simulations, parallel experiments, and controlled changes on production data for the first time. Zero-copy technology makes branch creation nearly free, and three-way merge lets branch results safely merge back to main.
-
Data branching + Ontology = simulating entire business worlds. Standalone data branching is just "copying data," but combined with Ontology, a single modification on a branch cascades through LinkType and DerivedProperty across the entire business graph. This elevates what-if analysis from "change one number, see one result" to "change one variable, see the chain reaction across the entire world."
-
coomia-dip delivers open-source data branching through Nessie + Iceberg. Using the World/Branch/Release three-level concept for data version management, with Iceberg snapshot isolation for zero-copy branching and Nessie for Git-style branch management APIs. This means enterprises don't need a Palantir commercial license to get equivalent data version control capabilities.
#Next Article
“S1-07: Palantir AIP: When LLMs Meet the Enterprise Data Operating System
If Ontology is Palantir's soul and Branching is its time-travel superpower, then AIP is the "brain" that drives it all with natural language. In 2023, Palantir launched AIP (Artificial Intelligence Platform), letting LLMs understand enterprise data through Ontology, execute business operations through Actions, and ensure safety through permissions. Next up, we dive deep into how Palantir turned LLMs from "chatbots" into "the intelligence layer of an enterprise operating system."
Tags: #Palantir #Branching #DataVersioning #WhatIfAnalysis #TimeTravel #Nessie #Iceberg #coomia-dip #GitForData #ThreeWayMerge