Palantir Data Branching Deep Dive: Managing Data Worlds Like Git

The Traditional Database Worldview
====================================

  Time T1:  inventory = 1000
                |
                | UPDATE inventory SET qty = 800
                v
  Time T2:  inventory = 800    (T1's state is gone forever)
                |
                | UPDATE inventory SET qty = 600
                v
  Time T3:  inventory = 600    (T2's state is also gone)

  Problems:
  1. Want to see T1's data? Sorry, it's gone
  2. Want to simulate "what if inventory drops to 200" without
     affecting production? Can't do it
  3. Want two teams to test different data change plans
     simultaneously? Impossible
  4. Want to safely merge an "experimental" data change
     into production? No mechanism for that

Capability Matrix Across Approaches
=====================================

                      Snapshot  Replica  TmpTbl  Temporal  Git-Branch
  View history          ~         x        x       Y         Y
  Create parallel       x         x        x       x         Y
    branches
  Independent           x         x        ~       x         Y
    modifications
  Compare two           x         x        x       x         Y
    branches
  Merge branch          x         x        x       x         Y
    back to main
  Conflict detection    x         x        x       x         Y
    & resolution
  Zero-copy (no         x         x        x       x         Y
    data duplication)

  Y = fully supported   ~ = partial   x = not supported

Git Code Branches vs. Palantir Data Branches
==============================================

Git (code):
  main ----o----o----o----o----o----o----> time
                     |                ^
                     | git branch     | git merge
                     v                |
  feature --------o----o----o---------+

Palantir (data):
  main ----[D1]--[D2]--[D3]--[D4]--[D5]--[D6]----> time
                        |                    ^
                        | branch             | merge
                        v                    |
  what-if --------[D3']--[D3'']--[D3''']-----+

  D1, D2... = different versions of data (dataset snapshots)
  D3' = modified version of D3 on the branch

Six Core Operations
====================

  1. BRANCH (Create Branch)
     Create an independent data copy from a point in time on main.
     Key: this is NOT a physical copy but a logical reference (zero-copy).

  2. MODIFY (Modify on Branch)
     Modify data on the branch independently, without affecting main.

  3. COMPARE (Diff Branches)
     View differences between a branch and main.

  4. MERGE (Merge Branch)
     Merge changes from a branch back into main.

  5. CONFLICT RESOLUTION
     When both main and branch modify the same record,
     conflicts must be resolved using three-way merge.

  6. TIME TRAVEL
     Query data state at any historical point in time.

     SELECT * FROM inventory AT TIMESTAMP '2025-03-15 10:30:00'

Supply Chain What-If Scenario
==============================

  1. Create a branch from main:
     main ---> branch "chip-shortage-scenario"

  2. Simulate supplier disruption on the branch:
     UPDATE suppliers SET status='DISRUPTED',
            delivery_capacity=0
     WHERE supplier_id = 'CHIP_VENDOR_A'
     ON BRANCH "chip-shortage-scenario"

  3. Let downstream derived properties auto-recompute:

     Main (Normal World):              Branch (Simulated World):
     +---------------------------+  +---------------------------+
     | Chip inventory: 50,000    |  | Chip inventory: 50,000    |
     | Daily consumption: 2,000  |  | Daily consumption: 2,000  |
     | Supplier delivery: 3,000  |  | Supplier delivery: 0      |
     | Days of stock: Restocking |  | Days of stock: STOCKOUT   |
     |                           |  |   IN 25 DAYS!!            |
     | Affected models: None     |  | Affected models: X, Y, Z  |
     | Affected orders: None     |  | Affected orders: 3,847    |
     | Est. loss: $0             |  | Est. loss: $284M          |
     +---------------------------+  +---------------------------+

  4. Test response plans on sub-branches:
     branch "chip-shortage-scenario"
       |
       +-- sub-branch "plan-A-switch-supplier"
       |     Result: stock days extended to 45, loss reduced to $120M
       |
       +-- sub-branch "plan-B-redesign-board"
       |     Result: 60 days for recertification, loss $95M
       |
       +-- sub-branch "plan-C-combined"
             Combine plan-A + plan-B
             Result: loss reduced to $45M (optimal plan)

  5. Decision: adopt plan-C-combined
     Merge contingency plan config from branch to main

Financial Stress Test Scenario
================================

  main (current market data)
    |
    +-- branch "rate-hike-300bp"
    |     Modify: benchmark rate 3.5% -> 6.5%
    |     Results: NPL 4.8%, capital adequacy 9.2%
    |
    +-- branch "rate-hike-200bp"
    |     Results: NPL 3.1%, capital adequacy 11.5%
    |
    +-- branch "rate-hike-500bp"
          Results: NPL 8.7%, capital adequacy 6.1%
                   (below regulatory threshold!)

  Three scenarios coexist, can be compared at any time.
  Delete branches when analysis is complete. Zero cost.

Data Change Git-Flow
=====================

  1. Data engineer creates a branch
  2. Execute data changes on the branch
  3. Automated validation generates impact report
  4. Submit Merge Request (like a Pull Request)
  5. Merge after approval with automatic audit log

Zero-Copy Branching Storage Internals
=======================================

  Storage comparison:
  +------------------------------------------+
  | Approach         | Cost of branching 1TB  |
  |------------------------------------------|
  | Full copy        | +1TB (100% overhead)   |
  | DB snapshot      | +200GB-1TB (varies)    |
  | Zero-copy branch | +1KB-100MB (delta only)|
  +------------------------------------------+

Three-Way Merge Algorithm
===========================

  Base (at branch creation):
    customer_001: {name: "Alice", credit: 750, city: "NYC"}

  Main (current):
    customer_001: {name: "Alice", credit: 780, city: "NYC"}

  Branch (current):
    customer_001: {name: "Alice Chen", credit: 750, city: "LA"}

  Three-way merge result:
    customer_001: {name: "Alice Chen", credit: 780, city: "LA"}
    (field-level merge: each field compared independently)

  Conflict resolution strategies:
  1. Auto: take the latest timestamp value
  2. Auto: take the higher value (conservative)
  3. Manual: flag conflict, let user decide
  4. Custom: business rules decide

Time-Travel Queries
====================

  -- View inventory on March 5
  SELECT * FROM inventory AT SNAPSHOT 'snap-002'

  -- View inventory trend from March 1 to March 15
  SELECT snapshot_time, qty FROM inventory
  BETWEEN SNAPSHOT 'snap-001' AND 'snap-004'

  -- Compare two points in time
  SELECT * FROM inventory
  DIFF BETWEEN 'snap-001' AND 'snap-004'

Three-Level Concept Mapping
=============================

  Concept               Nessie Concept     Git Analogy
  ================================================================
  World                 Repository         Repository
  Branch                Branch             Branch
  Release               Tag                Tag / Release

Format: {type}/{description}-{date-or-number}

Types:
- sim/     Simulation scenario
- fix/     Data fix
- etl/     ETL pipeline
- test/    Testing
- exp/     Experiment

Ontology Behavior on Branches
===============================

  main branch:
    Supplier CHIP_VENDOR_A: status="ACTIVE", risk_level="LOW"

  sim/chip-shortage branch:
    Supplier CHIP_VENDOR_A: status="DISRUPTED", risk_level="CRITICAL"
      (auto-recomputed!)

  Cascade effects (auto-propagated via LinkType):
    Supplier --SUPPLIES--> Product: chip_supply_status = "CRITICAL"
    Product --FULFILLS--> Order: at_risk = true
    Order --PLACED_BY--> Customer: satisfaction_risk = "HIGH"

  Modify one supplier's status on a branch,
  and derived properties across the entire Ontology
  graph auto-cascade and recompute.