Palantir's Data Lineage Deep Dive: Where Does Every Value Come From?

Data Lineage: A Food Safety Analogy
================================================

Olive Oil Traceability:
  Olive tree (Andalusia, Spain, Plot #A17)
    -> Harvested (2024-10-15, Worker Team #3)
    -> Pressed (cold-press, Machine #M7, 27 C)
    -> Filtered (triple filtration, Batch #B2024-1015)
    -> Bottled (500ml, Line #L2, Batch #F-88721)
    -> QA tested (acidity 0.3%, passed)
    -> Shipped (cold chain, Vehicle #T-9981)
    -> Supermarket shelf (the bottle you see)

Data Lineage:
  Raw data (ERP system, table sales_orders)
    -> Cleaned (deduplicated, null handling, Pipeline #P-001)
    -> Transformed (currency conversion, unit normalization, Transform #T-023)
    -> Aggregated (by region, monthly cumulative, Transform #T-024)
    -> Joined (with customer dimension table, Transform #T-025)
    -> Modeled (calculate customer LTV, Model #M-012)
    -> Displayed (CEO dashboard: "Monthly Revenue: $12.5M")

Entity-Level Lineage (Dataset-Level)
================================================

Tracks "relationships between datasets":

  Dataset A ------> Transform 1 ------> Dataset C
                                           |
  Dataset B ------> Transform 2 -----------+

  Question: "Which upstream datasets does Dataset C depend on?"
  Answer: "Dataset A and Dataset B"

  Granularity: Coarse  |  Cost: Low  |  Value: Moderate


Field-Level Lineage (Column-Level)
================================================

Tracks "relationships between fields":

  Dataset A                    Dataset C
  +----------------+          +----------------+
  | order_id       |--------->| order_id       |
  | raw_amount     |--+       |                |
  | currency       |--+-Xform>| amount_usd     |
  +----------------+  |       |                |
                      |       |                |
  Dataset B           |       |                |
  +----------------+  |       |                |
  | exchange_rate  |--+       |                |
  | customer_name  |--------->| customer_name  |
  +----------------+          +----------------+

  Question: "How was amount_usd calculated?"
  Answer: "raw_amount x exchange_rate, where raw_amount comes from
           Dataset A and exchange_rate comes from Dataset B"

  Granularity: Fine  |  Cost: High  |  Value: Very High

Compliance Scenario
================================================

Scenario: Banking Regulatory Review

Regulator: "The non-performing loan ratio of 2.3% you reported --
            how was this number calculated?"

Without lineage:
  Analyst: "Uh... let me check... probably from System A..."
  (3 days later) "We confirmed it's from System A, but there was
   manual processing by Department B. We need to verify the logic..."
  (1 week later) "Sorry, the person who owned this process left..."

  Result: Regulatory penalty, reputation damage

With lineage:
  Analyst: (clicks the number) "Here's the full calculation chain:
  1. Source: Core banking system, loan_master table
  2. Cleaning: Removed test accounts (Pipeline #P-1234, last run 2024-03-15)
  3. Classification: Overdue >90 days = non-performing (Transform #T-5678)
  4. Aggregation: NPL total / Total loans (Transform #T-5679)
  5. Result: 2.3% (Updated: 2024-03-20 08:00)"

  Result: Audit completed in 30 minutes

Debugging Scenario
================================================

Problem: "Why did the monthly sales report drop 30% suddenly?"

Without lineage:
  Must manually check every step:
  1. Report query logic -> seems fine
  2. Data warehouse table -> row counts look right
  3. ETL jobs -> which ETL? There are 47 of them...
  4. Source systems -> which source? There are 12...

  Time spent: 2-3 days

With lineage:
  1. Click report number -> see dependency chain
  2. Discover Transform #T-456's input dataset
     was last updated 3 days ago (previously daily)
  3. Trace upstream from Transform #T-456 ->
     find that CRM system data sync job failed
  4. Root cause: CRM system upgrade changed API format

  Time spent: 15 minutes

# Foundry Transform Example -- Automatic Lineage Capture

from transforms.api import Transform, Input, Output

@transform(
    order_input=Input("/datasets/raw/sales_orders"),
    rate_input=Input("/datasets/ref/exchange_rates"),
    output=Output("/datasets/clean/orders_usd")
)
def compute_orders_usd(order_input, rate_input, output):
    orders = order_input.dataframe()
    rates = rate_input.dataframe()

    # This JOIN is automatically recorded as field-level lineage
    result = orders.join(rates, on="currency")

    # This computation is automatically recorded:
    # amount_usd = amount * rate
    result = result.withColumn(
        "amount_usd",
        result["amount"] * result["rate"]
    )

    output.write_dataframe(result)

Ontology Lineage Tracking
================================================

Raw Data Layer:
  +---------+   +----------+   +---------+
  | CRM     |   | ERP      |   | IoT     |
  | System  |   | System   |   | Platform|
  +----+----+   +-----+----+   +----+----+
       |              |             |
       v              v             v
Cleaning / Transform Layer:
  +---------+   +----------+   +---------+
  |Customer |   |Order     |   |Device   |
  |Data     |   |Data      |   |Data     |
  |(cleaned)|   |(cleaned) |   |(cleaned)|
  +----+----+   +-----+----+   +----+----+
       |              |             |
       v              v             v
Ontology Object Layer:
  +------------------------------------------+
  |                                          |
  |  Customer Object       Order Object      |
  |  +----------------+   +---------------+  |
  |  | name           |   | order_id      |  |
  |  | address        |   | amount        |  |
  |  | lifetime_value |   | status        |  |
  |  | risk_score     |   | customer_ref  |  |
  |  +----------------+   +---------------+  |
  |         |                    |           |
  |         +---- places -------+           |
  |                                          |
  |  lifetime_value lineage:                 |
  |  = SUM(Order.amount)                     |
  |    WHERE Order.customer_ref = this       |
  |    Source: ERP.sales_orders.total_amount  |
  |    Transforms: FX conversion -> monthly  |
  |                aggregation -> cumulative |
  |                                          |
  +------------------------------------------+

Approach Comparison
================================================

Static Analysis (parse code/SQL):

  SQL: SELECT a.id, a.amount * b.rate AS amount_usd
       FROM orders a JOIN rates b ON a.currency = b.code

  Parse Result:
    amount_usd <- orders.amount, rates.rate  (multiplication)
    id         <- orders.id                   (passthrough)

  Pros: No execution needed, lineage available upfront
  Cons: Complex logic hard to parse (UDFs, dynamic SQL)

Runtime Capture (record during execution):

  Instrument the DataFrame engine with tracing:
    Every column operation records its sources

  result["amount_usd"] = df["amount"] * df["rate"]
  -> Record: amount_usd = multiply(orders.amount, rates.rate)

  Pros: Precise, supports complex logic
  Cons: Runtime overhead, requires actual execution

Foundry's Approach: Combines Both
  1. Static analysis for initial lineage (fast)
  2. Runtime validation and enrichment (precise)
  3. Cache results to avoid recomputation

Upstream Tracing
================================================

Question: "Where does 'quarterly_revenue' on the CEO
           dashboard come from?"

Query: trace_upstream("dashboard.quarterly_revenue")

Result (bottom to top):

  Layer 4 (Display): CEO Dashboard -> quarterly_revenue
       ^
  Layer 3 (Aggregation): Revenue Aggregation Transform
                  = SUM(monthly_revenue) WHERE quarter = Q1
       ^
  Layer 2 (Computation): Monthly Revenue Transform
                  = SUM(order_amount_usd) GROUP BY month
       ^
  Layer 1 (Cleaning): Order Cleaning Pipeline
                  = orders WHERE status != 'cancelled'
                    AND test_account = false
       ^
  Layer 0 (Source):   ERP System -> sales_orders table
                  Source: SAP S/4HANA
                  Update frequency: Real-time CDC
                  Last sync: 2024-03-20 07:55:00

Downstream Impact Analysis
================================================

Question: "If I change the update frequency of exchange_rates,
           what is affected?"

Query: trace_downstream("ref.exchange_rates")

Result (top to bottom):

  ref.exchange_rates
       |
       +--> orders_usd (Transform #T-023)
       |    +--> monthly_revenue (Transform #T-024)
       |    |    +--> CEO Dashboard (Report)
       |    |    +--> Board Report (Report)
       |    |    +--> SEC Filing (Report) [!] Compliance Critical
       |    |
       |    +--> customer_ltv (Model #M-012)
       |         +--> Customer Risk Score (Model #M-015)
       |         +--> Marketing Campaign Targeting (App)
       |
       +--> fx_pnl_report (Transform #T-067)
            +--> Treasury Dashboard (Report)

  Impact Summary:
  - Direct downstream: 2 Transforms
  - Indirect downstream: 7 datasets, 4 reports, 1 ML model
  - Compliance impact: SEC Filing (high priority)
  - Recommendation: Notify finance and compliance teams before change

Data Quality + Lineage Integration
================================================

Scenario: Automatically detect data quality issue propagation

  Quality Check Failed:
  +--------------------------------------+
  |  Dataset: customer_addresses         |
  |  Check: Address completeness         |
  |  Result: FAILED (23% missing zip)    |
  |  Time: 2024-03-20 08:00              |
  +------------------+-------------------+
                     | Auto-trigger impact analysis
                     v
  +--------------------------------------+
  |  Impact Report                       |
  |                                      |
  |  Direct Impact:                      |
  |  +- customer_geo_analysis (Transform)|
  |  +- delivery_routing (Model)         |
  |  +- regional_revenue (Report)        |
  |                                      |
  |  Indirect Impact:                    |
  |  +- territory_planning (App)         |
  |  +- logistics_optimization (Model)   |
  |                                      |
  |  Recommended Actions:                |
  |  1. Pause delivery_routing retraining|
  |  2. Add warning to regional_revenue  |
  |  3. Notify logistics team            |
  +--------------------------------------+

GDPR Data Subject Request + Lineage
================================================

Scenario: User requests data deletion (Right to be Forgotten)

Request: "Delete all data for user-12345"

Lineage Trace:
  user-12345's data appears in:

  1. CRM.customers (source)
     +- user_profile (cleaned)
        +- customer_360 (Ontology Object)
        |  +- customer_name: "John Smith"
        |  +- customer_email: "john@example.com"
        |  +- customer_phone: "+1-xxx"
        |
        +- marketing_targets (aggregated dataset)
        |  +- contains user-12345's behavioral data
        |
        +- ml_training_data (model training set)
           +- contains user-12345's features

  Locations to delete/anonymize: 4 datasets + 1 Ontology object
  Models to retrain: 1
  Compliance: Record complete audit log of deletion

Data Trust Pyramid
================================================

            /\
           /  \
          /Decision\       "I can make decisions based on this"
         / Trust    \
        /------------\
       / Data         \    "I understand what this data means"
      / Understanding  \
     /------------------\
    / Source              \  "I know where it came from"
   / Transparency         \
  /------------------------\
 / Quality                  \  "Data has been quality-checked"
/ Assurance                  \
+-----------------------------+
| Traceability                |  "Every step is recorded"
+-----------------------------+

No lineage = pyramid foundation missing = cannot build trust