“

Series: S1 Palantir Decoded · Article 15 | Level: Beginner | Reading Time: 15 min

Palantir's Search and Discovery: Finding What You Need Among Millions of Objects

#TL;DR

Foundry's search is not traditional "keyword-matching against tables" -- it searches Ontology objects, understands object types, properties, relationships, and permissions, letting users search enterprise data like Google, but with structured business objects as results.
Permission-aware search is Foundry's core differentiator -- you can only ever find objects you have permission to see, search results are automatically filtered, and no data you shouldn't see is ever exposed, which is critical for government and financial customers.
coomia-dip implements 6 search modes (BEST_MATCH/PREFIX/FUZZY/EXACT/WILDCARD/REGEX) + 3 faceted search types (TERMS/RANGE/DATE_RANGE) + Redis-based hot/recent search auto-suggestions, building an Ontology-aware search experience in the open-source ecosystem.

#1. Why Is Enterprise Search So Hard?

#1.1 Consumer Search vs Enterprise Search

We use Google and Bing every day to search the internet with a smooth, natural experience. But when you search for data inside an enterprise, the experience is often a disaster.

Code

Consumer Search vs Enterprise Search
================================================

Google Search:
  Input: "weather in Beijing"
  Result: Instant weather card with temperature, humidity, forecast
  Experience: *****

Traditional Enterprise Search:
  Input: "customer John's orders"
  Result: ???

  Scenario 1 (No search):
    "Please contact IT to file a ticket. Ticket processing time: 3-5 days."

  Scenario 2 (Basic search):
    Returns 47 results:
    - Customer records in CRM (3 entries)
    - Order data in ERP (12 entries)
    - Emails mentioning "John" (28 entries)
    - Some Excel files (4 files)

    Problem: Which of these refer to the same "John"?
    Problem: Do I have permission to see all this data?
    Problem: Which information is current?

  Foundry Search:
    Returns structured results:
    +-------------------------------------+
    | Customer: John Smith                |
    | ID: CUST-2024-78901                 |
    | Status: Active                      |
    | Related Orders: 12                  |
    | Latest Order: ORD-2024-56789 (open) |
    | Account Manager: Jane Doe           |
    | [View Details] [View Links] [Hist.] |
    +-------------------------------------+
    Experience: ****

#1.2 The Three Core Challenges of Enterprise Search

Code

Three Core Challenges of Enterprise Search
================================================

Challenge 1: Data Fragmentation
+------+ +------+ +------+ +------+ +------+
| CRM  | | ERP  | | HRM  | |Email | |Files |
+--+---+ +--+---+ +--+---+ +--+---+ +--+---+
   |        |        |        |        |
   v        v        v        v        v
 The same "customer" is represented 5 different
 ways across 5 systems -- different IDs, different
 fields, different update timestamps.

Challenge 2: Complex Permissions
+------------------------------------------+
|  User A (Sales): Can see contact info     |
|  User B (Finance): Can see billing info   |
|  User C (Compliance): Can see risk scores |
|  User D (Intern): Can only see names      |
|                                           |
|  Same search query, 4 people see          |
|  different results                        |
+------------------------------------------+

Challenge 3: Semantic Understanding
+------------------------------------------+
|  Search "big customer":                   |
|  - Customers with large order amounts?    |
|  - Customers with large headcount?        |
|  - Customers under the "Key Accounts"     |
|    department?                            |
|  - Customers with tier = "Enterprise"?    |
|                                           |
|  Without Ontology = ambiguity             |
|  With Ontology = precise semantics        |
+------------------------------------------+

#2. Foundry's Ontology-Aware Search

#2.1 Searching Objects, Not Tables

This is the fundamental difference between Foundry search and traditional database search:

Code

Traditional Search vs Ontology Search
================================================

Traditional Database Search:
  SELECT * FROM customers WHERE name LIKE '%John%'
  UNION
  SELECT * FROM orders WHERE customer_name LIKE '%John%'
  UNION
  SELECT * FROM tickets WHERE description LIKE '%John%'

  Problem: Cross-table search requires knowledge of table schemas
  Problem: Results are "rows", not "objects"
  Problem: Cannot understand relationships between objects

Foundry Ontology Search:
  Search("John", types=[Customer, Order, Ticket])

  The engine understands:
  - Customer is an object type with name, email, phone properties
  - Order has a "placed_by" link to Customer
  - Ticket has a "reported_by" link to Customer

  Returns:
  +-- Customer Object -----------------------+
  | John Smith (CUST-78901)                  |
  | +-- placed_by -> [12 orders]             |
  | +-- reported_by -> [3 tickets]           |
  | +-- managed_by -> Jane Doe (Employee)    |
  +-----------------------------------------+

#2.2 Search Architecture

Code

Foundry Search System Architecture
================================================

User Search Request
     |
     v
+---------------------------------------------+
|  Search Gateway                              |
|                                              |
|  1. Parse query (NLP / structured parsing)   |
|  2. Permission check (ACL filtering)         |
|  3. Query routing                            |
+--------+-----------------------+-------------+
         |                       |
    +----v------+           +----v------+
    | Full-Text |           | Structured|
    | Search    |           | Search    |
    | Engine    |           | Engine    |
    |           |           |           |
    |(Elastic-  |           |(Ontology  |
    | search)   |           | Index)    |
    +----+------+           +----+------+
         |                       |
         +-----------+-----------+
                     |
               +-----v------+
               |  Result     |
               |  Merger &   |
               |  Ranker     |
               |             |
               | Merge hits  |
               | Score/rank  |
               | Perm filter |
               | Facet agg   |
               +-----+------+
                     |
                     v
               Search Results (object list)

#2.3 How the Ontology Enriches Search

Code

Ontology-Enriched Search
================================================

When you type "customer revenue > 1M":

Without Ontology:
  The search engine treats this as text.
  Full-text matches documents containing
  "customer", "revenue", ">", "1M".
  Result: random documents with those words.

With Ontology:
  The engine knows:
  1. "Customer" is an ObjectType
  2. "revenue" is a numeric property of Customer
  3. "> 1M" is a numeric filter: revenue > 1,000,000

  It constructs a structured query:
  {
    object_type: "Customer",
    filter: { property: "revenue", operator: "gt", value: 1000000 },
    sort: { field: "revenue", order: "desc" }
  }

  Result: precisely the customers with revenue > $1M,
  sorted by revenue descending.

#3. Six Search Modes

#3.1 Mode Overview

Code

Six Search Modes
================================================

Mode 1: BEST_MATCH (default)
  Query: "John Smith"
  Behavior: Analyzes query, matches against all indexed
    fields, scores by relevance (TF-IDF / BM25)
  Best for: General-purpose search

Mode 2: PREFIX
  Query: "Joh"
  Behavior: Matches objects where any indexed field
    starts with "Joh" -- John, Johnson, Johannes
  Best for: Autocomplete / type-ahead

Mode 3: FUZZY
  Query: "Jonh Smth"  (typos)
  Behavior: Allows edit distance up to 2,
    finds "John Smith" despite typos
  Best for: Tolerating user typos

Mode 4: EXACT
  Query: "CUST-2024-78901"
  Behavior: Matches the exact string, no analysis
  Best for: ID lookups, exact value matching

Mode 5: WILDCARD
  Query: "CUST-2024-*"
  Behavior: * matches zero or more characters,
    ? matches exactly one character
  Best for: Pattern-based filtering

Mode 6: REGEX
  Query: "CUST-202[34]-\\d{5}"
  Behavior: Full regular expression matching
  Best for: Complex pattern matching (power users)

#3.2 Mode Selection Matrix

Code

+-------------+----------+--------+--------+----------+
| Scenario    | Mode     | Speed  | Recall | Precision|
+-------------+----------+--------+--------+----------+
| General     | BEST_    | Fast   | High   | Medium   |
| search      | MATCH    |        |        |          |
+-------------+----------+--------+--------+----------+
| Autocomplete| PREFIX   | V.Fast | Medium | High     |
+-------------+----------+--------+--------+----------+
| Typo-       | FUZZY    | Medium | V.High | Low-Med  |
| tolerant    |          |        |        |          |
+-------------+----------+--------+--------+----------+
| ID lookup   | EXACT    | V.Fast | Low    | V.High   |
+-------------+----------+--------+--------+----------+
| Pattern     | WILDCARD | Medium | Medium | High     |
| filter      |          |        |        |          |
+-------------+----------+--------+--------+----------+
| Complex     | REGEX    | Slow   | Medium | V.High   |
| pattern     |          |        |        |          |
+-------------+----------+--------+--------+----------+

#4. Faceted Search

Facets are aggregated counts that let users progressively narrow search results. Think of shopping on Amazon -- the left sidebar shows brand, price range, rating, and each option shows how many products match.

Code

Faceted Search Example
================================================

Search: "order" (all orders)
Total: 45,678 results

Left panel (facets):          Right panel (results):

Order Status (TERMS)          +-- Order ORD-2024-56789 --+
  [x] pending    (12345)      | Status: pending          |
  [ ] processing (8901)       | Amount: $2,340           |
  [ ] shipped    (15432)      | Customer: John Smith     |
  [ ] delivered  (9000)       +-------------------------+

Order Amount (RANGE)          +-- Order ORD-2024-56790 --+
  [ ] < $100     (5678)       | Status: pending          |
  [x] $100-$1K   (23456)      | Amount: $890             |
  [ ] $1K-$50K   (12345)      | Customer: Jane Doe       |
  [ ] $50K-$1M   (3456)       +-------------------------+
  [ ] > $1M      (532)
                              (After applying filters:
Customer Industry             pending + $100-$1K)
  [ ] Manufacturing (4521)    Showing: 8,234 results
  [ ] Finance      (3212)
  [ ] Retail       (2345)
  [ ] Technology   (5156)

Created Date (DATE_RANGE)
  [ ] Last 7 days  (1234)
  [x] Last 30 days (5678)
  [ ] Last 90 days (12345)

Code

Three Facet Types in Detail
================================================

TERMS Facet:
  Purpose: Count occurrences of each distinct value
  Example field: order_status
  Result: { "pending": 12345, "shipped": 15432, ... }
  Auto-generated from: Ontology enum properties

RANGE Facet:
  Purpose: Group numeric values into buckets
  Example field: order_amount
  Config: ranges = [0-100, 100-1000, 1000-50000, ...]
  Result: { "0-100": 5678, "100-1000": 23456, ... }
  Auto-generated from: Ontology numeric properties

DATE_RANGE Facet:
  Purpose: Group dates into time buckets
  Example field: created_at
  Config: ranges = [last_7d, last_30d, last_90d, ...]
  Result: { "last_7d": 1234, "last_30d": 5678, ... }
  Auto-generated from: Ontology date properties

#5. Permission-Aware Search

#5.1 You Can Only Find What You Can See

This is Foundry's most important search security feature:

Code

Permission-Aware Search
================================================

Objects actually in the database:
  Customer: [John, Jane, Bob, Alice, Eve]
  Order: [ORD-001, ORD-002, ORD-003, ..., ORD-100]

User A (East Region Sales Manager):
  Permission: East region customers + own orders
  Search "customer" -> Results: [John, Jane]  (2/5)
  Search "order"    -> Results: [ORD-001, ORD-023, ORD-045]

User B (National Sales Director):
  Permission: All regions + all orders
  Search "customer" -> Results: [John, Jane, Bob, Alice, Eve]
  Search "order"    -> Results: [ORD-001, ..., ORD-100]

User C (Compliance Auditor):
  Permission: All customers (compliance fields only) + high-risk orders
  Search "customer" -> Results: [John, Jane, Bob, Alice, Eve]
    But each customer shows only: name, risk_level, compliance_status
    Hidden fields: contact_info, transaction_history

Key Points:
  - Result COUNT differs (row-level permissions)
  - Result FIELDS differ (column-level permissions)
  - Facet counts adjust accordingly
  - Zero-knowledge: no leakage of unauthorized data

#5.2 Implementation Mechanism

Code

Permission-Aware Search Implementation
================================================

Search Request Processing Flow:

1. Receive search query
   query = "high value customer"
   user = User(id="user-A", roles=["east_region_sales"])

2. Retrieve user permissions
   permissions = ACL.get_permissions(user)
   -> {
       Customer: {
         row_filter: "region = 'east'",
         visible_fields: ["name", "email", "phone", "revenue"]
       },
       Order: {
         row_filter: "salesperson_id = 'user-A'",
         visible_fields: ["*"]
       }
     }

3. Build constrained search query
   search_query = {
     text: "high value customer",
     type_filters: {
       Customer: {
         must: [{ term: { region: "east" }}],
         source_includes: ["name", "email", "phone", "revenue"]
       }
     }
   }

4. Execute search (Elasticsearch)
   results = es.search(search_query)

5. Secondary permission check (guard against race conditions)
   filtered_results = [
     r for r in results
     if ACL.check_access(user, r.object_id)
   ]

#6. Search Suggestions and Autocomplete

#6.1 Types of Suggestions

Code

Search Suggestion Types
================================================

User types: "cust"
                        +------------------------------+
                        | Suggestions:                  |
                        |                               |
                        | [Type] Customer               |
                        |   "Search all Customer objects"|
                        |                               |
                        | [Recent] "customer churn"     |
                        |   "Your search from 2h ago"   |
                        |                               |
                        | [Hot] "customer satisfaction" |
                        |   "Trending (142 searches)"   |
                        |                               |
                        | [Object] CUST-2024-12345     |
                        |   "Customer: John Smith"      |
                        |                               |
                        | [Saved] "My VIP Customers"   |
                        |   "Saved search (shared)"     |
                        +------------------------------+

Five suggestion types:
  1. OBJECT_TYPE  - Suggests matching object types
  2. OBJECT       - Suggests specific matching objects
  3. RECENT       - User's recent searches (personalized)
  4. HOT          - Globally trending searches
  5. SAVED        - User's saved searches

#6.2 How Autocomplete Works

Code

Autocomplete Pipeline
================================================

User keystroke: "c" -> "cu" -> "cus" -> "cust"

For each keystroke (debounced ~150ms):

  1. Check Redis for recent searches matching prefix
     ZREVRANGE search:recent:{user_id} 0 4
     Filter: starts_with("cust")

  2. Check Redis for hot searches matching prefix
     ZREVRANGE search:hot 0 19
     Filter: starts_with("cust")

  3. Check Ontology type names matching prefix
     Filter ObjectTypes where name starts_with("cust")
     -> "Customer"

  4. (Optional) Quick prefix search in Elasticsearch
     For top-scoring objects matching prefix

  5. Merge, deduplicate, rank, return top 5-10
     Priority: Saved > Recent > Object > Hot > Type

#7. Comparison with Search Solutions

#7.1 Foundry vs Elasticsearch vs Algolia

Code

Feature Comparison
================================================

+---------------------+----------+----------+----------+
| Feature             | Foundry  | Elastic  | Algolia  |
+---------------------+----------+----------+----------+
| Full-text search    | Yes      | Yes      | Yes      |
+---------------------+----------+----------+----------+
| Fuzzy matching      | Yes      | Yes      | Yes      |
+---------------------+----------+----------+----------+
| Faceted search      | Yes      | Yes      | Yes      |
+---------------------+----------+----------+----------+
| Ontology-aware      | YES      | No       | No       |
+---------------------+----------+----------+----------+
| Object-level results| YES      | No (docs)| No (docs)|
+---------------------+----------+----------+----------+
| Permission-aware    | YES      | Manual   | Manual   |
+---------------------+----------+----------+----------+
| Row-level security  | Built-in | Plugin   | No       |
+---------------------+----------+----------+----------+
| Column-level secur. | Built-in | No       | No       |
+---------------------+----------+----------+----------+
| Relationship nav.   | YES      | No       | No       |
+---------------------+----------+----------+----------+
| Saved searches      | YES      | Kibana   | No       |
+---------------------+----------+----------+----------+
| Search analytics    | YES      | Kibana   | Yes      |
+---------------------+----------+----------+----------+
| Self-hosted         | Yes      | Yes      | No(SaaS) |
+---------------------+----------+----------+----------+
| Ontology auto-facet | YES      | Manual   | Manual   |
+---------------------+----------+----------+----------+

#7.2 Why Raw Elasticsearch Is Not Enough

Code

Elasticsearch Alone vs Ontology-Aware Search
================================================

With raw Elasticsearch:
  - You index documents manually
  - You define mappings manually
  - You build permission filters manually
  - You build facets manually for each query
  - You handle cross-index relationships manually
  - You build the UI from scratch

  Developer cost: Months of work per object type

With Foundry (or coomia-dip):
  - Define an Ontology type -> index auto-created
  - Properties define the mapping -> auto-generated
  - ACL model -> permission filters auto-injected
  - Property types -> facets auto-suggested
  - Link types -> relationship navigation built-in
  - Search UI -> auto-generated from Ontology schema

  Developer cost: Define the Ontology, search just works

#8. coomia-dip SearchService Implementation

#8.1 Architecture Overview

Code

coomia-dip SearchService
================================================

+------------------------------------------------+
|  SearchService (gRPC)                          |
|                                                |
|  Core Search RPCs:                             |
|  +------------------------------------------+ |
|  |  1. Search(SearchRequest)                 | |
|  |     -> Supports 6 search modes            | |
|  |     -> Supports faceted search            | |
|  |     -> Supports pagination and sorting    | |
|  |                                           | |
|  |  2. Suggest(SuggestRequest)               | |
|  |     -> Autocomplete suggestions           | |
|  |     -> Hot searches                       | |
|  |     -> Recent searches                    | |
|  |                                           | |
|  |  3. FacetSearch(FacetRequest)             | |
|  |     -> TERMS facets                       | |
|  |     -> RANGE facets                       | |
|  |     -> DATE_RANGE facets                  | |
|  +------------------------------------------+ |
|                                                |
|  Underlying Engines:                           |
|  +------------+  +----------+  +------------+  |
|  |Elasticsearch|  |  Redis   |  | Ontology   |  |
|  |(full-text   |  |(suggest  |  |(type-aware |  |
|  | index)      |  | cache)   |  | context)   |  |
|  +------------+  +----------+  +------------+  |
+------------------------------------------------+

#8.2 Search Request Model

PROTOBUF

// coomia-dip Search Protobuf (simplified)

syntax = "proto3";
package onto.search.v1;

message SearchRequest {
    string query = 1;                    // Search term
    SearchMode mode = 2;                 // Search mode
    repeated string object_types = 3;    // Restrict to object types
    repeated FacetSpec facets = 4;       // Facet definitions
    Pagination pagination = 5;          // Pagination
    repeated SortSpec sort = 6;         // Sorting
    map<string, string> filters = 7;    // Filter conditions
}

enum SearchMode {
    BEST_MATCH = 0;    // Overall best match
    PREFIX = 1;        // Prefix matching
    FUZZY = 2;         // Fuzzy matching
    EXACT = 3;         // Exact matching
    WILDCARD = 4;      // Wildcard
    REGEX = 5;         // Regular expression
}

message FacetSpec {
    string field = 1;          // Facet field
    FacetType type = 2;        // Facet type
    int32 size = 3;            // Return count
    RangeSpec range = 4;       // Config for RANGE type
}

enum FacetType {
    TERMS = 0;          // Terms facet (enum value counts)
    RANGE = 1;          // Numeric range facet
    DATE_RANGE = 2;     // Date range facet
}

message SearchResponse {
    repeated SearchHit hits = 1;
    int64 total_count = 2;
    repeated FacetResult facets = 3;
    float max_score = 4;
    int32 took_ms = 5;
}

message SearchHit {
    string object_id = 1;
    string object_type = 2;
    float score = 3;
    map<string, string> highlight = 4;  // Highlighted snippets
    map<string, google.protobuf.Value> fields = 5;
}

message SuggestRequest {
    string prefix = 1;          // User input prefix
    string user_id = 2;         // User ID (personalization)
    int32 max_results = 3;      // Max suggestions
}

message SuggestResponse {
    repeated Suggestion suggestions = 1;
}

message Suggestion {
    string text = 1;
    SuggestionType type = 2;
    string description = 3;
    float score = 4;
}

enum SuggestionType {
    OBJECT_TYPE = 0;    // Object type suggestion
    OBJECT = 1;         // Specific object suggestion
    RECENT = 2;         // Recent search
    HOT = 3;            // Hot/trending search
    SAVED = 4;          // Saved search
}

#8.3 Redis Search Suggestion Implementation

Python

# coomia-dip Search Suggestion Implementation

import redis.asyncio as redis
import time
from typing import List


class SearchSuggestService:
    """Redis-based search suggestion service."""

    def __init__(self, redis_client: redis.Redis):
        self.redis = redis_client
        self.HOT_KEY = "search:hot"
        self.RECENT_PREFIX = "search:recent:"
        self.HOT_WINDOW = 86400  # 24-hour rolling window

    async def record_search(self, user_id: str, query: str):
        """Record search behavior, update hot and recent searches."""
        now = time.time()

        # Update user's recent searches (ZSET, score=timestamp)
        recent_key = f"{self.RECENT_PREFIX}{user_id}"
        await self.redis.zadd(recent_key, {query: now})
        await self.redis.zremrangebyrank(recent_key, 0, -51)  # Keep last 50

        # Update global hot searches (ZSET, score=count)
        await self.redis.zincrby(self.HOT_KEY, 1, query)

        # Clean expired hot searches (hourly cleanup of old data)
        cutoff = now - self.HOT_WINDOW
        await self.redis.zremrangebyscore(
            self.HOT_KEY, "-inf", cutoff
        )

    async def get_suggestions(
        self,
        prefix: str,
        user_id: str,
        max_results: int = 10,
    ) -> List[dict]:
        """Get search suggestions."""
        suggestions = []

        # 1. Recent searches (personalized)
        recent_key = f"{self.RECENT_PREFIX}{user_id}"
        recent = await self.redis.zrevrange(
            recent_key, 0, 4, withscores=True
        )
        for query, score in recent:
            query_str = query.decode() if isinstance(query, bytes) else query
            if query_str.lower().startswith(prefix.lower()):
                suggestions.append({
                    "text": query_str,
                    "type": "RECENT",
                    "score": 0.8,
                })

        # 2. Hot searches (global)
        hot = await self.redis.zrevrange(
            self.HOT_KEY, 0, 19, withscores=True
        )
        for query, count in hot:
            query_str = query.decode() if isinstance(query, bytes) else query
            if query_str.lower().startswith(prefix.lower()):
                suggestions.append({
                    "text": query_str,
                    "type": "HOT",
                    "score": min(float(count) / 100, 1.0),
                })

        # 3. Deduplicate and sort
        seen = set()
        unique = []
        for s in sorted(suggestions, key=lambda x: -x["score"]):
            if s["text"] not in seen:
                seen.add(s["text"])
                unique.append(s)

        return unique[:max_results]

#9. Search Performance Optimization

#9.1 Indexing Strategy

Code

Search Index Optimization Strategies
================================================

Strategy 1: Multi-Level Index
+--------------------------------------------+
|  Level 1: In-Memory Cache (Redis)          |
|  - Cache hot search results                |
|  - TTL: 5 minutes                          |
|  - Hit rate: ~60%                          |
|  - Response time: <5ms                     |
+--------------------------------------------+
|  Level 2: Elasticsearch                    |
|  - Full-text + structured index            |
|  - Near real-time updates (1s refresh)     |
|  - Response time: 10-100ms                 |
+--------------------------------------------+
|  Level 3: Database (PostgreSQL)            |
|  - Exact query fallback                    |
|  - Complex join queries                    |
|  - Response time: 50-500ms                 |
+--------------------------------------------+

Strategy 2: Index Design
+--------------------------------------------+
|  One ES index per Ontology type:           |
|                                            |
|  onto_customer:                            |
|    mappings:                               |
|      name: text (analyzed) + keyword (raw) |
|      email: keyword                        |
|      revenue: long                         |
|      region: keyword                       |
|      created_at: date                      |
|      _all_text: text (merged text fields)  |
|                                            |
|  onto_order:                               |
|    mappings:                               |
|      order_id: keyword                     |
|      amount: double                        |
|      status: keyword                       |
|      description: text                     |
|      customer_id: keyword (for joins)      |
+--------------------------------------------+

Strategy 3: Query Optimization
+--------------------------------------------+
|  - Use filter context (cacheable) for      |
|    exact filters                           |
|  - Use query context for relevance scoring |
|  - Put permission filters in filter        |
|    context (no scoring impact)             |
|  - Use global aggregations for facets to   |
|    avoid redundant computation             |
|  - Use search_after instead of from+size   |
|    for large result sets                   |
+--------------------------------------------+

#9.2 Performance Benchmarks

Code

Expected Performance (coomia-dip targets)
================================================

+--------------------+----------+----------+----------+
| Operation          | p50      | p95      | p99      |
+--------------------+----------+----------+----------+
| Simple search      | 15ms     | 50ms     | 120ms    |
+--------------------+----------+----------+----------+
| Faceted search     | 30ms     | 80ms     | 200ms    |
+--------------------+----------+----------+----------+
| Autocomplete       | 5ms      | 15ms     | 40ms     |
+--------------------+----------+----------+----------+
| Regex search       | 100ms    | 500ms    | 2000ms   |
+--------------------+----------+----------+----------+
| Cross-type search  | 50ms     | 150ms    | 400ms    |
+--------------------+----------+----------+----------+

Index sizes (estimated):
  1M objects  -> ~2GB  ES index -> sub-100ms queries
  10M objects -> ~20GB ES index -> sub-200ms queries
  100M objects-> ~200GB ES index -> shard optimization needed

#10. Saved Searches and Search Subscriptions

#10.1 Saved Searches

Code

Saved Searches
================================================

Users often run the same complex searches. Save them:

Saved Search: "My high-priority pending orders"
+--------------------------------------------+
|  Definition:                               |
|  {                                         |
|    "query": "",                            |
|    "object_type": "Order",                 |
|    "filters": {                            |
|      "status": "pending",                  |
|      "priority": "high",                   |
|      "assigned_to": "${current_user}"      |
|    },                                      |
|    "sort": [{"field": "due_date",          |
|              "order": "asc"}]              |
|  }                                         |
|                                            |
|  Features:                                 |
|  - Dynamic variables (${current_user})     |
|  - Shareable with team members             |
|  - Pinnable to dashboards                  |
|  - Notification on result changes          |
+--------------------------------------------+

#10.2 Search Subscriptions

Code

Search Subscriptions (Alerts)
================================================

Subscribe to a search to get notified when results change:

Example: "Alert me when a new order > $100K appears"

Subscription:
  search: { type: "Order", filter: "amount > 100000" }
  trigger: ON_NEW_RESULT
  notify: [email, in-app, slack]
  frequency: REAL_TIME | HOURLY | DAILY

How it works:
  1. System stores the search definition
  2. On each index update, re-runs the search
  3. Compares with previous result set
  4. If new results found -> trigger notification

Use cases:
  - Compliance: "New transactions > $10K from flagged entities"
  - Sales: "New leads in my territory"
  - Operations: "Equipment with status changed to 'critical'"

#Key Takeaways

Foundry revolutionizes enterprise search by transforming "searching tables" into "searching objects" -- through Ontology awareness, the search engine understands business object types, properties, and relationships, returning structured business objects with full context instead of raw database rows, with permission filtering ensuring data security.
Permission-aware search is not "post-filter" but "built-in security" -- permission constraints are injected at query construction time, ensuring users can only ever discover data they are authorized to access, which is critical for government and financial industry compliance requirements.
coomia-dip builds a complete Ontology-aware search experience with 6 search modes + 3 facet types + Redis-based hot/recent search suggestions -- Elasticsearch provides search performance at the foundation, the Ontology type system provides semantic understanding at the top, and Redis delivers real-time personalized search suggestions.

#Next Article Preview

Article 16: Palantir's Pricing and Business Model -- Why Customers Pay $100M/Year

Palantir's single-customer annual fees can reach hundreds of millions of dollars, with net retention rates exceeding 118%. We will analyze their pricing strategy, business model, and why once you start using Palantir, it is very hard to stop.

Tags: palantir search discovery ontology elasticsearch faceted-search fuzzy-search coomia-dip redis permissions

Palantir's Search and Discovery: Finding What You Need Among Millions of Objects

Palantir's Search and Discovery: Finding What You Need Among Millions of Objects

#TL;DR

#1. Why Is Enterprise Search So Hard?

#1.1 Consumer Search vs Enterprise Search

#1.2 The Three Core Challenges of Enterprise Search

#2. Foundry's Ontology-Aware Search

#2.1 Searching Objects, Not Tables

#2.2 Search Architecture

#2.3 How the Ontology Enriches Search

#3. Six Search Modes

#3.1 Mode Overview

#3.2 Mode Selection Matrix

#4. Faceted Search

#4.1 What Are Facets?

#4.2 Three Facet Types

#5. Permission-Aware Search

#5.1 You Can Only Find What You Can See

#5.2 Implementation Mechanism

#6. Search Suggestions and Autocomplete

#6.1 Types of Suggestions

#6.2 How Autocomplete Works

#7. Comparison with Search Solutions

#7.1 Foundry vs Elasticsearch vs Algolia

#7.2 Why Raw Elasticsearch Is Not Enough

#8. coomia-dip SearchService Implementation

#8.1 Architecture Overview

#8.2 Search Request Model

#8.3 Redis Search Suggestion Implementation

#9. Search Performance Optimization

#9.1 Indexing Strategy

#9.2 Performance Benchmarks

#10. Saved Searches and Search Subscriptions

#10.1 Saved Searches

#10.2 Search Subscriptions

#Key Takeaways

#Next Article Preview