Back to Blog

S3-13 Search Engine Design: 6 Search Modes + Facets + Hot Suggestions

The coomia-dip SearchService provides 6 search modes (BESTMATCH / PREFIX / FUZZY / EXACT / WILDCARD / REGEX), faceted search (TERMS / RANGE / DATERANGE), Redis-backed hot query and recent history suggestions, saved searches, permission-aware filtering, and deep integration with Doris inverted indexes. This article walks through every layer of the search engine from architecture to implementation details.

CoomiaPublished on July 22, 202520 min read
Share this articleTwitter / X

S3-13 Search Engine Design: 6 Search Modes + Facets + Hot Suggestions

Series: S3 Data Foundation · Article 13 | Level: Advanced | Reading Time: 20 min

#TL;DR

The coomia-dip SearchService provides 6 search modes (BEST_MATCH / PREFIX / FUZZY / EXACT / WILDCARD / REGEX), faceted search (TERMS / RANGE / DATE_RANGE), Redis-backed hot query and recent history suggestions, saved searches, permission-aware filtering, and deep integration with Doris inverted indexes. This article walks through every layer of the search engine from architecture to implementation details.

#1. Why an Ontology Platform Needs Its Own Search Engine

In traditional data platforms, search is often degraded to "filter by name." But in an Ontology-driven system, search is the primary interaction entry point:

  • Object Discovery: Users need to quickly locate target objects among millions of entities
  • Relationship Exploration: Search results need to show relational context between objects
  • Metadata Retrieval: Search covers not just data but type definitions, property descriptions, tags, and other metadata
  • Action Gateway: Search results directly link to available Actions (approvals, updates, workflow triggers)

Palantir Foundry's search experience is one of its core competitive advantages. When users search in Workshop or Vertex, results include not just matching objects but also facet statistics, related relationships, and executable actions. coomia-dip's SearchService aims to achieve this level of experience.

Code
+------------------------------------------------------------------+
|                        SearchService                              |
|                                                                   |
|  +-----------+  +-----------+  +-----------+  +----------------+  |
|  |  Query    |  |  Facet    |  | Suggest   |  |  Saved Search  |  |
|  |  Engine   |  |  Engine   |  | Engine    |  |  Manager       |  |
|  +-----------+  +-----------+  +-----------+  +----------------+  |
|       |              |              |                |             |
|  +-----------+  +-----------+  +-----------+  +----------------+  |
|  |  Mode     |  |  Agg      |  |  Redis    |  |  PostgreSQL    |  |
|  |  Router   |  |  Builder  |  |  Hot/Hist |  |  Storage       |  |
|  +-----------+  +-----------+  +-----------+  +----------------+  |
|       |              |                                            |
|  +----+--------------+----------------------------------------+   |
|  |            Permission-Aware Filter Layer                   |   |
|  +------------------------------------------------------------+   |
|       |                                                           |
|  +------------------------------------------------------------+   |
|  |          Doris Inverted Index / Full-Text Engine            |   |
|  +------------------------------------------------------------+   |
+------------------------------------------------------------------+

#2. Search Request Model

#2.1 SearchRequest Definition

PROTOBUF
message SearchRequest {
  string world_id = 1;
  string query = 2;
  SearchMode mode = 3;
  repeated string object_types = 4;    // restrict search to types
  repeated string search_fields = 5;   // restrict search to fields
  repeated FacetRequest facets = 6;
  repeated FilterClause filters = 7;
  Pagination pagination = 8;
  SortSpec sort = 9;
  bool include_highlights = 10;
  bool include_relations = 11;
}

enum SearchMode {
  BEST_MATCH = 0;  // default: composite scoring
  PREFIX = 1;       // prefix matching
  FUZZY = 2;        // fuzzy matching (tolerant)
  EXACT = 3;        // exact matching
  WILDCARD = 4;     // wildcard patterns
  REGEX = 5;        // regular expressions
}

#2.2 SearchResponse Definition

PROTOBUF
message SearchResponse {
  repeated SearchHit hits = 1;
  int64 total_count = 2;
  repeated FacetResult facets = 3;
  repeated Suggestion suggestions = 4;
  SearchMetadata metadata = 5;
}

message SearchHit {
  string object_type = 1;
  string object_id = 2;
  double score = 3;
  map<string, Value> attributes = 4;
  repeated Highlight highlights = 5;
  repeated RelatedObject relations = 6;
}

#3. The 6 Search Modes in Detail

BEST_MATCH is the default and most complex mode. It does not simply match text but considers multiple signals for scoring:

Code
Score = w1 * text_relevance
      + w2 * field_boost
      + w3 * recency_score
      + w4 * popularity_score
      + w5 * type_priority

text_relevance is computed by the BM25 algorithm, natively supported by Doris inverted indexes. field_boost assigns different weights to different fields — title field weight 3.0, description 1.5, tags 2.0. recency_score decays based on the object's last modification time. popularity_score is based on view/reference counts.

Python
class BestMatchScorer:
    """Composite scorer for BEST_MATCH mode"""

    FIELD_WEIGHTS = {
        "title": 3.0,
        "display_name": 3.0,
        "description": 1.5,
        "tags": 2.0,
        "properties": 1.0,
    }

    def score(self, query: str, hit: RawHit) -> float:
        text_score = hit.bm25_score
        field_score = self.FIELD_WEIGHTS.get(hit.matched_field, 1.0)
        recency = self._recency_decay(hit.updated_at)
        popularity = math.log1p(hit.view_count) * 0.1
        type_priority = self._type_priority(hit.object_type)

        return (
            0.5 * text_score * field_score
            + 0.15 * recency
            + 0.1 * popularity
            + 0.25 * type_priority
        )

    def _recency_decay(self, updated_at: datetime) -> float:
        days_ago = (datetime.utcnow() - updated_at).days
        return math.exp(-0.01 * days_ago)  # half-life ~70 days

    def _type_priority(self, object_type: str) -> float:
        priorities = {"Customer": 1.0, "Order": 0.9, "Product": 0.85}
        return priorities.get(object_type, 0.5)

#3.2 PREFIX — Prefix Matching

Prefix search is the core of autocomplete scenarios. Typing "cust" matches "Customer," "Custom Field," etc.

SQL
-- Doris prefix query
SELECT * FROM ontology_objects
WHERE display_name LIKE 'cust%'
ORDER BY length(display_name) ASC, view_count DESC
LIMIT 10;

The key optimization for prefix search is field-level inverted indexes. Doris supports creating inverted indexes on VARCHAR columns, allowing prefix queries to use the index directly instead of full table scans:

SQL
CREATE INDEX idx_display_name ON ontology_objects(display_name)
  USING INVERTED PROPERTIES("parser" = "standard");

#3.3 FUZZY — Fuzzy Matching

Fuzzy search tolerates spelling errors. Typing "custmer" (missing an 'o') still matches "Customer."

Python
class FuzzyMatcher:
    """Edit-distance-based fuzzy matching"""

    def __init__(self, max_edit_distance: int = 2):
        self.max_edit_distance = max_edit_distance

    def match(self, query: str, candidates: list[str]) -> list[tuple[str, int]]:
        results = []
        for candidate in candidates:
            distance = self._levenshtein(query.lower(), candidate.lower())
            if distance <= self.max_edit_distance:
                results.append((candidate, distance))
        return sorted(results, key=lambda x: x[1])

    def _levenshtein(self, s1: str, s2: str) -> int:
        if len(s1) < len(s2):
            return self._levenshtein(s2, s1)
        if len(s2) == 0:
            return len(s1)
        prev_row = range(len(s2) + 1)
        for i, c1 in enumerate(s1):
            curr_row = [i + 1]
            for j, c2 in enumerate(s2):
                insertions = prev_row[j + 1] + 1
                deletions = curr_row[j] + 1
                substitutions = prev_row[j] + (c1 != c2)
                curr_row.append(min(insertions, deletions, substitutions))
            prev_row = curr_row
        return prev_row[-1]

Doris inverted indexes also support fuzzy queries through MATCH_PHRASE with slop parameter or MATCH_ALL for approximate matching.

#3.4 EXACT — Exact Matching

Exact matching is used when the exact value is known, such as searching by ID, code, or SKU:

SQL
SELECT * FROM ontology_objects
WHERE object_id = 'ORD-2024-001234'
   OR attributes->>'sku' = 'ORD-2024-001234';

Exact matching bypasses scoring logic and directly returns perfectly matched results. It is the fastest mode, going directly through primary key or unique indexes.

#3.5 WILDCARD — Wildcard Matching

Wildcard search supports * (any character sequence) and ? (single character):

Python
class WildcardMatcher:
    """Wildcard search converter"""

    def to_sql(self, pattern: str) -> str:
        """Convert wildcard pattern to SQL LIKE pattern"""
        sql_pattern = pattern.replace("*", "%").replace("?", "_")
        return f"display_name LIKE '{sql_pattern}'"

    def to_regex(self, pattern: str) -> str:
        """Convert wildcard pattern to regex"""
        regex = pattern.replace(".", r"\.") \
                       .replace("*", ".*") \
                       .replace("?", ".")
        return f"^{regex}$"

#3.6 REGEX — Regular Expression Matching

Regex search is the most flexible but also the most dangerous mode. It requires strict input validation and timeout controls:

Python
class RegexSearchHandler:
    """Regular expression search handler"""

    MAX_REGEX_LENGTH = 200
    TIMEOUT_SECONDS = 5

    DANGEROUS_PATTERNS = [
        r"(.+)+",     # exponential backtracking
        r"(a*)*",     # nested quantifiers
        r"(a|a)*",    # overlapping alternation
    ]

    def validate(self, pattern: str) -> None:
        if len(pattern) > self.MAX_REGEX_LENGTH:
            raise SearchError("Regex pattern too long")
        for dangerous in self.DANGEROUS_PATTERNS:
            if dangerous in pattern:
                raise SearchError("Potentially catastrophic regex pattern")
        try:
            re.compile(pattern)
        except re.error as e:
            raise SearchError(f"Invalid regex: {e}")

    def search(self, pattern: str, field: str) -> str:
        self.validate(pattern)
        return f"{field} REGEXP '{pattern}'"

#3.7 Mode Router

Python
class SearchModeRouter:
    """Routes to the appropriate handler based on search mode"""

    def __init__(self):
        self._handlers: dict[SearchMode, SearchHandler] = {
            SearchMode.BEST_MATCH: BestMatchHandler(),
            SearchMode.PREFIX: PrefixHandler(),
            SearchMode.FUZZY: FuzzyHandler(),
            SearchMode.EXACT: ExactHandler(),
            SearchMode.WILDCARD: WildcardHandler(),
            SearchMode.REGEX: RegexHandler(),
        }

    def route(self, request: SearchRequest) -> SearchResponse:
        handler = self._handlers.get(request.mode)
        if handler is None:
            raise SearchError(f"Unsupported search mode: {request.mode}")

        # Auto-detect mode (when mode=BEST_MATCH)
        if request.mode == SearchMode.BEST_MATCH:
            detected = self._auto_detect(request.query)
            if detected != SearchMode.BEST_MATCH:
                handler = self._handlers[detected]

        return handler.execute(request)

    def _auto_detect(self, query: str) -> SearchMode:
        if query.startswith('"') and query.endswith('"'):
            return SearchMode.EXACT
        if "*" in query or "?" in query:
            return SearchMode.WILDCARD
        if query.startswith("/") and query.endswith("/"):
            return SearchMode.REGEX
        return SearchMode.BEST_MATCH

#4.1 Facet Types

Faceted search lets users aggregate search results by dimensions to quickly narrow down results. SearchService supports three facet types:

Facet TypePurposeExample
TERMSDiscrete value statisticsBy object type: Customer(120), Order(89), Product(45)
RANGENumeric intervalsBy amount range: 0-1K(30), 1K-10K(55), 10K+(15)
DATE_RANGETime intervalsBy creation date: Last 7 days(20), Last 30 days(45), Older(35)

#4.2 Facet Request and Response

PROTOBUF
message FacetRequest {
  string field = 1;
  FacetType type = 2;
  int32 size = 3;            // TERMS: return top N values
  repeated double ranges = 4; // RANGE: interval boundaries
  repeated DateRange date_ranges = 5;
}

enum FacetType {
  TERMS = 0;
  RANGE = 1;
  DATE_RANGE = 2;
}

message FacetResult {
  string field = 1;
  FacetType type = 2;
  repeated FacetBucket buckets = 3;
}

message FacetBucket {
  string key = 1;
  int64 count = 2;
  double from = 3;   // RANGE
  double to = 4;     // RANGE
}

#4.3 Facet Query Generation

Facet queries need to execute aggregate statistics in addition to the main query without changing its results:

Python
class FacetQueryBuilder:
    """Facet query builder"""

    def build_terms_facet(self, field: str, size: int) -> str:
        return f"""
        SELECT {field} AS facet_key, COUNT(*) AS facet_count
        FROM ontology_objects
        WHERE {{base_where_clause}}
        GROUP BY {field}
        ORDER BY facet_count DESC
        LIMIT {size}
        """

    def build_range_facet(self, field: str, ranges: list[float]) -> str:
        cases = []
        for i in range(len(ranges) - 1):
            lo, hi = ranges[i], ranges[i + 1]
            cases.append(
                f"WHEN {field} >= {lo} AND {field} < {hi} "
                f"THEN '{lo}-{hi}'"
            )
        cases.append(f"WHEN {field} >= {ranges[-1]} THEN '{ranges[-1]}+'")

        case_sql = "\n            ".join(cases)
        return f"""
        SELECT
            CASE
                {case_sql}
            END AS facet_key,
            COUNT(*) AS facet_count
        FROM ontology_objects
        WHERE {{base_where_clause}}
        GROUP BY facet_key
        ORDER BY facet_key
        """

    def build_date_range_facet(
        self, field: str, ranges: list[dict]
    ) -> str:
        cases = []
        for r in ranges:
            label = r["label"]
            from_date = r.get("from", "1970-01-01")
            to_date = r.get("to", "2099-12-31")
            cases.append(
                f"WHEN {field} >= '{from_date}' "
                f"AND {field} < '{to_date}' "
                f"THEN '{label}'"
            )
        case_sql = "\n            ".join(cases)
        return f"""
        SELECT
            CASE
                {case_sql}
            END AS facet_key,
            COUNT(*) AS facet_count
        FROM ontology_objects
        WHERE {{base_where_clause}}
        GROUP BY facet_key
        """

#4.4 Facet-Filter Interaction

When a user clicks a facet value to filter, other facets' statistics need to update, but the clicked facet itself should not re-filter (otherwise only the selected value would show):

Python
class FacetFilterCoordinator:
    """Coordinates facet filtering with facet statistics"""

    def execute_with_facets(
        self,
        base_query: str,
        facet_requests: list[FacetRequest],
        active_filters: dict[str, list[str]],
    ) -> tuple[list[SearchHit], list[FacetResult]]:
        # Main query applies all filters
        main_results = self._execute_main(base_query, active_filters)

        # Each facet query excludes its own filter
        facet_results = []
        for facet in facet_requests:
            other_filters = {
                k: v for k, v in active_filters.items()
                if k != facet.field
            }
            facet_result = self._execute_facet(
                base_query, facet, other_filters
            )
            facet_results.append(facet_result)

        return main_results, facet_results

#5. Auto-Suggest

#5.1 Three-Layer Suggestion Sources

Search suggestions come from three layers, ordered by priority:

Code
+-------------------------------------------+
|  Layer 1: Hot Queries (Redis Sorted Set)  |  <-- global trending
+-------------------------------------------+
|  Layer 2: Recent History (per user)       |  <-- personal history
+-------------------------------------------+
|  Layer 3: Entity Name Index (Trie)        |  <-- entity names
+-------------------------------------------+

#5.2 Redis Hot Query Management

Python
class HotQueryManager:
    """Hot query management using Redis Sorted Sets"""

    HOT_QUERY_KEY = "search:hot_queries"
    RECENT_KEY_PREFIX = "search:recent:{user_id}"
    MAX_HOT_QUERIES = 1000
    MAX_RECENT = 50
    HOT_QUERY_TTL = 86400 * 7  # 7 days

    def __init__(self, redis: Redis):
        self._redis = redis

    async def record_query(
        self, query: str, user_id: str
    ) -> None:
        """Record a search query"""
        normalized = query.strip().lower()
        if len(normalized) < 2:
            return

        pipe = self._redis.pipeline()
        # Increment hot query count
        pipe.zincrby(self.HOT_QUERY_KEY, 1, normalized)
        # Personal history (most recent 50)
        pipe.lpush(f"{self.RECENT_KEY_PREFIX}:{user_id}", normalized)
        pipe.ltrim(f"{self.RECENT_KEY_PREFIX}:{user_id}", 0, self.MAX_RECENT - 1)
        pipe.expire(f"{self.RECENT_KEY_PREFIX}:{user_id}", self.HOT_QUERY_TTL)
        await pipe.execute()

        # Periodically clean low-frequency terms
        count = await self._redis.zcard(self.HOT_QUERY_KEY)
        if count > self.MAX_HOT_QUERIES:
            await self._redis.zremrangebyrank(
                self.HOT_QUERY_KEY, 0,
                count - self.MAX_HOT_QUERIES - 1
            )

    async def get_suggestions(
        self, prefix: str, user_id: str, limit: int = 10
    ) -> list[Suggestion]:
        """Get search suggestions"""
        suggestions = []

        # 1. Personal history (priority)
        recent = await self._redis.lrange(
            f"{self.RECENT_KEY_PREFIX}:{user_id}", 0, -1
        )
        for q in recent:
            q_str = q.decode() if isinstance(q, bytes) else q
            if q_str.startswith(prefix.lower()):
                suggestions.append(
                    Suggestion(text=q_str, source="RECENT", score=1.0)
                )
            if len(suggestions) >= limit // 3:
                break

        # 2. Global hot queries
        hot = await self._redis.zrevrangebyscore(
            self.HOT_QUERY_KEY, "+inf", "-inf",
            start=0, num=100, withscores=True
        )
        for q, score in hot:
            q_str = q.decode() if isinstance(q, bytes) else q
            if q_str.startswith(prefix.lower()):
                suggestions.append(
                    Suggestion(text=q_str, source="HOT", score=score)
                )
            if len(suggestions) >= limit:
                break

        return suggestions[:limit]

#5.3 Entity Name Prefix Index

Beyond hot queries and history, suggestions based on actual data are needed. This is achieved through an in-memory Trie or Doris prefix queries:

Python
class EntitySuggester:
    """Entity name-based search suggestions"""

    SUGGEST_SQL = """
    SELECT display_name, object_type, object_id
    FROM ontology_objects
    WHERE display_name LIKE '{prefix}%'
    ORDER BY view_count DESC
    LIMIT {limit}
    """

    async def suggest(
        self, prefix: str, limit: int = 5
    ) -> list[Suggestion]:
        rows = await self._doris.execute(
            self.SUGGEST_SQL.format(prefix=prefix, limit=limit)
        )
        return [
            Suggestion(
                text=row["display_name"],
                source="ENTITY",
                score=0.5,
                metadata={
                    "object_type": row["object_type"],
                    "object_id": row["object_id"],
                },
            )
            for row in rows
        ]

#6. Saved Searches

#6.1 Saved Search Model

Python
class SavedSearch(BaseModel):
    """A saved search configuration"""
    id: str = Field(default_factory=lambda: str(uuid4()))
    user_id: str
    name: str
    description: str | None = None
    query: str
    mode: SearchMode = SearchMode.BEST_MATCH
    object_types: list[str] = []
    filters: list[FilterClause] = []
    facet_selections: dict[str, list[str]] = {}
    sort: SortSpec | None = None
    is_shared: bool = False
    created_at: datetime = Field(default_factory=datetime.utcnow)
    updated_at: datetime = Field(default_factory=datetime.utcnow)
    last_executed_at: datetime | None = None
    execution_count: int = 0

#6.2 SavedSearchService

Python
class SavedSearchService:
    """Saved search management service"""

    async def create(
        self, user_id: str, request: CreateSavedSearchRequest
    ) -> SavedSearch:
        saved = SavedSearch(
            user_id=user_id,
            name=request.name,
            description=request.description,
            query=request.query,
            mode=request.mode,
            object_types=request.object_types,
            filters=request.filters,
        )
        await self._repo.save(saved)
        return saved

    async def execute(
        self, saved_id: str, user_id: str
    ) -> SearchResponse:
        saved = await self._repo.get(saved_id)
        if saved.user_id != user_id and not saved.is_shared:
            raise PermissionError("Cannot execute others' private search")

        request = SearchRequest(
            query=saved.query,
            mode=saved.mode,
            object_types=saved.object_types,
            filters=saved.filters,
        )

        saved.last_executed_at = datetime.utcnow()
        saved.execution_count += 1
        await self._repo.save(saved)

        return await self._search_service.search(request)

    async def list_by_user(
        self, user_id: str
    ) -> list[SavedSearch]:
        own = await self._repo.find_by_user(user_id)
        shared = await self._repo.find_shared()
        return own + [s for s in shared if s.user_id != user_id]

#7. Permission-Aware Search Filtering

#7.1 Search Permission Model

Search results must respect data permission policies. Even if Doris returns matching results, they cannot be displayed if the user lacks read permission for those objects:

Code
+--------------------------------------------------+
|  SearchRequest                                    |
|       |                                           |
|       v                                           |
|  Query Execution (Doris)                          |
|       |                                           |
|       v                                           |
|  Raw Results (N hits)                             |
|       |                                           |
|       v                                           |
|  Permission Filter Layer                          |
|  +----------------------------------------------+|
|  |  1. Object-type-level ACL                    ||
|  |  2. Row-level security (RLS)                 ||
|  |  3. Column-level masking                     ||
|  |  4. World-level isolation                    ||
|  +----------------------------------------------+|
|       |                                           |
|  Filtered Results (M hits, M <= N)                |
+--------------------------------------------------+

#7.2 Injecting Permission Clauses at Query Time

A more efficient approach is to inject permission conditions during query generation, avoiding fetching large volumes of invisible results only to filter them out:

Python
class PermissionAwareSearchFilter:
    """Permission-aware search filter"""

    async def inject_permission_clause(
        self, user: User, base_query: str
    ) -> str:
        # Get object types the user can access
        accessible_types = await self._acl_service.get_accessible_types(
            user.id, Permission.READ
        )
        type_clause = (
            f"object_type IN ({','.join(repr(t) for t in accessible_types)})"
        )

        # Get row-level security policies
        rls_policies = await self._policy_service.get_rls_policies(user.id)
        rls_clauses = []
        for policy in rls_policies:
            rls_clauses.append(policy.to_sql_clause())

        # Get World isolation condition
        world_clause = f"world_id = '{user.active_world_id}'"

        # Combine all permission conditions
        permission_clause = " AND ".join(
            [type_clause, world_clause] + rls_clauses
        )

        return f"{base_query} AND ({permission_clause})"

    async def mask_columns(
        self, user: User, hits: list[SearchHit]
    ) -> list[SearchHit]:
        """Apply column-level masking to search results"""
        masking_rules = await self._policy_service.get_masking_rules(user.id)

        for hit in hits:
            for field, rule in masking_rules.items():
                if field in hit.attributes:
                    hit.attributes[field] = rule.apply(
                        hit.attributes[field]
                    )
                # Highlights also need masking
                hit.highlights = [
                    h for h in hit.highlights
                    if h.field not in masking_rules
                ]

        return hits

#8. Doris Inverted Index Integration

#8.1 Why Doris Instead of Elasticsearch

ComparisonDorisElasticsearch
Architecture complexitySingle system, no data sync neededRequires data sync pipeline
Data consistencyStrong consistency (same data)Eventually consistent (sync delay)
Ops costLow (existing Doris cluster)High (additional cluster + sync)
Full-text search2.0+ supports inverted indexesNative support, more features
Aggregation analyticsStrongAlso strong
CJK supportNative Chinese tokenizationRequires plugins

In coomia-dip's scenario, Doris inverted indexes fully meet requirements while avoiding the data synchronization complexity of introducing Elasticsearch.

#8.2 Inverted Index Configuration

SQL
-- Create inverted indexes on the objects table
ALTER TABLE ontology_objects ADD INDEX idx_inv_display_name(display_name)
  USING INVERTED
  PROPERTIES(
    "parser" = "chinese",
    "lower_case" = "true"
  );

ALTER TABLE ontology_objects ADD INDEX idx_inv_description(description)
  USING INVERTED
  PROPERTIES(
    "parser" = "chinese",
    "lower_case" = "true",
    "support_phrase" = "true"
  );

ALTER TABLE ontology_objects ADD INDEX idx_inv_tags(tags)
  USING INVERTED
  PROPERTIES(
    "parser" = "comma"
  );

-- Create inverted index on JSON attributes column
ALTER TABLE ontology_objects ADD INDEX idx_inv_attrs(attributes)
  USING INVERTED
  PROPERTIES(
    "parser" = "chinese",
    "lower_case" = "true"
  );

#8.3 Full-Text Search Queries

Query syntax supported by Doris inverted indexes:

SQL
-- MATCH_ANY: any token matches
SELECT * FROM ontology_objects
WHERE display_name MATCH_ANY 'customer order';

-- MATCH_ALL: all tokens must match
SELECT * FROM ontology_objects
WHERE description MATCH_ALL 'data pipeline scheduler';

-- MATCH_PHRASE: phrase matching (preserves token order)
SELECT * FROM ontology_objects
WHERE description MATCH_PHRASE 'real-time data pipeline';

#8.4 Query Builder

Python
class DorisSearchQueryBuilder:
    """Doris full-text search query builder"""

    def build(self, request: SearchRequest) -> str:
        select_clause = self._build_select(request)
        where_clause = self._build_where(request)
        order_clause = self._build_order(request)
        limit_clause = self._build_limit(request.pagination)

        return f"""
        {select_clause}
        FROM ontology_objects
        WHERE {where_clause}
        {order_clause}
        {limit_clause}
        """

    def _build_where(self, request: SearchRequest) -> str:
        clauses = []

        # Search conditions
        if request.mode == SearchMode.BEST_MATCH:
            fields = request.search_fields or [
                "display_name", "description", "tags"
            ]
            match_clauses = [
                f"{f} MATCH_ANY '{request.query}'" for f in fields
            ]
            clauses.append(f"({' OR '.join(match_clauses)})")

        elif request.mode == SearchMode.PREFIX:
            fields = request.search_fields or ["display_name"]
            like_clauses = [
                f"{f} LIKE '{request.query}%'" for f in fields
            ]
            clauses.append(f"({' OR '.join(like_clauses)})")

        elif request.mode == SearchMode.EXACT:
            fields = request.search_fields or ["display_name"]
            exact_clauses = [
                f"{f} = '{request.query}'" for f in fields
            ]
            clauses.append(f"({' OR '.join(exact_clauses)})")

        # Type filter
        if request.object_types:
            types = ",".join(f"'{t}'" for t in request.object_types)
            clauses.append(f"object_type IN ({types})")

        # Additional filters
        for f in request.filters:
            clauses.append(self._filter_to_sql(f))

        return " AND ".join(clauses) if clauses else "1=1"

#9. Search Highlighting

#9.1 Highlight Implementation

Keyword highlighting in search results helps users quickly identify match positions:

Python
class HighlightGenerator:
    """Search result highlight generator"""

    PRE_TAG = "<em>"
    POST_TAG = "</em>"
    FRAGMENT_SIZE = 150

    def highlight(
        self, query: str, text: str, field: str
    ) -> list[Highlight]:
        if not text:
            return []

        tokens = self._tokenize(query)
        fragments = self._extract_fragments(text, tokens)

        return [
            Highlight(
                field=field,
                fragment=self._mark_tokens(fragment, tokens),
            )
            for fragment in fragments
        ]

    def _tokenize(self, query: str) -> list[str]:
        """Tokenize the query"""
        return [t.strip() for t in query.split() if t.strip()]

    def _extract_fragments(
        self, text: str, tokens: list[str]
    ) -> list[str]:
        """Extract text fragments containing matched tokens"""
        fragments = []
        text_lower = text.lower()

        for token in tokens:
            pos = text_lower.find(token.lower())
            if pos >= 0:
                start = max(0, pos - self.FRAGMENT_SIZE // 2)
                end = min(len(text), pos + len(token) + self.FRAGMENT_SIZE // 2)
                fragment = text[start:end]
                if start > 0:
                    fragment = "..." + fragment
                if end < len(text):
                    fragment = fragment + "..."
                fragments.append(fragment)

        return fragments[:3]  # max 3 fragments

    def _mark_tokens(self, text: str, tokens: list[str]) -> str:
        """Mark matched tokens in text"""
        result = text
        for token in tokens:
            pattern = re.compile(re.escape(token), re.IGNORECASE)
            result = pattern.sub(
                f"{self.PRE_TAG}\\g<0>{self.POST_TAG}", result
            )
        return result

#10. Performance Optimization Strategies

#10.1 Search Cache

Python
class SearchCache:
    """Search result cache"""

    CACHE_TTL = 300  # 5 minutes
    CACHE_PREFIX = "search:cache:"

    async def get_or_execute(
        self, request: SearchRequest, executor: Callable
    ) -> SearchResponse:
        cache_key = self._build_key(request)
        cached = await self._redis.get(cache_key)

        if cached:
            return SearchResponse.model_validate_json(cached)

        response = await executor(request)

        # Only cache queries that take significant time
        if response.metadata.query_time_ms > 100:
            await self._redis.setex(
                cache_key,
                self.CACHE_TTL,
                response.model_dump_json(),
            )

        return response

    def _build_key(self, request: SearchRequest) -> str:
        import hashlib
        content = request.model_dump_json()
        digest = hashlib.sha256(content.encode()).hexdigest()[:16]
        return f"{self.CACHE_PREFIX}{digest}"

#10.2 Performance Benchmarks

Search ModeTypical LatencySuitable Data Volume
EXACT< 5msAny
PREFIX< 20ms< 10M
BEST_MATCH< 100ms< 10M
FUZZY< 200ms< 1M
WILDCARD< 500ms< 1M
REGEX< 5000ms< 100K

#10.3 Query Timeout and Circuit Breaking

Python
class SearchCircuitBreaker:
    """Search query circuit breaker"""

    def __init__(
        self,
        timeout_ms: int = 5000,
        failure_threshold: int = 5,
        reset_timeout: int = 30,
    ):
        self._timeout_ms = timeout_ms
        self._failure_count = 0
        self._failure_threshold = failure_threshold
        self._reset_timeout = reset_timeout
        self._state = "CLOSED"
        self._last_failure: datetime | None = None

    async def execute(
        self, coro: Coroutine
    ) -> SearchResponse:
        if self._state == "OPEN":
            if self._should_reset():
                self._state = "HALF_OPEN"
            else:
                raise SearchError("Search service circuit breaker is OPEN")

        try:
            result = await asyncio.wait_for(
                coro, timeout=self._timeout_ms / 1000
            )
            self._on_success()
            return result
        except asyncio.TimeoutError:
            self._on_failure()
            raise SearchError(f"Search timed out after {self._timeout_ms}ms")
        except Exception:
            self._on_failure()
            raise

#11. gRPC Service Definition

PROTOBUF
service SearchService {
  // General search
  rpc Search(SearchRequest) returns (SearchResponse);

  // Search suggestions
  rpc Suggest(SuggestRequest) returns (SuggestResponse);

  // Facet statistics only
  rpc GetFacets(FacetOnlyRequest) returns (FacetOnlyResponse);

  // Saved searches
  rpc CreateSavedSearch(CreateSavedSearchRequest)
      returns (SavedSearch);
  rpc ListSavedSearches(ListSavedSearchesRequest)
      returns (ListSavedSearchesResponse);
  rpc ExecuteSavedSearch(ExecuteSavedSearchRequest)
      returns (SearchResponse);
  rpc DeleteSavedSearch(DeleteSavedSearchRequest)
      returns (google.protobuf.Empty);
}

#12. End-to-End Search Flow

Code
User types "customer Zhang"
       |
       v
  AutoSuggest (< 50ms)
  +-- Recent: "customer Zhang San" (user's history)
  +-- Hot: "customer management" (global hot)
  +-- Entity: "Zhang Wei" (Customer), "Zhang Ming" (Supplier)
       |
  User selects or presses Enter
       |
       v
  SearchRequest(query="customer Zhang San", mode=BEST_MATCH)
       |
       v
  ModeRouter -> auto-detect -> BEST_MATCH
       |
       v
  PermissionFilter.inject(user) -> add ACL/RLS clauses
       |
       v
  DorisQueryBuilder.build() ->
    SELECT * FROM ontology_objects
    WHERE (display_name MATCH_ALL 'customer Zhang San'
           OR description MATCH_ANY 'customer Zhang San')
      AND object_type IN ('Customer','Order')
      AND world_id = 'main'
    ORDER BY score DESC
    LIMIT 20
       |
       v
  Doris Inverted Index -> raw hits
       |
       v
  BestMatchScorer -> re-rank with field boost + recency
       |
       v
  HighlightGenerator -> mark matched tokens
       |
       v
  FacetQueryBuilder -> parallel facet aggregations
       |
       v
  SearchResponse {
    hits: [{
      object_type: "Customer",
      object_id: "cust-001",
      score: 0.95,
      attributes: {display_name: "Zhang San", ...},
      highlights: [{field: "display_name",
                    fragment: "<em>Zhang San</em>"}]
    }, ...],
    total_count: 42,
    facets: [
      {field: "object_type", buckets: [
        {key: "Customer", count: 30},
        {key: "Order", count: 12}
      ]}
    ]
  }
       |
       v
  HotQueryManager.record("customer Zhang San", user_id)

#Key Takeaways

  1. 6 search modes cover all scenarios — from exact lookups to regex, each mode has its own optimization path
  2. Faceted search is not post-processing — facet statistics execute in parallel during query time, following the "exclude own filter" principle
  3. Three-layer suggestion system — personal history > global hot queries > entity names, Redis Sorted Set achieves O(log N) updates
  4. Permission injection at query time — rather than filtering results post-query, permission conditions are injected into SQL, reducing wasted queries
  5. Doris inverted indexes replace ES — a single system solves OLAP + full-text search, avoiding data sync consistency issues
  6. Circuit breaking and caching ensure stability — high-risk modes like regex search have timeout and circuit breaker protection

#Next Article

Next up: S3-14 Metric System: Priority Routing of 6 Computation Strategies will dive deep into metric registration, computation strategy routing, and the OQL rewriter implementation.

Tags: #SearchEngine #FacetedSearch #InvertedIndex #Doris #AutoSuggest #Redis #PermissionAware #gRPC #OntologyPlatform