返回博客

查询优化:从逻辑计划到物理执行

Tags: #QueryOptimization #LogicalPlan #PhysicalPlan #CostModel #Vectorization #智策平台

Coomia发布于 2025年7月18日16 分钟阅读
分享本文Twitter / X

系列:S3 数据基座 · 第 9 篇 | 难度:高级 | 阅读时间:20 分钟

查询优化:从逻辑计划到物理执行

Tags: #QueryOptimization #LogicalPlan #PhysicalPlan #CostModel #Vectorization #智策平台

#TL;DR

OQL 查询经过解析和联邦路由后,最终要在具体引擎上执行。本文聚焦查询优化的核心环节:如何将一个"正确但低效"的逻辑计划转换为"正确且高效"的物理执行计划。涵盖谓词下推(Predicate Pushdown)、列裁剪(Column Pruning)、分区剪枝(Partition Pruning)、JOIN 重排序(Join Reordering)、子查询解关联(Subquery Decorrelation)、物化视图自动匹配以及 Doris 特有的向量化执行优化。通过基准测试展示各优化规则带来的性能提升。

#1. 优化器架构

#1.1 两阶段优化

Code
查询优化两阶段架构:

┌─────────────────────────────────────────────────┐
│                  OQL Query                       │
└──────────────────────┬──────────────────────────┘
                       ▼
┌─────────────────────────────────────────────────┐
│         Phase 1: Rule-Based Optimization (RBO)   │
│                                                  │
│  ┌──────────────┐  ┌──────────────┐             │
│  │ 谓词下推       │  │ 列裁剪        │             │
│  └──────────────┘  └──────────────┘             │
│  ┌──────────────┐  ┌──────────────┐             │
│  │ 常量折叠       │  │ 子查询解关联   │             │
│  └──────────────┘  └──────────────┘             │
│  ┌──────────────┐  ┌──────────────┐             │
│  │ 冗余消除       │  │ 表达式简化     │             │
│  └──────────────┘  └──────────────┘             │
└──────────────────────┬──────────────────────────┘
                       ▼
┌─────────────────────────────────────────────────┐
│         Phase 2: Cost-Based Optimization (CBO)   │
│                                                  │
│  ┌──────────────┐  ┌──────────────┐             │
│  │ JOIN 重排序    │  │ 索引选择       │             │
│  └──────────────┘  └──────────────┘             │
│  ┌──────────────┐  ┌──────────────┐             │
│  │ 物化视图匹配   │  │ 分区剪枝       │             │
│  └──────────────┘  └──────────────┘             │
│  ┌──────────────┐  ┌──────────────┐             │
│  │ 并行度决策     │  │ 聚合策略选择   │             │
│  └──────────────┘  └──────────────┘             │
└──────────────────────┬──────────────────────────┘
                       ▼
┌─────────────────────────────────────────────────┐
│              Physical Execution Plan              │
└─────────────────────────────────────────────────┘

#1.2 优化器接口

Python
from abc import ABC, abstractmethod

class OptimizationRule(ABC):
    """优化规则基类"""

    @abstractmethod
    def pattern(self) -> PlanPattern:
        """匹配的计划模式"""
        ...

    @abstractmethod
    def apply(self, plan: LogicalPlan) -> LogicalPlan:
        """应用优化规则"""
        ...

    @property
    def name(self) -> str:
        return self.__class__.__name__


class QueryOptimizer:
    """查询优化器"""

    def __init__(self):
        self._rbo_rules: list[OptimizationRule] = [
            PredicatePushdown(),
            ColumnPruning(),
            ConstantFolding(),
            SubqueryDecorrelation(),
            RedundantElimination(),
            ExpressionSimplification(),
        ]
        self._cbo_rules: list[CostBasedRule] = [
            JoinReordering(),
            IndexSelection(),
            MaterializedViewMatching(),
            PartitionPruning(),
            ParallelismDecision(),
            AggregationStrategy(),
        ]

    def optimize(
        self, plan: LogicalPlan, stats: TableStatistics
    ) -> PhysicalPlan:
        # Phase 1: RBO
        optimized = plan
        for rule in self._rbo_rules:
            optimized = self._apply_rule_recursive(optimized, rule)

        # Phase 2: CBO
        physical = self._cost_based_optimize(optimized, stats)

        return physical

#2. 基于规则的优化(RBO)

#2.1 谓词下推

Python
class PredicatePushdown(OptimizationRule):
    """谓词下推:将过滤条件推到尽可能靠近数据源的位置"""

    def apply(self, plan: LogicalPlan) -> LogicalPlan:
        if not isinstance(plan, Filter):
            return plan

        child = plan.child

        # Case 1: Filter 下面是 Join → 将谓词推到 Join 的子节点
        if isinstance(child, Join):
            left_preds, right_preds, join_preds = self._split_predicates(
                plan.predicate, child.left_schema, child.right_schema
            )

            new_left = child.left
            if left_preds:
                new_left = Filter(child.left, self._combine(left_preds))

            new_right = child.right
            if right_preds:
                new_right = Filter(child.right, self._combine(right_preds))

            new_join = Join(new_left, new_right, child.join_type, child.on)

            if join_preds:
                return Filter(new_join, self._combine(join_preds))
            return new_join

        # Case 2: Filter 下面是 Project → 交换顺序
        if isinstance(child, Project):
            if self._can_evaluate_before_project(
                plan.predicate, child.columns
            ):
                return Project(
                    Filter(child.child, plan.predicate),
                    child.columns
                )

        # Case 3: Filter 下面是 EntityScan → 合并为带过滤的 Scan
        if isinstance(child, EntityScan):
            return EntityScan(
                entity_type=child.entity_type,
                predicate=self._merge_predicates(
                    child.predicate, plan.predicate
                )
            )

        return plan
Code
谓词下推示例:

优化前:
  Filter(age > 30)
    └─ Join(Person.dept_id = Dept.id)
       ├─ EntityScan(Person)        ← 扫描所有 Person
       └─ EntityScan(Department)

优化后:
  Join(Person.dept_id = Dept.id)
    ├─ EntityScan(Person, age > 30) ← 只扫描 age > 30 的 Person
    └─ EntityScan(Department)

效果:减少 Join 的输入数据量 60-90%

#2.2 列裁剪

Python
class ColumnPruning(OptimizationRule):
    """列裁剪:只读取查询实际需要的列"""

    def apply(self, plan: LogicalPlan) -> LogicalPlan:
        required_columns = self._collect_required_columns(plan)
        return self._prune_recursive(plan, required_columns)

    def _collect_required_columns(
        self, plan: LogicalPlan
    ) -> set[str]:
        """自顶向下收集所有需要的列"""
        columns = set()

        if isinstance(plan, Project):
            columns.update(plan.columns)
        if isinstance(plan, Filter):
            columns.update(self._extract_column_refs(plan.predicate))
        if isinstance(plan, Sort):
            columns.update(col for col, _ in plan.order_by)
        if isinstance(plan, Join):
            columns.update(self._extract_column_refs(plan.on))

        for child in plan.children:
            columns.update(self._collect_required_columns(child))

        return columns

    def _prune_recursive(
        self, plan: LogicalPlan, needed: set[str]
    ) -> LogicalPlan:
        if isinstance(plan, EntityScan):
            # 只从存储读取需要的列
            return EntityScan(
                entity_type=plan.entity_type,
                predicate=plan.predicate,
                columns=list(needed & plan.available_columns)
            )
        return plan
Code
列裁剪示例:

OQL: FETCH Person SELECT name, age WHERE status = 'active'

优化前 (SELECT *):
  读取列:entity_id, entity_type, name, age, email, phone,
          address, status, created_at, updated_at, properties
  → 11 列,读取 2.2 GB

优化后 (列裁剪):
  读取列:entity_id, name, age, status
  → 4 列,读取 0.4 GB

效果:I/O 减少 82%

#2.3 常量折叠

Python
class ConstantFolding(OptimizationRule):
    """常量折叠:编译时计算常量表达式"""

    def apply(self, plan: LogicalPlan) -> LogicalPlan:
        if isinstance(plan, Filter):
            folded = self._fold_expression(plan.predicate)
            if isinstance(folded, LiteralExpr):
                if folded.value is True:
                    return plan.child  # 恒真条件,移除 Filter
                elif folded.value is False:
                    return EmptyResult()  # 恒假条件,空结果
            return Filter(plan.child, folded)
        return plan

    def _fold_expression(self, expr: Expression) -> Expression:
        if isinstance(expr, BinaryExpr):
            left = self._fold_expression(expr.left)
            right = self._fold_expression(expr.right)

            # 两个字面量可以直接计算
            if isinstance(left, LiteralExpr) and isinstance(right, LiteralExpr):
                result = self._evaluate(left.value, expr.operator, right.value)
                return LiteralExpr(value=result, literal_type='bool')

            # 特殊情况:x AND TRUE → x, x OR FALSE → x
            if expr.operator == 'AND':
                if isinstance(right, LiteralExpr) and right.value is True:
                    return left
                if isinstance(left, LiteralExpr) and left.value is True:
                    return right
            if expr.operator == 'OR':
                if isinstance(right, LiteralExpr) and right.value is False:
                    return left
                if isinstance(left, LiteralExpr) and left.value is False:
                    return right

            return BinaryExpr(left, expr.operator, right)
        return expr

#2.4 子查询解关联

Code
子查询解关联示例:

OQL:
  FETCH Person
  WHERE department_id IN (
    FETCH Department SELECT id WHERE location = 'Beijing'
  )

优化前(关联子查询):
  对每个 Person 行执行一次子查询 → O(N*M) 复杂度

优化后(解关联为 Semi-Join):
  Filter(Semi-Join)
    ├─ EntityScan(Person)
    └─ EntityScan(Department, location = 'Beijing')
  → O(N+M) 复杂度

效果:10000 Person × 500 Department
  优化前:5,000,000 次子查询执行
  优化后:1 次 Semi-Join(~500ms)

#3. 基于成本的优化(CBO)

#3.1 成本模型

Python
@dataclass
class QueryCost:
    """查询成本模型"""
    cpu_cost: float       # CPU 计算成本(指令数估算)
    io_cost: float        # I/O 成本(字节数估算)
    network_cost: float   # 网络传输成本(字节数估算)
    memory_cost: float    # 内存消耗(字节数估算)

    @property
    def total_cost(self) -> float:
        """加权总成本"""
        return (
            self.cpu_cost * 0.1 +
            self.io_cost * 1.0 +      # I/O 通常是瓶颈
            self.network_cost * 2.0 +  # 网络更贵
            self.memory_cost * 0.5
        )


class CostEstimator:
    """成本估算器"""

    def __init__(self, statistics: TableStatistics):
        self._stats = statistics

    def estimate_scan(self, scan: EntityScan) -> QueryCost:
        table_stats = self._stats.get(scan.entity_type)
        row_count = table_stats.row_count
        avg_row_size = table_stats.avg_row_size

        # 谓词选择率估算
        if scan.predicate:
            selectivity = self._estimate_selectivity(
                scan.predicate, table_stats
            )
        else:
            selectivity = 1.0

        filtered_rows = row_count * selectivity

        # 列裁剪后的行大小
        if scan.columns:
            col_size = sum(
                table_stats.column_stats[c].avg_size
                for c in scan.columns
            )
        else:
            col_size = avg_row_size

        return QueryCost(
            cpu_cost=filtered_rows * 10,  # 每行 ~10 指令
            io_cost=filtered_rows * col_size,
            network_cost=0,  # Scan 是本地操作
            memory_cost=min(filtered_rows, 8192) * col_size  # 批次大小
        )

    def _estimate_selectivity(
        self, predicate: Expression, stats: EntityStats
    ) -> float:
        """估算谓词选择率"""
        if isinstance(predicate, BinaryExpr):
            if predicate.operator == 'AND':
                left_sel = self._estimate_selectivity(predicate.left, stats)
                right_sel = self._estimate_selectivity(predicate.right, stats)
                return left_sel * right_sel  # 独立性假设

            if predicate.operator == 'OR':
                left_sel = self._estimate_selectivity(predicate.left, stats)
                right_sel = self._estimate_selectivity(predicate.right, stats)
                return left_sel + right_sel - left_sel * right_sel

            if predicate.operator == '=':
                col_stats = stats.column_stats.get(
                    predicate.left.property_name
                )
                if col_stats and col_stats.distinct_count > 0:
                    return 1.0 / col_stats.distinct_count

            if predicate.operator in ('<', '>', '<=', '>='):
                return 0.33  # 默认范围查询选择率

        return 0.5  # 默认选择率

#3.2 JOIN 重排序

Python
class JoinReordering(CostBasedRule):
    """JOIN 重排序:选择最优的 JOIN 执行顺序"""

    def optimize(
        self, joins: list[Join], stats: TableStatistics
    ) -> LogicalPlan:
        if len(joins) <= 2:
            return self._two_way_join(joins, stats)

        # 动态规划:枚举所有 JOIN 顺序
        n = len(joins)
        dp: dict[frozenset, tuple[LogicalPlan, QueryCost]] = {}

        # 初始化:单表
        for i, join in enumerate(joins):
            key = frozenset([i])
            dp[key] = (join.left, self._estimator.estimate(join.left))

        # DP 填表
        for size in range(2, n + 1):
            for subset in combinations(range(n), size):
                subset_key = frozenset(subset)
                best_plan = None
                best_cost = QueryCost(float('inf'), float('inf'), 0, 0)

                for split in self._enumerate_splits(subset_key):
                    left_key, right_key = split
                    if left_key not in dp or right_key not in dp:
                        continue

                    left_plan, left_cost = dp[left_key]
                    right_plan, right_cost = dp[right_key]

                    join_plan = Join(left_plan, right_plan, ...)
                    join_cost = self._estimator.estimate_join(
                        join_plan, left_cost, right_cost
                    )
                    total = left_cost.total_cost + right_cost.total_cost + join_cost.total_cost

                    if total < best_cost.total_cost:
                        best_plan = join_plan
                        best_cost = join_cost

                dp[subset_key] = (best_plan, best_cost)

        full_key = frozenset(range(n))
        return dp[full_key][0]
Code
JOIN 重排序示例:

三表 JOIN:Person ⋈ Department ⋈ Company
  Person:     100,000 行
  Department:   500 行
  Company:       50 行

优化前(按书写顺序):
  (Person ⋈ Department) ⋈ Company
  → 中间结果:100,000 行 ⋈ 500 = 最多 100,000 行
  → 最终 JOIN:100,000 ⋈ 50 行

优化后(最优顺序):
  Person ⋈ (Department ⋈ Company)
  → 中间结果:500 ⋈ 50 = 最多 500 行
  → 最终 JOIN:100,000 ⋈ 500 行(使用 Broadcast Join)

效果:中间结果减少 200 倍

#3.3 分区剪枝

Python
class PartitionPruning(CostBasedRule):
    """分区剪枝:跳过不相关的数据分区"""

    def apply(self, plan: EntityScan, stats: TableStatistics) -> EntityScan:
        if not plan.predicate:
            return plan

        partition_info = stats.get_partition_info(plan.entity_type)
        if not partition_info:
            return plan

        # 提取分区列上的谓词
        partition_predicates = self._extract_partition_predicates(
            plan.predicate, partition_info.partition_columns
        )

        if not partition_predicates:
            return plan

        # 评估每个分区是否可能包含匹配数据
        pruned_partitions = []
        for partition in partition_info.partitions:
            if self._partition_may_match(partition, partition_predicates):
                pruned_partitions.append(partition)

        return EntityScan(
            entity_type=plan.entity_type,
            predicate=plan.predicate,
            columns=plan.columns,
            partitions=pruned_partitions
        )
Code
分区剪枝示例:

entity_common 表按 entity_type 分区:
  Partition 1: entity_type = 'Person'      (100,000 行)
  Partition 2: entity_type = 'Device'      (500,000 行)
  Partition 3: entity_type = 'Document'    (200,000 行)
  Partition 4: entity_type = 'Company'     (10,000 行)

OQL: FETCH Person WHERE age > 30

分区剪枝后只扫描 Partition 1 → 跳过 87.7% 的数据

#3.4 物化视图匹配

Python
class MaterializedViewMatching(CostBasedRule):
    """物化视图自动匹配"""

    def __init__(self, mv_registry: MVRegistry):
        self._registry = mv_registry

    def apply(self, plan: LogicalPlan) -> LogicalPlan:
        """检查是否有物化视图可以替代当前查询"""
        for mv in self._registry.list_active_views():
            if self._can_answer_from_mv(plan, mv):
                # 物化视图可以回答此查询
                rewritten = self._rewrite_using_mv(plan, mv)
                original_cost = self._estimator.estimate(plan)
                mv_cost = self._estimator.estimate(rewritten)

                if mv_cost.total_cost < original_cost.total_cost * 0.5:
                    # 至少快 2 倍才使用 MV
                    return rewritten

        return plan

    def _can_answer_from_mv(
        self, plan: LogicalPlan, mv: MaterializedView
    ) -> bool:
        """检查 MV 是否包含回答查询所需的所有数据"""
        # 1. 查询的表必须是 MV 的子集
        query_tables = self._extract_tables(plan)
        mv_tables = self._extract_tables(mv.definition)
        if not query_tables.issubset(mv_tables):
            return False

        # 2. 查询的列必须在 MV 输出中
        query_columns = self._extract_required_columns(plan)
        mv_columns = set(mv.output_columns)
        if not query_columns.issubset(mv_columns):
            return False

        # 3. 查询的过滤条件必须能从 MV 的过滤条件推导
        return self._predicate_subsumes(mv.predicate, plan.predicate)

#4. 物理计划生成

#4.1 Doris 特有优化

Code
Doris 物理执行优化:

┌─────────────────────────────────────────────────┐
│                 Doris BE (Backend)                │
│                                                  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐      │
│  │ 列式存储   │  │ 向量化执行 │  │ SIMD 加速 │      │
│  │ (Columnar) │  │(Vectorized│  │(AVX2/512)│      │
│  └──────────┘  └──────────┘  └──────────┘      │
│                                                  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐      │
│  │ 分区裁剪   │  │ 索引过滤   │  │ 短路求值  │      │
│  │(Partition) │  │(Bitmap/   │  │(Short-   │      │
│  │            │  │ BloomFilter│  │ circuit) │      │
│  └──────────┘  └──────────┘  └──────────┘      │
└─────────────────────────────────────────────────┘

#4.2 JSON 属性提取优化

Code
三表模型中 JSON 属性查询优化:

问题:entity_common.properties 是 JSON 列
  每次查询都需要 JSON_EXTRACT → CPU 密集

优化 1:Bitmap 索引加速
  对常用属性值建立 Bitmap 索引:
  CREATE INDEX idx_status ON entity_common
  USING BITMAP(JSON_EXTRACT(properties, '$.status'));

优化 2:生成列(Generated Column)
  ALTER TABLE entity_common ADD COLUMN
  status VARCHAR(50) GENERATED ALWAYS AS
  JSON_EXTRACT(properties, '$.status');

  → JSON_EXTRACT 变成普通列读取,快 10-50 倍

优化 3:VARIANT 类型(Doris 2.1+)
  ALTER TABLE entity_common
  MODIFY COLUMN properties VARIANT;

  → 半结构化数据原生支持,自动子列提取

#5. 执行计划可视化

#5.1 EXPLAIN 输出

Code
OQL EXPLAIN 输出格式:

OQL: FETCH Person WHERE age > 30 AND department.name = 'Engineering'
     WITH METRIC direct_reports ORDER BY age DESC LIMIT 50

EXPLAIN:
┌─ Limit(50)                                    cost: 0.01
│  └─ Sort(age DESC)                             cost: 12.5
│     └─ MetricExpansion(direct_reports)          cost: 150.0
│        └─ HashJoin(Person.dept_id = Dept.id)    cost: 85.0
│           ├─ EntityScan(Person)                 cost: 45.0
│           │  ├─ Partition: entity_type='Person'
│           │  ├─ Predicate: age > 30 (sel: 0.33)
│           │  ├─ Columns: [entity_id, name, age, dept_id]
│           │  └─ Rows: 33,000 (of 100,000)
│           └─ EntityScan(Department)             cost: 0.5
│              ├─ Predicate: name = 'Engineering' (sel: 0.01)
│              ├─ Columns: [id, name]
│              └─ Rows: 5 (of 500)
│
│  Total Cost: 293.01
│  Estimated Rows: 50
│  Estimated Time: 280ms
└─ Optimizations Applied:
   ✓ Predicate Pushdown (age > 30 → Person scan)
   ✓ Predicate Pushdown (name = 'Engineering' → Dept scan)
   ✓ Column Pruning (11 → 4 columns for Person)
   ✓ Partition Pruning (4 → 1 partition)
   ✓ Join Strategy: Hash Join (Broadcast)

#6. 自适应查询执行(AQE)

#6.1 运行时统计反馈

Python
class AdaptiveQueryExecutor:
    """自适应查询执行:运行时调整执行计划"""

    async def execute(self, plan: PhysicalPlan) -> pa.Table:
        # 执行第一个阶段(通常是 Scan + Filter)
        stage1_result = await self._execute_stage(plan.stages[0])

        # 基于实际数据量调整后续阶段
        actual_rows = stage1_result.num_rows
        estimated_rows = plan.stages[0].estimated_rows

        if actual_rows < estimated_rows * 0.1:
            # 实际数据量远小于估计 → 切换为更轻量的策略
            plan = self._downgrade_plan(plan, actual_rows)
        elif actual_rows > estimated_rows * 10:
            # 实际数据量远大于估计 → 切换为更重量的策略
            plan = self._upgrade_plan(plan, actual_rows)

        # 执行剩余阶段
        return await self._execute_remaining(plan, stage1_result)

    def _downgrade_plan(
        self, plan: PhysicalPlan, actual_rows: int
    ) -> PhysicalPlan:
        """降级策略:数据量比预期小"""
        for stage in plan.stages[1:]:
            if isinstance(stage, SortMergeJoin) and actual_rows < 10000:
                stage = stage.convert_to(BroadcastJoin)
            if isinstance(stage, ExternalSort) and actual_rows < 100000:
                stage = stage.convert_to(InMemorySort)
        return plan

#7. 基准测试

#7.1 优化效果对比

Code
各优化规则性能提升:

┌─────────────────┬──────────┬──────────┬──────────┐
│ 优化规则           │ 无优化    │ 有优化    │ 提升      │
├─────────────────┼──────────┼──────────┼──────────┤
│ 谓词下推          │ 2.5s     │ 0.3s     │ 8.3x     │
│ 列裁剪            │ 1.8s     │ 0.4s     │ 4.5x     │
│ 分区剪枝          │ 3.2s     │ 0.4s     │ 8.0x     │
│ JOIN 重排序       │ 5.1s     │ 0.8s     │ 6.4x     │
│ 物化视图匹配      │ 2.3s     │ 0.05s    │ 46x      │
│ 子查询解关联      │ 12.0s    │ 0.5s     │ 24x      │
│ 常量折叠          │ 0.5s     │ 0.45s    │ 1.1x     │
├─────────────────┼──────────┼──────────┼──────────┤
│ 全部组合          │ 15.0s    │ 0.2s     │ 75x      │
└─────────────────┴──────────┴──────────┴──────────┘

测试环境:100万实体、500万边、Doris 3BE

#7.2 与 Palantir Foundry 对比

Code
查询性能对比(相似规模数据集):

┌─────────────────┬──────────┬──────────┬──────────┐
│ 查询类型           │ Foundry  │ coomia-dip│ 差距      │
├─────────────────┼──────────┼──────────┼──────────┤
│ 简单实体查询       │ 200ms    │ 50ms     │ 4x 领先   │
│ 图遍历 3 跳       │ 800ms    │ 500ms    │ 1.6x 领先 │
│ 聚合分析          │ 1.5s     │ 300ms    │ 5x 领先   │
│ 时间旅行          │ 2.0s     │ 1.5s     │ 1.3x 领先 │
│ 跨引擎联邦        │ 3.0s     │ 2.0s     │ 1.5x 领先 │
└─────────────────┴──────────┴──────────┴──────────┘

注:Foundry 使用 Spark,coomia-dip 使用 Doris
    Doris 在 OLAP 场景天然更快

#8. 测试策略

#8.1 优化器正确性测试

Python
class TestQueryOptimizer:

    def test_predicate_pushdown_through_join(self):
        plan = Filter(
            Join(
                EntityScan('Person'),
                EntityScan('Department'),
                on='dept_id = id'
            ),
            predicate=BinaryExpr(
                PropertyRef('age'), '>', LiteralExpr(30)
            )
        )
        optimized = optimizer.optimize(plan, stats)

        # 谓词应该被推到 Person scan 上
        assert isinstance(optimized, Join)
        assert isinstance(optimized.left, EntityScan)
        assert optimized.left.predicate is not None

    def test_optimization_preserves_semantics(self):
        """优化后的查询应该返回相同结果"""
        query = "FETCH Person WHERE age > 30 ORDER BY name"
        unoptimized = execute_without_optimization(query)
        optimized = execute_with_optimization(query)
        assert_tables_equal(unoptimized, optimized)

    def test_cost_model_accuracy(self):
        """成本模型估算应在 2x 误差范围内"""
        plan = parse_and_plan("FETCH Person WHERE age > 30")
        estimated = cost_estimator.estimate(plan)
        actual = measure_actual_cost(plan)

        assert estimated.io_cost / actual.io_cost < 2.0
        assert actual.io_cost / estimated.io_cost < 2.0

#Key Takeaways

  1. 两阶段优化(RBO + CBO)平衡了可预测性和最优性:RBO 处理确定性优化(谓词下推、列裁剪),CBO 处理需要统计信息的决策(JOIN 排序、索引选择)。

  2. 谓词下推是最重要的单一优化:将过滤推到数据源可以减少 80-90% 的 I/O,是所有优化中投入产出比最高的。

  3. 物化视图自动匹配带来数量级提升:当查询匹配到预计算的物化视图时,性能提升可达 40-50 倍。

  4. 三表模型的 JSON 属性查询需要特殊优化:生成列和 VARIANT 类型可以避免每次查询都做 JSON_EXTRACT。

  5. 自适应执行弥补了静态优化的不足:运行时统计反馈允许在执行过程中调整策略,应对估算偏差。

#Next Article

下一篇 S3-10《时间旅行:Iceberg 快照的深度应用》 将展示如何利用 Iceberg 的快照机制实现 OQL 的 AT TIME 子句,让用户查询任意历史时刻的 Ontology 数据。

Tags: #QueryOptimization #LogicalPlan #PhysicalPlan #PredicatePushdown #ColumnPruning #PartitionPruning #JoinReordering #MaterializedView #CostModel #智策平台 #coomia-dip #数据基座