S3-14 指标系统:6 种计算策略的优先级路由
智策平台的指标系统 MetricRegistryService 支持 6 种计算策略(REALTIME / VIRTUALCOLUMN / UDF / CACHED / ROLLUP / MATERIALIZED),通过 ComputationCoordinator 进行优先级路由,由 OQL 重写器将 SELECT metric('x') FROM Type 展开为具体的计算逻辑。本文完整拆解指标定义、注册、计算路由和查询重写的全链路。
S3-14 指标系统:6 种计算策略的优先级路由
“系列:S3 数据基座 · 第 14 篇 | 难度:高级 | 阅读时间:20 分钟
#TL;DR
智策平台的指标系统 MetricRegistryService 支持 6 种计算策略(REALTIME / VIRTUAL_COLUMN / UDF / CACHED / ROLLUP / MATERIALIZED),通过 ComputationCoordinator 进行优先级路由,由 OQL 重写器将 SELECT metric('x') FROM Type 展开为具体的计算逻辑。本文完整拆解指标定义、注册、计算路由和查询重写的全链路。
#1. 指标系统在 Ontology 平台中的位置
在 Palantir Foundry 中,指标(Metrics)是对原始数据的抽象计算层。用户不需要知道"月活跃客户数"是怎么算的——他们只需要在 OQL 中写 metric('monthly_active_customers'),系统自动选择最优的计算路径。
智策平台的指标系统需要解决三个核心问题:
- 定义标准化 — 一个指标有且只有一个权威定义,避免"相同名称不同口径"
- 计算路由 — 根据数据新鲜度、查询延迟要求、数据量自动选择最优策略
- 透明嵌入 — 指标在 OQL 中像普通属性一样使用,用户无感知
+------------------------------------------------------------------+
| Metric System Architecture |
| |
| +---------------------+ +-----------------------------+ |
| | MetricRegistryService| | ComputationCoordinator | |
| | | | | |
| | - register() | | - route(metric, context) | |
| | - update() | | - evaluate(strategy) | |
| | - delete() | | - fallback() | |
| | - list() | +-----------------------------+ |
| | - get() | | |
| +---------------------+ v |
| | +-----------------------------+ |
| v | Strategy Executors | |
| +------------------+ | | |
| | OQL Rewriter | | REALTIME -> live query | |
| | | | VIRTUAL_COL -> Doris VC | |
| | metric('x') ──> | | UDF -> custom func | |
| | expanded SQL | | CACHED -> Redis cache | |
| +------------------+ | ROLLUP -> pre-agg | |
| | MATERIALIZED-> MV table | |
| +-----------------------------+ |
+------------------------------------------------------------------+
#2. 指标定义模型
#2.1 MetricDefinition
class MetricDefinition(BaseModel):
"""指标定义"""
metric_id: str = Field(default_factory=lambda: str(uuid4()))
name: str # 唯一名称: monthly_active_customers
display_name: str # 显示名: 月活跃客户数
description: str | None = None
object_type: str # 所属对象类型: Customer
category: MetricCategory # 分类
data_type: MetricDataType # 返回值类型
expression: MetricExpression # 计算表达式
strategies: list[ComputationStrategy] # 允许的计算策略(按优先级排序)
freshness_requirement: FreshnessLevel # 新鲜度要求
cache_ttl: int = 300 # 缓存 TTL(秒)
tags: list[str] = []
owner: str | None = None
created_at: datetime = Field(default_factory=datetime.utcnow)
updated_at: datetime = Field(default_factory=datetime.utcnow)
version: int = 1
class MetricCategory(str, Enum):
BUSINESS = "BUSINESS" # 业务指标
OPERATIONAL = "OPERATIONAL" # 运营指标
FINANCIAL = "FINANCIAL" # 财务指标
TECHNICAL = "TECHNICAL" # 技术指标
CUSTOM = "CUSTOM" # 自定义
class MetricDataType(str, Enum):
INTEGER = "INTEGER"
DECIMAL = "DECIMAL"
PERCENTAGE = "PERCENTAGE"
CURRENCY = "CURRENCY"
DURATION = "DURATION"
class FreshnessLevel(str, Enum):
REALTIME = "REALTIME" # 实时(< 1s 延迟)
NEAR_REALTIME = "NEAR_REALTIME" # 准实时(< 1min)
HOURLY = "HOURLY" # 小时级
DAILY = "DAILY" # 天级
WEEKLY = "WEEKLY" # 周级
#2.2 MetricExpression
指标表达式定义了"怎么算":
class MetricExpression(BaseModel):
"""指标计算表达式"""
type: ExpressionType
sql: str | None = None # SQL 表达式
aggregation: AggregationType | None = None
source_field: str | None = None # 源字段
filter_clause: str | None = None # 过滤条件
group_by: list[str] = [] # 分组字段
window: WindowSpec | None = None # 窗口规格
class ExpressionType(str, Enum):
SIMPLE_AGG = "SIMPLE_AGG" # COUNT/SUM/AVG/MAX/MIN
SQL = "SQL" # 自定义 SQL
COMPOSITE = "COMPOSITE" # 组合其他指标
PYTHON_UDF = "PYTHON_UDF" # Python UDF
class AggregationType(str, Enum):
COUNT = "COUNT"
COUNT_DISTINCT = "COUNT_DISTINCT"
SUM = "SUM"
AVG = "AVG"
MAX = "MAX"
MIN = "MIN"
MEDIAN = "MEDIAN"
PERCENTILE = "PERCENTILE"
#2.3 指标定义示例
# 示例 1: 简单聚合 — 客户总订单数
order_count_metric = MetricDefinition(
name="customer_order_count",
display_name="客户总订单数",
object_type="Customer",
category=MetricCategory.BUSINESS,
data_type=MetricDataType.INTEGER,
expression=MetricExpression(
type=ExpressionType.SIMPLE_AGG,
aggregation=AggregationType.COUNT,
source_field="orders", # 关系名
),
strategies=[
ComputationStrategy.CACHED,
ComputationStrategy.ROLLUP,
ComputationStrategy.REALTIME,
],
freshness_requirement=FreshnessLevel.HOURLY,
cache_ttl=3600,
)
# 示例 2: SQL 表达式 — 客户 30 天 GMV
gmv_30d_metric = MetricDefinition(
name="customer_gmv_30d",
display_name="客户30天GMV",
object_type="Customer",
category=MetricCategory.FINANCIAL,
data_type=MetricDataType.CURRENCY,
expression=MetricExpression(
type=ExpressionType.SQL,
sql="""
SELECT SUM(o.total_amount)
FROM orders o
WHERE o.customer_id = {object_id}
AND o.created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY)
AND o.status != 'CANCELLED'
""",
),
strategies=[
ComputationStrategy.MATERIALIZED,
ComputationStrategy.CACHED,
ComputationStrategy.REALTIME,
],
freshness_requirement=FreshnessLevel.DAILY,
cache_ttl=86400,
)
# 示例 3: 组合指标 — 客均 GMV
avg_gmv_metric = MetricDefinition(
name="average_customer_gmv",
display_name="客均GMV",
object_type="Customer",
category=MetricCategory.FINANCIAL,
data_type=MetricDataType.CURRENCY,
expression=MetricExpression(
type=ExpressionType.COMPOSITE,
sql="metric('total_gmv') / metric('active_customer_count')",
),
strategies=[
ComputationStrategy.CACHED,
ComputationStrategy.REALTIME,
],
freshness_requirement=FreshnessLevel.DAILY,
)
#3. 6 种计算策略详解
#3.1 策略概览
| 策略 | 延迟 | 新鲜度 | 适用场景 | 资源消耗 |
|---|---|---|---|---|
| REALTIME | 100ms-10s | 实时 | 少量数据、关键决策 | 高 |
| VIRTUAL_COLUMN | 10-100ms | 实时 | 简单计算、Doris 原生支持 | 低 |
| UDF | 50ms-5s | 实时 | 复杂逻辑、Python 计算 | 中 |
| CACHED | 1-5ms | 准实时 | 高频访问、可容忍短延迟 | 低 |
| ROLLUP | 5-50ms | 小时/天级 | 大数据量聚合 | 预计算时高 |
| MATERIALIZED | 5-20ms | 定时刷新 | 复杂查询、大数据量 | 存储高 |
#3.2 REALTIME — 实时计算
实时策略直接在数据源上执行计算,每次查询都生成最新结果:
class RealtimeExecutor(StrategyExecutor):
"""实时计算执行器"""
async def execute(
self, metric: MetricDefinition, context: ComputeContext
) -> MetricResult:
sql = self._build_sql(metric, context)
start = time.monotonic()
result = await self._doris.execute(sql)
elapsed_ms = (time.monotonic() - start) * 1000
return MetricResult(
metric_name=metric.name,
value=result[0]["value"] if result else None,
computed_at=datetime.utcnow(),
strategy_used=ComputationStrategy.REALTIME,
latency_ms=elapsed_ms,
is_stale=False,
)
def _build_sql(
self, metric: MetricDefinition, context: ComputeContext
) -> str:
expr = metric.expression
if expr.type == ExpressionType.SIMPLE_AGG:
agg = expr.aggregation.value
source = expr.source_field
return f"""
SELECT {agg}({source}) AS value
FROM {metric.object_type.lower()}_objects
WHERE object_id = '{context.object_id}'
"""
elif expr.type == ExpressionType.SQL:
return expr.sql.replace("{object_id}", context.object_id)
raise MetricError(f"Unsupported expression type for REALTIME: {expr.type}")
#3.3 VIRTUAL_COLUMN — Doris 虚拟列
Doris 2.1+ 支持生成列(Generated Column),可以将简单的计算逻辑直接定义在表结构中:
-- 创建虚拟列
ALTER TABLE customer_objects ADD COLUMN
order_total_amount DECIMAL(18,2)
GENERATED ALWAYS AS (
SELECT COALESCE(SUM(amount), 0)
FROM order_objects
WHERE customer_id = customer_objects.object_id
) VIRTUAL;
class VirtualColumnExecutor(StrategyExecutor):
"""Doris 虚拟列执行器"""
async def execute(
self, metric: MetricDefinition, context: ComputeContext
) -> MetricResult:
column_name = self._get_virtual_column_name(metric)
sql = f"""
SELECT {column_name} AS value
FROM {metric.object_type.lower()}_objects
WHERE object_id = '{context.object_id}'
"""
result = await self._doris.execute(sql)
return MetricResult(
metric_name=metric.name,
value=result[0]["value"] if result else None,
computed_at=datetime.utcnow(),
strategy_used=ComputationStrategy.VIRTUAL_COLUMN,
latency_ms=0,
is_stale=False,
)
def _get_virtual_column_name(self, metric: MetricDefinition) -> str:
return f"vc_{metric.name}"
#3.4 UDF — 用户自定义函数
对于无法用 SQL 表达的复杂逻辑,通过 Python UDF 执行:
class UdfExecutor(StrategyExecutor):
"""UDF 计算执行器"""
def __init__(self, udf_registry: dict[str, Callable]):
self._udf_registry = udf_registry
async def execute(
self, metric: MetricDefinition, context: ComputeContext
) -> MetricResult:
udf_name = metric.expression.sql # UDF 名称存在 sql 字段
udf_func = self._udf_registry.get(udf_name)
if udf_func is None:
raise MetricError(f"UDF not found: {udf_name}")
# 获取计算所需的数据
data = await self._fetch_data(metric, context)
start = time.monotonic()
value = await udf_func(data, context)
elapsed_ms = (time.monotonic() - start) * 1000
return MetricResult(
metric_name=metric.name,
value=value,
computed_at=datetime.utcnow(),
strategy_used=ComputationStrategy.UDF,
latency_ms=elapsed_ms,
is_stale=False,
)
# UDF 注册示例
@register_metric_udf("customer_health_score")
async def compute_health_score(
data: MetricData, context: ComputeContext
) -> float:
"""计算客户健康分数 — 综合多个维度"""
order_frequency = data.get("order_frequency", 0)
avg_order_value = data.get("avg_order_value", 0)
days_since_last_order = data.get("days_since_last_order", 999)
support_tickets = data.get("support_tickets", 0)
frequency_score = min(order_frequency / 10, 1.0) * 30
value_score = min(avg_order_value / 10000, 1.0) * 25
recency_score = max(0, 1 - days_since_last_order / 365) * 30
satisfaction_score = max(0, 1 - support_tickets / 20) * 15
return round(frequency_score + value_score + recency_score + satisfaction_score, 2)
#3.5 CACHED — 缓存计算
缓存策略将计算结果存储在 Redis 中,在 TTL 内直接返回缓存值:
class CachedExecutor(StrategyExecutor):
"""缓存计算执行器"""
CACHE_PREFIX = "metric:cache:"
async def execute(
self, metric: MetricDefinition, context: ComputeContext
) -> MetricResult:
cache_key = self._cache_key(metric, context)
# 尝试读取缓存
cached = await self._redis.get(cache_key)
if cached:
data = json.loads(cached)
return MetricResult(
metric_name=metric.name,
value=data["value"],
computed_at=datetime.fromisoformat(data["computed_at"]),
strategy_used=ComputationStrategy.CACHED,
latency_ms=0,
is_stale=self._is_stale(data, metric),
)
# 缓存未命中,回退到实时计算
result = await self._realtime_executor.execute(metric, context)
# 写入缓存
await self._redis.setex(
cache_key,
metric.cache_ttl,
json.dumps({
"value": result.value,
"computed_at": result.computed_at.isoformat(),
}),
)
result.strategy_used = ComputationStrategy.CACHED
return result
def _cache_key(
self, metric: MetricDefinition, context: ComputeContext
) -> str:
return f"{self.CACHE_PREFIX}{metric.name}:{context.object_id}"
def _is_stale(
self, data: dict, metric: MetricDefinition
) -> bool:
computed = datetime.fromisoformat(data["computed_at"])
age = (datetime.utcnow() - computed).total_seconds()
return age > metric.cache_ttl * 0.8 # 80% TTL 后标记为 stale
#3.6 ROLLUP — 预聚合
ROLLUP 策略通过预计算的聚合表提供低延迟查询:
class RollupExecutor(StrategyExecutor):
"""预聚合执行器"""
async def execute(
self, metric: MetricDefinition, context: ComputeContext
) -> MetricResult:
rollup_table = self._get_rollup_table(metric)
sql = f"""
SELECT metric_value AS value, computed_at
FROM {rollup_table}
WHERE metric_name = '{metric.name}'
AND object_id = '{context.object_id}'
ORDER BY computed_at DESC
LIMIT 1
"""
result = await self._doris.execute(sql)
if not result:
# Rollup 表中无数据,回退
raise StrategyUnavailableError("No rollup data available")
return MetricResult(
metric_name=metric.name,
value=result[0]["value"],
computed_at=result[0]["computed_at"],
strategy_used=ComputationStrategy.ROLLUP,
latency_ms=0,
is_stale=self._check_staleness(result[0]["computed_at"], metric),
)
def _get_rollup_table(self, metric: MetricDefinition) -> str:
freshness = metric.freshness_requirement
if freshness in (FreshnessLevel.HOURLY, FreshnessLevel.NEAR_REALTIME):
return "metric_rollup_hourly"
elif freshness == FreshnessLevel.DAILY:
return "metric_rollup_daily"
else:
return "metric_rollup_weekly"
Rollup 表由定时任务维护:
-- 每小时 Rollup 任务
INSERT INTO metric_rollup_hourly (metric_name, object_id, metric_value, computed_at)
SELECT
'customer_order_count' AS metric_name,
c.object_id,
COUNT(o.object_id) AS metric_value,
NOW() AS computed_at
FROM customer_objects c
LEFT JOIN order_objects o ON o.customer_id = c.object_id
GROUP BY c.object_id
ON DUPLICATE KEY UPDATE
metric_value = VALUES(metric_value),
computed_at = VALUES(computed_at);
#3.7 MATERIALIZED — 物化视图
物化视图策略使用 Doris 的物化视图功能,将复杂查询的结果固化为表:
class MaterializedViewExecutor(StrategyExecutor):
"""物化视图执行器"""
async def execute(
self, metric: MetricDefinition, context: ComputeContext
) -> MetricResult:
mv_name = f"mv_{metric.name}"
sql = f"""
SELECT metric_value AS value
FROM {mv_name}
WHERE object_id = '{context.object_id}'
"""
result = await self._doris.execute(sql)
if not result:
raise StrategyUnavailableError(f"MV {mv_name} has no data")
return MetricResult(
metric_name=metric.name,
value=result[0]["value"],
computed_at=datetime.utcnow(), # MV 由 Doris 自动维护
strategy_used=ComputationStrategy.MATERIALIZED,
latency_ms=0,
is_stale=False,
)
#4. ComputationCoordinator — 计算路由器
#4.1 路由逻辑
ComputationCoordinator 是指标系统的核心,它根据指标定义和查询上下文选择最优的计算策略:
class ComputationCoordinator:
"""指标计算协调器 — 策略优先级路由"""
def __init__(self):
self._executors: dict[ComputationStrategy, StrategyExecutor] = {
ComputationStrategy.REALTIME: RealtimeExecutor(),
ComputationStrategy.VIRTUAL_COLUMN: VirtualColumnExecutor(),
ComputationStrategy.UDF: UdfExecutor(),
ComputationStrategy.CACHED: CachedExecutor(),
ComputationStrategy.ROLLUP: RollupExecutor(),
ComputationStrategy.MATERIALIZED: MaterializedViewExecutor(),
}
async def compute(
self,
metric: MetricDefinition,
context: ComputeContext,
) -> MetricResult:
"""按优先级依次尝试计算策略"""
errors: list[str] = []
for strategy in metric.strategies:
executor = self._executors.get(strategy)
if executor is None:
continue
# 检查策略是否可用
if not await self._is_strategy_available(strategy, metric, context):
errors.append(f"{strategy.value}: unavailable")
continue
try:
result = await executor.execute(metric, context)
# 检查结果新鲜度
if result.is_stale and self._has_fresher_strategy(
metric, strategy
):
# 结果过期且有更新鲜的策略,触发异步刷新
await self._schedule_refresh(metric, context)
return result
except StrategyUnavailableError as e:
errors.append(f"{strategy.value}: {e}")
continue
except Exception as e:
errors.append(f"{strategy.value}: error - {e}")
logger.warning(
f"Strategy {strategy.value} failed for {metric.name}: {e}"
)
continue
# 所有策略都失败
raise MetricComputeError(
f"All strategies failed for metric {metric.name}: "
+ "; ".join(errors)
)
async def _is_strategy_available(
self,
strategy: ComputationStrategy,
metric: MetricDefinition,
context: ComputeContext,
) -> bool:
"""检查策略是否可用"""
if strategy == ComputationStrategy.VIRTUAL_COLUMN:
return await self._has_virtual_column(metric)
if strategy == ComputationStrategy.MATERIALIZED:
return await self._has_materialized_view(metric)
if strategy == ComputationStrategy.ROLLUP:
return await self._has_rollup_data(metric, context)
if strategy == ComputationStrategy.UDF:
return metric.expression.type == ExpressionType.PYTHON_UDF
return True
def _has_fresher_strategy(
self,
metric: MetricDefinition,
current: ComputationStrategy,
) -> bool:
"""当前策略之后是否还有更实时的策略"""
fresher = {
ComputationStrategy.REALTIME,
ComputationStrategy.VIRTUAL_COLUMN,
ComputationStrategy.UDF,
}
idx = metric.strategies.index(current)
remaining = metric.strategies[idx + 1:]
return any(s in fresher for s in remaining)
#4.2 路由决策流程
ComputationCoordinator.compute(metric, context)
|
v
strategies = [MATERIALIZED, CACHED, ROLLUP, REALTIME]
|
v
[1] MATERIALIZED available? ──NO──> skip
|YES
v
Execute MV query
|
Success? ──NO──> [2] CACHED
|YES
v
Result stale? ──YES──> schedule async refresh
|NO
v
Return result
|
[2] CACHED available? ──YES──> check Redis
| |
| Hit? ──YES──> return cached
| |NO
| v
| Fallback to REALTIME, cache result
|
[3] ROLLUP available? ──YES──> query rollup table
| |
| Data found? ──YES──> return
| |NO
| v
v fallback
[4] REALTIME ──> execute live query ──> return
#4.3 批量计算优化
当查询多个对象的同一指标时,批量执行比逐个执行高效得多:
class BatchComputationCoordinator:
"""批量指标计算"""
async def compute_batch(
self,
metric: MetricDefinition,
object_ids: list[str],
) -> dict[str, MetricResult]:
# 先批量查缓存
cached_results = await self._batch_cache_lookup(metric, object_ids)
missing_ids = [
oid for oid in object_ids if oid not in cached_results
]
if not missing_ids:
return cached_results
# 对未命中的批量实时计算
batch_sql = self._build_batch_sql(metric, missing_ids)
rows = await self._doris.execute(batch_sql)
computed_results = {}
for row in rows:
oid = row["object_id"]
result = MetricResult(
metric_name=metric.name,
value=row["value"],
computed_at=datetime.utcnow(),
strategy_used=ComputationStrategy.REALTIME,
)
computed_results[oid] = result
# 写入缓存
await self._cache_result(metric, oid, result)
return {**cached_results, **computed_results}
#5. OQL 重写器
#5.1 什么是 OQL 指标展开
OQL(Ontology Query Language)允许用户在查询中直接引用指标。重写器负责将指标引用展开为实际的计算逻辑:
-- 用户写的 OQL
SELECT
customer.name,
metric('customer_order_count') AS order_count,
metric('customer_gmv_30d') AS gmv_30d
FROM Customer customer
WHERE metric('customer_gmv_30d') > 10000
-- 重写后的 SQL(策略=MATERIALIZED)
SELECT
c.name,
mv_order_count.metric_value AS order_count,
mv_gmv_30d.metric_value AS gmv_30d
FROM customer_objects c
LEFT JOIN mv_customer_order_count mv_order_count
ON mv_order_count.object_id = c.object_id
LEFT JOIN mv_customer_gmv_30d mv_gmv_30d
ON mv_gmv_30d.object_id = c.object_id
WHERE mv_gmv_30d.metric_value > 10000
#5.2 重写器实现
class OqlMetricRewriter:
"""OQL 指标重写器"""
METRIC_PATTERN = re.compile(r"metric\(\s*'([^']+)'\s*\)")
def __init__(
self,
registry: MetricRegistryService,
coordinator: ComputationCoordinator,
):
self._registry = registry
self._coordinator = coordinator
async def rewrite(self, oql: str, context: QueryContext) -> str:
"""重写 OQL 中的 metric() 调用"""
# 1. 提取所有指标引用
metric_refs = self.METRIC_PATTERN.findall(oql)
if not metric_refs:
return oql # 无指标引用,原样返回
# 2. 加载指标定义
metrics = {}
for name in metric_refs:
metric_def = await self._registry.get_by_name(name)
if metric_def is None:
raise OqlError(f"Unknown metric: {name}")
metrics[name] = metric_def
# 3. 为每个指标确定最优策略
strategy_map = {}
for name, metric_def in metrics.items():
strategy = await self._coordinator.select_strategy(
metric_def, context
)
strategy_map[name] = strategy
# 4. 根据策略生成重写后的 SQL
rewritten = oql
join_clauses = []
for name, metric_def in metrics.items():
strategy = strategy_map[name]
replacement, join = self._generate_replacement(
name, metric_def, strategy
)
rewritten = rewritten.replace(
f"metric('{name}')", replacement
)
if join:
join_clauses.append(join)
# 5. 插入 JOIN 子句
if join_clauses:
rewritten = self._inject_joins(rewritten, join_clauses)
return rewritten
def _generate_replacement(
self,
name: str,
metric: MetricDefinition,
strategy: ComputationStrategy,
) -> tuple[str, str | None]:
"""生成指标的 SQL 替换"""
if strategy == ComputationStrategy.MATERIALIZED:
alias = f"mv_{name}"
replacement = f"{alias}.metric_value"
join = (
f"LEFT JOIN mv_{name} {alias} "
f"ON {alias}.object_id = main.object_id"
)
return replacement, join
elif strategy == ComputationStrategy.VIRTUAL_COLUMN:
replacement = f"main.vc_{name}"
return replacement, None
elif strategy == ComputationStrategy.ROLLUP:
alias = f"rollup_{name}"
replacement = f"{alias}.metric_value"
join = (
f"LEFT JOIN metric_rollup_daily {alias} "
f"ON {alias}.object_id = main.object_id "
f"AND {alias}.metric_name = '{name}'"
)
return replacement, join
else:
# REALTIME / CACHED / UDF: 使用子查询
sub = metric.expression.sql.replace(
"{object_id}", "main.object_id"
)
replacement = f"({sub})"
return replacement, None
def _inject_joins(
self, sql: str, join_clauses: list[str]
) -> str:
"""在 FROM 子句后注入 JOIN"""
joins = "\n".join(join_clauses)
# 简单的 FROM 子句检测
from_pattern = re.compile(
r"(FROM\s+\w+\s+\w+)", re.IGNORECASE
)
match = from_pattern.search(sql)
if match:
insert_pos = match.end()
return sql[:insert_pos] + "\n" + joins + sql[insert_pos:]
return sql
#6. MetricRegistryService
#6.1 注册与管理
class MetricRegistryService:
"""指标注册与管理服务"""
async def register(
self, definition: MetricDefinition
) -> MetricDefinition:
"""注册新指标"""
# 验证
await self._validate(definition)
# 检查重名
existing = await self._repo.find_by_name(definition.name)
if existing:
raise MetricError(
f"Metric '{definition.name}' already exists"
)
# 保存定义
await self._repo.save(definition)
# 如果策略包含 MATERIALIZED,自动创建物化视图
if ComputationStrategy.MATERIALIZED in definition.strategies:
await self._mv_service.create_for_metric(definition)
# 如果策略包含 ROLLUP,注册 Rollup 任务
if ComputationStrategy.ROLLUP in definition.strategies:
await self._rollup_scheduler.register(definition)
# 如果策略包含 VIRTUAL_COLUMN,创建虚拟列
if ComputationStrategy.VIRTUAL_COLUMN in definition.strategies:
await self._doris_schema.add_virtual_column(definition)
logger.info(f"Registered metric: {definition.name}")
return definition
async def update(
self, metric_id: str, updates: MetricUpdateRequest
) -> MetricDefinition:
"""更新指标定义"""
existing = await self._repo.get(metric_id)
if existing is None:
raise MetricNotFoundError(metric_id)
# 应用更新
updated = existing.model_copy(update=updates.model_dump(exclude_unset=True))
updated.version = existing.version + 1
updated.updated_at = datetime.utcnow()
# 检查策略变更
old_strategies = set(existing.strategies)
new_strategies = set(updated.strategies)
added = new_strategies - old_strategies
removed = old_strategies - new_strategies
# 处理新增策略
if ComputationStrategy.MATERIALIZED in added:
await self._mv_service.create_for_metric(updated)
if ComputationStrategy.ROLLUP in added:
await self._rollup_scheduler.register(updated)
# 处理移除策略
if ComputationStrategy.MATERIALIZED in removed:
await self._mv_service.drop_for_metric(existing)
if ComputationStrategy.ROLLUP in removed:
await self._rollup_scheduler.unregister(existing)
await self._repo.save(updated)
return updated
async def delete(self, metric_id: str) -> None:
"""删除指标"""
existing = await self._repo.get(metric_id)
if existing is None:
raise MetricNotFoundError(metric_id)
# 检查是否被其他指标引用(组合指标)
dependents = await self._repo.find_dependents(existing.name)
if dependents:
names = [d.name for d in dependents]
raise MetricError(
f"Cannot delete: referenced by {names}"
)
# 清理关联资源
await self._cleanup(existing)
await self._repo.delete(metric_id)
async def _validate(self, definition: MetricDefinition) -> None:
"""验证指标定义"""
# 检查对象类型存在
obj_type = await self._ontology.get_type(definition.object_type)
if obj_type is None:
raise MetricError(
f"Object type '{definition.object_type}' not found"
)
# 检查 SQL 表达式语法
if definition.expression.type == ExpressionType.SQL:
await self._sql_validator.validate(definition.expression.sql)
# 检查组合指标的依赖
if definition.expression.type == ExpressionType.COMPOSITE:
deps = self.METRIC_PATTERN.findall(definition.expression.sql)
for dep in deps:
dep_metric = await self._repo.find_by_name(dep)
if dep_metric is None:
raise MetricError(
f"Dependent metric '{dep}' not found"
)
#6.2 gRPC 服务定义
service MetricRegistryService {
rpc RegisterMetric(RegisterMetricRequest) returns (MetricDefinition);
rpc UpdateMetric(UpdateMetricRequest) returns (MetricDefinition);
rpc DeleteMetric(DeleteMetricRequest) returns (google.protobuf.Empty);
rpc GetMetric(GetMetricRequest) returns (MetricDefinition);
rpc ListMetrics(ListMetricsRequest) returns (ListMetricsResponse);
rpc ComputeMetric(ComputeMetricRequest) returns (MetricResult);
rpc ComputeMetricBatch(ComputeMetricBatchRequest)
returns (ComputeMetricBatchResponse);
}
#7. 指标监控与可观测性
#7.1 策略使用统计
class MetricObservability:
"""指标系统可观测性"""
async def record_computation(
self, metric_name: str, strategy: ComputationStrategy,
latency_ms: float, success: bool,
) -> None:
labels = {
"metric": metric_name,
"strategy": strategy.value,
"success": str(success),
}
self._histogram.observe(latency_ms, labels)
self._counter.inc(labels)
async def get_strategy_stats(
self, metric_name: str
) -> dict[str, StrategyStats]:
"""获取各策略的使用统计"""
return {
"REALTIME": StrategyStats(
call_count=1200, avg_latency_ms=450,
success_rate=0.98, cache_hit_rate=0.0
),
"CACHED": StrategyStats(
call_count=8500, avg_latency_ms=2,
success_rate=0.99, cache_hit_rate=0.85
),
"MATERIALIZED": StrategyStats(
call_count=3200, avg_latency_ms=12,
success_rate=0.97, cache_hit_rate=0.0
),
}
#8. 端到端流程示例
User: SELECT name, metric('customer_gmv_30d') FROM Customer WHERE metric('customer_gmv_30d') > 50000
|
v
OqlMetricRewriter
├── Extract: ['customer_gmv_30d']
├── Load definition: gmv_30d (strategies=[MATERIALIZED, CACHED, REALTIME])
├── Select strategy: MATERIALIZED (MV exists + data fresh)
└── Rewrite:
SELECT c.name, mv.metric_value AS customer_gmv_30d
FROM customer_objects c
LEFT JOIN mv_customer_gmv_30d mv ON mv.object_id = c.object_id
WHERE mv.metric_value > 50000
|
v
Doris Execution (< 20ms, index scan on MV)
|
v
Result: [{name: "Acme Corp", customer_gmv_30d: 125000}, ...]
#Key Takeaways
- 6 种策略不是互斥的 — 每个指标可配置多种策略,按优先级自动降级
- ComputationCoordinator 是决策中枢 — 它综合策略可用性、数据新鲜度和查询延迟来路由
- OQL 重写是透明的 — 用户写
metric('x'),系统自动展开为最优 SQL - 注册即生效 — 注册指标时自动创建物化视图、Rollup 任务、虚拟列等基础设施
- 批量计算是关键优化 — 批量场景下先查缓存,未命中的再批量实时计算
- 组合指标支持递归依赖 —
metric('avg_gmv')可以引用metric('total_gmv')和metric('customer_count')
#Next Article
下一篇 S3-15 物化视图自动化:注册指标即创建物化视图 将深入物化视图的自动创建、刷新调度、过期检测和 Schema 变更时的级联失效。
Tags: #MetricSystem #ComputationRouting #OQLRewriter #MaterializedView #CacheStrategy #Doris #gRPC #OntologyPlatform