“

Series: S14 Engineering Stories · Article 10 | Level: Intermediate | Reading Time: 15 min

Testing Strategy Evolution

#TL;DR

coomia-dip 的测试策略经历了从无到有的演进：最初的零测试、手动验证阶段，到单元测试覆盖核心逻辑，到集成测试验证 Layer 间通信，再到场景化端到端测试。本文记录了每个阶段的驱动因素、技术选型和经验教训。

#1. Background

#1.1 Where It Started

Every engineering decision has a story behind it. coomia-dip, as an open-source project benchmarked against Palantir Foundry, went through countless "tear it down and rebuild" moments during technology selection and architecture evolution. This article documents one such typical case -- testing strategy evolution.

Building an enterprise PaaS platform with a 4-person team means every decision must balance "ideal" against "reality." Perfect architecture looks great on a whiteboard, but with limited manpower and time, you must make trade-offs.

#1.2 Problem Definition

coomia-dip 的测试策略经历了从无到有的演进：最初的零测试、手动验证阶段，到单元测试覆盖核心逻辑，到集成测试验证 Layer 间通信，再到场景化端到端测试。本文记录了每个阶段的驱动因素、技术选型和经验教训。. This seemingly simple problem actually involves technical challenges across multiple layers.

From a project management perspective, this type of problem typically exhibits:

• Initially underestimated: Seems "should be easy" during planning
• Mid-course depth discovery: Actual implementation reveals far more complexity than imagined
• Post-resolution experience: Solutions become valuable team knowledge assets

#2. Exploration Phase

#2.1 Solution Research

We first surveyed common industry approaches:

options = {
    "Option A": {
        "description": "Most direct approach",
        "pros": ["Simple to implement", "Good community support"],
        "cons": ["Performance ceiling", "Poor extensibility"],
        "effort": "1 week",
    },
    "Option B": {
        "description": "Middle-ground approach",
        "pros": ["Moderate performance", "Good maintainability"],
        "cons": ["Some custom components needed"],
        "effort": "2 weeks",
    },
    "Option C": {
        "description": "Ultimate approach",
        "pros": ["Best performance", "Full control"],
        "cons": ["Complex implementation", "High maintenance"],
        "effort": "4 weeks",
    },
}

#2.2 Prototype Validation

We chose Option B as the starting point and built a quick prototype in one week:

class PrototypeImplementation:
    async def initialize(self):
        assert self._check_assumption_1(), "Assumption 1 invalid"
        assert self._check_assumption_2(), "Assumption 2 invalid"

    async def run_benchmark(self) -> dict:
        results = {}
        start = time.perf_counter()
        for i in range(10000):
            await self.process(generate_test_data())
        elapsed = time.perf_counter() - start
        results["throughput"] = 10000 / elapsed

        latencies = []
        for i in range(1000):
            t0 = time.perf_counter()
            await self.process(generate_test_data())
            latencies.append((time.perf_counter() - t0) * 1000)
        latencies.sort()
        results["p50"] = latencies[500]
        results["p99"] = latencies[990]
        return results

#3. Implementation

#3.1 Week 1: Core Implementation

The first week focused on core functionality. The key surprise was discovering that what we assumed could be a simple mapping actually involved nested references requiring topological sorting.

class CoreImplementation:
    async def process(self, input_data: InputData) -> ProcessResult:
        with self.metrics.timer("process_duration"):
            validated = self._validate(input_data)
            transformed = await self._transform(validated)
            result = await self._persist(transformed)
            self.metrics.increment("processed_total")
            return result

    async def _transform(self, data):
        graph = self._build_dependency_graph(data)
        ordered = topological_sort(graph)
        results = []
        for node in ordered:
            result = await self._process_node(node, results)
            results.append(result)
        return TransformedData(nodes=results)

#3.2 Week 2: Edge Cases

Week two was the most painful -- all the edge cases surfaced: circular dependencies, data inconsistencies, timeout handling. Each required specific error handling strategies.

#3.3 Week 3: Optimization & Testing

benchmark_results = {
    "before": {"throughput": 500, "p50": 15, "p95": 120, "p99": 350},
    "after": {"throughput": 3500, "p50": 3, "p95": 18, "p99": 45},
}
# Techniques: batch processing, connection pooling, caching, async I/O

#4. Pitfalls Encountered

#4.1 Pitfall: False Atomicity Assumption

Symptom: Works in dev, intermittent integration test failures

Root Cause: Assumed operation atomicity, but race conditions exist under concurrency

Fix: Proper locking

# Wrong
async def update_if_exists(obj_id, data):
    obj = await repo.get(obj_id)
    if obj:
        await repo.save(obj.update(data))

# Right
async def update_if_exists(obj_id, data):
    async with repo.lock(obj_id):
        obj = await repo.get(obj_id)
        if obj:
            await repo.save(obj.update(data))

#4.2 Pitfall: Configuration Hell

Problem: Different behavior across environments due to config scattered across env vars, files, and code defaults

Fix: Unified configuration management with Pydantic Settings

#4.3 Pitfall: Log Flood

Problem: Too many logs in production make it impossible to find critical information

Fix: Structured logging + request tracing with correlation IDs

#5. Lessons Learned

#5.1 Technical Lessons

#5.2 Team Lessons

• Small team advantage: Fast decisions, low communication overhead, everyone understands the big picture
• Small team disadvantage: Limited capacity, can't parallelize too much, individual departures have high impact
• Key strategy: Prioritize subtraction (reduce unnecessary complexity) over addition

#5.3 If We Could Do It Again

1. Introduce automated testing earlier, not after "features are basically done"
2. Be stricter about dependency count -- think three times before adding any external dependency
3. Document the "why" of every major decision in ADRs, not just the "what"

#6. Advice for Readers

If you are building a similar system:

1. Don't pursue perfect architecture: Get it running first, then optimize. Good architecture evolves; it is not designed upfront
2. Record decision processes: ADRs (Architecture Decision Records) are the best team memory
3. Embrace constraints: Technical red lines are not restrictions but guardrails preventing bigger mistakes
4. Measure, don't guess: Performance issues need data, not intuition-based optimization

#Conclusion

coomia-dip 的测试策略经历了从无到有的演进：最初的零测试、手动验证阶段，到单元测试覆盖核心逻辑，到集成测试验证 Layer 间通信，再到场景化端到端测试。本文记录了每个阶段的驱动因素、技术选型和经验教训。 -- this seemingly simple problem ultimately taught us lessons far more important than the technology itself: how to make decisions under uncertainty, and how to deliver high-quality software with limited resources.

We hope this chronicle provides useful reference for teams facing similar challenges. The coomia-dip engineering journey continues.

Next: [S14-11] Previous: [S14-09]