“

Series: S14 Engineering Stories · Article 1 | Level: Intermediate | Reading Time: 15 min

Dev Diary: From Line 1 to 6000+ Tests

#TL;DR

This article is the complete development diary of the coomia-dip project, from inception to the first public beta. Over 14 months, we went from a single print("hello ontology") to building an ontology-driven intelligent decision PaaS platform benchmarked against Palantir Foundry. As of this writing, the project has over 6,000 automated test cases covering 8 architectural Layers, with more than 150,000 lines of code. This article walks through our timeline of key milestones, technical decisions, pitfalls encountered, and the lessons we learned.

#1. Origins: Why We Did This

#1.1 The Palantir Foundry Revelation

In early 2024, while evaluating data platform solutions for a manufacturing client, our team conducted a deep study of Palantir Foundry. We were profoundly struck by its core philosophy — "Ontology drives everything." Traditional data platforms remain at the "table-column-row" abstraction level, while Foundry elevates business concepts (devices, work orders, suppliers) to first-class citizens, with all data pipelines, analytics, permissions, and operations revolving around these ontology objects.

However, Foundry had two fatal flaws: price — starting at several million USD per year — and data sovereignty — the SaaS model meant data leaving national borders. For domestic manufacturing and financial clients, both were non-starters.

#1.2 The Go Decision

After two weeks of feasibility analysis, we concluded that Foundry's core architecture was replicable. The key lay in three core components: Ontology Runtime, Schema Registry, and Action Engine. We decided to launch coomia-dip — a privately deployable, open-source-friendly Palantir Foundry alternative.

#1.3 Team Formation

The initial team was just 4 people:

• 1 architect (responsible for overall design and Control Layer)
• 1 backend engineer (responsible for Data Layer)
• 1 AI engineer (responsible for Intelligence Layer)
• 1 full-stack engineer (responsible for SDK and frontend)

This lean team configuration was later proven to be both an advantage (fast decisions, low communication overhead) and a disadvantage (when shorthanded, overtime was the only option).

#2. Month One: Laying the Foundation (Day 1 – Day 30)

#2.1 Birth of the 8-Layer Architecture

The entire first week was spent on architecture discussions. We finalized an 8-Layer architecture:

Why 8 Layers? Because we believe in "separation of concerns." Each Layer has its own lifecycle, freedom to choose technology stacks, and ability to deploy independently. Of course, we later discovered that 8 Layers was too heavy during development — but that's a story for later (see S14-03).

#2.2 The First Line of Code

# ontology_sdk/__init__.py - 2024-02-15
print("hello ontology")
__version__ = "0.0.1"

This line still exists in the git history. It reminds us that every massive system starts from a simple beginning.

#2.3 Key Technology Decisions

gRPC over REST: All internal service communication uses gRPC. This was decided in the first week because of:

• Strongly-typed contracts (Protobuf IDL)
• Bidirectional streaming
• Code generation reducing boilerplate
• Performance (binary serialization + HTTP/2 multiplexing)

Gradle over Maven: All Java projects use Gradle uniformly. Maven's XML configuration is too verbose for multi-module projects.

Python for Intelligence Layer: The reasoning engine and Agent runtime use Python + FastAPI because the AI/ML ecosystem is almost entirely in Python.

#3. Months Two and Three: Core Components (Day 31 – Day 90)

#3.1 Ontology Runtime v1

The Ontology Runtime is the heart of the entire platform. It is responsible for:

• Managing ObjectType and LinkType definitions
• Maintaining Object instance lifecycles
• Providing the OQL (Ontology Query Language) query interface

The first version was rudimentary — in-memory HashMap storage, no persistence, no transactions. But it ran the core flow: define an ObjectType → create Objects → query via OQL.

#3.2 Schema Registry

The Schema Registry manages all ontology type metadata. We referenced Confluent Schema Registry's design but added ontology-specific extensions:

• Support for three first-class types: ObjectType, LinkType, ActionType
• Version management (each modification generates a new version; old versions are immutable)
• Compatibility checking (new versions must be backward compatible)

#3.3 The First Integration Test

On Day 60, we ran the first end-to-end integration test successfully:

1. Create ObjectType "Device" via SDK
2. Add properties name, status, location to Device
3. Create 3 Device instances
4. Query devices with status == "running" via OQL
5. Verify 2 results returned

Although simple, this test wired together the complete SDK → Control Layer → Data Layer chain. The team celebrated — this was the first time the system "came alive."

#3.4 Initial Testing Strategy

From the start, we committed to a "test-driven" approach. Each Layer independently maintains its own test suite:

• Unit tests: covering core business logic
• Integration tests: verifying intra-Layer component collaboration
• Contract tests: verifying gRPC interface compatibility
• End-to-end tests: cross-Layer complete business flows

By Day 90, the test count reached approximately 400.

#4. Months Four to Six: Feature Explosion (Day 91 – Day 180)

#4.1 Action Engine

The Action Engine is coomia-dip's "executor." It allows users to define Actions (such as "approve work order" or "adjust inventory") and performs permission checks, parameter validation, and side-effect management during execution.

The Action Engine's core design principle is "reversibility": every Action must define both apply and revert methods. This borrows from the Saga pattern — in a distributed environment, if an operation fails, the system needs to be able to roll back completed steps.

#4.2 Rule Engine

The Rule Engine implements forward-chaining inference. Users can define rules (such as "when device temperature > 80°C and runtime > 24h, trigger a maintenance work order"), and the system automatically evaluates rules and executes corresponding Actions when data changes.

We initially tried Drools but found it too heavy — it dragged in the entire KIE ecosystem. We ultimately built a lightweight rule engine based on a simplified version of the Rete algorithm.

#4.3 Data Pipelines

Data pipelines are the channels for importing external data into coomia-dip. We integrated DolphinScheduler as the scheduling engine, supporting:

• Batch imports (CSV, JSON, Parquet)
• Incremental sync (CDC via Debezium)
• Real-time streaming (Kafka Consumer)

#4.4 Permission Model

The permission model is the core of every enterprise-grade platform. We implemented a three-layer permission model:

• RBAC (Role-Based Access Control)
• ABAC (Attribute-Based Access Control)
• Row-level permissions (data filtering based on ontology attributes)

These three layers can be combined. For example: "Finance department (RBAC) can view (ABAC: read-only) work orders from their own department (row-level: department == user.department)."

#4.5 Test Count Explosion

By Day 180, the test count reached approximately 1,800. The fastest growth was in Action Engine and Rule Engine tests — their edge cases are numerous.

#5. Months Seven to Nine: Stability Hardening (Day 181 – Day 270)

#5.1 Performance Issues Surface

As test data volumes increased, performance issues started to surface:

• OQL queries exceeded 5 seconds at the 100K data scale
• Action Engine concurrent execution had deadlock risks
• Rule Engine evaluation grew exponentially with complex rule sets

#5.2 OQL Query Optimization

OQL query optimization was the biggest technical challenge of this phase (see S14-09 for details). Core improvements included:

• Introducing a query plan optimizer
• Implementing predicate pushdown to the storage layer
• Adding query result caching
• Optimizing JOIN strategies (from nested loops to Hash Join)

After optimization, the same query dropped from 5 seconds to 200ms — a 25x improvement.

#5.3 Concurrency Control

The Action Engine's deadlock issue stemmed from multiple Actions modifying the same Object simultaneously. We introduced Optimistic Concurrency Control (OCC):

• Each Object maintains a version number
• Actions check the version number during execution
• Automatic retry on conflict (up to 3 times)

#5.4 Storage Architecture Evolution

The storage architecture went through three major changes:

1. v1: PostgreSQL + MinIO (simple, but limited query capability)
2. v2: PostgreSQL + ClickHouse + MinIO (good OLAP performance, but operationally complex)
3. v3: Doris + Iceberg/Nessie + MinIO (unified OLTP/OLAP, versioned storage)

The specific reasons and processes for each migration are detailed in S14-02 and S14-04.

#5.5 Continuous Test Growth

By Day 270, the test count reached approximately 3,500. We began introducing performance and stress tests — using Locust to simulate 1,000 concurrent users.

#6. Months Ten to Twelve: Productization (Day 271 – Day 365)

#6.1 SDK 1.0

The Python SDK is the primary interface for users to interact with coomia-dip. The design goals for SDK 1.0 were:

• Intuitiveness: API naming should let developers who have never used an ontology platform guess the usage
• Type safety: All APIs have complete type annotations so IDEs can provide autocomplete
• Progressive: Simple scenarios handled in one line; complex scenarios composed via the Builder pattern

from ontology_sdk import OntoPlatform

platform = OntoPlatform.connect("localhost:8080")

# One line to create an object
device = platform.objects.create("Device", name="CNC-001", status="running")

# One line to query
running_devices = platform.objects.query("Device").where(status="running").list()

# One line to execute an action
platform.actions.execute("MaintenanceCheck", target=device)

#6.2 Deployment Options

We provided three deployment options:

• Development mode: docker compose up, all services on a single machine
• Testing mode: 3-node Docker Swarm, simulating a distributed environment
• Production mode: Kubernetes + Helm Chart, with horizontal scaling support

#6.3 Documentation System

Documentation is key to productization. We established a comprehensive documentation system:

• Quick start (5-minute walkthrough)
• API reference
• Architecture guide
• Industry case studies
• Technical article series (the very series you're reading)

#6.4 Test Milestone

By Day 365, the test count reached approximately 5,200. The pass rate stayed above 99.5%.

#7. Months Thirteen and Fourteen: Polish and Open Source (Day 366 – Day 420)

#7.1 Introducing AI-Assisted Development

From the thirteenth month onward, we adopted AI-assisted development at scale (see S14-11 for details). Claude was used for:

• Code review
• Test case generation
• Documentation writing
• Bug debugging

The introduction of AI significantly accelerated test case writing — from an average of 15 per day to 40.

#7.2 Open Source Preparation

Open sourcing isn't just putting code on GitHub. We did extensive preparation:

• Code audit (removing hardcoded secrets, internal IPs)
• License selection (Apache 2.0)
• Contributor guide
• Issue templates
• CI/CD pipeline (GitHub Actions)

#7.3 What 6000+ Tests Mean

As of Day 420, test cases exceeded 6,000. The distribution was:

6,000+ tests is more than just a number. It represents:

• Confidence with every commit that existing features won't break
• A safety net during refactoring (we did 3 major refactors, all backed by tests)
• "Executable documentation" for new team members during onboarding
• Regression bugs reduced to near zero

#8. Key Milestone Timeline

#9. Deepest Lessons Learned

#9.1 Premature vs. Late Optimization

We made the "premature optimization" mistake with our storage architecture — introducing ClickHouse in month two, only to discover it wasn't better than Doris for our use case and added operational complexity. But we also made the "late optimization" mistake with OQL queries — waiting until users complained before starting optimization, resulting in fix costs far exceeding what they would have been if we'd considered performance during the design phase.

Lesson: Make it work, make it right, then make it fast. But "then" doesn't mean "when users start complaining."

#9.2 Tests Are an Investment, Not a Cost

Every time someone proposed "skip tests for now and ship features," we insisted on writing tests. This did slow delivery in the short term, but during the month-seven stability hardening phase, those tests saved us — they let us refactor boldly without fear of breaking functionality.

Lesson: The ROI of tests becomes visible after month six and grows exponentially after that.

#9.3 Communication Costs of 4 People Were Underestimated

Although a 4-person team is lean, when each person owns a different Layer, the communication cost of interface changes far exceeded expectations. We later introduced Protobuf as the "interface contract" — any interface change required a PR to the .proto file first, which had to be approved by others before implementation.

Lesson: Small team doesn't mean no process.

#10. Final Thoughts

From line 1 to 6,000+ tests, the coomia-dip development journey was far more winding than this article can convey. We experienced repeated overturning of technical approaches, late-night debugging sessions, performance bottleneck anxieties, and also savored every moment of "it works!" triumph.

The subsequent articles in this series will dive deep into each key decision and technical challenge. If you're doing something similar — building a complex platform-level product — we hope our experience can help you avoid some detours.

#Key Takeaways

1. Architecture design needs room for evolution: The 8-Layer architecture was ideal early on, but required balancing between team size and development efficiency
2. Tests are the best investment: 6,000+ tests gave the team confidence for major refactoring
3. Small teams still need process: A 4-person team still needs interface contracts and code review
4. Be cautious with storage decisions: Every storage architecture migration cost 3-5x more than expected
5. Plan performance optimization: Don't wait for user complaints, but don't optimize prematurely either

#Next Article

Next: S14-02 Why We Abandoned ClickHouse — We'll detail the ClickHouse saga in our storage selection and why Doris ultimately won.

Tags: #coomia-dip #DevDiary #ArchitectureDesign #TestingStrategy #EngineeringStories #PalantirAlternative