Data Onboarding: From External Sources to Ontology Objects

“

Series: S4 Ontology Modeling · Article 11 | Level: Intermediate | Reading Time: 18 min

#TL;DR

Data Onboarding is the process of mapping external data sources to the Ontology model — raw data from CSV files, database tables, REST APIs, and message queues is transformed into ObjectType instances through mapping rules, bringing data "alive" in the Ontology.
Three synchronization modes (Full/Incremental/Real-time) cover every scenario from historical data migration to real-time data streams, each with clear applicability conditions, performance characteristics, and consistency guarantees.
Data Quality Gates execute validation before data enters the Ontology — type checks, required field checks, uniqueness checks, and referential integrity checks ensure "garbage data" never pollutes the Ontology.

#1. Why Data Onboarding Is a Key Challenge

#1.1 Diversity of Data Sources

Code

Typical enterprise data source landscape:

┌─────────────────────────────────────────────────┐
│         Enterprise Data Source Panorama           │
│                                                  │
│  Structured Data:                                │
│  ├── PostgreSQL (orders, customers, products)    │
│  ├── MySQL (inventory, warehousing)              │
│  ├── Oracle (ERP system)                         │
│  ├── SQL Server (financial system)               │
│  └── CSV/Excel (manual reports, historical data) │
│                                                  │
│  Semi-Structured Data:                           │
│  ├── REST API (third-party services)             │
│  ├── GraphQL API (partner platforms)             │
│  ├── JSON files (configs, logs)                  │
│  └── XML files (EDI, government data)            │
│                                                  │
│  Streaming Data:                                 │
│  ├── Kafka (event streams)                       │
│  ├── RabbitMQ (message queues)                   │
│  ├── WebSocket (real-time push)                  │
│  └── IoT device data streams                     │
│                                                  │
│  Challenges:                                     │
│  ├── Each data source has a different format     │
│  ├── Each data source has different connectivity │
│  ├── Data quality varies wildly                  │
│  └── All must be unified into the Ontology model │
└─────────────────────────────────────────────────┘

#1.2 coomia-dip Data Onboarding Architecture

Code

Data onboarding pipeline:

External Source → Connector → Mapper → Validator → Loader → Ontology

  Connector: Connects to data source, reads raw data
  Mapper: Maps raw fields to ObjectType properties
  Validator: Executes data quality checks
  Loader: Loads validated data into the Ontology

Each step is configurable, monitorable, and rollbackable.

#2. Connector: Data Source Connection

#2.1 Database Connector

Code

Database connector configuration:

connector:
  connectorId: "conn-pg-orders"
  type: DATABASE
  config:
    dialect: POSTGRESQL
    host: "orders-db.internal"
    port: 5432
    database: "orders"
    username: "${ORDERS_DB_USER}"
    password: "${ORDERS_DB_PASS}"
    schema: "public"
    ssl: true
    connectionPool:
      minSize: 2
      maxSize: 10
      idleTimeout: 300s

  tables:
    - tableName: "orders"
      targetObjectType: "Order"
      syncMode: INCREMENTAL
      incrementalColumn: "updated_at"
      primaryKey: "order_id"

    - tableName: "customers"
      targetObjectType: "Customer"
      syncMode: INCREMENTAL
      incrementalColumn: "modified_at"
      primaryKey: "customer_id"

  schedule:
    type: CRON
    expression: "*/5 * * * *"    # Every 5 minutes
    timezone: "UTC"

#2.2 REST API Connector

Code

REST API connector configuration:

connector:
  connectorId: "conn-api-weather"
  type: REST_API
  config:
    baseUrl: "https://api.weather.com/v3"
    authentication:
      type: API_KEY
      header: "X-API-Key"
      value: "${WEATHER_API_KEY}"
    rateLimit:
      requestsPerSecond: 10
      retryPolicy:
        maxRetries: 3
        backoffMultiplier: 2

  endpoints:
    - path: "/weather/current"
      method: GET
      parameters:
        city: "{{objectId}}"
      targetObjectType: "CityWeather"
      syncMode: FULL
      responseMapping:
        root: "$.data"

  schedule:
    type: INTERVAL
    interval: 15m

#2.3 Kafka Connector

Code

Kafka streaming connector configuration:

connector:
  connectorId: "conn-kafka-events"
  type: KAFKA
  config:
    bootstrapServers: "kafka-1:9092,kafka-2:9092"
    groupId: "coomia-dip-ingestion"
    topics:
      - topic: "order-events"
        targetObjectType: "Order"
        keyField: "orderId"
        format: JSON
      - topic: "user-activities"
        targetObjectType: "UserActivity"
        keyField: "userId"
        format: AVRO
        schemaRegistry: "http://schema-registry:8081"

    consumer:
      autoOffsetReset: EARLIEST
      maxPollRecords: 500
      sessionTimeout: 30s

    syncMode: REALTIME
    errorHandling:
      deadLetterTopic: "coomia-dip-dlq"
      maxRetries: 5

#2.4 CSV/File Connector

Code

File connector configuration:

connector:
  connectorId: "conn-csv-products"
  type: FILE
  config:
    source:
      type: S3
      bucket: "data-imports"
      prefix: "products/"
      filePattern: "*.csv"
    format:
      type: CSV
      delimiter: ","
      header: true
      encoding: "UTF-8"
      quoteChar: '"'
      escapeChar: '\\'
      nullValues: ["", "NULL", "N/A"]

    targetObjectType: "Product"
    syncMode: FULL
    primaryKey: "product_id"

    postProcess:
      archiveProcessed: true
      archivePath: "products/processed/"
      deleteAfterProcess: false

  schedule:
    type: FILE_WATCHER
    pollInterval: 30s

#3. Mapper: Field Mapping

#3.1 Mapping Rule Definition

Code

Field mapping configuration:

mapping:
  sourceConnector: "conn-pg-orders"
  sourceTable: "orders"
  targetObjectType: "Order"

  fieldMappings:
    # Direct mapping (field name and type match)
    - source: "order_id"
      target: "orderId"
      type: DIRECT

    # Type conversion mapping
    - source: "total_amount"
      target: "totalAmount"
      type: CAST
      castConfig:
        from: NUMERIC
        to: DECIMAL
        precision: 18
        scale: 4

    # Enum mapping
    - source: "status"
      target: "orderStatus"
      type: ENUM_MAP
      enumMapping:
        "0": "PENDING"
        "1": "CONFIRMED"
        "2": "SHIPPED"
        "3": "DELIVERED"
        "9": "CANCELLED"

    # Expression mapping
    - source: null
      target: "displayName"
      type: EXPRESSION
      expression: "CONCAT('Order #', order_id, ' - ', customer_name)"

    # Date format conversion
    - source: "created_at"
      target: "createdAt"
      type: DATE_FORMAT
      sourceFormat: "yyyy-MM-dd HH:mm:ss"
      targetFormat: "ISO8601"

    # JSON field extraction
    - source: "metadata"
      target: "shippingAddress"
      type: JSON_PATH
      jsonPath: "$.shipping.address"

    # Relation mapping
    - source: "customer_id"
      target: "customer"
      type: RELATION
      relationConfig:
        relationType: "OrderBelongsToCustomer"
        targetObjectType: "Customer"
        targetProperty: "customerId"

    # Constant value
    - source: null
      target: "dataSource"
      type: CONSTANT
      value: "orders-db"

#3.2 Multi-Source Merging

Code

Multiple data sources mapping to the same ObjectType:

Scenario: Customer data spread across 3 systems

Source 1: CRM System (basic info)
  customer_id → customerId
  full_name → name
  email → email
  phone → phone
  created_date → createdAt

Source 2: Order System (spending info)
  customer_id → customerId (join key)
  total_orders → orderCount
  total_spent → totalSpent
  last_order_date → lastOrderAt

Source 3: Support System (service info)
  cust_id → customerId (join key)
  satisfaction_score → satisfactionScore
  ticket_count → supportTicketCount
  last_contact → lastContactAt

Merge strategy:
  mergeStrategy:
    primarySource: "crm-system"     # Basic info from CRM as authority
    mergeKey: "customerId"          # Join via customerId
    conflictResolution:
      name: PREFER_PRIMARY           # Name from primary source
      email: MOST_RECENT             # Email uses most recent update
      phone: PREFER_PRIMARY           # Phone from primary source
      default: MOST_RECENT           # Other fields use most recent

Merged Customer object contains data from all 3 systems.
Any system's data update reflects in the unified Customer object.

#3.3 Mapping Validation

Code

Mapping rules are auto-validated at registration:

Validation checks:
├── Source field existence: Does the field exist in source table/API
├── Target property existence: Is the property defined in ObjectType
├── Type compatibility: Can source type safely convert to target type
├── Required coverage: All required properties have mappings
├── Primary key mapping: Is primary key mapping defined
├── Relation references: Does the target ObjectType exist
└── Expression syntax: Is the expression syntax correct

Validation report example:
┌─────────────────────────────────────────────────┐
│  Mapping Validation Report                       │
│  Source: orders (PostgreSQL)                     │
│  Target: Order (ObjectType)                      │
├─────────────────────────────────────────────────┤
│  Field mappings: 12                              │
│  Valid: 11                                       │
│  Warnings: 1                                    │
│  Errors: 0                                      │
│                                                  │
│  Warning:                                        │
│  ├── Field "discount_code" in source has no      │
│  │   mapping. Data will be ignored.              │
│                                                  │
│  Coverage:                                       │
│  ├── Source fields mapped: 11/15 (73%)           │
│  ├── Target properties covered: 12/14 (86%)     │
│  ├── Unmapped target: "priority" (has default)   │
│  │   "tags" (nullable)                           │
│  └── All required properties covered: YES        │
└─────────────────────────────────────────────────┘

#4. Validator: Data Quality Gate

#4.1 Quality Check Rules

Code

Layered data quality checking strategy:

Layer 1 — Format checks (fast, per-row):
├── Type matching: Can string parse to target type
├── Length limits: Does string exceed max length
├── Format matching: Date, email, URL format validation
├── Range checks: Is numeric value within allowed range
└── Null checks: Are required fields populated

Layer 2 — Semantic checks (slower, batch):
├── Uniqueness: Are primary keys unique
├── Referential integrity: Do referenced objects exist
├── Enum value check: Is value in enum list
├── Business rules: Custom validation logic
└── Cross-field checks: Logical relationships between fields

Layer 3 — Statistical checks (after batch completes):
├── Anomaly detection: Does value deviate from historical distribution
├── Completeness: Does missing rate exceed threshold
├── Consistency: Comparison with other data sources
└── Trend check: Are there abnormal value fluctuations

#4.2 Quality Check Configuration

Code

Quality check rule configuration:

qualityRules:
  objectType: "Order"
  rules:
    - ruleId: "qr-001"
      name: "Order amount non-negative"
      field: "totalAmount"
      check: RANGE
      config:
        min: 0
        max: 10000000
      severity: ERROR          # Reject if fails

    - ruleId: "qr-002"
      name: "Customer reference integrity"
      field: "customer"
      check: REFERENTIAL_INTEGRITY
      config:
        targetObjectType: "Customer"
        targetField: "customerId"
      severity: ERROR

    - ruleId: "qr-003"
      name: "Email format"
      field: "customerEmail"
      check: REGEX
      config:
        pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
      severity: WARNING        # Mark warning if fails

    - ruleId: "qr-004"
      name: "Order date reasonable"
      field: "orderDate"
      check: EXPRESSION
      config:
        expression: "orderDate <= NOW() AND orderDate >= '2020-01-01'"
      severity: ERROR

    - ruleId: "qr-005"
      name: "Amount anomaly detection"
      field: "totalAmount"
      check: ANOMALY
      config:
        method: Z_SCORE
        threshold: 3.0
        baseline: LAST_30_DAYS
      severity: WARNING

#4.3 Quality Report

Code

Quality report generated for each data onboarding batch:

┌──────────────────────────────────────────────────────┐
│  Data Quality Report                                  │
│  Source: orders (PostgreSQL)                          │
│  Batch: batch-20250115-001                           │
│  Records: 10,000                                     │
├──────────────────────────────────────────────────────┤
│                                                       │
│  Summary:                                             │
│  ├── Total records: 10,000                            │
│  ├── Passed: 9,823 (98.23%)                          │
│  ├── Warnings: 145 (1.45%)                           │
│  ├── Rejected: 32 (0.32%)                            │
│  └── Quality Score: 98.23 / 100                      │
│                                                       │
│  Rule Results:                                        │
│  ├── qr-001 (Amount non-neg): PASS 9,998 / FAIL 2   │
│  ├── qr-002 (Customer ref):  PASS 9,970 / FAIL 30   │
│  ├── qr-003 (Email format):  PASS 9,855 / WARN 145  │
│  ├── qr-004 (Date valid):    PASS 10,000 / FAIL 0   │
│  └── qr-005 (Amount anomaly): PASS 9,980 / WARN 20  │
│                                                       │
│  Rejected Records (sample):                           │
│  ├── Row 3421: totalAmount = -500 (qr-001)           │
│  ├── Row 5892: customer_id = "CUST-9999" not found   │
│  └── Row 7234: totalAmount = -200 (qr-001)           │
│                                                       │
│  Action:                                              │
│  ├── 9,823 records loaded to Ontology                │
│  ├── 145 records loaded with WARNING flag            │
│  ├── 32 records sent to quarantine                   │
│  └── Quarantine records available for manual review  │
└──────────────────────────────────────────────────────┘

#5. Synchronization Modes

#5.1 Full Sync

Code

Applicable scenarios:
├── First-time data import
├── Small data volume (< 1 million rows)
├── Data source has no incremental identifier
└── Requires full consistency

Flow:
  1. Read all data from source table
  2. Map + Validate
  3. Compare with existing Ontology data
  4. Execute INSERT / UPDATE / DELETE
  5. Record sync results

Comparison strategy:
  Source has + Ontology doesn't → INSERT
  Source has + Ontology has + values changed → UPDATE
  Source has + Ontology has + values same → SKIP
  Source doesn't + Ontology has → DELETE (or mark STALE)

Full sync configuration:
syncConfig:
  mode: FULL
  batchSize: 1000
  parallelism: 4
  deletePolicy: SOFT_DELETE    # Don't physically delete, mark inactive
  conflictResolution: SOURCE_WINS
  timeout: 30m

#5.2 Incremental Sync

Code

Applicable scenarios:
├── Data source has updated_at / version field
├── Medium to large data volume (1M ~ 100M rows)
├── Regular sync needed (every 5 min ~ every hour)
└── Brief inconsistency acceptable

Flow:
  1. Read last sync watermark
  2. Query records where updated_at > watermark
  3. Map + Validate
  4. Execute INSERT / UPDATE
  5. Update watermark

Watermark management:
  watermarkStore:
    connectorId: "conn-pg-orders"
    tableName: "orders"
    watermarkColumn: "updated_at"
    currentWatermark: "2025-01-15T10:25:00Z"
    lastSyncRecords: 342
    lastSyncDuration: 12.5s

Incremental sync pitfalls:
├── Deletes: Incremental sync can't detect DELETE in source
│   → Solution: Soft delete field (is_deleted) or periodic full compare
├── Backfill: If someone modifies records before updated_at
│   → Solution: Periodic full validation (e.g., daily)
├── Clock skew: Source database clock may be imprecise
│   → Solution: Watermark overlap (e.g., rewind 30 seconds)
└── Bulk updates: Updating 1M rows at once, incremental query is slow
    → Solution: Batch processing + timeout fallback to full mode

#5.3 Real-time Sync

Code

Applicable scenarios:
├── Second-level latency required
├── Data source supports CDC or message queues
├── Critical business data
└── Batch latency unacceptable

CDC (Change Data Capture) approach:
  1. Listen to database binlog / WAL
  2. Capture INSERT / UPDATE / DELETE events
  3. Map + Validate
  4. Write to Ontology in real-time

  Configuration example:
  connector:
    type: CDC
    config:
      database: POSTGRESQL
      host: "orders-db.internal"
      replicationSlot: "coomia-dip_cdc"
      publication: "orders_pub"
      tables: ["orders", "customers", "line_items"]

  Advantages:
  ├── Low latency (seconds)
  ├── Captures all changes (including DELETE)
  ├── No additional load on source database (reads WAL, not table)
  └── Guarantees ordering

Message queue approach:
  1. Application sends events to Kafka
  2. coomia-dip consumes events
  3. Map + Validate
  4. Write to Ontology

  Even lower latency (milliseconds)
  But requires application cooperation to send events

#6. Data Onboarding Orchestration

#6.1 Dependency Ordering

Code

Multiple ObjectType onboarding has ordering dependencies:

Scenario: E-commerce data onboarding

Dependencies:
  Customer → no dependency (import first)
  Product → no dependency (import first)
  Order → depends on Customer (Customer must exist first)
  LineItem → depends on Order + Product
  Payment → depends on Order

Onboarding order orchestration:
  Phase 1 (parallel): Customer, Product
  Phase 2 (parallel): Order (waits for Phase 1)
  Phase 3 (parallel): LineItem, Payment (waits for Phase 2)

Orchestration configuration:
orchestration:
  name: "ecommerce-full-import"
  phases:
    - phase: 1
      parallel: true
      connectors:
        - "conn-crm-customers"
        - "conn-pim-products"
    - phase: 2
      dependsOn: [1]
      connectors:
        - "conn-oms-orders"
    - phase: 3
      dependsOn: [2]
      parallel: true
      connectors:
        - "conn-oms-lineitems"
        - "conn-pay-payments"

  errorPolicy:
    phaseFailure: STOP_ALL    # Phase failure stops subsequent phases
    connectorFailure: CONTINUE # One connector failure doesn't affect others

#6.2 Backfill Strategies

Code

Backfill strategies for first-time historical data onboarding:

Scenario: Onboarding 5 years of historical order data (50M rows)

Strategy 1 — Time-sliced backfill:
  Slice 5 years of data by month
  Each slice imported independently
  Process multiple slices in parallel

  timeline:
    sliceBy: MONTH
    startDate: "2020-01-01"
    endDate: "2025-01-15"
    parallelSlices: 4

  Result:
  60 months x 830K rows/month
  4 parallel slices
  Estimated time: ~2 hours

Strategy 2 — Priority backfill:
  Import most recent data first (users can use immediately)
  Backfill historical data in background

  priority:
    - range: "LAST_30_DAYS"    # Import last 30 days first
      parallelism: 8
    - range: "LAST_365_DAYS"   # Then past year
      parallelism: 4
    - range: "ALL"             # Then all history
      parallelism: 2

Strategy 3 — Gradual backfill:
  Run at low load during business hours
  Run at high load during off-hours

  schedule:
    daytime:   # 09:00-18:00
      parallelism: 2
      batchSize: 500
    nighttime: # 18:00-09:00
      parallelism: 8
      batchSize: 5000

#7. Error Handling and Quarantine

#7.1 Quarantine Zone

Code

Records that fail validation enter the quarantine zone:

Quarantine purposes:
├── Prevent bad data from entering Ontology
├── Preserve original data for manual review
├── Record failure reasons
├── Support re-import after fixing
└── Provide quality trend analysis

Quarantine record:
{
  "quarantineId": "q-20250115-001",
  "connectorId": "conn-pg-orders",
  "batchId": "batch-20250115-001",
  "sourceRecord": {
    "order_id": "ORD-999",
    "customer_id": "CUST-9999",
    "total_amount": -500,
    "status": "1"
  },
  "targetObjectType": "Order",
  "failedRules": [
    {
      "ruleId": "qr-001",
      "ruleName": "Order amount non-negative",
      "field": "totalAmount",
      "value": -500,
      "reason": "Value -500 is below minimum 0"
    },
    {
      "ruleId": "qr-002",
      "ruleName": "Customer reference integrity",
      "field": "customer_id",
      "value": "CUST-9999",
      "reason": "Customer CUST-9999 not found in Ontology"
    }
  ],
  "quarantinedAt": "2025-01-15T10:30:00Z",
  "status": "PENDING_REVIEW"
}

Quarantine operations:
  Manual review → Fix data → Re-validate → Import to Ontology
  or
  Manual review → Confirm as garbage data → Permanently discard

#7.2 Retry Mechanism

Code

Retry strategy for transient errors:

Transient error types:
├── Network timeout
├── Database connection lost
├── Message queue unavailable
├── API rate limited (429)
└── Target storage temporarily unavailable

Retry configuration:
retryPolicy:
  maxRetries: 5
  backoff:
    type: EXPONENTIAL
    initialDelay: 1s
    maxDelay: 60s
    multiplier: 2
  retryableErrors:
    - CONNECTION_TIMEOUT
    - TEMPORARY_UNAVAILABLE
    - RATE_LIMITED
  nonRetryableErrors:
    - VALIDATION_FAILED
    - MAPPING_ERROR
    - AUTHENTICATION_FAILED

Retry example:
  Attempt 1: Failed (timeout) → wait 1s
  Attempt 2: Failed (timeout) → wait 2s
  Attempt 3: Failed (timeout) → wait 4s
  Attempt 4: Success ✓

Dead Letter Queue (DLQ):
  All 5 retries failed → message goes to DLQ
  DLQ messages are not auto-retried
  Requires manual investigation then manual replay

#8. Data Onboarding Monitoring

#8.1 Onboarding Status Dashboard

Code

Data onboarding monitoring panel:

┌──────────────────────────────────────────────────────┐
│  Data Onboarding Dashboard                            │
├──────────────────────────────────────────────────────┤
│                                                       │
│  Active Connectors: 12 / 15                          │
│  ├── Healthy: 10                                     │
│  ├── Warning: 2 (lag > 5min)                         │
│  └── Error: 0                                        │
│                                                       │
│  Last 24h Summary:                                   │
│  ├── Records processed: 2,345,678                    │
│  ├── Records loaded: 2,310,234 (98.5%)              │
│  ├── Records quarantined: 35,444 (1.5%)             │
│  ├── Average latency: 3.2s                          │
│  └── Peak throughput: 12,500 records/s              │
│                                                       │
│  Connector Health:                                   │
│  ┌──────────────────┬────────┬─────────┬──────────┐ │
│  │ Connector        │ Status │ Lag     │ Quality  │ │
│  ├──────────────────┼────────┼─────────┼──────────┤ │
│  │ pg-orders        │ OK     │ 30s    │ 99.2%    │ │
│  │ kafka-events     │ OK     │ 2s     │ 97.8%    │ │
│  │ api-weather      │ WARN   │ 8m     │ 100%     │ │
│  │ csv-products     │ OK     │ 0s     │ 98.5%    │ │
│  │ cdc-inventory    │ OK     │ 1s     │ 99.9%    │ │
│  └──────────────────┴────────┴─────────┴──────────┘ │
└──────────────────────────────────────────────────────┘

#8.2 Data Lineage Tracking

Code

Every Ontology record is traceable to its data source:

Data Lineage:

Query: Where did this Order's data come from?

GET /api/v1/objects/Order/ORD-001/lineage

{
  "objectId": "ORD-001",
  "objectType": "Order",
  "sources": [
    {
      "connector": "conn-pg-orders",
      "table": "orders",
      "primaryKey": "ORD-001",
      "lastSyncAt": "2025-01-15T10:30:00Z",
      "syncMode": "INCREMENTAL",
      "batchId": "batch-20250115-042"
    }
  ],
  "propertyLineage": {
    "orderId": {"source": "orders.order_id", "transform": "DIRECT"},
    "totalAmount": {"source": "orders.total_amount", "transform": "CAST(DECIMAL)"},
    "orderStatus": {"source": "orders.status", "transform": "ENUM_MAP(0→PENDING,1→CONFIRMED...)"},
    "customer": {"source": "orders.customer_id", "transform": "RELATION(Customer)"}
  },
  "qualityFlags": {
    "customerEmail": "WARNING: invalid format"
  }
}

Purposes:
├── Quality debugging: Where did this odd value come from?
├── Compliance audit: What's the source of this customer data?
├── Impact analysis: If source schema changes, which Ontology objects are affected?
└── Debugging: Why doesn't this object's value match the source data?

#9. Practical Example: E-Commerce Data Onboarding

#9.1 Complete Onboarding Plan

Code

E-commerce platform data onboarding panorama:

┌─────────────────────────────────────────────────────┐
│                                                      │
│  [CRM MySQL] ──CDC──→ Customer (real-time)           │
│  [PIM PostgreSQL] ──Incr──→ Product (5 min)          │
│  [OMS PostgreSQL] ──CDC──→ Order, LineItem (real-time)│
│  [Payment Gateway API] ──Incr──→ Payment (1 min)     │
│  [Kafka Events] ──RT──→ UserActivity (real-time)     │
│  [Logistics API] ──Incr──→ Shipment (15 min)         │
│  [CSV Reports] ──Full──→ FinancialReport (daily)     │
│  [IoT MQTT] ──RT──→ WarehouseDevice (real-time)      │
│                                                      │
│  Onboarding order:                                   │
│  Phase 1: Customer, Product (master data)            │
│  Phase 2: Order (depends on Customer)                │
│  Phase 3: LineItem, Payment, Shipment (depend on Order)│
│  Phase 4: UserActivity, WarehouseDevice (independent) │
│  Phase 5: FinancialReport (daily summary)            │
│                                                      │
│  Volume estimates:                                   │
│  Customer: 5M, incremental ~1,000/day                │
│  Product: 1M, incremental ~500/day                   │
│  Order: 50M, incremental ~50,000/day                 │
│  LineItem: 200M, incremental ~200,000/day            │
│  UserActivity: ~10M/day (event stream)               │
└─────────────────────────────────────────────────────┘

#10. Comparison with Palantir Foundry

Code

Data onboarding capability comparison:

┌──────────────────┬────────────────────┬────────────────────┐
│ Feature          │ coomia-dip          │ Palantir Foundry   │
├──────────────────┼────────────────────┼────────────────────┤
│ Connector types  │ DB/API/Kafka/File  │ 200+ connectors    │
│ Mapping approach │ Declarative YAML   │ Pipeline Builder   │
│ Quality checks   │ Multi-layer valid. │ Checks framework   │
│ CDC support      │ PostgreSQL/MySQL   │ Multi-database     │
│ Real-time stream │ Kafka/MQTT         │ Multiple platforms │
│ Data lineage     │ Property-level     │ Column-level       │
│ Quarantine       │ Yes                │ Yes                │
│ Multi-source     │ Yes                │ Yes                │
│ Orchestration    │ Phase-based        │ Pipeline DAG       │
│ Visualization    │ Dashboard          │ Pipeline visual    │
│ Community conn.  │ In development     │ Rich ecosystem     │
└──────────────────┴────────────────────┴────────────────────┘

coomia-dip data onboarding goal is not to replicate Foundry's 200+ connectors,
but to provide a flexible framework where 80% of common scenarios
(relational databases, REST APIs, Kafka, CSV) work out of the box.

#Key Takeaways

Data onboarding is the Ontology's "lifeline" — no matter how perfect the ObjectType definitions, without reliable data onboarding pipelines, the Ontology is an empty shell. The Connector + Mapper + Validator + Loader four-step pipeline ensures data flows reliably from external sources into the Ontology.
Three sync modes cover all scenarios — Full (first import, small data), Incremental (periodic sync, medium-large data), Real-time (CDC/Kafka, second-level latency). Choose based on business needs and data source capabilities.
Data Quality Gates are the core defense — format checks, semantic checks, and statistical checks in three layers ensure "garbage data" is intercepted outside the Ontology. The quarantine zone preserves original data for manual review and repair.
Multi-source merging creates a unified view — a Customer's data may come from CRM, order system, and support system. The merge strategy auto-handles conflicts, and consumers see a unified Customer object.
Data lineage is the foundation of trust — every piece of data traces back to its original source, mapping rules, and quality flags. When data issues arise, you can quickly pinpoint which data source and which sync batch introduced the problem.

#Next Article

The next article S4-12 Auto-Provisioning discusses automated infrastructure provisioning for the Ontology — when a new ObjectType is registered, the system automatically creates storage tables, indexes, API endpoints, and permission configurations, achieving "define and deploy" with zero manual intervention.

#ontology #data-onboarding #etl #connector #data-quality #cdc #incremental-sync #data-lineage #mapping