Skip to main content

Core Concepts

Understanding these foundational concepts will help you effectively use DataBridge data quality monitoring.

Data Quality Checks​

Data quality checks are assertions about your data that should always be true. They define the expected state of your data and automatically detect when reality diverges from expectations.

Check Hierarchy​

Checks are organized in three levels:

1. Schema-Level Checks

Validate the structure and presence of columns:

- schema_check:
    expect_columns: [id, name, created_at]
    desc: "Table must have required columns"

- schema_check:
    expect_columns_ordered: [id, name, email, created_at]
    desc: "Columns should be in the correct order"

- schema_check:
    columns_not_present: [deprecated_field]
    desc: "Old columns should be removed"

2. Table-Level Checks

Validate table-wide properties:

- row_count > 1000:
    desc: "Must have minimum data volume"

- row_count between 1000 and 1000000:
    desc: "Row count should be within expected range"

- raw_query: "SELECT COUNT(*) FROM orders WHERE status = 'pending'"
    equals: 0
    desc: "No orders should be stuck in pending"

3. Column-Level Checks

Validate individual column properties:

- not_null(user_id):
    desc: "User ID is mandatory"

- uniqueness(email):
    desc: "Emails must be unique"

- min(price) > 0:
    desc: "Price must be positive"

- freshness(updated_at) < 24h:
    desc: "Data should update daily"

Metrics and Thresholds​

Available Metrics​

DataBridge computes these metrics for your data:

Statistical Metrics

  • row_count - Total number of rows
  • min(column) - Minimum value
  • max(column) - Maximum value
  • avg(column) - Average value
  • sum(column) - Sum of values
  • stddev(column) - Standard deviation

Quality Metrics

  • not_null(column) - Count of null values
  • uniqueness(column) - Percentage of unique values
  • freshness(column) - Time since most recent value
  • Custom via raw_query - Any SQL calculation

Threshold Operators​

Define acceptable ranges using comparison operators:

# Equality
- row_count equals 1000

# Comparisons
- min(price) > 0
- max(quantity) < 10000
- avg(rating) >= 4.0

# Ranges
- row_count between 1000 and 50000
- freshness(updated_at) < 24h

# Time units
- freshness(created_at) < 1d   # 1 day
- freshness(created_at) < 12h  # 12 hours
- freshness(created_at) < 30m  # 30 minutes

Validation Workflows​

When to Validate​

1. Continuous Monitoring (Scheduled) Run checks on a schedule to catch degradation:

# Cron job - run every hour
0 * * * * dbqctl check --checks /path/to/checks.yaml

2. Post-ETL Validation (Event-Driven) Validate after data pipelines complete:

# In your ETL script
run_etl_pipeline()
dbqctl check --checks ./warehouse-checks.yaml || handle_failure()

3. CI/CD Integration (Pre-Deployment) Prevent deploying code that breaks data contracts:

# .github/workflows/test.yml
- name: Validate Data Quality
  run: dbqctl check --checks ./checks.yaml

4. Ad-Hoc Validation (Manual) Investigate issues interactively:

# Profile to understand the data
dbqctl profile -d pg --dataset public.suspicious_table

# Run targeted checks
dbqctl check --checks ./investigation.yaml

Check Execution Flow​

1. Connect to Data Source
   ↓
2. Load Dataset Metadata
   ↓
3. Execute Schema Checks
   ↓
4. Execute Table Checks
   ↓
5. Execute Column Checks
   ↓
6. Report Results
   ↓
7. Trigger Alerts (if failures)

Profiling vs. Assertion​

Understanding the difference between profiling and asserting is key to effective data quality monitoring.

Data Profiling​

Purpose: Understand what your data actually looks like

When to Use:

  • Initial exploration of new datasets
  • Investigating anomalies
  • Establishing baseline for thresholds
  • Documenting data characteristics

Example:

dbqctl profile -d pg --dataset public.users

Output: Descriptive statistics

  • Column types and counts
  • Null/blank percentages
  • Min/Max/Avg values
  • Most frequent values
  • Distribution characteristics

Data Assertion​

Purpose: Verify data meets defined expectations

When to Use:

  • Automated quality gates
  • Continuous monitoring
  • Production validation
  • Preventing bad data propagation

Example:

checks:
  - dataset: pg@[public.users]
    checks:
      - not_null(email):
          desc: "Email is required"

Output: Pass/Fail results

  • Which checks passed
  • Which checks failed
  • Actual vs. expected values
  • Actionable error messages

Workflow Recommendation​

1. Profile → Understand data characteristics
2. Define Checks → Based on profiling insights
3. Assert → Continuously validate expectations
4. Profile Again → When checks fail, investigate
5. Refine Checks → Adjust thresholds based on learnings

Data Sources & Connectors​

Connection Configuration​

Define data sources in dbq.yaml:

version: "1"
datasources:
  - id: production_pg
    type: postgresql
    configuration:
      host: prod-db.example.com
      port: 5432
      username: readonly_user
      password: ${DB_PASSWORD}
      database: analytics
    datasets:
      - public.users
      - public.orders

  - id: clickhouse_events
    type: clickhouse
    configuration:
      host: ch.example.com
      port: 9000
      username: default
      database: events
    datasets:
      - events.pageviews

  - id: mysql_app
    type: mysql
    configuration:
      host: mysql.example.com
      port: 3306
      username: dbq_user
      database: application
    datasets:
      - application.orders

Connection Best Practices​

Security:

  • Use environment variables for secrets
  • Use read-only database users
  • Enable SSL/TLS connections
  • Rotate credentials regularly

Performance:

  • Create database indexes on frequently checked columns
  • Use connection pooling for high-frequency checks
  • Limit row scans with appropriate WHERE clauses
  • Profile during off-peak hours when possible

Dataset References​

Reference datasets using the format: datasource_id@[schema.table]

version: "1"
rules:
  - dataset: pg@[public.users]
    checks: [...]
  - dataset: ch@[events.pageviews]
    checks: [...]
  - dataset: mysql@[ecommerce.orders]
    checks: [...]

Alerts & Notifications​

Alert Triggers​

Alerts are triggered when:

  • Any check fails
  • Threshold violations occur
  • Schema drift is detected
  • Custom alert conditions are met

Alert Configuration​

CLI (Exit Codes):

# dbqctl returns non-zero exit code on failure
dbqctl check --checks checks.yaml || send_alert "Quality check failed"

Cloud (Built-in Alerting):

# Configure in DataBridge Cloud UI
notifications:
  - type: slack
    channel: "#data-quality"
    on: [failure, warning]

  - type: email
    recipients: [team@example.com]
    on: [failure]

  - type: webhook
    url: https://api.example.com/alerts
    on: [failure, warning, success]

Alert Best Practices​

Actionable Alerts:

  • Include descriptive check descriptions
  • Specify expected vs. actual values
  • Link to investigation runbooks
  • Tag appropriate team members

Alert Fatigue Prevention:

  • Set realistic thresholds
  • Group related checks
  • Use warning levels for non-critical issues
  • Implement alert suppression for known issues

Escalation:

  • Page for critical data quality issues
  • Slack for standard failures
  • Email for daily summaries
  • Dashboards for trends

Data Quality Dimensions​

DataBridge checks cover these quality dimensions:

Completeness​

Are all expected values present?

  • Not null checks
  • Required field validation
  • Record count validation

Uniqueness​

Are identifiers truly unique?

  • Primary key uniqueness
  • Composite key validation
  • Deduplication verification

Freshness​

Is the data current?

  • Timestamp recency checks
  • Update frequency validation
  • Staleness detection

Validity​

Do values fall within expected ranges?

  • Min/max range checks
  • Enum value validation
  • Format/pattern matching

Consistency​

Does data remain structurally sound?

  • Schema drift detection
  • Type consistency
  • Cross-table referential integrity

Accuracy​

Does data reflect reality?

  • Business rule validation
  • Cross-source reconciliation
  • Statistical anomaly detection

Volume​

Are there unexpected spikes or drops?

  • Row count ranges
  • Growth rate validation
  • Partition balance checks

Next Steps​

Now that you understand the core concepts: