User Guide & CLI Reference

Cloud Dashboard is the recommended approach

Profiling and validation tasks are now managed through the DataBridge Dashboard and Agent. The CLI commands documented here (dbqctl) reflect the legacy local workflow.

See DataBridge Cloud and the Agent Guide for the current cloud-based setup.

Comprehensive guides and reference documentation for using dbqctl effectively.

How-To Guides

Connecting to Data Sources

PostgreSQL

# ~/.dbq.yaml
version: "1"
datasources:
  - id: prod_pg
    type: postgresql
    configuration:
      host: postgres.example.com
      port: 5432
      username: readonly_user
      password: ${PG_PASSWORD}
      database: analytics
    datasets:
      - public.users
      - public.orders

Test connection:

export PG_PASSWORD="your-password"
dbqctl ping -d prod_pg

ClickHouse

datasources:
  - id: clickhouse
    type: clickhouse
    configuration:
      host: clickhouse.example.com
      port: 9000
      username: default
      password: ${CH_PASSWORD}
      database: events
    datasets:
      - events.pageviews

Import tables:

dbqctl import -d clickhouse --filter "events"

MySQL

datasources:
  - id: mysql_app
    type: mysql
    configuration:
      host: mysql.example.com
      port: 3306
      username: dbq_user
      password: ${MYSQL_PASSWORD}
      database: application
    datasets:
      - application.orders

Using SSH Tunnels

For databases behind firewalls:

# Create SSH tunnel
ssh -L 5432:internal-db.example.com:5432 bastion.example.com -N &

# Configure datasource to use localhost
# ~/.dbq.yaml
datasources:
  - id: tunneled_pg
    type: postgres
    host: localhost
    port: 5432
    database: analytics
    user: readonly_user
    password: ${PG_PASSWORD}

Creating Validation Rules

Start with Profiling

Before writing checks, profile your data to understand its characteristics:

dbqctl profile -d pg --dataset public.users

Review the output to determine appropriate thresholds:

Null percentages → Set not_null checks
Value ranges → Set min/max checks
Unique counts → Set uniqueness checks
Freshness → Set freshness expectations

Write Your First Check

Create checks.yaml:

version: "1"
rules:
  - dataset: pg@[public.users]
    checks:
      # Start with simple checks
      - not_null(user_id):
          desc: "User ID is required"

      - row_count > 0:
          desc: "Table should not be empty"

Test the check:

dbqctl check --checks checks.yaml

Incrementally Add Checks

Add checks one at a time and validate:

version: "1"
rules:
  - dataset: pg@[public.users]
    checks:
      - not_null(user_id):
          desc: "User ID is required"

      - row_count > 1000:
          desc: "Should have at least 1000 users"

      # Add more checks as you validate each one
      - uniqueness(email):
          desc: "Emails must be unique"
          on_fail: error

Organize Checks by Dataset

Group related checks together:

version: "1"
rules:
  # User table checks
  - dataset: pg@[public.users]
    checks:
      - not_null(user_id)
      - uniqueness(email):
          on_fail: error
      - freshness(created_at) < 24h:
          on_fail: warn

  # Orders table checks
  - dataset: pg@[public.orders]
    checks:
      - not_null(order_id)
      - min(total) > 0
      - row_count > 100

Running Checks Automatically

CI/CD Integration

GitHub Actions:

# .github/workflows/data-quality.yml
name: Data Quality Checks

on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:  # Manual trigger

jobs:
  quality-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Install dbqctl
        run: |
          curl -L https://github.com/DataBridgeTech/dbqctl/releases/latest/download/dbqctl-linux-amd64 -o dbqctl
          chmod +x dbqctl
          sudo mv dbqctl /usr/local/bin/

      - name: Run Quality Checks
        env:
          PG_PASSWORD: ${{ secrets.PG_PASSWORD }}
        run: dbqctl check --checks ./data-quality/checks.yaml

      - name: Notify on Failure
        if: failure()
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: 'Data quality checks failed!'
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}

GitLab CI:

# .gitlab-ci.yml
data-quality:
  stage: test
  image: alpine:latest
  before_script:
    - apk add --no-cache curl
    - curl -L https://github.com/DataBridgeTech/dbqctl/releases/latest/download/dbqctl-linux-amd64 -o dbqctl
    - chmod +x dbqctl
  script:
    - ./dbqctl check --checks ./checks.yaml
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - if: '$CI_PIPELINE_SOURCE == "web"'

Cron Jobs

# Run checks hourly
# crontab -e
0 * * * * /usr/local/bin/dbqctl check --checks /path/to/checks.yaml || echo "Quality checks failed" | mail -s "Data Quality Alert" team@example.com

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email': ['team@example.com'],
    'email_on_failure': True,
    'retries': 1,
}

dag = DAG(
    'data_quality_checks',
    default_args=default_args,
    description='Run data quality checks',
    schedule_interval='0 */6 * * *',  # Every 6 hours
    start_date=datetime(2024, 1, 1),
    catchup=False,
)

run_checks = BashOperator(
    task_id='run_quality_checks',
    bash_command='dbqctl check --checks /opt/airflow/config/checks.yaml',
    dag=dag,
)

Post-ETL Validation

Shell Script:

#!/bin/bash
# etl-with-validation.sh

set -e  # Exit on error

echo "Running ETL pipeline..."
python run_etl.py

echo "Validating data quality..."
if dbqctl check --checks ./checks.yaml; then
    echo "✓ Quality checks passed"
    exit 0
else
    echo "✗ Quality checks failed"
    # Rollback or send alert
    send_alert "ETL validation failed"
    exit 1
fi

Python Script:

import subprocess
import sys

def run_etl():
    # Your ETL logic
    print("Running ETL...")

def validate_quality():
    result = subprocess.run(
        ['dbqctl', 'check', '--checks', './checks.yaml'],
        capture_output=True
    )
    return result.returncode == 0

if __name__ == '__main__':
    run_etl()

    if validate_quality():
        print("✓ Data quality validated")
        sys.exit(0)
    else:
        print("✗ Data quality check failed")
        send_alert("Quality validation failed")
        sys.exit(1)

Alerting & Notifications

Slack Notifications

#!/bin/bash
# run-checks-with-slack.sh

SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

if dbqctl check --checks checks.yaml; then
    # Success
    curl -X POST $SLACK_WEBHOOK \
        -H 'Content-Type: application/json' \
        -d '{"text":"✓ Data quality checks passed"}'
else
    # Failure
    curl -X POST $SLACK_WEBHOOK \
        -H 'Content-Type: application/json' \
        -d '{"text":"✗ Data quality checks FAILED! <!channel>","username":"DataBridge"}'
fi

Email Notifications

#!/bin/bash
# run-checks-with-email.sh

OUTPUT=$(dbqctl check --checks checks.yaml 2>&1)
EXIT_CODE=$?

if [ $EXIT_CODE -ne 0 ]; then
    echo "$OUTPUT" | mail -s "Data Quality Alert: Checks Failed" team@example.com
fi

exit $EXIT_CODE

PagerDuty Integration

#!/bin/bash
# run-checks-with-pagerduty.sh

PAGERDUTY_KEY="your-integration-key"

if ! dbqctl check --checks checks.yaml; then
    curl -X POST https://events.pagerduty.com/v2/enqueue \
        -H 'Content-Type: application/json' \
        -d "{
            \"routing_key\": \"$PAGERDUTY_KEY\",
            \"event_action\": \"trigger\",
            \"payload\": {
                \"summary\": \"Data quality checks failed\",
                \"severity\": \"error\",
                \"source\": \"dbqctl\"
            }
        }"
fi

Exporting Reports

JSON Output

# Export results as JSON
dbqctl check --checks checks.yaml --format json > results.json

HTML Reports

# Generate HTML report
dbqctl check --checks checks.yaml --format html > report.html

Custom Reporting

import subprocess
import json

# Run checks and capture output
result = subprocess.run(
    ['dbqctl', 'check', '--checks', 'checks.yaml', '--format', 'json'],
    capture_output=True,
    text=True
)

# Parse results
results = json.loads(result.stdout)

# Generate custom report
for dataset in results['datasets']:
    print(f"\nDataset: {dataset['name']}")
    for check in dataset['checks']:
        status = "✓" if check['passed'] else "✗"
        print(f"  {status} {check['description']}")

CLI Reference

Global Options

dbqctl [command] [options]

Global Options:
  --config <path>    Path to configuration file (default: ~/.dbq.yaml)
  --format <format>  Output format: text, json, html (default: text)
  --verbose, -v      Verbose output
  --quiet, -q        Quiet mode (errors only)
  --help, -h         Show help
  --version          Show version

Commands

check

Run data quality checks from a configuration file.

dbqctl check --checks <path> [options]

Options:
  --checks <path>    Path to checks YAML file (required)
  --dataset <name>   Run checks for specific dataset only
  --fail-fast        Stop on first failure
  --parallel <n>     Run checks in parallel (default: 1)

Examples:
  # Run all checks
  dbqctl check --checks ./checks.yaml

  # Run checks for specific dataset
  dbqctl check --checks ./checks.yaml --dataset pg@[public.users]

  # Fail fast mode
  dbqctl check --checks ./checks.yaml --fail-fast

  # Run in parallel
  dbqctl check --checks ./checks.yaml --parallel 4

Exit Codes:

0 - All checks passed
1 - One or more checks failed
2 - Configuration or connection error

profile

Generate statistical profile of a dataset.

dbqctl profile -d <datasource> --dataset <name> [options]

Options:
  -d, --datasource <id>  Datasource ID from config (required)
  --dataset <name>       Schema.table to profile (required)
  --sample <n>           Sample size for value analysis (default: 100)
  --limit <n>            Limit profiling to first N rows

Examples:
  # Profile entire table
  dbqctl profile -d pg --dataset public.users

  # Profile with sampling
  dbqctl profile -d pg --dataset public.orders --sample 1000

  # Profile first 10000 rows only
  dbqctl profile -d pg --dataset public.events --limit 10000

ping

Test connection to a data source.

dbqctl ping -d <datasource>

Options:
  -d, --datasource <id>  Datasource ID to test (required)

Examples:
  # Test PostgreSQL connection
  dbqctl ping -d pg

  # Test all configured datasources
  dbqctl ping -d all

import

Import and list available datasets from a data source.

dbqctl import -d <datasource> [options]

Options:
  -d, --datasource <id>  Datasource ID (required)
  --filter <pattern>     Filter schemas/tables by pattern
  --schema <name>        Import specific schema only

Examples:
  # Import all tables
  dbqctl import -d pg

  # Import with filter
  dbqctl import -d pg --filter "public"

  # Import specific schema
  dbqctl import -d ch --schema events

version

Print version information.

dbqctl version

Output:
  dbqctl version 1.0.0
  Build: abc123
  Go version: go1.21.0

Configuration File Reference

Complete Configuration Example

# ~/.dbq.yaml or ./dbq.yaml
version: "1"

# Data source connections
datasources:
  - id: prod_pg
    type: postgresql
    configuration:
      host: postgres.example.com
      port: 5432
      username: readonly_user
      password: ${PG_PASSWORD}
      database: analytics
    datasets:
      - public.users
      - public.orders

  - id: clickhouse_events
    type: clickhouse
    configuration:
      host: clickhouse.example.com
      port: 9000
      username: default
      password: ${CH_PASSWORD}
      database: events
    datasets:
      - events.pageviews

  - id: mysql_app
    type: mysql
    configuration:
      host: mysql.example.com
      port: 3306
      username: dbq_user
      password: ${MYSQL_PASSWORD}
      database: application
    datasets:
      - application.orders

Checks Configuration Format

# checks.yaml
version: "1"

rules:
  # Dataset definition
  - dataset: <datasource_id>@[schema.table]

    # Optional: Filter rows
    where: "created_at > '2024-01-01'"

    # Check definitions
    checks:
      # Schema checks
      - schema_check:
          expect_columns:
            columns: [col1, col2]
        desc: "Description"
        on_fail: error

      # Table checks
      - row_count > 1000:
          desc: "Description"
          on_fail: error

      # Column checks
      - not_null(column_name):
          desc: "Description"
          on_fail: warn

Best Practices

Organization

project/
├── data-quality/
│   ├── checks/
│   │   ├── production/
│   │   │   ├── users.yaml
│   │   │   ├── orders.yaml
│   │   │   └── events.yaml
│   │   └── staging/
│   │       └── staging-checks.yaml
│   ├── config/
│   │   ├── dbq.yaml
│   │   └── dbq.staging.yaml
│   └── scripts/
│       ├── run-checks.sh
│       └── alert.sh

Version Control

# Store checks in git
git add data-quality/
git commit -m "Add data quality checks for users table"

# Review changes
git diff data-quality/checks/users.yaml

Testing Checks

# Test on small dataset first
dbqctl profile -d pg --dataset public.users --limit 1000

# Run single check
dbqctl check --checks test-check.yaml --fail-fast

# Validate on staging before production
dbqctl check --config staging.yaml --checks checks.yaml

Performance Optimization

# Use sampling for large tables
- dataset: pg@[public.large_table]
  sample: 100000  # Check only 100K rows
  checks:
    - row_count > 1000000
    - not_null(id)

# Filter partitioned data
- dataset: pg@[public.events]
  filter: "WHERE date = CURRENT_DATE"
  checks:
    - freshness(timestamp) < 1h

Troubleshooting

Connection Issues

Problem: connection refused

Solution:

# Check network connectivity
telnet postgres.example.com 5432

# Verify credentials
psql -h postgres.example.com -U readonly_user -d analytics

# Check SSH tunnel if applicable
ps aux | grep ssh

Timeout Errors

Problem: query timeout exceeded

Solution:

# Increase timeout in config
settings:
  timeout: 600  # 10 minutes

# Or use sampling
- dataset: pg@[public.large_table]
  sample: 100000

Check Failures

Problem: Check failing unexpectedly

Solution:

# Profile the data first
dbqctl profile -d pg --dataset public.problematic_table

# Run with verbose output
dbqctl check --checks checks.yaml --verbose

# Test with raw SQL
psql -c "SELECT COUNT(*) FROM public.problematic_table WHERE column IS NULL"

Next Steps

Check Types Reference - See all available check types
Overview - Data quality overview
Community - Get help and share ideas

How-To Guides​

Connecting to Data Sources​

PostgreSQL​

ClickHouse​

MySQL​

Using SSH Tunnels​

Creating Validation Rules​

Start with Profiling​

Write Your First Check​

Incrementally Add Checks​

Organize Checks by Dataset​

Running Checks Automatically​

CI/CD Integration​

Cron Jobs​

Apache Airflow​

Post-ETL Validation​

Alerting & Notifications​

Slack Notifications​

Email Notifications​

PagerDuty Integration​

Exporting Reports​

JSON Output​

HTML Reports​

Custom Reporting​

CLI Reference​

Global Options​

Commands​

check​

profile​

ping​

import​

version​

Configuration File Reference​

Complete Configuration Example​

Checks Configuration Format​

Best Practices​

Organization​

Version Control​

Testing Checks​

Performance Optimization​

Troubleshooting​

Connection Issues​

Timeout Errors​

Check Failures​

Next Steps​

How-To Guides

Connecting to Data Sources

PostgreSQL

ClickHouse

MySQL

Using SSH Tunnels

Creating Validation Rules

Start with Profiling

Write Your First Check

Incrementally Add Checks

Organize Checks by Dataset

Running Checks Automatically

CI/CD Integration

Cron Jobs

Apache Airflow

Post-ETL Validation

Alerting & Notifications

Slack Notifications

Email Notifications

PagerDuty Integration

Exporting Reports

JSON Output

HTML Reports

Custom Reporting

CLI Reference

Global Options

Commands

check

profile

ping

import

version

Configuration File Reference

Complete Configuration Example

Checks Configuration Format

Best Practices

Organization

Version Control

Testing Checks

Performance Optimization

Troubleshooting

Connection Issues

Timeout Errors

Check Failures

Next Steps