DataBridge Agent Guide

The DataBridge Agent is a lightweight service that runs in your infrastructure, connecting directly to your data warehouse to profile tables and run quality checks. It communicates results back to DataBridge Cloud, so your credentials and data never leave your network.

Use when: You want to keep warehouse credentials entirely within your infrastructure, or you don't want to give DataBridge access to your database (follows the principle of least privilege).

Scope

The self-hosted agent currently supports data quality at rest monitoring and checks only. Data ingestion via the agent is on the roadmap.

How the Agent Works

The agent operates on a simple poll-execute-report loop:

DataBridge Cloud                        Agent (your infrastructure)
      |                                        |
      |  <--- poll for tasks (every 10s) ----  |
      |  ---- task ------------------------->  |
      |                                        |--- connect to your DWH
      |                                        |--- execute task
      |  <--- submit results ----------------  |

1. The agent polls DataBridge Cloud for pending tasks
2. When a task arrives (profile a table or run quality checks), the agent connects to your data warehouse locally
3. Results are submitted back to DataBridge Cloud for display in the dashboard
4. If polling fails, the agent backs off and retries automatically

Configuration

The agent is configured through a config.yaml file and environment variables. Environment variables take precedence over file values.

Agent Mode Configuration

Add a connections section to define local data warehouse credentials. Each key is a connection alias that must match the alias configured on the corresponding destination in the DataBridge UI.

loglevel: info
databridge:
  agentId: "my-agent"
  cloud:
    host: "https://cloud.databridge.tech/cloud-api/v1"
    apiSecret: "your-cloud-api-secret"

connections:
  prod-clickhouse:
    type: "clickhouse"
    protocol: "http"
    hosts:
      - "clickhouse.internal:8123"
    username: "default"
    password: "secret"
    database: "analytics"

  prod-postgres:
    type: "postgresql"
    host: "pg.internal"
    port: 5432
    username: "readonly"
    password: "secret"
    database: "warehouse"
    schema: "public"
    sslmode: "require"

  prod-mysql:
    type: "mysql"
    host: "mysql.internal"
    port: 3306
    username: "readonly"
    password: "secret"
    database: "app"
    sslmode: "disable"

warning

When using agent mode, the dbconfig section is not required. The agent will skip connecting to the DataBridge database and resolve all credentials locally.

Environment Variables

All configuration values can be set via environment variables:

Environment Variable	Config Path	Description
LOG_LEVEL	loglevel	Log level: debug, info, warn, error
DBRG_AGENT_ID	databridge.agentId	Unique identifier for this agent
DBRG_CLOUD_HOST	databridge.cloud.host	DataBridge Cloud API endpoint
DBRG_CLOUD_API_SECRET	databridge.cloud.apiSecret	API secret for authenticating with Cloud

Connection Type Reference

PostgreSQL

Field	Required	Default	Description
type	Yes	-	"postgresql" or "postgres"
host	Yes	-	Hostname
port	Yes	-	Port number
username	Yes	-	Database user
password	Yes	-	Database password
database	Yes	-	Database name
schema	No	"public"	Schema to profile
sslmode	No	"disable"	SSL mode (disable, require, verify-ca, verify-full)

ClickHouse

Field	Required	Default	Description
type	Yes	-	"clickhouse"
protocol	No	"http"	Protocol: http, https, clickhouse, clickhouse-secure
hosts	Yes	-	List of host:port entries
username	Yes	-	Database user
password	Yes	-	Database password
database	Yes	-	Database name
cluster	No	-	ClickHouse cluster name
kv_params	No	{}	Additional key-value driver parameters

MySQL

Field	Required	Default	Description
type	Yes	-	"mysql"
host	Yes	-	Hostname
port	Yes	-	Port number
username	Yes	-	Database user
password	Yes	-	Database password
database	Yes	-	Database name
schema	No	-	Schema name
sslmode	No	"disable"	SSL mode

Setup

Prerequisites

• A DataBridge Cloud account with an API secret
• Network access from the agent to your data warehouse(s)
• Network access from the agent to the DataBridge Cloud API

Running the Agent

Coming Soon

Docker support is coming soon. For now, run the agent as a standalone binary.

1. Create a config.yaml in the agent's working directory (or at cmd/agent/config.yaml).

2. Set required environment variables (or include them in the config file):

export DBRG_CLOUD_API_SECRET="your-secret"

3. Start the agent:

./agent

The agent logs its configuration on startup:

level=INFO msg="Starting DataBridge Quality Agent..." cloud_host=https://cloud.databridge.tech/cloud-api/v1 agent_id=my-agent local_connections=2
level=INFO msg="Running in agent mode with local connections" count=2
level=INFO msg="Agent loop starting" poll_interval=10s

Matching Connection Aliases

When using agent mode, the connection alias in your config.yaml must match the Connection Alias field configured on the destination in the DataBridge UI.

For example, if your config has:

connections:
  prod-ch:
    type: "clickhouse"
    # ...

Then the destination in DataBridge must have its connection alias set to prod-ch.

Monitoring

The agent exposes a health and metrics endpoint on port 8095:

Endpoint	Description
GET /sys/health	Returns {"status": "ok"} when the agent is running
GET /sys/metrics	Prometheus metrics endpoint

Prometheus Metrics

Metric	Type	Description
databridge_agent_tasks_polled_total	Counter	Total tasks polled, labeled by task type
databridge_agent_tasks_completed_total	Counter	Total tasks completed, labeled by type and status (success/error)
databridge_agent_task_duration_seconds	Histogram	Time spent executing each task
databridge_agent_poll_errors_total	Counter	Network/auth errors when polling for tasks
databridge_agent_consecutive_errors	Gauge	Current consecutive poll error count (drives backoff)

Troubleshooting

Agent is not picking up tasks

• Verify DBRG_CLOUD_HOST and DBRG_CLOUD_API_SECRET are correct.
• Check that the agent can reach the Cloud API endpoint (network/firewall).
• Confirm the destination has datasets configured with profiling or checks enabled.
• Look at the agent logs for "No tasks available" (normal when idle) vs error messages.

Connection errors

• In agent mode, verify the connection alias matches exactly between the config and the DataBridge UI.
• Check that the agent can reach the data warehouse host and port.
• Verify credentials are correct by testing with a database client from the same network.
• For ClickHouse, confirm the protocol matches your server configuration (http vs https).

Agent keeps backing off

Consecutive poll errors trigger exponential backoff (10s, 20s, 40s, ... up to 5 minutes). Common causes:

• Cloud API is unreachable (network issue).
• API secret is invalid or expired.
• The DataBridge backend is down.

Check the logs for the specific error message after "Error fetching task".

Task execution fails but agent keeps running

Task execution errors are reported to the backend and do not cause the agent to back off. The agent marks the task as failed and continues polling.