Skip to main content

Configure Data Pipeline

A Data Pipeline connects your data sources to destinations, routing validated events through optional transformations. Pipelines are the glue that ties together schemas, sources and warehouses into a complete data flow.

What is a Data Pipeline?​

A pipeline defines the complete data flow:

Source → [Schema Validation] → [Functions/Transforms] → Destination(s)

Each pipeline includes:

  • One source where events originate
  • Schema validation against your registry
  • Optional transformations (Functions)
  • One or more destinations for delivery
  • Monitoring and alerting for health tracking

Pipeline Components​

1. Source​

The origin of events (website, mobile app, backend service)

2. Schema Validation​

Every event is validated against JSON Schema before processing:

  • Valid events continue through pipeline
  • Invalid events go to dead-letter queue

3. Functions (Optional)​

Transform, enrich, or filter events in real-time:

  • Add derived fields (e.g., total_with_tax)
  • Enrich with external data (geo-location, user profiles)
  • Filter unwanted events (bots, test traffic)
  • Anonymize PII for compliance

4. Destination(s)​

Where validated events are delivered:

  • Data warehouses (ClickHouse, MySQL, PostgreSQL)
  • Multiple destinations supported per pipeline
  • Each with its own delivery settings

Creating Your First Pipeline​

Step 1: Navigate to Pipelines​

Go to Pipelines in DataBridge Cloud and click Create Pipeline.

Step 2: Configure Pipeline Details​

Pipeline Name

  • Example: Production Events to ClickHouse
  • Use descriptive names indicating source → destination

Description (optional)

  • Add context about this pipeline's purpose
  • Example: "Routes production web events to ClickHouse for analytics"

Step 3: Select Source​

Choose the data source you created earlier:

  • Production Website
  • Mobile App (iOS)
  • Backend API
  • etc.

The pipeline will receive all events from this source.

Step 4: Configure Transformations (Optional)​

Add Functions to transform events before delivery:

Built-in transformations:

  • PII Pseudonymization - Hash email, phone, SSN automatically
  • IP Anonymization - Remove last IP octets for GDPR
  • Geo-Enrichment - Add country, region, city from IP
  • User-Agent Parsing - Extract browser, OS, device info
  • UTM Attribution - Capture marketing campaign parameters

Custom Functions: Write JavaScript to transform events:

function transform($event) {
  // Add derived fields
  $event.total_with_tax = $event.amount * 1.08;
  $event.is_high_value = $event.amount > 1000;

  // Filter events
  if ($event.user_id && $event.user_id.startsWith('test_')) {
    return null; // Don't deliver test events
  }

  // Anonymize PII
  if ($event.email) {
    $event.email_hash = hashEmail($event.email);
    delete $event.email;
  }

  return $event;
}

See Data Transformation for detailed examples.

Step 5: Select Destination(s)​

Multi-destination setup: Events are cloned and sent to each destination independently. This allows:

  • Different warehouses for different teams
  • Real-time + batch delivery modes
  • Different transformations per destination

Step 6: Set Up Alerts (Optional)​

Configure alerts for pipeline issues:

Alert Conditions:

  • Validation error rate > 5%
  • Delivery failure rate > 1%
  • Event volume drops > 50%
  • Pipeline lag > 5 minutes

Alert Channels:

  • Email
  • Telegram
  • Slack webhook
  • Custom webhook

Step 7: Save and Activate​

Click Create Pipeline to save configuration.

The pipeline starts in Active state and begins processing events immediately.

Pipeline Status and Monitoring​

Pipeline States​

Active

  • Processing events normally
  • All components healthy
  • Delivering to destinations

Paused

  • Temporarily stopped
  • Events still accepted but buffered
  • Resume when ready

Monitoring Dashboard​

Track pipeline health in real-time:

Volume Metrics:

  • Events received per hour/day
  • Events validated (success vs failures)
  • Events delivered to each destination
  • Events filtered/sampled

Quality Metrics:

  • Validation success rate (target: >99%)
  • Schema match rate
  • Dead-letter queue size
  • Transformation errors

Performance Metrics:

  • End-to-end latency (source → warehouse)
  • Validation latency
  • Transformation processing time
  • Delivery latency per destination

Error Tracking:

  • Validation errors by schema
  • Transformation failures
  • Destination delivery failures
  • Root cause analysis

Pipeline Patterns​

Pattern 1: Single Source, Single Destination​

Simple analytics pipeline:

Production Website → Snowflake

Use case:

  • Basic event tracking
  • One team, one warehouse
  • Simplest setup

Pattern 2: Single Source, Multiple Destinations​

Multi-warehouse delivery:

Production Website
  ↓
  ├→ ClickHouse (real-time analytics and ML workflows)
  ├→ PostgreSQL (team A)
  └→ MySQL (team B)

Use case:

  • Different teams with different tools
  • Hot/cold storage separation
  • Real-time + batch processing

Pattern 3: Multiple Sources, Single Destination​

Unified data warehouse:

Website Events      ─┐
Mobile App Events   ─┤
Backend API Events  ─┤→ ClickHouse
IoT Device Events   ─┘

Use case:

  • Centralized analytics
  • Cross-platform insights
  • Unified data model

Advanced Pipeline Features​

Dead-Letter Queue (DLQ)​

Invalid events are sent to DLQ for review:

What goes to DLQ:

  • Events failing schema validation
  • Events failed transformation functions
  • Events that can't be delivered (retries exhausted)

DLQ Management:

  1. Review failed events in dashboard
  2. Identify root cause (schema mismatch, bad data, etc.)
  3. Fix source code or update schema

Best Practices​

1. Start Simple, Add Complexity Gradually​

Begin with:

Source → Validation → Destination

Then add:

  • Transformations
  • Multiple destinations
  • Advanced routing

2. Use Descriptive Naming​

Good: prod-web-events-to-clickhouse-analytics

Bad: pipeline1, test, new-pipeline

3. Test Transformations Before Production​

  1. Create test pipeline with sample data
  2. Verify transformation logic
  3. Check output in warehouse
  4. Deploy to production

4. Implement Proper Error Handling​

In transformation functions:

function transform($event) {
  try {
    // Your transformation logic
    $event.enriched_field = complexCalculation($event);
    return $event;
  } catch (error) {
    // Log error but don't drop event
    console.error('Transformation error:', error);
    $event._transform_error = error.message;
    return $event;
  }
}

5. Use Sampling for High-Volume Events​

For page_view events with millions/day:

  • Sample 10% for analytics
  • Sample 100% for critical events (purchases, signups)

Troubleshooting Pipelines​

Events Not Arriving​

Check:

  1. Pipeline is Active (not in the Paused state)
  2. Source is sending events (check source metrics)
  3. Events pass validation (check DLQ)
  4. Destination is reachable (test connection)

High Validation Error Rate​

Diagnose:

  1. Check DLQ for error details
  2. Review recent schema changes
  3. Compare event structure with schema
  4. Update schema or fix source

Solutions:

  • Update schema to accept new fields
  • Fix event generation code
  • Add transformation to normalize data

Slow Event Delivery​

Symptoms:

  • High latency from source to warehouse
  • Events delayed by minutes/hours

Solutions:

  1. Check destination performance (warehouse slow?)
  2. Reduce batch size (deliver more frequently)
  3. Scale destination warehouse
  4. Sample traffic if applicable

Transformation Failures​

Diagnose:

  1. Check transformation logs
  2. Test function with sample events
  3. Review JavaScript errors

Solutions:

  • Add try-catch error handling
  • Validate input data before transformation
  • Use defensive coding (check undefined/null)

Next Steps​

Now that you've configured a pipeline:

  1. Set Up Quality Alerts for proactive monitoring
  2. Add Transformations to enrich your data
  3. Enable Quality Monitoring for warehouse tables
  4. Monitor pipeline health and optimize performance
  5. Query your data in the warehouse!

Your end-to-end data pipeline is now live and delivering high-quality events! 🎉