Delivery Pipeline

Beaco delivery is asynchronous, queue-driven, and designed for failure recovery.

This page explains how notifications move from accepted events to delivered (or dead-letter) outcomes.

Delivery Pipeline Overview

Lifecycle:

1. Event accepted by API.
2. Notifications created (one per recipient × channel).
3. Notifications queued to Redis-backed Celery queues.
4. Channel workers process delivery attempts.
5. Notification status updated after each attempt.

Pipeline summary:

Event accepted
  -> Notification records created
  -> Enqueued to Redis
  -> Celery worker picks task
  -> Channel adapter delivers
  -> Status persisted and observable

Notification Status Model

Notifications move through these statuses:

Event status is aggregate. Notification status is per delivery unit and should be used for operational detail.

Async Processing with Celery + Redis

Beaco uses Celery workers with Redis queues for background processing.

Key characteristics:

• API returns quickly after enqueue.
• Worker execution is decoupled from request thread.
• Channel queues isolate provider incidents.

Queue Isolation by Channel

Each channel has dedicated queue/worker routing:

• email queue -> email worker
• sms queue -> sms worker
• webhook queue -> webhook worker

Isolation benefits:

• one channel outage does not halt others
• per-channel scaling and tuning
• clearer failure blast radius

Retry Logic

Transient failures are retried automatically.

Backoff Formula

Beaco uses exponential backoff with jitter:

delay = base_delay × 2^retry_count + jitter

Where:

• base_delay is configurable per channel
• retry_count starts at 0 for first retry
• jitter adds random spread to reduce synchronized retries

Per-Channel Retry Policy Fields

Retry Decision Rules

Policies can be configured to retry on:

• timeout
• 5xx responses
• 4xx responses

Recommended defaults:

• retry on timeout: enabled
• retry on 5xx: enabled
• retry on 4xx: disabled unless explicitly needed

Enabling retries for all 4xx errors can create noisy loops for permanent failures (invalid recipient, auth errors).

Example Retry Timeline

Assume:

• base_delay_seconds=10
• max_retries=4
• jitter enabled

Possible schedule:

• attempt 1 fails -> retry in ~10s
• attempt 2 fails -> retry in ~20s
• attempt 3 fails -> retry in ~40s
• attempt 4 fails -> retry in ~80s
• final failure -> dead letter

Dead Letter Queue (DLQ)

If all retries are exhausted, notification enters dead_letter.

A DLQ entry is created for operator review and action.

DLQ Entry Contains

• original notification reference
• terminal error message
• retry/failure details

This data supports root-cause analysis and controlled recovery.

DLQ Operations

Manual retry:

POST /dead-letter/{id}/retry

Discard entry:

POST /dead-letter/{id}/discard

DLQ Statuses

Example: Retry a Dead-Letter Message

curl -X POST https://beaco.michaelogunjimi.com/api/v1/dead-letter/dlq_01j4zdh4k2x9m6f2p7r9w3s8ab/retry \
  -H "X-API-Key: YOUR_MASTER_KEY"

Example: Discard a Dead-Letter Message

curl -X POST https://beaco.michaelogunjimi.com/api/v1/dead-letter/dlq_01j4zdh4k2x9m6f2p7r9w3s8ab/discard \
  -H "X-API-Key: YOUR_MASTER_KEY"

Suppressions

Suppressions prevent sends to specific recipients.

Typical use cases:

• hard bounce
• spam complaint
• manual compliance action

Create Suppression

Endpoint:

POST /suppressions

Example:

curl -X POST https://beaco.michaelogunjimi.com/api/v1/suppressions \
  -H "Content-Type: application/json" \
  -H "X-API-Key: YOUR_MASTER_KEY" \
  -d '{
    "channel": "email",
    "recipient": "[email protected]",
    "reason": "hard_bounce"
  }'

Suppressed notifications are skipped during delivery attempts.

Suppression Reasons

Idempotency in Delivery Context

Idempotency prevents duplicate processing when producers retry submissions.

How it works:

1. Producer sends event with idempotency_key.
2. Beaco stores first successful event response.
3. Duplicate submission with same key returns original event (200 OK).
4. No duplicate notifications are created.

This is especially important under network retries and at-least-once transport conditions.

Observability and Operations

Track delivery using:

• GET /events/{id} for event-level state
• GET /notifications for delivery-level status
• Beaco dashboard at https://beaco.michaelogunjimi.com

Operational checks:

1. monitor queue depth
2. monitor retry volume
3. inspect DLQ growth
4. watch channel-specific failure rates

Failure Modes and Recommended Actions

End-to-End Example Flow

# 1) Submit idempotent event
curl -X POST https://beaco.michaelogunjimi.com/api/v1/events \
  -H "Content-Type: application/json" \
  -H "X-API-Key: YOUR_PROJECT_KEY" \
  -d '{
    "event_type":"order.shipped",
    "idempotency_key":"order-shipped-10019-v1",
    "recipients":[{"channels":["email"],"email":"[email protected]"}],
    "payload":{"order_id":"ord_10019"}
  }'

# 2) Inspect notification states
curl -X GET https://beaco.michaelogunjimi.com/api/v1/notifications \
  -H "X-API-Key: YOUR_PROJECT_KEY"

# 3) Retry dead-letter item if needed
curl -X POST https://beaco.michaelogunjimi.com/api/v1/dead-letter/DLQ_ID/retry \
  -H "X-API-Key: YOUR_MASTER_KEY"

Best Practices

1. Use idempotency keys for all producer requests.
2. Tune retry policies by channel, not globally.
3. Keep retry-on-4xx disabled unless error classes are known transient.
4. Monitor DLQ and suppression trends as first-class SLO indicators.
5. Use queue isolation to scale bottleneck channels independently.

Related Docs

• Introduction for platform architecture overview.
• Events for event submission and lifecycle.
• Channels for channel constraints and configuration.
• Templates for render-time content behavior.
• Quickstart for local setup and first event.