System Design: Building a Multi-Channel Notification Service

A subscription payment fails. The billing service saves the failure and sends an email. Then the email provider times out.

Did the email go out?

The billing service does not know. It retries, and the customer receives two messages. Meanwhile, the in-app warning never appears because the code stopped at the email error.

This small failure explains why a notification service needs more thought than a function called sendNotification(). The hard part is keeping a clear record of what should happen, what has happened, and what remains uncertain across several systems.

Let's design a service for email, SMS, mobile push, and an in-app inbox. We will follow that failed payment from the billing transaction to the customer's screen, including the points where work can be delayed, repeated, or lost.

Start with the promise to the user

Our example is a subscription SaaS product. When a payment fails, the account's billing contact should receive an in-app warning and an email. SMS and push are available when the customer has enabled them for billing alerts. We do not automatically send every message through every channel.

The first promise is that a committed payment failure will create durable notification work. The second is that a problem with one channel will not prevent the others from making progress.

We also need some limits. Old reminders should not be sent after the payment has been resolved. A customer who disables an optional channel should not keep receiving queued messages through it. Support should be able to explain a missing notification without searching raw provider logs.

These requirements already tell us several things about the design. Sending must happen outside the billing request. Delivery needs a record per channel. Preferences need to be checked close to sending. A message needs a useful lifetime.

“Exactly once” sounds like an attractive additional promise, but we cannot casually offer it for an external email or SMS send. We can deduplicate our own records. A provider timeout can still leave us uncertain about an external side effect. We will design for that uncertainty explicitly.

Give the system a realistic workload

Assume the product creates 100,000 notification events a day. Each event produces an average of two channel deliveries. That is 200,000 deliveries a day, or about 2.3 per second on average.

The average is modest. The burst is more interesting. A billing run might create 30,000 events in ten minutes. With two deliveries per event, that means 100 deliveries per second during that period.

These are design assumptions, not measured traffic. They tell us to investigate provider quotas, worker concurrency, and backlog recovery before choosing infrastructure.

Suppose a provider accepts 40 requests per second while 100 requests per second arrive for its channel. The backlog grows by roughly 60 requests per second. Adding workers cannot fix the provider's limit. We need enough queue capacity, a delivery-delay target, and an honest policy for work that becomes too old to send.

For this example, aim to make an in-app warning available within a minute and attempt external delivery within five minutes under normal conditions. These are internal objectives. They do not guarantee when a phone comes online or when a person opens an email.

Follow one event through the service

The design has a small number of responsibilities:

Billing transaction
  -> payment state + billing outbox
  -> outbox relay
  -> notification intake
  -> notification + delivery records + dispatch outbox
  -> dispatcher
       -> email queue  -> email worker  -> provider
       -> SMS queue    -> SMS worker    -> provider
       -> push queue   -> push worker   -> provider
       -> in-app queue -> inbox worker  -> database

Provider callbacks -> callback handler -> delivery history

The notification intake decides what work exists. Channel workers decide how to perform that work through their provider. A failure in an SMS adapter should not roll back the creation of an in-app item.

These are logical boundaries. They do not require eight separately deployed services. One application can host the intake and callback endpoints, with workers deployed as separate processes. The modular monolith decision framework is relevant here: choose independent deployments when they solve a real scaling or ownership problem.

The event carries an identity and enough business context to interpret the intent:

export type PaymentFailedEvent = {
  schemaVersion: 1
  eventId: string
  tenantId: string
  type: 'billing.payment_failed'
  occurredAt: string
  expiresAt: string
  recipientId: string
  data: {
    invoiceId: string
    amountMinor: number
    currency: string
  }
}

This is an internal contract, not a runtime validator. Validate received data before using it. Authenticate the producer and authorise its tenant and event type; possession of a tenant ID in JSON is not permission to send notifications for that tenant.

Keep card details and arbitrary message bodies out of the event. Resolve permitted destinations through the recipient profile, and generate links from trusted application routes. Otherwise, a compromised producer could turn the service into a general-purpose messaging tool.

Close the gap between the database and the queue

The billing service should write the payment failure and an outbox event in the same database transaction. Either both records commit or neither does.

A relay later publishes committed outbox events. If the broker is unavailable, the outbox record remains available for another attempt. If publishing succeeds but the relay crashes before marking the record complete, it may publish the event again.

That duplicate is expected. The transactional outbox pattern solves the dual-write gap; it does not remove the need for idempotent consumers.

At intake, use a database uniqueness rule on (tenant_id, event_id). Inside one transaction, insert the notification, create the intended deliveries, and create dispatch-outbox records. If the event already exists, return the existing notification without creating another set of deliveries.

Check that a repeated event ID has the same meaningful payload. Quietly accepting the same ID for a different invoice would hide a producer bug. Store a canonical payload hash or compare the relevant persisted fields and flag mismatches.

There is a second queue boundary here. Creating delivery rows and then publishing jobs as unrelated operations would recreate the original problem. The dispatch outbox closes that gap. For a smaller system, workers could instead poll durable delivery rows directly. Both approaches can work; mixing them accidentally leaves holes.

Separate the notification from its deliveries

A notification represents one business intent for one recipient. A delivery represents an attempt to fulfil that intent through a specific channel and destination.

For our example, the important records are:

Record	What it owns
Notification	Tenant, source event, recipient, type, business data, expiry, template version
Delivery	Notification, channel, destination identity, current status, next attempt time
Attempt	Delivery, attempt number, start time, result, provider message ID, error classification
Inbox item	Recipient-visible content, notification reference, read time

A uniqueness rule on (notification_id, channel, destination_key) prevents duplicate delivery rows. For email, the destination key might represent the selected billing-contact address version. For push, it represents a device registration. One recipient may have several devices, so one push row per user is not always enough.

Keep attempt history separate from current status. If an email succeeds after two failed attempts, support should see both the success and the earlier failures.

Useful delivery states include pending, sending, accepted, delivered, retry_wait, unknown, failed, suppressed, and expired. They mean different things:

• accepted means the provider accepted the request.
• delivered means the channel supplied a delivery receipt with a defined meaning.
• unknown means a request may have succeeded, but we lack enough evidence.
• suppressed means policy or preferences prevented sending.

Do not require every channel to reach delivered. Some providers cannot prove delivery. An in-app record can be available without being read, while an email server accepting a message does not prove that a human saw it. Keep those distinctions visible in the support interface.

Make preferences and templates part of the design

At intake, select channels from the event type's policy and the recipient's settings. Store the decision so you can explain why a delivery exists or was suppressed.

Check suppression again before sending. A queued message may be several hours old, and the recipient may have disabled SMS in the meantime. If a destination has changed, do not blindly keep sending to a stale address or device token. Define whether to suppress the old delivery and create a replacement through an explicit, auditable action.

Use a policy table for event categories. Billing alerts, product announcements, and security alerts have different purposes. Avoid a universal “ignore preferences” switch. The product must define which channels are required for which event and communicate that behaviour to its users.

Store template versions with the notification. If a message retries tomorrow, an unrelated template edit should not silently change its meaning. Resolve the locale with a documented fallback and validate required variables before the delivery reaches a worker.

For example, billing.payment_failed version 3 might require an invoice reference, a formatted amount, and a trusted billing-page URL. If a required variable is missing, treat that as a permanent content error to investigate. Retrying the same broken template every minute will not repair it.

Escape variables for their destination. HTML email, plain text, and push payloads have different constraints. Keep sensitive billing details out of push previews that may appear on a locked phone. The notification can say that action is needed and direct the recipient to an authenticated page.

Retry the work without pretending duplicates are impossible

A worker receives a delivery ID and atomically claims the delivery with a lease. The lease gives it temporary ownership; it is not a guarantee that the process will remain alive.

The worker checks the current state, preferences, expiry, and relevant business status. A paid invoice should suppress a stale failed-payment reminder. Cache or fetch only the information needed for that check, and define what happens if the authoritative service is unavailable. For this reminder, delaying the send is preferable to knowingly sending an obsolete warning.

After claiming work, make the provider request outside a long database transaction. Set a request timeout shorter than the work lease, renew ownership when necessary, and use a claim token when updating the result. A late worker must not overwrite a result recorded by a newer owner. A lease can protect database state, but cannot revoke a provider call already in flight.

Classify results before deciding to retry:

Result	Response
Invalid destination or invalid template	Stop and record a permanent failure
Provider throttling	Respect the provider's retry delay and local quota
Temporary provider failure	Schedule a bounded retry with jitter
Authentication failure	Pause affected work and alert the operator
Timeout after the request may have been sent	Record uncertainty and reconcile if possible

Here is a small scheduling helper. The attempt argument counts failed attempts, starting at one. A missing result means the retry budget or useful lifetime is exhausted.

export function nextRetryAt(
  attempt: number,
  nowMs: number,
  expiresAtMs: number,
  retryAfterMs = 0,
  random: () => number = Math.random
): number | null {
  if (!Number.isInteger(attempt) || attempt < 1 || attempt >= 5) {
    return null
  }
  const ceilingMs = Math.min(60_000, 1_000 * 2 ** (attempt - 1))
  const jitterMs = Math.floor(random() * ceilingMs)
  const delayMs = Math.max(250, retryAfterMs, jitterMs)
  const nextMs = nowMs + delayMs
  return nextMs < expiresAtMs ? nextMs : null
}

Inputs must already be validated finite millisecond values; the injected random function must return a value from zero up to, but excluding, one. Parse the provider's Retry-After representation in its adapter before calling this function. The numbers are example policy, not a universal recommendation. Real delivery windows and quotas should determine the retry budget.

The helper does not solve duplicate sends. If the provider supports an idempotency key, use a stable key based on the delivery ID across retries. Check the provider's retention window and payload rules before relying on it.

Without provider support, a timeout after acceptance leaves a choice: retry and risk a duplicate, or stop and risk a missing message. For our billing reminder, first try provider lookup or await a callback within a bounded window. If neither can resolve the result, follow an explicit policy that permits a possible duplicate and records that risk.

Do not automatically switch providers on an ambiguous timeout. The second provider cannot know whether the first already sent the message. Failover is useful for known failures, but it can also create duplicate notifications.

Treat delivery reports as events

Email and SMS providers may send callbacks after accepting a request. Verify the callback's signature using the provider's documented process, then durably record it before acknowledging receipt.

Callbacks can be repeated and arrive out of order. Use a provider event identity to deduplicate them where available. Keep the raw event history with appropriate access controls and retention, then map it into your delivery model.

Do not let a late “accepted” callback overwrite an already recorded “delivered” state. Also do not assume that every provider has the same simple progression: a bounce or complaint may arrive later and describe a different concern. Define the mapping per adapter and preserve information you cannot safely reduce to one status.

For push, provider acceptance is especially easy to overinterpret. A device can be offline, and a message can expire before it reaches the device. Firebase's message lifespan documentation explains TTL and storage behaviour. Set the provider TTL to the remaining useful lifetime of the notification rather than giving an old reminder a fresh lifetime on every retry.

Keep one channel from blocking the rest

Separate channel queues and concurrency limits let email continue when SMS is throttled. They also let the in-app path remain responsive without competing with slow network calls.

Within each channel, apply fairness across tenants. A large customer's batch should not consume every available worker slot. Per-tenant limits and bounded batches can be enough at this scale; a separate queue for every tenant is not an automatic requirement.

Queue acknowledgement should follow a durable outcome. If a worker records retry_wait, the scheduler must have a reliable way to find that work again. If it crashes after a successful provider call and before recording the result, replay must enter the uncertainty handling described earlier.

At-least-once delivery is a normal queue contract. For example, Amazon SQS standard queues can deliver a message more than once. A queue does not remove your responsibility to define safe replay.

After the retry budget is exhausted, retain the failure for investigation. A dead-letter queue is useful for diagnosis, but replay must recheck expiry and preferences. Bulk replaying last week's expired reminders would turn recovery into a new incident.

Build the support view before you need it

When a customer asks why they did not receive a message, support needs a timeline: event received, channels selected, delivery suppressed or attempted, provider response, and final known status.

Use the source event ID, notification ID, delivery ID, and provider message ID to join that story. Log identifiers and error classes rather than full message bodies or destination addresses. Restrict access to sensitive details in the support view.

Measure queue age per channel, time from event creation to first attempt, provider error and throttling rates, uncertain outcomes, expired work, and dead-letter growth. Track in-app availability separately from external acceptance.

Queue length alone is misleading. A thousand messages processed in seconds may be healthy; ten messages stuck for two hours may indicate a broken partition or destination adapter. Alert on delay and failure patterns that affect the promises you made to users.

Test the failure windows

Use fake providers for deterministic failure tests, then provider sandboxes for adapter integration. The useful tests cross boundaries:

1. Commit the billing transaction while publishing is unavailable. The event must remain recoverable.
2. Deliver the same source event twice. There should be one notification and one intended delivery per destination.
3. Crash after provider acceptance but before saving the result. Confirm the idempotency or uncertainty policy.
4. Disable a channel after enqueueing. The worker should suppress the pending delivery.
5. Resolve the invoice before a retry. The stale reminder should stop.
6. Send callbacks twice and in a different order. The history should remain accurate.
7. Throttle SMS while processing in-app work. One channel should not stop the other.

Test the retry helper at the expiry boundary too. Scheduling an attempt exactly at expiry is too late under the policy above. Use an injected random value so the test does not depend on chance.

Start with a system your team can operate

For the stated workload, I would start with PostgreSQL, an outbox relay, a managed queue with separate channel work streams, and a small set of workers. I would keep templates and preferences in the notification application and build the delivery timeline early.

I would split services further only when ownership, throughput, or failure isolation required it. The first useful scaling move might be reducing provider calls, adjusting quotas, or improving tenant fairness rather than introducing another platform.

When the next email timeout happens, this design still cannot manufacture certainty the provider never supplied. What it can do is preserve the work, isolate the failure, apply a deliberate retry policy, and tell the team exactly what is known. That is the foundation of a notification service people can rely on.