pedro.balbinoblog.home301server.com.br · ANONYMIZED ARCHITECTURE · pattern catalogEVENT-DRIVEN RETAIL BACKENDper-channel BFFs · transactional broker · dispatcher side-car · 18-month strangler-figfig. 1 — Container view · BFF + Broker + Dispatcherrev. 2026-05-17DATA PLANEchannel inputs · BFF mesh · broker · dispatcher · downstream servicesCONTROL PLANEsaga orchestration · schema contracts · outbox poller · idempotency keysCOMPLIANCE PLANEtransactional outbox · hash-chained replay log · RTBF tombstone (LGPD-aligned)Mobile appiOS · AndroidCustomer webNext.js · WebAdmin consoleInternal SPAPOS terminalPOS terminal · 4h offline queueBFF-mobileNode + GraphQLBFF-webNode + RESTBFF-adminNode + RESTBFF-posNode + RESTone BFF per channel — no shared client SDKBROKERtransactional outbox + saga orchestrator entry pointKafka · per-order_id partition · idempotent producerDISPATCHER side-carfan-out · retry · DLQGo · Sarama · DLQ + replay UIpublish event (sync)consumepaymentsGo · Stripe / CieloinventoryGo · Postgres 14logisticsNode · 3PL adaptersloyaltyNode · points ledgershippingGo · carrier APILegacy monolith8-yr Rails · ~38% LOC remainingconsume topic (async)strangler-figSaga orchestratorTemporal · per-order workflowSchema registryConfluent SR · Avro contractsOutbox pollerDebezium CDC · poll lag p95 ≤ 200msIdempotency storeRedis · 24h TTL keysOutbox tablePostgres 16 · transactional rowappend-only · cryptographically anchoredOrder-event replay loghash-chained · SHA-256 · LGPDappend-only · cryptographically anchoredPII tombstone workflowTemporal cron · RTBF workflowappend-only · cryptographically anchoredMEASURED · steady-state production≤350 msp99 order latencyBFF → broker ack100%payment-timeout recoveryno orphan paid orders~62%monolith strangledLOC migrated · 18 mo4 hPOS offline tolerancelocal replay queue4 + 1BFFs · shared meshchannels × backendANTI-PATTERN · no shared client SDK across channels — each BFF owns its contractstack: TypeScript · Node.js · NestJS · Go · Kafka · Postgres · Redis · Temporal · Debezium · OpenTelemetry
1×
Architecture diagram — event-driven retail backend with per-channel BFFs, transactional broker, dispatcher side-car, three-plane topology

Anonymized writeup. Client name omitted under non-compete. Architecture and stack patterns are fully discussable.

Context

Tier-1 LATAM multi-channel retailer. Four consumer surfaces: native mobile (iOS + Android), responsive customer web, admin console for store operators, physical-POS at flagship stores. Inventory spans an online warehouse + ~150 physical stores with independent stock pools. Payment side: 4 local LATAM acquirers + 1 international card network. Loyalty earns/burns across every channel.

Stack reality pre-decomposition: Rails monolith, 8+ years old, 1M+ LOC, 40+ engineers shipping into it, 30-min deploy cycle gated by manual QA, one shared Postgres with every team’s tables intermixed. Peak load: ~250 orders/min, ~4 events/order fanned out downstream.

Problem statement

Three concrete problems forcing decomposition:

  1. Team-autonomy ceiling. 40+ engineers + 1 deploy pipeline = deploy-lock contention. Mobile waits on loyalty. Payments cannot ship an acquirer fix without admin-console changes piggy-backing in. Lead-time-for-changes had crept past 5 days median (DORA 2024 elite tier is < 1 day ).
  2. Channel fragmentation. Mobile wants customer + inventory@store + loyalty-summary pre-joined per screen. Admin wants the raw entity tree + audit metadata. POS wants offline-first with eventual sync. One API surface serving all three = lowest-common-denominator design; 3 of 4 channels paid an over-fetch tax on every request.
  3. Transactional consistency under partial failure. order-placed fans out to payment-capture + inventory-reserve + loyalty-accrue + ship-label + warehouse-pick — all-or-none. An acquirer timeout at 03:00 BRT cannot leave a paid order without reserved inventory.

Constraints

  • Checkout p99 ≤ 500ms end-to-end (mobile page-load perception threshold)
  • Payment-provider outages must NOT cascade to customer-visible errors — retry + degrade gracefully
  • Single end-to-end trace across all services for finance audit replay (Central Bank card-acquirer reconciliation, BACEN Circular 3.978/2020 )
  • Zero lost events: at-least-once delivery via Outbox + idempotent consumers
  • Per-channel team ships independently — no shared deploy lock
  • Language pragma: TypeScript across BFFs (FE/BE shared types + hire-pool); Go for Broker + Dispatcher (static binary + ~5ms per-event overhead); Rails monolith decommissioned incrementally, never flag-day

Architecture

Event flow: four channel apps through per-channel BFFs into Core services; the orders service commits to the outbox table in the same transaction; the Broker polls the outbox and publishes to Kafka; the Dispatcher consumes and calls payments, inventory, loyalty and shipping with idempotency keys, dead-lettering failures to the DLQ with replay

fig. 2 — event flow · BFF → Core → outbox → Broker → Kafka → Dispatcher

Six components carry most of the weight:

  1. BFF-per-channel. Each channel team owns a thin TypeScript / NestJS BFF that aggregates Core into the exact payload its frontend needs. Mobile pre-joins customer + inventory@my-store + loyalty-summary in one round-trip; Admin emits the raw entity tree + audit metadata; POS supports offline-first via local SQLite + sync-on-reconnect. Each BFF stays < 15 KLOC, deployed independently by its channel team.
  2. Core services. Orders, payments, inventory, loyalty, shipping. Each owns its own Postgres schema with zero cross-schema FKs. Public contract: protobuf-defined gRPC (Core ↔ Core) + JSON/REST (BFF ↔ Core). Every Core service stays < 20 KLOC so one engineer holds the whole model in head.
  3. Outbox table. Every Core write commits domain rows + outbox row in the same transaction: BEGIN; UPDATE orders SET status='placed'; INSERT INTO outbox(event_id, type, payload, created_at); COMMIT;. Atomicity guarantees a successful domain write will produce an event.
  4. Broker. Go service polls each outbox at 200ms (or subscribes via Postgres logical replication where p99-latency-sensitive), publishes to Kafka with partition_key = order_id for per-order ordering. Broker is the ONLY component allowed to read outbox tables — no direct-publish from Core.
  5. Dispatcher. Stateless Go consumer. Calls downstream Core services with idempotency-key = event_id (UUIDv7). Retry: exponential backoff 100ms → 30s over 8 attempts, then route to DLQ. All retry state lives in Kafka offsets + a 24h-TTL Redis attempt counter — Dispatcher itself stores nothing.
  6. DLQ + replay UI. Ops inspects failed payloads, traces the full saga chain via trace_id, re-queues or marks-resolved. Replay preserves the original idempotency-key so downstream never double-processes. No event is ever truly lost.

Decision narrative

Four decisions worth calling out — each with explicit trade-offs.

Decision 1: BFF-per-channel over single API gateway. A single gateway (Kong , AWS API Gateway ) would centralize ops + flatten the deploy story. Rejected because the real pain wasn’t routing — it was payload shape. Every channel wanted different joins; a gateway can’t do that without custom transform logic that drifts into BFF territory anyway. Per-channel BFF makes team boundaries explicit and lets Mobile ship 12×/week while Admin ships 2×/week without coordination. Pattern grounded in Sam Newman’s BFF writeup and the Newman + O’Reilly BFF chapter, Building Microservices 2nd ed. ch.4 . Reconsider when the team has only 1 consumer surface — at that point the BFF is overhead, not abstraction.

Decision 2: Outbox over direct-publish + dual-write. Dual-write (app commits DB, then publishes to Kafka in a separate step) is race-condition central: DB commit + broker crash = silent lost event. Outbox stores the event in the same DB transaction as the domain write; Broker publishes from outbox with at-least-once delivery; downstream deduplicates on idempotency-key. Cost: median publish latency rises from ~5ms (direct) to ~120ms (poll interval 200ms + replication lag). Benefit: zero lost events across ~18 months of production traffic. Pattern documented in Chris Richardson’s Microservices Patterns, ch.3 . Reconsider the polling interval (drop to 50ms or switch to logical replication) when end-to-end fan-out latency starts trespassing on the 500ms checkout budget.

Decision 3: Saga over 2-phase commit for multi-service transactions. order-placed touches 5 services (orders + payments + inventory + loyalty + shipping). 2PC would lock all 5 during commit — availability disaster the moment any one is slow. Saga: an orchestrated sequence of local transactions with compensating actions on failure (order-cancel reverses payment-capture, restores inventory hold, claws back loyalty points). Cost: every action needs a designed compensation, which roughly doubles the design effort per workflow. Benefit: per-service availability is unlocked, and any one downstream’s 5xx only escalates the saga to compensate — never blocks the user. Grounded in Garcia-Molina + Salem’s original SAGAS paper, SIGMOD 1987 — still the load-bearing reference 39 years later. Reconsider only for hard-real-time semantic invariants (e.g. live auction bid acceptance) where eventual consistency is unacceptable.

Decision 4: Strangler-fig over big-bang rewrite. The Rails monolith was NOT rewritten in a single cutover. Instead, Fowler’s strangler-fig pattern : new code lives in Core services behind BFFs, old code stays in monolith, routes migrate per-feature behind feature-flags over ~18 months.

Cost: temporary duplication (a feature lives in both monolith + Core during cutover). Benefit: zero flag-day risk, and any individual feature migration can be reverted in < 5 min by toggling its flag.

Lessons learned

  • Idempotency keys are first-class, not a retrofit. Every event carries a UUIDv7. Every downstream persists (idempotency_key, result) in a 24h-TTL dedup table. Retry becomes boring.
  • Outbox-poller drift is the single failure mode worth alerting on. oldest_unprocessed_outbox_age > 60s per service catches Broker stalls before customers notice. Ours fired exactly twice in 18 months — both real, both pre-empted.
  • BFF-per-channel drifts toward duplication; tend it actively. Mobile + Web BFFs shared ~70% of the customer-profile query. Extracted into a versioned @company/customer-client package, re-linted every sprint. Team autonomy without active de-duplication = three copies of the same query.
  • DLQ replay UI is the saga’s invariant guarantee. No event is ever truly lost; ops can always reconstruct the chain from order-placed through the failed step. Audits become a one-query reconciliation instead of a forensic.
  • Trace-id propagation is binary — you have it everywhere or nowhere. One missing hop = a 3-hour debug session. OpenTelemetry baked into BFF middleware + gRPC interceptors + Kafka headers from day 1. Bolt-on later costs 10× more.
  • Anti-pattern to refuse on sight: a single shared client SDK across all four channels. It looks DRY for one sprint, then becomes the new shared deploy lock — exactly the thing BFF-per-channel exists to dissolve.

Stack

  • BFF layer: TypeScript 5 + Node.js 20 + NestJS 10 , gRPC clients to Core, Zod for runtime validation at every BFF↔Core boundary
  • Core services: TypeScript + Node for CRUD-heavy (orders, loyalty); Go 1.22 for perf-sensitive (payments, inventory); Postgres 14 with per-service schema + zero cross-schema FKs
  • Broker + Dispatcher: Go 1.22 + IBM/Sarama Kafka client, stateless, horizontally scaled per-topic
  • Event bus: Kafka (managed Confluent Cloud ) + partition_key = order_id for per-order ordering
  • Dedup store: Redis 7 — 24h TTL on idempotency keys + per-event attempt counter
  • Observability: OpenTelemetry → Honeycomb (trace-first), Grafana dashboards for saga_health, outbox_age_p99, dlq_depth
  • Deploy: Kubernetes (GKE) + Helm per-service, GitHub Actions matrix CI, progressive rollout via Argo Rollouts
  • Migration: strangler-fig over ~18 months, every route migration feature-flagged + reversible in < 5 min

When this pattern fits

  • Multi-channel commerce / fintech / loyalty: ≥ 3 distinct consumer surfaces with diverging payload shapes
  • ≥ 5 independent engineering teams reaching deploy-lock contention (DORA lead-time > 1 day)
  • Transactional consistency with availability priority (saga acceptable, 2PC not)
  • Financial audit-replay requirement — the event log gives you that for free

When it doesn’t

  • Single-channel SaaS. BFF-per-channel is pure tax; one well-versioned REST API is fine.
  • Team < 8 engineers per BFF. Below that, dependency churn dominates and the channel team can’t sustain a service surface without burning out.
  • Strict synchronous semantic requirement (e.g. real-time auction bid acceptance, regulated batch jobs). Saga’s eventual consistency isn’t acceptable.
  • < 100 KLOC monolith with < 10 engineers. A modular monolith with selective event seams is cheaper to operate and easier to reason about — the strangler-fig payoff doesn’t materialize until you cross the team-coordination threshold.