📝 Note: This post was edited with AI assistance for clarity and structure. The system design, implementation decisions, and technical thinking are entirely my own.

TL;DR

• Integrated a third-party payment provider with a dual-flow architecture (server-to-server token exchange + client-side browser SDK), where webhooks are the primary state update mechanism and polling is the fallback.

• Discovered that webhook delivery order does not guarantee event order — SUCCESS webhooks arrive before PROCESSING, FAILURE before PROCESSING — silently corrupting the raw provider status table while internal state guards rejected the bad transitions downstream.

• Modelled payment states into three groups: intermediate states, final success states, and final failure states — and derived all valid transitions from group membership alone.

• Enforced transition guards at two layers: DB update procedures and Java service layer, rejecting reverse transitions and redundant same-state updates at both.

• Added operational alerting on out-of-order webhook arrivals to make the failure mode observable rather than silent.

Introduction

When integrating a third-party payment provider, most of the engineering attention goes to the obvious hard parts: authentication, request construction, error handling, and retry design. State management tends to feel like a solved problem — the provider tells you what happened, you update your records accordingly.

The assumption hidden in that sentence is that the provider tells you what happened in the order it happened. In practice, with webhook-based integrations, that assumption does not hold. Webhook delivery is not guaranteed to be ordered. Network conditions, provider-side queuing, and retry mechanisms mean that a SUCCESS notification for a payment can arrive at your server before the PROCESSING notification that logically preceded it.

This post is about the failure mode that surfaces when you trust webhook delivery order, the state modelling pattern that fixes it, and the specific enforcement approach we implemented.

Integration Architecture

The payment provider integration uses a dual-flow design:

1. Server-to-server token exchange — our backend calls the provider API to generate a short-lived token, which is passed to the client.

2. Client-side browser SDK — the client uses the token to complete the payment flow directly with the provider. At the end of this flow, the provider returns a result event to the browser.

3. Webhook delivery — simultaneously and independently of the browser event, the provider POSTs webhook notifications to our server as the payment progresses through its internal pipeline.

Webhooks are the primary mechanism for server-side state updates because they are event-driven and near real-time. Polling against the provider's status API runs as a fallback — batched, periodic, and intended to catch payments where webhooks were missed or dropped.

On the storage side, we maintain two separate tables:

• Provider status table — stores the raw payment status as reported by the provider (webhook payloads, poll responses).

• Internal payment status table — stores our own payment lifecycle state, derived from provider statuses via a sync process.

A background sync process reads from the provider status table, maps provider-specific statuses to our internal status model and applies the transition to the internal table. The internal table is the source of truth for all downstream systems.

State Model

Provider Statuses

The provider exposes a set of statuses that cover the full lifecycle of a payment:

Internal Statuses

These map to our domain model, independent of provider-specific terminology:

State Grouping: The Core Design Decision

The first step in designing the transition model was not enumerating individual allowed transitions between statuses. That approach scales poorly — with N statuses, you potentially have N² pairs to reason about, and it becomes easy to miss a case or introduce an inconsistency.

Instead, we grouped statuses by their role in the payment lifecycle:

Intermediate states — the payment is still in progress; further transitions are expected. PENDING, IN_PROGRESS, ON_HOLD

Final success states — the payment has settled successfully; no further provider-driven transitions are valid.COMPLETED, RETURNED (a return is a terminal outcome, not a reversal back to intermediate)

Final failure states — the payment has terminated unsuccessfully. FAILED, CANCELLED

From these three groups, all valid transitions follow from four rules:

And two explicit rejections:

Note that final-to-same-final is treated as a no-op, not an error. A duplicate SUCCESS webhook for an already-COMPLETED payment is expected behaviour given webhook retry semantics. Silently ignoring it is the correct response — rejecting it as an error would generate false alerts.

State Transition Diagram

What Broke Before This Model

Before the transition guard was applied uniformly to both tables, the raw provider status table was updated on every incoming webhook without validation.

The failure scenario played out as follows:

The internal state was correct because the Java service layer already had transition guards. But the provider table had been overwritten to a stale state. Any process reading the provider table directly — monitoring queries, reconciliation jobs, support tooling — would see a payment stuck in PROCESSING that had already settled.

The only way to reconstruct the actual sequence of events was to go through the state transition audit log manually and trace each webhook arrival by timestamp.

The internal guards held. The problem was that the provider table had no equivalent protection, so the corruption happened one layer above where the guards lived.

The Fix: Transition Guards at Every Layer

The fix was to apply the same group-based transition model to the provider status table, not just the internal one.

Layer 1 — DB Update Procedures

Direct writes to either status table are not permitted from application code. All updates go through stored procedures. We added transition validation inside the procedures:

-- Pseudocode representation of the guard logic
IF current_group = 'FINAL' AND new_group = 'INTERMEDIATE' THEN
    -- Reverse transition: reject silently, raise alert
    RETURN;
END IF;
 
IF current_group = 'FINAL_SUCCESS' AND new_group = 'FINAL_FAILURE' THEN
    -- Cross-terminal transition: reject, raise alert
    RETURN;
END IF;
 
IF current_group = 'FINAL_FAILURE' AND new_group = 'FINAL_SUCCESS' THEN
    -- Cross-terminal transition: reject, raise alert
    RETURN;
END IF;
 
IF current_status = new_status THEN
    -- Idempotent duplicate: reject silently, no alert
    RETURN;
END IF;
 
-- Valid transition: proceed with update
UPDATE payment_status SET status = new_status ...

The DB layer is the last line of defence. If the application layer fails to catch something, the procedure will.

Layer 2 — Java Service Layer

The service layer validates transitions before issuing any DB call, using an explicit group classification and an allowed-transition check:

public enum PaymentStatusGroup {
    INTERMEDIATE, FINAL_SUCCESS, FINAL_FAILURE
}
 
public enum InternalPaymentStatus {
    PENDING(INTERMEDIATE),
    IN_PROGRESS(INTERMEDIATE),
    ON_HOLD(INTERMEDIATE),
    COMPLETED(FINAL_SUCCESS),
    RETURNED(FINAL_SUCCESS),
    FAILED(FINAL_FAILURE),
    CANCELLED(FINAL_FAILURE);
 
    private final PaymentStatusGroup group;
 
    InternalPaymentStatus(PaymentStatusGroup group) {
        this.group = group;
    }
 
    public PaymentStatusGroup getGroup() {
        return group;
    }
}

public TransitionResult validateTransition(
        InternalPaymentStatus current,
        InternalPaymentStatus next) {
 
    PaymentStatusGroup currentGroup = current.getGroup();
    PaymentStatusGroup nextGroup = next.getGroup();
 
    // Idempotent: same status, nothing to do
    if (current == next) {
        return TransitionResult.IDEMPOTENT;
    }
 
    // Final → Intermediate: reverse transition
    if (currentGroup != INTERMEDIATE && nextGroup == INTERMEDIATE) {
        return TransitionResult.REJECTED_REVERSE;
    }
 
    // Cross-terminal: final success ↔ final failure
    if (currentGroup == FINAL_SUCCESS && nextGroup == FINAL_FAILURE
            || currentGroup == FINAL_FAILURE && nextGroup == FINAL_SUCCESS) {
        return TransitionResult.REJECTED_CROSS_TERMINAL;
    }
 
    // All other transitions are valid
    return TransitionResult.ALLOWED;
}

The service layer acts on the TransitionResult:

• ALLOWED → proceed, call DB procedure

• IDEMPOTENT → no-op, return without error

• REJECTED_REVERSE or REJECTED_CROSS_TERMINAL → reject, fire alert

Alerting

Silent rejection is operationally dangerous. If out-of-order webhooks are arriving and being silently dropped, you have no visibility into how frequently this happens, whether it is a transient network issue or a systematic provider problem, or which specific payments are affected.

We fire an alert on every rejected transition (excluding idempotent duplicates). The alert payload includes the payment ID, the current status, the rejected incoming status, the webhook timestamp, and the arrival timestamp — enough to reconstruct the ordering discrepancy without manual log archaeology.

Key Takeaways

• External payment providers do not guarantee webhook delivery order. Treat every incoming status update as potentially out-of-order.

• Model payment states by group — intermediate, final success, final failure — and derive allowed transitions from group membership. This is more maintainable than enumerating individual valid pairs, and makes it trivial to classify any new status correctly.

• Apply transition guards at every layer that stores payment state, not just the authoritative internal one. A raw provider status table with no guards will silently corrupt even when your internal table is protected.

• Idempotent transitions (final → same final) should be silent no-ops, not errors. Duplicate webhook delivery is expected behaviour, not a bug.

• Invalid transitions should fire operational alerts, not be silently dropped. Frequency and pattern of bad-order arrivals is genuinely useful signal for diagnosing provider behaviour.

• Polling as a fallback is not optional. Webhooks can be missed or dropped; polling is what recovers payments that would otherwise get stuck in intermediate states indefinitely.

📝 Note: This post was edited with AI assistance for clarity and structure. The system design, implementation decisions, and technical thinking are entirely my own.

TL;DR

Built a full internal admin platform — health dashboards, approval workflows, RBAC, query consoles, batch job management — using Spring Boot, Thymeleaf, and HTMX. Single deployable JAR. Zero React.

• Replaced a legacy administrative interface with limited access controls and no audit trail with a dedicated, properly secured application.

• Chose HTMX over React for three concrete reasons: backend team skillset, single deployable requirement, and internal dependency approval overhead. AI made HTMX + Tailwind the obvious choice — Thymeleaf templates are straightforward to prompt for, and we went from zero to working prototype faster than we could have set up a React project.

• Used hx-trigger polling for live health dashboards, partial swaps for search, and modals for Prometheus snapshots.

• Built a config approval workflow with a git-like diff view — JSON CLOB deserialized against the same Java class for validation.

• Layered custom RBAC on top of SSO using Spring Security's built-in authority model.

• Discovered (the hard way) that DOM ID collisions are HTMX's scaling gotcha — solved with namespace conventions on Thymeleaf fragments.

Introduction

When we migrated a backend service from a legacy Java 8 EJB monolith to modern Spring Boot, we inherited a problem that had nothing to do with the migration itself:

A legacy administrative interface secured by a single shared password known to the entire team.

No audit trail. No per-user accountability. No roles. Just a URL and a password in an internal wiki.

When the new service went live, we decided we weren't dragging that interface along. We'd build a dedicated internal admin platform — properly secured, decoupled from the application it managed, and extensible enough to absorb future operational needs.

What started as a simple tracking UI grew into:

• Health check dashboard — live status for dozens of backend services and their dependencies, with Prometheus/Micrometer metrics

• Audit log viewer — searchable by record ID, live lifecycle status

• Query console — prebuilt debugging queries for non-engineering teams

• Spring Batch job management — trigger, monitor, and inspect batch job runs

• Config management with approval workflows — diff views, async approval, RBAC-gated persistence

• User management — role assignment, internal auth integration, server-side integrity-validated RBAC changes

Why Not React?

Before describing what we built, it's worth being direct about the decision not to reach for React or Vue.

React was never seriously considered. Here's the actual reasoning:

Team skillset. We're a backend team. Java, Spring, SQL, distributed systems. Nobody had strong React experience, and a migration project is not the right moment to introduce a framework none of us know well.

Single deployable. A React SPA means a client-side application with its own build pipeline, deployment target, and hosting, talking to a Spring Boot backend over JSON APIs. That's two things to deploy, two things to monitor, two things to break at 2am. We wanted one JAR that runs with java -jar.

Internal dependency overhead. On internal systems, adding new libraries isn't always frictionless. A typical React project can pull in hundreds of transitive npm packages. Getting security approval for all of them — especially those with known CVEs — is a real process that takes real time. HTMX is a single JavaScript file with no transitive dependencies. That matters operationally.

No JSON translation layer. With Thymeleaf, the server renders HTML directly. With React, the server renders JSON, the client consumes it, and the client renders HTML. For an internal dashboard that displays operational data and executes actions, that translation layer adds complexity without adding value.

The question wasn't "HTMX vs React." It was: does the complexity React brings actually serve our requirements? The answer was no.

AI as a deciding factor. One thing worth calling out explicitly: AI-assisted development made HTMX + Tailwind an even clearer choice. Thymeleaf templates are plain HTML with attributes — easy to describe, easy to prompt for, and the output is immediately readable and debuggable. We went from zero to a working prototype faster than we could have bootstrapped a React project. With React, you're prompting for components, hooks, state management, and build config. With Thymeleaf + HTMX, you're prompting for HTML. The feedback loop is tighter.

Dynamic Interfaces with HTMX

HTMX's model is simple: HTML attributes trigger server requests and swap fragments into the DOM. Here's where it delivered concretely.

Health Dashboard — Polling Without JavaScript

The health dashboard shows live status for dozens of internal services. Each card displays current health and key Prometheus metrics. Clicking a card opens a modal with a full point-in-time stats snapshot.

The polling setup is four attributes:

<div id="health-grid"
     hx-get="/admin/health"
     hx-trigger="every 10s"
     hx-swap="innerHTML">
  <!-- Server renders updated cards as a Thymeleaf fragment -->
</div>

Every 10 seconds, HTMX fires a GET, the server renders the updated fragment, and only that region of the DOM is replaced. No WebSockets. No SSE setup. No JavaScript.

The Prometheus metrics are point-in-time snapshots — a live scrape of /actuator/prometheus at request time. We deliberately chose not to store time-series data. For our use case ("is this service healthy right now?"), a snapshot at the moment of viewing was enough. Storing historical time-series would have introduced infrastructure we didn't need.

The modal for drill-down stats follows the same pattern — hx-get on the card element, hx-target pointing at a modal container, server returns a rendered fragment.

Server-Driven Search

The query console — used by non-engineering teams to look up records — has a search input that filters results in place as you type:

<input type="text"
       name="query"
       hx-get="/admin/console/search"
       hx-trigger="input changed delay:400ms"
       hx-target="#results-table"
       hx-swap="innerHTML"
       placeholder="Search...">

Debounced, server-driven search. The results table updates without a page reload. Four attributes, no JavaScript event handlers.

The audit log viewer works differently — it's primarily a developer debugging tool. You enter a specific record ID and retrieve the full lifecycle event trace for that record. No live filtering; a deliberate lookup. HTMX handles the form submission and swaps the result panel in place, but the interaction model is submit-and-display rather than type-and-filter.

This pattern — debounced input triggering a partial server render — is where HTMX makes the "mostly static server-rendered site" feel genuinely interactive.

Config Management with Approval Workflows

Configuration changes — application parameters and feature flags — go through a two-step async flow:

1. A requestor submits a proposed config change via the UI

2. An approver reviews the diff and approves or rejects it at their convenience

The interesting implementation detail: we store config state as a JSON CLOB in the database, serialized from the same Java class that represents the config at runtime.

When an approver reviews a pending change, we deserialize both the current and proposed config, diff them, and render the result as a structured diff view:

  feature.rateLimitEnabled:  true
- feature.maxRequestsPerMin: 100
+ feature.maxRequestsPerMin: 250
  feature.retryOnFailure:    true

On approval, the proposed JSON is deserialized back against the config class, validated, and persisted. The same class handles serialization and deserialization — no schema drift, no separate migration scripts for config format changes.

We deliberately did not build live UI updates for approval status. When something gets approved, the requestor finds out via internal notification. The HTMX live-update complexity wasn't justified for a workflow that operates on the timescale of minutes-to-hours, not seconds.

Authentication and Authorization

Internal Auth + Custom RBAC

We integrated with the organization's internal authentication system — users log in with the same credentials they use across internal tooling. The shared password was gone on day one.

The internal auth system handled authentication cleanly. Authorization was a different story.

Its role model was too coarse-grained for what we needed. It could tell us who someone was, not what they were allowed to do inside our specific dashboard. We layered our own RBAC on top using Spring Security's built-in authority model.

Users can hold multiple roles simultaneously. Roles map to Spring Security authorities, which gives us sec:authorize in Thymeleaf templates:

<!-- Only rendered for users with the APPROVER role -->
<button sec:authorize="hasRole('APPROVER')"
        hx-post="/admin/config/approve"
        hx-target="#approval-status">
  Approve Change
</button>

And @PreAuthorize on controller methods for actual enforcement (template visibility alone is never sufficient):

@PostMapping("/config/approve")
@PreAuthorize("hasRole('APPROVER')")
public String approveConfigChange(...) {
    // ...
}

Privileged roles — approvers, admins — are pinned to specific credentials managed at the server level. RBAC changes are validated using server-side authorization and integrity checks before any role modification takes effect.

DOM ID Collisions at Scale

As the codebase grew — more fragments, more interactive regions, more reused components — a class of bugs started appearing that were time-consuming to track down: event listeners firing multiple times, interactions behaving unpredictably, state leaking between sections of the page.

The root cause was straightforward once identified: ID collisions across Thymeleaf fragments.

When multiple instances of the same fragment exist on a page — a service health card rendered for dozens of services, for example — and each fragment contains id="status-badge" or id="refresh-btn", the DOM behaves unpredictably. The browser doesn't enforce ID uniqueness; it quietly picks one and ignores the rest. HTMX targets by ID, so the wrong element gets updated.

The fix is a namespacing convention: propagate a context identifier from the calling template into every fragment, and use it as the dynamic suffix of every element ID.

<!-- Caller passes a context ID -->
<div th:replace="~{fragments/service-card :: card(serviceId=${service.id})}"></div>

<!-- Fragment namespaces all IDs using that context -->
<div th:fragment="card(serviceId)">
  <div th:id="'status-' + ${serviceId}">...</div>

  <button th:id="'refresh-' + ${serviceId}"
          th:attr="hx-get='/admin/health/' + ${serviceId},
                   hx-target='#status-' + ${serviceId}">
    Refresh
  </button>
</div>

Static prefix, dynamic suffix. Safe fragment reuse, predictable HTMX targeting.

This is the discipline that React's component model enforces by default through scoped rendering. With HTMX and Thymeleaf, you enforce it yourself — as a team convention established early, before the codebase is large enough to make retroactive fixes painful.

When This Stack Makes Sense

For an internal tool on a backend team, the calculus is clear: one deployable, native Spring Security integration, no build pipeline, and you ship faster. With AI assistance, the gap widens further — Thymeleaf templates are plain HTML that's easy to prompt for and immediately readable. By the time a React project is bootstrapped, you've already shipped v1.

The threshold where we'd reconsider:

The shared password is gone. The legacy interface is gone. And we never had to write a Redux reducer.

Key Takeaways

• Single deployable wins for internal tools. Two servers means two failure domains, two deployment pipelines, and double the on-call surface area.

• HTMX's model is hx-get + hx-trigger + hx-swap. Most interactive requirements — polling, search, modals, in-place updates — reduce to these three attributes.

• AI + HTMX is a fast prototyping stack. Thymeleaf templates are plain HTML — easy to describe, easy to prompt for, and immediately debuggable. The feedback loop is significantly tighter than with component-based frameworks.

• Template-level visibility (sec:authorize) is never sufficient. Always enforce authorization at the controller layer with @PreAuthorize. UI hiding is UX, not security.

• Store config as a serialized domain object. Deserializing proposed changes against the same Java class at approval time means the validation layer and the storage layer share a single source of truth — no schema drift.

• Establish a fragment ID namespacing convention early. DOM ID collisions are HTMX's main scaling pitfall. Dynamic suffixes from context identifiers keep fragment reuse safe and HTMX targeting predictable.

• React is the right tool for rich client-side state. It's the wrong default for every web interface.

📝 Note: This post was edited with AI assistance for clarity and structure. The system design, implementation decisions, and technical thinking are entirely my own.

TL;DR

• Designed a dual-signature model for a multi-currency real-time payments system:

  • HTTP Signatures (RFC 9421 / draft-cavage) for in-transit integrity, authentication, and non-repudiation.

  • HMAC persisted alongside transfer records for at-rest tamper detection.

• Eliminated mTLS, OAuth, and shared-secret signing after a systematic security requirements analysis.

• Built idempotent retries with a client-generated correlation ID to prevent duplicate transfers during network timeouts.

• A background integrity job re-validates persisted HMACs; mismatches halt retries and raise alerts.

• Isolated currency-specific request construction behind a factory + strategy pattern, keeping core logic currency-agnostic.

Introduction

Our payments platform previously routed interbank fund transfers over traditional correspondent banking rails. While reliable, settlement could take several hours. Integrating with a real-time payments API gave us near-instant settlement, but introduced a hard security requirement:

Every request must be authenticated, tamper-proof in transit, tamper-evident at rest, and non-repudiable.

Security Requirements

Before choosing an authentication mechanism, we defined what we actually needed:

• Sender authentication — cryptographic proof the request originated from our service.

• Message integrity (in transit) — any modification to payload or headers in flight must be detectable.

• Integrity at rest — if a persisted transfer record is modified after the fact, we must know before retrying it.

• Non-repudiation — the authorizing party cannot later deny that the request was made with those exact parameters.

• Replay protection — a captured valid request must not be replayable.

Why Not JWT, OAuth, or mTLS?

• Requires asymmetric keys for non-repudiation and timestamps/nonces with server-side validation for replay protection.

Key Eliminations

• mTLS protects the transport channel, not the message. Once a request is stored in a database, mTLS has done its job and is gone.

• HMAC requires a shared secret. Since both parties can generate valid signatures, it does not provide non-repudiation.

• JWT and OAuth operate at the token level, not the HTTP request level. They authenticate principals but do not protect individual headers, request targets, or payload integrity.

HTTP Signatures (draft-cavage-http-signatures-12, later standardized as RFC 9421) was the only mechanism that met all our requirements without additional round-trips or a separate verification service.

How HTTP Signatures Work

The specification defines a signing string — a canonical, newline-separated concatenation of selected HTTP components — which is then signed and base64-encoded into the request headers.

Here's what the signing string looks like for a typical transfer request — this exact string is what gets signed with your private key:

Signing String

(request-target): post /v1/transfers
(created): 1718300000
host: api.example.com
digest: SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
content-length: 142
x-nonce: a3f9c21b-7e84-4d12-b001-9e5c3d8f0a72

Authorization Header

Authorization: Signature
  keyId="payments-service-prod",
  algorithm="hs2019",
  headers="(request-target) (created) host digest content-length x-nonce",
  signature="Base64(Signature(signing-string))"

The header order in the signature string is a bilateral contract with the payment provider. Our agreed canonical order is:

(request-target) → (created) → host → digest → content-length → x-nonce

Component Breakdown

RFC 9421 vs draft-cavage-http-signatures-12

The mental model is nearly identical. The terminology evolved:

The investment in understanding draft-cavage-http-signatures-12 transfers directly to RFC 9421. The model remains the same; the specification simply tightened the details.

With that context established, here's how both mechanisms fit together in practice.

System Architecture

We use two distinct signing mechanisms serving two different concerns.

1. HTTP Signatures (In Transit)

The outbound request to the payment provider is signed using HTTP Signatures with an asymmetric keypair (e.g., Ed25519). The private key signs the signing string; the provider verifies with our public key. This provides non-repudiation — we cannot later deny having generated the request. The signature covers:

• (request-target)

• (created)

• digest

• x-nonce

• agreed headers

This protects the request while it traverses networks and intermediary infrastructure.

2. HMAC Stored in the Database (At Rest)

The business intent is independently signed with an HMAC (symmetric, shared only between our signing and verification services) and persisted alongside the transfer record.

{
  sourceIdentifier,
  destinationIdentifier,
  amount,
  currency,
  timestamp
}

This HMAC serves as an at-rest tamper-detection mechanism and is entirely separate from the HTTP Signature. It cannot provide non-repudiation (the verifying side also knows the key), but that’s not its job — it exists solely to detect unauthorized modification after the fact.

Request Lifecycle

Internal Transfer Initiator
     │
     │  Input: Source ID, Destination ID, Amount, Currency
     ▼
  Payments Service
     │
     ├─► Resolve payment details
     │
     ├─► Compute HMAC over
     │    {source, destination, amount, currency, timestamp}
     │
     ├─► Persist
     │    {transfer payload + HMAC}
     │
     ├─► Construct outbound request
     │    (fresh timestamp, nonce, correlation ID)
     │
     ├─► Sign outbound request via HTTP Signatures
     │
     └─► POST to payment provider
          │
          ├─ 2xx → update state
          ├─ 4xx → fail request
          └─ timeout / transient failure
                 → asynchronous retry workflow

Why Sign the Input Instead of the Outbound Request?

During retries, the outbound request is reconstructed:

• New timestamp

• New nonce

• New correlation identifiers

These changes are legitimate.

What must never change is the business intent:

Who is transferring what amount to whom.

Signing the input at receipt cryptographically locks the business intent at the moment of authorization.

Before every retry:

1. Recompute HMAC from persisted values.

2. Compare with stored HMAC.

3. If they differ, halt processing and raise an operational alert.

A modified amount or substituted beneficiary never reaches the payment provider.

Retry Design: Avoiding Double Posts

A timeout during a real-time payment operation is one of the most dangerous states in fintech. Retrying blindly risks duplicate transfers.

Idempotency

The payment provider supports idempotent requests through a client-generated correlation identifier.

The service:

1. Generates the identifier during the first attempt.

2. Persists it alongside the transfer record.

3. Reuses the same identifier for every retry.

This guarantees duplicate submissions resolve to the original transaction rather than creating additional transfers.

The HMAC verification gate serves a dual purpose:

• security validation, and

• correctness validation that retries are reconstructing the identical business intent.

Multi-Currency Request Construction

A single transfer endpoint often supports multiple currencies, but field requirements vary considerably:

• Fields mandatory for currency A may be optional or invalid for currency B.

• Corridor-specific requirements may apply only to certain regions.

• Some fields must be explicitly omitted for specific currency pairs.

A branching approach quickly becomes unmaintainable:

if (currency == USD) { ... }
else if (currency == GBP) { ... }
else if (currency == EUR) { ... }

Instead, we implemented a factory and generator pattern:

public interface PaymentRequestGenerator {
    PaymentRequest generate(TransferInput input);
}

public class UsdPaymentRequestGenerator
        implements PaymentRequestGenerator { ... }

public class GbpPaymentRequestGenerator
        implements PaymentRequestGenerator { ... }

public class EurPaymentRequestGenerator
        implements PaymentRequestGenerator { ... }

public class PaymentRequestGeneratorFactory {
    public static PaymentRequestGenerator
            forCurrency(Currency currency) {

        return switch (currency) {
            case USD -> new UsdPaymentRequestGenerator();
            case GBP -> new GbpPaymentRequestGenerator();
            case EUR -> new EurPaymentRequestGenerator();
            default ->
                throw new UnsupportedCurrencyException(currency);
        };
    }
}

The signing, retry, and tamper-detection layers operate entirely on TransferInput and remain currency-agnostic.

Adding a new payment corridor becomes a single new generator implementation with no changes to core logic.

Key Takeaways

• HTTP Signatures provide authentication, message integrity, and—when asymmetric keys are used—non-repudiation in a stateless mechanism.

• Replay protection requires timestamps and nonce validation; neither alone is sufficient.

• Sign what you receive, not what you send. Outbound requests legitimately evolve across retries; business intent must not.

• Treat in-transit signatures and at-rest integrity verification as separate concerns. They answer different threat questions.

• A factory plus generator pattern cleanly isolates currency-specific rules and keeps core logic currency-agnostic.

• Header ordering is a bilateral contract between producer and consumer. Establish it before writing code.