Nothing destroys customer trust faster than charging them twice. In payment systems, this usually happens during retries: a network timeout, a crashed server, a user who double-clicks the pay button. The solution is idempotency: making operations safe to retry without duplicate effects.
But "use idempotency keys" is easier said than done. Here are six patterns that actually work in production, with the tradeoffs each involves.
The Fundamental Problem
Consider what happens when a payment request times out:
- . Client sends payment request
- . Server receives request, starts processing
- . Server calls payment processor (Stripe, etc.)
- . Network times out before response reaches server
- . Client assumes failure, retries
- . Server processes the retry as a new request
- . Customer gets charged twice
The payment processor already processed the first charge. The second request creates a duplicate. The customer is unhappy. Your support team is unhappy. Your reputation suffers.
Pattern 1: Client-Generated Idempotency Keys
The simplest pattern: clients generate a unique key for each logical payment attempt, and servers reject duplicates.
How it works: ``` POST /payments X-Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
{ "amount": 9999, "currency": "usd", "customer": "cus_123" } ```
Server behavior: - First request with this key: process and store result - Subsequent requests with same key: return stored result
Implementation requirements: - Store idempotency key → response mapping - Check for existing key before processing - Handle race conditions (two simultaneous requests with same key)
Tradeoffs: - Clients must generate truly unique keys (UUIDs work) - Need to decide how long to keep keys (24 hours? 7 days?) - Storage requirements grow with transaction volume - Doesn't prevent issues if client generates new key on retry
Pattern 2: Request Fingerprinting
When you can't trust clients to send idempotency keys (or want defense in depth), generate them from request content.
How it works: Hash the important parts of the request to create a fingerprint: ``` fingerprint = SHA256( customer_id + amount + currency + timestamp_bucket + merchant_id ) ```
Treat requests with matching fingerprints within a time window as duplicates.
The time bucket trick: Round timestamps to a bucket (e.g., 5-minute windows). This catches retries without blocking legitimate repeat purchases.
``` bucket = floor(timestamp / 300) * 300 ```
Tradeoffs: - Can block legitimate repeat purchases (same amount, same customer) - Time window tuning is tricky (too short: misses retries; too long: blocks valid requests) - Requires careful selection of fingerprint components - Usually used alongside explicit idempotency keys, not instead of
Pattern 3: Optimistic Locking with Version Numbers
For operations that modify state (like capturing an authorized payment), version numbers prevent double-processing.
How it works: ```sql -- Each payment has a version number UPDATE payments SET status = 'captured', version = version + 1 WHERE id = 'pay_123' AND version = 3 ```
If the version changed since you read it, someone else already processed the operation. Your update affects zero rows, and you know to return the existing result.
In application code: ``` payment = fetch_payment(id) if payment.version != expected_version: return existing_result # Someone beat us to it
result = process_capture(payment) update_with_version(payment, result, payment.version + 1) ```
Tradeoffs: - Requires version column on every relevant table - Needs retry logic for legitimate concurrent modifications - Doesn't work for initial creation (use Pattern 1 or 2 for that) - Can cause failures under high concurrency (may need backoff/retry)
Pattern 4: State Machine Transitions
Model payments as state machines where invalid transitions are rejected.
Payment states: ``` pending → authorized → captured → settled ↓ cancelled ```
Each transition is only valid from specific states. Attempting to capture an already-captured payment fails.
Implementation: ```sql UPDATE payments SET status = 'captured' WHERE id = 'pay_123' AND status = 'authorized' RETURNING * ```
If the payment isn't in 'authorized' state, the update affects zero rows.
Combine with explicit transition history: ```sql INSERT INTO payment_transitions (payment_id, from_state, to_state, idempotency_key) VALUES ('pay_123', 'authorized', 'captured', 'idem_456') ON CONFLICT (idempotency_key) DO NOTHING RETURNING * ```
The transition either succeeds (first attempt) or does nothing (retry). Query the existing transition to return the consistent result.
Tradeoffs: - Requires well-defined state machines (good practice anyway) - State transitions must be atomic (database transactions) - Complex operations may span multiple states - Need to handle edge cases (what if payment stuck in intermediate state?)
Pattern 5: Distributed Locking for Multi-Step Operations
Some operations span multiple systems. For these, you need distributed locks.
Example: Creating a subscription 1. Create customer in payment processor 2. Create subscription in payment processor 3. Provision access in your system 4. Send confirmation email
If step 3 fails and the client retries, you don't want to create another subscription.
Implementation with Redis: ``` lock_key = "subscription_create:user_123:plan_456" lock = acquire_lock(lock_key, timeout=30s)
if !lock: # Another request is processing, wait and return its result wait_for_result(lock_key) return cached_result
try: result = create_subscription_flow() cache_result(lock_key, result) return result finally: release_lock(lock_key) ```
Tradeoffs: - Distributed locks add complexity - Must handle lock expiration (what if holder crashes?) - Lock contention under high load - Need Redis or similar (another system to maintain) - Timeout tuning is critical (too short: lock expires mid-operation; too long: blocked requests timeout)
Pattern 6: Event Sourcing with Deduplication
For complex payment systems, event sourcing provides natural idempotency.
How it works: Instead of storing current state, store events with unique IDs: ``` { "event_id": "evt_789", "type": "payment.captured", "payment_id": "pay_123", "amount": 9999, "timestamp": "2025-01-08T10:30:00Z" } ```
Events with duplicate IDs are rejected. Current state is derived by replaying events.
Deduplication at ingestion: ```sql INSERT INTO events (event_id, type, data) VALUES ('evt_789', 'payment.captured', '...') ON CONFLICT (event_id) DO NOTHING ```
Tradeoffs: - Architectural shift (can't just add to existing system) - Event replay for state reconstruction can be slow - Snapshots needed for performance - More complex queries (aggregate current state from events) - Excellent audit trail and debugging
Combining Patterns
Production systems typically combine multiple patterns:
E-commerce checkout: 1. Client-generated idempotency key (Pattern 1) 2. State machine for order status (Pattern 4) 3. Request fingerprinting as defense in depth (Pattern 2)
Subscription billing: 1. Distributed locks for subscription creation (Pattern 5) 2. Optimistic locking for subscription updates (Pattern 3) 3. Event sourcing for billing history (Pattern 6)
One-time payments: 1. Idempotency keys to payment processor (Pattern 1) 2. State machine for payment lifecycle (Pattern 4) 3. Optimistic locking for refunds (Pattern 3)
Implementation Checklist
Before launching any payment feature:
Client-side: - Generate idempotency keys for all payment operations - Store keys locally to use on retry - Disable submit buttons after click - Show clear pending states during processing
Server-side: - Implement at least one idempotency pattern - Store idempotency key → response mappings - Return stored response on duplicate requests - Log duplicates for monitoring
Database: - Use transactions for multi-step operations - Add version columns for mutable payment records - Index idempotency keys for fast lookup - Plan for storage cleanup (old keys)
Monitoring: - Alert on duplicate payment attempts - Track idempotency key collision rates - Monitor for double-charges escaping your systems - Audit trail for all payment state changes
Testing Idempotency
You can't just hope it works. Test these scenarios:
Retry scenarios: - Request timeout, retry with same key → same result - Network error, retry → same result - Server crash mid-processing, retry → consistent result
Race conditions: - Two simultaneous requests with same key → only one processes - Rapid succession requests → only first succeeds
Edge cases: - Idempotency key collision (different requests, same key) - Key expiration (retry after key TTL) - Mixed success/failure in multi-step operations
Chaos engineering: - Kill servers mid-transaction - Partition network between services - Delay payment processor responses - Inject duplicate webhook deliveries
The Cost of Getting It Wrong
Double-charges cost more than refunds: - Customer support time - Payment processor fees (often not refunded) - Reputation damage - Potential regulatory issues (especially with disputes)
The teams that build reliable payment systems invest in idempotency from day one. It's not an afterthought you can bolt on. It's a fundamental architectural decision.
Building a payment system that needs to be bulletproof? [Let's architect it right](/contact).