Cross-Service Consistency: Compensation Beats Distributed Transactions

When one operation writes to orders, inventory and finance at once, strong consistency is hard to achieve. We start by admitting that, then design mechanisms that detect and repair divergence.

Three pragmatic principles

Every step is traceable: timestamp, result and request id, so divergence can be located precisely.

Key states are eventually consistent: a failed inventory deduction retries automatically and escalates to a human queue after a threshold.

Reconciliation as a safety net: a daily cross-system comparison generates tickets automatically instead of waiting for complaints.

A real example

After a payment succeeded, the inventory call timed out. The order was marked paid while stock was never deducted. Our fix publishes the inventory change as a message, retries idempotently, and alerts past a threshold.

In six months the mechanism repaired a dozen divergences without manual intervention.

Distributed systems will always be briefly inconsistent. The goal is not zero errors, but detecting and repairing them without drama.