Architecture, Consistency, and Failure Handling

When a customer taps "Place order", the screen usually changes within a second or two. Behind that moment sits a business process that can run for days. Stock has to be set aside, money has to be secured, a warehouse has to pick and pack the goods, and a carrier has to deliver them. Each of these activities belongs to a different system, and any one of those systems can be slow, unavailable, or briefly wrong. The software that holds this process together is the Order Management System, and its design decides whether a retailer oversells stock, charges customers twice, or ships orders that were already cancelled.

This article designs an Order Management System for a hypothetical e-commerce retailer. It is not a whiteboard exercise that sketches boxes and stops. The aim is to follow the design into the places where real systems struggle: concurrent purchases of the last unit in stock, payment requests whose outcome is unknown, messages that arrive twice or out of order, and cancellations that race against a warehouse. Along the way, the article explains why each decision was made, which consistency guarantee it needs, and what it costs.

The article builds the design in the order a reader needs it. It begins with the role of the OMS, the retailer's assumptions, and the invariants the design must protect. It then defines the order lifecycle, the architecture, and the data model, and follows a single example order from submission to delivery. From there, it goes deep into inventory consistency, distributed workflow coordination, and failure handling, which are the places where the design is tested hardest. It closes with scalability and observability, and then with a summary of the architectural decisions and the trade-offs behind them.

Understanding the Order Management System

What is an Order Management System?

An Order Management System (OMS) manages and coordinates the lifecycle of customer orders. The lifecycle begins when an order is created and confirmed, and it continues through fulfillment until the order is delivered or cancelled. Throughout that journey, the OMS keeps the authoritative record of the order, meaning the single record that every other system treats as the truth about what the customer bought and where the order stands.

The OMS does not do this work alone. In a typical e-commerce platform, several independent systems take part in processing a single order:

• Catalog and Pricing System: provides product information and current prices.
• Inventory System: maintains available stock and the reservations held against it.
• Payment System: manages payment authorization, capture, voids, and refunds.
• Fulfillment System: coordinates warehouse operations such as picking and packing.
• Shipping System: creates shipments, integrates with carriers, and tracks parcels.
• Notification System: keeps customers informed as the order progresses.

It helps to separate order placement from order management, because the two are often blurred together. Order placement is the customer's statement of intent to buy specific products. Order management is the coordination of all the business activities needed to turn that intent into a delivered parcel.

An OMS is therefore much more than a CRUD service, which is a service that only creates, reads, updates, and deletes records. At the same time, the OMS does not need to own inventory, payment transactions, or warehouse operations. Its job is to coordinate the systems that do own them, while keeping the order itself accurate and consistent.

Business context

A design is only meaningful against a concrete business. We will design the OMS for a hypothetical e-commerce retailer that sells physical goods, and we will hold the following assumptions constant for the rest of the article:

• Customers place orders through web and mobile applications.
• An order can contain multiple products.
• Inventory is held across multiple warehouses.
• In the initial design, all items in an order are allocated to one warehouse.
• Customers pay online through an external payment provider.
• For supported payment methods, payment authorization and payment capture are separate operations. Authorization asks the provider to set the money aside on the customer's account. Capture is the later step that actually collects that money.
• Orders can be cancelled before a defined fulfillment cutoff, which is the point after which warehouse work can no longer be stopped. The Fulfillment Service decides when an order crosses the cutoff.
• Order processing involves both synchronous operations, where the caller waits for the answer, and asynchronous operations, where the answer arrives later.
• The system must handle concurrent orders, retried requests, and failures in downstream services.
• The initial deployment serves a single geographic operating region.

Scope boundaries

An OMS sits close to many other capabilities, so it is worth stating clearly what this design covers. The article treats the following areas in depth: order creation and validation, order lifecycle management, inventory reservation, coordination of payment authorization and capture, order confirmation, fulfillment initiation and tracking, order cancellation, reliable event processing, recovery and reconciliation, and finally scalability and observability.

Several neighbouring topics are deliberately left out of detailed treatment. These are product search and recommendations, dynamic pricing algorithms, the internals of fraud detection, complex tax calculation, warehouse picking optimisation, split shipments across warehouses, returns and exchanges, and multi-region active-active deployment. Returns will appear briefly where they help explain why some actions cannot simply be undone. Refunds are a different matter. A refund sends captured money back to the customer, and it is covered wherever a cancellation follows payment capture.

With the role of the OMS and the retailer's context in place, the next step is to turn that context into precise requirements and, more importantly, into the rules the system must never break.

Requirements and Design Considerations

Functional requirements

Functional requirements describe what the system must do. At its core, the OMS must create orders containing one or more products. It validates product identifiers and quantities, establishes authoritative prices and totals, and persists each order under a unique identifier, without creating a duplicate when a client retries the same submission. It must also manage the order lifecycle by maintaining the current status, enforcing valid transitions, recording significant lifecycle events, and supporting retrieval of individual orders and order history.

Around that core, the OMS coordinates work owned by other systems. On the inventory side, it identifies a warehouse that can fulfill the order, reserves stock before confirmation, releases stock when an order is rejected or cancelled, prevents overselling during concurrent reservations, and manages reservation expiry. On the payment side, it requests authorization and tracks the outcome, captures payment according to business policy, voids authorizations or refunds captured payments when appropriate, and handles unsuccessful and uncertain outcomes.

On the fulfillment side, it initiates fulfillment for confirmed orders, tracks warehouse acceptance and progress, receives shipment and delivery updates, and prevents fulfillment from continuing after a cancellation has taken effect.

Cancellation and customer queries complete the picture. The OMS decides whether an order is still eligible for cancellation, coordinates the cancellation with fulfillment, releases inventory where applicable, voids the payment authorization or refunds a captured payment as the situation requires, and preserves the cancellation history. Customers, in turn, must be able to see their order details and history, a customer-facing status, and shipment tracking once it is available.

Non-functional requirements

Non-functional requirements describe how well the system must behave. They often shape the architecture more than the functional requirements do, because they decide where the system needs durability, isolation, or room to grow.

For this OMS, the most important qualities follow directly from what an order represents. Order creation and order retrieval must remain available when parts of the platform fail, and critical business invariants must remain consistent. Committed orders and workflow decisions must be durable enough to survive crashes, and interrupted workflows must be recoverable. An order submission should receive a timely acknowledgement, even when processing continues afterwards. Processing workloads and query workloads must be able to scale independently.

Operational qualities matter just as much. Operators must be able to observe business progress as well as technical failures, and significant state transitions must be preserved so that they can be audited later. The design should make it easy to add payment methods, warehouses, and sales channels. It must also be secure, enforcing authorization on every request and protecting sensitive customer information.

Business invariants

Requirements tell us what to build. Invariants tell us what must never go wrong. An invariant is a condition that must remain true regardless of concurrency, retries, or failures. Invariants are the most useful tool in this design, because every later decision about locking, transactions, and messaging can be traced back to one of them. The invariants below are the ones this OMS protects.

Invariant 1: Inventory cannot be over-reserved. A SKU, or stock-keeping unit, is the identifier for one specific sellable item, such as a particular laptop model in a particular configuration. For every SKU in every warehouse, the quantity reserved for orders must never exceed the stock that is eligible to be sold.

ReservedQuantity <= EligibleStockQuantity

Invariant 2: Duplicate submissions must not create duplicate orders. Clients retry requests when a network call times out, and customers sometimes press a button twice. An idempotency key is a unique value the client attaches to a request so that the server can recognise repeats of the same logical request. Within its scope, such as a single customer, the same idempotency key must never produce more than one order.

Invariant 3: An order cannot be confirmed unless its prerequisites are satisfied. In this design, an order may move to CONFIRMED only when its inventory reservation is RESERVED and its payment authorization is AUTHORIZED. Crucially, both conditions must still be valid at the moment of confirmation. A reservation that expired a minute earlier does not count, even if it was valid when the workflow last checked.

CONFIRMED is allowed only when: Inventory Reservation = RESERVED AND Payment Authorization = AUTHORIZED

Invariant 4: Payment operations must not be duplicated. Retrying the same logical authorization, capture, or refund must never create an additional successful payment transaction. A customer should never be charged twice, or refunded twice, because a network call was repeated.

Invariant 5: Fulfillment cannot proceed after cancellation takes effect. Cancellation and fulfillment must coordinate through explicit authorization or cutoff rules, so that a warehouse never dispatches an order the customer has successfully cancelled.

Invariant 6: Historical order values must remain stable. Later changes to a product's price or description must not alter the commercial details of orders that have already been placed.

Technical nuance: Not every operation needs strong consistency across the whole platform. Reserving inventory needs atomic concurrency control. Updating a customer-facing shipment-tracking view can usually tolerate a short, bounded delay. Sending an order-confirmation email can happen asynchronously.

A sound architecture applies strong coordination where business correctness requires it, instead of imposing global consistency on every operation.

Several of these invariants talk about an order "being confirmed" or "being cancelled". Before we can protect them, we need a precise definition of the states an order can be in and the rules for moving between them. That is the job of the order lifecycle.

Order Lifecycle and State Management

A simplified order state machine

A state machine is a model that lists every state something can be in and the specific events that are allowed to move it from one state to another. Modelling an order as a state machine makes illegal moves, such as shipping an order that was never confirmed, impossible by construction. The diagram below shows the complete lifecycle used in this article, including the happy path, the rejection path, and the cancellation path.

Notice that cancellation passes through CANCELING instead of jumping straight to CANCELED, and that the dashed return path exists because the system must never report a cancellation it cannot guarantee.

On the normal path, an order starts as PENDING. It becomes CONFIRMED once inventory is reserved and payment is authorized. It moves to FULFILLING when the warehouse accepts it, to SHIPPED when payment has been captured and the parcel has left the warehouse, and finally to DELIVERED.

An order that cannot be fulfilled takes the rejection path. If stock is unavailable or the payment provider declines the authorization, the order moves from PENDING to REJECTED. That move happens only after any required cleanup has completed, such as releasing stock that was already reserved.

Cancellation is the most delicate path. A customer can request cancellation while the order is PENDING, CONFIRMED, or FULFILLING, provided the fulfillment cutoff has not passed. Once eligibility is verified, the order enters an intermediate CANCELING state. The order reaches CANCELED only after every cleanup step that applies to it has been confirmed. If the OMS has already sent the order to fulfillment, the fulfillment service must confirm that it has stopped work. If stock was reserved, the reservation must be released.

If payment was authorized but not yet captured, the authorization must be voided. If payment has already been captured, the captured amount must be refunded. A PENDING order that never reached the warehouse therefore needs only the inventory and payment cleanup. If cancellation cannot be guaranteed, the system must not report CANCELED. It keeps or restores the appropriate active state and tells the customer what actually happened.

In this simplified design, DELIVERED, CANCELED, and REJECTED are terminal states, which means no further transitions leave them. A more comprehensive OMS would add workflows for returns, exchanges, and post-delivery disputes, and those workflows would extend the lifecycle beyond DELIVERED.

Why one status field is insufficient

It is tempting to describe an order with a single status column. The trouble is that an order has several independent business dimensions, and each dimension changes on its own schedule. Payment can be authorized while stock is still reserved and the warehouse has not yet started. Later, payment can be captured while the warehouse is still packing. The diagram below shows two perfectly valid snapshots of the same order.

Notice that each row changes independently, so a single combined status would need a separate value for every possible combination.

A single enumeration that tries to name every combination quickly grows into values like "CONFIRMED_AUTHORIZED_NOT_PICKED", and it becomes hard to maintain and easy to get wrong. The recommended approach is to keep separate state machines for the order lifecycle, the payment lifecycle, the inventory reservation lifecycle, and the fulfillment lifecycle. Each service owns the authoritative state for its own lifecycle. The OMS holds references to those records and maintains a coordinated view that it uses to make order-level decisions.

State transitions and concurrency

A state machine defines which transitions are legal. It does not, by itself, stop two transitions from happening at the same time. Consider a customer who requests cancellation at the same moment that the fulfillment service starts processing the order. Both operations may read the order as CONFIRMED, and without some form of concurrency control, both may go ahead.

One common mechanism is optimistic concurrency control. Each order row carries a version number. A writer reads the row, remembers the version it saw, and then updates the row only if the version is still the same. If another writer got there first, the version will have changed and the update will affect zero rows.

UPDATE Orders SET Status = 'CANCELING', Version = Version + 1 WHERE OrderId = @OrderId AND Status = 'CONFIRMED' AND Version = @ExpectedVersion;

Only the writer whose expected version matches succeeds, so the OMS can never record two conflicting transitions for the same order. However, this protection stops at the edge of the OMS database. Updating the OMS record alone does not prevent a separate fulfillment service from acting on stale information.

The warehouse may already have read CONFIRMED and started picking. To close that gap, the fulfillment service must take part in a cancellation-and-dispatch coordination protocol built around the fulfillment cutoff. We return to that protocol in the section on distributed transactions and workflow coordination.

Technical nuance: An order state machine enforces valid transitions within its own consistency boundary, which is the set of data that can be updated together in one transaction. It does not automatically coordinate concurrent operations across independent services.

The lifecycle tells us what an order can do. The next question is which components carry out that lifecycle, who owns which data, and how those components talk to each other.

High-Level Architecture

Core components

The architecture is easiest to understand as a map first and a set of responsibilities second. The diagram below shows the logical components and how requests and messages flow between them. The rest of this section then explains what each component owns.

Notice that the OMS keeps its order data, workflow state, and outbox in one database, while every participating service keeps its own store and the broker connects them for asynchronous work.

The message broker is drawn separately for readability. In practice, the OMS and the participating services all use the broker wherever their interactions are asynchronous. The arrows show logical relationships, not a single transport for every call. The outbox shown in the OMS database is a table of messages waiting to be published. The reliability section explains why it shares a database with the order data.

The API / Checkout layer is the front door. It accepts customer requests, authenticates the caller, checks that the caller is allowed to perform the action, and validates the shape of each request. Working with the OMS, it makes order submission idempotent, so that a retried submission never creates a second order, and it returns the order identifier and status to the client.

The Order Management Service, which the rest of this article calls the Order Service for short, owns the order records. It maintains the order lifecycle, enforces order-level business rules, and exposes the APIs for creating, retrieving, and cancelling orders. In the diagram, this responsibility appears as the Order Lifecycle Manager inside the OMS.

The Order Workflow Coordinator drives the multi-step processing of each order. It keeps a durable record of how far each workflow has progressed, issues commands to the participating services, and handles retries, timeouts, and compensating actions, which undo the effect of earlier steps when something fails. It does not need to be a separate deployment. At first it can live as a module inside the OMS, as the diagram shows.

The Inventory Service owns stock availability. It decides which warehouse should serve an order, creates and releases reservations, and enforces the concurrency rules that stop stock from being oversold.

The Payment Service integrates with the external payment provider, often called the payment gateway. It manages authorization, capture, void, and refund operations, stores every payment attempt and its outcome, and handles provider callbacks and reconciliation.

The Fulfillment Service coordinates warehouse operations, usually through a Warehouse Management System (WMS), which is the software that runs day-to-day activity inside the warehouse. The Fulfillment Service tracks picking, packing, and dispatch, enforces the rules that decide when cancellation is still possible, and publishes progress updates.

The Message Broker provides durable asynchronous communication. It stores messages until consumers process them, carries events and, where appropriate, commands, and allows each service to process work on its own schedule. Finally, the Notification Service sends confirmation and shipment messages to customers and retries failed deliveries independently, so a slow email provider never holds up the core order workflow.

The Catalog and Pricing System and the Shipping System from the opening list do not appear as separate boxes, because the OMS never coordinates them directly. The checkout layer calls the catalog to establish authoritative prices when an order is submitted. The Fulfillment Service handles carrier integration and shipment tracking on behalf of the order.

Communication strategy

Not every interaction deserves the same style of communication. A synchronous call keeps the caller waiting until the answer arrives, which suits short operations where the caller needs the result immediately.

An asynchronous interaction hands the work off and delivers the result later, which suits long-running or non-critical work. The table below shows how this design chooses between them. It also names the kind of message involved. A command asks one specific component to do something, while an event announces something that has already happened. The next subsection explains why that difference matters.

Commands versus events

Messages in this system are either commands or events, and confusing the two is a common source of design bugs. A command expresses an intention for one specific component to perform an action. ReserveInventory, AuthorizePayment, InitiateFulfillment, and CancelFulfillment are commands. Because a command is a request, the receiving component may reject it.

An event reports something that has already happened. InventoryReserved, PaymentAuthorized, OrderConfirmed, FulfillmentAccepted, and OrderShipped are events. An event should represent an established fact published by the component that owns that fact, so consumers cannot reject it. They can only react to it.

The practical rule is simple. Do not use events as disguised commands. If the OMS publishes an event called "OrderReadyForPacking" and quietly expects the warehouse to act on it, nobody clearly owns the outcome, and nobody is responsible when the warehouse ignores it. When an action is required, send a command to the owner, and let the owner publish an event once the action is done.

Modular monolith versus microservices

The architecture diagram shows separate services, but it does not require separate deployments. A modular monolith is a single deployable application whose code is divided into well-separated modules with clear interfaces. An early version of this platform could run as one e-commerce backend containing an order module, an inventory module, a payment module, and a fulfillment module, all deployed together.

Notice that the module boundaries are identical in both panels; only the deployment boundary changes.

A modular monolith keeps operations simple. There is one deployment pipeline, and modules can share a database transaction when they genuinely need to. As the platform grows, there are good reasons to split some modules into independently owned services.

Different modules may need to scale at very different rates. Teams may need to release changes without coordinating with each other. A module may belong to a separate team or organisation. External integrations, such as payment gateways, may bring enough complexity to deserve isolation. A failure in one area may need to be contained so that it cannot take down the rest of the platform.

Microservices are not automatically better. They bring network failures, distributed data, and harder debugging, and every one of those costs is real. The decision should follow the pressures above, not fashion.

Technical nuance: Logical service boundaries and physical deployment boundaries are separate architectural decisions. Defining them independently makes an OMS much easier to evolve.

Components and boundaries only become concrete when we decide what data each component stores. The next section turns the ownership lines in the architecture into a data model.

Data Modeling and Storage Design

Core entities

An entity is a distinct thing the system stores and tracks, usually as a database table. The OMS is built around the Order, OrderItem, and OrderWorkflow entities. The Order entity holds the header-level facts about a purchase, meaning the facts that apply to the order as a whole instead of to any single item. It records a unique OrderId, a CustomerId that is left empty for guest orders, the current lifecycle Status, the Currency, and the TotalAmount. The total is stored in minor currency units, such as paise for rupees, to avoid rounding errors from decimal arithmetic. The Order also carries a Version number for optimistic concurrency, along with CreatedAt and UpdatedAt timestamps.

The OrderItem entity holds one line of the order. Each line records what was bought: the parent OrderId, a ProductId that refers to the catalog, the SKU of the specific inventory item, and the Quantity. It also records the exact commercial terms that applied at the moment of purchase, which are the UnitPrice, the product description, the DiscountAmount, the TaxAmount, and the resulting LineTotal. These purchase-time values become important when we look at order-time snapshots.

The OrderWorkflow entity is what makes long-running processing recoverable. It records how far each order's workflow has progressed, so that a restarted worker can pick up exactly where the previous one stopped. Each workflow instance has a WorkflowId and belongs to an OrderId. Its CurrentStep names the processing step reached so far, and its WorkflowStatus is Running, Waiting, Compensating, or Completed. A RetryCount and a NextAttemptAt time tell the system how many retries the current step has used and when to try again. An UpdatedAt timestamp records the last change.

Several supporting entities complete the model. OrderStatusHistory records every significant transition for auditing. OrderPaymentReference, OrderReservationReference, and OrderFulfillmentReference link the order to the records held by the other services. OutboxMessage stores events waiting to be published, and IdempotencyRecord remembers submissions so that retries can be recognised. The outbox and idempotency records are explained in detail in the reliability section. The reference entities hold identifiers and the latest known status, but they should not duplicate the authoritative transaction history owned by the inventory, payment, and fulfillment services.

Data relationships

Notice that only OrderItem requires at least one row; every other relationship allows zero or many, which leaves room for retries and repeated attempts.

Every order has at least one item, and every other relationship allows zero or many records. A simple order may end up with exactly one payment and one reservation. In the real world, however, a card may be declined and retried with another card, or a reservation may expire and need to be recreated. Modelling these references as zero-or-many from the start avoids building in the assumption that every attempt succeeds the first time.

Order-time snapshots

Invariant 6 says that historical order values must not change. Suppose a customer buys a laptop for ₹50,000, and a week later the catalog price rises to ₹53,000. The customer's order, invoice, and any future refund must still be based on ₹50,000. If the order simply pointed at the catalog's current price, the order would silently change every time the catalog did.

The solution is a snapshot, which is a copy of the relevant values taken at the moment of purchase. The OMS keeps the ProductId as a reference, but it also stores the purchased unit price, the purchased description, the quantity, the applied discount, and the tax amount on the OrderItem. The catalog remains free to change its current price and description without touching any existing order.

Notice that the catalog price moves after purchase while the locked snapshot keeps the ₹50,000 the customer actually agreed to pay.

Data ownership

Each service is the single source of truth for its own business facts. The Order Service owns orders, order items, and order history. The Inventory Service owns stock levels, reservations, and warehouse allocations. The Payment Service owns payment attempts, authorizations, captures, and refunds. The Fulfillment Service owns fulfillment jobs and shipment progress.

The OMS may keep a denormalized order summary, meaning a read-friendly copy that combines data from several sources, so that customer queries do not need to call four services. That summary is a convenience view. When there is a disagreement, the owning service is right, and the summary must be corrected.

Database choice

A relational database is a reasonable default for core order data. The order domain benefits from transactional updates, referential integrity between orders and their items, consistency constraints such as uniqueness, flexible queries over order history, and reliable conditional updates for status transitions.

A relational database provides all of these directly. Other workloads around the OMS, such as search, analytics, event processing, and very high-contention inventory counters, may justify different storage technologies. Those choices belong to the workloads that need them, not to the core order record.

Essential APIs

The OMS exposes a small set of illustrative APIs. Clients create an order, fetch a single order, list a customer's orders, and request cancellation.

POST /orders GET /orders/{orderId} GET /customers/{customerId}/orders POST /orders/{orderId}/cancel

An order submission carries the idempotency key in a request header, together with the items, a reference to a saved payment method, and a shipping address identifier.

POST /orders Idempotency-Key: request-12345 Content-Type: application/json { "items": [ { "productId": "P101", "quantity": 1 }, { "productId": "P205", "quantity": 2 } ], "paymentMethodReference": "PM456", "shippingAddressId": "ADDR789" }

The request contains no prices. That is deliberate. The checkout process must establish authoritative prices and totals, and the OMS must never trust prices supplied by the client, because anything sent from a browser or mobile app can be altered. The response returns the new order identifier with a status of PENDING.

{ "orderId": "ORD10001", "status": "PENDING" }

Returning a pending order allows the API to respond quickly while downstream processing continues in the background. The usual protections still apply around these APIs: client authentication, authorization checks so that customers can see only their own orders, request validation, and careful handling of payment references.

Technical nuance: A successfully accepted order request is not necessarily a confirmed order. An API response and a business state transition represent different guarantees.

We now have components, data, and APIs. The best way to see them work together is to follow a single order from the moment it is submitted to the moment it is delivered.

End-to-End Order Processing

Example scenario

We will follow one order for the rest of the article. The customer submits the request shown earlier: one laptop (product P101) and two wireless mice (product P205). The OMS assigns the order the identifier ORD10001. The retailer holds stock in several warehouses, and, in line with our assumptions, the Inventory Service picks a single warehouse that can supply the whole order.

The order moves through a fixed sequence. The OMS validates the request and creates a pending order. It then reserves inventory, authorizes payment, and confirms the order. After confirmation, it initiates fulfillment, captures payment before dispatch, and finally tracks the shipment until delivery.

The order processing sequence

Notice that the OMS sits at the centre of every exchange, and that payment is captured only after the warehouse reports the order ready, not at confirmation.

The sequence diagram shows the logical order of steps, not how each message travels. Individual requests and results may travel as API calls, durable commands, or events, depending on the interaction choices made in the communication strategy. To keep the picture readable, the diagram leaves out the consumption of the reservation, the individual warehouse progress events, and some intermediate states. The sections that follow fill in each step.

Order submission

The customer submits the order through the checkout API. Before anything is stored, the system authenticates the customer or validates the guest session, checks that the products exist and the quantities are sensible, calculates authoritative prices and totals, validates the shipping address, and checks the idempotency key to see whether this submission has been seen before.

The OMS then creates order ORD10001 in the PENDING state. The same database transaction also records whatever instruction is needed to start downstream processing reliably. For example, the OMS can write an outbox record, which is a row describing a message that still has to be sent, in the same commit as the order. The reliability section explains this pattern in full.

Persisting the pending order first is a deliberate choice. The stored order is a durable record of what the customer submitted. Its identifier lets every later step be tracked and correlated. It gives the system a way to recognise retries of the same submission. It also provides a starting point for recovery if a downstream service fails. Without a durable pending record, the system can lose track of a checkout whose external side effects, such as a stock reservation, have already happened.

Inventory reservation

The OMS next asks the Inventory Service to reserve stock for ORD10001. The Inventory Service selects a warehouse that can supply the laptop and both mice, checks availability for every item, attempts to reserve the requested quantities, persists the reservation, and returns a reservation identifier along with the outcome.

A successful reservation does not mean the goods have been picked off a shelf. It is an allocation commitment: a promise that this stock is set aside for this order and will not be sold to anyone else. Because this design follows an all-or-nothing policy, the reservation must cover every item in the order. If the Inventory Service cannot reserve all items atomically, it must track any partial successes and release them safely when the full reservation fails. The inventory section shows how.

Payment authorization

Once stock is reserved, the OMS requests payment authorization. Reserving inventory first means the customer's funds are not held for an order whose stock is already known to be unavailable. Funds can still be held for an order that later fails, for example when a reservation expires before a slow authorization completes, and the reservation expiry section explains how that case is resolved.

The Payment Service creates a payment attempt, or finds an existing one if this is a retry. It sends the authorization request to the provider, persists the provider's response or the fact that the response is still pending, and then returns or publishes the outcome.

The authorization comes back as AUTHORIZED, DECLINED, PENDING, or UNKNOWN. AUTHORIZED means the funds are held. DECLINED means the provider explicitly refused. PENDING means the provider will decide later, as can happen with some payment methods. UNKNOWN usually means the request timed out, and a timeout does not mean the provider rejected the payment, because the provider may have approved it and only the response was lost. The Payment Service must never respond to an UNKNOWN outcome by blindly sending a second authorization. The reliability section explains how it establishes what actually happened.

Order confirmation

When both prerequisites are in place, the OMS confirms the order, exactly as Invariant 3 requires. The inventory reservation must be RESERVED, the payment authorization must be AUTHORIZED, and both must still be valid at the moment of confirmation. To make sure the reservation cannot expire underneath the order, the Workflow Coordinator first asks the Inventory Service to secure the reservation, a step explained in the reservation expiry section. The OMS then moves ORD10001 to CONFIRMED and, in the same transaction, records an OrderConfirmed event in the outbox for reliable publication. From this point, the order is accepted for fulfillment under the retailer's business rules.

Fulfillment initiation

With the order confirmed, the OMS sends an InitiateFulfillment command. The Fulfillment Service accepts the order, creates its own fulfillment record, coordinates the warehouse work, and reports progress back as events. The fulfillment lifecycle has more detail than the order lifecycle, as the diagram below shows.

Notice that several detailed warehouse states collapse into a single FULFILLING state for the customer, so the customer view stays simple while the warehouse keeps full detail.

The OMS does not need to mirror every warehouse state. It can map the detailed fulfillment states onto the simpler customer-facing order states, which keeps the order lifecycle stable even when the warehouse adds new internal steps.

Payment capture and dispatch

This design assumes that payment must be captured before the order leaves the warehouse. When the Fulfillment Service reports that ORD10001 is ready for dispatch, the Workflow Coordinator asks the Payment Service to capture the authorized amount. The Payment Service confirms the outcome, and only then is dispatch permitted. The Fulfillment Service dispatches the parcel and records the shipment.

If the capture fails, dispatch stays blocked while the workflow decides whether to retry, reconcile an uncertain result, or escalate to a person. In practice, the right moment for capture depends on what the payment method supports, how long the provider keeps an authorization valid, and the retailer's own policies. An authorization that expires while the order is still in the warehouse is a real possibility, and the workflow must have an explicit rule for it, such as re-authorizing or escalating.

Capturing payment does not close the door on cancellation. The dispatch permission described above is a payment precondition, not the fulfillment cutoff. The cutoff is a separate decision owned by the Fulfillment Service, and in this design it can fall after capture, for example while a packed parcel can still be pulled back from the dispatch area.

A customer may therefore cancel after payment has been captured but before the cutoff. In that case, voiding the authorization is no longer possible, because the money has already been collected. The cleanup must issue a refund instead, which is a payment operation that sends captured money back to the customer. A refund is not the same as a return. A return is the separate post-delivery workflow in which goods come back to the retailer, and it is outside the scope of this article.

Shipment tracking and delivery

After dispatch, the Fulfillment Service receives updates from the warehouse and shipping integrations and passes them on as events. A ShipmentDispatched event moves ORD10001 to SHIPPED, and the customer's order view reflects the change. A later ShipmentDelivered event moves the order to DELIVERED. The Notification Service consumes the same events independently and emails the customer at each milestone, without the OMS ever waiting for it.

Technical nuance: The order workflow must never be implemented as one long-running HTTP request. Long-running business operations need durable state, explicit progress tracking, and recovery mechanisms.

The walkthrough made inventory reservation look like a single tidy step. In reality, it is where the hardest concurrency problem in the whole system lives, because many customers can compete for the same last few units at the same moment.

Inventory Management and Consistency

Understanding inventory quantities

Inventory reasoning starts with the OnHand, Reserved, and Available quantities. OnHand is the physical quantity currently recorded as held in the warehouse. Reserved is the quantity already allocated to active reservations. Available is the quantity that can still be promised to new orders, and in a simplified model it is calculated as follows.

Available = OnHand - Reserved Warehouse A, Product: Laptop OnHand = 10 Reserved = 7 ----------------- Available = 3

Warehouse A holds ten laptops, seven of which are promised to other orders, so only three can be offered to new customers. In this simplified model, all on-hand stock is eligible to be sold, so Invariant 1 becomes the rule that Reserved never exceeds OnHand. Put another way, Available must never fall below zero. Production inventory models also track damaged stock, safety stock, committed allocations, and inbound deliveries, and some of these reduce the stock that is eligible to be sold. The simplified model is enough to explain the concurrency problems that matter here.

Preventing overselling

Suppose only one laptop is available and two customers, A and B, try to buy it at the same moment. A naive implementation reads the available quantity, checks that it is large enough, and then writes the reservation as a separate step.

Notice that each customer's check is correct on its own; the failure comes entirely from the gap between checking and reserving.

Both customers read an available quantity of one before either reservation is written. Both checks pass, both reservations go through, and the warehouse has promised two laptops when it holds one. The root cause is that the availability check and the reservation update are not protected as one atomic operation. An atomic operation is one that happens completely or not at all, with no point in between at which another operation can interfere.

Atomic inventory reservation

One way to close the gap is to fold the check into the update itself, using a conditional update. The database increases the reserved quantity only if enough stock remains, and it evaluates that condition and applies the change as one operation.

UPDATE Inventory SET Reserved = Reserved + @Quantity WHERE SKU = @SKU AND WarehouseId = @WarehouseId AND OnHand - Reserved >= @Quantity;

The application then checks how many rows were updated. If one row was updated, the reservation capacity was acquired. If zero rows were updated, the requested quantity could not be reserved under current conditions. In the laptop race, only one of the two customers would see a row updated. The statement must run with transactional guarantees suited to the chosen database, so that concurrent statements cannot both pass the condition.

Adjusting a counter is not enough on its own. The system also needs a durable reservation record tied to the order, so that it knows which order holds which stock and can recognise a retried request. Conceptually, the Inventory Service performs the following inside one transaction.

BEGIN TRANSACTION 1. Check whether ReservationId already exists. 2. Atomically reserve available stock. 3. Persist the reservation record. COMMIT If any required step fails: ROLLBACK

The ReservationId column carries a uniqueness constraint, which means the database refuses to store two rows with the same value. If a retried ReserveInventory command arrives for a reservation that already exists, the constraint lets the service recognise the retry and return the existing result, instead of reserving the same stock twice.

Multiple-item reservations

Order ORD10001 contains two different items, and our all-or-nothing policy requires both to be reserved. Imagine the laptop reservation succeeds but the mice cannot be reserved because only one is left. The laptop reservation must then be released, and the order's reservation as a whole is reported as failed.

Notice that the amber release step exists only because the two reservations were committed separately; one shared transaction would make it unnecessary.

Whether the release step is needed at all depends on the consistency boundary. This design serves each order from a single warehouse. If that warehouse's stock lives in one transactional inventory database, the Inventory Service can reserve every item inside one database transaction. Either all items are reserved or none are, and no release step is needed. If the items were held in independently managed stock stores, the reservation would become a coordinated workflow, and the Inventory Service would need explicit compensation to release earlier successes. The implementation follows from where the consistency boundary lies.

The reservation lifecycle

A reservation has its own small state machine.

• Once created, it is RESERVED, which means the stock is allocated to the order. Every reservation then ends in exactly one terminal outcome.
• It becomes CONSUMED when the reserved stock is turned into an actual inventory movement for fulfillment.
• It becomes RELEASED when the order is cancelled or rejected and the stock returns to the available pool.
• It becomes EXPIRED when its deadline passes before the reservation has been secured for a confirmed order.

In this simplified model, these outcomes are mutually exclusive.

Consuming inventory

When reserved items physically leave the warehouse, the system must take care not to count them twice. In this article, we assume that the reservation is consumed at dispatch. Suppose that before dispatch a warehouse holds 10 units, of which 3 are reserved, leaving 7 available. Two of the reserved units are then shipped.

Before dispatch: OnHand = 10 Reserved = 3 Available = 7 After dispatch: OnHand = 8 Reserved = 1 Available = 7

Both OnHand and Reserved fall by two, so Available stays at 7. That is correct, because the shipped units were never available to other customers in the first place. If the system reduced only OnHand, it would show OnHand = 8, Reserved = 3, and Available = 5, wrongly removing two sellable units from the store. Real warehouses may choose to consume reservations at picking, packing, or dispatch, depending on their accounting model. Whatever the choice, the consumption point must be defined explicitly and used consistently.

Reservation expiry

A reservation cannot hold stock forever. Otherwise, abandoned checkouts and stuck payments would slowly lock away sellable inventory. For illustration, assume that a reservation is held for 10 minutes while payment authorization is pending. The real duration is a business decision that must account for payment-processing latency and the retailer's requirements.

Notice that a late payment authorization does not lead to confirmation; it leads to payment reconciliation, because the stock it depended on is no longer held.

If authorization succeeds before the deadline, the order is confirmed as normal. If the deadline passes first, the reservation expires, and the payment must be reconciled. Any authorization that arrives later must be voided, retried against a fresh reservation, or otherwise compensated. The OMS cannot confirm an order merely because a late payment authorization arrives. It must first verify that a valid inventory reservation still exists, which is precisely what Invariant 3 demands.

Reservation expiry also creates a race condition worth naming. Expiry and confirmation can be attempted at almost the same instant, and only one of them may win. The order record lives in the OMS database and the reservation lives in the inventory database, so the decision has to be made inside the Inventory Service, which owns the reservation. Before the OMS moves an order to CONFIRMED, the Workflow Coordinator asks the Inventory Service to secure the reservation. The Inventory Service clears the reservation's expiry deadline with a conditional state update that succeeds only if the reservation is still RESERVED and its deadline has not passed.

The expiry process uses the matching condition and changes the reservation to EXPIRED only if it is still RESERVED and its deadline has passed. Equivalent transactional controls work as well. Whichever update commits first wins, and the other finds the reservation no longer in the state it expected. The OMS confirms the order only when the Inventory Service reports that the reservation was secured.

Technical nuance: Reservation expiry is a business-state transition, not merely the deletion of a cache entry. An expired reservation must never be resurrected accidentally by a delayed workflow message.

Inside the Inventory Service, a single database transaction can protect stock. The order workflow, however, crosses the inventory database, the payment provider, and the warehouse, and no single transaction spans all of them. The next section deals with that wider problem.

Distributed Transactions and Workflow Coordination

The distributed transaction problem

The order workflow touches the order database, the inventory database, an external payment provider, and the fulfillment system. Each of these commits its own changes independently, and they cannot simply be committed or rolled back together as one database transaction.

Consider what happens to ORD10001 if things go partly wrong. The Order Service successfully creates the order, the Inventory Service successfully reserves the laptop and mice, and then the payment provider declines the authorization. Two systems have committed real changes, and the third has refused. The OMS now owns a partially completed workflow, and it must decide how to bring it to a clean, correct end.

Why not use one global transaction?

A traditional distributed transaction tries to make several participants commit or abort together under a single commit protocol. The best-known protocol, two-phase commit, asks every participant to prepare and then tells them all to commit. That approach struggles badly in this setting. The participants are independently deployed services, an external payment gateway, a message broker, and warehouse activities that can last for hours. Many of these do not support such a protocol at all.

Even where the protocol is available, it would hold locks across several systems for the whole duration of the workflow, tie the availability of every service to the slowest participant, and couple the services tightly together. For this OMS, an application-level workflow, where the application itself manages progress and recovery, is the more suitable choice.

The saga pattern

A saga is a sequence of local operations, each of which commits inside a single service's boundary. If a later step cannot proceed, the saga runs compensating actions, which are business operations that undo or offset the effect of earlier steps. For ORD10001, the happy path is a saga that creates the order, reserves inventory, authorizes payment, and confirms the order. The interesting part is what happens on the unhappy paths.

Notice that every amber compensation corresponds to an earlier emerald step, and that the cancellation lane reaches CANCELED only after both cleanup actions are confirmed.

When payment is declined after inventory has been reserved, the saga releases the reservation and then marks the order REJECTED. When a customer cancels an order whose stock is reserved and whose payment is authorized, but which has not yet been sent to the warehouse, the saga releases inventory and voids the authorization. These cleanup actions can run in parallel.

The saga then confirms that both completed before it marks the order CANCELED. If either cleanup step remains uncertain, the workflow must keep tracking it until recovery succeeds, instead of declaring cleanup complete too early. An order that has already been sent to the warehouse needs one more step first.

The Fulfillment Service must confirm that the order has not crossed the fulfillment cutoff and that work has stopped. If payment was captured before the cancellation, the saga then refunds the captured amount instead of voiding the authorization. The discussion of saga isolation limitations later in this section explains why only the Fulfillment Service can confirm the cutoff.

Saga orchestration

Sagas can be coordinated in two classical styles: orchestration and choreography. In orchestration, a central coordinator decides the sequence. In our architecture, that coordinator is the Order Workflow Coordinator. It sends commands to the Inventory, Payment, and Fulfillment services and keeps a durable record of the current step, the operations already completed, the commands still pending, the retry state, any required compensations, and the final outcome.

Orchestration suits the core of this OMS because confirming and cancelling an order involve an explicit sequence of business decisions. Stock must be reserved before payment is authorized. Both must be valid before confirmation. Cancelling an order that has already reached the warehouse must wait for the Fulfillment Service to confirm that the order has not crossed the fulfillment cutoff. Having that logic in one place makes it easier to read, change, and debug.

Saga choreography

In choreography, there is no central coordinator. Each service listens for events and reacts by doing its own part. The Order Service publishes OrderCreated. The Inventory Service reserves stock and publishes InventoryReserved. The Payment Service reacts by authorizing payment and publishing PaymentAuthorized. The Order Service then confirms the order and publishes OrderConfirmed, which the Fulfillment Service picks up.

Notice that in choreography the workflow exists only as a chain of reactions, so no single component can show how far an order has progressed.

Choreography works well for simple workflows, and it keeps services loosely coupled. As interdependencies, recovery paths, and cancellation rules pile up, however, the overall sequence becomes spread across many event handlers, and it becomes harder to understand and operate. The table below compares the two styles.

The recommended split for this OMS uses both styles. Use orchestration for the core order confirmation and cancellation workflows, where sequence and recovery matter. Use asynchronous events for notifications, analytics, and any other consumer that simply needs to react to what has happened.

Compensating transactions

A compensating transaction attempts to reverse or offset the business effect of an earlier operation. Each forward step in the order workflow has a matching compensation.

A compensation is not the same as a database rollback. A rollback makes it as if the change never happened. A compensation is a new business action that leaves its own trace. Some actions cannot be compensated cleanly at all. Once a shipment has left the warehouse, its physical movement cannot be undone, and the business needs a return or customer-service workflow instead. This is why the design works so hard to stop cancelled orders from being dispatched in the first place.

Saga isolation limitations

A database transaction offers isolation, meaning concurrent transactions cannot see or interfere with each other's half-finished work. A saga offers no such protection, because each of its steps commits independently and is immediately visible. This brings us back to the race from the lifecycle section: a cancellation workflow and a dispatch workflow running at the same time. If the OMS cancels ORD10001 while the warehouse independently dispatches it, the business ends up with a cancelled order sitting in a delivery van.

Notice that the decision to cancel or dispatch is made inside the Fulfillment Service, under a single lock, and the OMS only records the outcome it is told.

The solution is to coordinate dispatch eligibility explicitly. The Fulfillment Service is the authority on whether an order has crossed the fulfillment cutoff, the point after which warehouse work can no longer be stopped, because it is the only component that knows what is physically happening in the warehouse. The OMS acknowledges a cancellation request as pending, which is the CANCELING state, until fulfillment confirms that the request can be honoured.

Inside the fulfillment boundary, cancel and dispatch decisions are serialised or otherwise coordinated atomically, so that exactly one of them can win for any order. The OMS finalises the cancellation only after the required services have confirmed the outcome. If the fulfillment cutoff has passed, the OMS restores the order's active state and tells the customer honestly that the order will ship.

Real systems place this decision either with the warehouse side or with the OMS. When the warehouse side decides, the OMS sends a cancellation request, and the fulfillment service accepts or rejects it based on how far the physical work has progressed. Shopify's fulfillment API follows this model, and some warehouse systems stop accepting cancellations once picking or shipping has begun.

When the OMS decides, it cancels the order on its own and instructs the warehouse to stop. That model is safe only if the warehouse cannot ship without the OMS's permission. Otherwise the parcel may already be leaving when the instruction arrives. This article lets the warehouse side decide, for the reason given above. The payment check before dispatch does not change that choice. That check only confirms that money has been collected, and the cancel-or-dispatch decision stays with the Fulfillment Service.

Technical nuance: Sagas solve workflow coordination, not every concurrency problem. Business invariants must still be enforced by the services that own the relevant data and decisions.

A saga assumes that each step either succeeds or fails cleanly. Real networks are less tidy. Responses get lost, messages arrive twice, and processes crash between two writes. The next section makes the workflow robust against exactly these failures.

Reliability and Failure Handling

Idempotent order creation

An operation is idempotent when performing it several times has the same effect as performing it once. Order creation needs this property because of a very ordinary failure. The customer's app sends POST /orders, the OMS creates ORD10001, and the response is lost on the way back. The app sees a timeout and retries.

Notice that the retry returns the original order instead of creating a new one, because the idempotency key ties both requests to the same logical submission.

Without idempotency, the retry would create a second order, and the customer might be charged twice. With it, the OMS recognises the key and returns ORD10001. To make this work, the OMS stores an idempotency record for each submission. The record holds the key itself, the scope it belongs to (such as the customer or client), a request fingerprint (a hash of the request body), the status of the operation, the resulting order identifier, the stored response, and creation and expiry times.

The details matter. Two retries carrying the same key can arrive at the same moment, so the system must claim the key atomically, for example by inserting the record under a uniqueness constraint. The fingerprint also catches a subtler problem. If a client reuses a key with a different request body, that is not a retry. It is a mistake, and the OMS should reject it instead of returning an unrelated order.

Idempotency across participating services

Idempotency at the front door is not enough if the commands sent downstream can also be duplicated. The Workflow Coordinator may retry a command after a timeout, and many message brokers guarantee only at-least-once delivery, which means a message is never lost but may be delivered more than once. Each command therefore carries a stable operation identifier that the receiving service can use to recognise repeats.

ReserveInventory(OrderId, ReservationId) AuthorizePayment(OrderId, PaymentAttemptId) CapturePayment(PaymentId, CaptureOperationId) RefundPayment(PaymentId, RefundOperationId) InitiateFulfillment(OrderId, FulfillmentRequestId)

The ReservationId is the same identifier whose uniqueness constraint protected the Inventory Service earlier. The other identifiers play the same role for payment and fulfillment, which is how Invariant 4 is kept. A repeated capture with the same CaptureOperationId collects the money only once, and a repeated refund with the same RefundOperationId returns it only once.

Ambiguous payment outcomes

Payments produce the most dangerous kind of uncertainty. The OMS asks the Payment Service to authorize ORD10001. The Payment Service forwards the request to the gateway, and the gateway approves it. The response then fails to reach the OMS, and the OMS sees only a timeout.

Notice that the money was actually authorized at the gateway; only the OMS's knowledge of that fact was lost.

The OMS must not assume that the authorization failed. A decline and a timeout are different facts. A decline means the provider explicitly refused. A timeout means the final outcome is unknown.

The Payment Service resolves the uncertainty in a disciplined way. It persists every attempt with a stable operation identifier before calling the gateway. When an outcome is unknown, it checks whether the provider completed the operation, using the provider's query APIs or its callbacks. It then returns the established result and never creates a duplicate authorization along the way.

The same reasoning applies to inventory reservations, fulfillment initiation, payment capture, and refunds. Whenever an outcome is unknown, the system finds out what actually happened before acting again.

Reliable event publication

Uncertainty also arises inside the OMS itself. Suppose the OMS updates ORD10001 to CONFIRMED, commits the database transaction, and then crashes just before publishing the OrderConfirmed event.

The order is confirmed in the database, but the event never reaches the broker, so the Fulfillment Service never learns that the order exists. This is the dual-write problem: writing to two separate systems, here the database and the broker, without any guarantee that both writes happen.

The transactional outbox

Notice that the risky step has moved: committing the state change and recording the event now succeed or fail together, and only publication, which is safe to retry, happens later.

The transactional outbox closes the gap. Instead of publishing directly, the Order Service writes the event as a row in an outbox table within the same local transaction that updates the order. Either both the status change and the outbox row are committed, or neither is. A separate outbox publisher then reads unpublished rows and sends them to the broker, retrying until each send succeeds.

The outbox guarantees that no event is lost. It does not guarantee that no event is duplicated. The publisher may deliver an event successfully and then crash before marking the row as published. When it restarts, it will send that event again. Consumers must therefore tolerate repeated delivery.

Idempotent consumers

An idempotent consumer processes each distinct message only once, however many times the message arrives. Suppose the Fulfillment Service receives OrderConfirmed with event identifier E123 and creates a fulfillment job. If E123 arrives again, the consumer sees that it has already processed E123 and does nothing, so no duplicate fulfillment is created.

The consumer typically records processed event identifiers in a table. Ideally, it writes the deduplication record in the same transaction as the business change, so the two can never drift apart. A business-level uniqueness constraint adds a second safeguard. For example, the fulfillment table can allow only one fulfillment job per FulfillmentRequestId.

Technical nuance: The goal is effectively-once business side effects, not an assumption that every message will be physically delivered exactly once.

Out-of-order events

Duplicates are not the only delivery anomaly. Messages can also arrive in a different order from the one in which they were produced. Suppose the Fulfillment Service publishes PACKED as version 12 of ORD10001's fulfillment record, and DISPATCHED as version 13. Because of delivery delays, the OMS receives version 13 first and sets the fulfillment status to DISPATCHED. When version 12 arrives later, applying it would move the order backwards to PACKED.

When the source assigns versions that only increase for a given record, the consumer can compare them and reject any update older than the one it already holds. The comparison works only within a single sequence, however. Payment and fulfillment events have independent version sequences, so the OMS must not compare a payment version against a fulfillment version as if they shared one global clock. The sequencing contract should be defined per record and per source, and validation of state transitions remains necessary as a second line of defence.

Retry strategies

Retrying is the natural response to failure, but only some failures deserve a retry. A transient failure is temporary and may succeed on another attempt, such as a brief outage or a network blip. A permanent business failure will fail every time, such as a declined card or an item that is out of stock. Retrying a permanent failure wastes capacity and delays the customer's answer.

Exponential backoff means waiting progressively longer between attempts, so that a struggling service is not flooded with retries. Jitter adds a small random variation to each wait, so that many clients do not all retry at the same instant. Every retry policy should also have a ceiling, either a maximum number of attempts or a time budget, and a durable record of unfinished work, so that nothing is forgotten when the ceiling is reached and the case is escalated.

Reconciliation

Even with durable workflows and idempotency everywhere, services can temporarily disagree. A payment may be AUTHORIZED while its order is still PENDING. Stock may still be RESERVED for an order whose payment was declined, because the release has not yet completed. An order may be CONFIRMED while its fulfillment is still NOT_STARTED. Some of these mismatches are simply in-flight work. Others are genuine gaps left by a crash or a lost message.

Notice that reconciliation only examines work that has exceeded its expected duration, so normal asynchronous delays are not mistaken for faults.

Reconciliation is a scheduled process that finds and repairs these gaps. It looks for orders and operations that have taken longer than expected, asks the authoritative services what actually happened, and compares that with what the OMS believes. If the two agree, nothing is done. If they disagree, the process decides on a safe action, which may be resuming the workflow, compensating, or escalating to a person.

Reconciliation must distinguish genuinely inconsistent states from normal delays, and it must keep enough context to avoid replaying a business operation incorrectly, such as authorizing a payment that already exists.

Technical nuance: A reliable system needs both prevention and recovery. Idempotency, transactions, and durable messaging reduce the likelihood of errors. Reconciliation detects and resolves the incomplete outcomes that still occur.

A design that behaves correctly for one order must also behave correctly for thousands of orders at once, and operators must be able to see when it does not. That brings us to scale and observability.

Scalability, Performance, and Observability

Workload characteristics

An OMS serves order creation, order retrieval, and order processing, and these workloads have very different shapes.

Order creation is a write-heavy transactional workload. It needs reliable persistence, validation, and workflow initiation, and each new order triggers further downstream work.

Order retrieval is read-heavy. Customer apps, support tools, and operational dashboards all use it, and some fields can tolerate being slightly out of date.

Order processing is event-driven or workflow-driven. It calls inventory, payment, and fulfillment, and a single workflow may run for seconds, minutes, or much longer. Because these workloads differ so much, they should be able to scale independently.

Scaling the OMS

Notice that the crimson bottleneck markers sit on shared resources, which do not grow when more API instances or workers are added.

The deployment separates request handling from long-running workflow processing. The API instances are stateless, meaning they keep no order data in memory between requests, so more instances can be added behind the load balancer as traffic grows. Workflow workers can also scale with demand, but their concurrency must be coordinated so that two workers never advance the same order's workflow at the same time.

One common approach gives each workflow instance a lease, which is a time-limited claim that only one worker can hold at a time. If a worker crashes, its lease lapses and another worker can take over the workflow instance. Locking each workflow instance achieves the same protection. Adding workers has a limit, though. The order database, the Inventory Service, and the payment integrations can become bottlenecks no matter how many OMS workers are running.

Read and write workloads

A common next step is to separate read-heavy order views from transactional processing. This is a CQRS-style design, short for Command Query Responsibility Segregation, in which writes and reads use different models. The Order Service writes to the primary order database and publishes order events. A read-model updater consumes those events and maintains a separate order query store, which the customer order API reads from.

The benefits are real. Read queries can be optimised independently, the transactional model stays focused on correct writes, and customer order history can use denormalized structures that are fast to display. The costs are real too. The read model may lag behind the authoritative state, the event processing that feeds it must be reliable, and rebuilding a read model adds operational work.

CQRS is an option, not a requirement for every OMS. It earns its place when read traffic clearly outgrows what the transactional store can serve comfortably.

Handling peak demand

During a flash sale, order traffic can rise sharply, and pressure appears in several places at once: contention on popular products, database write capacity, payment-provider throughput, workflow backlogs, and downstream fulfillment capacity. A message broker can separate the rate at which work arrives from the rate at which it is processed, but it does not remove downstream limits.

To see why, assume 5,000 operations per second arriving and a processing capacity of 2,000 operations per second. The backlog then grows by 3,000 operations every second. After one minute, 180,000 operations are waiting. A durable backlog can absorb a short burst, but it cannot sustain that imbalance indefinitely. The system needs backpressure, which slows producers when consumers fall behind, bounded concurrency, which caps how much work runs against each downstream service at once, and admission control, which limits how many new orders are accepted when capacity is exhausted. Adding a queue does not, on its own, solve a scalability problem.

Hot inventory items

A hot item is a single popular SKU that attracts far more purchase attempts than it has stock. Assume a limited-edition laptop with 100 units available and 10,000 concurrent purchase attempts. At most 100 of those attempts can succeed, and the other 9,900 must fail cleanly. Even if the OMS can accept thousands of requests per second, every reservation attempt contends for the same inventory record. The atomic conditional update from the inventory section keeps the count correct, but it also means that those attempts are processed one after another at that record.

Which optimisation fits depends on the requirements. Options include atomic conditional updates, partitioning stock by SKU or by warehouse, routing selected hot items through a serialised reservation queue, applying admission control before expensive downstream calls are made, and specialised reservation counters whose consistency guarantees are well understood. The central lesson is that scaling the OMS horizontally does not remove contention at the inventory consistency boundary.

Observability

Observability is the ability to understand what a system is doing from the signals it emits. For an OMS, technical metrics such as CPU and memory alone cannot reveal whether orders are being processed correctly. The system also needs business metrics.

Distributed tracing follows a single request as it passes through many services. It relies on a correlation identifier, a value attached to every operation belonging to the same order. For our example, correlation ID CORR10001 links order ORD10001, reservation RES10001, payment attempt PAY10001, and fulfillment FUL10001. An operator investigating a stuck order can then see the whole journey without searching disconnected logs by hand. Sensitive customer and payment details should be kept out of logs and traces unless they are strictly needed.

Technical nuance: Infrastructure health and business-process health are different things. Every service can be running while hundreds of orders sit stuck in intermediate states. OMS observability must therefore measure business progress, not just CPU, memory, and API availability.

We have now made a long series of decisions, each one justified locally. The final section brings them together so that the design can be judged as a whole.

Architectural Decisions and Trade-offs

Decision summary

Strong versus eventual consistency

Strong consistency means every reader sees the latest committed value immediately. Eventual consistency means different parts of the system may briefly disagree but will converge on the same value once in-flight updates are processed. This design deliberately uses both, as the diagram below summarises.

Notice that the operations protecting stock and order state sit at the strong end, while everything that only reports on those facts sits at the eventual end.

Eventual consistency is often misread as permission to be wrong. It is not acceptance of invalid business outcomes. The customer's order page may show a slightly stale status for a few seconds, but no stock is ever over-reserved, and no payment is ever duplicated in the meantime. Those rules are always enforced by the services that own the relevant data.

Key architectural lessons

• Order management is coordination, not ownership of everything. The OMS owns the order lifecycle. It should not absorb the internal responsibilities of the inventory, payment, and fulfillment systems, because each of those systems is the authority on its own facts.
• Business invariants determine consistency boundaries. An inventory reservation needs stronger guarantees than an asynchronous notification. Consistency requirements should be derived from business rules, not applied uniformly across the platform.
• Long-running workflows need durable progress tracking. A request can finish while order processing carries on for days. Workflow state must survive application restarts, timeouts, and transient failures.
• Acknowledgement does not imply completion. Accepting an order, reserving inventory, authorizing payment, capturing payment, and completing delivery are separate milestones, and each carries its own guarantee.
• Failure recovery is a normal part of distributed business workflows. Retries, compensations, and reconciliation should be designed alongside the happy path, not added after the first production incident.

An Order Management System looks simple from the checkout button and complex from the inside. The complexity is not accidental. It reflects the fact that an order is a promise spanning stock, money, and physical goods, made across systems that fail independently. A design that names its invariants, places each one inside the service that can enforce it, and treats uncertainty and recovery as ordinary work will keep that promise reliably, even on the day the last laptop sells out.