If you’ve spent any time managing fulfillment operations, you know the feeling. Someone in a meeting asks, “What would happen if we sourced more orders from the East Coast DCs?” and the room goes quiet. Maybe someone pulls together a spreadsheet model. Maybe you run a simulation. And then you make a decision worth millions of dollars based on your best guess.
There’s a better way. A/B testing the order sourcing logic in your distributed order management system (OMS) lets you stop guessing and start knowing—with real orders, real inventory, and real customers. Here’s why it matters and how to do it right.
What Is A/B Testing in an OMS System?
Your OMS is the brain behind order sourcing. When an order comes in, it decides which warehouse, store, or fulfillment center ships it… and how. Sourcing logic is the set of rules that drives those decisions. It might prioritize the closest node, the lowest cost option, the location with the most inventory, capacity, or some weighted combination of these and many more.
A/B testing in this context means running two versions of your sourcing logic at the same time. Version A might be your current rules. Version B is the new logic you want to evaluate. Your OMS system routes a portion of live orders through each version, and then you compare results across a defined set of metrics. No guessing. No projections. Actual outcomes from actual orders.
Why Live Testing Beats Simulation Every Time
Simulation has its place, but it has a fundamental flaw when applied to fulfillment: you can never perfectly recreate your inventory picture at any given moment in time.
Think about what’s happening across your fulfillment network right now. Inventory counts are being updated by receiving teams. Stock is being sold in stores. Returns are being processed and put back into available-to-promise pools. Transfers are in transit. Flash sale demand is draining stock at three DCs faster than your replenishment cycle can respond. No simulation model captures all of that simultaneously, with full fidelity, in real time.
Live A/B testing routes actual orders through actual conditions. If your test logic routes orders to a DC that turns out to be oversold on a key SKU, you see exactly what happens. And you measure it. That split fulfillment? That delayed shipment? That margin hit? It shows up in your data. A simulation would have assumed perfect inventory accuracy and missed it entirely.
Live testing also captures carrier performance in real conditions. Lane delays, dimensional weight surcharges, pickup cutoff misses. These are things your simulation assumes away but your customers definitely notice.
The Power of Testing at Small Volume
One of the most under-appreciated features of a well-built A/B testing framework is the ability to control what percentage of your order volume runs through the experimental logic. You don’t have to flip the switch on 100% of orders to learn something valuable.
Starting at 1% or 10% of order volume is a risk management strategy as much as it is a testing strategy. At 1%, if your new logic has an unintended flaw (say, it’s over-indexing to a node that’s about to go out of stock) the damage is contained. You catch it early, adjust, and move on. At 10%, you’re generating statistically significant data much faster while still protecting 90% of your volume from any downside.
This also means you can run tests continuously and in parallel without disrupting operations or requiring a big-bang rollout. You can have one test running at 5% of volume evaluating a new freight carrier preference rule, while another test at 2% evaluates a new split order threshold. Your operations team barely feels it. Your data team has a constant stream of insights.
And when results look good, you scale up confidence gradually rather than making a leap of faith. From 1% to 5%, to 25%, to full rollout.
What Types of Sourcing Logic Should You Test?
The short answer is: anything that influences which node fulfills an order, how it ships, and at what cost. Here are some of the most impactful categories.
Node prioritization rules. What happens if you shift priority from “closest to customer” to “lowest fulfillment cost”? Or add a new tier that prioritizes nodes with excess inventory to reduce carrying costs? Testing these changes tells you how much delivery time you’re trading for cost savings. Or vice versa.
Split order thresholds. At what point does it make sense to split an order across two nodes versus waiting for a single node that has all items? Tightening or loosening this threshold has direct impact on your split fulfillment rate, delivery cost, and customer experience. Testing it live shows you where the real break-even point sits.
Carrier and service level assignment. If your OMS logic influences carrier selection, you can test whether routing more volume to a regional carrier in a specific geography saves money without hurting on-time delivery. Or test whether upgrading a segment of orders to a faster service level impacts repeat purchase rates.
Store fulfillment eligibility. Expanding or contracting which stores are eligible to fulfill ecommerce orders is a high-stakes decision. A live test at low volume lets you evaluate pick accuracy, fulfillment time, and margin impact before committing to a wider rollout.
Distance and zone caps. Testing whether capping fulfillment beyond a certain shipping zone improves margin without materially harming OTIF or delivery time is something simulations struggle to model accurately given carrier performance variability.
How to Measure What Matters
A good A/B test is only as valuable as the metrics you use to evaluate it. Here are the ten metrics that give you a full picture of what your sourcing logic change actually did—and the trade-off decisions each one informs.
Average Gross Margin per Order tells you the raw dollar contribution left after product cost and direct fulfillment expenses. If your test logic improves this number, you’re sourcing more profitably. If it drops, you’re paying more to fulfill than you’re gaining elsewhere.
Average Gross Margin % normalizes margin across your order mix, which matters when your test group and control group have slightly different average order values. A 5-point improvement here means your logic is structurally more efficient, not just luckier on high-AOV orders.
Average Net Margin Value and Net Margin % go a layer deeper, folding in overhead allocations, returns costs, and other below-the-line expenses. A sourcing rule that looks great on gross margin can erode quickly at the net level if it’s driving higher return rates or more customer service contacts. These two metrics are your sanity check.
Average Delivery Cost per Order is your most direct read on the financial efficiency of your sourcing decision. Lower delivery cost is usually the goal of any optimization test, but it should never be read in isolation. A sourcing logic change that cuts delivery cost by $1.50 per order but tanks your On Time In Full (OTIF) rate is not a win.
Average Delivery Distance is a leading indicator for both cost and speed. If your test logic is routing orders from farther nodes on average, expect to see it show up in cost and time metrics downstream. Conversely, if distance is dropping, you’re likely to see margin improvement and faster delivery—assuming carrier performance holds.
Average Order-to-Door Time is the customer-facing version of your speed story. This is the number your customers actually experience. A sourcing logic test that shortens this metric tends to have downstream benefits on satisfaction and repeat purchase behavior. The trade-off question it forces is: how much are you willing to spend to shave a day off delivery time?
On Time In Full (OTIF) Rate is arguably the most important customer experience metric in this set. It measures whether orders arrived when promised and arrived complete. A sourcing logic change that improves margins but drops OTIF is a problem, because the hidden cost (customer dissatisfaction, returns, lost lifetime value, etc.) rarely shows up cleanly in margin calculations. OTIF is your guardrail metric. If it moves negative in a test, the test probably isn’t ready to scale.
Average Fulfillment Time measures the internal speed story. From order receipt to shipment handoff. This is where store fulfillment tests often reveal operational limits. A new node eligibility rule might look good on paper, but if those locations are adding two days to fulfillment time, your order-to-door time will follow.
Split Fulfillment Rate is the metric that sits at the intersection of cost, experience, and complexity. Every split shipment costs more to send and creates a worse customer experience. A sourcing logic test that reduces your split rate without sacrificing OTIF is close to a pure win. Tests that improve margin by increasing split rate require more careful analysis. You’re essentially asking whether the cost savings justify the customer experience hit.
Putting It All Together: Making Real Decisions
The real value of a well-instrumented A/B test isn’t any single metric. It’s the ability to look at all ten together and make an informed trade-off decision.
Say your test logic prioritizes low-cost nodes over proximity. After two weeks at 10% volume, you see: delivery cost is down $1.80 per order, gross margin is up 2.3 points, average delivery distance is up 180 miles, order-to-door time has increased by 0.8 days, and OTIF has dipped 1.4 percentage points. Is that a win?
That’s a real business decision. For a segment of customers who have shown low sensitivity to delivery speed, it might be an easy yes. For a premium customer segment or a high-velocity category, that OTIF dip might be disqualifying. The point is that you’re making that call with data, not instinct.
Without A/B testing, you’d have launched that logic change across 100% of orders, hoped for the best, and spent three months trying to untangle what actually happened.
Getting Started
You don’t need a perfect setup to start. Pick one sourcing logic hypothesis, something your team has been debating, and build a simple two-variant test at 5% of order volume. Define your success metrics in advance. Run it for two to four weeks. Read the data honestly.
The fulfillment operations managers who are going to win the next few years aren’t the ones with the best instincts. They’re the ones who have built systems that let them learn faster than the competition. A/B testing your sourcing logic is one of the highest-leverage ways to do exactly that.
For more information on how you can A/B test your order sourcing logic with Fluent Order Management, contact us today.
