Azure · Workers · Batch
Async workers for high-volume catalog updates
Updating a few dozen products is trivial. Updating tens or hundreds of thousands — with price, stock and attribute changes arriving in waves — requires an architecture designed for volume. Asynchronous workers are the standard way to do it in a controlled manner.
Why not do it inline
Processing a large catalog inside a single request or a single monolithic job leads to timeouts, spikes and total failure on the first error. Breaking the work into small, independent units, queued and processed by workers, makes the system resilient and scalable.
Queues and units of work
Each change (a product, a variation group, a batch) becomes a message in a queue. Workers consume messages in parallel. Benefits: failures isolate a single unit (which lands in dead-letter), load is distributed, and you can resume from where you stopped.
Batching: the right trade-off
Calling the API once per product is inefficient; sending everything at once is risky. Batching groups N items per call, leveraging marketplace bulk operations (e.g. eBay Feed/bulk operations). Batch size must be tuned to API limits and payload size.
Controlled concurrency
| Lever | Effect |
|---|---|
| Number of workers | How many messages in parallel |
| Max parallelism per host | Avoids saturating CPU/connections |
| Shared rate limiter | Respects the global API budget |
| Batch size | Balances throughput and risk |
The goal is to maximise throughput without exceeding the limits of downstream systems.
Bulk and asynchronous operations
Many marketplaces offer asynchronous bulk operations: you upload a set of changes and get a job to monitor. Workers must know how to send the batch, track the job and read the result per item, reporting failures for reprocessing.
Recovering from failures
In a mass update, something will always fail. The key is granularity: an error on one product must not invalidate the whole batch. With independent messages, idempotency and dead-letter, failed items are reprocessed without starting over.
Architecture on Azure
On Microsoft Azure: a Service Bus queue for units of work, Azure Functions or containers as workers with configured parallelism, a shared rate limiter, and storage for state and logs. An orchestration Function can split a large update into batches and track its progress.
Common mistakes
- monolithic jobs that fail entirely on the first error;
- one call per product, inefficient and slow;
- unbounded concurrency that saturates APIs and infrastructure;
- no granular recovery from failures.
Conclusion
Asynchronous workers with queues, batching, controlled concurrency and granular recovery are the reliable way to update high-volume catalogs. They turn a risky operation into a predictable, observable process, even with hundreds of thousands of products.