What Is Batch Inference?

Batch inference is running an AI model over a large set of inputs all at once, on a schedule or in bulk, rather than responding to each request the moment it arrives. It trades immediacy for efficiency, processing many items together to lower cost and raise throughput.

How batch inference works

Instead of calling the model once per request in real time, batch inference collects inputs into a group and runs them through the model together, often as a scheduled job. Scoring every customer in a database overnight, tagging a whole library of documents, or generating embeddings for a large dataset are typical jobs. Because the work is not waiting on a person, it can be scheduled for when compute is cheaper and run at higher volume.

Batch inference vs real-time inference

The difference is timing. Real-time inference, also called online inference, answers a single request immediately, which a chat reply needs. Batch inference processes many inputs together with no live user waiting, which suits large, non-urgent jobs. Cost per item is usually lower in batch, because the work can be packed and scheduled.