Model Serving

> Increasing batch size from 4 to 16 improves throughput by 33% — and makes average latency 20% worse. Throughput and latency are not the same goal, and this chapter measures exactly where they pull apart.