Introduction

Model inference workloads vary: quiet periods with minimal load, market-open bursts requiring thousands of predictions/second. Serverless computing (AWS Lambda, Google Cloud Functions) auto-scales to handle burst loads without over-provisioning for peak capacity, reducing costs while maintaining availability.

Serverless for ML Inference

Deploy lightweight model servers as Lambda/Cloud Function instances. Trigger on API requests. Auto-scale: 0 instances idle, thousands under peak load. No ops overhead; managed by cloud provider. Pay only for actual usage (compute seconds), not reserved capacity.

Trade-offs

Advantages: cost efficiency, auto-scaling, low ops overhead. Disadvantages: cold start latency (first invocation has delay), limited execution time (typically < 15 minutes), not suitable for long-running batch jobs. Best for short, stateless predictions.

Hybrid Approach

Use serverless for bursty workloads (market microstructure models). Use dedicated servers for baseline load (large-cap portfolio models). Combination optimizes cost and latency across diverse workloads.

Conclusion

Serverless architectures cost-effectively handle bursty ML inference workloads, reducing infrastructure costs without sacrificing availability.