Llama 3.2 Vision-Instruct Inference Speed on A100 or H100 GPU

garage_greg

Can anyone provide an estimated time of how long does it take for Llama-3.2 Vision-Instruct 11-B model to: process an image size of 1-MB and prompt size of 1000 words and generate a response of 500 words The GPUs used for inference could be A100, A6000, or H100.

Originally asked by Adil on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

On the A100, Llama 3.2 Vision-Instruct achieves moderate inference speeds, leveraging FP16 and TensorRT optimizations. However, the H100 significantly boosts performance with FP8 support, Transformer Engine, and higher memory bandwidth, enabling faster multi-modal processing. For real-time deployment, H100 with quantization (e.g., INT4, INT8) and tensor parallelism offers the best latency and throughput

Full answer by Emily Carter: https://ai.stackexchange.com/a/48127 (CC BY-SA 4.0).

Need it done for your business? UphillTech.