L40S GPU Instance

Use H100 PCIe GPU Instances for Medium to Large scale Foundational model training but harness L40S capabilities to fine tune in hours and train in days small LLMs.

• An infrastructure powered by L40S GPUs can train Models in days
  To train Llama 2-7B (100B tokens) it would require 64 L40S GPUs and take 2.9 days (versus 1 day with H100 NVlink GPUs, like on Nabu2023)
• Fine-tune Models in hours
  To fine tune Llama 2-70B SFT (1T tokens), it will require 64 L40S GPUs L40S and take 8.2 hours (versus 2.5hours with H100 NVlink GPUs, like on Nabu2023)

Source: NVIDIA L40S Product Deck, October 2023

L40S GPU Instance, with 1 GPU, delivers inference performance that enables you to put Llama 2-7B main use cases in production efficiently:

• Chat: generate immediate responses to prompts with context
  A L40S* delivers a response to an input sequence of 225 with an output of 20 in 16msec for 1st token latency and 246 msec for End-to-end latency
• Q&A: generate quick responses to short questions
  L40S* delivers a response to an input sequence of 16 with an output of 64 in 12msec for 1st token latency and 771 msec for End-to-end latency
• Summarization and RAG: retrieve augmented responses based on interpretation/summarization of multi page content
  L40S* delivers a response to an input sequence of 2048 with an output of 128 in 77msec for 1st token latency and 1703 msec for End-to-end latency

*serving a LLama 2-7B model in FP8

Rendering is essential for bringing visual elements together, from lighting effects to textures, to generate the final result. Leveraging the Lovelace architecture, the L40S GPU Instance demonstrates superior performance compared to the Ampere architecture, particularly in rendering and graphics tasks. Some concrete results for you to compare:

• Rendering: the L40S is up to 2.4 times faster than a A40 for rendering (on Blender Cycles render engine)
• Graphics: the L40S is up to 2.3 times faster than a A40 for graphics (Graphics: Viewperf2020; snx composite 1080p, energy composite 4k)

Source: NVIDIA L40S Product Deck, October 2023