How much VRAM does OLMo 2 13B need?

How the memory is worked out

A local model needs three things to fit in your graphics card’s memory (VRAM):

• The model file. The weights are loaded as they are stored, so this is the download size its page shows. A smaller quantization (Q4 rather than Q8) is a smaller file, at some cost in quality.
• The context. For every token in the conversation the model keeps a key and a value for each layer and key/value head. We read the layers, heads and head width from the publisher’s own config file and count two bytes per number, which is how llama.cpp (and so Ollama and LM Studio) stores them by default. Double the context and this part doubles.
• An allowance of 0.5 GB for the program running the model. This one is our estimate, not a published figure.

Download pages give file sizes in decimal gigabytes (1,000,000,000 bytes), while graphics card and Mac memory is sold in binary ones (1,073,741,824 bytes). We show everything in the binary unit so it compares directly with your card, which is why a “5.2 GB” download reads as 4.8 GB here.

Graphics cards and Macs

A graphics card can give a model all of its VRAM, as the maker lists it. A Mac shares one pool of memory with macOS and your other apps, so we count 75% of it (also our estimate). Cards and Macs with the same memory get the same answer; how fast they run differs, but nobody publishes speed figures we could cite. See our methodology.