How to Run a Hugging Face Model Locally with Transformers

The Hub hosts close to two million models now, and most of them are not worth running locally. Picking one that fits your hardware is 80% of the job; the actual from_pretrained call is four lines.

Direct answer: to run a Hugging Face model locally, install transformers and torch, load with AutoModelForCausalLM.from_pretrained("model-id", torch_dtype="auto", device_map="auto"), then call model.generate() on tokenised input. Rule of thumb: 4-bit quantisation needs roughly 1 GB of VRAM per 1B parameters, 16-bit needs about 2 GB per 1B.

How much VRAM does a model need?

Model size	fp16	8-bit	4-bit	Fits on
1B	~2 GB	~1.2 GB	~0.8 GB	any 4 GB GPU
7B	~14 GB	~8 GB	~4.5 GB	RTX 3060 12 GB at 4-bit
13B	~26 GB	~14 GB	~8 GB	RTX 4090
70B	~140 GB	~70 GB	~40 GB	2× A100 40 GB

These figures exclude the KV cache, which grows with context length. A 7B model at 8K context adds 2-3 GB. Budget 20% headroom or you will hit out-of-memory on the third request rather than the first.

What is the fastest install?

pip install transformers accelerate bitsandbytes torch

accelerate handles device_map="auto", which shards a model across GPUs and spills to CPU RAM when VRAM runs out. That spill is why a 7B model sometimes “fits” on an 8 GB card — and why it then runs at 2 tokens per second.

On Apple Silicon, install torch from the nightly channel and set device_map="mps". It works, though 4-bit quantisation via bitsandbytes is CUDA-only, so Mac users are limited to fp16 and thus smaller models.

How do you load and generate?

from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "Qwen/Qwen2.5-3B-Instruct"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(
    mid, torch_dtype="auto", device_map="auto"
)

msgs = [{"role": "user", "content": "Summarise the CAP theorem in two sentences."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Use apply_chat_template rather than hand-writing prompt strings. Every instruct model on the Hub expects a slightly different format, and getting it wrong produces rambling output that looks like a broken model.

Set do_sample=False for deterministic summaries and classification. Sampling temperature above 0.8 on a 3B model produces confident nonsense.

How do you make it fast?

Three changes, in order of impact. First, load in 4-bit:

from transformers import BitsAndBytesConfig
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype="bfloat16")
model = AutoModelForCausalLM.from_pretrained(mid, quantization_config=bnb, device_map="auto")

Second, batch your requests. A GPU at batch size 1 wastes most of its throughput; batch 8 typically triples tokens per second with no quality change.

Third, cap max_new_tokens. Models will happily generate 2,000 tokens when 150 would do, and generation is linear in output length.

Why is my first request so slow?

The model has to be read from disk into VRAM. For a 7B model that is 14 GB of I/O — 20 to 60 seconds on a normal SSD, minutes on a network mount. Loading happens once per process, so a long-running server pays it once. A script that loads a model per request pays it every time.

That is the argument for text-generation-inference or vLLM in production. Both keep the model resident and add continuous batching. transformers alone is a research tool; it is not a serving layer.

Who should just use an API?

Most people. An API call costs a fraction of a cent and runs on hardware you do not own. Local inference makes sense when: data cannot leave your network, you need a fine-tuned model nobody hosts, you run more than about 2 million tokens a month, or you want no network dependency at all.

Running a 3B model locally to save API costs is a false economy on a laptop — the electricity and time exceed the bill. Running it because patient records cannot go to a third party is a completely different calculation.

Key Takeaways

• Rough VRAM guide: 4-bit ≈ 1 GB per 1B parameters, fp16 ≈ 2 GB per 1B.
• apply_chat_template prevents the prompt-format bugs that make models look broken.
• device_map="auto" spills to CPU when VRAM runs out — it works, but at 2 tokens/sec.
• Batch size 8 typically triples throughput over batch size 1.
• Load once per process; for production serving use vLLM or TGI instead of raw Transformers.