If you’ve been searching for QLoRA Explained: Quantized Fine-Tuning for Huge LLMs, you’ve likely hit the same wall most practitioners do: fine-tuning a 70-billion parameter model requires hardware that most teams simply don’t have. QLoRA — introduced in the 2023 paper “QLoRA: Efficient Finetuning of Quantized LLMs” by Dettmers et al. — solves this by combining two techniques that individually are useful but together are transformative. The result: you can fine-tune a 65B model on a single 48GB GPU, or a 7B model on a consumer-grade GPU with 24GB VRAM.
This tutorial walks through exactly how QLoRA works, then gives you production-ready code to fine-tune a model on your own dataset.
Here’s the minimal QLoRA setup you’ll build toward — load a quantized model and attach trainable adapters in under 20 lines:
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Meta-Llama-3-8B",
quantization_config=bnb_config,
device_map="auto",
)
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM")
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~40M || all params: ~8B || trainable%: ~0.50%The rest of this guide explains every parameter, covers the full training loop, and shows you how to merge and deploy the result.
What Makes QLoRA Different from Standard Fine-Tuning
To understand QLoRA, you need to understand the problem it solves. Standard full fine-tuning loads every weight into GPU memory in 32-bit or 16-bit floating point. For a 7B parameter model, that’s roughly 14GB in float16 — and that’s before you add the optimizer states, gradients, and activations needed during training.
LoRA (Low-Rank Adaptation) — covered in the LoRA Paper Explained article — addresses the weight update problem by freezing the original weights and injecting small trainable rank decomposition matrices. But LoRA still loads the base model at full precision.
QLoRA takes LoRA further by:
- 4-bit NormalFloat quantization (NF4) — A new data type that stores weights in 4 bits using a distribution optimally suited for normally distributed neural network weights. This reduces base model memory by ~75% compared to
float16. - Double quantization — Quantizes the quantization constants themselves, saving an additional ~0.37 bits per parameter.
- Paged optimizers — Uses NVIDIA unified memory to page optimizer states between GPU and CPU RAM, preventing out-of-memory crashes during long training runs.
- LoRA adapters trained in
bfloat16— The small adapter matrices remain in high precision so gradients don’t degrade.
flowchart TD
A[Base Model Weights\nfloat16 / 32GB] --> B[4-bit NF4 Quantization\n~8GB on GPU]
B --> C[Frozen Quantized Base]
C --> D[Forward Pass\nDequantize on the fly → bfloat16]
D --> E[LoRA Adapter Layers\nbfloat16, trainable]
E --> F[Loss Computation]
F --> G[Backprop through adapters only]
G --> H[Paged AdamW Optimizer\nStates paged to CPU if needed]
H --> EThe key insight: during the forward pass, weights are dequantized block-by-block from NF4 back to bfloat16 just before the matrix multiply, then discarded. Only the tiny LoRA adapter weights are updated. This gives near full-precision adapter training at a fraction of the memory cost.
Setting Up Your Environment
You’ll need the bitsandbytes, peft, transformers, and trl libraries. The trl library provides the SFTTrainer class which integrates with QLoRA cleanly.
pip install transformers==4.40.0 \
bitsandbytes==0.43.0 \
peft==0.10.0 \
trl==0.8.6 \
accelerate==0.29.3 \
datasets==2.19.0 \
torch==2.2.2Verify your GPU is visible and check VRAM:
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")Core Concepts: NF4 Quantization and LoRA Config
Before writing training code, it’s worth understanding the two configuration objects you’ll use everywhere.
BitsAndBytesConfig — Controlling Quantization
from transformers import BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True, # Enable 4-bit loading
bnb_4bit_quant_type="nf4", # Use NormalFloat4 (optimal for LLM weights)
bnb_4bit_compute_dtype=torch.bfloat16, # Dequantize to bfloat16 for compute
bnb_4bit_use_double_quant=True, # Double quantization for extra savings
)The bnb_4bit_compute_dtype is separate from storage dtype. Weights are stored in NF4 but dequantized to bfloat16 during the matrix multiplication. Using bfloat16 rather than float16 reduces the risk of overflow in accumulation.
LoraConfig — Defining Adapter Architecture
from peft import LoraConfig, TaskType
lora_config = LoraConfig(
r=16, # Rank: controls adapter size. Higher = more capacity
lora_alpha=32, # Scaling factor (effective lr = alpha/r)
target_modules=[ # Which weight matrices to attach adapters to
"q_proj", "k_proj",
"v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"
],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
)r (rank) is the most impactful hyperparameter. A rank of 8–64 covers most use cases. Higher rank means more trainable parameters — a rank-16 adapter on a 7B model adds roughly 40M trainable parameters versus ~7B frozen ones.
target_modules determines which projections get adapters. For maximum quality, target all attention and MLP projections. For faster training, target only q_proj and v_proj.
Full QLoRA Fine-Tuning Implementation
This example fine-tunes meta-llama/Llama-3-8B on a simple instruction dataset. You can swap the model and dataset for your use case.
import torch
from datasets import load_dataset
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
# ── 1. Configuration ─────────────────────────────────────────────────────────
MODEL_ID = "meta-llama/Meta-Llama-3-8B"
DATASET_NAME = "HuggingFaceH4/ultrachat_200k"
OUTPUT_DIR = "./qlora-llama3-8b"
MAX_SEQ_LEN = 1024
# ── 2. Quantization config ────────────────────────────────────────────────────
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
# ── 3. Load tokenizer ─────────────────────────────────────────────────────────
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right" # Prevent warning with batch training
# ── 4. Load model in 4-bit ────────────────────────────────────────────────────
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
quantization_config=bnb_config,
device_map="auto", # Automatically split across available GPUs
trust_remote_code=True,
torch_dtype=torch.bfloat16,
)
model.config.use_cache = False # Required during training
model.config.pretraining_tp = 1 # Disable tensor parallelism for training
# ── 5. Prepare for k-bit training (enables gradient checkpointing + cast) ─────
model = prepare_model_for_kbit_training(model)
# ── 6. Inject LoRA adapters ───────────────────────────────────────────────────
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Expected output: trainable params: ~40M || all params: ~8B || trainable%: ~0.5%
# ── 7. Dataset preparation ────────────────────────────────────────────────────
dataset = load_dataset(DATASET_NAME, split="train_sft[:5000]")
def format_chat(example):
"""Convert dataset messages to a single training string."""
messages = example["messages"]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=False,
)
return {"text": text}
dataset = dataset.map(format_chat, remove_columns=dataset.column_names)
# ── 8. Training arguments ─────────────────────────────────────────────────────
training_args = TrainingArguments(
output_dir=OUTPUT_DIR,
num_train_epochs=1,
per_device_train_batch_size=2,
gradient_accumulation_steps=4, # Effective batch size = 2 * 4 = 8
gradient_checkpointing=True, # Trade compute for memory
optim="paged_adamw_32bit", # Paged optimizer — prevents OOM
save_steps=100,
logging_steps=25,
learning_rate=2e-4,
weight_decay=0.001,
fp16=False,
bf16=True, # Use bfloat16 for training
max_grad_norm=0.3,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
report_to="none",
)
# ── 9. Train ──────────────────────────────────────────────────────────────────
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
dataset_text_field="text",
tokenizer=tokenizer,
max_seq_length=MAX_SEQ_LEN,
packing=False,
)
trainer.train()
trainer.save_model(OUTPUT_DIR)
print(f"Model saved to {OUTPUT_DIR}")Loading and Merging the Adapter After Training
The trained artifact is only the adapter weights (~80MB), not the full model. To deploy, you can either load base + adapter at inference time, or merge them into a single model:
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
BASE_MODEL = "meta-llama/Meta-Llama-3-8B"
ADAPTER_PATH = "./qlora-llama3-8b"
MERGED_PATH = "./qlora-llama3-8b-merged"
# Load base in bfloat16 (full precision for merging)
base_model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Load and merge adapter
model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
model = model.merge_and_unload() # Fuses adapter weights into base
model.save_pretrained(MERGED_PATH)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.save_pretrained(MERGED_PATH)
print("Merged model saved. Ready for deployment or quantization.")The merged model can then be quantized with llama.cpp or GGUF tools for CPU/edge deployment — a common pattern when building privacy-sensitive applications like those explored in OpenJarvis Use Cases: Local AI Agents for Privacy-Sensitive Tasks.
Practical Tips for Production QLoRA Training
Rank selection: Start with r=8 for quick experiments, r=16 for production. Ranks above 64 rarely improve quality and significantly increase adapter size and training time.
Learning rate: 2e-4 works well for most instruction fine-tuning tasks. If the loss diverges early, reduce to 1e-4. If convergence is slow, try 3e-4.
Gradient checkpointing: Always enable during training (gradient_checkpointing=True). It recomputes activations during backward pass instead of storing them, cutting activation memory by ~60%.
Multi-GPU training: QLoRA works with accelerate + device_map="auto". For a 70B model, you’ll want at least 2× A100 80GB or 4× A40 48GB. The paged_adamw optimizer pages states to CPU RAM, so ensure your system RAM is at least 2× GPU VRAM.
Monitoring training quality: Log gradient norms alongside loss. Exploding gradients (norm > 10) indicate learning rate is too high. Flat gradients indicate the model is not learning — increase rank or check your dataset format.
QLoRA-fine-tuned models integrate naturally into agent architectures. When building pipelines that need specialized domain models — like the multi-agent workflows in Getting Started with CrewAI: Multi-Agent Workflows in Python — you can deploy a fine-tuned QLoRA model as a specialized worker agent alongside general-purpose models, splitting tasks by domain expertise.
Frequently Asked Questions
What is the difference between QLoRA and LoRA?
LoRA freezes the base model weights and adds trainable low-rank adapter matrices, but the base model is still loaded in float16 or bfloat16. QLoRA extends this by also quantizing the frozen base model to 4-bit NF4 format, dramatically reducing memory usage. The adapter matrices themselves are still trained in bfloat16. The result is that QLoRA achieves nearly the same fine-tuning quality as LoRA but at roughly one-quarter the GPU memory cost.
How much GPU memory does QLoRA actually require?
A 7B model requires approximately 5–6 GB VRAM for the 4-bit weights, plus 2–3 GB for activations and adapter gradients, totaling around 8–10 GB. A 13B model needs roughly 10–14 GB. A 70B model needs approximately 40–48 GB, fitting on a single A100 80GB. These are estimates — actual usage depends on batch size, sequence length, and whether gradient checkpointing is enabled.
Can I use QLoRA for models other than LLaMA?
Yes. QLoRA is model-agnostic and works with any architecture supported by bitsandbytes and peft. This includes Mistral, Mixtral, Falcon, Gemma, Phi, and most Hugging Face-hosted models. The target_modules list will differ by architecture — check the model’s config for the correct attention and MLP projection names.
Should I merge the adapter before deployment?
It depends. If you need the smallest possible artifact or want to swap adapters dynamically (e.g., different fine-tunes for different tasks), keep them separate. PeftModel.from_pretrained loads them together at inference time with negligible overhead. If you need maximum inference throughput or want to quantize with GGUF tools for CPU deployment, merge first — merging eliminates the LoRA overhead during the forward pass.
How does QLoRA affect inference quality compared to full fine-tuning?
The original paper reports that QLoRA with rank-64 adapters on LLaMA-65B achieves performance within 1–2% of full 16-bit fine-tuning on most benchmarks. For most practical applications — instruction following, domain adaptation, style transfer — the quality difference is imperceptible. The main trade-off is that very task-specific behaviors requiring deep weight modification may benefit from full fine-tuning, but such cases are rare.