Fine-Tuning Large Language Models with Python and Hugging Face
Pre-trained large language models are impressive out of the box, but they get much better when you fine-tune them for a specific domain or task. If you need a model that understands medical terminology, writes code in a particular style, or classifies customer support tickets, fine-tuning closes the gap between a general-purpose model and one that actually fits your needs.
I will walk through fine-tuning an LLM using Hugging Face Transformers, the PEFT library for parameter-efficient fine-tuning with LoRA, and the datasets library for data preparation.
Why Fine-Tune Instead of Prompt Engineering?
Prompt engineering works well for many tasks, but it has clear limitations. You are constrained by context window size, every inference call carries the cost of your lengthy prompt, and some behaviors simply cannot be reliably coaxed through prompting alone. Fine-tuning bakes the knowledge directly into the model weights, resulting in faster inference, more consistent outputs, and better performance on specialized tasks.
Setting Up the Environment
Start by installing the necessary libraries:
pip install transformers datasets peft accelerate bitsandbytes trl torch
Make sure you have a CUDA-capable GPU available. Fine-tuning even small models on CPU is prohibitively slow.
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"Device: {torch.cuda.get_device_name(0)}")
Preparing Your Dataset
The Hugging Face datasets library makes it simple to load and prepare training data. For this example, we will fine-tune a model for instruction-following using a subset of a publicly available dataset:
from datasets import load_dataset
dataset = load_dataset("databricks/databricks-dolly-15k", split="train")
print(f"Dataset size: {len(dataset)}")
print(f"Columns: {dataset.column_names}")
print(dataset[0])
For instruction fine-tuning, you need to format each example into a consistent prompt template. This is critical — the model learns the pattern during training, and you must use the same pattern during inference.
def format_instruction(example):
"""Format a single example into an instruction-response pair."""
if example["context"]:
prompt = (
f"### Instruction:\n{example['instruction']}\n\n"
f"### Context:\n{example['context']}\n\n"
f"### Response:\n{example['response']}"
)
else:
prompt = (
f"### Instruction:\n{example['instruction']}\n\n"
f"### Response:\n{example['response']}"
)
return {"text": prompt}
formatted_dataset = dataset.map(format_instruction)
Split the data for training and validation:
split_dataset = formatted_dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]
print(f"Training examples: {len(train_dataset)}")
print(f"Validation examples: {len(eval_dataset)}")
Loading the Base Model with Quantization
Loading a full-precision LLM requires enormous GPU memory. 4-bit quantization via bitsandbytes dramatically reduces the memory footprint while preserving most of the model's capability:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_name = "meta-llama/Llama-2-7b-hf"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
The nf4 quantization type is specifically designed for normally distributed weights, which is typical in transformer models. Double quantization (bnb_4bit_use_double_quant) further reduces memory by quantizing the quantization constants themselves.
Configuring LoRA with PEFT
LoRA (Low-Rank Adaptation) is the key technique that makes fine-tuning practical. Instead of updating all model parameters (which would require enormous compute), LoRA freezes the original weights and injects small trainable rank-decomposition matrices into specific layers:
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
The output will show something like: trainable params: 33,554,432 || all params: 6,771,970,048 || trainable%: 0.4956. You are training less than 0.5% of the total parameters, which is what makes this approach so efficient.
Key LoRA hyperparameters to understand:
- r (rank): Controls the size of the low-rank matrices. Higher values capture more information but increase compute. 8-64 is typical.
- lora_alpha: A scaling factor, usually set to 2x the rank.
- target_modules: Which layers to apply LoRA to. Targeting all linear layers in the attention and MLP blocks gives the best results.
Training with the SFTTrainer
The trl library's SFTTrainer simplifies supervised fine-tuning:
from transformers import TrainingArguments
from trl import SFTTrainer
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
weight_decay=0.01,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
logging_steps=25,
save_strategy="epoch",
evaluation_strategy="epoch",
fp16=True,
optim="paged_adamw_8bit",
report_to="none",
)
trainer = SFTTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
dataset_text_field="text",
max_seq_length=1024,
tokenizer=tokenizer,
args=training_args,
packing=False,
)
trainer.train()
A few important configuration choices here:
- paged_adamw_8bit: An optimizer that pages optimizer states to CPU when GPU memory is scarce, preventing out-of-memory errors.
- gradient_accumulation_steps: Simulates a larger batch size without the memory cost. Effective batch size = per_device_batch_size * gradient_accumulation_steps.
- cosine scheduler with warmup: Starts with a linear warmup, then gradually decreases the learning rate following a cosine curve. This works well for fine-tuning.
Saving and Loading the Fine-Tuned Model
After training, save the LoRA adapter weights:
trainer.model.save_pretrained("./fine-tuned-adapter")
tokenizer.save_pretrained("./fine-tuned-adapter")
To load and use the fine-tuned model later:
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
)
fine_tuned_model = PeftModel.from_pretrained(base_model, "./fine-tuned-adapter")
def generate_response(instruction, max_length=512):
prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = fine_tuned_model.generate(
**inputs,
max_new_tokens=max_length,
temperature=0.7,
top_p=0.9,
do_sample=True,
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
return response.split("### Response:\n")[-1].strip()
print(generate_response("Explain the difference between TCP and UDP."))
Evaluating Your Fine-Tuned Model
Always evaluate systematically. Compare outputs before and after fine-tuning on a held-out test set:
from tqdm import tqdm
def evaluate_model(model, eval_data, num_samples=50):
results = []
for example in tqdm(eval_data.select(range(num_samples))):
prediction = generate_response(example["instruction"])
results.append({
"instruction": example["instruction"],
"expected": example["response"],
"predicted": prediction,
})
return results
For quantitative evaluation, consider metrics like ROUGE scores for summarization tasks, exact match for classification, or BLEU scores for translation. For instruction-following, human evaluation or LLM-as-judge approaches often provide the most meaningful signal.
Conclusion
Fine-tuning LLMs has become much more accessible thanks to PEFT techniques like LoRA and the Hugging Face ecosystem. With 4-bit quantization and LoRA, you can fine-tune a 7B parameter model on a single consumer GPU. The key ingredients are a well-formatted dataset, sensible LoRA configuration, and careful training hyperparameters. Start with a small experiment, evaluate rigorously, and iterate.