Fine-Tuning LLMs

Fine-tuning adapts a pre-trained LLM to your specific task or domain by training it further on your own data. But fine-tuning is not always the right answer. This lesson covers the decision framework, parameter-efficient methods like LoRA, dataset preparation, and practical training with Hugging Face.

When to Use Each Approach

Full Fine-Tuning vs Parameter-Efficient Methods

Full Fine-Tuning

Updates all model parameters. For a 7B parameter model, this requires:

Full fine-tuning is rarely practical for most teams.

Parameter-Efficient Fine-Tuning (PEFT)

Only updates a small subset of parameters while freezing the rest. This dramatically reduces memory, compute, and data requirements.

LoRA: Low-Rank Adaptation

LoRA is the most popular PEFT method. Instead of updating the full weight matrix W, LoRA adds two small matrices (A and B) that approximate the weight update.

How LoRA Works

Original: y = Wx
LoRA:     y = Wx + BAx
Where:
  W is the frozen original weight matrix (e.g., 4096 x 4096)
  A is a small matrix (4096 x r) - initialized randomly
  B is a small matrix (r x 4096) - initialized to zeros
  r is the "rank" (typically 8-64) << 4096

Instead of updating 16.7 million parameters (4096 x 4096), LoRA only trains 524,288 parameters (4096 x 64 + 64 x 4096 with rank=64). That is a 97% reduction in trainable parameters.

Where LoRA Is Applied

LoRA adapters are typically added to the attention weight matrices in each transformer layer:

LoRA Rank Selection

QLoRA: Quantized LoRA

QLoRA combines LoRA with 4-bit quantization of the base model. The frozen base weights are stored in 4-bit precision (NormalFloat4), while the LoRA adapters are trained in 16-bit.

Memory Savings

from transformers import BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",         # NormalFloat4
    bnb_4bit_compute_dtype="bfloat16", # Compute in bf16
    bnb_4bit_use_double_quant=True,     # Double quantization
)

Dataset Preparation

Fine-tuning data must be formatted in the structure the model expects.

Instruction Format (Alpaca-style)

{
  "instruction": "Summarize the following article.",
  "input": "The Federal Reserve announced today that...",
  "output": "The Fed raised interest rates by 0.25% citing..."
}

Chat Format (ChatML / Llama)

{
  "messages": [
    {"role": "system", "content": "You are a medical assistant."},
    {"role": "user", "content": "What are the symptoms of flu?"},
    {"role": "assistant", "content": "Common flu symptoms include fever..."}
  ]
}

Dataset Quality Guidelines

Evaluating Fine-Tuned Models

Perplexity

Perplexity measures how "surprised" the model is by the test data. Lower = better.

import math
def compute_perplexity(model, tokenizer, text):
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        outputs = model(inputs, labels=inputs["input_ids"])
    return math.exp(outputs.loss.item())

Task-Specific Metrics

The gold standard: run both the base model and fine-tuned model on the same inputs, then have humans (or a strong judge model) rate which is better. Report the win rate of the fine-tuned model.

Deploying Fine-Tuned Models

1. Merge adapter into base model (for simpler deployment):

   merged_model = model.merge_and_unload()
   merged_model.save_pretrained("./merged-model")
   

2. Serve with adapter (for multi-tenant, multiple adapters on one base model): - Load base model once - Hot-swap LoRA adapters per request - Tools: vLLM, LoRAX, Hugging Face TGI

3. Quantize for deployment** (GGUF/GPTQ for smaller, faster inference)