Fine-Tuning an Open Model on a Single GPU

Fine-tuning used to sound like a data-centre project. Then came LoRA — low-rank adaptation — which freezes the original model and trains only a tiny set of extra parameters. A 7 billion parameter model can be adapted on a single consumer GPU in hours.

When to fine-tune (and when not to)

Fine-tuning changes style and behaviour, not knowledge. Use it to teach a model your tone, your output format, or a specialised task like summarising legal clauses. It cannot reliably inject new facts — that is what RAG is for.

If you only need a few examples, try few-shot prompting first. Fine-tuning pays off when you have hundreds or thousands of examples and a consistent task.

Prepare the data

Collect pairs of input and ideal output. Quality beats quantity: 500 clean, consistent examples usually beat 5,000 noisy ones. Format matters more than size.

[{"instruction": "Summarize the meeting notes",
  "input": "...",
  "output": "..."}]

Consistency is the secret. If some outputs are bullet points and others are prose, the model will mix them. Decide the format first, then write every example in it.

Choose a base model

Pick an open-weight model in the 7B–14B range: LLaMA, Mistral or Qwen families are solid starting points. Smaller models fine-tune faster and deploy on less hardware; larger models generalise better on harder tasks. Start small and scale only if quality demands it.

Configure LoRA training

With the peft library, LoRA configuration is a few lines:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,            # rank — bigger captures more, costs more
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()  # typically ~1-2% of weights

Only 1–2% of parameters are trainable. That is the entire trick: the model keeps its learned knowledge, and you add a lightweight adaptation layer on top.

Train on one GPU

Use 4-bit quantisation to fit the base model into memory, batch size 1–2 with gradient accumulation, and a low learning rate around 2e-4. Watch the loss on a held-out validation split. Two to four epochs is a common sweet spot; more epochs can overfit your style until it sounds like a parody of itself.

Evaluate, then deploy

Hold back 10% of examples the model never trained on. Compare its outputs against your ideal outputs on a few dimensions: format compliance, tone, factual accuracy. Run the fine-tuned model side by side with the base model on the same prompts.

When quality looks right, merge the LoRA weights into the base model and export. You now have a compact, deployable model that writes in your voice.

Fine-tuning with LoRA is the closest thing to a free lunch in applied AI: a few hours of training, a tiny model delta, and a dramatically more consistent assistant.