Fine-Tuning Llama 3 with LoRA & Unsloth

Parameter-efficient fine-tuning (QLoRA) on custom domain JSONL datasets with 4-bit quantization, GPU VRAM optimization, and GGUF export for local deployment.

Python 3.10+ (PyTorch) LLMOps Intermediate ⏱️ 20 min lab • 6 Steps End-to-End

CUDA Setup & Unsloth Package Installation

Unsloth accelerates fine-tuning by 2x while consuming 70% less VRAM compared to standard HuggingFace PEFT models.

Terminal - Install Unsloth & PyTorch

pip install --no-deps "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install torch trl peft datasets bitsandbytes

PYTHON • check_cuda.py

import torch

print(f"PyTorch Version: {torch.__version__}")
print(f"CUDA Available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"Device Name: {torch.cuda.get_device_name(0)}")

Formatting Custom JSONL Instruction Dataset in Python

Convert enterprise Q&A dataset into Llama 3 ChatML / Alpaca prompt template format using HuggingFace datasets.

PYTHON • prepare_dataset.py

from datasets import Dataset

prompt_style = """Below is an instruction that describes a task, paired with an input context.

### Instruction:
{}

### Input:
{}

### Response:
{}"""

def format_prompts(examples):
    instructions = examples["instruction"]
    inputs       = examples["input"]
    outputs      = examples["output"]
    texts = []
    for instr, inp, out in zip(instructions, inputs, outputs):
        text = prompt_style.format(instr, inp, out)
        texts.append(text)
    return { "text" : texts }

dataset_data = {
    "instruction": ["Classify security alert", "Generate Python script"],
    "input": ["Unauthorized SSH attempt", "Parse CSV log"],
    "output": ["SEVERITY: HIGH", "import csv..."]
}

dataset = Dataset.from_dict(dataset_data)
formatted_dataset = dataset.map(format_prompts, batched=True)

Loading 4-Bit Quantized Llama 3 Base Model

Load unsloth/llama-3-8b-Instruct-bnb-4bit into 4-bit precision to fit within consumer GPU VRAM limits (under 8GB VRAM).

PYTHON • load_model.py

from unsloth import FastLanguageModel

def get_base_model():
    max_seq_length = 2048
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name = "unsloth/llama-3-8b-Instruct-bnb-4bit",
        max_seq_length = max_seq_length,
        load_in_4bit = True,
    )
    return model, tokenizer, max_seq_length

Injecting Low-Rank Adaptation (LoRA) Target Modules

Attach trainable LoRA rank matrices to query, key, value, and projection linear layers.

PYTHON • lora_setup.py

from unsloth import FastLanguageModel

def apply_lora_adapters(model):
    model = FastLanguageModel.get_peft_model(
        model,
        r = 16,
        target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                          "gate_proj", "up_proj", "down_proj"],
        lora_alpha = 16,
        lora_dropout = 0,
        bias = "none",
        random_state = 3407,
    )
    return model

Configuring SFTTrainer & Execution Training Loop

Set up HuggingFace SFTTrainer for supervised fine-tuning with cosine learning rate schedule.

PYTHON • train_llama.py

from trl import SFTTrainer
from transformers import TrainingArguments

def train_model(model, tokenizer, formatted_dataset, max_seq_length):
    trainer = SFTTrainer(
        model = model,
        tokenizer = tokenizer,
        train_dataset = formatted_dataset,
        dataset_text_field = "text",
        max_seq_length = max_seq_length,
        args = TrainingArguments(
            per_device_train_batch_size = 2,
            gradient_accumulation_steps = 4,
            warmup_steps = 5,
            max_steps = 60,
            learning_rate = 2e-4,
            optim = "adamw_8bit",
            output_dir = "outputs",
        ),
    )
    return trainer.train()

Exporting 4-Bit GGUF & Local Inference Test Script

Export fine-tuned weights directly into 4-bit GGUF format for execution in Ollama or llama.cpp.

PYTHON • export_and_infer.py

def export_to_gguf(model, tokenizer):
    model.save_pretrained_gguf("llama3_echo_custom", tokenizer, quantization_method = "q4_k_m")
    print("Successfully exported GGUF model: llama3_echo_custom-unsloth.Q4_K_M.gguf")

Expected Training Output

[INFO] Unsloth: Fast Llama 3 4-bit loaded successfully.
Step 60/60 - Loss: 0.142 - Training Complete in 42.1s
Successfully exported GGUF model: llama3_echo_custom-unsloth.Q4_K_M.gguf