{"id":1026,"date":"2026-06-03T08:51:10","date_gmt":"2026-06-03T00:51:10","guid":{"rendered":"https:\/\/connectword.dpdns.org\/?p=1026"},"modified":"2026-06-03T08:51:10","modified_gmt":"2026-06-03T00:51:10","slug":"how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-tutorial-on-google-colab","status":"publish","type":"post","link":"https:\/\/connectword.dpdns.org\/?p=1026","title":{"rendered":"How to Fine-Tune LFM2 Using QLoRA and DPO: A Complete Step-by-Step Coding Tutorial on Google Colab"},"content":{"rendered":"

In this tutorial, we fine-tune Liquid AI\u2019s LFM2<\/strong><\/a> model through a complete open-source workflow. We start by loading the base LFM2 checkpoint with QLoRA, preparing a chat-style supervised fine-tuning dataset, training a lightweight LoRA adapter using TRL and PEFT, and then merging the adapter back into the model. We also extend the workflow with DPO to show how we can improve response preference using chosen and rejected answers. At the end, we have a practical pipeline that moves from a base LFM2 model to an SFT-tuned, preference-aligned checkpoint, ready for further testing or deployment.<\/p>\n

\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
!pip install -q -U \"transformers>=4.55\" \"trl>=0.12\" \"peft>=0.13\" \"datasets>=2.20\" \"accelerate>=0.34\" bitsandbytes\n\n\nimport torch, gc\nfrom datasets import load_dataset, Dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\nfrom peft import LoraConfig, PeftModel, prepare_model_for_kbit_training\nfrom trl import SFTConfig, SFTTrainer, DPOConfig, DPOTrainer\n\n\nMODEL_ID    = \"LiquidAI\/LFM2-1.2B\"\nUSE_4BIT    = True\nRUN_DPO     = True\nSFT_SAMPLES = 500\nSFT_STEPS   = 60\nDPO_STEPS   = 40\nMAX_LEN     = 1024\n\n\nBF16 = torch.cuda.is_available() and torch.cuda.is_bf16_supported()\nDTYPE = torch.bfloat16 if BF16 else torch.float16\nassert torch.cuda.is_available(), \"No GPU detected \u2014 set Runtime > Change runtime type > GPU\"\nprint(f\"GPU: {torch.cuda.get_device_name(0)} | dtype={DTYPE} | 4bit={USE_4BIT}\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We install all the required libraries for fine-tuning LFM2 inside Google Colab. We import the core tools from Transformers, TRL, PEFT, datasets, bitsandbytes, and PyTorch. We also define the main training settings, detect available GPUs, and select the appropriate precision for efficient training.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
def load_base(four_bit: bool):\n   quant_cfg = None\n   if four_bit:\n       quant_cfg = BitsAndBytesConfig(\n           load_in_4bit=True,\n           bnb_4bit_quant_type=\"nf4\",\n           bnb_4bit_use_double_quant=True,\n           bnb_4bit_compute_dtype=DTYPE,\n       )\n   model = AutoModelForCausalLM.from_pretrained(\n       MODEL_ID,\n       device_map=\"auto\",\n       dtype=DTYPE,\n       quantization_config=quant_cfg,\n   )\n   model.config.use_cache = False\n   return model\n\n\ntokenizer = AutoTokenizer.from_pretrained(MODEL_ID)\nif tokenizer.pad_token is None:\n   tokenizer.pad_token = tokenizer.eos_token\n\n\nmodel = load_base(USE_4BIT)\n\n\n@torch.no_grad()\ndef chat(m, user_msg, system=None, max_new_tokens=200):\n   msgs = ([{\"role\": \"system\", \"content\": system}] if system else []) + \n          [{\"role\": \"user\", \"content\": user_msg}]\n   inputs = tokenizer.apply_chat_template(\n       msgs,\n       add_generation_prompt=True,\n       return_tensors=\"pt\",\n       tokenize=True,\n       return_dict=True,\n   ).to(m.device)\n   m.config.use_cache = True\n   out = m.generate(\n       **inputs,\n       max_new_tokens=max_new_tokens, do_sample=True,\n       temperature=0.3, min_p=0.15, repetition_penalty=1.05,\n       pad_token_id=tokenizer.pad_token_id,\n   )\n   m.config.use_cache = False\n   prompt_len = inputs[\"input_ids\"].shape[-1]\n   return tokenizer.decode(out[0, prompt_len:], skip_special_tokens=True)\n\n\nPROBE = \"Explain what makes the LFM2 architecture good for on-device AI, in 2 sentences.\"\nprint(\"n=== BASELINE (before fine-tuning) ===n\", chat(model, PROBE))<\/code><\/pre>\n<\/div>\n<\/div>\n
We load the LFM2 base model with optional 4-bit quantization to reduce GPU memory usage. We prepare the tokenizer, set the padding token, and define a chat function for testing model responses. We then run a baseline prompt to compare the model\u2019s behavior before and after fine-tuning.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
sft_ds = load_dataset(\"HuggingFaceTB\/smoltalk\", \"all\", split=f\"train[:{SFT_SAMPLES}]\")\nsft_ds = sft_ds.select_columns([\"messages\"])\nprint(\"nSFT example messages:\", sft_ds[0][\"messages\"][:2])\n\n\nlora_sft = LoraConfig(\n   r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n   task_type=\"CAUSAL_LM\", target_modules=\"all-linear\",\n)\n\n\nsft_cfg = SFTConfig(\n   output_dir=\"outputs\/sft\/lfm2_demo\",\n   max_length=MAX_LEN,\n   per_device_train_batch_size=2,\n   gradient_accumulation_steps=4,\n   learning_rate=2e-5,\n   warmup_ratio=0.03,\n   lr_scheduler_type=\"cosine\",\n   max_steps=SFT_STEPS,\n   logging_steps=10,\n   save_strategy=\"no\",\n   gradient_checkpointing=True,\n   gradient_checkpointing_kwargs={\"use_reentrant\": False},\n   bf16=BF16, fp16=not BF16,\n   optim=\"paged_adamw_8bit\" if USE_4BIT else \"adamw_torch\",\n   packing=False,\n   report_to=\"none\",\n)\n\n\nsft_trainer = SFTTrainer(\n   model=model,\n   args=sft_cfg,\n   train_dataset=sft_ds,\n   peft_config=lora_sft,\n   processing_class=tokenizer,\n)\nsft_trainer.train()\nsft_trainer.save_model(\"outputs\/sft\/lfm2_adapter\")\nprint(\"n=== AFTER SFT ===n\", chat(sft_trainer.model, PROBE))<\/code><\/pre>\n<\/div>\n<\/div>\n
We load a chat-formatted supervised fine-tuning dataset and keep only the messages column. We configure LoRA for lightweight adapter-based training and define the SFT training settings. We then train the model with SFT, save the LoRA adapter, and test the improved model response.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
del sft_trainer, model\ngc.collect(); torch.cuda.empty_cache()\n\n\nbase_fp16 = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\", dtype=DTYPE)\nsft_merged = PeftModel.from_pretrained(base_fp16, \"outputs\/sft\/lfm2_adapter\").merge_and_unload()\nsft_merged.save_pretrained(\"outputs\/sft\/lfm2_merged\")\ntokenizer.save_pretrained(\"outputs\/sft\/lfm2_merged\")\nprint(\"Merged SFT model saved -> outputs\/sft\/lfm2_merged\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We clear the earlier training objects from memory to free GPU resources. We reload the base LFM2 model in fp16 or bf16 and attach the trained SFT LoRA adapter. We then merge the adapter into the base model and save the merged SFT checkpoint for the next stage.<\/p>\n
\n
\n
\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
Copy Code<\/span>Copied<\/span>Use a different Browser<\/span><\/a><\/div>\n<\/div>\n
if RUN_DPO:\n   pref_rows = [\n       {\"prompt\":  [{\"role\": \"user\", \"content\": \"Reply to a customer whose order is late.\"}],\n        \"chosen\":  [{\"role\": \"assistant\", \"content\": \"I'm sorry your order is delayed. I've checked your tracking and it will arrive within 2 days \u2014 here's a 10% credit for the inconvenience.\"}],\n        \"rejected\":[{\"role\": \"assistant\", \"content\": \"Orders are sometimes late. Please wait.\"}]},\n       {\"prompt\":  [{\"role\": \"user\", \"content\": \"Summarize the benefit of edge AI in one line.\"}],\n        \"chosen\":  [{\"role\": \"assistant\", \"content\": \"Edge AI runs models locally, giving low latency, offline reliability, and stronger privacy.\"}],\n        \"rejected\":[{\"role\": \"assistant\", \"content\": \"Edge AI is AI on the edge of things and it is good.\"}]},\n       {\"prompt\":  [{\"role\": \"user\", \"content\": \"Decline a meeting politely.\"}],\n        \"chosen\":  [{\"role\": \"assistant\", \"content\": \"Thanks for the invite \u2014 I have a conflict then. Could we find another slot this week?\"}],\n        \"rejected\":[{\"role\": \"assistant\", \"content\": \"No.\"}]},\n   ] * 20\n   pref_ds = Dataset.from_list(pref_rows)\n\n\n   lora_dpo = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n                         task_type=\"CAUSAL_LM\", target_modules=\"all-linear\")\n   dpo_cfg = DPOConfig(\n       output_dir=\"outputs\/dpo\/lfm2_demo\",\n       per_device_train_batch_size=1,\n       gradient_accumulation_steps=4,\n       learning_rate=5e-6,\n       beta=0.1,\n       max_length=MAX_LEN,\n       max_prompt_length=512,\n       max_steps=DPO_STEPS,\n       logging_steps=10,\n       save_strategy=\"no\",\n       gradient_checkpointing=True,\n       gradient_checkpointing_kwargs={\"use_reentrant\": False},\n       bf16=BF16, fp16=not BF16,\n       report_to=\"none\",\n   )\n   dpo_trainer = DPOTrainer(\n       model=sft_merged,\n       ref_model=None,\n       args=dpo_cfg,\n       train_dataset=pref_ds,\n       processing_class=tokenizer,\n       peft_config=lora_dpo,\n   )\n   dpo_trainer.train()\n   final = dpo_trainer.model.merge_and_unload()\n   final.save_pretrained(\"outputs\/final\/lfm2_sft_dpo\")\n   tokenizer.save_pretrained(\"outputs\/final\/lfm2_sft_dpo\")\n   print(\"n=== AFTER SFT + DPO ===n\", chat(dpo_trainer.model, PROBE))\n   print(\"Final model saved -> outputs\/final\/lfm2_sft_dpo\")\n\n\nprint(\"nDone. Compare the BASELINE vs AFTER-SFT(+DPO) outputs above.\")<\/code><\/pre>\n<\/div>\n<\/div>\n
We optionally run DPO using prompt-chosen-and-rejected response pairs. We configure another LoRA adapter for preference tuning and train the SFT-merged model with DPO. We finally merge the DPO adapter, save the final model checkpoint, and compare the result against earlier outputs.<\/p>\n
In conclusion, we built a full fine-tuning pipeline for LFM2 using only open-source tools, including Transformers, TRL, PEFT, datasets, and bitsandbytes. We used QLoRA to make training efficient on Colab GPUs, applied supervised fine-tuning to chat-formatted data, merged the trained adapter into the base model, and optionally further improved the model through DPO. It gives us a clear view of how modern LLM fine-tuning works in practice, from loading the model to producing a final checkpoint that can be compared against the original baseline and prepared for deployment.<\/p>\n
\n
\n<\/p>
\n<\/p>
Check out\u00a0the\u00a0Codes with Notebook here<\/a>.\u00a0<\/strong>Also,\u00a0feel free to follow us on\u00a0Twitter<\/mark><\/a><\/strong>\u00a0and don\u2019t forget to join our\u00a0150k+ ML SubReddit<\/a><\/strong>\u00a0and Subscribe to\u00a0our Newsletter<\/a><\/strong>. Wait! are you on telegram?\u00a0now you can join us on telegram as well.<\/a><\/strong><\/p>\n
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.?\u00a0Connect with us<\/mark><\/a><\/strong><\/p>\n
The post How to Fine-Tune LFM2 Using QLoRA and DPO: A Complete Step-by-Step Coding Tutorial on Google Colab<\/a> appeared first on MarkTechPost<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
In this tutorial, we fine-tune…<\/p>\n","protected":false},"author":1,"featured_media":29,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1026","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/1026","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1026"}],"version-history":[{"count":0,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/posts\/1026\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=\/wp\/v2\/media\/29"}],"wp:attachment":[{"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1026"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1026"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/connectword.dpdns.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1026"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}