This repository contains a script for training Qwen2-VL, Qwen2.5-VL , Qwen3-VL and Qwen3.5 with only using HuggingFace and Liger-Kernel.
liger_kernel==0.8.0. Liger 0.8.0 adds official patches for qwen3_5 / qwen3_5_moe and ships LigerExperts, a fused MoE expert kernel that auto-accelerates qwen3_vl_moe and qwen3_5_moe under --use_liger_kernel True. The 0.7-era hardcoded fallback that force-disabled Liger for Qwen3.5 in SFT/DPO/GRPO has been removed, and the mm_token_type_ids GRPO wrapper is now skipped automatically on Liger ≥ 0.8.0 (kept as a no-op shim for older installs).transformers==5.3.0[!WARNING] Read Training Notes before running any training script. It contains required settings and compatibility notes for
Qwen3.5,QLoRA + vision,QLoRA + liger,DeepSpeed, and video training.
vision_model while using LoRATo simplfy the setting process for training, you could use the provided pre-build environments.
The settings are done in the conda env named train.
You could find more information about the image here.
docker pull john119/vlm
docker run --gpus all -it -v /host/path:/docker/path --name vlm --ipc=host john119/vlm /bin/bash
Install the required packages using environment.yaml.
requirements.txtpip install -r requirements.txt -f https://download.pytorch.org/whl/cu128
pip install qwen-vl-utils
pip install flash-attn --no-build-isolation
environment.yamlconda env create -f environment.yaml
conda activate train
pip install qwen-vl-utils
pip install flash-attn --no-build-isolation
Note: You should install flash-attn after installing the other packages.
Qwen3.5 series: use --disable_flash_attn2 True for now. In local testing, Flash Attention 2 raised CUDA errors while sdpa was stable. This applies to SFT, CLS, DPO, and GRPO.QLoRA + vision: do not combine quantization (--bits 4 / --bits 8) with vision training (--vision_lora True, --freeze_vision_tower False, or --unfreeze_topk_vision > 0). Use --bits 16 if you want to train vision-related modules.QLoRA + liger: disable liger when using QLoRA.DeepSpeed: zero2 is usually faster and often more stable than zero3, but it uses more memory.Video: do not set fps and nframes at the same time.Top-k unfreeze: if you use --unfreeze_topk_llm or --unfreeze_topk_vision, keep the corresponding base module frozen first with --freeze_llm True or --freeze_vision_tower True.Learning rates: vision_model usually works better with a learning rate about 5x to 10x smaller than language_model.The script requires a dataset formatted according to the LLaVA specification. The dataset should be a JSON file where each entry contains information about conversations and images. Ensure that the image paths in the dataset match the provided --image_folder.
When using a multi-image dataset, the image tokens should all be <image>, and the image file names should have been in a list.
Please see the example below and follow format your data.
Example for single image dataset
[
{
"id": "000000033471",
"image": "000000033471.jpg",
"conversations": [
{
"from": "human",
"value": "<image>\nWhat are the colors of the bus in the image?"
},
{
"from": "gpt",
"value": "The bus in the image is white and red."
},
{
"from": "human",
"value": "What feature can be seen on the back of the bus?"
},
{
"from": "gpt",
"value": "The back of the bus features an advertisement."
},
{
"from": "human",
"value": "Is the bus driving down the street or pulled off to the side?"
},
{
"from": "gpt",
"value": "The bus is driving down the street, which is crowded with people and other vehicles."
}
]
}
...
]
Example for multi image dataset
[
{
"id": "000000033471",
"image": ["000000033471.jpg", "000000033472.jpg"],
"conversations": [
{
"from": "human",
"value": "<image>\n<image>\nIs the perspective of the camera differnt?"
},
{
"from": "gpt",
"value": "Yes, It the perspective of the camera is different."
}
]
}
...
]
Example for video dataset
[
{
"id": "sample1",
"video": "sample1.mp4",
"conversations": [
{
"from": "human",
"value": "<video>\nWhat is going on in this video?"
},
{
"from": "gpt",
"value": "A man is walking down the road."
}
]
}
...
]
Example for DPO dataet
[
{
"id": "e4cd017f-3854-4019-9cc1-9dacacb5d424",
"image": "example_0_0.png",
"prompt": "<image>\nDescribe the key features of the image in great detail.",
"chosen": "The image features a modern, multi-colored train parked on a train track, occupying a significant portion of the scene. The train has a unique and striking design, with red, white, and blue colors, as well as yellow accents. It appears to be a high-speed train, indicating its advanced engineering and potential for fast travel.\n\nThe train is situated next to a platform, suggesting that it may be at a train station. The overall scene conveys a sense of transportation and modern technology, with the vibrant and sleek train as the main subject.",
"rejected": "The image features a modern, colorful train parked on a set of train tracks. The train has a striking design, with red, white, and blue colors as well as yellow accents. It appears to be a high-speed train, ready for departure.\n\nThe train spans a significant portion of the image, from the left to the right side. The train tracks can be seen clearly beneath the train, emphasizing its position on the railway line. The overall scene gives off an impression of a contemporary and efficient mode of transportation."
},
{
"id": "5e19e647-e5d3-4bcf-82e9-d262570743ae",
"image": "example_1_0.png",
"prompt": "<image>\nIs this bus in the USA?",
"chosen": "Yes, based on the image, it can be assumed that this bus is in the USA. The location of the bus cannot be accurately determined.",
"rejected": "No, it's not in the USA. The image does not provide specific information on where the bus is located. However, we can say that it's not in the United States."
}
...
]
Example for GRPO dataset
[
{
"id": "06bc8a17-bb1c-4007-8c08-92c41e2628b2",
"image": "image_2.jpg",
"conversations": [
{
"from": "human",
"value": "<image>\nBased on the image, which geometric method is used to determine the bearing angle, and why is it the most appropriate choice?"
},
{
"from": "gpt",
"reasoning": "Let's analyze the image step-by-step. The image shows a right-angled triangle with points B, C, and A. The angle at point B is a right angle, indicating that trigonometric functions can be applied. To find the bearing angle, we need to relate the sides of the triangle. The tangent function is suitable here because it relates the opposite side (BC) to the adjacent side (AB) in a right-angled triangle. By using the tangent function, we can calculate the angle at point A, which is the bearing angle. Therefore, the most appropriate geometric method is the use of trigonometric functions.",
"value": "<answer>A</answer>"
}
]
}
...
]
You can keep using the normal dataset format, but if you want to train with an explicit reasoning trace you should add a separate reasoning field instead of manually concatenating <think>...</think> into value.
Use --enable_reasoning True only for the following model families:
Qwen/Qwen3-VL-*-ThinkingQwen/Qwen3.5-*When --enable_reasoning True is enabled, the dataset pipeline follows the official chat template behavior for supported models:
<|im_start|>assistant\n<think>\n, and the label starts from the reasoning body.reasoning field is inserted into the reasoning block.value field is treated as the final answer body after the reasoning block.This is intended to make training-time formatting match the model's default inference-time chat template as closely as possible for supported reasoning models.
For unsupported models such as Qwen2-VL, Qwen2.5-VL, and non-thinking Qwen3-VL-Instruct, --enable_reasoning True raises an error on purpose.
Qwen3.5 special case
Qwen3.5 is the only supported family where samples may mix reasoning and non-reasoning data under --enable_reasoning True.Qwen3.5 sample has a reasoning field, the prompt uses the open thinking scaffold and the label starts from the reasoning body.Qwen3.5 sample does not have a reasoning field, the dataset uses the official non-thinking scaffold <think>\n\n</think>\n\n as prompt-only and trains only on the $ claude mcp add Qwen-VL-Series-Finetune \
-- python -m otcore.mcp_server <graph>