RESEARCH 001 · ARXIV PREPRINT
GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation
21 October 2026 · Revised 22 October 2026 · v2
How much of a large language model’s capability can a smaller model retain? GoiBERT explores this question through two compact Vietnamese encoders that learn from two teacher models. They process raw Vietnamese text without a separate word-segmentation step and are released for task-specific fine-tuning.
01 / THE RESEARCH
Learning more.
Carrying less.
Summary of the paper
Training happens in two stages. A Vietnamese encoder with the same tokenizer teaches general representations. A larger teacher then provides task-level guidance through soft labels. Across eight benchmarks, the paper finds that compression preserves sentiment and classification performance more effectively than inference and named-entity recognition. Those remaining gaps point to the next research questions.
02 / RELEASED BACKBONES
Two sizes.
A shared idea.
These are encoder backbones for task-specific fine-tuning. Choose the model and evaluate it on your own task and data.
The paper is licensed under CC BY 4.0. Model weights have separate usage terms. Read the model license before use.
03 / CITATION
Build on this work.
Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh, Thanh Danh Le, Khang Nhat Hoang Vo, and Huynh Dai Nhan Tran. GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation. arXiv:2610.30067, 2026. DOI
BibTeX
@misc{goibert,
title = {GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation},
author = {Quang Anh Kiet Tran and Nam Anh Tran and Manh Hung Trinh and Thanh Danh Le and Khang Nhat Hoang Vo and Huynh Dai Nhan Tran},
year = {2026},
eprint = {2610.30067},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.30067},
doi = {10.48550/arXiv.2610.30067},
}