Parameter-Efficient Fine-Tuning of IndoBERT Using LoRA in EmoJournal Web Application for Student Emotion Classification

Marsha Faine Sabina(1), Yeka Hendriyani(2),
(1) Universitas Negeri Padang  Indonesia
(2) Universitas Negeri Padang  Indonesia

Corresponding Author


DOI : https://doi.org/10.24036/voteteknika.v14i3.139436

Abstract


Emotion classification in Indonesian text presents significant linguistic complexity due to non-standard slang, pervasive abbreviations, colloquial particles, and subtle implicit expressions in student daily reflective narratives. Full fine-tuning of large pre-trained contextual Transformer models such as IndoBERT demands substantial graphics memory and computational infrastructure, while introducing severe vulnerability to overfitting and catastrophic forgetting when adapted to domain-specific corpora. This study aims to design, evaluate, and deploy an end-to-end parameter-efficient emotion classification pipeline across six discrete emotional categories (happiness, disgust, anger, sadness, fear, and surprise) by implementing Low-Rank Adaptation (LoRA) on IndoBERT using PyTorch, integrated natively into the Django-based EmoJournal web architecture. The methodology combines computational deep learning experimentation with an Agile Kanban software engineering lifecycle. Text processing encompasses case folding, tokenization via IndoBERT-base-p1, sequence length bounding to 128 tokens, and stratified dataset partitioning. Low-rank decomposition matrices ($r=8, \alpha=16$) are injected into the multi-head self-attention query and value projections, optimizing merely 299,526 trainable parameters out of 124,745,484 total parameters (0.24%). The fine-tuned IndoBERT-LoRA model achieved an internal test accuracy of 98.40%, a macro-F1 score of 0.9840, and a test loss of 0.0577 across 6,141 validation instances. External generalization benchmarking against the out-of-domain Emo Tweet dataset (2,214 samples) yielded an accuracy of 61.92% and a macro-F1 of 0.6286, successfully mapping the boundaries of cross-domain linguistic shift. The serialized adapter model was integrated into EmoJournal via PeftModel, facilitating real-time inference with sub-second latency to generate dynamic confidence scores, seven-day affective longitudinal visualization dashboards, and an Early Warning System (EWS) for institutional counseling units. In conclusion, LoRA adaptation provides a scalable, computationally viable paradigm for deploying deep contextual NLP within expressive digital health platforms without sacrificing core semantic representation.

Keywords— IndoBERT, Low-Rank Adaptation (LoRA), Emotion Classification, Natural Language Processing, EmoJournal, PyTorch, Django.


References


J. J. Gross, "Emotion regulation: Current status and future prospects," Psychol. Inq., vol. 26, no. 1, pp. 1–26, 2015.

P. Ekman, "An argument for basic emotions," Cogn. Emot., vol. 6, no. 3–4, pp. 169–200, 1992.

D. Eisenberg, E. Golberstein, and S. E. Hunt, "Mental health and academic success in college," B.E. J. Econom. Anal. Policy, vol. 9, no. 1, pp. 1–35, 2009.

J. W. Pennebaker, "Writing about emotional experiences as a therapeutic process," Psychol. Sci., vol. 8, no. 3, pp. 162–166, 1997.

K. A. Fissel, "Online emotional disclosure: A review of expressive writing in digital mental health," Comput. Human Behav., vol. 101, pp. 199–209, 2019.

M. S. Saputri, R. Mahendra, and M. Adriani, "Emotion classification on Indonesian Twitter dataset using Support Vector Machines," in Proc. Int. Conf. Asian Lang. Process. (IALP), 2018, pp. 90–95.

S. M. Mohammad, "Sentiment analysis: Detecting valence, emotions, and other affectual states from text," in Emotion Measurement, H. L. Meiselman, Ed. Duxford, UK: Woodhead Publishing, 2016, pp. 201–237.

F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, "IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP," in Proc. 28th Int. Conf. Comput. Linguist., 2020, pp. 757–770.

B.Wilie et al., "IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding," in Proc. 1st Conf. Asia-Pacific Chapter Assoc. Comput. Linguist., 2020, pp. 843–857.

A. Vaswani et al., "Attention is all you need," in Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 5998–6008

J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. NAACL-HLT, 2019, pp. 4171–4186.

E. J. Hu et al., "LoRA: Low-rank adaptation of large language models," in Proc. Int. Conf. Learn. Represent. (ICLR), 2022, pp. 1–26.

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "QLoRA: Efficient finetuning of quantized LLMs," in Adv. Neural Inf. Process. Syst. (NeurIPS), 2023, pp. 10088–10115.

N. Ding et al., "Delta tuning: Comprehensive tracking and analysis of parameter-efficient methods for pre-trained language models," Nat. Comput. Sci., vol. 3, no. 9, pp. 779–791, 2023.

Q. Zhang et al., "Adaptive budget allocation for parameter-efficient fine-tuning," in Proc. Int. Conf. Learn. Represent. (ICLR), 2023, pp. 1–18.

P. Burnap and M. L. Williams, "Us and them: Identifying cyber hate on Twitter across multiple protected characteristics," EPJ Data Sci., vol. 5, no. 1, p. 11, 2016.

I. Sommerville, Software Engineering, 10th ed. Boston, MA: Pearson, 2015.

M. Mubai, K. Rukun, E. Tasrif, and A. Huda, "Augmented Reality (AR) - Based Learning Media on the Subject of Computer Network Installation," J. Pendidik. dan Pengajaran, vol. 53, no. 2, pp. 213–226, 2020.

D. Ramadhani, A. Purwarianti, and D. H. Widyantoro, "Domain adaptation in Indonesian emotion classification using contextualized embeddings," IEEE Access, vol. 9, pp. 132450–132461, 2021.

S. M. Metev and V. P. Veiko, Laser Assisted Microtechnology, 2nd ed., R. M. Osgood, Jr., Ed. Berlin, Germany: Springer-Verlag, 1998.


Article Metrics

 Abstract Views : 129 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.