Enhancing Semantic Coherence in Image Captioning via a Parameter-Efficient Refinement Framework

Online First: 23/06/2026

Các tác giả

Email tác giả liên hệ:

dinhnt@huit.edu.vn

DOI:

https://doi.org/10.54644/jte.2026.2126

Từ khóa:

Image Captioning, Semantic Coherence, Caption Refinement, Parameter-Efficient Learning, Two-Stage Framework

Tóm tắt

Large-scale pre-trained visual-linguistic models have achieved significant progress in image annotation generation. However, the generated descriptions often suffer from limitations in semantic consistency, a lack of key image elements, and structural coherence. To overcome these limitations without requiring high-cost end-to-end refinement, this study proposes a two-stage parameter-efficient refinement framework. In the first stage, the pre-trained visual-linguistic model is fixed to generate initial annotations from the input image. In the second stage, the problem is redefined as a conditional text generation task, where the pre-trained linguistic model is adjusted using low-order adaptive techniques to improve grammatical structure and enhance semantic coherence, while preserving previously learned knowledge. Experimental results on the Flickr30k dataset, using BLEU-n and METEOR scales, demonstrate that the proposed method significantly enhances the quality of expression and semantic consistency compared to the baseline model, while maintaining a low number of trained parameters. The proposed framework offers a cost-effective solution for enhancing the quality of semantically oriented annotations, and also lays the groundwork for further research on fine-tuning efficient parameters in multimodal language generation models.

Tải xuống: 0

Dữ liệu tải xuống chưa có sẵn.

Tiểu sử của Tác giả

Huynh Anh Ngan Ha, Ho Chi Minh City University of Industry and Trade, Vietnam

Huynh Anh Ngan Ha is currently pursuing a Bachelor’s degree in Software Engineering at Ho Chi Minh City University of Industry and Trade, Ho Chi Minh City, Vietnam, expected to graduate in 2026. This month, she started an internship as a Business Analyst at UNIT while still being a student at the Faculty of Information Technology, Ho Chi Minh City University of Industry and Trade, Vietnam. Her research focuses on Software Engineering, Deep Learning and Business Process.

Email: hahuynhanhngan@gmail.com. ORCID:  https://orcid.org/0009-0003-5613-4630

Lam Hoang Phu Bui, Ho Chi Minh City University of Industry and Trade, Vietnam

Lam Hoang Phu Bui is currently pursuing a Bachelor’s degree in Software Engineering at Ho Chi Minh City University of Industry and Trade, Ho Chi Minh City, Vietnam, expected to graduate in 2026. He is currently working as a freelance AI Engineer. His research focuses on Software Engineering, Deep Learning and Automation.

Email: hoangphu130404@gmail.com. ORCID:  https://orcid.org/0009-0005-2651-7053

Minh Sam Lam, Ho Chi Minh City University of Industry and Trade, Vietnam

Minh Sam Lam expected to receive my Bachelor’s degree in Software Engineering from Ho Chi Minh City University of Industry and Trade (HUIT) in 2026. His core research interests include Software Engineering and Deep Learning, with practical experience in developing language models and data processing. He is currently focusing on optimizing software development lifecycles through advanced machine learning algorithms.

Email: lamminhsam123@gmail.com. ORCID:  https://orcid.org/0009-0000-3951-4699

Thi Dinh Nguyen, Ho Chi Minh City University of Industry and Trade, Vietnam

Thi Dinh Nguyen graduated in Pedagogy Informatics Ho Chi Minh City University of Education in 2006, and received a Master's degree in industry Data transmission and computer network at Ho Chi Minh City Institute of Post and Telecommunications Technology Ho Chi Minh City in 2011. In 2023, she received a PhD degree in Computer Science from the University of Science, Hue, Vietnam. Her field research includes image processing, image retrieval, and information system.

Email: dinhnt@huit.edu.vn. ORCID:  https://orcid.org/0000-0003-3428-3101

Tài liệu tham khảo

J. Li, N. Xu, W. Nie, and S. Zhang, “Image captioning with multi-level similarity-guided semantic matching,” Vis. Informatics, vol. 5, no. 4, pp. 41–48, 2021. DOI: https://doi.org/10.1016/j.visinf.2021.11.003

J. C. Hu, R. Cavicchioli, and A. Capotondi, “Exploiting multiple sequence lengths in fast end-to-end training for image captioning,” in Proc. IEEE Int. Conf. Big Data (BigData), 2023, pp. 2173–2182. DOI: https://doi.org/10.1109/BigData59044.2023.10386812

R. Muzaffar, S. Y. Arafat, J. Rashid, J. Kim, and U. Naseem, “UICD: A new dataset and approach for Urdu image captioning,” PLOS ONE, vol. 20, no. 6, p. e0320701, 2025. DOI: https://doi.org/10.1371/journal.pone.0320701

A. C. Hoang, D. C. Nguyen, and H. L. Nguyen, “Performance evaluation of CNN-based encoders for image captioning,” in Proc. Int. Conf. Control, Autom. Inf. Sci. (ICCAIS), 2023, pp. 212–217. DOI: https://doi.org/10.1109/ICCAIS59597.2023.10382370

J. Li, Z. Mao, H. Li, W. Chen, and Y. Zhang, “Exploring visual relationships via transformer-based graphs for enhanced image captioning,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 20, no. 5, pp. 1–23, 2024. DOI: https://doi.org/10.1145/3638558

Y. Zhang, X. Shi, S. Mi, and X. Yang, “Image captioning with transformer and knowledge graph,” Pattern Recognit. Lett., vol. 143, pp. 43–49, 2021. DOI: https://doi.org/10.1016/j.patrec.2020.12.020

S. S. Santiesteban, S. Atito, M. Awais, Y. Z. Song, and J. Kittler, “Improved image captioning via knowledge graph-augmented models,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2024, pp. 4290–4294. DOI: https://doi.org/10.1109/ICASSP48485.2024.10447637

I. Al Badarneh, B. H. Hammo, and O. Al-Kadi, “An ensemble model with attention-based mechanism for image captioning,” Comput. Electr. Eng., vol. 123, p. 110077, 2025. DOI: https://doi.org/10.1016/j.compeleceng.2025.110077

A. Abdussalam, Z. Ye, A. Hawbani, M. Al-Qatf, and R. Khan, “NumCap: A number-controlled multi-caption image captioning network,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 19, no. 4, pp. 1–24, 2023. DOI: https://doi.org/10.1145/3576927

L. Wang, M. Jiao, Z. Li, M. Zhang, H. Wei, and Y. Ma, “Image captioning model based on multi-step cross-attention cross-modal alignment and external commonsense knowledge augmentation,” Electronics, vol. 14, no. 16, p. 3325, 2025. DOI: https://doi.org/10.3390/electronics14163325

M. J. Parseh and S. Ghadiri, “Graph-based image captioning with semantic and spatial features,” Signal Process. Image Commun., vol. 133, p. 117273, 2025. DOI: https://doi.org/10.1016/j.image.2025.117273

Y. A. Thakare, K. H. Walse, M. Atique, and V. M. Thakare, “Insightful analysis of image captioning models with Image Captions100,” AIP Conf. Proc., vol. 3327, no. 1, p. 020010, 2025. DOI: https://doi.org/10.1063/5.0289777

M. Limbu and D. Banerjee, “MedBLIP: Fine-tuning BLIP for medical image captioning,” arXiv:2505.14726, 2025.

Y. Wang, J. Xu, and Y. Sun, “End-to-end transformer-based model for image captioning,” in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 3, pp. 2585–2594, 2022. DOI: https://doi.org/10.1609/aaai.v36i3.20160

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, and S. Wang, “LoRA: Low-rank adaptation of large language models,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2022.

M. A. A. Khan, Z. U. Rehman, J. Ma, and H. Ma, “Optimization of LoRa for BIoT based on ML: A case of ESL,” Alex. Eng. J., vol. 85, pp. 185–206, 2023. DOI: https://doi.org/10.1016/j.aej.2023.10.064

J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in Proc. Int. Conf. Mach. Learn. (ICML), 2022, pp. 12888–12900.

M. Kalimuthu, A. Mogadala, M. Mosbach, and D. Klakow, “Fusion models for improved image captioning,” in Proc. Int. Conf. Pattern Recognit. (ICPR), 2021, pp. 381–395. DOI: https://doi.org/10.1007/978-3-030-68780-9_32

Y. Ma, J. Ji, X. Sun, Y. Zhou, and R. Ji, “Towards local visual modeling for image captioning,” Pattern Recognit., vol. 138, p. 109420, 2023. DOI: https://doi.org/10.1016/j.patcog.2023.109420

Z. Zhang, Q. Wu, Y. Wang, and F. Chen, “Exploring region relationships implicitly: Image captioning with visual relationship attention,” Image Vis. Comput., vol. 109, p. 104146, 2021. DOI: https://doi.org/10.1016/j.imavis.2021.104146

H. Tian, H. Mo, and L. Jiang, “Image caption generation using multi-level semantic context information,” Symmetry, vol. 13, no. 7, p. 1184, 2021. DOI: https://doi.org/10.3390/sym13071184

J. Seo, S. Lee, L. Liu, and W. Choi, “TA-SBERT: Token attention sentence-BERT for improving sentence representation,” IEEE Access, vol. 10, pp. 39119–39128, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3164769

Tải xuống

Đã Xuất bản

2026-06-23

Cách trích dẫn

[1]
H. A. N. . Ha, L. H. P. Bui, M. S. Lam, và T. D. Nguyen, “Enhancing Semantic Coherence in Image Captioning via a Parameter-Efficient Refinement Framework: Online First: 23/06/2026”, JTE, tháng 6 2026.

Số

2026: Bài đang xuất bản

Chuyên mục

Bài báo khoa học

Categories

Giấy phép

Bản quyền (c) 2026 Tạp chí Khoa học Giáo dục Kỹ Thuật

Giấy phép Creative Commons

Tác phẩm này được cấp phép theo Giấy phép quốc tế Creative Commons Attribution-NonCommercial-NoDeri Phái sinh 4.0 .

Bản quyền thuộc về JTE.

Crossref
Scopus
Google Scholar
Europe PMC

Các bài báo được đọc nhiều nhất của cùng tác giả