Enhancing Semantic Coherence in Image Captioning via a Parameter-Efficient Refinement Framework
Online First: 23/06/2026
Corressponding author's email:
dinhnt@huit.edu.vnDOI:
https://doi.org/10.54644/jte.2026.2126Keywords:
Image Captioning, Semantic Coherence, Caption Refinement, Parameter-Efficient Learning, Two-Stage FrameworkAbstract
Large-scale pre-trained visual-linguistic models have achieved significant progress in image annotation generation. However, the generated descriptions often suffer from limitations in semantic consistency, a lack of key image elements, and structural coherence. To overcome these limitations without requiring high-cost end-to-end refinement, this study proposes a two-stage parameter-efficient refinement framework. In the first stage, the pre-trained visual-linguistic model is fixed to generate initial annotations from the input image. In the second stage, the problem is redefined as a conditional text generation task, where the pre-trained linguistic model is adjusted using low-order adaptive techniques to improve grammatical structure and enhance semantic coherence, while preserving previously learned knowledge. Experimental results on the Flickr30k dataset, using BLEU-n and METEOR scales, demonstrate that the proposed method significantly enhances the quality of expression and semantic consistency compared to the baseline model, while maintaining a low number of trained parameters. The proposed framework offers a cost-effective solution for enhancing the quality of semantically oriented annotations, and also lays the groundwork for further research on fine-tuning efficient parameters in multimodal language generation models.
Downloads: 0
References
J. Li, N. Xu, W. Nie, and S. Zhang, “Image captioning with multi-level similarity-guided semantic matching,” Vis. Informatics, vol. 5, no. 4, pp. 41–48, 2021. DOI: https://doi.org/10.1016/j.visinf.2021.11.003
J. C. Hu, R. Cavicchioli, and A. Capotondi, “Exploiting multiple sequence lengths in fast end-to-end training for image captioning,” in Proc. IEEE Int. Conf. Big Data (BigData), 2023, pp. 2173–2182. DOI: https://doi.org/10.1109/BigData59044.2023.10386812
R. Muzaffar, S. Y. Arafat, J. Rashid, J. Kim, and U. Naseem, “UICD: A new dataset and approach for Urdu image captioning,” PLOS ONE, vol. 20, no. 6, p. e0320701, 2025. DOI: https://doi.org/10.1371/journal.pone.0320701
A. C. Hoang, D. C. Nguyen, and H. L. Nguyen, “Performance evaluation of CNN-based encoders for image captioning,” in Proc. Int. Conf. Control, Autom. Inf. Sci. (ICCAIS), 2023, pp. 212–217. DOI: https://doi.org/10.1109/ICCAIS59597.2023.10382370
J. Li, Z. Mao, H. Li, W. Chen, and Y. Zhang, “Exploring visual relationships via transformer-based graphs for enhanced image captioning,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 20, no. 5, pp. 1–23, 2024. DOI: https://doi.org/10.1145/3638558
Y. Zhang, X. Shi, S. Mi, and X. Yang, “Image captioning with transformer and knowledge graph,” Pattern Recognit. Lett., vol. 143, pp. 43–49, 2021. DOI: https://doi.org/10.1016/j.patrec.2020.12.020
S. S. Santiesteban, S. Atito, M. Awais, Y. Z. Song, and J. Kittler, “Improved image captioning via knowledge graph-augmented models,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2024, pp. 4290–4294. DOI: https://doi.org/10.1109/ICASSP48485.2024.10447637
I. Al Badarneh, B. H. Hammo, and O. Al-Kadi, “An ensemble model with attention-based mechanism for image captioning,” Comput. Electr. Eng., vol. 123, p. 110077, 2025. DOI: https://doi.org/10.1016/j.compeleceng.2025.110077
A. Abdussalam, Z. Ye, A. Hawbani, M. Al-Qatf, and R. Khan, “NumCap: A number-controlled multi-caption image captioning network,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 19, no. 4, pp. 1–24, 2023. DOI: https://doi.org/10.1145/3576927
L. Wang, M. Jiao, Z. Li, M. Zhang, H. Wei, and Y. Ma, “Image captioning model based on multi-step cross-attention cross-modal alignment and external commonsense knowledge augmentation,” Electronics, vol. 14, no. 16, p. 3325, 2025. DOI: https://doi.org/10.3390/electronics14163325
M. J. Parseh and S. Ghadiri, “Graph-based image captioning with semantic and spatial features,” Signal Process. Image Commun., vol. 133, p. 117273, 2025. DOI: https://doi.org/10.1016/j.image.2025.117273
Y. A. Thakare, K. H. Walse, M. Atique, and V. M. Thakare, “Insightful analysis of image captioning models with Image Captions100,” AIP Conf. Proc., vol. 3327, no. 1, p. 020010, 2025. DOI: https://doi.org/10.1063/5.0289777
M. Limbu and D. Banerjee, “MedBLIP: Fine-tuning BLIP for medical image captioning,” arXiv:2505.14726, 2025.
Y. Wang, J. Xu, and Y. Sun, “End-to-end transformer-based model for image captioning,” in Proc. AAAI Conf. Artif. Intell., vol. 36, no. 3, pp. 2585–2594, 2022. DOI: https://doi.org/10.1609/aaai.v36i3.20160
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, and S. Wang, “LoRA: Low-rank adaptation of large language models,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2022.
M. A. A. Khan, Z. U. Rehman, J. Ma, and H. Ma, “Optimization of LoRa for BIoT based on ML: A case of ESL,” Alex. Eng. J., vol. 85, pp. 185–206, 2023. DOI: https://doi.org/10.1016/j.aej.2023.10.064
J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in Proc. Int. Conf. Mach. Learn. (ICML), 2022, pp. 12888–12900.
M. Kalimuthu, A. Mogadala, M. Mosbach, and D. Klakow, “Fusion models for improved image captioning,” in Proc. Int. Conf. Pattern Recognit. (ICPR), 2021, pp. 381–395. DOI: https://doi.org/10.1007/978-3-030-68780-9_32
Y. Ma, J. Ji, X. Sun, Y. Zhou, and R. Ji, “Towards local visual modeling for image captioning,” Pattern Recognit., vol. 138, p. 109420, 2023. DOI: https://doi.org/10.1016/j.patcog.2023.109420
Z. Zhang, Q. Wu, Y. Wang, and F. Chen, “Exploring region relationships implicitly: Image captioning with visual relationship attention,” Image Vis. Comput., vol. 109, p. 104146, 2021. DOI: https://doi.org/10.1016/j.imavis.2021.104146
H. Tian, H. Mo, and L. Jiang, “Image caption generation using multi-level semantic context information,” Symmetry, vol. 13, no. 7, p. 1184, 2021. DOI: https://doi.org/10.3390/sym13071184
J. Seo, S. Lee, L. Liu, and W. Choi, “TA-SBERT: Token attention sentence-BERT for improving sentence representation,” IEEE Access, vol. 10, pp. 39119–39128, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3164769
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2026 Journal of Technical Education Science
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright © JTE.
