Abstract

Reducing the language disparity in AI through efficient multilingual natural language inference.

This project addresses the challenge of multilingual Natural Language Inference (NLI), particularly in low-resource languages. It proposes a scalable, efficient system that leverages LoRA (Low-Rank Adaptation) techniques for parameter-efficient fine-tuning of pre-trained transformer models. The system aims to enhance cross-lingual understanding by minimizing computational load while improving performance across diverse languages. It supports tasks such as classification, summarization, and translation validation, bridging the gap between high- and low-resource language capabilities.

System Components

Core modules enabling efficient multilingual inference.

• Data Preprocessor: Cleans, normalizes, and formats multilingual text (includes Bengali-specific logic)
• LoRA-enhanced Transformer: Uses LoRA-based fine-tuning to enable efficient adaptation to new languages
• XNLI Dataset Adapter: Supports stratified sampling and balanced dataset preparation for multilingual training
• Evaluation Pipeline: Computes class-wise precision, recall, F1, and confusion matrix metrics
• Inference Layer: Handles single prediction inputs with calibrated confidence scores

Architecture

From dataset loading to fine-tuning and inference.

Training Pipeline

• Data Loading: Uses HuggingFace XNLI datasets (fallbacks to alternatives if unavailable)
• Text Preprocessing: Language-specific cleaning, Unicode normalization, punctuation handling
• Tokenization: Sentence pair tokenization with RoBERTa/XLM-R rules
• LoRA-based PEFT: Wraps AutoModelForSequenceClassification using peft with LoRA configs