Author: Merilin Sousa Silva

About

This project explores the capabilities of large language models and multilingual masked/ discriminative language models to detect loanwords in a given sentence from the ConLoan dataset[1]. Additionally, the language models ELECTRA-base, mBERT-base and XLM_RoBERTa base and large were trained and tested on the ConLoan dataset to check for an improvement.

Loanwords are "Loanword (or lexical borrowing) is here defined as a word that at some point in the history of a language entered its lexicon as a result of borrowing (or transfer, or copying)" (Haspelmath 36).[2]

---

The ConLoan dataset

This dataset contains annotated loanwords with their native alternatives for 10 languages, which are German, Portuguese, Spanish, Greek, Russian, Italian, Icelandic, French, Northern-Kurdish and Chinese. For each language one human annotator was provided with a definition for what a loanword is and tasked with the annotation and replacement of loanwords in multiple thousand sentences.

For this project the json files were used and the language models received the raw sentences for detection and training and the evaluation was done through comparison with the loanword list provided in each json element.

Please cloan the ConLoan-main and watch out for double .git files, you might need to delete the one in the ConLoan-main if you want to keep it nested. It is recommended to keep them separate and change the file paths in the scripts accordingly. (github repository [1])

---

Requirements

This project contains mainly jupyter notebooks. The Llama notebook was run on Google colab with a A100 GPU (L4 GPU suffices too). Additionally, all training and testing of the fine-tuned language models was done on Google colab with a v5e-1 TPU. The rest was done locally on a MacBook Pro with a Apple M2 Pro Chip and 16 GB of RAM.

All needed packages can be found in the requirements.txt file, please install the packages.

---

Testing of Large Language Models

Zero-Shot & Few-Shot Evaluation of Large Language Models

This section documents the prompt-based evaluation of three LLMs—Gemini-2.5-Flash-Lite, GPT-4.1, and Meta-Llama-3-8B-Instruct—for token/span-level loanword detection without supervised fine-tuning. The goal is to establish prompt-only baselines, assess how well state-of-the art large language models perform with loanword detection and to see if the definition of a loanword is diverges between human linguists and large language models.

---

Models & Prompting Regimes

• Models: Gemini-2.5-Flash-Lite, GPT-4.1, Meta-Llama-3-8B-Instruct.

• Regimes:

  • Zero-shot: model is prompted without examples.
  • Few-shot: the prompt includes two in-language exemplars (from the currently processed language) to condition the model’s behavior.

---

Prompt Suite (Unchanged)

Each model is evaluated under three prompt variants in both zero-shot and few-shot settings. The prompts are kept exactly as below.

Prompt 1

You are a loanword detection system.

Instructions:
- Output must be ONLY a valid Python list of strings.
- If no loanwords are found, return an empty list.
- Do not add explanations or any other text.

Now process this sentence:
Sentence: "{sentence}"
Output:

Prompt 2

You are a loanword detection system.

Loanword (or lexical borrowing) is here defined as a word that at some point in the history of a language entered its lexicon as a result of borrowing (or transfer, or copying).

Instructions:
- Output must be ONLY a valid Python list of strings.
- If no loanwords are found, return an empty list.
- Do not add explanations or any other text.

Now process this sentence:
Sentence: "{sentence}"
Output:

Prompt 3

You are a loanword detection system.

From the point of view of an entire language (not that of a single speaker), a loanword is a word that can conventionally be used as part of the language. In particular, it can be used in situations where no code-switching occurs, e.g. in the speech of monolinguals. This is the simplest and most reliable criterion for distinguishing loanwords from single-word switches.

Instructions:
- Output must be ONLY a valid Python list of strings.
- If no loanwords are found, return an empty list.
- Do not add explanations or any other text.

Now process this sentence:
Sentence: "{sentence}"
Output:

---

Output Post-Processing & Matching

All models are instructed to return a Python list of strings. The raw response is parsed, normalized (lowercasing/whitespace), and compared against the gold loanword list per sentence. To accommodate realistic variations, two evaluation protocols are applied:

• Strict evaluation

  • Set-based comparison where tokenization of predicted loanwords must match the gold segmentation.
  • Order is irrelevant (set semantics), but a multiword loanword must appear as one item if the gold has it as one item.

• Relaxed evaluation

  • The gold list is expanded by splitting multiword loanwords into their token constituents; predictions are left unchanged.
  • As a result, both ['social media'] and ['social', 'media'] can be counted as correct, improving tolerance to segmentation variance.

For both strict and relaxed protocols, precision, recall, and F1-score are computed at the sentence level and aggregated.

---

Caching & Determinism Controls

To avoid repeated API calls and ensure reproducibility:

• Response caching: each (sentence, prompt, regime, model) output is stored in a JSON cache keyed by an MD5 hash. On re-runs, the cached value is used if present.
• Parsing robustness: if the model returns auxiliary text, the pipeline attempts to recover a valid list via conservative parsing; invalid outputs default to [] and are logged for error analysis.

---

Here’s an updated version of your README section with a new bullet describing what the compute_llm_summary.py script produces — including the new LaTeX average tables:

---

What Gets Produced

Running the notebooks under models/geminiAPI/, models/openAIAPI/, and models/llamaAPI/ (one per vendor) yields:

1. Overall metrics per prompt (zero-shot & few-shot; strict & relaxed):

   • OverallMetrics_<Vendor>_<Regime>_<Strict|Relaxed>_Prompt<k>.xlsx
   • Contains precision, recall, and F1 aggregated across all languages.

2. Per-language F1 tables per prompt (zero-shot & few-shot; strict & relaxed):

   • LanguageEvaluation_<Vendor>_<Regime>_<Strict|Relaxed>_Prompt<k>.xlsx
   • Rows = languages; values = F1-score.

3. Error inspection sheets per prompt:

   • ErrorInspection_<Vendor>_Prompt<k>.xlsx
   • Up to two mispredictions per language with sentence context for qualitative analysis.

4. Caches

   • JSON files under caches/ storing normalized model outputs to prevent redundant API calls.

5. Language-Level Aggregation Tables

• Generated by tools/compute_language_llm_avg.py after the per-language F1 tables (merged_language_f1_prompt*.xlsx) have been created for all prompts.

• Produces average F1-score summaries focused on language-level performance patterns across prompts and models.

• Output files are saved in results/averages/:

  • avg_by_model.xlsx / avg_by_model.tex

    • Rows = languages
    • Columns = models (Gemini, OpenAI, Llama)
    • Each cell represents the average F1-score for that language–model pair, averaged across all prompt, shot, and evaluation configurations.

  • avg_by_prompt.xlsx / avg_by_prompt.tex

    • Rows = languages
    • Columns = prompts (Prompt1, Prompt2, Prompt3)
    • Each cell represents the average F1-score for that language across all models and configurations within the given prompt.

• Both .tex tables include descriptive captions and labels for direct inclusion in LaTeX documents.

6. Summary and LaTeX tables

   • Generated by compute_llm_summary.py after merging the individual .tex result files.

   • Produces:

     • llm_metrics_summary.xlsx — an Excel workbook containing:

       • Full extracted data (All Data)
       • Averages by model (Avg by Model)
       • Averages by prompt (Avg by Prompt)
       • Averages by model and prompt (Avg by Model+Prompt)
       • Overall averages (Overall Avg)

     • llm_metrics_summary_tables.tex — a LaTeX file containing formatted tables for:

       • Average metrics per prompt
       • Average metrics per model
       • Overall average across all models and prompts

   • These tables include Precision, Recall, and F1-Score values, suitable for direct inclusion in reports or publications.

---

Reproducing the Evaluation

1. Environment

   • Python ≥ 3.10 with the packages in requirements.txt.

2. Credentials

   • Configure API keys for Gemini, OpenAI, and (if applicable) hosted Llama access.
   • Verify rate limits and set conservative concurrency to avoid throttling.

3. Data layout

   • Place the sentence-level evaluation files (e.g., from ConLoan) under each notebook’s expected datasets/ directory, or change the path accordinly.
   • Each file’s stem denotes the language ID used in per-language reporting.

4. Run notebooks

   • Execute the zero-shot and few-shot notebooks for each vendor.
   • For few-shot, the notebook selects two in-language examples automatically to populate the prompt.

5. Merge & report

   • To build cross-model summary tables (overall and per-language) in Excel and LaTeX, run:

     tools/create_metric_llm_table.py
     

   • This script consolidates the generated .xlsx metrics into merged workbooks and TeX tables suitable for inclusion in reports.

---

Rationale

This prompt-only evaluation establishes task-agnostic baselines for loanword detection across languages and prompt formulations. The comparison between strict and relaxed scoring quantifies sensitivity to multiword segmentation. These baselines motivate and calibrate subsequent supervised fine-tuning efforts (see the fine-tuning section) by indicating where prompt-only performance saturates and where model adaptation is beneficial.

---

Testing multilingual language models on loanword detection without fine-tuning them

Zero-Shot Baselines with Multilingual Encoders (mBERT, XLM-R Base/Large, ELECTRA)

This section establishes strong zero-shot baselines for loanword detection using widely adopted multilingual encoders—mBERT, XLM-RoBERTa (base & large), and ELECTRA (base multilingual)—without any task-specific fine-tuning (e.g., no fine-tuning on ConLoan). The goal is to quantify how well pretrained multilingual models transfer to token-level loanword detection out-of-the-box, and to determine whether additional fine-tuning is warranted.

Approach (high level)

• Each model is used as a frozen feature extractor for token-level sequences.

• Sentences are tokenized into word pieces; model hidden states are computed for each subword token.

• A zero-shot decision rule maps model outputs to BIO loanword tags. The rule is purely deterministic and does not use any supervised updates on the target dataset. Typical ingredients include:

  • subword continuity and wordpiece boundaries (to reassemble word-level spans),
  • simple confidence/score transforms derived from the model’s token-level outputs,
  • language-agnostic normalization (lowercasing, punctuation stripping).

• Predictions are compared against gold spans; Precision/Recall/F1 are computed overall and per language.

A single reference notebook (e.g., xlm_roberta_base_inference.ipynb) is provided; the mBERT, XLM-R large, and ELECTRA notebooks are structure-identical, differing only in the model_name checkpoint.

What the notebook(s) do

1. Load & normalize data

   • Reads multiple *.json/*.csv files from a dataset directory.
   • Extracts plain source text and gold loanword spans.
   • Cleans tokens (lowercasing, punctuation removal) and aligns gold spans to word indices.

2. Tokenization & alignment

   • Applies the model’s tokenizer with is_split_into_words=True.
   • Tracks word_ids to map subword tokens back to word-level positions.
   • Builds BIO labels for evaluation (gold spans → B-LOAN/I-LOAN/O).

3. Zero-shot inference

   • Runs the frozen encoder to obtain contextual token representations (no gradient updates).
   • Applies the deterministic decision rule to assign BIO tags at word level.

4. Scoring

   • Computes precision/recall/F1 overall and per language.
   • Generates error inspections (sentences/tokens where predicted BIO tags differ from gold).

5. Artifacts

   • Saves overall metrics (OverallMetrics_<Model>.xlsx).
   • Saves per-language metrics (LanguageEvaluation_<Model>.xlsx).
   • Saves error analysis sheet with token-level mismatches (ErrorInspection_<Model>.xlsx).

How to reproduce

Prerequisites

• Python 3.10+ and recent transformers, datasets, pandas, seqeval.
• GPU strongly recommended for throughput (CPU is supported but slower).

Steps

1. Install dependencies

   pip install -r requirments.txt

2. Place data

   • Put ConLoan (or any compatible BIO-derivable dataset) files under datasets/.
   • File names (minus extension) are treated as language IDs for per-language reporting.

3. Open a notebook

   • Use one of:

     • mbart_base_inference.ipynb (mBERT)
     • xlm_roberta_base_inference.ipynb (XLM-R base)
     • xlm_roberta_large_inference.ipynb (XLM-R large)
     • electra_base_multilingual_loanword_detection.ipynb (ELECTRA base multilingual)

   • Set data_dir = Path("datasets") (or point to your folder).

4. Select the model checkpoint

   • Example for XLM-R base:

     MODEL_NAME = "xlm-roberta-base"

   • Other options:

     • bert-base-multilingual-cased
     • xlm-roberta-large
     • google/electra-base-discriminator

5. Run all cells

   • The notebook handles loading, tokenization, zero-shot tagging, scoring, and export.

6. Inspect outputs (written to the working directory unless changed)

   • Overall: OverallMetrics_<Model>.xlsx
   • Per language: LanguageEvaluation_<Model>.xlsx
   • Errors: ErrorInspection_<Model>.xlsx

Why zero-shot first?

Zero-shot performance quantifies the raw transferability of multilingual encoders to the loanword detection objective, independent of in-domain supervision. These baselines serve as a reference floor: if results are insufficient for downstream use, they justify task-specific fine-tuning and/or data augmentation (e.g., on ConLoan). Conversely, if certain languages already reach competitive F1, fine-tuning efforts can be focused on the low-resource or hard languages identified in the per-language tables.

Expected deliverables

• Per-model overall metrics (precision/recall/F1) as Excel spreadsheets.
• Per-language F1 breakdowns to diagnose cross-lingual transfer.
• Token-level error sheets to review systematic failure modes (boundary errors, multi-word spans, script mixing, etc.).

These outputs provide a consistent, replicable baseline for subsequent experiments (e.g., few-shot adapters, full fine-tuning, CRF decoding, prompt-conditioned variants).

---

Training detection models on ConLoan Dataset

Fine-Tuning Token-Level Loanword Detectors (mBERT, XLM-R Base/Large, ELECTRA)

This section details the supervised fine-tuning procedure used to adapt multilingual encoders—mBERT, XLM-RoBERTa (base & large), and ELECTRA (base multilingual)—to the BIO loanword tagging task. One reference notebook (e.g., xlm_roberta_base_loanword_detection.ipynb) is provided; the others are structure-identical, differing only by the model checkpoint.

The objective is to establish task-aware detectors on top of pretrained encoders, quantify gains over zero-shot baselines, and produce reusable checkpoints and evaluation artifacts.

p.s. The model weights can be found on Hugging Face: msousa/Multilingual_Loanword_Detection_Models

Approach (high level)

• Task: sequence labeling with BIO tags (O, B-LOAN, I-LOAN) at the word level.

• Model head: a single token-classification layer on top of the multilingual encoder.

• Tokenization & alignment:

  • Sentences are split into words; model tokenizer is run with is_split_into_words=True.

  • word_ids are used to propagate gold word-level tags to subword pieces:

    • first subword → B-LOAN/O
    • subsequent subwords of a loanword → I-LOAN
    • special tokens → label -100 (ignored in loss)

• Loss: cross-entropy over visible subword tokens.

• Metrics: seqeval precision/recall/F1 aggregated at the entity level on the eval split and per-language diagnostics.

What the notebook(s) do

1. Load & normalize data

   • Reads multiple *.json/*.csv from datasets/.
   • Uses file stem as language ID for per-language reporting.
   • Extracts source_plain and words_in_L_tags (loanword spans) and normalizes surface forms.

2. Label projection to tokens

   • Builds BIO tags at the word level from span annotations.
   • Tokenizes with the chosen model tokenizer (e.g., xlm-roberta-base) and aligns to subwords via word_ids.
   • Assigns -100 to non-word positions so they are excluded from loss.

3. Dataset & split

   • Wraps examples in a datasets.DatasetDict with an 80/20 train/test split (random_state=42).

4. Model & collator

   • Loads the encoder with a token-classification head (AutoModelForTokenClassification), with:

     • num_labels = 3
     • label2id/id2label consistent with ["O", "B-LOAN", "I-LOAN"]

   • Uses DataCollatorForTokenClassification for dynamic padding and label alignment.

5. Training configuration (hyperparameters)

   • num_train_epochs = 10
   • per_device_train_batch_size = 16
   • weight_decay = 0.01
   • optim = "adamw_torch"
   • logging_steps = 50
   • (Default learning rate from TrainingArguments is used unless explicitly overridden in the notebook.)

6. Training & evaluation

   • Runs supervised fine-tuning on the training split.
   • Evaluates at the end of each epoch with seqeval precision/recall/F1.
   • After training, computes per-language metrics by filtering the eval set by language and re-running prediction.

7. Error analysis

   • Exports a token-level sheet of mismatched tags (gold vs. predicted) with the full sentence context for inspection.

8. Artifact export

   • Hugging Face training artifacts under out-detection-model/:

     • best checkpoint (pytorch_model.bin or model.safetensors, config.json)
     • tokenizer files
     • trainer_state.json, trainer_config.json
     • epoch-wise eval_results.json (if enabled by version)

   • Overall metrics: OverallMetrics_<Model>.xlsx (precision/recall/F1)

   • Per-language metrics: LanguageEvaluation_<Model>.xlsx (F1 and optionally P/R)

   • Error inspection: wrong_predictions.xlsx (per-token mismatches with sentences)

How to reproduce

Prerequisites

• Python 3.10+ and recent versions of:

  pip install -r requirments.txt

• GPU/TPU recommended for throughput (the notebooks are configured to be TPU-friendly if used). All the notebooks were also trained on Google Colab with a v5e-1 TPU.

Steps

1. Prepare data

   • Place the dataset files under datasets/. Each file’s stem (e.g., de.json) is treated as the language name.

2. Open a notebook

   • Use one of:

     • xlm_roberta_base_loanword_detection.ipynb
     • (analogous notebooks exist for mBERT, XLM-R large, ELECTRA)

3. Set the encoder checkpoint

   MODEL_NAME = "xlm-roberta-base"     # or:
   # "xlm-roberta-large"
   # "bert-base-multilingual-cased"
   # "google/electra-base-discriminator"

4. Run all cells

   • The notebook performs: loading → alignment → training → evaluation → exports.

5. Inspect results

   • Check out-detection-model/ for checkpoints and logs.
   • Review OverallMetrics_<Model>.xlsx and LanguageEvaluation_<Model>.xlsx.
   • Inspect wrong_predictions.xlsx for qualitative errors (boundary errors, missed multi-word spans, etc.).

Expected outputs

• Model artifacts (in out-detection-model/):

  • Fine-tuned checkpoint(s) with config and tokenizer.
  • Trainer metadata (trainer_state.json, logs).

• Metrics (Excel):

  • OverallMetrics_<Model>.xlsx — overall precision/recall/F1 on the held-out split.
  • LanguageEvaluation_<Model>.xlsx — per-language scores (F1; optionally P/R).

• Diagnostics:

  • wrong_predictions.xlsx — token-level mismatches with sentence context for error analysis.

These fine-tuned detectors provide task-specific performance references against the zero-shot baselines and can be directly deployed or further improved (e.g., with CRF decoding, span post-processing, or language-adaptive fine-tuning).

Merge Results The python script tools/merge_data.py can be run to merge the language-specific and overall performance results of the fine-tuned and base multilingual models. One has to change file paths etc.

---

Results

LLM results

Common mistakes

Gemini

Gemini’s errors cluster into a few clear patterns. First, it frequently over-predicts salient content words that are topical or technical but not borrowed (e.g., French “pétrolier,” “casse”; German “EU,” “europäischen”; Portuguese “imprensa,” “televisão”) while missing the annotated borrowings such as cargo, rail, bomba, comité, trust, or Latinisms like ex ante/ex post. Second, it often splits or expands multi-word borrowings (e.g., Kurdish social media → “social, media”; Spanish monitoring of the implementation → “monitoring, implementation”), creating partial matches or extra false positives. Third, it exhibits script/orthography heuristics: in Chinese it selects native content strings like “花樣, 年華” and ignores short particles; in Icelandic and Russian it targets morphologically rich common nouns rather than genuine loan stems. Finally, Gemini sometimes labels named entities or domain nouns (e.g., “plan,” “power,” “director,” “assembleia/deputado”) as borrowings. These behaviors are consistent with its internal loanword heuristic—words that “are adopted from one language into another” and may retain or slightly adapt pronunciation, often motivated by cultural contact or lexical gaps—which likely biases the model toward rare, technical, or “foreign-looking” tokens rather than etymologically borrowed lexemes per se.

Llama

Across prompts, LLaMA’s false positives skew toward salient content nouns and capitalized items (e.g., institutional terms, topical nouns, acronyms, proper names) and toward partial spans of multiword expressions (returning one token of a compound). False negatives are concentrated in: (i) multiword loanwords/Latinisms where the model emits only a subset or reorders tokens; (ii) script/segmentation challenges (Chinese, Greek) where token boundaries are unclear; (iii) inflected forms in morphologically rich languages (Russian, Kurdish, Icelandic) where the surface deviates from the canonical dictionary form; and (iv) domain jargon and calques, which the model treats as native even when the gold labels mark borrowing. These patterns align with the model’s own definition of a loanword—“a word that is borrowed from another language…often with little or no modification” —which biases detection toward items that look foreign orthographically or phonotactically and away from integrated, morphologically adapted, or translated borrowings. Prompting helps (few-shot reduces span fragmentation), but residual errors suggest the model relies on surface heuristics (capitalization, rarity, Latin/Greek roots) rather than robust cross-lingual lexical evidence, leading to over-tagging of salient nouns and under-tagging of integrated borrowings.

OpenAI

OpenAI’s own definition frames a loanword as “a word that has been borrowed from one language and adopted into the vocabulary of another,” emphasizing adoption into the recipient lexicon but giving little guidance on boundary cases such as named entities, multi-word borrowings, or calques. In the error sheets this shows up in three recurrent ways. (1) Over-selection of salient content words in Romance/Germanic texts: the model frequently tags topical nouns from EU administrative prose (e.g., plan, rapport, commission, transports, power) as loanwords alongside the true target (e.g., trust), which inflates false positives. (2) Multi-word expressions (MWEs) are often only partially recovered (market failures → market, failures) or expanded with related terms (monitoring of the implementation → monitoring, implementation), hurting strict scoring and even relaxed scoring when extraneous items dominate. (3) Named-entity and script effects: in Chinese, Greek, and Russian examples the model either returns nothing or treats proper names/ordinary nouns as borrowings (e.g., selecting generic items like 作品, комитет, or person surnames), suggesting the heuristic “foreign-looking or domain-salient ⇒ loanword.” A smaller but consistent issue is domain drift: English technical Anglicisms that do exist in French/Portuguese are often over-generated beyond the gold label set, reducing precision. Overall, GPT-4.1 tends to equate perceived “foreignness” or topic salience with borrowing, struggles to keep MWEs intact, and is conservative/empty for non-Latin scripts—yielding modest F1 and indicating that lexicon-aware prompts, stricter output constraints, and (ultimately) task-specific fine-tuning or post-hoc filtering would likely be required to reach dependable loanword detection performance.

Language dependend scores

General observational remarks

• Who wins overall? Averaged across languages, Gemini is strongest, especially for prompt 1 and 3, OpenAI is second. Llama performs the worst, being the only Large Language model to achieve zero scores.

• Languages that tend to score higher: Chinese, French, Italian repeatedly show the best means across prompts. • Prompt 1 top-3: Chinese (≈0.605), French (≈0.474), Italian (≈0.469). • Prompt 2 top-3: Chinese (≈0.616), Italian (≈0.509), French (≈0.503). • Prompt 3 top-3: Chinese (≈0.552), Italian (≈0.466), Greek (≈0.431). Likely reasons: clearer cues for borrowed lexemes (proper names, technical terms, Latin/Romance roots), and tokenization that aligns well with the prompts.

• Languages that tend to score lower: German, Icelandic, and Portuguese are repeatedly near the bottom. • Prompt 1 bottom-3: German (≈0.141), Icelandic (≈0.307), Portuguese (≈0.325). • Prompt 2 bottom-3: German (≈0.172), Icelandic (≈0.318), Portuguese (≈0.368). • Prompt 3 bottom-3: German (≈0.144), Icelandic (≈0.300), Portuguese (≈0.315). Likely reasons: (i) gold sets include fully nativized items (“solide”, “präzise”, “bomba”) that LLMs treat as native; (ii) compounding/morphology (German, Icelandic) makes boundary detection and etymological inference harder.

• Model-by-language tendencies: • Gemini is most robust across almost all languages, particularly Chinese and Italian, and usually leads in French and Greek. • OpenAI narrows the gap on some Romance languages (Italian/French) and occasionally in Russian, but still lags Gemini overall. • Llama underperforms broadly; its drops are most visible in German, Icelandic, and Northern-Kurdish where morphology and compounding are prominent or the language was less prominent in the training data of the LLM.

• Why Prompt 3 underperforms: Its longer, more theory-laden definition seems to encourage conservative behavior and semantic over-reasoning, reducing recall. Prompts 1/2 give more condensed extraction behavior.

• Zero- vs. Few-Shot prompting Generally, Few-Shot prompts (2 shots) yield in better results. This indicates that by showing the LLM, what human Linguists regard as loanwords, it becomes better at replicating the gold truth of the ConLoan dataset.

Overall scores dependend scores

• Across all runs, Gemini is the strongest overall, OpenAI comes second, and Llama trails by a clear margin.
• Few-shot prompting generally helps, especially for Gemini and, to a lesser extent, OpenAI, by nudging the models toward the extraction format and raising recall without a catastrophic precision drop. Llama shows little benefit from few-shot in these settings.
• Counter-intuitively, the relaxed scorer (which accepts tokenized variants like ["social", "media"] for ["social media"]) does not guarantee higher F1: it raises recall but often inflates false positives (models over-select salient nouns and named entities), so strict F1 is frequently higher in your results.
• Put simply: Gemini > OpenAI > Llama, few-shot ≥ zero-shot for most models, and strict scoring typically beats relaxed on F1 because relaxed precision falls faster than recall rises.

Non-finetuned multilingual model perfomance

Common mistakes

• High false negatives on inflected/compounded forms. Models miss loanwords once they’re morphologically integrated (e.g., German compounds, Slavic case endings, Icelandic inflection). Without task-specific supervision, they don’t reliably link loan base ⇄ surface form (e.g., computer → computern/kompjuteru/-sins).
• Multiword loans get split or dropped. Phrases like social media, ex post, know-how are often partially detected (only one token) or ignored because the models don’t learn span boundaries for multi-token borrowings in zero-shot.
• Named entities vs. loanwords confusion. They over-flag proper names, acronyms, and titles (ISO, EU, personal surnames) as “loanwords,” and under-flag genuine loans that are fully nativized.
• Script & tokenization effects. In languages without whitespace tokenization or with rich clitic behavior (CJK, Greek, Kurdish orthographies), subword splitting seems to make span alignment brittle; loans fused with affixes are missed.
• Orthographic assimilation hurts recall. Once diacritics or phonology are localized, the model stops associating the word with the donor language (mazout, súkkulaði), so detection drops.
• Domain terms vs. loans. Technical vocabulary (cargo, monitoring, bug) gets inconsistently treated—sometimes correctly as loans, sometimes rejected as native/terminological.
• Sensitivity to casing & punctuation. Lower/upper case, hyphens, and quotes alter subword segmentation and produce unstable predictions across otherwise identical contexts.
• Language imbalance. Performance is typically higher for high-resource, Latin-script languages (EN-adjacent Romance/Germanic) and lower for morphologically rich or low-resource languages (e.g., Icelandic, Russian inflectional variants; Kurdish orthography variants).
• Span order is unstable. Even when the right items are found, ordering and boundary consistency vary example-to-example, amplifying metric penalties if evaluation is strict.

Bottom line: without fine-tuning, these encoders behave like general lexical/semantic matchers rather than borrowed-lexeme recognizers. They miss many true loans (recall issues)—especially inflected, compounded, or multiword cases—and over-flag proper names and cognates (precision issues).

Language-dependent performances

Overall scores are very low across the board (expected in zero-resource detection), but there’s clear structure. Russian comes out best on average (≈0.033 F1), followed by Northern-Kurdish (~0.024), then Italian, Portuguese, and French (~0.019–0.012). At the lower end sit Spanish (~0.010), German (~0.009), Chinese (~0.006), and especially Icelandic/Greek (~0.003).

By model, untrained mBERT has the strongest mean F1 and “wins” the largest number of languages (best score in 5 languages), XLM-R large is next (wins 3 languages), ELECTRA-base shows occasional wins (2), while XLM-R base lags considerably in average F1.

These patterns are consistent with: (i) tokenization and script coverage—mBERT’s WordPiece vocabulary [4] and broad multilingual pretraining seem to generalize slightly better to borrowed-word substrings; (ii) orthographic cues—languages where loanwords tend to surface as transparent Romanized forms or with diagnostic affixes (e.g., Italian/Portuguese/French) fare better than ones with compounding or rich inflection that obscures loanword boundaries (German, Icelandic, Greek); (iii) multi-word expressions and capitalization—untrained encoders often miss multi-token borrowings and proper-name borrowings, depressing F1 for languages with many such cases; and (iv) domain mismatch—if the pretraining distribution underrepresents certain languages/genres (e.g., Northern-Kurdish still does surprisingly well, but several scripts/genres are sparse), recall is especially brittle. The headline: without task supervision, mBERT > XLM-R large > ELECTRA-base >> XLM-R base, and language ranking reflects how saliently loanwords appear in surface form and how much morphology dilutes those cues.

Overall performances

Evaluated without any task-specific fine-tuning, the multilingual backbones show a consistent ranking in aggregate F1 across languages and prompts: mBERT (cased/uncased) > XLM-RoBERTa-large > ELECTRA-base (multilingual) > XLM-RoBERTa-base. While absolute scores remain modest for all models in the strict setting (as expected for zero-shot sequence tagging), the relative gaps are stable across prompts.

Overall performance of base multilingual encoders (zero-shot loanword detection)

Evaluated without any task-specific fine-tuning, the multilingual backbones show a consistent ranking in aggregate F1 across languages and prompts: mBERT (cased/uncased) > XLM-RoBERTa-large > ELECTRA-base (multilingual) > XLM-RoBERTa-base. Absolute scores are modest in the strict regime (expected for zero-shot sequence tagging), but this ordering is stable across prompts.

Why this ordering?

• mBERT on top.

  • Domain alignment: mBERT is trained on Wikipedia only; many evaluation sentences (parliamentary proceedings, newsy prose) are closer to Wikipedia’s register than CommonCrawl. That alignment helps lexical priors for named entities, internationalisms, and ISO-style terms (e.g., ISO, cargo).
  • WordPiece behavior: mBERT’s ~110k WordPiece vocabulary tends to conservatively segment OOV forms into a few stable chunks, which often preserves meaningful prefixes/stems of loanwords. This reduces spurious inside/outside flips and yields cleaner B-/I-span decisions in zero-shot.
  • Mature training recipe: Despite its age, mBERT’s multilingual balancing and cased/uncased variants provide robust coverage for high-resource Latin-script languages that dominate the dataset, boosting macro F1.

• XLM-R-large second.

  • Capacity vs. domain: XLM-R-large (1024-d, 24L) brings strong context modeling, which helps on morphologically rich or non-Latin scripts. However, CommonCrawl domain drift (noisier, varied orthography) can make it more permissive on content words that are not true loanwords, slightly hurting precision under strict scoring.
  • Tokenizer trade-offs: SentencePiece (250k) improves coverage but can over-merge frequent web n-grams into units that do not align with gold spans, creating boundary mismatches (penalized in strict evaluation).

• ELECTRA-base ahead of XLM-R-base.

  • Objective bias helps zero-shot tagging: ELECTRA’s Replaced-Token Detection trains a sharper token-level discriminator. In zero-shot, that bias often translates into higher precision on loanword spans (it avoids over-tagging), even if recall lags.
  • Smaller yet decisive: Although its capacity is similar to mBERT, the discriminator objective yields cleaner token decisions than XLM-R-base’s MLM at the same scale.

• XLM-R-base last.

  • Under-capacity for this task: With smaller width/depth than XLM-R-large, XLM-R-base lacks the contextual headroom to disambiguate borrowings from semantically related natives, especially in compounding or heavy inflection.
  • Span boundary issues: The same SentencePiece vocabulary as the large model but less capacity means more boundary errors survive to prediction time, which strict scoring punishes.

Takeaway

• Even without fine-tuning, mBERT offers the best zero-shot baseline here—likely due to Wikipedia domain alignment and stable WordPiece segmentation that matches gold spans.
• XLM-R-large is a close second and the most resilient on morphologically rich or non-Latin scripts; with task adaptation it would be expected to overtake.
• ELECTRA-base provides precise but conservative tagging, outperforming XLM-R-base, which appears capacity-limited for this span-sensitive task.
• However, overall the performance is very poor. The models mainly tend to randomly tag loanwords and non-loanwords, which leads to almost untracable trends and is probable cause by containing an untrained model detection head.

Finetuned multilingual model performance

Common mistakes

XLM-RoBERTa-Base

1) False negatives (gold=LOAN → pred=O)

Typical misses: golf, lápis, açúcar, megabytes, really, pogromo, mandatório Patterns

• Orthographic assimilation: Long-established borrowings with native spelling (golf, açúcar, lápis) are treated as in-vocabulary natives. The model’s subword segmentation and priors favor “native” readings when form and distribution match the host language.
• Scientific/technical items: Units and domain terms (megabytes, pegmatít, summa) often look like transparent compounds or transliterations; without explicit task signals the model under-tags these.
• Proper names & cultural terms: Borrowed ethnonyms/toponyms and culture-specific items (pogromo, Lehdê) are missed when frequency is low or when context supports a named-entity reading.
• Embedded English tokens: High-frequency English adverbs (really) inside non-English sentences are discounted as code-switch noise rather than lexical borrowings.

Trend: Under-detection when the borrowing is fully assimilated, domain-specific, or named-entity-like.

2) False positives (gold=O → pred=LOAN)

Typical FPs: krem, champanhe, filosofia, bolo, sala, set, baríum, 尼日利亚 Patterns

• International/Greco-Latin look-alikes: Native realizations of classical morphemes (filosofia, baríum, klóríð) are mistaken for loans because form overlaps cross-lingually.
• Short high-ambiguity words: bolo, sala, set share surface forms with other languages; short tokens are easily over-generalized.
• Proper nouns & toponyms: Place/person names (e.g., 尼日利亚) are over-tagged as loans when the model conflates onomastics with lexical borrowing.
• Recent but conventionalized terms: Items like computador (PT) or vegan (DE) are treated as “foreign” despite being the only conventional option in the language.

Trend: Over-tagging whenever orthographic similarity to international vocabulary is high, or proper names are present.

XLM-RoBERTa-Large

Relative to base

• Recall improves: The larger model captures more true loans, especially longer or more technical forms (capacity helps contextual disambiguation).
• Precision plateaus: It still over-predicts for Greco-Latin look-alikes and proper names; the fundamental ambiguity remains.
• Shared failure modes: Proper nouns, assimilated loans, and classical morphology remain the hardest cases.

Net effect: Higher recall with similar precision, so overall fewer FNs but persistent FP patterns.

ELECTRA-Base (multilingual)

Aggregate behavior

• Many FNs and FPs (observed ≈560 each), indicating conservative detection in some contexts and over-flagging in others.

False negatives

• Misses Slavic technical/economic items and East-Asian tokens (e.g., девальвировать, стенд, фондовый, 飛船上), suggesting weaker multilingual lexical priors than RoBERTa.

False positives

• Over-tags frequent natives (e.g., molhado, меры, ближайший) and brand/technical terms (Vodka, pólýester) as loans.
• ELECTRA’s replaced-token detection objective encourages sharp token decisions; absent task supervision, that bias can manifest as over-confident mislabels.

mBERT-Base

Relative positioning

• Better overall than ELECTRA; below XLM-R-large.
• You observed FN≈385, FP≈387.

False negatives

• Misses more than XLM-R-large (and slightly more than XLM-R-base) on Slavic/East-Asian borrowings (e.g., национальную, saldo, 飛船上), but not as severely as ELECTRA.

False positives

• Similar rate to XLM-R-base (~387), concentrated in short Mandarin tokens and morphologically complex natives, reflecting subword boundary and script-mixing challenges.

Boundary errors

• Occasional O→I-LOAN mismatches, consistent with WordPiece segmentation interacting with multi-character words in non-Latin scripts.

Language dependent results

Overall pattern. Performance varies systematically by language and model. Languages with transparent orthography and high overlap with international vocabulary (e.g., Portuguese, Italian, Spanish, French) tend to score higher, while morphologically rich or low-resource languages (e.g., Icelandic, Northern-Kurdish) and non-segmented scripts (e.g., Chinese) rank lower. German often falls in the lower–middle range due to productive compounding and high ambiguity between native compounds and internationalisms.

Model differences.

• XLM-RoBERTa-large achieves the best recall across most languages after fine-tuning, particularly improving on long/technical loanwords (Romance and Russian).
• mBERT-base is competitive on Latin-script languages and relatively stable on Slavic; it remains more brittle on non-Latin scripts where WordPiece segmentation increases boundary errors.
• XLM-RoBERTa-base tracks mBERT closely but lags on recall for longer multiword or technical borrowings.
• ELECTRA-base shows the largest language variance: it can perform adequately on higher-resource Latin-script languages, but it is more error-prone on Chinese (tokenization) and Icelandic/Northern-Kurdish (complex morphology, sparser pretraining signal).

Compared to untrained baselines. Fine-tuning yields substantial gains for every language. The biggest absolute improvements appear in languages that were weakest without task supervision—Chinese (token boundary learning), Icelandic (rich inflection), and Northern-Kurdish (data scarcity, orthographic variation). Languages that already looked reasonable in zero-shot (e.g., Portuguese/Italian) still benefit, but with smaller relative gains. The overall ordering by difficulty remains similar pre- vs. post-training (Romance > Slavic ≈ Germanic > Kurdish/Greek > Chinese), but the gaps narrow once the models learn span boundaries and the task definition.

Takeaway. After fine-tuning, XLM-R-large leads on most languages, mBERT/XLM-R-base form the middle tier, and ELECTRA-base remains the most variable across scripts. The largest post-training gains accrue to languages where segmentation and morphology previously dominated errors, confirming that task-supervised span learning is essential for reliable loanword detection in multilingual settings.

Overall performance

After supervised fine-tuning on the loanword tagging objective, the overall ranking across languages is consistent: XLM-RoBERTa-large > mBERT-base ≈ XLM-RoBERTa-base > ELECTRA-base (multilingual). Absolute F1 improves substantially for all models versus their untrained variants, but the relative gaps remain stable.

Model-by-model takeaways

• XLM-RoBERTa-large achieves the strongest macro-F1 and the best recall. The larger capacity and sentence-piece vocabulary yield better span discovery for long/technical borrowings and multiword items. Precision is competitive; most residual errors are proper nouns and internationalisms with native derivations.

• mBERT-base is a close second overall. It maintains balanced precision/recall on Latin-script languages and Slavic, with modest degradation on non-segmented scripts. Its WordPiece segmentation sometimes fragments Chinese and highly inflected forms, limiting recall on short spans.

• XLM-RoBERTa-base trails mBERT slightly. It benefits from the same multilingual pretraining recipe as XLM-R-large but has less capacity, which shows up as lower recall on rarer or morphologically integrated loans. Precision is similar to mBERT.

• ELECTRA-base (multilingual) improves markedly with fine-tuning but remains last overall. Its discriminative pretraining tends to over-flag “foreign-looking” tokens (Greco-Latin scientific vocabulary, proper names) while still missing assimilated borrowings, leading to a less favorable precision–recall trade-off.

Effect of supervision. Compared to the untrained classifiers, fine-tuning delivers:

• Large recall gains from learning span boundaries and task semantics;
• Moderate precision gains from suppressing false alarms on high-overlap international terms;
• The biggest absolute improvements in languages that were previously hardest (Chinese, Icelandic, Northern-Kurdish), narrowing cross-language gaps.

Error profile (remaining). Residual errors are consistent across models: (i) proper nouns and organization names, (ii) assimilated loans with native orthography/morphology, and (iii) Greco-Latin scientific terms that look international but are conventional in the target language. These patterns suggest that adding gazetteers/lexicons, morphology-aware tagging, or span-level contrastive objectives could further improve precision without sacrificing recall.

---

Project summary & next steps

Scope. This project investigates automatic loanword detection across 10 languages using two complementary paradigms:

1. Prompted Large Language Models (LLMs) — Gemini-2.5-Flash-Lite, GPT-4.1, and Meta-Llama-3-8B-Instruct — evaluated in zero-shot and few-shot regimes under three prompt variants (with/without explicit definitions of “loanword”).
2. Multilingual token-classification encoders — mBERT, XLM-RoBERTa (base/large), and ELECTRA-base (multilingual) — tested without fine-tuning as baselines and then fine-tuned on ConLoan for BIO tagging (O, B-LOAN, I-LOAN).

Dataset. ConLoan provides sentence-level inputs with human-annotated loanword spans and native alternatives for German, Portuguese, Spanish, Greek, Russian, Italian, Icelandic, French, Northern-Kurdish, Chinese. Evaluation is span-based with Precision/Recall/F1, in strict form (exact span segmentation) and a relaxed variant (tolerant to tokenization of multi-word loans, e.g., ["social media"] ≡ ["social","media"]).

Methodological highlights.

• LLMs were constrained to return a Python list of strings; outputs were cached and parsed deterministically.
• For multilingual encoders, data were aligned to subwords via word_ids, labels projected to subpieces (first = B-LOAN, subsequent = I-LOAN, specials = -100), and models trained with cross-entropy; metrics computed with seqeval.
• Per-language breakdowns and error-inspection sheets were produced to diagnose failure modes (proper nouns, assimilated forms, multi-word spans, script/tokenization effects).

Key findings.

• LLMs (prompting only): Few-shot generally improves over zero-shot, but absolute F1 remains modest. Gemini tends to lead overall, OpenAI next, Llama behind; performance varies notably by language and prompt. Strict vs. relaxed can be counter-intuitive: relaxed boosts recall but often hurts precision (over-selection of salient nouns), so strict F1 is sometimes higher.
• Untrained multilingual encoders: As expected for zero-shot sequence labeling, scores are low. A stable ranking emerges: mBERT > XLM-R-large > ELECTRA-base > XLM-R-base. Errors concentrate on assimilated loans, multi-word items, proper names, and rich morphology.
• Fine-tuned encoders: Supervision yields substantial gains for all models. Overall ordering after fine-tuning stabilizes at XLM-R-large > mBERT ≈ XLM-R-base > ELECTRA-base. The largest absolute improvements appear in languages previously hardest (Chinese, Icelandic, Northern-Kurdish), indicating that span learning + task semantics address core failure modes.

What the pipeline produces.

• Overall metrics per model/prompt/regime (precision/recall/F1) as .xlsx.
• Per-language F1 tables (strict & relaxed).
• Error sheets with sentence context and mismatched tokens.
• Merged cross-model summaries in Excel and LaTeX for reporting.
• Saved checkpoints for fine-tuned encoders and JSON caches for LLM outputs.

Current limitations.

• Named entities vs. loans remain ambiguous without external knowledge.
• Multi-word expressions and morphological variants still cause boundary errors.
• LLMs rely on surface heuristics (capitalization, rarity, “foreign-looking” orthography), leading to FP bursts in administrative or technical prose.
• Cross-language data imbalance and tokenization (CJK, Kurdish orthographies, German/Icelandic compounding) still depress recall.

Where to go next (research & engineering roadmap)

1. Span-aware decoding. Add a CRF or span-classification head; experiment with boundary objectives (e.g., token-pair scoring) to reduce fragmentation of multi-word loans.

2. Lexicon-augmented training. Inject gazetteers/etymological lexicons (e.g., ISO lists, onomastic dictionaries) as features or soft constraints to disambiguate proper nouns and internationalisms.

3. Language-adaptive fine-tuning (LAFT). Per-language adapters or LoRA modules to target low-resource or morphologically rich languages; optionally mix with continued pretraining on in-domain text.

4. Contrastive objectives. Encourage separation between calques/native cognates vs. true borrowings using contrastive pairs (loan vs. near-native synonym).

5. Prompt-engineering for LLMs.

   • Structure outputs as JSON with char offsets, not tokens, to avoid segmentation ambiguity.
   • Calibrate with few-shot exemplars that emphasize multi-word integrity and exclusion of named entities.
   • Add self-consistency or reranking with a span validator.

6. Evaluation extensions.

   • Report entity-level micro/macro metrics and partial-match scores (IoU-style).
   • Add calibration curves and threshold sweeps for models that output probabilities.

7. Human-in-the-loop. Build an annotation UI, where a text can be inserted and loanwords will be detected. The user can then add any corrections that are found, which will help in the fine-tuning.

8. Model distillation. Distill fine-tuned large models into compact student models for efficient deployment across languages.

Bottom line. The project establishes reproducible baselines and fine-tuned detectors for multilingual loanword identification, clarifies common error modes, and provides a clear path toward higher precision/recall—via span-aware modeling, lexicon integration, language-adaptive tuning, and smarter prompting/decoding for LLMs.

---

Sources

[1] https://github.com/merilinsilva/CLoAn.git

[2] Haspelmath, Martin. (2009) "Lexical borrowing: Concepts and issues." Loanword Typology, S. 36, 10.1515/9783110218442.

[4] https://github.com/google-research/bert?utm_source=chatgpt.com