Gold laurel award crest

ACM Multimedia 2026 · Oral Presentation

Outstanding Paper Award

34th ACM International Conference on Multimedia
Rio de Janeiro, Brazil · November 10–14, 2026

ViSeLAVision-Grounded Semantic Learning and Alignment
for Scalable Multilingual Text-Based Person Retrieval

Xinyu Zhoua · Fan Lia · Shasha Chena · Min Caoa,* · Ahtamjan Ahmatb · Qiaoming Zhua · Mang Yec

aSoochow UniversitybThe Xinjiang Technical Institute of Physics & Chemistry, CAScWuhan University

*Corresponding author

MultiPedes corpus10.00MImage–Text Pairs1.25M Images · EN, ZH, FR, DE
CUHK-PEDES(M)77.94%Avg R@170.16% Avg mAP
+1.63 mAP over Bi-IRRA
UFineBench(M)91.52%Avg R@190.56% Avg mAP
91.65% English-only R@1
ICFG-PEDES(M)69.66%Avg R@1English-only: 70.53% R@1
44.65% mAP

Paper results with person-centric pretraining · Tables 1–3, arXiv v1 · Avg denotes the four-language mean

Overview

Multilingual text-based person retrieval

Abstract

Multilingual Text-Based Person Retrieval (TBPR) aims to retrieve individuals with texts in multiple languages. Despite its practical importance, it has been overlooked by existing research, which mainly focuses on English-only retrieval. The only prior attempt at multilingual TBPR suffers from (1) the absence of a large-scale, multilingual pretraining dataset and (2) a suboptimal model architecture for multilingual text encoding and cross-modal alignment, which employs a shared encoder that overlooks linguistic heterogeneity and relies on English as a pivot for cross-modal alignment, introducing English-centric bias. For this, we contribute MultiPedes, the first large-scale, pedestrian-centric multilingual pretraining dataset. It contains 10 million image-text pairs across four languages, constructed by a multilingual quality-aware data construction pipeline. We also propose ViSeLA (Vision-grounded Semantic Learning and Alignment), a novel framework featuring: (1) a semantic-aware multilingual learning module that extends the mixture-of-experts architecture by routing language-invariant semantics to specialized experts for improved multilingual feature learning; and (2) a vision-grounded cross-modal alignment module that employs a semantic probe to capture fine-grained semantics across modalities, enabling direct cross-modal alignment without relying on a pivot language. Extensive experiments on both multilingual and English-only TBPR datasets demonstrate the effectiveness of ViSeLA. The dataset and code are available at https://github.com/Flame-Chasers/ViSeLA.

Figure 1. (a) Multilingual TBPR, where person images are retrieved with texts in multiple languages. (b) Existing method adopts a shared text encoder for all languages and relies on an English-centered knowledge transfer paradigm for cross-lingual cross-modal alignment. (c) Our ViSeLA framework introduces Se-L for multilingual learning and Vi-A for vision-grounded cross-modal alignment.
01

MultiPedes 10M Corpus

A pedestrian-centric corpus with 10 million image–text pairs in four languages, built through consensus-guided filtering and multilingual translation.

02

Se-L: Semantic-Aware Multilingual MoE

Route language-invariant semantics to specialized experts to learn from equivalent concepts across linguistically diverse descriptions.

03

Vi-A: Vision-Grounded Cross-Modal Alignment

Use the image as a shared semantic anchor and align fine-grained multilingual cues directly through a learnable semantic probe.

Method

Semantic routing with a visual anchor

ViSeLA comprises a Semantic-aware multilingual Learning (Se-L) module for self-adaptive multilingual encoding and a Vision-grounded Cross-modal Alignment (Vi-A) module for fine-grained cross-modal alignment.

Figure 4. The Se-L module uses semantic information to route text features to specialized experts, while Vi-A uses the semantic probe to capture fine-grained cross-modal semantics and enable image-grounded alignment.
Semantic-aware learning

Se-L · Route by meaning

The semantic probe ranks token–query attention and selects the smallest expert set whose cumulative attention reaches the threshold δ. Selected experts are combined with normalized routing weights.

∑k ∈ Kᵢ Pi,k ≥ δ
Vision-grounded alignment

Vi-A · Share the visual anchor

Shared learnable queries probe both modalities through symmetric multi-head cross-attention. Corresponding semantic cues are aligned with LogSumExp fine-grained contrastive learning.

Sf = log ∑i exp(cos(v̂i, t̂i))
Training & retrieval

Align, shortlist, re-rank

Jointly train coarse and fine-grained contrastive alignment, masked language modeling and image–text matching. Retrieval first shortlists by global similarity, then re-ranks candidates with cross-modal matching.

ℒ = ℒcN-ITC + ℒfN-ITC
+ ℒMLM + λℒITM

Dataset

MultiPedes: a world of descriptions

MultiPedes contains 10 million image–text pairs across English, Chinese, French, and German. The construction pipeline has two stages: consensus-guided filtering of English image–text pairs, followed by multilingual text translation.

Table 1 · Comparison with mainstream TBPR pretraining datasets

Dataset#Images#TextsAvg. LenLanguageTTRCLIPScore
LUPerson-T1.30M1.30M27.39En0.880.22
LUPerson-M1.00M4.00M28.61En0.800.36
HAM-PEDES1.00M2.00M26.30En0.800.40
SynTBPR1.63M6.62M34.10En0.810.43
WebPerson5M10M23.34En0.810.40
MultiPedes1.25M10.00M38.11En, ZH, Fr, De0.870.40

Open research

Data & checkpoints

Release v1.0.0

The baseline router is a separate released configuration. Main-table ViSeLA uses MultiPedes pretraining and LogSumExp similarity. Load each checkpoint with its embedded configuration. Download SHA256 checksums.

Full benchmark images are obtained separately under the original dataset terms. The separate HAM JSON and the aligned Parquet archive are distinct annotation sources.

Four languages. One visual anchor.

See the shared semantics

Explore real CUHK-PEDES(M) test images and their parallel benchmark descriptions. Select a person to read the same sample across four languages.

CUHK-PEDES(M) · Test split
CUHK-PEDES test person 12216, original benchmark photograph
Person ID 12216train_query/p11144_s13323.jpg

EnglishEN

A woman wearing a purple shirt, a pair of black shoes, a long gray skirt and a black bag in her right hand.

中文ZH

一个穿着紫色衬衫、黑色鞋子、灰色长裙和右手拿着黑色包的女人。

FrançaisFR

Une femme portant un chemisier violet, des chaussures noires, une jupe grise longue et un sac noir dans sa main droite.

DeutschDE

Eine Frau trägt ein lila Hemd, ein Paar schwarze Schuhe, einen langen grauen Rock und einen schwarzen Beutel in ihrer rechten Hand.

CUHK-PEDES test person 12004, original benchmark photograph
Person ID 12004train_query/p8848_s17661.jpg

EnglishEN

A man wearing a blue and white stripe tank top, a pair of green pants and a pair of pink shoes.

中文ZH

一个男人穿着蓝色和白色条纹的背心,一条绿色的裤子和一双粉红色的鞋子。

FrançaisFR

Un homme portant un tank top à rayures bleues et blanches, un pantalon vert et des chaussures roses.

DeutschDE

Ein Mann trägt ein blau-weiß gestreiftes Tanktop, eine grüne Hose und rosa Schuhe.

CUHK-PEDES test person 12007, original benchmark photograph
Person ID 12007train_query/p260_s495.jpg

EnglishEN

The woman wears a teal t-shirt, black pants, and blue tennis shoes. She wears glasses and carries a red backpack.

中文ZH

这位女士穿着一件天蓝色的T恤,黑色裤子和蓝色网球鞋。她戴着眼镜,背着一个红色的背包。

FrançaisFR

La femme porte un t-shirt teal, des pantalons noirs et des chaussures de tennis bleues. Elle porte des lunettes et transporte un sac à dos rouge.

DeutschDE

Die Frau trägt ein teal-grünes T-Shirt, schwarze Hosen und blaue Tennisschuhe. Sie trägt Brille und trägt ein rotes Rucksack.

CUHK-PEDES test person 12648, original benchmark photograph
Person ID 12648train_query/p6995_s9272.jpg

EnglishEN

A woman wearing a pink shirt, a pair of black pants and a pair of white shoes.

中文ZH

一个女人穿着一件粉色的衬衫、一条黑色的裤子和一双白色的鞋子。

FrançaisFR

Une femme portant une chemise rose, un pantalon noir et des chaussures blanches.

DeutschDE

Eine Frau trägt ein rosa Hemd, ein Paar schwarze Hosen und ein Paar weiße Schuhe.

Original images and benchmark wording, including translation choices, are preserved. Color highlights mark corresponding attribute phrases. Source: v1.0.0 multilingual annotations · Caption 1 for each image · Images subject to CUHK-PEDES research terms.

Experiments

Benchmark results

Explore the full multilingual and English-only comparisons, then inspect the pretraining and component ablations. Values are reproduced from arXiv:2611.15328v1, Tables 2–5.

Bold cells mark each numeric column’s maximum within its comparison group. Pretraining settings remain separate.

Table 2 · Performance comparison on multilingual TBPR

Interactive benchmark explorer

Compare the paper results

Table 2 · arXiv v1
Benchmark

Avg R@1 (%)

Avg mAP (%)

Both bar scales start at 0 and end at 100%. Averages cover English, Chinese, French and German.
Scroll horizontally to inspect every language
Dataset / settingMethodEnglishChineseFrenchGermanAvg
R@1R@10mAPR@1R@10mAPR@1R@10mAPR@1R@10mAPR@1R@10mAP
CUHK-PEDES(M) · Without pre-training
CUHK-PEDES(M)CCLM74.6893.4265.7870.8392.8563.1471.1792.3062.8970.1992.2462.4771.7292.7063.57
CUHK-PEDES(M)X²-VLM76.9794.9667.8674.8194.1766.0075.1894.1866.1774.7194.3065.7375.4294.4066.44
CUHK-PEDES(M)CCRK70.1192.5362.1068.3191.9160.6968.2991.3660.3367.0190.7459.5468.4391.6460.67
CUHK-PEDES(M)Bi-IRRA78.8295.4769.6876.4394.7967.7976.4694.3567.2675.5794.3167.0776.8294.7367.95
CUHK-PEDES(M)ViSeLA78.6595.2170.3376.0894.1768.3076.6594.2368.3575.9294.1867.8776.8394.4568.71
CUHK-PEDES(M) · With pre-training
CUHK-PEDES(M)Bi-IRRA79.4395.6870.5176.5594.4968.0576.6094.3167.6575.7994.3867.8977.0994.7268.53
CUHK-PEDES(M)ViSeLA79.3795.1171.6877.2993.9469.7277.6694.1069.7177.4594.0269.5377.9494.2970.16
ICFG-PEDES(M) · Without pre-training
ICFG-PEDES(M)CCLM64.1485.5035.5361.8383.9834.0160.9583.5833.1661.4283.7233.8462.0984.2034.14
ICFG-PEDES(M)X²-VLM68.3987.5840.5766.5486.2839.1365.7485.6938.5966.3486.0238.7766.7586.3939.27
ICFG-PEDES(M)CCRK60.3183.3732.3257.9881.9930.9157.5781.6530.2056.5280.6329.2858.1081.9130.68
ICFG-PEDES(M)Bi-IRRA68.5387.7941.8266.7986.4440.2666.9086.4039.4467.0586.6239.9167.3286.8140.36
ICFG-PEDES(M)ViSeLA70.2887.8043.5768.2186.4941.9767.6086.4741.4568.1186.7141.7968.5586.8742.20
ICFG-PEDES(M) · With pre-training
ICFG-PEDES(M)Bi-IRRA70.6388.4743.1567.8186.6141.2467.5486.8641.0067.5186.8641.0968.3787.2041.62
ICFG-PEDES(M)ViSeLA71.3588.5144.8369.2687.0343.3768.8087.1642.5569.2387.1043.0269.6687.4543.45
UFineBench(M) · Without pre-training
UFineBench(M)CCLM85.8696.6284.1984.9896.4483.3783.5895.8481.8083.4295.7781.4984.4696.1782.71
UFineBench(M)X²-VLM89.3298.0287.8288.2097.9087.1287.9897.3986.2186.8697.2985.5288.0997.6586.67
UFineBench(M)CCRK81.0695.1479.2179.5894.4977.9878.6194.2076.8477.6393.9875.8779.2294.4577.48
UFineBench(M)Bi-IRRA90.4598.1889.6689.9898.0589.2289.3097.7388.3289.3997.7588.0289.7897.9388.81
UFineBench(M)ViSeLA91.4598.1990.1890.4198.0589.3489.9797.8988.8789.8797.6688.6190.4297.9589.25
UFineBench(M) · With pre-training
UFineBench(M)Bi-IRRA91.2698.2690.0190.6798.1389.7889.9397.8889.0389.3497.9288.3290.3098.0589.29
UFineBench(M)ViSeLA92.2798.3591.3491.5598.2090.6091.1798.0990.2091.1097.9990.0891.5298.1690.56

Table 3 · Performance comparison on English TBPR

MethodCUHK-PEDESICFG-PEDESUFineBench
R@1R@5R@10mAPR@1R@5R@10mAPR@1R@5R@10mAP
Without large-scale person-centric pretraining
IRRA73.3889.9393.7166.1363.4680.2585.8238.0683.5392.9495.9582.79
RaSa76.5190.2994.2569.3865.2880.4085.1241.2986.1394.5596.7384.17
TBPS-CLIP73.5488.1992.3565.3865.0580.3485.4739.8376.2188.9293.0975.07
CFAM75.6090.5394.3667.2765.3881.1786.3539.4288.5195.5897.4987.09
RDE75.9490.1494.1267.5667.6882.4787.3640.0685.3194.1596.7284.05
TAG-CLIP74.3888.3092.5967.1866.7881.3686.2343.60----
ViSeLA77.6291.2695.0069.4667.9481.9986.7941.7290.1096.3497.8588.76
With large-scale person-centric pretraining
PLIP75.3690.8694.87-66.1783.3788.94-78.8991.0294.6576.89
APTM76.5390.0494.1566.9168.5182.9987.5641.2287.5995.4297.2285.75
AUL77.2390.4394.41-69.1683.3288.37-84.6094.2896.6983.06
MLLM+IRRA76.8291.1694.4669.5567.0582.1687.3341.5189.1396.1097.9088.38
HAM77.9991.3495.0369.7269.9583.8888.3942.7290.9496.7798.2490.13
CAMeL77.2491.8095.1668.3268.7083.1188.3241.5878.8692.7996.1778.87
MRA77.2190.6694.4668.5068.9383.4688.0741.38----
ViSeLA79.4292.1195.4871.5370.5383.5288.1044.6591.6597.0098.2690.83

Table 4 · Ablation study on pretraining datasets

MethodPretraining datasetEnglishChineseFrenchGermanAvg
R@1R@10mAPR@1R@10mAPR@1R@10mAPR@1R@10mAPR@1R@10mAP
Bi-IRRA-78.8295.4769.6876.4394.7967.7976.4694.3567.2675.5794.3167.0776.8294.7367.95
LUPerson-M79.4395.6870.5176.5594.4968.0576.6094.3167.6575.7994.3867.8977.0994.7268.53
MultiPedes79.6495.5570.6076.8694.5768.5676.7794.5468.3876.6894.5768.0577.4994.8168.90
ViSeLA-78.6595.2170.3376.0894.1768.3076.6594.2368.3575.9294.1867.8776.8394.4568.71
LUPerson-M78.9595.1370.7676.3694.2068.5776.8994.2268.5476.2393.8668.4177.1194.3569.07
MultiPedes79.3795.1171.6877.2993.9469.7277.6694.1069.7177.4594.0269.5377.9494.2970.16

Table 5 · Ablation study on ViSeLA components

Se-L
No.Vi-AS-MoEV-MoEL-MoER@1R@5R@10mAP
1✓77.6790.9794.3969.54
2✓77.2190.7394.2069.56
3✓✓77.4991.0094.3669.78
4✓✓77.3790.9194.4269.61
5✓✓77.9490.9494.2970.16

S-MoE, V-MoE, and L-MoE denote semantic-aware, vanilla, and language-aware MoE, respectively. Results are averaged over CUHK-PEDES.

Reproduction

Quick Start

The commands below follow the repository workflow. The CPU dry run uses randomly initialized weights for pipeline smoke testing.

bash
git clone https://github.com/Flame-Chasers/ViSeLA.git
cd ViSeLA
conda env create -f environment_tbps_py310.yml
conda activate tbps_py310

# CPU pipeline smoke test with randomly initialized weights
python tools/demo_retrieval.py --dry-run --top-k 3

# Retrieval demo with a released checkpoint
python tools/demo_retrieval.py \
  --checkpoint /path/to/visela_cuhk_pedes_baseline_router.pth \
  --text-encoder FacebookAI/xlm-roberta-base \
  --top-k 3
bash
python scripts/convert_multipedes_parquet.py \
  --parquet-dir /path/to/multipedes_parquet \
  --image-root /path/to/images \
  --output-dir /path/to/multipedes_jsonl
bash
# Pretrain on MultiPedes (4 GPUs)
cp config/examples/pretrain_multipedes.yaml local_pretrain_multipedes.yaml
torchrun --nproc_per_node=4 Pretrain.py \
  --config local_pretrain_multipedes.yaml \
  --checkpoint /path/to/cclm_x2vlm_base.th \
  --output_dir output/pretrain/visela \
  --epoch 2

# Fine-tune on CUHK-PEDES(M)
cp config/examples/visela_cuhk_pedes.yaml local_visela_cuhk_pedes.yaml
torchrun --nproc_per_node=4 main.py --config local_visela_cuhk_pedes.yaml

# Evaluate a checkpoint
python main.py --config local_visela_cuhk_pedes.yaml --eval-only --resume /path/to/checkpoint.pth

Citation

BibTeX

bibtex
@inproceedings{zhou2026visela,
  title={Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval},
  author={Zhou, Xinyu and Li, Fan and Chen, Shasha and Cao, Min and Ahmat, Ahtamjan and Zhu, Qiaoming and Ye, Mang},
  booktitle={Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
  year={2026},
  doi={10.1145/3767308.3836472},
  url={https://arxiv.org/abs/2611.15328},
  note={Oral Presentation, Outstanding Paper Award}
}