MultiPedes 10M Corpus
A pedestrian-centric corpus with 10 million image–text pairs in four languages, built through consensus-guided filtering and multilingual translation.
Paper results with person-centric pretraining · Tables 1–3, arXiv v1 · Avg denotes the four-language mean
Overview
Multilingual Text-Based Person Retrieval (TBPR) aims to retrieve individuals with texts in multiple languages. Despite its practical importance, it has been overlooked by existing research, which mainly focuses on English-only retrieval. The only prior attempt at multilingual TBPR suffers from (1) the absence of a large-scale, multilingual pretraining dataset and (2) a suboptimal model architecture for multilingual text encoding and cross-modal alignment, which employs a shared encoder that overlooks linguistic heterogeneity and relies on English as a pivot for cross-modal alignment, introducing English-centric bias. For this, we contribute MultiPedes, the first large-scale, pedestrian-centric multilingual pretraining dataset. It contains 10 million image-text pairs across four languages, constructed by a multilingual quality-aware data construction pipeline. We also propose ViSeLA (Vision-grounded Semantic Learning and Alignment), a novel framework featuring: (1) a semantic-aware multilingual learning module that extends the mixture-of-experts architecture by routing language-invariant semantics to specialized experts for improved multilingual feature learning; and (2) a vision-grounded cross-modal alignment module that employs a semantic probe to capture fine-grained semantics across modalities, enabling direct cross-modal alignment without relying on a pivot language. Extensive experiments on both multilingual and English-only TBPR datasets demonstrate the effectiveness of ViSeLA. The dataset and code are available at https://github.com/Flame-Chasers/ViSeLA.
A pedestrian-centric corpus with 10 million image–text pairs in four languages, built through consensus-guided filtering and multilingual translation.
Route language-invariant semantics to specialized experts to learn from equivalent concepts across linguistically diverse descriptions.
Use the image as a shared semantic anchor and align fine-grained multilingual cues directly through a learnable semantic probe.
Method
ViSeLA comprises a Semantic-aware multilingual Learning (Se-L) module for self-adaptive multilingual encoding and a Vision-grounded Cross-modal Alignment (Vi-A) module for fine-grained cross-modal alignment.
The semantic probe ranks token–query attention and selects the smallest expert set whose cumulative attention reaches the threshold δ. Selected experts are combined with normalized routing weights.
Shared learnable queries probe both modalities through symmetric multi-head cross-attention. Corresponding semantic cues are aligned with LogSumExp fine-grained contrastive learning.
Jointly train coarse and fine-grained contrastive alignment, masked language modeling and image–text matching. Retrieval first shortlists by global similarity, then re-ranks candidates with cross-modal matching.
Dataset
MultiPedes contains 10 million image–text pairs across English, Chinese, French, and German. The construction pipeline has two stages: consensus-guided filtering of English image–text pairs, followed by multilingual text translation.
| Dataset | #Images | #Texts | Avg. Len | Language | TTR | CLIPScore |
|---|---|---|---|---|---|---|
| LUPerson-T | 1.30M | 1.30M | 27.39 | En | 0.88 | 0.22 |
| LUPerson-M | 1.00M | 4.00M | 28.61 | En | 0.80 | 0.36 |
| HAM-PEDES | 1.00M | 2.00M | 26.30 | En | 0.80 | 0.40 |
| SynTBPR | 1.63M | 6.62M | 34.10 | En | 0.81 | 0.43 |
| WebPerson | 5M | 10M | 23.34 | En | 0.81 | 0.40 |
| MultiPedes | 1.25M | 10.00M | 38.11 | En, ZH, Fr, De | 0.87 | 0.40 |
Open research
297 aligned HAM-PEDES & WebPerson shards. 10,000,000 image–text pairs across four languages.
Parquet · tar.gzAnnotation archive4 languages999,007 images with separate English, Chinese, French and German annotations.
JSON · tar.gzDownstream annotations32 JSONsTrain/test annotations for CUHK-PEDES, ICFG-PEDES, RSTPReid and UFineBench in four languages.
JSON · tar.gzDownstream annotations16 JSONsMultilingual code-switch training annotations for the same four downstream benchmarks.
JSON · tar.gzModel weightsEpoch 1HAM multilingual + LUPerson bilingual. Four experts, four queries, LogSumExp similarity; routing temperature 1.0.
FP32 · .thModel weightsBaseline router76.78% Avg R@1 · 68.67% Avg mAP. CCLM-X2VLM initialization, average similarity, routing temperature 0.85.
FP32 · .pthThe baseline router is a separate released configuration. Main-table ViSeLA uses MultiPedes pretraining and LogSumExp similarity. Load each checkpoint with its embedded configuration. Download SHA256 checksums.
Full benchmark images are obtained separately under the original dataset terms. The separate HAM JSON and the aligned Parquet archive are distinct annotation sources.
Hugging Face Hub
Explore aligned multilingual annotations in the Dataset Viewer, stream them with 🤗 Datasets, and download checkpoints with their configurations.
2.50M aligned four-language tuples from HAM-PEDES and WebPerson, plus supplementary HAM annotations.
Parquet · Viewer · Streaming 🤗 Dataset4 benchmarksCUHK-PEDES, ICFG-PEDES, RSTPReid, and UFineBench annotations in English, Chinese, French, and German.
Parquet · Original JSON archives 🤗 ModelEpoch 1Multilingual pretraining weights for downstream fine-tuning, with four experts, four queries, and LogSumExp similarity.
PyTorch · Configurations · Checksums 🤗 ModelBaseline router76.78% Avg R@1 · 68.67% Avg mAP, with CCLM-X2VLM initialization and average similarity.
PyTorch · Configurations · ChecksumsFour languages. One visual anchor.
Explore real CUHK-PEDES(M) test images and their parallel benchmark descriptions. Select a person to read the same sample across four languages.

A woman wearing a purple shirt, a pair of black shoes, a long gray skirt and a black bag in her right hand.
一个穿着紫色衬衫、黑色鞋子、灰色长裙和右手拿着黑色包的女人。
Une femme portant un chemisier violet, des chaussures noires, une jupe grise longue et un sac noir dans sa main droite.
Eine Frau trägt ein lila Hemd, ein Paar schwarze Schuhe, einen langen grauen Rock und einen schwarzen Beutel in ihrer rechten Hand.

A man wearing a blue and white stripe tank top, a pair of green pants and a pair of pink shoes.
一个男人穿着蓝色和白色条纹的背心,一条绿色的裤子和一双粉红色的鞋子。
Un homme portant un tank top à rayures bleues et blanches, un pantalon vert et des chaussures roses.
Ein Mann trägt ein blau-weiß gestreiftes Tanktop, eine grüne Hose und rosa Schuhe.

The woman wears a teal t-shirt, black pants, and blue tennis shoes. She wears glasses and carries a red backpack.
这位女士穿着一件天蓝色的T恤,黑色裤子和蓝色网球鞋。她戴着眼镜,背着一个红色的背包。
La femme porte un t-shirt teal, des pantalons noirs et des chaussures de tennis bleues. Elle porte des lunettes et transporte un sac à dos rouge.
Die Frau trägt ein teal-grünes T-Shirt, schwarze Hosen und blaue Tennisschuhe. Sie trägt Brille und trägt ein rotes Rucksack.

A woman wearing a pink shirt, a pair of black pants and a pair of white shoes.
一个女人穿着一件粉色的衬衫、一条黑色的裤子和一双白色的鞋子。
Une femme portant une chemise rose, un pantalon noir et des chaussures blanches.
Eine Frau trägt ein rosa Hemd, ein Paar schwarze Hosen und ein Paar weiße Schuhe.
Original images and benchmark wording, including translation choices, are preserved. Color highlights mark corresponding attribute phrases. Source: v1.0.0 multilingual annotations · Caption 1 for each image · Images subject to CUHK-PEDES research terms.
Experiments
Explore the full multilingual and English-only comparisons, then inspect the pretraining and component ablations. Values are reproduced from arXiv:2611.15328v1, Tables 2–5.
Bold cells mark each numeric column’s maximum within its comparison group. Pretraining settings remain separate.
Interactive benchmark explorer
| Dataset / setting | Method | English | Chinese | French | German | Avg | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@10 | mAP | R@1 | R@10 | mAP | R@1 | R@10 | mAP | R@1 | R@10 | mAP | R@1 | R@10 | mAP | ||
| CUHK-PEDES(M) · Without pre-training | ||||||||||||||||
| CUHK-PEDES(M) | CCLM | 74.68 | 93.42 | 65.78 | 70.83 | 92.85 | 63.14 | 71.17 | 92.30 | 62.89 | 70.19 | 92.24 | 62.47 | 71.72 | 92.70 | 63.57 |
| CUHK-PEDES(M) | X²-VLM | 76.97 | 94.96 | 67.86 | 74.81 | 94.17 | 66.00 | 75.18 | 94.18 | 66.17 | 74.71 | 94.30 | 65.73 | 75.42 | 94.40 | 66.44 |
| CUHK-PEDES(M) | CCRK | 70.11 | 92.53 | 62.10 | 68.31 | 91.91 | 60.69 | 68.29 | 91.36 | 60.33 | 67.01 | 90.74 | 59.54 | 68.43 | 91.64 | 60.67 |
| CUHK-PEDES(M) | Bi-IRRA | 78.82 | 95.47 | 69.68 | 76.43 | 94.79 | 67.79 | 76.46 | 94.35 | 67.26 | 75.57 | 94.31 | 67.07 | 76.82 | 94.73 | 67.95 |
| CUHK-PEDES(M) | ViSeLA | 78.65 | 95.21 | 70.33 | 76.08 | 94.17 | 68.30 | 76.65 | 94.23 | 68.35 | 75.92 | 94.18 | 67.87 | 76.83 | 94.45 | 68.71 |
| CUHK-PEDES(M) · With pre-training | ||||||||||||||||
| CUHK-PEDES(M) | Bi-IRRA | 79.43 | 95.68 | 70.51 | 76.55 | 94.49 | 68.05 | 76.60 | 94.31 | 67.65 | 75.79 | 94.38 | 67.89 | 77.09 | 94.72 | 68.53 |
| CUHK-PEDES(M) | ViSeLA | 79.37 | 95.11 | 71.68 | 77.29 | 93.94 | 69.72 | 77.66 | 94.10 | 69.71 | 77.45 | 94.02 | 69.53 | 77.94 | 94.29 | 70.16 |
| ICFG-PEDES(M) · Without pre-training | ||||||||||||||||
| ICFG-PEDES(M) | CCLM | 64.14 | 85.50 | 35.53 | 61.83 | 83.98 | 34.01 | 60.95 | 83.58 | 33.16 | 61.42 | 83.72 | 33.84 | 62.09 | 84.20 | 34.14 |
| ICFG-PEDES(M) | X²-VLM | 68.39 | 87.58 | 40.57 | 66.54 | 86.28 | 39.13 | 65.74 | 85.69 | 38.59 | 66.34 | 86.02 | 38.77 | 66.75 | 86.39 | 39.27 |
| ICFG-PEDES(M) | CCRK | 60.31 | 83.37 | 32.32 | 57.98 | 81.99 | 30.91 | 57.57 | 81.65 | 30.20 | 56.52 | 80.63 | 29.28 | 58.10 | 81.91 | 30.68 |
| ICFG-PEDES(M) | Bi-IRRA | 68.53 | 87.79 | 41.82 | 66.79 | 86.44 | 40.26 | 66.90 | 86.40 | 39.44 | 67.05 | 86.62 | 39.91 | 67.32 | 86.81 | 40.36 |
| ICFG-PEDES(M) | ViSeLA | 70.28 | 87.80 | 43.57 | 68.21 | 86.49 | 41.97 | 67.60 | 86.47 | 41.45 | 68.11 | 86.71 | 41.79 | 68.55 | 86.87 | 42.20 |
| ICFG-PEDES(M) · With pre-training | ||||||||||||||||
| ICFG-PEDES(M) | Bi-IRRA | 70.63 | 88.47 | 43.15 | 67.81 | 86.61 | 41.24 | 67.54 | 86.86 | 41.00 | 67.51 | 86.86 | 41.09 | 68.37 | 87.20 | 41.62 |
| ICFG-PEDES(M) | ViSeLA | 71.35 | 88.51 | 44.83 | 69.26 | 87.03 | 43.37 | 68.80 | 87.16 | 42.55 | 69.23 | 87.10 | 43.02 | 69.66 | 87.45 | 43.45 |
| UFineBench(M) · Without pre-training | ||||||||||||||||
| UFineBench(M) | CCLM | 85.86 | 96.62 | 84.19 | 84.98 | 96.44 | 83.37 | 83.58 | 95.84 | 81.80 | 83.42 | 95.77 | 81.49 | 84.46 | 96.17 | 82.71 |
| UFineBench(M) | X²-VLM | 89.32 | 98.02 | 87.82 | 88.20 | 97.90 | 87.12 | 87.98 | 97.39 | 86.21 | 86.86 | 97.29 | 85.52 | 88.09 | 97.65 | 86.67 |
| UFineBench(M) | CCRK | 81.06 | 95.14 | 79.21 | 79.58 | 94.49 | 77.98 | 78.61 | 94.20 | 76.84 | 77.63 | 93.98 | 75.87 | 79.22 | 94.45 | 77.48 |
| UFineBench(M) | Bi-IRRA | 90.45 | 98.18 | 89.66 | 89.98 | 98.05 | 89.22 | 89.30 | 97.73 | 88.32 | 89.39 | 97.75 | 88.02 | 89.78 | 97.93 | 88.81 |
| UFineBench(M) | ViSeLA | 91.45 | 98.19 | 90.18 | 90.41 | 98.05 | 89.34 | 89.97 | 97.89 | 88.87 | 89.87 | 97.66 | 88.61 | 90.42 | 97.95 | 89.25 |
| UFineBench(M) · With pre-training | ||||||||||||||||
| UFineBench(M) | Bi-IRRA | 91.26 | 98.26 | 90.01 | 90.67 | 98.13 | 89.78 | 89.93 | 97.88 | 89.03 | 89.34 | 97.92 | 88.32 | 90.30 | 98.05 | 89.29 |
| UFineBench(M) | ViSeLA | 92.27 | 98.35 | 91.34 | 91.55 | 98.20 | 90.60 | 91.17 | 98.09 | 90.20 | 91.10 | 97.99 | 90.08 | 91.52 | 98.16 | 90.56 |
| Method | CUHK-PEDES | ICFG-PEDES | UFineBench | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@5 | R@10 | mAP | R@1 | R@5 | R@10 | mAP | R@1 | R@5 | R@10 | mAP | |
| Without large-scale person-centric pretraining | ||||||||||||
| IRRA | 73.38 | 89.93 | 93.71 | 66.13 | 63.46 | 80.25 | 85.82 | 38.06 | 83.53 | 92.94 | 95.95 | 82.79 |
| RaSa | 76.51 | 90.29 | 94.25 | 69.38 | 65.28 | 80.40 | 85.12 | 41.29 | 86.13 | 94.55 | 96.73 | 84.17 |
| TBPS-CLIP | 73.54 | 88.19 | 92.35 | 65.38 | 65.05 | 80.34 | 85.47 | 39.83 | 76.21 | 88.92 | 93.09 | 75.07 |
| CFAM | 75.60 | 90.53 | 94.36 | 67.27 | 65.38 | 81.17 | 86.35 | 39.42 | 88.51 | 95.58 | 97.49 | 87.09 |
| RDE | 75.94 | 90.14 | 94.12 | 67.56 | 67.68 | 82.47 | 87.36 | 40.06 | 85.31 | 94.15 | 96.72 | 84.05 |
| TAG-CLIP | 74.38 | 88.30 | 92.59 | 67.18 | 66.78 | 81.36 | 86.23 | 43.60 | - | - | - | - |
| ViSeLA | 77.62 | 91.26 | 95.00 | 69.46 | 67.94 | 81.99 | 86.79 | 41.72 | 90.10 | 96.34 | 97.85 | 88.76 |
| With large-scale person-centric pretraining | ||||||||||||
| PLIP | 75.36 | 90.86 | 94.87 | - | 66.17 | 83.37 | 88.94 | - | 78.89 | 91.02 | 94.65 | 76.89 |
| APTM | 76.53 | 90.04 | 94.15 | 66.91 | 68.51 | 82.99 | 87.56 | 41.22 | 87.59 | 95.42 | 97.22 | 85.75 |
| AUL | 77.23 | 90.43 | 94.41 | - | 69.16 | 83.32 | 88.37 | - | 84.60 | 94.28 | 96.69 | 83.06 |
| MLLM+IRRA | 76.82 | 91.16 | 94.46 | 69.55 | 67.05 | 82.16 | 87.33 | 41.51 | 89.13 | 96.10 | 97.90 | 88.38 |
| HAM | 77.99 | 91.34 | 95.03 | 69.72 | 69.95 | 83.88 | 88.39 | 42.72 | 90.94 | 96.77 | 98.24 | 90.13 |
| CAMeL | 77.24 | 91.80 | 95.16 | 68.32 | 68.70 | 83.11 | 88.32 | 41.58 | 78.86 | 92.79 | 96.17 | 78.87 |
| MRA | 77.21 | 90.66 | 94.46 | 68.50 | 68.93 | 83.46 | 88.07 | 41.38 | - | - | - | - |
| ViSeLA | 79.42 | 92.11 | 95.48 | 71.53 | 70.53 | 83.52 | 88.10 | 44.65 | 91.65 | 97.00 | 98.26 | 90.83 |
| Method | Pretraining dataset | English | Chinese | French | German | Avg | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@10 | mAP | R@1 | R@10 | mAP | R@1 | R@10 | mAP | R@1 | R@10 | mAP | R@1 | R@10 | mAP | ||
| Bi-IRRA | - | 78.82 | 95.47 | 69.68 | 76.43 | 94.79 | 67.79 | 76.46 | 94.35 | 67.26 | 75.57 | 94.31 | 67.07 | 76.82 | 94.73 | 67.95 |
| LUPerson-M | 79.43 | 95.68 | 70.51 | 76.55 | 94.49 | 68.05 | 76.60 | 94.31 | 67.65 | 75.79 | 94.38 | 67.89 | 77.09 | 94.72 | 68.53 | |
| MultiPedes | 79.64 | 95.55 | 70.60 | 76.86 | 94.57 | 68.56 | 76.77 | 94.54 | 68.38 | 76.68 | 94.57 | 68.05 | 77.49 | 94.81 | 68.90 | |
| ViSeLA | - | 78.65 | 95.21 | 70.33 | 76.08 | 94.17 | 68.30 | 76.65 | 94.23 | 68.35 | 75.92 | 94.18 | 67.87 | 76.83 | 94.45 | 68.71 |
| LUPerson-M | 78.95 | 95.13 | 70.76 | 76.36 | 94.20 | 68.57 | 76.89 | 94.22 | 68.54 | 76.23 | 93.86 | 68.41 | 77.11 | 94.35 | 69.07 | |
| MultiPedes | 79.37 | 95.11 | 71.68 | 77.29 | 93.94 | 69.72 | 77.66 | 94.10 | 69.71 | 77.45 | 94.02 | 69.53 | 77.94 | 94.29 | 70.16 | |
| Se-L | ||||||||
|---|---|---|---|---|---|---|---|---|
| No. | Vi-A | S-MoE | V-MoE | L-MoE | R@1 | R@5 | R@10 | mAP |
| 1 | ✓ | 77.67 | 90.97 | 94.39 | 69.54 | |||
| 2 | ✓ | 77.21 | 90.73 | 94.20 | 69.56 | |||
| 3 | ✓ | ✓ | 77.49 | 91.00 | 94.36 | 69.78 | ||
| 4 | ✓ | ✓ | 77.37 | 90.91 | 94.42 | 69.61 | ||
| 5 | ✓ | ✓ | 77.94 | 90.94 | 94.29 | 70.16 | ||
S-MoE, V-MoE, and L-MoE denote semantic-aware, vanilla, and language-aware MoE, respectively. Results are averaged over CUHK-PEDES.
Reproduction
The commands below follow the repository workflow. The CPU dry run uses randomly initialized weights for pipeline smoke testing.
git clone https://github.com/Flame-Chasers/ViSeLA.git
cd ViSeLA
conda env create -f environment_tbps_py310.yml
conda activate tbps_py310
# CPU pipeline smoke test with randomly initialized weights
python tools/demo_retrieval.py --dry-run --top-k 3
# Retrieval demo with a released checkpoint
python tools/demo_retrieval.py \
--checkpoint /path/to/visela_cuhk_pedes_baseline_router.pth \
--text-encoder FacebookAI/xlm-roberta-base \
--top-k 3python scripts/convert_multipedes_parquet.py \
--parquet-dir /path/to/multipedes_parquet \
--image-root /path/to/images \
--output-dir /path/to/multipedes_jsonl# Pretrain on MultiPedes (4 GPUs)
cp config/examples/pretrain_multipedes.yaml local_pretrain_multipedes.yaml
torchrun --nproc_per_node=4 Pretrain.py \
--config local_pretrain_multipedes.yaml \
--checkpoint /path/to/cclm_x2vlm_base.th \
--output_dir output/pretrain/visela \
--epoch 2
# Fine-tune on CUHK-PEDES(M)
cp config/examples/visela_cuhk_pedes.yaml local_visela_cuhk_pedes.yaml
torchrun --nproc_per_node=4 main.py --config local_visela_cuhk_pedes.yaml
# Evaluate a checkpoint
python main.py --config local_visela_cuhk_pedes.yaml --eval-only --resume /path/to/checkpoint.pthCitation
@inproceedings{zhou2026visela,
title={Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval},
author={Zhou, Xinyu and Li, Fan and Chen, Shasha and Cao, Min and Ahmat, Ahtamjan and Zhu, Qiaoming and Ye, Mang},
booktitle={Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
year={2026},
doi={10.1145/3767308.3836472},
url={https://arxiv.org/abs/2611.15328},
note={Oral Presentation, Outstanding Paper Award}
}