tomaarsen/reranker-msmarco-v1.1-ModernBERT-base-bce

CrossEncoder based on answerdotai/ModernBERT-base

This is a Cross Encoder model finetuned from answerdotai/ModernBERT-base on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("sentence_transformers_model_id")
# Get scores for pairs...
pairs = [
    ['what popular gemstone sounds like peridotite', 'Peridot (pronounced pair-a-doe) is the gem variety of olivine. Olivine, which is actually not an official mineral, is composed of two minerals: fayalite and forsterite. Fayalite is the iron rich member with a pure formula of Fe2SiO4. Forsterite is the magnesium rich member with a pure formula of Mg2SiO4.'],
    ['how much does it cost to keep an inmate in jail', 'The cost of each prison varies, depending on the types of inmates who are housed there. For example, it costs $99.12 a day to house an inmate at a reception center, because the inmates residing there are being evaluated and tested medically, psychologically, academically, vocationally, etc. In contrast, a typical adult male facility costs just $44.96 per day to house an inmate (excluding private prisons). When you average all types of state prison facilities together like those listed in the chart below, the daily cost to house an inmate is $55.09.'],
    ['what process is found in both cellular respiration and photosynthesis', 'Cellular respiration takes place in the same way in both plants and animals. Living cells obtain the products of photosynthesis (sugar molecules) and undergo cellular respiration to produce ATP molecules. Some cells respire aerobically, using oxygen, while others undergo anaerobic respiration, without using oxygen. ☞ Cellular respiration takes place in the cytoplasm and mitochondria of the cell. ☛ Photosynthesis uses water, sunlight, and CO 2 from the atmosphere to create glucose molecules, and releases oxygen as a by-product.'],
    ["how to keep someone at arm's length", "keep at arm's length from someone or something. Fig. to retain a degree of physical or social remoteness from someone or something. I try to keep at arm's length from Larry, since our disagreement. I keep Tom at arm's length because we don't get along. 1 keep someone or something around. 2  keep someone or something at arm's length. 3  keep someone or something at bay. 4  keep someone or something hanging. 5  keep someone or something off. 6  keep someone or something on track. 7  keep someone or something under control"],
    ['which is a characteristic of something in the domain bacteria', 'Archaea were only shown to be a separate domain—through analysis of their RNA—in 1977. Many archaea thrive under the extreme conditions of hot sulfur pools or in minerals and rock deep inside the Earth. '],
]
scores = model.predict(pairs)
print(scores.shape)
# [5]

# ... or rank different texts based on similarity to a single text
ranks = model.rank(
    'what popular gemstone sounds like peridotite',
    [
        'Peridot (pronounced pair-a-doe) is the gem variety of olivine. Olivine, which is actually not an official mineral, is composed of two minerals: fayalite and forsterite. Fayalite is the iron rich member with a pure formula of Fe2SiO4. Forsterite is the magnesium rich member with a pure formula of Mg2SiO4.',
        'The cost of each prison varies, depending on the types of inmates who are housed there. For example, it costs $99.12 a day to house an inmate at a reception center, because the inmates residing there are being evaluated and tested medically, psychologically, academically, vocationally, etc. In contrast, a typical adult male facility costs just $44.96 per day to house an inmate (excluding private prisons). When you average all types of state prison facilities together like those listed in the chart below, the daily cost to house an inmate is $55.09.',
        'Cellular respiration takes place in the same way in both plants and animals. Living cells obtain the products of photosynthesis (sugar molecules) and undergo cellular respiration to produce ATP molecules. Some cells respire aerobically, using oxygen, while others undergo anaerobic respiration, without using oxygen. ☞ Cellular respiration takes place in the cytoplasm and mitochondria of the cell. ☛ Photosynthesis uses water, sunlight, and CO 2 from the atmosphere to create glucose molecules, and releases oxygen as a by-product.',
        "keep at arm's length from someone or something. Fig. to retain a degree of physical or social remoteness from someone or something. I try to keep at arm's length from Larry, since our disagreement. I keep Tom at arm's length because we don't get along. 1 keep someone or something around. 2  keep someone or something at arm's length. 3  keep someone or something at bay. 4  keep someone or something hanging. 5  keep someone or something off. 6  keep someone or something on track. 7  keep someone or something under control",
        'Archaea were only shown to be a separate domain—through analysis of their RNA—in 1977. Many archaea thrive under the extreme conditions of hot sulfur pools or in minerals and rock deep inside the Earth. ',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO, NanoNFCorpus and NanoNQ
• Evaluated with CERerankingEvaluator

Metric	NanoMSMARCO	NanoNFCorpus	NanoNQ
map	0.4939 (+0.0044)	0.3115 (+0.0411)	0.5612 (+0.1405)
mrr@10	0.4840 (+0.0065)	0.4840 (-0.0158)	0.5607 (+0.1340)
ndcg@10	0.5549 (+0.0145)	0.3503 (+0.0252)	0.6091 (+0.1085)

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with CENanoBEIREvaluator

Metric	Value
map	0.4555 (+0.0620)
mrr@10	0.5096 (+0.0416)
ndcg@10	0.5048 (+0.0494)

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 82,326 training samples
• Columns: query, passage, and label
• Approximate statistics based on the first 1000 samples:

  	query	passage	label
  type	string	string	int
  details	min: 11 characters mean: 34.37 characters max: 99 characters	min: 61 characters mean: 415.74 characters max: 949 characters	0: ~87.50% 1: ~12.50%

• Samples:

  query	passage	label
  honor meaning	Honor is also employed to signify integrity in a judge, courage in a soldier, and chastity in a woman. To deprive a woman of her honor is, in some cases, punished as a public wrong, and by an action for the recovery of damages done to the relative rights of a husband or a father. Vide Criminal conversation. 2.	0
  what happens if xmas day falls on weekend	1 If the holiday falls on a Saturday or Sunday and that day would not otherwise be a working day for the employee, the holiday is transferred to the following Monday or Tuesday so that the employee still gets a paid day off if the employee would usually work on these days.	0
  what is the longest recorded flight time of a chicken	According to several chicken trivia sites like Feathersite, Poultryhelp and also including the Guinness World Book of World records, the longest verified flight of a domestic … chicken is 13 seconds and 301.5 feet distance.	0

• Loss: BinaryCrossEntropyLoss

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 82,326 evaluation samples
• Columns: query, passage, and label
• Approximate statistics based on the first 1000 samples:

  	query	passage	label
  type	string	string	int
  details	min: 11 characters mean: 34.76 characters max: 99 characters	min: 92 characters mean: 420.95 characters max: 980 characters	0: ~85.50% 1: ~14.50%

• Samples:

  query	passage	label
  what popular gemstone sounds like peridotite	Peridot (pronounced pair-a-doe) is the gem variety of olivine. Olivine, which is actually not an official mineral, is composed of two minerals: fayalite and forsterite. Fayalite is the iron rich member with a pure formula of Fe2SiO4. Forsterite is the magnesium rich member with a pure formula of Mg2SiO4.	0
  how much does it cost to keep an inmate in jail	The cost of each prison varies, depending on the types of inmates who are housed there. For example, it costs $99.12 a day to house an inmate at a reception center, because the inmates residing there are being evaluated and tested medically, psychologically, academically, vocationally, etc. In contrast, a typical adult male facility costs just $44.96 per day to house an inmate (excluding private prisons). When you average all types of state prison facilities together like those listed in the chart below, the daily cost to house an inmate is $55.09.	0
  what process is found in both cellular respiration and photosynthesis	Cellular respiration takes place in the same way in both plants and animals. Living cells obtain the products of photosynthesis (sugar molecules) and undergo cellular respiration to produce ATP molecules. Some cells respire aerobically, using oxygen, while others undergo anaerobic respiration, without using oxygen. ☞ Cellular respiration takes place in the cytoplasm and mitochondria of the cell. ☛ Photosynthesis uses water, sunlight, and CO 2 from the atmosphere to create glucose molecules, and releases oxygen as a by-product.	0

• Loss: BinaryCrossEntropyLoss

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_ndcg@10	NanoNFCorpus_ndcg@10	NanoNQ_ndcg@10	NanoBEIR_mean_ndcg@10
-1	-1	-	-	0.0381 (-0.5023)	0.3036 (-0.0214)	0.0388 (-0.4619)	0.1268 (-0.3285)
0.0002	1	0.8716	-	-	-	-	-
0.0384	200	0.4492	-	-	-	-	-
0.0768	400	0.382	-	-	-	-	-
0.1153	600	0.3863	-	-	-	-	-
0.1537	800	0.3776	-	-	-	-	-
0.1921	1000	0.364	0.3614	0.4437 (-0.0967)	0.2752 (-0.0498)	0.5125 (+0.0118)	0.4105 (-0.0449)
0.2305	1200	0.3529	-	-	-	-	-
0.2690	1400	0.3491	-	-	-	-	-
0.3074	1600	0.346	-	-	-	-	-
0.3458	1800	0.3411	-	-	-	-	-
0.3842	2000	0.3456	0.3465	0.5523 (+0.0119)	0.3359 (+0.0109)	0.5412 (+0.0406)	0.4765 (+0.0211)
0.4227	2200	0.3476	-	-	-	-	-
0.4611	2400	0.3391	-	-	-	-	-
0.4995	2600	0.3392	-	-	-	-	-
0.5379	2800	0.3355	-	-	-	-	-
0.5764	3000	0.349	0.3417	0.5289 (-0.0115)	0.3129 (-0.0122)	0.5651 (+0.0644)	0.4690 (+0.0136)
0.6148	3200	0.3342	-	-	-	-	-
0.6532	3400	0.3352	-	-	-	-	-
0.6916	3600	0.341	-	-	-	-	-
0.7301	3800	0.3383	-	-	-	-	-
0.7685	4000	0.3353	0.3364	0.5549 (+0.0145)	0.3503 (+0.0252)	0.6091 (+0.1085)	0.5048 (+0.0494)
0.8069	4200	0.3339	-	-	-	-	-
0.8453	4400	0.3275	-	-	-	-	-
0.8838	4600	0.3283	-	-	-	-	-
0.9222	4800	0.3293	-	-	-	-	-
0.9606	5000	0.3368	0.3351	0.5314 (-0.0090)	0.3221 (-0.0030)	0.5952 (+0.0946)	0.4829 (+0.0275)
0.9990	5200	0.3298	-	-	-	-	-
-1	-1	-	-	0.5549 (+0.0145)	0.3503 (+0.0252)	0.6091 (+0.1085)	0.5048 (+0.0494)

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.10
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.49.0.dev0
• PyTorch: 2.6.0.dev20241112+cu121
• Accelerate: 1.2.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}