IoannisKat1/bge_reranker_ft

CrossEncoder based on BAAI/bge-reranker-v2-m3

This is a Cross Encoder model finetuned from BAAI/bge-reranker-v2-m3 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: BAAI/bge-reranker-v2-m3
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['Who must be subject to a duty of professional secrecy?', '1.Each Member State shall provide by law for all of the following: (a)  the establishment of each supervisory authority; 4.5.2016 L 119/66   (b)  the qualifications and eligibility conditions required to be appointed as member of each supervisory authority; (c)  the rules and procedures for the appointment of the member or members of each supervisory authority; (d)  the duration of the term of the member or members of each supervisory authority of no less than four years, except for the first appointment after 24 May 2016, part of which may take place for a shorter period where that is necessary to protect the independence of the supervisory authority by means of a staggered appointment procedure; (e)  whether and, if so, for how many terms the member or members of each supervisory authority is eligible for reappointment; (f)  the conditions governing the obligations of the member or members and staff of each supervisory authority, prohibitions on actions, occupations and benefits incompatible therewith during and after the term of office and rules governing the cessation of employment.\n2.The member or members and the staff of each supervisory authority shall, in accordance with Union or Member State law, be subject to a duty of professional secrecy both during and after their term of office, with regard to any confidential information which has come to their knowledge in the course of the performance of their tasks or exercise of their powers. During their term of office, that duty of professional secrecy shall in particular apply to reporting by natural persons of infringements of this Regulation. Section 2 Competence, tasks and powers'],
    ['Who must be subject to a duty of professional secrecy?', 'Whoever, without authorization, copies, uses, discloses to another person, or in any way violates data of a computer or a computer program, which belong to the realm of state secrets, private secrets, business secrets, trade secrets, or privacy, shall be punished by imprisonment from three months to five years. Private computer data or programs should be considered all the data and programs that the legal holder keeps secret with justified interest, especially if the owner had taken security measures.\nIf the offender is in the service of the legal holder of the data, or the secret computer data and programs have a great economic value, the act shall be punished by imprisonment from one year to five years.\nIf the secret computer data and programs belong to the realm of military or diplomatic secrets, or of the security of the state, the act shall be punished according to Articles 146–147.\nThe offences of paragraphs 1–2 are prosecuted only upon complaint.\n'],
    ['Who must be subject to a duty of professional secrecy?', "1.Without prejudice to other tasks set out under this Regulation, each supervisory authority shall on its territory: (a)  monitor and enforce the application of this Regulation; (b)  promote public awareness and understanding of the risks, rules, safeguards and rights in relation to processing. Activities addressed specifically to children shall receive specific attention; (c)  advise, in accordance with Member State law, the national parliament, the government, and other institutions and bodies on legislative and administrative measures relating to the protection of natural persons' rights and freedoms with regard to processing; (d)  promote the awareness of controllers and processors of their obligations under this Regulation; (e)  upon request, provide information to any data subject concerning the exercise of their rights under this Regulation and, if appropriate, cooperate with the supervisory authorities in other Member States to that end; (f)  handle complaints lodged by a data subject, or by a body, organisation or association in accordance with Article 80, and investigate, to the extent appropriate, the subject matter of the complaint and inform the complainant of the progress and the outcome of the investigation within a reasonable period, in particular if further investigation or coordination with another supervisory authority is necessary; (g)  cooperate with, including sharing information and provide mutual assistance to, other supervisory authorities with a view to ensuring the consistency of application and enforcement of this Regulation; (h)  conduct investigations on the application of this Regulation, including on the basis of information received from another supervisory authority or other public authority; (i)  monitor relevant developments, insofar as they have an impact on the protection of personal data, in particular the development of information and communication technologies and commercial practices; (j)  adopt standard contractual clauses referred to in Article 28(8) and in point (d) of Article 46(2); (k)  establish and maintain a list in relation to the requirement for data protection impact assessment pursuant to Article 35(4); (l)  give advice on the processing operations referred to in Article 36(2); (m)  encourage the drawing up of codes of conduct pursuant to Article 40(1) and provide an opinion and approve such codes of conduct which provide sufficient safeguards, pursuant to Article 40(5); (n)  encourage the establishment of data protection certification mechanisms and of data protection seals and marks pursuant to Article 42(1), and approve the criteria of certification pursuant to Article 42(5); (o)  where applicable, carry out a periodic review of certifications issued in accordance with Article 42(7); 4.5.2016 L 119/68   (p)  draft and publish the criteria for accreditation of a body for monitoring codes of conduct pursuant to Article 41 and of a certification body pursuant to Article 43; (q)  conduct the accreditation of a body for monitoring codes of conduct pursuant to Article 41 and of a certification body pursuant to Article 43; (r)  authorise contractual clauses and provisions referred to in Article 46(3); (s)  approve binding corporate rules pursuant to Article 47; (t)  contribute to the activities of the Board; (u)  keep internal records of infringements of this Regulation and of measures taken in accordance with Article 58(2); and (v)  fulfil any other tasks related to the protection of personal data.\n2.Each supervisory authority shall facilitate the submission of complaints referred to in point (f) of paragraph 1 by measures such as a complaint submission form which can also be completed electronically, without excluding other means of communication.\n3.The performance of the tasks of each supervisory authority shall be free of charge for the data subject and, where applicable, for the data protection officer.\n4.Where requests are manifestly unfounded or excessive, in particular because of their repetitive character, the supervisory authority may charge a reasonable fee based on administrative costs, or refuse to act on the request. The supervisory authority shall bear the burden of demonstrating the manifestly unfounded or excessive character of the request."],
    ['Who must be subject to a duty of professional secrecy?', '1.Union or Member State law to which the data controller or processor is subject may restrict by way of a legislative measure the scope of the obligations and rights provided for in Articles 12 to 22 and Article 34, as well as Article 5 in so far as its provisions correspond to the rights and obligations provided for in Articles 12 to 22, when such a restriction respects the essence of the fundamental rights and freedoms and is a necessary and proportionate measure in a democratic society to safeguard: (a)  national security; (b)  defence; (c)  public security; 4.5.2016 L 119/46   (d)  the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, including the safeguarding against and the prevention of threats to public security; (e)  other important objectives of general public interest of the Union or of a Member State, in particular an important economic or financial interest of the Union or of a Member State, including monetary, budgetary and taxation a matters, public health and social security; (f)  the protection of judicial independence and judicial proceedings; (g)  the prevention, investigation, detection and prosecution of breaches of ethics for regulated professions; (h)  a monitoring, inspection or regulatory function connected, even occasionally, to the exercise of official authority in the cases referred to in points (a) to (e) and (g); (i)  the protection of the data subject or the rights and freedoms of others; (j)  the enforcement of civil law claims.\n2.In particular, any legislative measure referred to in paragraph 1 shall contain specific provisions at least, where relevant, as to: (a)  the purposes of the processing or categories of processing; (b)  the categories of personal data; (c)  the scope of the restrictions introduced; (d)  the safeguards to prevent abuse or unlawful access or transfer; (e)  the specification of the controller or categories of controllers; (f)  the storage periods and the applicable safeguards taking into account the nature, scope and purposes of the processing or categories of processing; (g)  the risks to the rights and freedoms of data subjects; and (h)  the right of data subjects to be informed about the restriction, unless that may be prejudicial to the purpose of the restriction. CHAPTER IV Controller and processor Section 1 General obligations'],
    ['Who must be subject to a duty of professional secrecy?', "1.Processing of personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person's sex life or sexual orientation shall be prohibited.\n2.Paragraph 1 shall not apply if one of the following applies: (a)  the data subject has given explicit consent to the processing of those personal data for one or more specified purposes, except where Union or Member State law provide that the prohibition referred to in paragraph 1 may not be lifted by the data subject; (b)  processing is necessary for the purposes of carrying out the obligations and exercising specific rights of the controller or of the data subject in the field of employment and social security and social protection law in so far as it is authorised by Union or Member State law or a collective agreement pursuant to Member State law providing for appropriate safeguards for the fundamental rights and the interests of the data subject; (c)  processing is necessary to protect the vital interests of the data subject or of another natural person where the data subject is physically or legally incapable of giving consent; (d)  processing is carried out in the course of its legitimate activities with appropriate safeguards by a foundation, association or any other not-for-profit body with a political, philosophical, religious or trade union aim and on condition that the processing relates solely to the members or to former members of the body or to persons who have regular contact with it in connection with its purposes and that the personal data are not disclosed outside that body without the consent of the data subjects; (e)  processing relates to personal data which are manifestly made public by the data subject; (f)  processing is necessary for the establishment, exercise or defence of legal claims or whenever courts are acting in their judicial capacity; (g)  processing is necessary for reasons of substantial public interest, on the basis of Union or Member State law which shall be proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and the interests of the data subject; (h)  processing is necessary for the purposes of preventive or occupational medicine, for the assessment of the working capacity of the employee, medical diagnosis, the provision of health or social care or treatment or the management of health or social care systems and services on the basis of Union or Member State law or pursuant to contract with a health professional and subject to the conditions and safeguards referred to in paragraph 3; (i)  processing is necessary for reasons of public interest in the area of public health, such as protecting against serious cross-border threats to health or ensuring high standards of quality and safety of health care and of medicinal products or medical devices, on the basis of Union or Member State law which provides for suitable and specific measures to safeguard the rights and freedoms of the data subject, in particular professional secrecy; 4.5.2016 L 119/38   (j)  processing is necessary for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes in accordance with Article 89(1) based on Union or Member State law which shall be proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and the interests of the data subject.\n3.Personal data referred to in paragraph 1 may be processed for the purposes referred to in point (h) of paragraph 2 when those data are processed by or under the responsibility of a professional subject to the obligation of professional secrecy under Union or Member State law or rules established by national competent bodies or by another person also subject to an obligation of secrecy under Union or Member State law or rules established by national competent bodies.\n4.Member States may maintain or introduce further conditions, including limitations, with regard to the processing of genetic data, biometric data or data concerning health."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'Who must be subject to a duty of professional secrecy?',
    [
        '1.Each Member State shall provide by law for all of the following: (a)  the establishment of each supervisory authority; 4.5.2016 L 119/66   (b)  the qualifications and eligibility conditions required to be appointed as member of each supervisory authority; (c)  the rules and procedures for the appointment of the member or members of each supervisory authority; (d)  the duration of the term of the member or members of each supervisory authority of no less than four years, except for the first appointment after 24 May 2016, part of which may take place for a shorter period where that is necessary to protect the independence of the supervisory authority by means of a staggered appointment procedure; (e)  whether and, if so, for how many terms the member or members of each supervisory authority is eligible for reappointment; (f)  the conditions governing the obligations of the member or members and staff of each supervisory authority, prohibitions on actions, occupations and benefits incompatible therewith during and after the term of office and rules governing the cessation of employment.\n2.The member or members and the staff of each supervisory authority shall, in accordance with Union or Member State law, be subject to a duty of professional secrecy both during and after their term of office, with regard to any confidential information which has come to their knowledge in the course of the performance of their tasks or exercise of their powers. During their term of office, that duty of professional secrecy shall in particular apply to reporting by natural persons of infringements of this Regulation. Section 2 Competence, tasks and powers',
        'Whoever, without authorization, copies, uses, discloses to another person, or in any way violates data of a computer or a computer program, which belong to the realm of state secrets, private secrets, business secrets, trade secrets, or privacy, shall be punished by imprisonment from three months to five years. Private computer data or programs should be considered all the data and programs that the legal holder keeps secret with justified interest, especially if the owner had taken security measures.\nIf the offender is in the service of the legal holder of the data, or the secret computer data and programs have a great economic value, the act shall be punished by imprisonment from one year to five years.\nIf the secret computer data and programs belong to the realm of military or diplomatic secrets, or of the security of the state, the act shall be punished according to Articles 146–147.\nThe offences of paragraphs 1–2 are prosecuted only upon complaint.\n',
        "1.Without prejudice to other tasks set out under this Regulation, each supervisory authority shall on its territory: (a)  monitor and enforce the application of this Regulation; (b)  promote public awareness and understanding of the risks, rules, safeguards and rights in relation to processing. Activities addressed specifically to children shall receive specific attention; (c)  advise, in accordance with Member State law, the national parliament, the government, and other institutions and bodies on legislative and administrative measures relating to the protection of natural persons' rights and freedoms with regard to processing; (d)  promote the awareness of controllers and processors of their obligations under this Regulation; (e)  upon request, provide information to any data subject concerning the exercise of their rights under this Regulation and, if appropriate, cooperate with the supervisory authorities in other Member States to that end; (f)  handle complaints lodged by a data subject, or by a body, organisation or association in accordance with Article 80, and investigate, to the extent appropriate, the subject matter of the complaint and inform the complainant of the progress and the outcome of the investigation within a reasonable period, in particular if further investigation or coordination with another supervisory authority is necessary; (g)  cooperate with, including sharing information and provide mutual assistance to, other supervisory authorities with a view to ensuring the consistency of application and enforcement of this Regulation; (h)  conduct investigations on the application of this Regulation, including on the basis of information received from another supervisory authority or other public authority; (i)  monitor relevant developments, insofar as they have an impact on the protection of personal data, in particular the development of information and communication technologies and commercial practices; (j)  adopt standard contractual clauses referred to in Article 28(8) and in point (d) of Article 46(2); (k)  establish and maintain a list in relation to the requirement for data protection impact assessment pursuant to Article 35(4); (l)  give advice on the processing operations referred to in Article 36(2); (m)  encourage the drawing up of codes of conduct pursuant to Article 40(1) and provide an opinion and approve such codes of conduct which provide sufficient safeguards, pursuant to Article 40(5); (n)  encourage the establishment of data protection certification mechanisms and of data protection seals and marks pursuant to Article 42(1), and approve the criteria of certification pursuant to Article 42(5); (o)  where applicable, carry out a periodic review of certifications issued in accordance with Article 42(7); 4.5.2016 L 119/68   (p)  draft and publish the criteria for accreditation of a body for monitoring codes of conduct pursuant to Article 41 and of a certification body pursuant to Article 43; (q)  conduct the accreditation of a body for monitoring codes of conduct pursuant to Article 41 and of a certification body pursuant to Article 43; (r)  authorise contractual clauses and provisions referred to in Article 46(3); (s)  approve binding corporate rules pursuant to Article 47; (t)  contribute to the activities of the Board; (u)  keep internal records of infringements of this Regulation and of measures taken in accordance with Article 58(2); and (v)  fulfil any other tasks related to the protection of personal data.\n2.Each supervisory authority shall facilitate the submission of complaints referred to in point (f) of paragraph 1 by measures such as a complaint submission form which can also be completed electronically, without excluding other means of communication.\n3.The performance of the tasks of each supervisory authority shall be free of charge for the data subject and, where applicable, for the data protection officer.\n4.Where requests are manifestly unfounded or excessive, in particular because of their repetitive character, the supervisory authority may charge a reasonable fee based on administrative costs, or refuse to act on the request. The supervisory authority shall bear the burden of demonstrating the manifestly unfounded or excessive character of the request.",
        '1.Union or Member State law to which the data controller or processor is subject may restrict by way of a legislative measure the scope of the obligations and rights provided for in Articles 12 to 22 and Article 34, as well as Article 5 in so far as its provisions correspond to the rights and obligations provided for in Articles 12 to 22, when such a restriction respects the essence of the fundamental rights and freedoms and is a necessary and proportionate measure in a democratic society to safeguard: (a)  national security; (b)  defence; (c)  public security; 4.5.2016 L 119/46   (d)  the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, including the safeguarding against and the prevention of threats to public security; (e)  other important objectives of general public interest of the Union or of a Member State, in particular an important economic or financial interest of the Union or of a Member State, including monetary, budgetary and taxation a matters, public health and social security; (f)  the protection of judicial independence and judicial proceedings; (g)  the prevention, investigation, detection and prosecution of breaches of ethics for regulated professions; (h)  a monitoring, inspection or regulatory function connected, even occasionally, to the exercise of official authority in the cases referred to in points (a) to (e) and (g); (i)  the protection of the data subject or the rights and freedoms of others; (j)  the enforcement of civil law claims.\n2.In particular, any legislative measure referred to in paragraph 1 shall contain specific provisions at least, where relevant, as to: (a)  the purposes of the processing or categories of processing; (b)  the categories of personal data; (c)  the scope of the restrictions introduced; (d)  the safeguards to prevent abuse or unlawful access or transfer; (e)  the specification of the controller or categories of controllers; (f)  the storage periods and the applicable safeguards taking into account the nature, scope and purposes of the processing or categories of processing; (g)  the risks to the rights and freedoms of data subjects; and (h)  the right of data subjects to be informed about the restriction, unless that may be prejudicial to the purpose of the restriction. CHAPTER IV Controller and processor Section 1 General obligations',
        "1.Processing of personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person's sex life or sexual orientation shall be prohibited.\n2.Paragraph 1 shall not apply if one of the following applies: (a)  the data subject has given explicit consent to the processing of those personal data for one or more specified purposes, except where Union or Member State law provide that the prohibition referred to in paragraph 1 may not be lifted by the data subject; (b)  processing is necessary for the purposes of carrying out the obligations and exercising specific rights of the controller or of the data subject in the field of employment and social security and social protection law in so far as it is authorised by Union or Member State law or a collective agreement pursuant to Member State law providing for appropriate safeguards for the fundamental rights and the interests of the data subject; (c)  processing is necessary to protect the vital interests of the data subject or of another natural person where the data subject is physically or legally incapable of giving consent; (d)  processing is carried out in the course of its legitimate activities with appropriate safeguards by a foundation, association or any other not-for-profit body with a political, philosophical, religious or trade union aim and on condition that the processing relates solely to the members or to former members of the body or to persons who have regular contact with it in connection with its purposes and that the personal data are not disclosed outside that body without the consent of the data subjects; (e)  processing relates to personal data which are manifestly made public by the data subject; (f)  processing is necessary for the establishment, exercise or defence of legal claims or whenever courts are acting in their judicial capacity; (g)  processing is necessary for reasons of substantial public interest, on the basis of Union or Member State law which shall be proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and the interests of the data subject; (h)  processing is necessary for the purposes of preventive or occupational medicine, for the assessment of the working capacity of the employee, medical diagnosis, the provision of health or social care or treatment or the management of health or social care systems and services on the basis of Union or Member State law or pursuant to contract with a health professional and subject to the conditions and safeguards referred to in paragraph 3; (i)  processing is necessary for reasons of public interest in the area of public health, such as protecting against serious cross-border threats to health or ensuring high standards of quality and safety of health care and of medicinal products or medical devices, on the basis of Union or Member State law which provides for suitable and specific measures to safeguard the rights and freedoms of the data subject, in particular professional secrecy; 4.5.2016 L 119/38   (j)  processing is necessary for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes in accordance with Article 89(1) based on Union or Member State law which shall be proportionate to the aim pursued, respect the essence of the right to data protection and provide for suitable and specific measures to safeguard the fundamental rights and the interests of the data subject.\n3.Personal data referred to in paragraph 1 may be processed for the purposes referred to in point (h) of paragraph 2 when those data are processed by or under the responsibility of a professional subject to the obligation of professional secrecy under Union or Member State law or rules established by national competent bodies or by another person also subject to an obligation of secrecy under Union or Member State law or rules established by national competent bodies.\n4.Member States may maintain or introduce further conditions, including limitations, with regard to the processing of genetic data, biometric data or data concerning health.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Dataset: gooaq-dev
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": false
  }
  

Metric	Value
map	0.5495 (+0.1120)
mrr@10	0.5495 (+0.1120)
ndcg@10	0.5642 (+0.0848)

Cross Encoder Classification

• Dataset: cls-dev
• Evaluated with CrossEncoderClassificationEvaluator

Metric	Value
accuracy	0.9975
accuracy_threshold	0.0001
f1	0.9987
f1_threshold	0.0001
precision	1.0
recall	0.9975
average_precision	1.0

Training Details

Training Dataset

Unnamed Dataset

• Size: 6,763 training samples
• Columns: query, response, and label
• Approximate statistics based on the first 1000 samples:

  	query	response	label
  type	string	string	int
  details	min: 26 characters mean: 75.78 characters max: 174 characters	min: 121 characters mean: 2527.85 characters max: 12618 characters	0: ~74.60% 1: ~25.40%

• Samples:

  query	response	label
  Who must be subject to a duty of professional secrecy?	1.Each Member State shall provide by law for all of the following: (a) the establishment of each supervisory authority; 4.5.2016 L 119/66 (b) the qualifications and eligibility conditions required to be appointed as member of each supervisory authority; (c) the rules and procedures for the appointment of the member or members of each supervisory authority; (d) the duration of the term of the member or members of each supervisory authority of no less than four years, except for the first appointment after 24 May 2016, part of which may take place for a shorter period where that is necessary to protect the independence of the supervisory authority by means of a staggered appointment procedure; (e) whether and, if so, for how many terms the member or members of each supervisory authority is eligible for reappointment; (f) the conditions governing the obligations of the member or members and staff of each supervisory authority, prohibitions on actions, occupations and benefits inco...	1
  Who must be subject to a duty of professional secrecy?	Whoever, without authorization, copies, uses, discloses to another person, or in any way violates data of a computer or a computer program, which belong to the realm of state secrets, private secrets, business secrets, trade secrets, or privacy, shall be punished by imprisonment from three months to five years. Private computer data or programs should be considered all the data and programs that the legal holder keeps secret with justified interest, especially if the owner had taken security measures. If the offender is in the service of the legal holder of the data, or the secret computer data and programs have a great economic value, the act shall be punished by imprisonment from one year to five years. If the secret computer data and programs belong to the realm of military or diplomatic secrets, or of the security of the state, the act shall be punished according to Articles 146–147. The offences of paragraphs 1–2 are prosecuted only upon complaint.	0
  Who must be subject to a duty of professional secrecy?	1.Without prejudice to other tasks set out under this Regulation, each supervisory authority shall on its territory: (a) monitor and enforce the application of this Regulation; (b) promote public awareness and understanding of the risks, rules, safeguards and rights in relation to processing. Activities addressed specifically to children shall receive specific attention; (c) advise, in accordance with Member State law, the national parliament, the government, and other institutions and bodies on legislative and administrative measures relating to the protection of natural persons' rights and freedoms with regard to processing; (d) promote the awareness of controllers and processors of their obligations under this Regulation; (e) upon request, provide information to any data subject concerning the exercise of their rights under this Regulation and, if appropriate, cooperate with the supervisory authorities in other Member States to that end; (f) handle complaints lodged by a data ...	0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 20
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• dataloader_num_workers: 4
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 20
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	gooaq-dev_ndcg@10	cls-dev_average_precision
-1	-1	-	0.5434 (+0.0640)	-
0.0024	1	0.1676	-	-
0.0473	20	0.249	-	-
0.0946	40	0.2914	-	-
0.1418	60	0.2758	-	-
0.1891	80	0.2923	-	-
0.2364	100	0.1707	-	-
0.2837	120	0.2452	-	-
0.3310	140	0.1872	-	-
0.3783	160	0.2231	-	-
0.4255	180	0.242	-	-
0.4728	200	0.2	-	-
0.5201	220	0.2104	-	-
0.5674	240	0.1601	-	-
0.6147	260	0.2107	-	-
0.6619	280	0.1892	-	-
0.7092	300	0.1564	-	-
0.7565	320	0.2265	-	-
0.8038	340	0.1537	-	-
0.8511	360	0.1611	-	-
0.8983	380	0.1893	-	-
0.9456	400	0.1901	-	-
0.9929	420	0.2341	-	-
1.0402	440	0.1475	-	-
1.0875	460	0.1256	-	-
1.1348	480	0.1482	-	-
1.1820	500	0.1877	-	-
1.2293	520	0.17	-	-
1.2766	540	0.1331	-	-
1.3239	560	0.1613	-	-
1.3712	580	0.1549	-	-
1.4184	600	0.1173	-	-
1.4657	620	0.2092	-	-
1.5130	640	0.2167	-	-
1.5603	660	0.0821	-	-
1.6076	680	0.1738	-	-
1.6548	700	0.2319	-	-
1.7021	720	0.194	-	-
1.7494	740	0.1161	-	-
1.7967	760	0.1973	-	-
1.8440	780	0.2089	-	-
1.8913	800	0.1717	-	-
1.9385	820	0.1449	-	-
1.9858	840	0.097	-	-
2.0331	860	0.1914	-	-
2.0804	880	0.122	-	-
2.1277	900	0.1189	-	-
2.1749	920	0.061	-	-
2.2222	940	0.2266	-	-
2.2695	960	0.0962	-	-
2.3168	980	0.1157	-	-
2.3641	1000	0.1509	0.5634 (+0.0839)	1.0
2.4113	1020	0.1773	-	-
2.4586	1040	0.1545	-	-
2.5059	1060	0.092	-	-
2.5532	1080	0.1731	-	-
2.6005	1100	0.0803	-	-
2.6478	1120	0.0969	-	-
2.6950	1140	0.1143	-	-
2.7423	1160	0.0911	-	-
2.7896	1180	0.1787	-	-
2.8369	1200	0.1111	-	-
2.8842	1220	0.1136	-	-
2.9314	1240	0.142	-	-
2.9787	1260	0.1501	-	-
3.0260	1280	0.0939	-	-
3.0733	1300	0.0578	-	-
3.1206	1320	0.0904	-	-
3.1678	1340	0.1505	-	-
3.2151	1360	0.0675	-	-
3.2624	1380	0.0886	-	-
3.3097	1400	0.0907	-	-
3.3570	1420	0.0819	-	-
3.4043	1440	0.1033	-	-
3.4515	1460	0.0929	-	-
3.4988	1480	0.0817	-	-
3.5461	1500	0.1282	-	-
3.5934	1520	0.0762	-	-
3.6407	1540	0.0554	-	-
3.6879	1560	0.0482	-	-
3.7352	1580	0.1593	-	-
3.7825	1600	0.0755	-	-
3.8298	1620	0.0943	-	-
3.8771	1640	0.0932	-	-
3.9243	1660	0.1	-	-
3.9716	1680	0.0805	-	-
4.0189	1700	0.0518	-	-
4.0662	1720	0.0723	-	-
4.1135	1740	0.0636	-	-
4.1608	1760	0.0482	-	-
4.2080	1780	0.0777	-	-
4.2553	1800	0.037	-	-
4.3026	1820	0.0525	-	-
4.3499	1840	0.0731	-	-
4.3972	1860	0.0759	-	-
4.4444	1880	0.0401	-	-
4.4917	1900	0.0523	-	-
4.5390	1920	0.0737	-	-
4.5863	1940	0.0903	-	-
4.6336	1960	0.0415	-	-
4.6809	1980	0.1643	-	-
4.7281	2000	0.0565	0.5572 (+0.0777)	1.0
4.7754	2020	0.1068	-	-
4.8227	2040	0.1043	-	-
4.8700	2060	0.059	-	-
4.9173	2080	0.049	-	-
4.9645	2100	0.0697	-	-
5.0118	2120	0.1064	-	-
5.0591	2140	0.0135	-	-
5.1064	2160	0.0352	-	-
5.1537	2180	0.0478	-	-
5.2009	2200	0.0445	-	-
5.2482	2220	0.0464	-	-
5.2955	2240	0.0414	-	-
5.3428	2260	0.0978	-	-
5.3901	2280	0.0441	-	-
5.4374	2300	0.0841	-	-
5.4846	2320	0.0296	-	-
5.5319	2340	0.0602	-	-
5.5792	2360	0.04	-	-
5.6265	2380	0.0832	-	-
5.6738	2400	0.0349	-	-
5.7210	2420	0.0536	-	-
5.7683	2440	0.0402	-	-
5.8156	2460	0.1068	-	-
5.8629	2480	0.057	-	-
5.9102	2500	0.0912	-	-
5.9574	2520	0.0555	-	-
6.0047	2540	0.0738	-	-
6.0520	2560	0.0754	-	-
6.0993	2580	0.0517	-	-
6.1466	2600	0.023	-	-
6.1939	2620	0.0254	-	-
6.2411	2640	0.0541	-	-
6.2884	2660	0.0202	-	-
6.3357	2680	0.0465	-	-
6.3830	2700	0.0577	-	-
6.4303	2720	0.0912	-	-
6.4775	2740	0.0233	-	-
6.5248	2760	0.0189	-	-
6.5721	2780	0.0613	-	-
6.6194	2800	0.0563	-	-
6.6667	2820	0.0311	-	-
6.7139	2840	0.0131	-	-
6.7612	2860	0.0861	-	-
6.8085	2880	0.035	-	-
6.8558	2900	0.1242	-	-
6.9031	2920	0.088	-	-
6.9504	2940	0.0292	-	-
6.9976	2960	0.0559	-	-
7.0449	2980	0.0267	-	-
7.0922	3000	0.0503	0.5642 (+0.0848)	1.0
7.1395	3020	0.031	-	-
7.1868	3040	0.0502	-	-
7.2340	3060	0.0336	-	-
7.2813	3080	0.0286	-	-
7.3286	3100	0.0541	-	-
7.3759	3120	0.0244	-	-
7.4232	3140	0.1121	-	-
7.4704	3160	0.0019	-	-
7.5177	3180	0.012	-	-
7.5650	3200	0.0853	-	-
7.6123	3220	0.0301	-	-
7.6596	3240	0.0582	-	-
7.7069	3260	0.0549	-	-
7.7541	3280	0.059	-	-
7.8014	3300	0.0066	-	-
7.8487	3320	0.0221	-	-
7.8960	3340	0.0395	-	-
7.9433	3360	0.0515	-	-
7.9905	3380	0.0513	-	-
8.0378	3400	0.0005	-	-
8.0851	3420	0.051	-	-
8.1324	3440	0.0013	-	-
8.1797	3460	0.0127	-	-
8.2270	3480	0.0413	-	-
8.2742	3500	0.0438	-	-
8.3215	3520	0.0203	-	-
8.3688	3540	0.016	-	-
8.4161	3560	0.0352	-	-
8.4634	3580	0.0145	-	-
8.5106	3600	0.0612	-	-
8.5579	3620	0.0146	-	-
8.6052	3640	0.0427	-	-
8.6525	3660	0.0833	-	-
8.6998	3680	0.0556	-	-
8.7470	3700	0.0429	-	-
8.7943	3720	0.094	-	-
8.8416	3740	0.0133	-	-
8.8889	3760	0.0162	-	-
8.9362	3780	0.0105	-	-
8.9835	3800	0.0418	-	-
9.0307	3820	0.0004	-	-
9.0780	3840	0.0319	-	-
9.1253	3860	0.0436	-	-
9.1726	3880	0.0146	-	-
9.2199	3900	0.0342	-	-
9.2671	3920	0.0271	-	-
9.3144	3940	0.0248	-	-
9.3617	3960	0.0231	-	-
9.4090	3980	0.0029	-	-
9.4563	4000	0.0316	0.5556 (+0.0761)	1.0000
9.5035	4020	0.0318	-	-
9.5508	4040	0.0018	-	-
9.5981	4060	0.0808	-	-
9.6454	4080	0.0389	-	-
9.6927	4100	0.0031	-	-
9.7400	4120	0.0452	-	-
9.7872	4140	0.0163	-	-
9.8345	4160	0.0576	-	-
9.8818	4180	0.0441	-	-
9.9291	4200	0.0398	-	-
9.9764	4220	0.0392	-	-
10.0236	4240	0.0038	-	-
10.0709	4260	0.0035	-	-
10.1182	4280	0.0006	-	-
10.1655	4300	0.0286	-	-
10.2128	4320	0.0665	-	-
10.2600	4340	0.0303	-	-
10.3073	4360	0.0149	-	-
10.3546	4380	0.0194	-	-
10.4019	4400	0.0139	-	-
10.4492	4420	0.0079	-	-
10.4965	4440	0.0236	-	-
10.5437	4460	0.0322	-	-
10.5910	4480	0.0274	-	-
10.6383	4500	0.0097	-	-
10.6856	4520	0.0284	-	-
10.7329	4540	0.0072	-	-
10.7801	4560	0.0131	-	-
10.8274	4580	0.0553	-	-
10.8747	4600	0.051	-	-
10.9220	4620	0.0851	-	-
10.9693	4640	0.0471	-	-
11.0165	4660	0.0016	-	-
11.0638	4680	0.0161	-	-
11.1111	4700	0.0236	-	-
11.1584	4720	0.022	-	-
11.2057	4740	0.0462	-	-
11.2530	4760	0.03	-	-
11.3002	4780	0.0077	-	-
11.3475	4800	0.0496	-	-
11.3948	4820	0.0015	-	-
11.4421	4840	0.0037	-	-
11.4894	4860	0.007	-	-
11.5366	4880	0.0588	-	-
11.5839	4900	0.0175	-	-
11.6312	4920	0.0311	-	-
11.6785	4940	0.037	-	-
11.7258	4960	0.0021	-	-
11.7730	4980	0.063	-	-
11.8203	5000	0.0071	0.5493 (+0.0698)	1.0
11.8676	5020	0.0729	-	-
11.9149	5040	0.0104	-	-
11.9622	5060	0.0318	-	-
12.0095	5080	0.0292	-	-
12.0567	5100	0.0252	-	-
12.1040	5120	0.0215	-	-
12.1513	5140	0.0379	-	-
12.1986	5160	0.0068	-	-
12.2459	5180	0.0253	-	-
12.2931	5200	0.0324	-	-
12.3404	5220	0.0379	-	-
12.3877	5240	0.0288	-	-
12.4350	5260	0.039	-	-
12.4823	5280	0.0175	-	-
12.5296	5300	0.0264	-	-
12.5768	5320	0.0172	-	-
12.6241	5340	0.0198	-	-
12.6714	5360	0.03	-	-
12.7187	5380	0.0563	-	-
12.7660	5400	0.0011	-	-
12.8132	5420	0.0175	-	-
12.8605	5440	0.0316	-	-
12.9078	5460	0.0376	-	-
12.9551	5480	0.0051	-	-
13.0024	5500	0.0214	-	-
13.0496	5520	0.0236	-	-
13.0969	5540	0.0489	-	-
13.1442	5560	0.01	-	-
13.1915	5580	0.0264	-	-
13.2388	5600	0.0029	-	-
13.2861	5620	0.024	-	-
13.3333	5640	0.0495	-	-
13.3806	5660	0.0055	-	-
13.4279	5680	0.0382	-	-
13.4752	5700	0.0447	-	-
13.5225	5720	0.0127	-	-
13.5697	5740	0.0252	-	-
13.6170	5760	0.0263	-	-
13.6643	5780	0.0227	-	-
13.7116	5800	0.0062	-	-
13.7589	5820	0.0255	-	-
13.8061	5840	0.0685	-	-
13.8534	5860	0.0295	-	-
13.9007	5880	0.0161	-	-
13.9480	5900	0.0158	-	-
13.9953	5920	0.0416	-	-
14.0426	5940	0.0067	-	-
14.0898	5960	0.0226	-	-
14.1371	5980	0.0179	-	-
14.1844	6000	0.0084	0.5591 (+0.0796)	1.0
14.2317	6020	0.0125	-	-
14.2790	6040	0.0025	-	-
14.3262	6060	0.0649	-	-
14.3735	6080	0.0763	-	-
14.4208	6100	0.0041	-	-
14.4681	6120	0.0007	-	-
14.5154	6140	0.0214	-	-
14.5626	6160	0.0182	-	-
14.6099	6180	0.0383	-	-
14.6572	6200	0.025	-	-
14.7045	6220	0.0068	-	-
14.7518	6240	0.0492	-	-
14.7991	6260	0.0118	-	-
14.8463	6280	0.0083	-	-
14.8936	6300	0.0056	-	-
14.9409	6320	0.0366	-	-
14.9882	6340	0.018	-	-
15.0355	6360	0.0016	-	-
15.0827	6380	0.0014	-	-
15.1300	6400	0.0136	-	-
15.1773	6420	0.0046	-	-
15.2246	6440	0.0322	-	-
15.2719	6460	0.0187	-	-
15.3191	6480	0.0081	-	-
15.3664	6500	0.0049	-	-
15.4137	6520	0.0286	-	-
15.4610	6540	0.0178	-	-
15.5083	6560	0.007	-	-
15.5556	6580	0.0154	-	-
15.6028	6600	0.0313	-	-
15.6501	6620	0.0054	-	-
15.6974	6640	0.0239	-	-
15.7447	6660	0.0012	-	-
15.7920	6680	0.0628	-	-
15.8392	6700	0.0356	-	-
15.8865	6720	0.0026	-	-
15.9338	6740	0.0149	-	-
15.9811	6760	0.0041	-	-
16.0284	6780	0.0288	-	-
16.0757	6800	0.0137	-	-
16.1229	6820	0.0052	-	-
16.1702	6840	0.0192	-	-
16.2175	6860	0.0004	-	-
16.2648	6880	0.012	-	-
16.3121	6900	0.0104	-	-
16.3593	6920	0.0373	-	-
16.4066	6940	0.0101	-	-
16.4539	6960	0.0081	-	-
16.5012	6980	0.0006	-	-
16.5485	7000	0.0038	0.5613 (+0.0818)	1.0
16.5957	7020	0.0261	-	-
16.6430	7040	0.0053	-	-
16.6903	7060	0.0035	-	-
16.7376	7080	0.0068	-	-
16.7849	7100	0.0082	-	-
16.8322	7120	0.0071	-	-
16.8794	7140	0.0209	-	-
16.9267	7160	0.0105	-	-
16.9740	7180	0.0004	-	-
17.0213	7200	0.0316	-	-
17.0686	7220	0.0069	-	-
17.1158	7240	0.0308	-	-
17.1631	7260	0.0009	-	-
17.2104	7280	0.0007	-	-
17.2577	7300	0.0601	-	-
17.3050	7320	0.0004	-	-
17.3522	7340	0.027	-	-
17.3995	7360	0.0079	-	-
17.4468	7380	0.0005	-	-
17.4941	7400	0.0069	-	-
17.5414	7420	0.023	-	-
17.5887	7440	0.002	-	-
17.6359	7460	0.0211	-	-
17.6832	7480	0.0056	-	-
17.7305	7500	0.0041	-	-
17.7778	7520	0.0081	-	-
17.8251	7540	0.0054	-	-
17.8723	7560	0.0115	-	-
17.9196	7580	0.0013	-	-
17.9669	7600	0.0107	-	-
18.0142	7620	0.0078	-	-
18.0615	7640	0.0036	-	-
18.1087	7660	0.006	-	-
18.1560	7680	0.0105	-	-
18.2033	7700	0.0022	-	-
18.2506	7720	0.0021	-	-
18.2979	7740	0.0088	-	-
18.3452	7760	0.0002	-	-
18.3924	7780	0.0233	-	-
18.4397	7800	0.0002	-	-
18.4870	7820	0.0202	-	-
18.5343	7840	0.0012	-	-
18.5816	7860	0.0186	-	-
18.6288	7880	0.0037	-	-
18.6761	7900	0.0093	-	-
18.7234	7920	0.0033	-	-
18.7707	7940	0.0196	-	-
18.8180	7960	0.0129	-	-
18.8652	7980	0.0175	-	-
18.9125	8000	0.026	0.5402 (+0.0607)	1.0
18.9598	8020	0.0149	-	-
19.0071	8040	0.0058	-	-
19.0544	8060	0.007	-	-
19.1017	8080	0.0086	-	-
19.1489	8100	0.0133	-	-
19.1962	8120	0.0053	-	-
19.2435	8140	0.0097	-	-
19.2908	8160	0.017	-	-
19.3381	8180	0.0089	-	-
19.3853	8200	0.0048	-	-
19.4326	8220	0.0036	-	-
19.4799	8240	0.0002	-	-
19.5272	8260	0.0096	-	-
19.5745	8280	0.0089	-	-
19.6217	8300	0.0002	-	-
19.6690	8320	0.002	-	-
19.7163	8340	0.0154	-	-
19.7636	8360	0.0054	-	-
19.8109	8380	0.0022	-	-
19.8582	8400	0.0078	-	-
19.9054	8420	0.0002	-	-
19.9527	8440	0.0078	-	-
20.0	8460	0.0086	-	-
-1	-1	-	0.5642 (+0.0848)	1.0

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.1.2
• Transformers: 4.51.3
• PyTorch: 2.8.0+cu126
• Accelerate: 1.11.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}