shreyaspullehf/superconductor-search-v2

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'superconductivity',
    'We highlight the reproducibility and level of control over the electrical\nproperties of YBa$_2$Cu$_3$O$_7$ Josephson junctions fabricated with\nirradiation from a focused helium ion beam. Specifically, we show the results\nof electrical transport properties for several junctions fabricated using a\nlarge range of irradiation doses. At the lower end of this range, junctions\nexhibit superconductor-normal metal-superconductor (SNS) Josephson junction\nproperties. However, as dose increases there is a transition to electrical\ncharacteristics consistent with superconductor-insulator-superconductor (SIS)\njunctions. To investigate the uniformity of large numbers of helium ion\nJosephson junctions we fabricate arrays of both SNS and SIS Josephson junctions\ncontaining 20 connected in series. Electrical transport properties for these\narrays reveal very uniform junctions with no appreciable spread in critical\ncurrent or resistance.',
    'Non-invasive magnetic field sensing using optically - detected magnetic\nresonance of nitrogen-vacancy (NV) centers in diamond was used to study spatial\ndistribution of the magnetic induction upon penetration and expulsion of weak\nmagnetic fields in several representative superconductors. Vector magnetic\nfields were measured on the surface of conventional, Pb and Nb, and\nunconventional, LuNi$_2$B$_2$C, Ba$_{0.6}$K$_{0.4}$Fe$_2$As$_2$,\nBa(Fe$_{0.93}$Co$_{0.07}$)$_2$As$_2$, and CaKFe$_4$As$_4$, superconductors,\nwith diffraction - limited spatial resolution using variable - temperature\nconfocal system. Magnetic induction profiles across the crystal edges were\nmeasured in zero-field-cooled (ZFC) and field-cooled (FC) conditions. While all\nsuperconductors show nearly perfect screening of magnetic fields applied after\ncooling to temperatures well below the superconducting transition, $T_c$, a\nrange of very different behaviors was observed for Meissner expulsion upon\ncooling in static magnetic field from above $T_c$. Substantial conventional\nMeissner expulsion is found in LuNi$_2$B$_2$C, paramagnetic Meissner effect\n(PME) is found in Nb, and virtually no expulsion is observed in iron-based\nsuperconductors. In all cases, good correlation with macroscopic measurements\nof total magnetic moment is found. Our measurements of the spatial distribution\nof magnetic induction provide insight into microscopic physics of the Meissner\neffect.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3505, 0.3544],
#         [0.3505, 1.0000, 0.6777],
#         [0.3544, 0.6777, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: superconductor-eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7
cosine_accuracy@3	0.8
cosine_accuracy@5	0.9
cosine_accuracy@10	1.0
cosine_precision@1	0.7
cosine_precision@3	0.3667
cosine_precision@5	0.24
cosine_precision@10	0.13
cosine_recall@1	0.5
cosine_recall@3	0.7
cosine_recall@5	0.8
cosine_recall@10	0.9
cosine_ndcg@10	0.7651
cosine_mrr@10	0.7861
cosine_map@100	0.7098

Training Details

Training Dataset

Unnamed Dataset

• Size: 4,091 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 3 tokens mean: 7.98 tokens max: 41 tokens	min: 8 tokens mean: 160.85 tokens max: 256 tokens

• Samples:

  sentence_0	sentence_1
  codimension two lump solutions in string field theory and ta	We present some solutions for lumps in two dimensions in level-expanded string field theory, as well as in two tachyonic theories: pure tachyonic string field theory and pure $\phi^3$ theory. Much easier to handle, these theories might be used to help understanding solitonic features of string field theory. We compare lump solutions between these theories and we discuss some convergence issues.
  superconductivity explained	We review the current understanding of superconductivity in the quasi-one-dimensional organic conductors of the Bechgaard and Fabre salt families. We discuss the interplay between superconductivity, antiferromagnetism, and charge-density-wave fluctuations. The connection to recent experimental observations supporting unconventional pairing and the possibility of a triplet-spin order parameter for the superconducting phase is also presented.
  erez berg	Erez Berg- Theory of Strange Metals | Understanding "strange metal" phenomena - metallic behavior that deviates from that expected of an ordinary Fermi liquid down to the lowest measurable temperatures - is among the most puzzling open problems in condensed matter physics. Such phenomena are observed across many different strongly correlated materials. They seem tied to other interesting phenomena, such as quantum criticality and unconventional superconductivity. I will describe theoretical advances in understanding the possible or

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 4
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	superconductor-eval_cosine_ndcg@10
1.0	256	0.7651

Framework Versions

• Python: 3.9.6
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.8.0
• Accelerate: 1.10.1
• Datasets: 4.3.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}