--- license: apache-2.0 base_model: - meta-llama/Meta-Llama-3-8B-Instruct language: - es tags: - BEL - retrieval - entity-retrieval - named-entity-disambiguation - entity-disambiguation - named-entity-linking - entity-linking - text2text-generation - biomedical - healthcare - synthetic-data - causal-lm - llm library_name: transformers finetuning_task: - text2text-generation - entity-linking --- # SynCABEL: Synthetic Contextualized Augmentation for Biomedical Entity Linking ## SynCABEL **SynCABEL** is a novel framework that addresses data scarcity in biomedical entity linking through **synthetic data generation**. The method, introduced in our [paper] ## SynCABEL (SPACCC Edition) This is a **finetuned version of LLaMA-3-8B** trained on **SPACCC** using **[SynthSPACCC](https://huggingface.co/datasets/AnonymousARR42/SynCABEL)** (our synthetic dataset generated via the SynCABEL framework). | | | |--------|---------| | **Base Model** | [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | | **Training Data** | SPACCC (real) + [SynthSPACCC](https://huggingface.co/datasets/AnonymousARR42/SynCABEL) (synthetic) | | **Fine-tuning** | [Supervised Fine-Tuning](https://huggingface.co/docs/trl/en/sft_trainer) | ## Training Data Composition The model is trained on a mix of **human-annotated** and **synthetic** data: ``` SPACCC (human) : 27,799 examples SynthSPACCC (synthetic) : 1,813,463 examples ``` To ensure balanced learning, **human data is upsampled during training** so that each batch contains: ``` 50% human-annotated data 50% synthetic data ``` In other words, although SynthMM is larger, the model always sees a **1:1 ratio of human to synthetic examples**, preventing synthetic data from overwhelming human supervision. ## Usage ### Loading ```python import torch from transformers import AutoModelForCausalLM # Load the model (requires trust_remote_code for custom architecture) model = AutoModelForCausalLM.from_pretrained( "AnonymousARR42/SynCABEL_SPACCC", trust_remote_code=True, device_map="auto" ) ``` ### Unconstrained Generation ```python # Let the model freely generate concept names sentences = [ "El paciente con [embolia pulmonar masiva]{ENFERMEDAD} presentó signos de dificultad respiratoria.", "El paciente se sometió a una [angioplastia coronaria]{PROCEDIMIENTO} para restaurar el flujo sanguíneo." ] results = model.sample( sentences=sentences, constrained=False, num_beams=2, ) for i, beam_results in enumerate(results): print(f"Input: {sentences[i]}") mention = beam_results[0]["mention"] print(f"Mention: {mention}") for j, result in enumerate(beam_results): print( f"Beam {j+1}:\n" f"Predicted concept name:{result['pred_concept_name']}\n" f"Predicted code: {result['pred_concept_code']}\n" f"Beam score: {result['beam_score']:.3f}\n" ) ``` **Output:** ``` Input: El paciente con [embolia pulmonar masiva]{ENFERMEDAD} presentó signos de dificultad respiratoria. Mention: embolia pulmonar masiva Beam 1: Predicted concept name:tromboembolia pulmonar masiva aguda Predicted code: NO_CODE Beam score: 0.818 Beam 2: Predicted concept name:tromboembolia masiva Predicted code: 58417008 Beam score: 0.816 Input: El paciente se sometió a una [angioplastia coronaria]{PROCEDIMIENTO} para restaurar el flujo sanguíneo. Mention: angioplastia coronaria Beam 1: Predicted concept name:operaciones transluminales en arteria coronaria Predicted code: NO_CODE Beam score: 0.764 Beam 2: Predicted concept name:procedimiento en arteria coronaria Predicted code: NO_CODE Beam score: 0.728 ``` ### Constrained Decoding (Recommended for Entity Linking) ```python # Constrained to valid biomedical concepts sentences = [ "El paciente con [embolia pulmonar masiva]{ENFERMEDAD} presentó signos de dificultad respiratoria.", "El paciente se sometió a una [angioplastia coronaria]{PROCEDIMIENTO} para restaurar el flujo sanguíneo." ] results = model.sample( sentences=sentences, constrained=True, num_beams=2, ) for i, beam_results in enumerate(results): print(f"Input: {sentences[i]}") mention = beam_results[0]["mention"] print(f"Mention: {mention}") for j, result in enumerate(beam_results): print( f"Beam {j+1}:\n" f"Predicted concept name:{result['pred_concept_name']}\n" f"Predicted code: {result['pred_concept_code']}\n" f"Beam score: {result['beam_score']:.3f}\n" ) ``` **Output:** ``` Input: El paciente con [embolia pulmonar masiva]{ENFERMEDAD} presentó signos de dificultad respiratoria. Mention: embolia pulmonar masiva Beam 1: Predicted concept name:tromboembolia masiva Predicted code: 58417008 Beam score: 0.816 Beam 2: Predicted concept name:tromboembolia pulmonar aguda Predicted code: 707414004 Beam score: 0.763 Input: El paciente se sometió a una [angioplastia coronaria]{PROCEDIMIENTO} para restaurar el flujo sanguíneo. Mention: angioplastia coronaria Beam 1: Predicted concept name:operaciones transluminales en arteria pulmonar Predicted code: 175266007 Beam score: 0.238 Beam 2: Predicted concept name:operaciones transluminales en la arteria femoral o poplítea Predicted code: 265530008 Beam score: 0.182 ``` ## Scores Entity linking performance (Recall@1) on biomedical benchmarks. The best results are shown in **bold**, the second-best results are underlined, and the "Average" column reports the mean score across the four benchmarks. | Model | MM-ST21PV
(english) | QUAERO-MEDLINE
(french) | QUAERO-EMEA
(french) | SPACCC
(spanish) | Avg. | | :--- | :---: | :---: | :---: | :---: | :---: | | SciSpacy | 53.8 | 40.5 | 37.1 | 13.2 | 36.2 | | SapBERT | 51.1 | 50.6 | 49.8 | 33.9 | 46.4 | | CODER-all | 56.6 | 58.7 | 58.1 | 43.7 | 54.3 | | SapBERT-all | 64.6 | 74.7 | 67.9 | 47.9 | 63.8 | | ArboEL | 74.5 | 70.9 | 62.8 | 49.0 | 64.2 | | mBART-large | 65.5 | 61.5 | 58.6 | 57.7 | 60.8 | | + Guided inference | 70.0 | 72.8 | 71.1 | 61.8 | 68.9 | | **+ SynCABEL (Our method)** | 71.5 | 77.1 | 75.3 | 64.0 | 72.0 | | Llama-3-8B | 69.0 | 66.4 | 65.5 | 59.9 | 65.2 | | + Guided inference | 74.4 | 77.5 | 72.9 | 64.2 | 72.3 | | **+ SynCABEL (Our method)** | **75.4** | **79.7** | **79.0** | **67.0** | **75.3** | Here, we provide the source repositories for the baselines: - [**SciSpacy**](https://github.com/allenai/scispacy) - [**SapBERT**](https://hf.co/cambridgeltl/SapBERT-from-PubMedBERT-fulltext) - [**SapBERT-all**](https://hf.co/cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR) - [**CODER-all**](https://hf.co/GanjinZero/coder_all) - [**ArboEL**](https://github.com/dhdhagar/arboEL) - [**mBART-large**](https://hf.co/facebook/mbart-large-50) - [**LLaMA-3-8B**](https://hf.co/meta-llama/Meta-Llama-3-8B-Instruct). ### Speed and Memory | Model | Model (GB) | Cand. (GB) | Speed (/s) | |--------------|------------|------------|------------| | SapBERT | 2.1 | 20.1 | **575.5** | | ArboEL | **1.2** | 7.1 | 38.9 | | mBART | 2.3 | **5.4** | 51.0 | | Llama-3-8B | 28.6 | **5.4** | 19.1 | *Measured on single H100 GPU, constrained decoding*