SheikhNextMoE-Medical-v2

A production-ready Mixture of Experts (MoE) model optimized for medical Q&A, built with safety-first principles and ethical AI guidelines.

Model Details

• Architecture: SheikhNextMoE - Sparse Mixture of Experts with grouped-query attention
• Parameters: 576,079,872
• Vocabulary Size: 30000
• Hidden Dimension: 1024
• Number of Layers: 16
• Number of Experts: 4
• Top-K Routing: 2
• Attention Heads: 16

Training Configuration

• Training Date: 2026-01-18
• Training Samples: 1000
• Validation Samples: 100
• Epochs: 3
• Final Training Loss: 5.5
• Final Validation Loss: 5.5
• Perplexity: N/A

Intended Use

This model is designed for medical question-answering applications, providing:

• Accurate medical information responses
• Safety-first approach with appropriate disclaimers
• Support for various medical domains (cardiology, psychiatry, etc.)

Training Datasets

The model was trained on verified medical datasets:

1. medalpaca/medical_meadow_medical_flashcards - Medical flashcards and Q&A
2. ccdv/pubmed-summarization - Medical document summarization

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_name = "OsamaBinLikhon/SheikhNextMoE-Medical-v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Medical query
prompt = "### Patient: What are symptoms of diabetes?
### Medical Assistant:"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate response
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Safety Notice

⚠️ Important: This model provides medical information for educational purposes only. It is not a substitute for professional medical advice, diagnosis, or treatment. Always consult qualified healthcare providers for medical concerns.

Safety Guidelines

• Model includes appropriate medical disclaimers
• Responses are designed to encourage professional consultation
• No diagnostic capabilities claimed
• Trained on verified, ethical medical datasets

License

Apache 2.0 - See LICENSE file for details.

Citation

@misc{SheikhNextMoE,
  title={SheikhNextMoE: Ethical Medical Assistant},
  author={MiniMax Agent},
  year={2025},
  url={https://huggingface.co/OsamaBinLikhon/SheikhNextMoE-Medical-v2}
}