sharvesh007/mt5-small-tamil-tts

mT5-Small — Tamil TTS (Vocab-Pruned, Tying-Fixed)

Vocab-pruned google/mt5-small for Tamil TTS training.

Critical Fix (v2)

Weight tying is now correct. Only shared.weight is stored in the checkpoint. lm_head.weight, encoder.embed_tokens.weight, and decoder.embed_tokens.weight are all tied to shared.weight at load time via tie_word_embeddings=True.

Previous version saved both shared.weight and lm_head.weight, causing HuggingFace to UNTIE them silently, leading to divergent matrices during training and exploding logits.

Why mT5 for TTS?

Text -> audio tokens is a true seq2seq problem:

• Encoder reads Tamil text (bidirectional, deep understanding)
• Decoder generates FSQ audio tokens (cross-attention to text)
• Cross-attention lets each audio token attend to specific characters

What Changed

	Original	Pruned
Vocab	250,112	50,001
shared.weight	250,112x512	50,001x512
lm_head	separate	tied to shared
Transformer	~44M	~44M (all 8+8 layers untouched)
Total params	556,291,456	69,662,592
Reduction	--	87%

Training Notes

IMPORTANT: When adding CV3 tokens, resize in float32 then cast:

model = AutoModelForSeq2SeqLM.from_pretrained("sharvesh007/mt5-small-tamil-tts", torch_dtype=torch.float32)
tokenizer.add_tokens([f"<cv3_{i}>" for i in range(6561)])
model.resize_token_embeddings(len(tokenizer))
# Random init for new tokens (break mean-init symmetry)
with torch.no_grad():
    cv3_start = tokenizer.convert_tokens_to_ids("<cv3_0>")
    d = model.config.d_model
    std = (2.0 / (d + 1)) ** 0.5
    model.shared.weight[cv3_start:].normal_(0, std)
# NOW cast to bf16
model = model.to(torch.bfloat16)

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("sharvesh007/mt5-small-tamil-tts")
model = AutoModelForSeq2SeqLM.from_pretrained("sharvesh007/mt5-small-tamil-tts")
ids = tokenizer.encode("வணக்கம்", add_special_tokens=False)
print(f"IDs: {ids}, max: {max(ids)}")