KaniTTS2-en

Text-to-Speech Model with Frame-level Position Encodings, optimized for Realtime Conversations

Overview

KaniTTS2 uses a two-stage pipeline combining a LLM with an FSQ audio codec.

Key Specifications:

Model Size: 400M parameters
Sample Rate: 22kHz
Languages: English

Audio Examples

Speaker	Audio
🇺🇸 Frank from Boston
🇺🇸 Jermaine from Oakland
🏴󠁧󠁢󠁳󠁣󠁴󠁿 Rory from Glasgow
🏴󠁧󠁢󠁥󠁮󠁧󠁿 Baddy from Liverpool
🇺🇸 Chelsea from New York
🇺🇸 Andrew from San Francisco

Installation

pip install kani-tts-2
pip install -U "transformers==4.56.0"

Quickstart

from kani_tts import KaniTTS

# Initialize model
model = KaniTTS('repo/model')

# Generate speech
audio, text = model("Hello, world!")

# Save to file
model.save_audio(audio, "output.wav")

Voice Cloning

from kani_tts import KaniTTS, SpeakerEmbedder

# Initialize models
model = KaniTTS('repo/model')
embedder = SpeakerEmbedder()

# Extract speaker embedding from reference audio
speaker_embedding = embedder.embed_audio_file("reference_voice.wav")

# Generate speech with cloned voice
audio, text = model(
    "This is a cloned voice speaking!",
    speaker_emb=speaker_embedding
)
model.save_audio(audio, "cloned_voice.wav")

For more examples including language tags, generation parameters, and the full API reference, see the kani-tts-2 package.

Performance

Nvidia RTX 5080 Benchmarks:

RTF: ~0.2 (using custom executor)
Memory: 3GB GPU VRAM

Pretraining:

Dataset: ~10k hours
Hardware: 8x H100 GPUs, 6 hours training.

Use Cases

Conversational AI: Real-time speech for chatbots and virtual assistants
Research: Fine-tuning for specific voices, accents, or emotions

Limitations

Performance degrades with inputs exceeding ~40sec
May inherit biases from training data in prosody or pronunciation
Optimized primarily for English; other languages may require continual pretraining

Optimization Tips

Multilingual Performance: Continually pretrain on target language datasets and fine-tune NanoCodec
Batch Processing: Use batches of 8-16 for high-throughput scenarios
Hardware: Optimized for NVIDIA Blackwell architecture GPUs

Resources

Models: Pretrained Model, English Model

Github: kani-tts-2 package.

Pretraining Framework Train your own TTS model on your language or accent from scratch using this open-source pretraining framework: KaniTTS2-Pretrain.

Example Dataset: https://huggingface.co/datasets/nineninesix/kanitts2-es-nano-codec-speaker-emb-dataset

Links: Website, Discord

vLLM Integration

Coming soon.

Tech Report

Coming soon.

Acknowledgments

Built on top of LiquidAI LFM2 350M as the backbone and Nvidia NanoCodec for audio processing.

Training data provided by the LAION team through the Emilia and EmoNet-Voice datasets.

Speaker embeddings extracted using Orange/Speaker-wavLM-tbr.

Citation


@article{liquidai2025lfm2,
 title={LFM2 Technical Report},
 author={Liquid AI},
 journal={arXiv preprint arXiv:2511.23404},
 year={2025}
}

@inproceedings{emilialarge,
  author={He, Haorui and Shang, Zengqiang and Wang, Chaoren and Li, Xuyuan and Gu, Yicheng and Hua, Hua and Liu, Liwei and Yang, Chen and Li, Jiaqi and Shi, Peiyang and Wang, Yuancheng and Chen, Kai and Zhang, Pengyuan and Wu, Zhizheng},
  title={Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation},
  booktitle={arXiv:2501.15907},
  year={2025}
}

@article{emonet_voice_2025,
  author={Schuhmann, Christoph and Kaczmarczyk, Robert and Rabby, Gollam and Friedrich, Felix and Kraus, Maurice and Nadi, Kourosh and Nguyen, Huu and Kersting, Kristian and Auer, Sören},
  title={EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection},
  journal={arXiv preprint arXiv:2506.09827},
  year={2025}
}

@inproceedings{gengembre24_interspeech,
  title     = {Disentangling prosody and timbre embeddings via voice conversion},
  author    = {Nicolas Gengembre and Olivier {Le Blouch} and Cédric Gendrot},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {2765--2769},
  doi       = {10.21437/Interspeech.2024-207},
  issn      = {2958-1796},
}

Responsible Use

Prohibited activities include:

Illegal content or harmful, threatening, defamatory, or obscene material
Hate speech, harassment, or incitement of violence
Generating false or misleading information
Impersonating individuals without consent
Malicious activities such as spamming, phishing, or fraud

By using this model, you agree to comply with these restrictions and all applicable laws.

If you use this code in your research, please cite:

@software{kani_tts_2,
  author = {Nineninesix},
  title = {KaniTTS2: Text-to-Speech Model with Frame-level Position Encoding},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://github.com/nineninesix-ai/kani-tts-2}},
  note = {Open-source TTS model}
}