---
license: apache-2.0
datasets:
- Skylion007/openwebtext
language:
- en
pipeline_tag: text-generation
tags:
- research
- convolutional
- fft
- transformer-alternative
- causal-lm
---

# GCLM — Global Convolutional Language Model

## Model Summary

**GCLM (Global Convolutional Language Model)** is an experimental causal language model that replaces traditional self-attention with a hybrid **local + global convolutional architecture**.

Instead of attention heads, GCLM uses:
- **Local depthwise convolutions** for short-range context
- **FFT-based global convolutions** for long-range sequence modeling

This design explores whether **global receptive fields** can be achieved efficiently *without* quadratic attention, while remaining compatible with standard autoregressive language modeling.

> GCLM is a transformer alternative — not a transformer replacement.

---

## Architecture Overview

- Token + learned positional embeddings
- Stacked convolutional blocks:
  - Local depthwise + pointwise convolution
  - Optional global FFT convolution every *N* layers
  - Feedforward MLP
  - Residual connections + LayerNorm
- Causal language modeling head

**Key properties:**
- No attention mechanism
- No KV cache
- Linear memory scaling with sequence length
- Extremely long-context friendly (tested up to 8k+ tokens)

---

## Training Data

The model was trained on:
- **Skylion007/openwebtext**

This dataset contains raw, unfiltered internet text and may include biased, incorrect, or unsafe content.

---

## Intended Use

**Primary use cases:**
- Research into transformer alternatives
- Long-context modeling experiments
- Architectural ablation studies
- Educational exploration of non-attention sequence models

**Not intended for:**
- Safety-critical applications
- Medical, legal, or financial advice
- Deployment as a production chatbot without additional alignment work

---

## Limitations

- This model is **research-grade**, not instruction-tuned
- Outputs may be:
  - Incoherent
  - Factually incorrect
  - Biased or unsafe
- Performance characteristics differ significantly from transformer LMs
- No reinforcement learning or alignment tuning applied

---

## Ethical Considerations

GCLM was trained on publicly available web data and may reflect societal biases present in that data.

Users are responsible for:
- Applying appropriate filtering
- Avoiding harmful or misleading use cases
- Evaluating outputs critically

---

## License

This model is released under the **Apache License 2.0**.

You are free to:
- Use
- Modify
- Distribute
- Use commercially

Attribution and license preservation are required.  
Patent rights are explicitly granted under this license.

---

## Citation

If you use GCLM in your research, please cite or reference the project.


## Important

The model will not be put in the repo until it has finished training.