Upload HyperLlama-180M-swedish-1x

README.md

@@ -1,199 +1,81 @@
----
-library_name: transformers
-tags: []
----
+# Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation
 
-# Model Card for Model ID
+## About the Model
 
-<!-- Provide a quick summary of what the model is/does. -->
+This model was developed for the paper **_Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation_**.
 
+It is based on the [SmolLM2](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) architecture, 
+but instead of the original English tokenizer, 
+it uses the multilingual [Llama 3.3](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) tokenizer with a 128K vocabulary. 
+This results in a 180M-parameter base model.
 
+The base models were trained on:
 
-## Model Details
+- [Deduplicated FineWeb-Edu (English)](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus/viewer/fineweb-edu-dedup)
+- [Python-Edu (code subset of the SmolLM2 pre-training corpus)](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus/viewer/python-edu)
 
-### Model Description
+There exist 2 English base models: A 180M base model, and a 572M base model that is 
+the 180M base model [HyperCloned](https://arxiv.org/html/2409.12903v1) after 80% of the training steps, 
+and trained on the remaining 20% of the data afterwards. 
 
-<!-- Provide a longer summary of what this model is. -->
+Models were adapted to each target language using the corresponding [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) subset.
+The mutilingual models were adapted the same way, but on the FineWeb-2 subsets for all target languages at once. 
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+We evaluate three adaptation setups:
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **1×** — Continue pretraining the 180M base model on the target language
+- **1× (cloned)** — [HyperClone](https://arxiv.org/html/2409.12903v1) the 180M base model to 572M, then continue pretraining  on the target language
+- **2×** — Continue pretraining the 572M base model on the target language
 
-### Model Sources [optional]
+For more details on our setups, please refer to our paper.
 
-<!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+## Usage
 
-## Uses
+The model can be loaded like this. But note that this is a base model; it is not instruction-tuned. 
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-### Direct Use
+model_id = "liu-nlp/model-id"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+prompt = "To be hypercloned feels"
 
-[More Information Needed]
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=200,
+    do_sample=True,
+    temperature=0.7,
+)
 
-### Downstream Use [optional]
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+## Limitations 
 
-[More Information Needed]
+Users should apply the model responsibly, particularly in high-stakes or user-facing applications. The model is not guaranteed to provide factual, neutral, or safe outputs. Additional safety filtering, human oversight, or RLHF-style alignment may be required depending on the deployment context.
+Models of such sm all sizes are expected to produce hallucinations, especially for topics underrepresented or inconsistently represented in the training data. 
 
-### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+While the base model’s pre-training corpus presumably excludes overtly unethical or harmful material due to its educational focus, the continued pre-training phase introduces additional considerations. Large-scale corpora for our target languages are comparatively limited in availability and curation quality. As a result, the datasets used for domain or language adaptation may contain inappropriate, biased, or otherwise undesirable content.
 
-[More Information Needed]
 
-## Bias, Risks, and Limitations
+## Citation
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+If you use this model, please cite:
 
-[More Information Needed]
+```bibtex
+@misc{glocker2025growup,
+    title        = {Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation},
+    author       = {Glocker, Kevin and Kukk, Kätriin and Oji, Romina and Bollmann, Marcel and Kuhlmann, Marco and Kunz, Jenny},
+    year         = {2025},
+    eprint       = {XXXX.XXXXX},
+    archivePrefix= {arXiv},
+    primaryClass = {cs.CL},
+    note         = {Coming Soon},
+}
+```
 
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]

config.json

@@ -1,11 +1,10 @@
 {
   "architectures": [
-    "LlamaModel"
+    "LlamaForCausalLM"
   ],
   "attention_bias": false,
   "attention_dropout": 0.0,
   "bos_token_id": 0,
-  "dtype": "float32",
   "eos_token_id": 0,
   "head_dim": 64,
   "hidden_act": "gelu",
@@ -23,7 +22,8 @@
   "rope_scaling": null,
   "rope_theta": 10000.0,
   "tie_word_embeddings": true,
-  "transformers_version": "4.57.3",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.3",
   "use_cache": true,
   "vocab_size": 128256
 }

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.51.3"
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:431fbf457c2a22d73de246fca554871a6419ec5e7a5d1d2a49588d05c4335ae2
-size 720344392
+oid sha256:b1298840f7474452430d43edb48f3712e15b57afead943e446abe0b949027b9b
+size 360188472

tokenizer_config.json

@@ -2050,6 +2050,7 @@
     }
   },
   "bos_token": "<|begin_of_text|>",
+  "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n    {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n    {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n    {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n    {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content']|trim %}\n    {%- set messages = messages[1:] %}\n{%- else %}\n    {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n    {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n    {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n    {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n    {{- \"Do not use variables.\\n\\n\" }}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n    {#- Extract the first user message so we can plug it in here #}\n    {%- if messages | length != 0 %}\n        {%- set first_user_message = messages[0]['content']|trim %}\n        {%- set messages = messages[1:] %}\n    {%- else %}\n        {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n    {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n    {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n    {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n    {{- \"Do not use variables.\\n\\n\" }}\n    {%- for t in tools %}\n        {{- t | tojson(indent=4) }}\n        {{- \"\\n\\n\" }}\n    {%- endfor %}\n    {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n    {%- elif 'tool_calls' in message %}\n        {%- if not message.tool_calls|length == 1 %}\n            {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n        {%- endif %}\n        {%- set tool_call = message.tool_calls[0].function %}\n        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n            {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n            {%- for arg_name, arg_val in tool_call.arguments | items %}\n                {{- arg_name + '=\"' + arg_val + '\"' }}\n                {%- if not loop.last %}\n                    {{- \", \" }}\n                {%- endif %}\n                {%- endfor %}\n            {{- \")\" }}\n        {%- else  %}\n            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n            {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n            {{- '\"parameters\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- \"}\" }}\n        {%- endif %}\n        {%- if builtin_tools is defined %}\n            {#- This means we're in ipython mode #}\n            {{- \"<|eom_id|>\" }}\n        {%- else %}\n            {{- \"<|eot_id|>\" }}\n        {%- endif %}\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n        {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n        {%- if message.content is mapping or message.content is iterable %}\n            {{- message.content | tojson }}\n        {%- else %}\n            {{- message.content }}\n        {%- endif %}\n        {{- \"<|eot_id|>\" }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|eot_id|>",
   "extra_special_tokens": {},
@@ -2059,5 +2060,5 @@
   ],
   "model_max_length": 131072,
   "pad_token": "<|finetune_right_pad_id|>",
-  "tokenizer_class": "PreTrainedTokenizerFast"
+  "tokenizer_class": "PreTrainedTokenizer"
 }