new decoder path

README.md

@@ -0,0 +1,76 @@
+---
+language: es
+---
+
+# es-seq2seq-gender (decoder)
+
+This is a seq2seq model (decoder half) to "flip" gender in Spanish sentences.
+The model can augment your existing Spanish data, or generate counterfactuals
+to test a model's decisions (would changing the gender of the subject or speaker change output?).
+
+Intended Examples:
+
+- el profesor viejo => la profesora vieja (article, noun, adjective all flip)
+- una actriz => un actor (irregular noun)
+- el lingüista => la lingüista (irregular noun)
+- la biblioteca => la biblioteca (no person, no flip)
+
+People's names are unchanged in this version, but you can use packages
+such as https://pypi.org/project/gender-guesser/
+
+## Sample code
+
+https://colab.research.google.com/drive/1Ta_YkXx93FyxqEu_zJ-W23PjPumMNHe5
+
+```
+import torch
+from transformers import AutoTokenizer, EncoderDecoderModel
+
+model = EncoderDecoderModel.from_encoder_decoder_pretrained("monsoon-nlp/es-seq2seq-gender-encoder", "monsoon-nlp/es-seq2seq-gender-decoder")
+tokenizer = AutoTokenizer.from_pretrained('monsoon-nlp/es-seq2seq-gender-decoder') # all are same as BETO uncased original
+
+input_ids = torch.tensor(tokenizer.encode("la profesora vieja")).unsqueeze(0)
+generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)
+tokenizer.decode(generated.tolist()[0])
+> '[PAD] el profesor viejo profesor viejo profesor...'
+```
+
+## Training
+
+I originally developed 
+<a href="https://github.com/MonsoonNLP/el-la">a gender flip Python script</a>
+with 
+<a href="https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased">BETO</a>,
+the Spanish-language BERT from Universidad de Chile,
+and spaCy to parse dependencies in sentences.
+
+More about this project: https://medium.com/ai-in-plain-english/gender-bias-in-spanish-bert-1f4d76780617
+
+The seq2seq model is trained on gender-flipped text from that script run on the
+<a href="https://huggingface.co/datasets/muchocine">muchocine dataset</a>, 
+and the first 6,853 lines from the
+<a href="https://oscar-corpus.com/">OSCAR corpus</a>
+(Spanish ded-duped).
+
+The encoder and decoder started with weights and vocabulary from BETO (uncased).
+
+## Non-binary gender
+
+This model is useful to generate male and female text samples, but falls
+short of capturing gender diversity in the world and in the Spanish
+language. Some communities prefer the plural -@s to represent
+-os and -as, or -e and -es for gender-neutral or mixed-gender plural,
+or use fewer gendered professional nouns (la juez and not jueza). This is not yet
+embraced by the Royal Spanish Academy 
+and is not represented in the corpora and tokenizers used to build this project.
+
+This seq2seq project and script could, in the future, help generate more text samples 
+and prepare NLP models to understand us all better.
+
+#### Sources
+
+- https://www.nytimes.com/2020/04/15/world/americas/argentina-gender-language.html
+- https://www.washingtonpost.com/dc-md-va/2019/12/05/teens-argentina-are-leading-charge-gender-neutral-language/?arc404=true
+- https://www.theguardian.com/world/2020/jan/19/gender-neutral-language-battle-spain
+- https://es.wikipedia.org/wiki/Lenguaje_no_sexista
+- https://remezcla.com/culture/argentine-company-re-imagines-little-prince-gender-neutral-language/

config.json

@@ -0,0 +1,24 @@
+{
+  "add_cross_attention": true,
+  "architectures": [
+    "BertLMHeadModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": true,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "type_vocab_size": 2,
+  "vocab_size": 31002
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7cb79f2638210339579e4c8ddc4bcbd74e850d86e495405f44c3eefaf7d1205
+size 553140698

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "special_tokens_map_file": "/root/.cache/huggingface/transformers/78141ed1e8dcc5ff370950397ca0d1c5c9da478f54ec14544187d8a93eff1a26.dd8bd9bfd3664b530ea4e645105f557769387b3da9f79bdb55ed556bdd80611d", "name_or_path": "dccuchile/bert-base-spanish-wwm-uncased", "do_basic_tokenize": true, "never_split": null}