Upload LlamaMoEUpscalingForCausalLM

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,51 @@
+{
+  "_name_or_path": "ds-8x6.7b-top-2-universal-evol-instruct-2e-5_bs_64_epoch_4_e_4",
+  "add_weight_norm": false,
+  "architectures": [
+    "LlamaMoEUpscalingForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 32013,
+  "calculator_type": "UniversalCalculator",
+  "capacity_factor": 1.25,
+  "drop_tokens": true,
+  "dropped_padding": "zero",
+  "eos_token_id": 32014,
+  "ffn_folder_path": "ds-8x6.7b-top-2-universal-evol-instruct-2e-5_bs_64_epoch_4_e_4_ffn",
+  "gate_add_noise": true,
+  "gate_balance_loss_weight": 0.01,
+  "gate_network": "mlp",
+  "gate_noise_epsilon": 0.01,
+  "gate_type": "TopKUniversalBalancedNoisyGate",
+  "gate_use_balance": true,
+  "gate_use_softmax": true,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 16384,
+  "model_type": "llama_moe_upscaling",
+  "multiply_gate_scores": true,
+  "num_attention_heads": 32,
+  "num_experts": 8,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "num_selects": 2,
+  "pad_token_id": 32014,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "factor": 4.0,
+    "type": "linear"
+  },
+  "rope_theta": 10000.0,
+  "score_scale_factor": 1.0,
+  "shared_expert_weight": 0.75,
+  "size_experts": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.40.1",
+  "use_cache": true,
+  "vocab_size": 32256
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 32013,
+  "eos_token_id": 32014,
+  "pad_token_id": 32014,
+  "transformers_version": "4.40.1"
+}

model-00001-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3504c353f238238e961ae29c9253d1872192215077bc5e0b05eac8747d25d425
+size 4945318076

model-00002-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6900391fc6a7201a40464e8925ad702d57867d5f30ad3451235fda03bead4d5
+size 4978939004

model-00003-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a8d1e84909f4ad10212118d5fb1d4e875fc6293e88523014567b0eb4155a248
+size 4844917976

model-00004-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab915ab77c561a993ebd2701eecb4b38a9345af06ee2d93ea97c3b7430e30e54
+size 4957934276

model-00005-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0eb734da37b42fb4bd06e02115314820d756ae5cbc305b643953ad14ef052311
+size 4957934276

model-00006-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd19b93799e2a301ff5f06c51b811c818070dc78f7bff83b0e84de72f1328417
+size 4957934276

model-00007-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82307ffb4f1f432a8035182e9740bb2584169cb144fc1a1b7e3c7a2ed9468444
+size 4957934276

model-00008-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32bcf621da61b6303b76e4e7be113a4ded0bca8cab78376dd337b6692b65d7f2
+size 4957934276

model-00009-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c91cefe52c5d2b5ab438741a3d1a7900a879119c24e0b6eae8eb1f0393a7145
+size 4957934276

model-00010-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1f746dd0f05e95ea3055e06174d4f2eee07e41d2f66ecabc19b09fccd5df49a
+size 4957934308

model-00011-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d04620eac4a95d65186c297954fd8b9dc8e83fdd6112c17c44878b616d186ef3
+size 4957934308

model-00012-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34ee0671512e4160b5aae2b1d09302ba6ec592941e4966a21237a53fb24096f0
+size 4957934308

model-00013-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3d676924c399132937644a3890589a81c13ce3d9f9fae259a13cef25bf4e985
+size 4957934308

model-00014-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93625016ef207d5110ad4f458ed4b5a94a07c1dbf13afcfa3531bb909413fb76
+size 4957934308

model-00015-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e6d333d6a27306ce8212aadd84749ace67f4f8d663f5d8c9179839f841e1001
+size 4978939036

model-00016-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1016e8ec51b63bbdf6d337adc23821d1ca5f83a464ca81e3c60cc5d6a9b40e26
+size 4844918016

model-00017-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:deda933efe04b899b8a47e33993bf52d0c0f33749f109884ea5f0882cabb6b99
+size 4957934308

model-00018-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c77f4629281a55d4e585d3d815531c69341232304c597a9f7c786b0c63e6088d
+size 4957934308

model-00019-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b5b52beb0f5eb38e6043f31fb495ffd979daa3187eba8504d88c055b0018b70
+size 4957934308

model-00020-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17ef3138650fb5572ea26779b28f389d2a331c2d7dd0095c07de4e9f1cd81ade
+size 4957934308

model-00021-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:267265893923c60cb11144b0d1990825eb205fb6df46bd25622b985665835730
+size 4957934308

model-00022-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16093f43062c8e7dbc421903afe8970b2b10ab04750fc84245f51112f879c373
+size 4957934308

model-00023-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a45c347ae4be2411258866959375b66882300412b3b1b768a4081f814ce99c2e
+size 4957934308

model-00024-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f5a36083c208d2522d6cf1587eb9c833c27c23b382c0d050de2fa88b8fa34ff
+size 4957934308

model-00025-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61881f6793d0c135ef125ab5f21df768b1ce98fd17598a19d1509c4b4f85a2c5
+size 4957934308

model-00026-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3379b529bee41107906966a4f94ec09cfc047abf2836949f5b0dd5acfe5309ef
+size 4957934308

model-00027-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f96221719ca59df39518263ecd91db01b009147a7cd1bd1f308155da5fa7f88f
+size 4957934308

model-00028-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e3c533fb5cece5cb2108ee8ee09bda78702391521fb0ad0473d4b85f375ade6
+size 4978939036

model-00029-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0040845c9bea3c0480cb001d9a997d9359563debe9d803ee8cbb9f4a1505cf1c
+size 4844918016

model-00030-of-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2dbfa536f80ce330facbfed5c53daf4966fc300173dc2207d27d3dd643491aa0
+size 4676701712