Update README.md

README.md

@@ -11,6 +11,40 @@
 - **Bitlinear Layers:** Reduces weights to values of 1, 0, and -1. ➖
 - **Optimized for:** Fast inference and memory efficiency ⚡
 
+## Architecture
+```bash
+Model size: 1.604B parameters
+2024-10-08 14:53:07 - INFO - 🔢 Number of parameters in the model after extracting weights: 1
+2024-10-08 14:53:07 - INFO - 📏 Reduced model structure:
+LlamaForCausalLM(
+  (model): LlamaModel(
+    (embed_tokens): Embedding(128256, 4096)
+    (layers): ModuleList(
+      (0-5): 6 x LlamaDecoderLayer(
+        (self_attn): LlamaSdpaAttention(
+          (q_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
+          (k_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
+          (v_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
+          (o_proj): BitLinear(in_features=4096, out_features=4096, bias=False)
+          (rotary_emb): LlamaRotaryEmbedding()
+        )
+        (mlp): LlamaMLP(
+          (gate_proj): BitLinear(in_features=4096, out_features=2048, bias=False)
+          (up_proj): BitLinear(in_features=4096, out_features=2048, bias=False)
+          (down_proj): BitLinear(in_features=2048, out_features=4096, bias=False)
+          (act_fn): SiLU()
+        )
+        (input_layernorm): Identity()
+        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
+      )
+    )
+    (norm): LlamaRMSNorm((4096,), eps=1e-05)
+    (rotary_emb): LlamaRotaryEmbedding()
+  )
+  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
+)
+```
+
 ## Requirements 📦
 Make sure you have the following libraries installed: