nomic-ai/nomic-embed-text-v2-moe
Sentence Similarity
•
Updated
•
45.3k
•
268
nicolo
nicolollo
AI & ML interests
None yet
Recent Activity
liked
a model
12 days ago
nomic-ai/nomic-embed-text-v2-moe
liked
a model
14 days ago
tomg-group-umd/huginn-0125
Organizations
nicolollo's activity
liked
a
model
12 days ago
reacted to
grimjim's
post with ❤️
13 days ago
Post
2036
This recent paper points to an explanation for the unreasonable effectiveness of Frankenmerges:
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth
Approach (2502.05171)
Specifically, the duplication of layers in Frankenmerges serves a purpose similar to what occurs in their recurrent-depth architecture. Successful frankenmerges that operate without additional fine-tuning are able to recover or "heal" from any damage due to abrupt transitions between layer blocks. Operational replicated layer blocks can provide functional benefits grounded in latent reasoning. Frankenmerges can also result in hybrid reasoning, by splicing together the latent reasoning of different models.
Back in April 2024, I was able to duplicate a few layers in the Llama 3 8B model, turning it into a 9B model, without harming benchmarks significantly, despite any transition damage.
grimjim/llama-3-experiment-v1-9B
My informal experimentation suggested that latent reasoning circuits could occupy continguous stacks of 2-4 layers, though the result was highly sensitive to the choice of transition location between layers.
Specifically, the duplication of layers in Frankenmerges serves a purpose similar to what occurs in their recurrent-depth architecture. Successful frankenmerges that operate without additional fine-tuning are able to recover or "heal" from any damage due to abrupt transitions between layer blocks. Operational replicated layer blocks can provide functional benefits grounded in latent reasoning. Frankenmerges can also result in hybrid reasoning, by splicing together the latent reasoning of different models.
Back in April 2024, I was able to duplicate a few layers in the Llama 3 8B model, turning it into a 9B model, without harming benchmarks significantly, despite any transition damage.
grimjim/llama-3-experiment-v1-9B
My informal experimentation suggested that latent reasoning circuits could occupy continguous stacks of 2-4 layers, though the result was highly sensitive to the choice of transition location between layers.
- 1 reply
liked
a
model
14 days ago
liked
3
models
about 1 month ago
upvoted
2
collections
about 1 month ago
updated
a
model
about 1 month ago
published
a
model
about 1 month ago
updated
a
model
about 1 month ago
published
a
model
about 1 month ago
liked
a
model
about 1 month ago
upvoted
a
collection
about 2 months ago
reacted to
merve's
post with ❤️🚀🔥
about 2 months ago
Post
4875
supercharge your LLM apps with smolagents 🔥
however cool your LLM is, without being agentic it can only go so far
enter smolagents: a new agent library by Hugging Face to make the LLM write code, do analysis and automate boring stuff!
Here's our blog for you to get started https://huggingface.co./blog/smolagents
however cool your LLM is, without being agentic it can only go so far
enter smolagents: a new agent library by Hugging Face to make the LLM write code, do analysis and automate boring stuff!
Here's our blog for you to get started https://huggingface.co./blog/smolagents
liked
a
model
about 2 months ago
liked
2
models
2 months ago
