LoraHub

sivil_taram

sail-sg

Activity Feed Request to join this org

AI & ML interests

lora module composition, parameter-efficient tuning

Recent Activity

SivilTaram authored a paper about 1 month ago

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

SivilTaram authored a paper about 1 month ago

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

SivilTaram authored a paper about 1 month ago

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

View all activity

lorahub's activity

SivilTaram

authored 3 papers about 1 month ago

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

Paper • 2411.13476 • Published Nov 20 • 15

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

Paper • 2407.10956 • Published Jul 15 • 6

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

Paper • 2411.04905 • Published Nov 7 • 111

yuchenlin

authored a paper about 2 months ago

On Memorization of Large Language Models in Logical Reasoning

Paper • 2410.23123 • Published Oct 30 • 18

SivilTaram

authored a paper 2 months ago

Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

Paper • 2410.07137 • Published Oct 9 • 7

SivilTaram

authored a paper 3 months ago

Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale

Paper • 2409.17115 • Published Sep 25 • 60

SivilTaram

authored a paper 5 months ago

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Paper • 2407.13623 • Published Jul 18 • 53

SivilTaram

posted an update 6 months ago

Post

2539

Still following your human intuition to mix corpora from different sources for pre-training 🧠? Everyone says that data mixture has a big impact on model performance, but how - and why🕵️? Did you know that web corpora are actually highly impactful for downstream tasks 🏆?

Check out our preprint "RegMix: Data Mixture as Regression for Language Model Pre-training" 📄

🔬 In this paper, we've proposed an automatic data mixture method RegMix that achieves a 6.3% improvement over human selection on the widely used HellaSwag benchmark - and it only needs a 2% extra training FLOPs! 📈

📄 Paper: RegMix: Data Mixture as Regression for Language Model Pre-training (2407.01492)
💻 Code: https://github.com/sail-sg/regmix
📊 Collection: sail/regmix-data-mixture-as-regression-6682b6caab37b9442877f0ce
🎮 Demo: https://huggingface.co./spaces/sail/RegMix