updated content on different tabs - Adithya S K

app.py

@@ -153,34 +153,150 @@ def main():
     # About tab
     with About_tab:
         st.markdown('''
-            ### About Indic LLM Leaderboard
-            
-            ### Indic Eval
-            
-            ### Contribute
+## **Why a Indic LLM Leaderboard is Required ?**
+
+In recent months, there has been considerable progress in the Indic large language model (LLM) space. Major startups like Sarvam and Krutrim are building LLMs in this area.
+Simultaneously, the open-source community is also adapting pretrained models, such as Llama, Mistral, and Gemma, for Indic languages.
+Despite the influx of new models, there is a lack of a unified method to evaluate and compare them. This makes it challenging to track progress and determine what is working and what is not.
+
+> This is the alpha release of the Indic LLM Leaderboard, and modifications will be made to the leaderboard in the future.
+> 
+
+## **Who We Are**
+
+I'm [Adithya S K](https://linktr.ee/adithyaskolavi), the founder of [CognitiveLab](https://www.cognitivelab.in/). We provide AI solutions at scale and undertake research-based tasks.
+
+One initiative we have taken is to create a unified platform where Indic LLMs can be compared using specially crafted datasets. Although initially developed for internal use, we are now open-sourcing this framework to further aid the Indic LLM ecosystem.
+
+After releasing [Amabri, a 7b parameter English-Kannada bilingual LLM](https://www.cognitivelab.in/blog/introducing-ambari), we wanted to compare it with other open-source LLMs to identify areas for improvement. As there wasn't an existing solution, we built the Indic LLM suite, which consists of three projects:
+
+- [Indic-llm](https://github.com/adithya-s-k/Indic-llm): An open-source framework designed to adapt pretrained LLMs, such as Llama, Mistral, and Mixtral, to a wide array of domains and languages.
+- [Indic-Eval](https://github.com/adithya-s-k/indic_eval): A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks, aiding in performance assessment and comparison within the Indian language context.
+- [Indic LLM Leaderboard](https://huggingface.co/spaces/Cognitive-Lab/indic_llm_leaderboard): Utilizes the [indic_eval](https://github.com/adithya-s-k/indic_eval) evaluation framework, incorporating state-of-the-art translated benchmarks like ARC, Hellaswag, MMLU, among others. Supporting seven Indic languages, it offers a comprehensive platform for assessing model performance and comparing results within the Indic language modeling landscape.
+
+## **Upcoming implementations**
+
+- [ ] Support to add VLLM for faster evaluation and inference
+- [ ] SkyPilot installation to quickly run indic_eval on any cloud provider
+- [ ] Add support for onboard evaluation just like OpenLLM Leaderboard
+
+**Contribute**
+
+All the projects are completely open source with different licenses, so anyone can contribute.
+
+The current leaderboard is in alpha release, and many more changes are forthcoming:
+
+- More robust benchmarks tailored for Indic languages.
+- Easier integration with [indic_eval](https://github.com/adithya-s-k/indic_eval).
         ''')
         
     # FAQ tab
     with FAQ_tab:
         st.markdown('''
-            ### FAQ 
-                        
-            ### SUBMISSIONS
-            
-            
-            ### RESULTS
-            
-            
-            ### EDITING SUBMISSIONS
-            
-            
-            ### OTHER
+**What is the minimum requirement for GPUs to run the evaluation?**
+
+- The evaluation can easily run on a single A100 GPU, but the framework also supports multi-GPU based evaluation to speed up the process.
+
+**What languages are supported by the evaluation framework?**
+
+- The following languages are supported by default: English, Kannada, Hindi, Tamil, Telugu, Gujarati, Marathi, Malayalam.
+
+**How can I put my model on the leaderboard?**
+
+- Please follow the steps shown in the Submit tab or refer to the indic_eval for more details.
+
+**How does the leaderboard work?**
+
+- After running indic_eval on the model of your choice, the results are pushed to a server and stored in a database. The Frontend Leaderboard accesses the server and retrieves the latest models in the database along with their respective benchmarks and metadata. The entire system is deployed in India and is as secure as possible.
+
+**How is it different from the Open LLM leaderboard?**
+
+- This project was mainly inspired by the Open LLM leaderboard. However, due to limited computation resources, we standardized the evaluation library with standard benchmarks. You can run the evaluation on your GPUs and the leaderboard will serve as a unified platform to compare models. We used indictrans2 and other translation APIs to translate the benchmarking dataset into seven Indian languages to ensure reliability and consistency in the output.
+
+**Why does it take so much time to load the results?**
+
+- We are running the server on a serverless instance which has a cold start problem, so it might sometimes take a while.
+
+**What benchmarks are offered?**
+
+- The current Indic Benchmarks offered by the indic_eval library can be found in this collection: https://huggingface.co/collections/Cognitive-Lab/indic-llm-leaderboard-eval-suite-660ac4818695a785edee4e6f. They include ARC Easy, ARC Challenge, Hellaswag, Boolq, and MMLU.
+
+**How much time does it take to run the evaluation using indic_eval?**
+
+- Depending on which GPU you are running, the time for evaluation varies.
+- From our testing, it takes 3 to 4 hours to run the whole evaluation on a single GPU.
+- It's much faster when using multiple GPUs.
+
+**How does the verification step happen?**
+
+- While running the evaluation, you are given an option to push results to the leaderboard with `-push_to_leaderboard <[email protected]>`. You will need to provide an email address through which we can contact you. If we find any anomaly in the evaluation score, we will contact you through this email for verification of results.
         ''')
 
     # Submit tab
     with Submit_tab:
         st.markdown('''
-            ### Submit Your Model
+Here are the steps you will have to follows to put your model on the Indic LLM leaderboard 
+
+Clone the repo:
+
+```bash
+git clone <https://github.com/adithya-s-k/indic_eval>
+cd indic_eval
+
+```
+
+Create a virtual environment using virtualenv or conda depending on your preferences. We require Python 3.10 or above:
+
+```bash
+conda create -n indic-eval-venv python=3.10 && conda activate indic-eval-venv
+
+```
+
+Install the dependencies. For the default installation, you just need:
+
+```bash
+pip install .
+
+```
+
+If you want to evaluate models with frameworks like `accelerate` or `peft`, you will need to specify the optional dependencies group that fits your use case (`accelerate`,`tgi`,`optimum`,`quantization`,`adapters`,`nanotron`):
+
+```bash
+pip install '.[optional1,optional2]'
+
+```
+
+The setup tested most is:
+
+```bash
+pip install '.[accelerate,quantization,adapters]'
+
+```
+
+If you want to push your results to the Hugging Face Hub, don't forget to add your access token to the environment variable `HUGGING_FACE_HUB_TOKEN`. You can do this by running:
+
+```
+huggingface-cli login
+```
+
+## Command to Run Indic Eval and Push to Indic LLM Leaderboard
+
+```bash
+accelerate launch run_indic_evals_accelerate.py \\
+    --model_args="pretrained=<path to model on the hub>" \\
+    --tasks indic_llm_leaderboard \\
+    --output_dir output_dir \\
+    --push_to_leaderboard <[email protected]> \\
+
+```
+
+It's as simple as that.👍
+
+For `--push_to_leaderboard`, provide an email id through which we can contact you in case of verification. This email won't be shared anywhere. It's only required for future verification of the model's scores and for authenticity.
+
+After you have installed all the required packages, run the following command:
+
+For multi-GPU configuration, please refer to the docs of [Indic_Eval](https://github.com/adithya-s-k/indic_eval).
         ''')