LLM-Merging

LLM-Merging

LLM-Merging: Building LLMs Efficiently through Merging

Stars: 122

Visit
 screenshot

LLM-Merging is a repository containing starter code for the LLM-Merging competition. It provides a platform for efficiently building LLMs through merging methods. Users can develop new merging methods by creating new files in the specified directory and extending existing classes. The repository includes instructions for setting up the environment, developing new merging methods, testing the methods on specific datasets, and submitting solutions for evaluation. It aims to facilitate the development and evaluation of merging methods for LLMs.

README:

LLM-Merging: Building LLMs Efficiently through Merging

This repository contains the starter code for the LLM-Merging competition.

Setup Environment

The library was tested on CUDA 10.1 on an A6000.

conda env create -f environment.yml --name llm-merging
conda activate llm-merging
export PYTHONPATH=`pwd`

Authentication tokens are required for certain models like Llama2, which require users to agree to specific terms. You can find the authentication token here.

export HF_AUTH_TOKEN=""

Developing New Merging Methods

Do not modify any files other than the new file you create and setup.py. Doing so can result in the grounds for invalidating your submission. If you need to change code in other files, feel free to open a pull request.

  1. To add a new merging method, create a new file in llm_merging/merging.

    This file should implement __init__.py and merge.py functions and extend llm_merging/merging/Merges. See llm_merging/merging/FlanT5Avg.py or llm_merging/merging/LlamaAvg.py for examples.

  2. Modify setup.py and add an entry with the merging method in llm_merging.merging.Merges.

    For example, the entry llama_avg = llm_merging.merging.LlamaAvg:LlamaAvg indicates the method is called llama_avg and the file is at llm_merging/merging/LlamaAvg.

    Any additional required libraries can be specified in setup.py.

Test Method

python llm_merging/setup.py install
python llm_merging/main.py -m {merging_method}

The datasets (CosmosQA and XSum) are mainly included to ensure the merging method (with evaluation on those datasets) runs in under the 1-hour time limit. Our results on llama_avg are {"cosmos_qa": {"accuracy": 0.234}, "xsum": {"rouge1": 0.123, "rouge2": 0.023, "rougeL": 0.093, "rougeLsum": 0.102}}, which run in about 25 minutes on our A6000.

Submissions

After modifying the file, tar the file into a tarball using the command:

tar -cvf llm-merging.tar LLM-Merging

Submit the tar file using this form

Leaderboard

The leaderboard of the submitted solutions can be found here. Please note that your submission might not appear on the leaderboard immediately, as it is updated every few days. If you encounter any issues, please contact us.

Note: This submission method is only temporary and another automatic submission method should be comming soon.

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for LLM-Merging

Similar Open Source Tools

For similar tasks

For similar jobs