OpenGPTAndBeyond

OpenGPTAndBeyond

Open efforts to implement ChatGPT-like models and beyond.

Stars: 102

Visit
screenshot

README:

ChatGPT：开源与超越

开源类ChatGPT模型的实现与超越之路

LLaMA权重意外泄露、以及斯坦福小羊驼用以self-instruct方式从gpt-3 api构建的数据对LLaMA进行指令微调取得令人印象深刻的表现以来，开源社区对实现ChatGPT水平的大语言模型感到越来越有希望。

这个repo就是记录这个复刻与超越的过程，为社区提供一个概览。

包括：相关技术进展、基础模型、领域模型、训练、推理、技术、数据、多语言、多模态，等等

# 目录

Base Models

contributor model/project license language main feature
Meta LLaMA/LLaMA2 multi LLaMA-13B outperforms GPT-3(175B) and LLaMA-65B is competitive to PaLM-540M.
Base model for most follow-up works.
HuggingFace-BigScience BLOOM multi an autoregressive Large Language Model (LLM) trained by HuggingFace BigScience.
HuggingFace-BigScience BLOOMZ multi instruction-finetuned version of BLOOM & mT5 pretrained multilingual language models on crosslingual task mixture.
EleutherAI GPT-J en transformer model trained using Ben Wang'sMesh Transformer JAX.
Meta OPT en Open Pre-trained Transformer Language Models, aim in developing this suite of OPT models is to enable reproducible
and responsible research at scale, and to bring more voices to the table in studying the impact of these LLMs.
Cerebras Systems Cerebras-GPT en Pretrained LLM, GPT-3 like, Commercially available, efficiently trained on theAndromeda AI supercomputer,
trained in accordance withChinchilla scaling laws (20 tokens per model parameter) which is compute-optimal.
EleutherAI pythia en combine interpretability analysis and scaling laws to understand how knowledge develops
and evolves during training in autoregressive transformers.
Stability-AI StableLM en Stability AI Language Models
FDU MOSS en/zh An open-source tool-augmented conversational language model from Fudan University.
ssymmetry & FDU BBT-2 zh 12B open-source LM.
@mlfoundations OpenFlamingo en An open-source framework for training large multimodal models.
EleutherAI GPT-NeoX-20B en Its architecture intentionally resembles that of GPT-3, and is almost identical to that ofGPT-J- 6B.
UCB OpenLLaMA Apache-2.0 en An Open Reproduction of LLaMA.
MosaicML MPT Apache-2.0 en MPT-7B is a GPT-style model, and the first in the MosaicML Foundation Series of models.
Trained on 1T tokens of a MosaicML-curated dataset, MPT-7B is open-source,
commercially usable, and equivalent to LLaMa 7B on evaluation metrics.
TogetherComputer RedPajama-INCITE-Base-3B-v1 Apache-2.0 en A 2.8B parameter pretrained language model, pretrained onRedPajama-Data-1T,
together with an Instruction-tuned Version and a Chat Version.
Lightning-AI Lit-LLaMA Apache-2.0 - Independent implementation ofLLaMA that is fully open source under the Apache 2.0 license.
@conceptofmind PaLM MIT License en An open-source implementation of Google PaLM models.
TII Falcon-7B TII Falcon LLM License en a 7B parameters causal decoder-only model built byTII and trained on 1,500B tokens of RefinedWeb enhanced with curated corpora.
TII Falcon-40B TII Falcon LLM License multi a 40B parameters causal decoder-only model built byTII and trained on 1,000B tokens of RefinedWeb enhanced with curated corpora.
TigerResearch TigerBot Apache-2.0 en/zh a multi-language and multitask LLM.
BAAI Aquila / Aquila2 BAAI_Aquila_Model_License en/zh The Aquila language model inherits the architectural design advantages of GPT-3 and LLaMA, replacing a batch of more efficient underlying
operator implementations and redesigning the tokenizer for Chinese-English bilingual support.
OpenBMB CPM-Bee 通用模型许可协议-来源说明-宣传限制-商业授权 en/zh CPM-Bee is a fully open-source, commercially-usable Chinese-English bilingual base model with a capacity of ten billion parameters.
And has been pre-trained on an extensive corpus of trillion-scale tokens.
Baichuan baichuan-7B Apache-2.0 en/zh It has achieved the best performance among models of the same size on standard
Chinese and English authoritative benchmarks (C-EVAL, MMLU, etc).
Tencent lyraChatGLM MIT License en/zh To the best of our knowledge, it is thefirst accelerated version of ChatGLM-6B.
The inference speed of lyraChatGLM has achieved 300x acceleration upon the early original version.
We are still working hard to further improve the performance.
SalesForce XGen Apache-2.0 multi Salesforce open-source LLMs with 8k sequence length
Shanghai AI Lab InternLM Apache-2.0 en/zh InternLM has open-sourced a 7 billion parameter base model and a chat model tailored for practical scenarios. The model has the following characteristics:
It leverages trillions of high-quality tokens for training to establish a powerful knowledge base.
It supports an 8k context window length, enabling longer input sequences and stronger reasoning capabilities.
It provides a versatile toolset for users to flexibly build their own workflows.
xverse-ai XVERSE Apache-2.0 multi Multilingual LLMs developed by XVERSE Technology Inc.
Writer palmyra Apache-2.0 en extremely powerful while being extremely fast. This model excels at many nuanced tasks
such as sentiment classification and summarization.
Mistral AI Mistral Apache-2.0 en Mistral 7B is a 7.3B parameter model that:
1. Outperforms Llama 2 13B on all benchmarks
2. Outperforms Llama 1 34B on many benchmarks
3. Approaches CodeLlama 7B performance on code, while remaining good at English tasks
4. Uses Grouped-query attention (GQA) for faster inference
5. Uses Sliding Window Attention (SWA) to handle longer sequences at smaller cost
SkyworkAI Skywork - en/zh In major evaluation benchmarks, Skywork-13B is at the forefront of Chinese open source models and is the optimal level under the same parameter scale;
it can be used commercially without application; it has also open sourced a 600G (150 billion tokens) Chinese data set.
01.AI Yi - en/zh TheYi series models are large language models trained from scratch by developers at 01.AI.
IEIT Systems Yuan-2.0 - en/zh In this work, the Localized Filtering-based Attention (LFA) is introduced to incorporate prior knowledge of local dependencies of natural language into Attention.
Based on LFA, we develop and release Yuan 2.0, a large language model with parameters ranging from 2.1 billion to 102.6 billion. A data filtering and generation method
is presented to build pretraining and fine-tuning dataset in high quality. A distributed training method with non-uniform pipeline parallel, data parallel, and optimizer parallel is proposed,
which greatly reduces the bandwidth requirements of intra-node communication, and achieves good performance in large-scale distributed training.
Yuan 2.0 models display impressive ability in code generation, math problem-solving, and chat compared with existing models.
Nanbeige Nanbeige Apache-2.0 en/zh Nanbeige-16B is a 16 billion parameter language model developed by Nanbeige LLM Lab. It uses 2.5T Tokens for pre-training. The training data includes a large amount of high-quality internet corpus, various books, code, etc. It has achieved good results on various authoritative evaluation data sets. This release includes the Base, Chat, Base-32k and Chat-32k.
deepseek-ai deepseek-LLM MIT License en/zh an advanced language model comprising 67 billion parameters. It has been trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese.
LLM360 LLM360 - - Most open-source LLM releases include model weights and evaluation results. However, additional information is often needed to genuinely understand a model's behavior—and this information is not typically available to most researchers. Hence, we commit to releasing all of the intermediate checkpoints (up to 360!) collected during training, all of the training data (and its mapping to checkpoints), all collected metrics (e.g., loss, gradient norm, evaluation results), and all source code for preprocessing data and model training. These additional artifacts can help researchers and practitioners to have a deeper look into LLM’s construction process and conduct research such as analyzing model dynamics. We hope that LLM360 can help make advanced LLMs more transparent, foster research in smaller-scale labs, and improve reproducibility in AI research.
FDU, etc. CT-LLM - zh/en focusing on the Chinese language. Starting from scratch, CT-LLM primarily uses Chinese data from a 1,200 billion token corpus, including 800 billion Chinese, 300 billion English, and 100 billion code tokens. By open-sourcing CT-LLM's training process, including data processing and the Massive Appropriate Pretraining Chinese Corpus (MAP-CC), and introducing the Chinese Hard Case Benchmark (CHC-Bench), we encourage further research and innovation, aiming for more inclusive and adaptable language models.

Domain Models

contributor model domain language base model main feature
UT Southwestern/
UIUC/OSU/HDU		 ChatDoctor medical en LLaMA Maybe the first domain-specific chat model tuned on LLaMA.
Cambridge Visual Med-Alpaca biomedical en LLaMA-7B a multi-modal foundation model designed specifically for the biomedical domain.
HIT BenTsao / ChatGLM-Med medical zh LLaMA/ChatGLM fine-tuned with Chinese medical knowledge dataset, which is generated by using gpt3.5 api.
ShanghaiTech, etc. DoctorGLM medical en/zh ChatGLM-6B Chinese medical consultation model fine-tuned on ChatGLM-6B.
THU AIR BioMedGPT-1.6B biomedical en/zh - a pre-trained multi-modal molecular foundation model with 1.6B parameters that associates 2D molecular graphs with texts.
@LiuHC0428 LawGPT_zh legal zh ChatGLM-6B a general model in Chinese legal domain, trained on data generated via Reliable-Self-Instruction.
SJTU MedicalGPT-zh medical zh ChatGLM-6B a general model in Chinese medical domain, a diverse data generated via self-instruct.
SJTU PMC-LLaMA medical zh LLaMA Continue Training LLaMA on Medical Papers.
HuggingFace StarCoder code generation en - a language model (LM) trained on source code and natural language text. Its training data incorporates more than
80 different programming languages as well as text extracted from GitHub issues and commits and from notebooks.
@CogStack NHS-LLM medical en not clear A conversational model for healthcare trained usingOpenGPT.
@pengxiao-song LaWGPT legal zh LLaMA/ChatGLM expand the vocab with Chinese legal terminologies, instruction fine-tuned on data generated using self-instruct.
Duxiaoman XuanYuan finance zh BLOOM-176B A Large Chinese Financial Chat Model with Hundreds of Billions Parameters.
CUHK HuatuoGPT medical zh not clear HuatuoGPT, a large language model (LLM) trained on a vast Chinese medical corpus. Our objective with HuatuoGPT is
to construct a more professional ‘ChatGPT’ for medical consultation scenarios.
PKU Lawyer LLaMA legal zh LLaMA continue pretraining on Chinese legal data, insturction tuned on legal exams and legal consulting qa pairs.
THU LexiLaw legal zh ChatGLM-6B trained on a mixture of general data (BELLE 1.5M) and legal data
THU, etc. taoli education zh LLaMA A large model for international Chinese education. It extends specific vocabulary on the base model,
and uses the domain's proprietary data set for instruction fine-tuning.
NUS Goat arithmetic en LLaMA a fine-tuned LLaMA model that significantly outperforms GPT-4 on a range of arithmetic tasks.
Fine-tuned on a synthetically generated dataset, Goat achieves state-ofthe-art performance on BIG-bench arithmetic sub-task.
CU/NYU FinGPT finance en - an end-to-end open-source framework for financial large language models (FinLLMs).
microsoft WizardCoder code generation en StarCoder trained with78k evolved code instructions. surpasses Claude-Plus (+6.8) , Bard (+15.3) and InstructCodeT5+ (+22.3) on the HumanEval Benchmarks.
UCAS Cornucopia finance zh LLaMA finetune LLaMA on Chinese financial knowledge,
PKU ChatLaw legal zh Ziya / Anima Chinese legal domain model.
@michael-wzhu ChatMed medical zh LLaMA Chinese medical LLM based on LLaMA-7B.
SCUT SoulChat mental health zh ChatGLM-6B Chinese dialogue LLM in mental health domain, based on ChatGLM-6B.
@shibing624 MedicalGPT medical zh ChatGLM-6B Training Your Own Medical GPT Model with ChatGPT Training Pipeline.
BJTU TransGPT transportation zh LLaMA-7B Chinese transportation model.
BAAI AquilaCode code generation multi Aquila AquilaCode-multi is a multi-language model that supports high-accuracy code generation for various programming languages, including Python/C++/Java/Javascript/Go, etc.
It has achieved impressive results in HumanEval (Python) evaluation, with Pass@1, Pass@10, and Pass@100 scores of 26/45.7/71.6, respectively. In the HumanEval-X
multi-language code generation evaluation, it significantly outperforms other open-source models with similar parameters (as of July 19, 2023).
AquilaCode-py, on the other hand, is a single-language Python version of the model that focuses on Python code generation.
It has also demonstrated excellent performance in HumanEval evaluation, with Pass@1, Pass@10, and Pass@100 scores of 28.8/50.6/76.9 (as of July 19, 2023).
Meta CodeLLaMA code generation multi LLaMA-2 a family of large language models for code based onLlama 2 providing state-of-the-art performance among open models, infilling capabilities,
support for large input contexts, and zero-shot instruction following ability for programming tasks.
UNSW, etc Darwin natural science en LLaMA-7B the first open-source LLM for natural science, mainly in physics, chemistry and material science.
alibaba EcomGPT e-commerce en/zh BLOOMZ An Instruction-tuned Large Language Model for E-commerce.
TIGER-AI-Lab MAmmoTH math en LLaMA2/CodeLLaMA a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct,
a meticulously curated instruction tuning dataset that is lightweight yet generalizable. MathInstruct is compiled from 13 math rationale datasets,
six of which are newly curated by this work. It uniquely focuses on the hybrid use of chain-of-thought (CoT) and program-of-thought (PoT) rationales,
and ensures extensive coverage of diverse mathematical fields.
SJTU abel math en LLaMA2 We proposeParental Oversight* , A Babysitting Strategy for Supervised Fine-tuning, Parental Oversight is not limited to any specific data processing method. Instead, it defines the data processing philosophy that should guide supervised fine-tuning in the era of Generative AI GAI).
FDU DISC-LawLLM legal zh Baichuan-13B FudanDISC has released DISC-LawLLM, a Chinese intelligent legal system driven by a large language model.
The system can provide various legal services for different user groups. In addition, DISC-Law-Eval is constructed to evaluate the large legal language model from both objective and subjective aspects.
The model has obvious advantages compared with the existing large legal models.
The team also made available a high-quality Supervised fine-tuning (SFT) dataset of 300,000, DISC-Law-SFT.
HKU, etc ChatPsychiatrist mental health en LLaMA-7B This repo open-sources the Instruct-tuned LLaMA-7B model that has been fine-tuned with counseling domian instruction data.
To construct our 8K size instruct-tuning dataset, we collected real-world counseling dialogue examples and employed GPT-4 as an extractor and filter.
In addition, we have introduced a comprehensive set of metrics, specifically tailored to the LLM+Counseling domain, by incorporating counseling domain evaluation criteria.
These metrics enable the assessment of performance in generating language content that involves multi-dimensional counseling skills.
CAS StarWhisper astronomical zh - StarWhisper, a large astronomical model, significantly improves the reasoning logic and integrity of the model through the fine-tuning of astrophysical corpus labeled by experts,
logical long text training, and direct preference optimization. In the CG-Eval jointly published by the Keguei AI Research Institute and LanguageX AI Lab, it reached the second place overall,
just below GPT-4, and its mathematical reasoning and astronomical capabilities are close to or exceed the GPT 3.5 Turbo.
ZhiPuAI FinGLM finance zh ChatGLM solutions of SMP2023-ELMFT(The Evaluation of Large Model of Finance Technology).
PKU, etc CodeShell code generation en/zh - CodeShell is a code large language model (LLM) developed jointly by theKnowledge Computing Lab at Peking University and the AI team of Sichuan Tianfu Bank. CodeShell has 7 billion parameters,
was trained on 500 billion tokens, and has a context window length of 8192. On authoritative code evaluation benchmarks (HumanEval and MBPP), CodeShell achieves the best performance for models of its scale.
FDU DISC-FinLLM finance zh Baichuan-13B-Chat DISC-FinLLM is a large language model in the financial field. It is a multi-expert intelligent financial system composed of four modules for different financial scenarios: financial consulting,
financial text analysis, financial calculation, and financial knowledge retrieval and question answering.
Deepseek Deepseek Coder code generation en/zh - Deepseek Coder comprises a series of code language models trained on both 87% code and 13% natural language in English and Chinese, with each model pre-trained on 2T tokens.
For coding capabilities, Deepseek Coder achieves state-of-the-art performance among open-source code models on multiple programming languages and various benchmarks.
microsoft MathOctopus math multi LLaMA2 This work pioneers exploring and building powerful Multilingual Math Reasoning (xMR) LLMs. To accomplish this, we make the following works:
1. MGSM8KInstruct, the first multilingual math reasoning instruction dataset, encompassing ten distinct languages, thus addressing the issue of training data scarcity in xMR tasks.
2. MSVAMP, an out-of-domain xMR test dataset, to conduct a more exhaustive and comprehensive evaluation of the model’s multilingual mathematical capabilities.
3. MathOctopus, our effective Multilingual Math Reasoning LLMs, training with different strategies, which notably outperform conventional open-source LLMs and exhibit superiority over ChatGPT in few-shot scenarios.
ITREC Zh-MT-LLM maritime en/zh ChatGLM3-6b The training data use the maritime domain data Zh-mt-sft organized for three main segments, and 30w general conversation datamoss-003-sft-data. Zh-mt-sft specifically Contains CrimeKgAssitant-1.8w, Zh-law-qa, and Zh-law-court related to maritime laws and regulations Q&A, Zh-edu-qa and Zh-edu-qb related to maritime education and training, and Zh-mt-qa related to maritime specialized knowledge Q&A.

some medical models: here

some domain llms: Awesome-Domain-LLM

healcare models: Awesome-Healthcare-Foundation-Models

General Domain Instruction Models

contributor model/project language base model main feature
Stanford Alpaca en LLaMA/OPT use 52K instruction-following data generated by Self-Instructt techniques to fine-tune 7B LLaMA,
the resulting model,  Alpaca, behaves similarly to the text-davinci-003 model on the Self-Instruct instruction-following evaluation suite.
Alpaca has inspired many follow-up models.
LianJiaTech BELLE en/zh BLOOMZ-7B1-mt maybe the first Chinese model to follow Alpaca.
THU ChatGLM-6B en/zh - well-known Chinese model.
Databricks Dolly en GPT-J 6B use Alpaca data to fine-tune a 2-year-old model: GPT-J, which exhibits surprisingly high quality
instruction following behavior not characteristic of the foundation model on which it is based.
@tloen Alpaca-LoRA en LLaMA-7B trained within hours on a single RTX 4090,
reproducing the Stanford Alpaca results using low-rank adaptation (LoRA),
and can run on a Raspberry pi.
ColossalAI Coati7B en/zh LLaMA-7B a large language model developed by the ColossalChat project
Shanghai AI Lab LLaMA-Adapter en LLaMA-7B Fine-tuning LLaMA to follow instructions within 1 Hour and 1.2M Parameters
AetherCortex Llama-X en LLaMA Open Academic Research on Improving LLaMA to SOTA LLM.
TogetherComputer OpenChatKit en GPT-NeoX-20B OpenChatKit provides a powerful, open-source base to create both specialized and general purpose chatbots for various applications.
The kit includes an instruction-tuned language models, a moderation model, and an extensible retrieval system for including
up-to-date responses from custom repositories.
nomic-ai GPT4All en LLaMA trained on a massive collection of clean assistant data including code, stories and dialogue
@ymcui Chinese-LLaMA-Alpaca en/zh LLaMA-7B/13B expand the Chinese vocabulary based on the original LLaMA and use Chinese data for secondary pre-training,
further enhancing Chinese basic semantic understanding. Additionally, the project uses Chinese instruction data
for fine-tuning on the basis of the Chinese LLaMA, significantly improving the model's understanding and execution of instructions.
UC Berkley
Stanford
CMU		 Vicuna en LLaMA-13B Impressing GPT-4 with 90% ChatGPT Quality.
UCSD/SYSU baize en/zh LLaMA fine-tuned withLoRA. It uses 100k dialogs generated by letting ChatGPT chat with itself.
Alpaca's data is also used to improve its performance.
UC Berkley Koala en LLaMA Rather than maximizingquantity by scraping as much web data as possible, the team focus on collecting a small high-quality dataset.
@imClumsyPanda langchain-ChatGLM en/zh ChatGLM-6B local knowledge based ChatGLM with langchain.
@yangjianxin1 Firefly zh bloom-1b4-zh
bloom-2b6-zh		 Instruction Tuning on Chinese dataset. Vocabulary pruning, ZeRO, and tensor parallelism
are used to effectively reduce memory consumption and improve training efficiency.
microsoft GPT-4-LLM en/zh LLaMA aims to share data generated by GPT-4 for building an instruction-following LLMs with supervised learning and reinforcement learning.
Hugging Face StackLLaMA en LLaMA trained on StackExchange data and the main goal is to serve as a tutorial and walkthrough on
how to train model with RLHF and not primarily model performance.
Nebuly ChatLLaMA en - a library that allows you to create hyper-personalized ChatGPT-like assistants using your own data and the least amount of compute possible.
@juncongmoo ChatLLaMA en LLaMA LLaMA-based RLHF model, runnable in a single GPU.
@juncongmoo minichatgpt en GPT/OPT ... To Train ChatGPT In 5 Minutes with ColossalAI.
@LC1332 Luotuo-Chinese-LLM zh LLaMA/ChatGLM Instruction fine-tuned Chinese Language Models, with colab provided!
@Facico Chinese-Vicuna zh LLaMA A Chinese Instruction-following LLaMA-based Model, fine-tuned with Lora, cpp inference supported, colab provided.
@yanqiangmiffy InstructGLM en/zh ChatGLM-6B ChatGLM based instruction-following model, fine-tuned on a variety of data sources, supports deepspeed accelerating and LoRA.
alibaba Wombat en LLaMA a novel learning paradigm called RRHF, as an alternative of RLHF,  is proposed, which scores responses generated by
different sampling policies and learns to align them with human preferences through ranking loss. And the performance
is comparable to RLHF, with less models used in the process.
@WuJunde alpaca-glassoff en LLaMA a mini image-acceptable Chat AI can run on your own laptop,  based onstanford-alpaca and alpaca-lora.
@JosephusCheung Guanaco multi LLaMA-7B A Multilingual Instruction-Following Language Model.
@FreedomIntelligence LLM Zoo multi BLOOMZ/LLaMA a project that provides data, models, and evaluation benchmark for large language models.
model released: Phoenix, Chimera
SZU Linly en/zh LLaMA expand the Chinese vocabulary, full fine-tuned models, largest LLaMA-based Chinese models, aggregation of Chinese instruction data, reproduceable details..
@lamini-ai lamini multi - data generator for generating instructions to train instruction-following LLMs.
Stability-AI StableVicuna en LLaMA a further instruction fine tuned and RLHF trained version of Vicuna v0 13b, with better performance than Vicuna.
Hugging Face HuggingChat en LLaMA seems to be the first one available to access as a platform that appears similar to ChatGPT.
microsoft WizardLM en LLaMA trained with 70k evolved instructions,Evol-Instruct is a novel method using LLMs instead of humans to automatically mass-produce
open-domain instructions of various difficulty levels and skills range, to improve the performance of LLMs.
FDU OpenChineseLLaMA en/zh LLaMA-7B further pretrain LLaMA on Chinese data, improving LLaMA preformance on Chinese tasks.
@chenfeng357 open-Chinese-ChatLLaMA en/zh LLaMA The complete training code of the open-source Chinese-Llama model, including the full process from pre-training instructing and RLHF.
@FSoft-AI4Code CodeCapybara en LLaMA Open Source LLaMA Model that Follow Instruction-Tuning for Code Generation.
@mbzuai-nlp LaMini-LM en LLaMA/Flan-T5 ... A Diverse Herd of Distilled Models from Large-Scale Instructions.
NTU Panda en/zh LLaMA further pretraining on Chinese data, full-size of LLaMA models.
IBM/CMU/MIT Dromedary en LLaMA-65B Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision.
@melodysdreamj WizardVicunaLM multi Vicuna Wizard's dataset + ChatGPT's conversation extension + Vicuna's tuning method,
achieving approximately 7% performance improvement over Vicuna.
sambanovasystems BLOOMChat multi BLOOM BLOOMChat is a 176 billion parameter multilingual chat model. It is instruction tuned fromBLOOM (176B) on
assistant-style conversation datasets and supports conversation, question answering and generative answers in multiple languages.
TII Falcon-7B-Instruct en Falcon-7B a 7B parameters causal decoder-only model built byTII based on Falcon-7B and finetuned on a mixture of chat/instruct datasets.
TII Falcon-40B-Instruct multi Falcon-40B a 40B parameters causal decoder-only model built byTII based on Falcon-40B and finetuned on a mixture of Baize.
USTC, etc. ExpertLLaMA en LLaMA use In-Context Learning to automatically write customized expert identity and find the quality quite satisfying.
We then prepend corresponding expert identity to each instruction to produce augmented instruction-following data.
We refer to the overall framework as ExpertPrompting, find more details in our paper.
ZJU CaMA en/zh LLaMA further pretrained on Chinese courpus without expansion of vocabulary; optimized on the Information Extraction (IE) tasks.
pre-training script is available, which includes transformations, construction, and loading of large-scale corpora, as well as the LoRA instruction fine-tuning script.
THU UltraChat en LLaMA First, the UltraChat dataset provides a rich resource for the training of chatbots. Second, by fine-tuning the LLaMA model,
the researchers successfully created a dialogue model UltraLLaMA with superior performance.
RUC YuLan-Chat en/zh LLaMA developed based on fine-tuning LLaMA with high-quality English and Chinese instructions.
AI2 Tülu en LLaMA/Pythia/OPT a suite of LLaMa models fully-finetuned on a strong mix of datasets.
KAIST SelFee en LLaMA Iterative Self-Revising LLM Empowered by Self-Feedback Generation.
@lyogavin Anima en/zh LLaMA trained based on QLoRA's33B guanaco, finetuned for 10000 steps.
THU ChatGLM2-6B en/zh - ChatGLM2 -6B is the second-generation version of the open-source bilingual (Chinese-English) chat model ChatGLM-6B.
It retains the smooth conversation flow and low deployment threshold of the first-generation model, while introducing the following new features:
- Stronger Performance
- Longer Context
- More Efficient Inference- More Open License
OpenChat OpenChat en LLaMA, etc. a series of open-source language models fine-tuned on a small, yet diverse and high-quality dataset of multi-round conversations.
Specifically, we utilize only ~6K GPT-4 conversations directly filtered from the ~90K ShareGPT conversations.
Despite the small size of the dataset, OpenLLMs has demonstrated remarkable performance.
CAS BayLing multi LLaMA BayLing is an English/Chinese LLM equipped with advanced language alignment,
showing superior capability in English/Chinese generation, instruction following and multi-turn interaction.
stabilityai FreeWilly/FreeWilly2 en LLaMA/LLaMA2 FreeWilly is a Llama65B model fine-tuned on an Orca style Dataset.
FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset.
FreeWilly2 outperforms Llama2 70B on the huggingface Open LLM leaderboard.
alibaba Qwen-7B en/zh - 7B-parameter version of the large language model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud.
ZJU KnowLM en/zh LLaMA With the rapid development of deep learning technology, large language models such as ChatGPT have made substantial strides in the realm of natural language processing.
However, these expansive models still encounter several challenges in acquiring and comprehending knowledge, including the difficulty of updating knowledge and potential knowledge
discrepancies and biases, collectively known asknowledge fallacies .
The KnowLM project endeavors to tackle these issues by launching an open-source large-scale knowledgable language model framework and releasing corresponding models.
NEU TechGPT en/zh LLAMA TechGPT mainly strengthens the following three types of tasks:
- Various information extraction tasks such as relation triplet extraction with "knowledge graph construction" as the core
- Various intelligent question-and-answer tasks centered on "reading comprehension".
- Various sequence generation tasks such as keyword generation with "text understanding" as the core.
@MiuLab Taiwan-LLaMa en/zh LLaMA2 Traditional Chinese LLMs for Taiwan.
Xwin-LM Xwin-LM en LLaMA2 Xwin-LM aims to develop and open-source alignment technologies for large language models, including supervised fine-tuning (SFT),
reward models (RM), reject sampling, reinforcement learning from human feedback (RLHF), etc. Our first release, built-upon on the
Llama2 base models, rankedTOP-1 on AlpacaEval. Notably, it's the first to surpass GPT-4 on this benchmark.
wenge-research YaYi en/zh LLaMA/LLaMA2 YaYi was fine-tuned on millions of artificially constructed high-quality domain data. This training data covers five key domains:
media publicity, public opinion analysis, public safety, financial risk control, and urban governance, encompassing over a hundred natural language instruction tasks.
HuggingFace zephyr en Mistral Zephyr is a series of language models that are trained to act as helpful assistants. Zephyr-7B-α is the first model in the series, and is a fine-tuned version of
mistralai/Mistral-7B-v0.1 that was trained on on a mix of publicly available, synthetic datasets using Direct Preference Optimization (DPO).
Cohere Command-R / Command R+ multi - Command-R has the capability for multilingual generation evaluated in 10 languages and highly performant RAG capabilities.
XAI grok en - 314B MoE; context length: 8192
databricks dbrx-instruct - - afine-grained mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input. It was pre-trained on 12T tokens of text and code data. Compared to other open MoE models like Mixtral-8x7B and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral-8x7B and Grok-1 have 8 experts and choose 2.

Model Merging

contributor model/method main feature main feature
FuseAI FuseChat Firstly, it undertakes pairwise knowledge fusion for source LLMs to derive multiple target LLMs of identical structure and size via lightweight fine-tuning. Then, these target LLMs are merged within the parameter space, wherein we propose a novel method VaRM for determining the merging weights based on the variation ratio of parameter matrices before and after fine-tuning. a fusion of three prominent chat LLMs with diverse architectures and scales, namelyNH2-Mixtral-8x7B, NH2-Solar-10.7B, and OpenChat-3.5-7B. FuseChat-7B-VaRM achieves an average performance of 8.22 on MT-Bench, outperforming various powerful chat LLMs at 7B and 34B scales like Starling-7B and Yi-34B-Chat, even surpassing GPT-3.5 (March), Claude-2.1, and approaching Mixtral-8x7B-Instruct.
arcee-ai mergekit Tools for merging pretrained large language models.
SakanaAI EvoLLM Evolutionary Optimization of Model Merging Recipes.

Alternatives To Transformer

(maybe successors?)

contributor method main feature
BlinkDL RWKV-LM RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable).
So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
msra RetNet simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention.
Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent.
Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-costO(1) inference, which improves decoding throughput,
latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity,
where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results,
parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models.
stanford Bapcpack ABackpack is a drop-in replacement for a Transformer that provides new tools for interpretability-through-control while still enabling strong language models.
Backpacks decompose the predictive meaning of words into components non-contextually, and aggregate them by a weighted sum, allowing for precise, predictable interventions.
stanford, etc. Monarch Mixer (M2) The basic idea is to replace the major elements of a Transformer with Monarch matrices — which are a class of structured matrices that generalize the FFT and are sub-quadratic,
hardware-efficient, and expressive. In Monarch Mixer, we use layers built up from Monarch matrices to do both mixing across the sequence (replacing the Attention operation) and mixing across the model dimension (replacing the dense MLP).
CMU, etc. Mamba Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress onstructured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.
TogetherComputer StripedHyena StripedHyena is thefirst alternative model competitive with the best open-source Transformers of similar sizes in short and long-context evaluations.
StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged inHyena blocks, different from traditional decoder-only Transformers.
1. Costant memory decoding in Hyena blocks via representation of convolutions as state-space models (modal or canonical form), or as truncated filters.
2. Low latency, faster decoding and higher throughput than Transformers.
3. Improvement to training and inference-optimal scaling laws, compared to optimized Transformer architectures such as Llama-2.
4. Trained on sequences of up to 32k, allowing it to process longer prompts.
microsoft bGPT bGPT supports generative modelling via next byte prediction on any type of data and can perform any task executable on a computer, showcasing the capability to simulate all activities within the digital world, with its potential only limited by computational resources and our imagination.
DeepMind Griffin-Jax Jax + Flax implementation of theGriffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, not official code(official code is not released yet);
the RG-LRU layer, a novel gated linear recurrent layer, around which we design a new recurrent block to replace MQA. We build two new models using this recurrent block: Hawk, a model which interleaves MLPs with recurrent blocks, and Griffin, a hybrid model which interleaves MLPs with a mixture of recurrent blocks and local attention
Griffin-3B outperforms Mamba-3B, and Griffin-7B and Griffin-14B achieve performance competitive with Llama-2, despite being trained on nearly 7 times fewer tokens; Griffin can extrapolate on sequences significantly longer than those seen during training.
AI21 Jamba Jamba is the first production-scale Mamba implementation. It’s a pretrained, mixture-of-experts (MoE) generative text model, with 12B active parameters and a total of 52B parameters across all experts. It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.
Meta Megalodon Megalodon inherits the architecture of Mega (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability and stability, including complex exponential moving average (CEMA), timestep normalization layer, normalized attention mechanism and pre-norm with two-hop residual configuration. In a controlled head-to-head comparison with Llama2, Megalodon achieves better efficiency than Transformer in the scale of 7 billion parameters and 2 trillion training tokens.

MoE

contributor model/project main feature
mistralai Mixtral-8x7B The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. The Mistral-8x7B outperforms Llama 2 70B on most benchmarks we tested.
Shanghai AI Lab, etc. LLaMA-MoE A small and affordable MoE model based onLLaMA and SlimPajama. The number of activated model parameters is only 3.0~3.5B, which is friendly for deployment and research usage.
NUS, etc. OpenMoE A family of open-sourced Mixture-of-Experts (MoE) Large Language Models.
Snowflake Arctic Arctic uses a unique Dense-MoE Hybrid transformer architecture. It combines a 10B dense transformer model with a residual 128x3.66B MoE MLP resulting in 480B total and 17B active parameters chosen using a top-2 gating.

Multi-Modal

contributor project language base model main feature
BaihaiAIen IDPChat en/zh LLaMA-13B
Stable Diffusion		 Open Chinese multi-modal model, single GPU runnable, easy to deploy, UI provided.
KAUST MiniGPT-4 en/zh LLaMA MiniGPT-4 aligns a frozen visual encoder from BLIP-2 with a frozen LLM, Vicuna, using just one projection layer,
and yields many emerging vision-language capabilities similar to those demonstrated in GPT-4.
MSR, etc. LLaVA en LLaMA visual instruction tuning is proposed, towards building large language and vision models with GPT-4 level capabilities.
NUS/THU VPGTrans en LLaMA/OPT/
Flan-T5/BLIP-2
...		 transferring VPG across LLMs to build VL-LLMs at significantly lower cost. The GPU hours
can be reduced over 10 times and the training data can be reduced to around 10%.
Two novel VL-LLMs are released via VPGTrans, including VL-LLaMA and VL-Vicuna.
VL-LLaMA is a multimodal version LLaMA by transferring the BLIP-2 OPT-6.7B to LLaMA via VPGTrans.
VL-Vicuna is a GPT-4-like multimodal chatbot, based on the Vicuna LLM.
CAS, etc X-LLM en/zh ChatGLM-6B X-LLM converts multi-modalities (images, speech, videos) into foreign languages using X2L interfaces and feed them into
a large Language Model (ChatGLM) to accomplish a Multimodal LLM, achieving impressive multimodal chat capabilities.
NTU Otter en OpenFlamingo a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo),
trained on MIMIC-IT and showcasing improved instruction-following ability and in-context learning.
Futhermore, optimize OpenFlamingo's implementation, democratizing the required
training resources from 1x A100 GPU to 4x RTX-3090 GPUs.
XMU LaVIN en LLaMA propose a novel and affordable solution for vision-language instruction tuning, namely Mixture-of-Modality Adaptation (MMA).
Particularly, MMA is an end-to-end optimization regime, which connects the image encoder and LLM via lightweight adapters.
Meanwhile, we also propose a novel routing algorithm in MMA, which can help the model automatically shifts the reasoning paths
for single- and multi-modal instructions.
USTC Woodpecker - - the first work to correct hallucination in multimodal large language models.
hpcaitech Open-Sora - - open source alternative to Openai Sora.

see also: awesome-Multimodal-Large-Language-Models

Data

Pretrain Data

contributor data/project language main feature
TogetherComputer RedPajama-Data en An Open Source Recipe to Reproduce LLaMA training dataset.
@goldsmith Wikipedia multi A Pythonic wrapper for the Wikipedia API.

Instruction Data

see Alpaca-CoT data collection

contributor data language main feature
salesforce DialogStudio en DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection and Instruction-Aware Models for Conversational AI.

Synthetic Data Generation

contributor method main feature
UW, etc. self-instruct using the model's own generations to create a large collection of instructional data.
@LiuHC0428 Reliable-Self-Instruction use ChatGPT to generate some questions and answers based on a given text.
PKU Evol-Instruct a novel method, proposed inWizardLM,  by using LLMs instead of humans to automatically mass-produce open-domain
instructions of various difficulty levels and skills range, to improve the performance of LLMs.
KAUST, etc. CAMEL a novel communicative agent framework namedrole-playing is proposed, which involves using inception prompting to guide chat agents
toward task completion while maintaining consistency with human intentions.
role-playing can be used to generate conversational data in a specific task/domain.
@chatarena ChatArena a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social interactions.
it provides a flexible framework to define multiple players, environments and the interactions between them, based on Markov Decision Process.

Evaluation

contributor method main feature
- human evaluation -
OpenAI GPT-4/ChatGPT -
PKU/CMU/MSRA ... PandaLM Reproducible and Automated Language Model Assessment.
UCB Chatbot Arena Chat with two anonymous models side-by-side and vote for which one is better,
then use the Elo rating system to calculate the relative performance of the models.
Stanford AlpacaEval GPT-4/Claude evaluation onAlpacaFarm dataset.
clueai SuperCLUElyb Chinese version ofChatbot Arena developed by clueai.
SJTU, etc. Auto-J a new open-source generative judge that can effectively evaluate different LLMs on how they align to human preference.
CMU CodeBERTScore an automatic metric for code generation, based onBERTScore.
As BERTScore, CodeBERTScore leverages the pre-trained contextual embeddings from a model such as CodeBERT and matches words in candidate and reference sentences by cosine similarity.
Differently from BERTScore, CodeBERTScore also encodes natural language input or other context along with the generated code, but does not use that context to compute cosine similarities.

Benchmark

国内大模型测评现状

contributor benchmark main feature
princeton SWE-bench a benchmark for evaluating large language models on real world software issues collected from GitHub. Given acodebase and an issue,
a language model is tasked with generating a patch that resolves the described problem.
microsoft AGIEval a human-centric benchmark specifically designed to evaluate the general abilities of foundation models in tasks pertinent to human cognition and problem-solving.
clueai SuperCLUE-Agent Agent evaluation benchmark based on Chinese native tasks.
bytedance GPT-Fathom GPT-Fathom is an open-source and reproducible LLM evaluation suite, benchmarking 10+ leading open-source and closed-source LLMs as well as OpenAI's earlier models on 20+ curated benchmarks under aligned settings.

LeaderBoard

opencompass, huggingface

Framework/ToolKit/Platform

contributor project main feature
CAS Alpaca-CoT extend CoT data to Alpaca to boost its reasoning ability.
aims at building an instruction finetuning (IFT) platform with extensive instruction collection (especially the CoT datasets)
and a unified interface for various large language models.
@hiyouga ChatGLM-Efficient-Tuning efficient fine-tuning ChatGLM-6B with PEFT.
@hiyouga LLaMA-Efficient-Tuning Fine-tuning LLaMA with PEFT (PT+SFT+RLHF with QLoRA).
@jianzhnie Efficient-Tuning-LLMs Efficient Finetuning of QLoRA LLMs.
ColossalAI ColossalChat An open-source low cost solution for cloningChatGPT with a complete RLHF pipeline.
microsoft deepspeed-chat Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales.
LAION-AI Open Assistant a project meant to give everyone access to a great chat based large language model.
HKUST LMFlow an extensible, convenient, and efficient toolbox for finetuning large machine learning models,
designed to be user-friendly, speedy and reliable, and accessible to the entire community.
UCB EasyLM EasyLM is a one stop solution for pre-training, finetuning, evaluating and serving LLMs in JAX/Flax.
EasyLM can scale up LLM training to hundreds of TPU/GPU accelerators by leveraging JAX's pjit functionality.
@CogStack OpenGPT A framework for creating grounded instruction based datasets and training conversational domain expert Large Language Models (LLMs).
HugAILab HugNLP a unified and comprehensive NLP library based on HuggingFace Transformer.
ProjectD-AI LLaMA-Megatron-DeepSpeed Ongoing research training transformer language models at scale, including: BERT & GPT-2.
@PanQiWei AutoGPTQ An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
alibaba swift SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning) is an extensible framwork designed to faciliate lightweight model fine-tuning and inference.
It integrates implementations for various efficient fine-tuning methods, by embracing approaches that is parameter-efficient, memory-efficient, and time-efficient.
alibaba Megatron-LLaMA to facilitate the training of LLaMA-based models and reduce the cost on occupying hardware resources,
Alibaba decides to release the internal optimized Megatron-LLaMA training framework to the community.
@OpenLLMAI OpenRLHF OpenRLHF aims to develop aHigh-performance RLHF training framework based on Ray and DeepSpeed.
OpenRLHF is the Simplest high-performance RLHF librarythat supports 34B models RLHF training with Single DGXA100 (script)).
The key idea of OpenRLHF is to distribute the Actor Model, Reward Model, Reference Model, and the Critic Model onto separate GPUs using Ray,
while placing the Adam Optimizer on the CPU. This enables full-scale fine-tuning of 7B models across multiple 24GB RTX4090 GPUs
(or 34B models with multiple A100 80G), with high training efficiency thanks to the ability to use a large generate batch size with Adam Offload and Ray.
Our PPO performance with the 13B llama2 models is 4 times that of DeepSpeedChat.
@zejunwang1 LLMTuner LLMTuner is an LLM instruction tuning tool that supports LoRA, QLoRA and full parameter fine-tuning. During training, flash attention and xformers attention technologies
can be used to improve training efficiency, and combined with technologies such as LoRA, DeepSpeed ZeRO, gradient checkpointing and 4-bit quantification, to effectively
reduce video memory usage and achieve the same goal on a single consumer-grade graphics card (A100/A40/A30 /RTX3090/V100) to fine-tune 7B/13B/34B large models.
Shanghai AI Lab XTuner A toolkit for efficiently fine-tuning LLM (InternLM, Llama, Baichuan, QWen, ChatGLM2).
alibaba MFTCoder CodeFuse-MFTCoder is an open-source project of CodeFuse for multitasking Code-LLMs(large language model for code tasks),
which includes models, datasets, training codebases and inference guides.
facebook llama-recipes Examples and recipes for Llama 2 model.
microsoft MS-AMP The FP8-LM framework is highly optimized and uses the FP8 format throughout the forward and backward passes, which greatly reduces the system's computing, memory and communication overhead.

Alignment

contributor method used in main feature
- IFT ChatGPT Instruction Fine-Tuning.
- RLHF ChatGPT RL from Human Feedback.
Anthropic RLAIF Claude RL from AI Feedback.
alibaba RRHF Wombat a novel learning paradigm called RRHF, as an alternative of RLHF,  is proposed, which scores responses generated by
different sampling policies and learns to align them with human preferences through ranking loss. And the performance
is comparable to RLHF, with less models used in the process.
HKUST RAFT - RAFT is a new alignment algorithm, which is more efficient than conventional (PPO-based) RLHF.
IBM/CMU/MIT SELF-ALIGN Dromedary combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.
PKU CVA Beaver Constrained Value Alignment via Safe RLHF.
tencent RLTF - Reinforcement Learning from Unit Test Feedback.
stanford DPO - implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a KL-divergence constraint) but is simple to implement and straightforward to train. Intuitively,
the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective.
THU BPO - The central idea behind BPO is to create an automatic prompt optimizer that rewrites human prompts, which are usually less organized or ambiguous, to prompts that better deliver human intent.
Consequently, these prompts could be more LLM-preferred and hence yielding better human-preferred responses.and the empirical results demonstrate that the BPO-aligned ChatGPT yields a 22% increase in the win rate against its original version, and 10% for GPT-4.
AI2, etc. URIAL - URIAL is a simple,tuning-free alignment method, URIAL (Untuned LLMs with Restyled In-context ALignment). URIAL achieves effective alignment purely through in-context learning (ICL), requiring as few as three constant stylistic examples and a system prompt for achieving a comparable performance with SFT/RLHF.
openai weak-to-strong - naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors.

Multi-Language

vocabulary expansion

according to the official FAQ in LLaMA repo, there's not many tokens other than latin languages, so one of the efforts is to expand the vocabulary, some works are shown below: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

contributor model/project language base model main feature
@ymcui Chinese-LLaMA-Alpaca zh LLaMA
SZU Linly en/zh LLaMA full-size LLaMA, further pretrained on Chineses Corpus.
@Neutralzz BiLLa en/zh LLaMA-7B further pretrained onWudao、PILE、WMT.
@pengxiao-song LaWGPT zh LLaMA/ChatGLM expand the vocab with Chinese legal terminologies, instruction fine-tuned on data generated using self-instruct.
IDEA Ziya en/zh LLaMA large-scale pre-trained model based on LLaMA with 13 billion parameters.
We optimizes LLaMAtokenizer on chinese, and incrementally train 110 billion tokens of data based on LLaMa-13B model,
which significantly improved the understanding and generation ability on Chinese.
OpenBuddy OpenBuddy multi LLaMA/Falcon ... Built upon Tii's Falcon model and Facebook's LLaMA model, OpenBuddy is fine-tuned to include an extended vocabulary,
additional common characters, and enhanced token embeddings. By leveraging these improvements and multi-turn dialogue datasets,
OpenBuddy offers a robust model capable of answering questions and performing translation tasks across various languages.
FDU CuteGPT en/zh LLaMA CuteGPT expands the Chinese vocabulary and performs pre-training on the Llama model, improving its ability to understand Chinese.
Subsequently, it is fine-tuned with conversational instructions to enhance the model's ability to understand instructions.
FlagAlpha FlagAlpha en/zh LLaMA/LLaMA2 based on largs-scale Chinese data, and starting from pre-training, the Chinese abilities of the models are being continuously and iteratively upgraded.

Efficient Training/Fine-Tuning

contributor method main feature
microsoft LoRA Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices
into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
stanford Prefix Tuning a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen
and instead optimizes a sequence of continuous task-specific vectors, which we call the prefix.
THU P-Tuning P-tuning leverages few continuous free parameters to serve as prompts fed as the input to the pre-trained language models.
We then optimize the continuous prompts using gradient descent as an alternative to discrete prompt searching.
THU, etc. P-Tuning v2 a novel empirical finding that properly optimized prompt tuning can be comparable to fine-tuning universally across various model scales and NLU tasks.
Technically, P-tuning v2 is not conceptually novel. It can be viewed as an optimized and adapted implementation of Deep Prompt Tuning.
Google Prompt Tuning a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.
Prompt Tuning can be seen as a simplification of "prefix tuning".
microsoft, etc. AdaLoRA adaptively allocates the parameter budget among weight matrices according to their importance score.
In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition.
UW QLoRA an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving
full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA).
FDU LOMO a new optimizer,LOw-Memory Optimization ( LOMO ), which fuses the gradient computation and the parameter update in one step to reduce memory usage,
which enables the full parameter fine-tuning of a 7B model on a single RTX 3090, or a 65B model on a single machine with 8×RTX 3090, each with 24GB memory.
MBZUAI, etc. GLoRA Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations,
providing more flexibility and capability across diverse tasks and datasets.
UMass Lowell ReLoRA ReLoRA performs a high-rank update and achieves performance similar to regular neural network training.
 The components of ReLoRA include initial full-rank training of the neural network, LoRA training, restarts, a jagged learning rate schedule, and partial optimizer resets.
Huawei QA-LoRA equips the original LoRA with two-fold abilities:
(i) during fine-tuning, the LLM’s weights are quantized (e.g., into INT4) to reduce time and memory usage;
(ii) after fine-tuning, the LLM and auxiliary weights are naturally integrated into a quantized model without loss of accuracy.
UMD, etc. NEFTune we propose to add random noise to the embedding vectors of the training data during the forward pass of fine-tuning. We show that this simple trick
can improve the outcome of instruction fine-tuning, often by a large margin, with no additional compute or data overhead.
THU SoRA sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. We achieve this through the incorporation of
a gate unit optimized with proximal gradient method in the training stage, controlling the cardinality of rank under the sparsity of the gate. In the subsequent inference stage,
we eliminate the parameter blocks corresponding to the zeroed-out ranks, to reduce each SoRA module back to a concise yet rank-optimal LoRA.
experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
FDU, etc. O-LoRA O-LoRA mitigates catastrophic forgetting of past task knowledge by constraining the gradient updates of the current task to be orthogonal to the gradient subspace of the past tasks.
TUDB-Labs mLoRA m-LoRA (a.k.a Multi-Lora Fine-Tune) is an open-source framework for fine-tuning Large Language Models (LLMs) using the efficient multiple LoRA/QLoRA methods. Key features of m-LoRA include:
1. Efficient LoRA/QLoRA: Optimizes the fine-tuning process, significantly reducing GPU memory usage by leveraging a shared frozen-based model.
2. Multiple LoRA Adapters: Support for concurrent fine-tuning of multiple LoRA/QLoRA adapters.
3. LoRA based Mix-of-Expert: Support for MixLoRA, which implements Mix-of-Expert architecture based on multiple LoRA adapters for frozen FFN layer.

Low-Cost Inference

quantization

contributor algorithm main feature
UW, etc. SpQR a new compressed format and quantization technique which enables for the first time near-lossless compression of LLMs across model scales,
while reaching similar compression levels to previous methods.
THU Train_Transformers_with_INT4 For forward propagation, we identify the challenge of outliers and propose a Hadamard quantizer to suppress the outliers.
For backpropagation, we leverage the structural sparsity of gradients by proposing bit splitting and leverage score sampling techniques to quantize gradients accurately.
INTEL neural-compressor targeting to provide unified APIs for network compression technologies, such as low precision quantization, sparsity, pruning, knowledge distillation,
across different deep learning frameworks to pursue optimal inference performance.
INTEL intel-extension-for-transformers Intel® Extension for Transformers is an innovative toolkit to accelerate Transformer-based models on Intel platforms, in particular, effective on 4th Intel Xeon Scalable processor Sapphire Rapids.
UCB KVQuant Per-channel, Pre-RoPE Key quantization to better match the outlier channels in Keys; Non-Uniform Quantization ( NUQ ) to better represent the non-uniform activations; Dense-and-Sparse Quantization to mitigate the impacts of numerical outliers on quantization difficulty; Q-Norm to mitigate distribution shift at ultra low precisions (eg. 2-bit); KVQuant enables serving the LLaMA-7B model with 1M context length on a single A100-80GB GPU , or even the LLaMA-7B model with 10M context length on an 8-GPU system 🔥

projects

contributor project main feature
@ggerganov llama.cpp c/cpp implementation for llama and some other models, using quantization.
@NouamaneTazi bloomz.cpp C++ implementation for BLOOM inference.
@mlc-ai MLC LLM a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications,
plus a productive framework for everyone to further optimize model performance for their own use cases.  
alibaba ChatGLM-MNN converts the ChatGLM-6B model to MNN and performs inference using C++.
Jittor JittorLLMs Significantly reduce hardware costs (by 80%), currently known as the lowest-cost deployment library, supports multiple platforms.
OpenBMB BMInf BMInf supports running models with more than 10 billion parameters on a single NVIDIA GTX 1060 GPU in its minimum requirements.
 In cases where the GPU memory supports the large model inference (such as V100 or A100),
BMInf still has a significant performance improvement over the existing PyTorch implementation.
hpcaitech EnergonAI With tensor parallel operations, pipeline parallel wrapper, distributed checkpoint loading, and customized CUDA kernel,
EnergonAI can enable efficient parallel inference for larges-scale models.
MegEngine InferLLM a lightweight LLM model inference framework that mainly references and borrows fromthe llama.cpp project.
llama.cpp puts almost all core code and kernels in a single file and use a large number of macros, making it difficult for developers to read and modify.
@saharNooby rwkv.cpp a port ofBlinkDL/RWKV-LM to ggerganov/ggml.
FMInference FlexGen FlexGen is a high-throughput generation engine for running large language models with limited GPU memory.
FlexGen allowshigh-throughput generation by IO-efficient offloading, compression, and large effective batch sizes .
huggingface
bigcode-project		 starcoder.cpp C++ implemention for 💫 StarCoder inference using theggml library.
CMU SpecInfer SpecInfer is an open-source distributed multi-GPU system that accelerates generative LLM inference withspeculative inference and token tree verification.
A key insight behind SpecInfer is to combine various collectively boost-tuned small speculative models (SSMs) to jointly predict the LLM’s outputs.
@ztxz16 fastllm full-platform pure c++ llm acceleration library, supports moss, chatglm, baichuan models,  runs smoothly on mobile phones.
UCB vllm a fast and easy-to-use library for LLM inference and serving. fast with Efficient management of attention key and value memory withPagedAttention.
stanford mpt-30B-inference Run inference on the latest MPT-30B model using your CPU. This inference code uses aggml quantized model.
Shanghai AI Lab lmdeploy a toolkit for compressing, deploying, and serving LLM.
@turboderp ExLlama / ExLlamaV2 A fast inference library for running LLMs locally on modern consumer-class GPUs
PyTorch ExecuTorch End-to-end solution for enabling on-device AI across mobile and edge devices for PyTorch models.
Xorbitsai Xinference a powerful and versatile library designed to serve language, speech recognition, and multimodal models.
With Xorbits Inference, you can effortlessly deploy and serve your or state-of-the-art built-in models using just a single command.
NVIDIA TensorRT-LLM TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and buildTensorRT engines that contain
state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes
that execute those TensorRT engines. It also includes a backend for integration with the NVIDIA Triton Inference Server; a production-quality system to serve LLMs.
Models built with TensorRT-LLM can be executed on a wide range of configurations going from a single GPU to
multiple nodes with multiple GPUs (using Tensor Parallelism and/or Pipeline Parallelism).
@sabetAI Batched LoRAs Maximize GPU util by routing inference through multiple LoRAs in the same batch.
huggingface TGI Text Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation for the most popular open-source LLMs,
including Llama, Falcon, StarCoder, BLOOM, GPT-NeoX, andmore. TGI implements many features, such as:
- Simple launcher to serve most popular LLMs
- Production ready (distributed tracing with Open Telemetry, Prometheus metrics)
- Tensor Parallelism for faster inference on multiple GPUs
- Token streaming using Server-Sent Events (SSE)
- Continuous batching of incoming requests for increased total throughput
- Optimized transformers code for inference using Flash Attention and Paged Attention on the most popular architectures
microsoft DeepSpeed-MII Under-the-hood MII is powered byDeepSpeed-Inference. Based on model type, model size, batch size, and available hardware resources, MII automatically applies the appropriate set of
system optimizations from DeepSpeed-Inference to minimize latency and maximize throughput. It does so by using one of many pre-specified model injection policies, that allows MII and
DeepSpeed-Inference to identify the underlying PyTorch model architecture and replace it with an optimized implementation. In doing so, MII makes the expansive set of
optimizations in DeepSpeed-Inference automatically available for thousands of popular models that it supports.
flexflow FlexFlow A key technique that enables FlexFlow Serve to accelerate LLM serving is speculative inference, which combines various collectively boost-tuned small speculative models (SSMs)
to jointly predict the LLM’s outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences
represented by a token tree is verified against the LLM’s output in parallel using a novel tree-based parallel decoding mechanism. FlexFlow Serve uses an LLM as a token tree verifier instead of
an incremental decoder, which largely reduces the end-to-end inference latency and computational requirement for serving generative LLMs while provably preserving model quality.
BentoML BentoML an open platform for machine learning in production. It simplifies model packaging and model management, optimizes model serving workloads
to run at production scale, and accelerates the creation, deployment, and monitoring of prediction services.
@ModelTC LightLLM a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
@FasterDecoding Medusa Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads.
UCB, etc. S-LoRA S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation,
S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths.
Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA
to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving),
S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models
and offers the potential for large-scale customized fine-tuning services.
@lyogavin AirLLM When executing at a certain layer, the corresponding layer will be loaded from the hard drive, and the calculation of that layer will be performed. Once the calculation is complete,
the memory of that layer can be completely released. This way, the GPU memory usage will only be approximately the size of one layer of transformer parameters.
UW, etc. Punica We present Punica, a system to serve multiple LoRA models in a shared GPU cluster. Punica contains a new CUDA kernel design that allows batching of GPU operations for different LoRA models.
This allows a GPU to hold only a single copy of the underlying pre-trained model when serving multiple, different LoRA models, significantly enhancing GPU efficiency in terms of both memory and computation.
Our scheduler consolidates multi-tenant LoRA serving workloads in a shared GPU cluster. With a fixed-sized GPU cluster, our evaluations show that Punica achieves 12x higher throughput in serving multiple LoRA models
compared to state-of-the-art LLM serving systems while only adding 2ms latency per token.
alibaba MergeLM In this work, we uncover that Language Models (LMs), either encoder- or decoder-based, canobtain new capabilities by assimilating the parameters of homologous models without the need for retraining or GPUs.
1. We introduce a novel operation called DARE to directly set most of (90% or even 99%) the delta parameters to zeros without affecting the capabilities of SFT LMs.
2. We sparsify delta parameters of multiple SFT homologous models with DARE as a general preprocessing technique and subsequently merge them into a single model by parameter averaging.
ETH Zürich UltraFastBERT a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraFastBERT selectively engages just 12 out of 4095 neurons for each layer inference.
 This is achieved by replacing feedforward networks with fast feedforward networks (FFFs).
UCB, etc. LookaheadDecoding Lookahead decoding breaks the sequential dependency in autoregressive decoding by concurrently extracting and verifying n-grams directly with the LLM, utilizing theJacobi iteration method.
Lookahead decoding functions without the need for a draft model or a data store. It linearly decreases the number of decoding steps directly correlating with the log(FLOPs) used per decoding step.
Intel BigDL bigdl-llm is a library for running LLM (large language model) on Intel XPU (from Laptop to GPU to Cloud ) using INT4/FP4/INT8/FP8 with very low latency  (for any PyTorch model).
SenseTime, etc. LightLLM 1. Tri-process asynchronous collaboration: tokenization, model inference, and detokenization are performed asynchronously, leading to a considerable improvement in GPU utilization.
2. Token Attention: implements token-wise's KV cache memory management mechanism, allowing for zero memory waste during inference.
3. High-performance Router: collaborates with Token Attention to meticulously manage the GPU memory of each token, thereby optimizing system throughput.
THU, etc. SoT to guide LLMs to generate the skeleton of the answer, and then conducts parallel API calls or batched decoding to complete the contents of each skeleton point in parallel. Not only does SoT provide considerable speed-ups across 12 LLMs, but it can also potentially improve the answer quality on several question categories.
ollama ollama docker-style interation of llm inference.
alibaba RTP-LLM Alibaba's high-performance LLM inference engine for diverse applications. The project is mainly based onFasterTransformer, and on this basis, we have integrated some kernel implementations from TensorRT-LLM. FasterTransformer and TensorRT-LLM have provided us with reliable performance guarantees. Flash-Attention2 and cutlass have also provided a lot of help in our continuous performance optimization process. Our continuous batching and increment decoding draw on the implementation of vllm; sampling draws on transformers, with speculative sampling integrating Medusa's implementation, and the multimodal part integrating implementations from llava and qwen-vl.
腾讯 一念 一念LLM是面向LLM推理和服务的高性能和高易用的推理引擎。
高性能和高吞吐:
使用极致优化的 CUDA kernels, 包括来自 vllm, TensorRT-LLM, FastTransformer 等工作的高性能算子
基于PagedAttention实现地对注意力机制中key和value的高效显存管理
对任务调度和显存占用精细调优的动态batching
(实验版) 支持前缀缓存(Prefix caching)

Prompt Compression

contributor project main feature
microsoft LLMLingua LLMLingua, that uses a well-trained small language model after alignment, such as GPT2-small or LLaMA-7B, to detect the unimportant tokens in the prompt and enable inference with the compressed prompt in black-box LLMs, achieving up to 20x compression with minimal performance loss.

Prompting

Prompt Engineering Guide

contributor method main feature
Google CoT a technique that allows large language models (LLMs) to solve a problem as a series of intermediate steps before giving a final answer.
Princeton, etc. ToT(Yao et el. (2023) and Long (2023)) ToT maintains a tree of thoughts, where thoughts represent coherent language sequences that serve as intermediate steps toward solving a problem.
This approach enables an LM to self-evaluate the progress intermediate thoughts make towards solving a problem through a deliberate reasoning process.
SJTU, etc. GoT we propose Graph-of-Thought (GoT) reasoning, which models human thought processes not only as a chain but also as a graph. By representing thought units as nodes
and connections between them as edges, our approach captures the non-sequential nature of human thinking and allows for a more realistic modeling of thought processes.
Princeton, etc. ReAct LLMs are used to generate bothreasoning traces and task-specific actions in an interleaved manner.
SJTU Meta-CoT Meta-CoT is a generalizable CoT prompting method in mixed-task scenarios where the type of input questions is unknown. It consists of three phases:
(i) scenario identification : categorizes the scenario of the input question;
(ii) demonstration selection : fetches the ICL demonstrations for the categorized scenario;
(iii) answer derivation : performs the answer inference by feeding the LLM with the prompt comprising the fetched ICL demonstrations and the input question.
UCLA RaR we present a method named `Rephrase and Respond' (RaR), which allows LLMs to rephrase and expand questions posed by humans and provide responses in a single prompt.
Our experiments demonstrate that our methods significantly improve the performance of different models across a wide range to tasks.
CAS, etc. EmotionPrompt Our automatic experiments show that LLMs have a grasp of emotional intelligence, and their performance can be improved with emotional prompts (which we call “EmotionPrompt” that combines the original prompt with emotional stimuli),
Our human study results demonstrate that EmotionPrompt significantly boosts the performance of generative tasks.
Meta S2A S2A regenerates the input context to only include the relevant portions, before attending to the regenerated context to elicit the final response.
Google Step-Back Prompting We present STEP-BACK PROMPTING, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide the reasoning steps, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of STEP-BACK PROMPTING with PaLM-2L models and observe substantial performance gains on a wide range of challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, STEP-BACK PROMPTING improves PaLM-2L performance on MMLU Physics and Chemistry by 7% and 11%, TimeQA by 27%, and MuSiQue by 7%.

Safety

contributor method main feature
thu-coai Safety-Prompts Chinese safety prompts for evaluating and improving the safety of LLMs.

Truthfulness

contributor method main feature
Harvard ITI ITI operates by shifting model activations during inference, following a set of directions across a limited number of attention heads.
This intervention significantly improves the performance of LLaMA models on the TruthfulQA benchmark.
On an instruction-finetuned LLaMA called Alpaca, ITI improves its truthfulness from 32.5 to 65.1.

Exceeding Context Window

https://zhuanlan.zhihu.com/p/670280576

Extending Context Window

contributor method main feature
UW, etc. ALiBi Instead of adding position embeddings at the bottom of the transformer stack,
ALiBi adds a linear bias to each attention score, allowing the model to be trained on,
for example, 1024 tokens, and then do inference on 2048 (or much more) tokens without any finetuning.
DeepPavlov, etc. RMT use a recurrent memory to extend the context length.
bytedance SCM unleash infinite-length input capacity for large-scale language models.
Meta Position Interpolation extends the context window sizes of RoPE-based  pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps).
Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond
the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism.
UCB LongChat Instead of forcing the LLaMA model to adapt to position_ids > 2048, we condense position_ids > 2048 to be within 0 to 2048 (the same machenism asPosition Interpolation, surprisingly!).
we observed that our LongChat-13B-16K model reliably retrieves the first topic, with comparable accuracy to gpt-3.5-turbo.
microsoft LongNet replaces the attention of vanilla Transformers with a novel component nameddilated attention, and successfully scale the sequence length to 1 billion tokens.
IDEAS NCBR, etc. LongLLaMA LongLLaMA is built upon the foundation ofOpenLLaMA and fine-tuned using the Focused Transformer (FoT) method, and is capable of handling long contexts of 256k tokens or even more.
Abacus.AI Giraffe a range of experiments with different schemes for extending context length capabilities of Llama are conducted.
TogetherComputer Llama-2-7B-32K-Instruct long-context chat model finetuned fromLlama-2-7B-32K, over high-quality instruction and chat data.
Jianlin Su ReRoPE set a window with size$w$, the interval between positions inside the window is 1, while the interval outside the window is $\frac 1 k$.
CUHK/MIT longlora an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost.

Without Extending Context Window

contributor method main feature
MIT/Meta/CMU StreamingLLM/
SwiftInfer 		deploy LLMs forinfinite-length inputs without sacrificing efficiency and performance.
an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning.
We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment.
In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup.
SwiftInfer：implement StreamingLLM based on TensorRT-LLM.
UCB Ring Attention We present a distinct approach, Ring Attention, which leverages blockwise computation of self-attention to distribute long sequences across multiple devices while overlapping the communication of
key-value blocks with the computation of blockwise attention. Ring Attention enables training and inference of sequences that are up to device count times longer than those of prior memory-efficient Transformers,
effectively eliminating the memory constraints imposed by individual devices. Extensive experiments on language modeling tasks demonstrate the effectiveness of Ring Attention in allowing large sequence input size
and improving performance.
UCB MemGPT a system that intelligently manages different memory tiers in LLMs in order to effectively provide extended context within the LLM's limited context window.
For example, MemGPT knows when to push critical information to a vector database and when to retrieve it later in the chat, enabling perpetual conversations.
FDU, etc. ScalingRoPE we first observe that fine-tuning a RoPE-based LLM with either a smaller or larger base in pre-training context length could significantly enhance its extrapolation performance.
After that, we propose Scaling Laws of RoPE-based Extrapolation, a unified framework from the periodic perspective,
to describe the relationship between the extrapolation performance and base value as well as tuning context length.
THU InfLLM training-free, memory-base, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Even when the sequence length is scaled to 1, 024K, InfLLM still effectively captures long-distance dependencies.

Knowledge Editing

Must-read Papers on Model Editing: ModelEditingPapers

contributor method main feature
MIT, etc. ROME First, we trace the causal effects of hidden state activations within GPT using causal mediation analysis to identify the specific modules that mediate recall of a fact about a subject.
Our analysis reveals that feedforward MLPs at a range of middle layers are decisive when processing the last token of the subject name.
Second, we test this finding in model weights by introducing a Rank-One Model Editing method (ROME) to alter the parameters that determine a feedfoward layer’s behavior at the decisive token.
Despite the simplicity of the intervention, we find that ROME is similarly effective to other modelediting approaches on a standard zero-shot relation extraction benchmark.

Implementations

contributor project main feature
PKU FastEdit injectingfresh and customized knowledge into large language models efficiently using one single command.
ZJU EasyEdit a Python package for edit Large Language Models (LLM) like GPT-J, Llama, GPT-NEO, GPT2, T5(support models from 1B to 65B ),
the objective of which is to alter the behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. It is designed to be easy to use and easy to extend.

External Knowledge

allowing the model to access external knowledge, such as internet、KG、databases.

contributor project main feature
@jerryjliu LlamaIndex provides a central interface to connect your LLM's with external data.
@imClumsyPanda langchain-ChatGLM local knowledge based ChatGLM withlangchain.
@wenda-LLM wenda an LLM calling platform designed to find and design automatic execution actions for small model plug-in
knowledge bases to achieve the same generation ability as large models.
@csunny DB-GPT build a complete private large model solution for all database-based scenarios.
THU, BAAI, ZJU ChatDB a novel framework integrating symbolic memory with LLMs. ChatDB explores ways of augmenting LLMs with symbolic memory to handle contexts of arbitrary lengths.
Such a symbolic memory framework is instantiated as an LLM with a set of SQL databases. The LLM generates SQL instructions to manipulate the SQL databases
autonomously (including insertion, selection, update, and deletion), aiming to complete a complex task requiring multi-hop reasoning and long-term symbolic memory.
IDEA Ziya-Reader "Ziya-Reader-13B-v1.0" is a knowledge question-answering model. It can accurately answer questions given questions and knowledge documents,
and is suitable for both multi-document and single-document question-answering. The model has an 8k context window, and compared to models with longer windows,
we have achieved victory in evaluations across multiple long-text tasks. The tasks include multi-document question-answering, synthetic tasks (document retrieval), and long-text summarization.
Additionally, the model also demonstrates excellent generalization capabilities, enabling it to be used for general question-answering.
Its performance on our general ability evaluation set surpassed that of Ziya-Llama-13B.
docker GenAI Stack significantly simplify the entire process by integrating Docker with the Neo4j graph database, LangChain model linking technology, and Ollama for running Large Language models (LLM)
ARC53 DocsGPT GPT-powered chat for documentation, chat with your documents.
UW, etc. Self-RAG Unlike a widely-adopted Retrieval-Augmented Generation (RAG) approach,Self-RAG retrieves on demand (e.g., can retrieve multiple times or completely skip retrieval) given diverse queries,
and criticize its own generation from multiple fine-grained aspects by predicting reflection tokens as an integral part of generation.
RUC StructGPT Inspired by the studies on tool augmentation for LLMs, we develop an Iterative Reading-thenReasoning (IRR) framework to solve question answering tasks based on structured data, called StructGPT.
In this framework, we construct the specialized interfaces to collect relevant evidence from structured data (i.e., reading), and let LLMs concentrate on the reasoning task based on the collected information (i.e., reasoning).
Specially, we propose an invokinglinearization-generation procedure to support LLMs in reasoning on the structured data with the help of the interfaces. By iterating this procedure with provided interfaces,
our approach can gradually approach the target answers to a given query. Experiments conducted on three types of structured data show that StructGPT greatly improves the performance of LLMs,
under the few-shot and zero-shot settings.
BUPT ChatKBQA A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned LLMs.
ZJU KnowPAT Knowledgeable Preference AlignmenT (KnowPAT) is a new pipeline to align LLMs with human's knowledge preference.
KnowPAT incorporates domain knowledge graphs to construct preference set and design new alignment objective to fine-tune the LLMs.
LeptonAI Search with Lepton A conversation-based search demo, like perplexity.ai.
NetEase QAnything a local knowledge base question-answering system designed to support a wide range of file formats and databases, allowing for offline installation and use. With QAnything, you can simply drop any locally stored file of any format and receive accurate, fast, and reliable answers. Currently supported formats include: PDF(pdf) , Word(docx) , PPT(pptx) , XLS(xlsx) , Markdown(md) , Email(eml) , TXT(txt) , Image(jpg，jpeg，png) , CSV(csv) ,Web links(html) and more formats coming soon…
InfiniFlow RAGFlow an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
dify.ai Dify Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.
sealos.io FastGPT FastGPT is a knowledge-based Q&A system built on the LLM, offers out-of-the-box data processing and model invocation capabilities, allows for workflow orchestration through Flow visualization!

Chat with Docs

contributor project main feature
@arc53 DocsGPT GPT-powered chat for documentation, chat with your documents

more at: funNLP

内容解析

贡献者 项目 主要特征
阿里 OmniParser 设计了一个通用模型OmniParser，可以同时处理3个典型的视觉情境文本解析任务：文本识别、关键信息提取和表格识别。在OmniParser中，所有任务共享统一的编码器-解码器架构，统一的目标：条件文本生成，以及统一的输入和输出表示：提示和结构化序列。广泛的实验表明，OmniParser在三个视觉位置的文本解析任务的7个数据集上取得了最先进的(SOTA)或极具竞争力的性能。

Vector DataBase

contributor db main feature
milvus-io milvus a cloud-native vector database with storage and computation separated by design.
Meta faiss It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU.
nmslib hnswlib Header-only C++ HNSW implementation with python bindings, insertions and updates.
MyScale MyScaleDB An open-source, high-performance SQL vector database built on ClickHouse.
chroma Chroma the AI-native open-source embedding database.
Weaviate Weaviate stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database.

External Tools

Using Existing Tools

allowing the model to access external tools, such as search engine、api.

contributor project base model main feature
UCB/microsoft Gorilla LLaMA invokes 1,600+ (and growing) API calls accurately while reducing hallucination.
THU ToolLLaMA LLaMA This project aims to constructopen-source, large-scale, high-quality instruction tuning SFT data to facilitate the construction
of powerful LLMs with general tool-use capability. We provide the dataset, the corresponding training and evaluation scripts,
and a capable model ToolLLaMA fine-tuned on ToolBench.

Make New Tools

contributor project main feature
Google, etc. LATM LLMs create their own reusable tools for problem-solving.

Agent

contributor project main feature
@Significant-Gravitas Auto-GPT chains together LLM "thoughts", to autonomously achieve whatever goal you set.
@yoheinakajima BabyAGI The main idea behind this system is that it creates tasks based on the result of previous tasks and a predefined objective.
The script then uses OpenAI's natural language processing (NLP) capabilities to create new tasks based on the objective,
and Chroma/Weaviate to store and retrieve task results for context.
microsoft HuggingGPT Language serves as an interface for LLMs to connect numerous AI models for solving complicated AI tasks!
microsoft/NCSU ReWOO detaches the reasoning process from external observations, thus significantly reducing token consumption.
Stanford generative_agents Generative Agents: Interactive Simulacra of Human Behavior.
THU, etc. AgentVerse 🤖 AgentVerse 🪐 provides a flexible framework that simplifies the process of building custom multi-agent environments for large language models (LLMs).
BUAA, etc. TrafficGPT By seamlessly intertwining large language model and traffic expertise, TrafficGPT not only advances traffic
management but also offers a novel approach to leveraging AI capabilities in this domain.
microsoft, etc. ToRA ToRA is a series of Tool-integrated Reasoning LLM Agents designed to solve challenging mathematical reasoning problems by interacting with tools.
HKU OpenAgents an open platform for using and hosting language agents in the wild of everyday life.
THU XAgent an open-source experimental Large Language Model (LLM) driven autonomous agent that can automatically solve various tasks.
It is designed to be a general-purpose agent that can be applied to a wide range of tasks.
Nvidia, etc. Eureka ahuman-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement
capabilities of state-of-the-art LLMs, such as GPT-4, to perform in-context evolutionary optimization over reward code. The resulting rewards can then be used to
acquire complex skills via reinforcement learning. Eureka generates reward functions that outperform expert human-engineered rewards without any task-specific
prompting or pre-defined reward templates. In a diverse suite of 29 open-source RL environments that include 10 distinct robot morphologies,
Eureka outperforms human expert on 83% of the tasks leading to an average normalized improvement of 52% .
THU AgentTuning AgentTuning represents the very first attempt to instruction-tune LLMs using interaction trajectories across multiple agent tasks.
Evaluation results indicate that AgentTuning enables the agent capabilities of LLMs with robust generalization on unseen agent tasks while remaining good on general language abilities.
microsoft AutoGen AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks. AutoGen agents are
customizable, conversable, and seamlessly allow human participation. They can operate in various modes that employ combinations of LLMs, human inputs, and tools.
PKU RestGPT we connect LLMs withRESTful APIs and tackle the practical challenges of planning, API calling, and response parsing. To fully evaluate the performance of RestGPT, we propose RestBench,
a high-quality benchmark which consists of two real-world scenarios and human-annotated instructions with gold solution paths.
RestGPT adopts an iterative coarse-to-fine online planning framework and uses an executor to call RESTful APIs.
microsoft MusicAgent a music domain agent powered by large language models (LLMs). Its goal is to help developers and non-professional music creators automatically analyze user requests and select appropriate tools to solve the problem.
HW, etc. LEGO-Prover the first automated theorem prover powered by the LLM that constructs the proof in a block-by-block manner.
alibaba ModelScope-Agent An agent framework connecting models in ModelScope with the world.
CMU, etc. RoboGen A generative and self-guided robotic agent that endlessly propose and master new skills.
PKU, etc. LLaMA-Rider A LLM training framework that enables LLMs to autonomously explore open worlds based on environmental feedback and their own abilities, and to efficiently learn from collected experiences. In the Minecraft environment,
it has demonstrated better multitasking capabilities than other methods, including ChatGPT task planners. This adaptability to open worlds has been a major achievement for LLMs.
Additionally, LLaMA-Rider's ability to use past task experiences to solve new tasks demonstrates the potential of this method for lifelong exploration and learning in large models.
IDEA, etc. ToG Think-on-Graph (ToG), in which the LLM agent iteratively executes beam search on KG, discovers the most promising reasoning paths, and returns the most likely reasoning results.
Yale, etc. ToolkenGPT represents eachtool as a token (toolken) and learns an embedding for it, enabling tool calls in the same way as generating a regular word token. Once a toolken is triggered, the LLM is prompted to complete arguments for the tool to execute.
tencent AppAgent Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps
Stanford, etc. Meta-Prompting This approach transforms a single LM into a multi-faceted conductor, adept at managing and integrating multiple independent LM queries. By employing high-level instructions, meta-prompting guides the LM to break down complex tasks into smaller, more manageable subtasks. These subtasks are then handled by distinct "expert" instances of the same LM, each operating under specific, tailored instructions.
tencent More-Agents-Is-All-You-Need We find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty.
Pythagora-io gpt-pilot GPT Pilot aims to research how much LLMs can be utilized to generate fully working, production-ready apps while the developer oversees the implementation.
DeepWisdom, etc. MetaGPT A Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming.
OpenBMB XAgent An Autonomous LLM Agent for Complex Task Solving.
CrewAI crewAI Cutting-edge framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
stition.ai devika an Agentic AI Software Engineer that can understand high-level human instructions, break them down into steps, research relevant information, and write code to achieve the given objective. Devika aims to be a competitive open-source alternative toDevin by Cognition AI.
OpenDevin OpenDevin an open-source project aiming to replicate Devin, an autonomous AI software engineer who is capable of executing complex engineering tasks and collaborating actively with users on software development projects. This project aspires to replicate, enhance, and innovate upon Devin through the power of the open-source community.
alibaba AgentScope 结合丰富的语法工具、内置资源和用户友好的交互，AgentScope 的通信机制显著降低了开发和理解的障碍。为了实现健壮和灵活的多智能体应用，AgentScope 提供了内置和可定制的容错机制，同时也具备系统级支持多模态数据生成、存储和传输的能力。此外，设计了一个基于 actor 的分布式框架，使得本地和分布式部署之间的轻松转换以及自动并行优化成为可能，无需额外努力。通过这些特性，AgentScope 赋予开发者构建充分发挥智能代理潜力的应用程序的能力。

paper list : LLM-Agent-Paper-List

Papers / Repos / Blogs / ... : Awesome LLM-Powered Agent

LLMs as XXX

contributor LLM as repo main feature
Google DeepMind optimizer OPRO Optimization by PROmpting (OPRO), a simple and effective approach to LLMs
as optimizers, where the optimization task is described in natural language. In each optimization step, the LLM generates new solutions from the prompt that
contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step.
HKU, etc. part of graph tasks Awesome-LLMs-in-Graph-tasks A curated collection of research papers exploring the utilization of LLMs for graph-related tasks.

Similar Collections

collections of open instruction-following llms
开源微调大型语言模型（LLM）合集
机器之心SOTA!模型
Awesome Totally Open Chatgpt
LLM-Zoo
Awesome-LLM
🤗 Open LLM Leaderboard
Open LLMs
Awesome-Chinese-LLM
Awesome Pretrained Chinese NLP Models
LLMSurvey

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for OpenGPTAndBeyond

Similar Open Source Tools

OpenGPTAndBeyond Screenshot
OpenGPTAndBeyond

github
: 102
Pearl Screenshot
Pearl

Pearl is a production-ready Reinforcement Learning AI agent library open-sourced by the Applied Reinforcement Learning team at Meta. It enables researchers and practitioners to develop Reinforcement Learning AI agents that prioritize cumulative long-term feedback over immediate feedback and can adapt to environments with limited observability, sparse feedback, and high stochasticity. Pearl offers a diverse set of unique features for production environments, including dynamic action spaces, offline learning, intelligent neural exploration, safe decision making, history summarization, and data augmentation.

github
: 2.6k
MMLU-Pro Screenshot
MMLU-Pro

MMLU-Pro is an enhanced benchmark designed to evaluate language understanding models across broader and more challenging tasks. It integrates more challenging, reasoning-focused questions and increases answer choices per question, significantly raising difficulty. The dataset comprises over 12,000 questions from academic exams and textbooks across 14 diverse domains. Experimental results show a significant drop in accuracy compared to the original MMLU, with greater stability under varying prompts. Models utilizing Chain of Thought reasoning achieved better performance on MMLU-Pro.

github
: 106
free-for-life Screenshot
free-for-life

A massive list including a huge amount of products and services that are completely free! ⭐ Star on GitHub • 🤝 Contribute # Table of Contents * APIs, Data & ML * Artificial Intelligence * BaaS * Code Editors * Code Generation * DNS * Databases * Design & UI * Domains * Email * Font * For Students * Forms * Linux Distributions * Messaging & Streaming * PaaS * Payments & Billing * SSL

github
: 755
sqlcoder Screenshot
sqlcoder

Defog's SQLCoder is a family of state-of-the-art large language models (LLMs) designed for converting natural language questions into SQL queries. It outperforms popular open-source models like gpt-4 and gpt-4-turbo on SQL generation tasks. SQLCoder has been trained on more than 20,000 human-curated questions based on 10 different schemas, and the model weights are licensed under CC BY-SA 4.0. Users can interact with SQLCoder through the 'transformers' library and run queries using the 'sqlcoder launch' command in the terminal. The tool has been tested on NVIDIA GPUs with more than 16GB VRAM and Apple Silicon devices with some limitations. SQLCoder offers a demo on their website and supports quantized versions of the model for consumer GPUs with sufficient memory.

github
: 2.8k
Timestamp Screenshot
Timestamp

This repository is designed to inject backdoors into Language Model Models (LLMs) for code. The injected backdoors serve as timestamps for the training dataset of the LLMs. The code is randomly generated and includes watermark backdoors to show specific behaviors. A script automatically updates the repository with a new backdoor every month. Validating the existence of the backdoor can infer when the training dataset was collected. The backdoors are constructed in a specific format, and verifying them may require multiple tries. The repository keeps a record of backdoors injected along with associated dates.

github
: 109
awesome-generative-ai-apis Screenshot
awesome-generative-ai-apis

Awesome Generative AI & LLM APIs is a curated list of useful APIs that allow developers to integrate generative models into their applications without building the models from scratch. These APIs provide an interface for generating text, images, or other content, and include pre-trained language models for various tasks. The goal of this project is to create a hub for developers to create innovative applications, enhance user experiences, and drive progress in the AI field.

github
: 70
Awesome-LLM Screenshot
Awesome-LLM

Awesome-LLM is a curated list of resources related to large language models, focusing on papers, projects, frameworks, tools, tutorials, courses, opinions, and other useful resources in the field. It covers trending LLM projects, milestone papers, other papers, open LLM projects, LLM training frameworks, LLM evaluation frameworks, tools for deploying LLM, prompting libraries & tools, tutorials, courses, books, and opinions. The repository provides a comprehensive overview of the latest advancements and resources in the field of large language models.

github
: 17.5k
GenAIExamples Screenshot
GenAIExamples

This project provides a collective list of Generative AI (GenAI) and Retrieval-Augmented Generation (RAG) examples such as chatbot with question and answering (ChatQnA), code generation (CodeGen), document summary (DocSum), etc.

github
: 243
flowgen Screenshot
flowgen

FlowGen is a tool built for AutoGen, a great agent framework from Microsoft and a lot of contributors. It provides intuitive visual tools that streamline the construction and oversight of complex agent-based workflows, simplifying the process for creators and developers. Users can create Autoflows, chat with agents, and share flow templates. The tool is fully dockerized and supports deployment on Railway.app. Contributions to the project are welcome, and the platform uses semantic-release for versioning and releases.

github
: 123
redis-ai-resources Screenshot
redis-ai-resources

A curated repository of code recipes, demos, and resources for basic and advanced Redis use cases in the AI ecosystem. It includes demos for ArxivChatGuru, Redis VSS, Vertex AI & Redis, Agentic RAG, ArXiv Search, and Product Search. Recipes cover topics like Getting started with RAG, Semantic Cache, Advanced RAG, and Recommendation systems. The repository also provides integrations/tools like RedisVL, AWS Bedrock, LangChain Python, LangChain JS, LlamaIndex, Semantic Kernel, RelevanceAI, and DocArray. Additional content includes blog posts, talks, reviews, and documentation related to Vector Similarity Search, AI-Powered Document Search, Vector Databases, Real-Time Product Recommendations, and more. Benchmarks compare Redis against other Vector Databases and ANN benchmarks. Documentation includes QuickStart guides, official literature for Vector Similarity Search, Redis-py client library docs, Redis Stack documentation, and Redis client list.

github
: 123
specification Screenshot
specification

OWASP CycloneDX is a full-stack Bill of Materials (BOM) standard that provides advanced supply chain capabilities for cyber risk reduction. The specification supports various types of Bill of Materials including Software, Hardware, Machine Learning, Cryptography, Manufacturing, and Operations. It also includes support for Vulnerability Disclosure Reports, Vulnerability Exploitability eXchange, and CycloneDX Attestations. CycloneDX helps organizations accurately inventory all components used in software development to identify risks, enhance transparency, and enable rapid impact analysis. The project is managed by the CycloneDX Core Working Group under the OWASP Foundation and is supported by the global information security community.

github
: 348
taranis-ai Screenshot
taranis-ai

Taranis AI is an advanced Open-Source Intelligence (OSINT) tool that leverages Artificial Intelligence to revolutionize information gathering and situational analysis. It navigates through diverse data sources like websites to collect unstructured news articles, utilizing Natural Language Processing and Artificial Intelligence to enhance content quality. Analysts then refine these AI-augmented articles into structured reports that serve as the foundation for deliverables such as PDF files, which are ultimately published.

github
: 240
AI-Gateway Screenshot
AI-Gateway

The AI-Gateway repository explores the AI Gateway pattern through a series of experimental labs, focusing on Azure API Management for handling AI services APIs. The labs provide step-by-step instructions using Jupyter notebooks with Python scripts, Bicep files, and APIM policies. The goal is to accelerate experimentation of advanced use cases and pave the way for further innovation in the rapidly evolving field of AI. The repository also includes a Mock Server to mimic the behavior of the OpenAI API for testing and development purposes.

github
: 155
generative-ai-cdk-constructs Screenshot
generative-ai-cdk-constructs

The AWS Generative AI Constructs Library is an open-source extension of the AWS Cloud Development Kit (AWS CDK) that provides multi-service, well-architected patterns for quickly defining solutions in code to create predictable and repeatable infrastructure, called constructs. The goal of AWS Generative AI CDK Constructs is to help developers build generative AI solutions using pattern-based definitions for their architecture. The patterns defined in AWS Generative AI CDK Constructs are high level, multi-service abstractions of AWS CDK constructs that have default configurations based on well-architected best practices. The library is organized into logical modules using object-oriented techniques to create each architectural pattern model.

github
: 341
InternLM-XComposer Screenshot
InternLM-XComposer

InternLM-XComposer2 is a groundbreaking vision-language large model (VLLM) based on InternLM2-7B excelling in free-form text-image composition and comprehension. It boasts several amazing capabilities and applications: * **Free-form Interleaved Text-Image Composition** : InternLM-XComposer2 can effortlessly generate coherent and contextual articles with interleaved images following diverse inputs like outlines, detailed text requirements and reference images, enabling highly customizable content creation. * **Accurate Vision-language Problem-solving** : InternLM-XComposer2 accurately handles diverse and challenging vision-language Q&A tasks based on free-form instructions, excelling in recognition, perception, detailed captioning, visual reasoning, and more. * **Awesome performance** : InternLM-XComposer2 based on InternLM2-7B not only significantly outperforms existing open-source multimodal models in 13 benchmarks but also **matches or even surpasses GPT-4V and Gemini Pro in 6 benchmarks** We release InternLM-XComposer2 series in three versions: * **InternLM-XComposer2-4KHD-7B** 🤗: The high-resolution multi-task trained VLLM model with InternLM-7B as the initialization of the LLM for _High-resolution understanding_ , _VL benchmarks_ and _AI assistant_. * **InternLM-XComposer2-VL-7B** 🤗 : The multi-task trained VLLM model with InternLM-7B as the initialization of the LLM for _VL benchmarks_ and _AI assistant_. **It ranks as the most powerful vision-language model based on 7B-parameter level LLMs, leading across 13 benchmarks.** * **InternLM-XComposer2-VL-1.8B** 🤗 : A lightweight version of InternLM-XComposer2-VL based on InternLM-1.8B. * **InternLM-XComposer2-7B** 🤗: The further instruction tuned VLLM for _Interleaved Text-Image Composition_ with free-form inputs. Please refer to Technical Report and 4KHD Technical Reportfor more details.

github
: 2.5k

For similar tasks

No tools available

For similar jobs

No tools available