Efficient-Multimodal-LLMs-Survey

Efficient Multimodal Large Language Models: A Survey

Stars: 206

Visit

Efficient Multimodal Large Language Models: A Survey provides a comprehensive review of efficient and lightweight Multimodal Large Language Models (MLLMs), focusing on model size reduction and cost efficiency for edge computing scenarios. The survey covers the timeline of efficient MLLMs, research on efficient structures and strategies, and their applications, while also discussing current limitations and future directions.

README:

Efficient-Multimodal-LLMs-Survey

Efficient Multimodal Large Language Models: A Survey [arXiv]

Yizhang Jin¹², Jian Li¹, Yexin Liu³, Tianjun Gu⁴, Kai Wu¹, Zhengkai Jiang¹, Muyang He³, Bo Zhao³, Xin Tan⁴, Zhenye Gan¹, Yabiao Wang¹, Chengjie Wang¹, Lizhuang Ma²

¹Tencent YouTu Lab, ²SJTU, ³BAAI, ⁴ECNU

⚡We will actively maintain this repository and incorporate new research as it emerges. If you have any questions, please contact [email protected]. Welcome to collaborate on academic research and writing papers together.(欢迎学术合作).

@misc{jin2024efficient,
      title={Efficient Multimodal Large Language Models: A Survey}, 
      author={Yizhang Jin and Jian Li and Yexin Liu and Tianjun Gu and Kai Wu and Zhengkai Jiang and Muyang He and Bo Zhao and Xin Tan and Zhenye Gan and Yabiao Wang and Chengjie Wang and Lizhuang Ma},
      year={2024},
      eprint={2405.10739},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

📌 What is This Survey About?

In the past year, Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning. However, the extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. Thus, studying efficient and lightweight MLLMs has enormous potential, especially in edge computing scenarios. In this survey, we provide a comprehensive and systematic review of the current state of efficient MLLMs. Specifically, we summarize the timeline of representative efficient MLLMs, research state of efficient structures and strategies, and the applications. Finally, we discuss the limitations of current efficient MLLM research and promising future directions.

Summary of 17 Mainstream Efficient MLLMs

Model	Vision Encoder	Resolution	Vision Encoder Parameter Size	LLM	LLM Parameter Size	Vision-LLM Projector	Timeline
MobileVLM	CLIP ViT-L/14	336	0.3B	MobileLLaMA	2.7B	LDP	2023-12
LLaVA-Phi	CLIP ViT-L/14	336	0.3B	Phi-2	2.7B	MLP	2024-01
Imp-v1	SigLIP	384	0.4B	Phi-2	2.7B	-	2024-02
TinyLLaVA	SigLIP-SO	384	0.4B	Phi-2	2.7B	MLP	2024-02
Bunny	SigLIP-SO	384	0.4B	Phi-2	2.7B	MLP	2024-02
MobileVLM-v2-3B	CLIP ViT-L/14	336	0.3B	MobileLLaMA	2.7B	LDPv2	2024-02
MoE-LLaVA-3.6B	CLIP-Large	384	-	Phi-2	2.7B	MLP	2024-02
Cobra	DINOv2, SigLIP-SO	384	0.3B+0.4B	Mamba-2.8b-Zephyr	2.8B	MLP	2024-03
Mini-Gemini	CLIP-Large	336	-	Gemma	2B	MLP	2024-03
Vary-toy	CLIP	224	-	Qwen	1.8B	-	2024-01
TinyGPT-V	EVA	224/448	-	Phi-2	2.7B	Q-Former	2024-01
SPHINX-Tiny	DINOv2 , CLIP-ConvNeXt	448	-	TinyLlama	1.1B	-	2024-02
ALLaVA-Longer	CLIP-ViT-L/14	336	0.3B	Phi-2	2.7B	-	2024-02
MM1-3B-MoE-Chat	CLIP_DFN-ViT-H	378	-	-	3B	C-Abstractor	2024-03
LLaVA-Gemma	DinoV2	-	-	Gemma-2b-it	2B	-	2024-03
Mipha-3B	SigLIP	384	-	Phi-2	2.7B	-	2024-03
VL-Mamba	SigLIP-SO	384	-	Mamba-2.8B-Slimpj	2.8B	VSS-L2	2024-03
MiniCPM-V 2.0	SigLIP	-	0.4B	MiniCPM	2.7B	Perceiver Resampler	2024-03
DeepSeek-VL	SigLIP-L	384	0.4B	DeepSeek-LLM	1.3B	MLP	2024-03
KarmaVLM	SigLIP-SO	384	0.4B	Qwen1.5	0.5B	-	2024-02
moondream2	SigLIP	-	-	Phi-1.5	1.3B	-	2024-03
Bunny-v1.1-4B	SigLIP	1152	-	Phi-3-Mini-4K	3.8B	-	2024-02

Efficient MLLMs

Architecture

Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv, 2023 [Paper]
Llava-phi: Efficient multi-modal assistant with small language model. arXiv, 2024 [Paper]
Imp-v1: An emprical study of multimodal small language models. arXiv, 2024 [Paper]
TinyLLaVA: A Framework of Small-scale Large Multimodal Models. arxiv, 2024 [Paper]
(Bunny)Efficient multimodal learning from data-centric perspective.arXiv, 2024 [Paper]
Gemini: a family of highly capable multimodal modelsarXiv, 2023 [Paper]
Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv, 2024 [Paper]
Moe-llava: Mixture of experts for large vision-language models. arXiv, 2024 [Paper]
Cobra:Extending mamba to multi-modal large language model for efficient inference. arXiv, 2024 [Paper]
Mini-gemini: Mining the potential of multi-modality vision language models. arXiv, 2024 [Paper]
(Vary-toy)Small language model meets with reinforced vision vocabulary. arXiv, 2024 [Paper]
Tinygpt-v: Efficient multimodal large language model via small backbones.arXiv, 2023 [Paper]
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models.arXiv, 2024 [Paper]
ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model.arXiv, 2024 [Paper]
Mm1: Methods, analysis & insights from multimodal llm pre-training.arXiv, 2024 [Paper]
LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model.arXiv, 2024 [Paper]
Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models.arXiv, 2024 [Paper]
VL-Mamba: Exploring State Space Models for Multimodal Learning.arXiv, 2024 [Paper]
MiniCPM-V 2.0: An Efficient End-side MLLM with Strong OCR and Understanding Capabilities.github, 2024 [Github]
DeepSeek-VL: Towards Real-World Vision-Language Understanding .arXiv, 2024 [Paper]
KarmaVLM: A family of high efficiency and powerful visual language model.github, 2024 [Github]
moondream: tiny vision language model.github, 2024 [Github]

Vision Encoder

Multiple Vision Encoders

Broadening the visual encoding of vision-language models, arXiv, 2024 [Paper]
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference, arXiv, 2024 [Paper]
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models, arXiv, 2024 [Paper]

Lightweight Vision Encoder

ViTamin: Designing Scalable Vision Models in the Vision-Language Era. arXiv, 2024 [Paper]

Vision-Language Projector

MLP-based

Visual Instruction Tuning. arXiv, 2023 [Paper]
Improved baselines with visual instruction tuning. arXiv, 2023 [Paper]
TokenPacker: Efficient Visual Projector for Multimodal LLM. arXiv, 2024 [Paper]

Attention-based

Flamingo: a Visual Language Model for Few-Shot Learning, arXiv, 2022 [Paper]
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, arXiv, 2023 [Paper]
Broadening the visual encoding of vision-language models, arXiv, 2024 [Paper]

CNN-based

MobileVLM V2: Faster and Stronger Baseline for Vision Language Model, arXiv, 2023 [Paper]
Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv, 2023 [Paper]

Mamba-based

Vl-mamba: Exploring state space models for multimodal learning.arXiv, 2024 [Paper]

Hybrid Structure

Honeybee: Locality-enhanced projector for multimodal llm.arXiv, 2023 [Paper]

Small Language Models

Llama: Open and efficient foundation language models. arXiv, 2023 [Paper]
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.website, 2023 [[web](https://vicuna. lmsys. org)]
Phi-2: The surprising power of small language models. blog 2023 [[blog](Microsoft Research Blog)]
Gemma: Open models based on gemini research and technology. arXiv, 2024 [Paper]
Phi-3 technical report: A highly capable language model locally on your phone. 2024

Vision Token Compression

Multi-view Input

Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv, 2024 [Paper]
A pioneering large vision- language model handling resolutions from 336 pixels to 4k hd. arXiv, 2024 [Paper]

Token processing

Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv, 2024 [Paper]
Texthawk: Exploring efficient fine-grained perception of multimodal large language models. arXiv, 2024 [Paper]
Tiny- chart: Efficient chart understanding with visual token merging and program-of-thoughts learning.
Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv, 2024 [Paper]
Madtp: Multi- modal alignment-guided dynamic token pruning for accelerating vision-language transformer. arXiv, 2024 [Paper]
CROSSGET: CROSS-GUIDED ENSEMBLE OF TOKENS FOR ACCELERATING VISION-LANGUAGE TRANSFORMERS. ICML, 2024 [Paper]
Matryoshka Query Transformer for Large Vision-Language Models. arxiv, 2024 [Paper]

Multi-Scale Information Fusion

Mini-gemini: Mining the potential of multi-modality vision language models. arXiv, 2024 [Paper]
When do we not need larger vision models? arXiv, 2024 [Paper] arXiv, 2023 [Paper]

Vision Expert Agents

Plug-and-play grounding of reasoning in multimodal large language models. arXiv, 2024 [Paper]
Mova: Adapting mixture of vision experts to multimodal context. arXiv, 2024 [Paper]

Video-Specific Methods

Elysium: Exploring object-level perception in videos via mllm. arXiv, 2024 [Paper]
Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv, 2023 [Paper]
Video-llava: Learning united visual representation by alignment before projection. arXiv, 2023 [Paper]

Efficient Structures

Mixture of Experts

Moe-llava: Mixture of experts for large vision-language models. arXiv, 2024 [Paper]
Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv, 2024 [Paper]
Mixtral of experts. arXiv, 2024 [Paper]

Mamba

Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference, arXiv, 2024 [Paper]
Mamba: Linear-time sequence modeling with selective state spaces. arXiv, 2023 [Paper]
Vl-mamba: Exploring state space models for multimodal learning. arXiv, 2024 [Paper]

Inferece Acceleration

On speculative decoding for multimodal large language models. arXiv, 2024 [Paper]
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. arXiv, 2024 [Paper]
Boosting multimodal large language models with visual tokens withdrawal for rapid inference. arXiv, 2024 [Paper]

Training

Pre-Training

Which part to unfreeze

Tinyllava: A framework of small-scale large multimodal models. arXiv, 2024 [Paper]
Vila: On pre-training for visual language models. arXiv, 2023 [Paper]
Sharegpt4v: Improving large multi-modal models with better captions. arXiv, 2023 [Paper]

Multi-stage pre-training

What matters when building vision- language models? arXiv, 2024 [Paper]

Instruction-Tunining

Efficient IT

Cheap and quick: Efficient vision-language instruction tuning for large language models. nips, 2023 [Paper]
Hyperllava: Dynamic visual and language expert tuning for multimodal large language models. arXiv, 2024 [Paper]

Diverse Training Steps

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models.arXiv, 2024 [Paper]
Cobra:Extending mamba to multi-modal large language model for efficient inference. arXiv, 2024 [Paper]
Tinygpt-v: Efficient multimodal large language model via small backbones.arXiv, 2023 [Paper]

Parameter Efficient Transfer Learning

Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models. arXiv, 2024 [Paper]
Memory-space visual prompting for efficient vision-language fine-tuning. arXiv, 2024 [Paper]

Applications

Biomedical Analysis

Training small multimodal models to bridge biomedical competency gap: A case study in radiology imaging. arXiv, 2024 [[Paper]]
Moe-tinymed: Mixture of experts for tiny medical large vision-language models. arXiv, 2024 [Paper]

Document Understanding

Texthawk: Exploring efficient fine-grained perception of multimodal large language models. arXiv, 2024 [Paper]
Tiny- chart: Efficient chart understanding with visual token merging and program-of-thoughts learning. arXiv, 2024 [Paper]
Monkey: Image resolution and text label are important things for large multi-modal models. arXiv, 2024 [Paper]
Hrvda: High-resolution visual document assistant. arXiv, 2023 [Paper]

Video Comprehension

mplug-2: A modular- ized multi-modal foundation model across text, image and video. arXiv, 2023 [Paper]
Video-llava: Learning united visual representation by alignment before projection. arXiv, 2023 [Paper]
Ma-lmm: Memory-augmented large multimodal model for long-term video under- standing. arXiv, 2024 [Paper]
Llama-vid: An image is worth 2 tokens in large language models. arXiv, 2023 [Paper]

For Tasks:

Click tags to check more tools for each tasks

analyze multimodal data build efficient models optimize model size reduce training costs explore edge computing

For Jobs:

research scientist machine learning engineer data scientist ai researcher academic researcher

Alternative AI tools for Efficient-Multimodal-LLMs-Survey

Similar Open Source Tools

Efficient-Multimodal-LLMs-Survey

github

: 206

Efficient-Multimodal-LLMs-Survey

Efficient Multimodal Large Language Models: A Survey provides a comprehensive review of efficient and lightweight Multimodal Large Language Models (MLLMs), focusing on model size reduction and cost efficiency for edge computing scenarios. The survey covers the timeline of efficient MLLMs, research on efficient structures and strategies, and applications. It discusses current limitations and future directions in efficient MLLM research.

github

: 103

ChatGPT-Plugins

ChatGPT-Plugins is a repository containing plugins for ChatGPT-Next-Web. These plugins provide additional functionalities and features to enhance the ChatGPT experience. Users can easily deploy these plugins using Vercel or HuggingFace. The repository includes README files in English and Vietnamese for user guidance.

github

: 72

HuatuoGPT-II

HuatuoGPT2 is an innovative domain-adapted medical large language model that excels in medical knowledge and dialogue proficiency. It showcases state-of-the-art performance in various medical benchmarks, surpassing GPT-4 in expert evaluations and fresh medical licensing exams. The open-source release includes HuatuoGPT2 models in 7B, 13B, and 34B versions, training code for one-stage adaptation, partial pre-training and fine-tuning instructions, and evaluation methods for medical response capabilities and professional pharmacist exams. The tool aims to enhance LLM capabilities in the Chinese medical field through open-source principles.

github

: 308

LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing

LLM-PowerHouse is a comprehensive and curated guide designed to empower developers, researchers, and enthusiasts to harness the true capabilities of Large Language Models (LLMs) and build intelligent applications that push the boundaries of natural language understanding. This GitHub repository provides in-depth articles, codebase mastery, LLM PlayLab, and resources for cost analysis and network visualization. It covers various aspects of LLMs, including NLP, models, training, evaluation metrics, open LLMs, and more. The repository also includes a collection of code examples and tutorials to help users build and deploy LLM-based applications.

github

: 617

Awesome_Multimodel_LLM

github

: 231

dive-into-llms

The 'Dive into Large Language Models' series programming practice tutorial is an extension of the 'Artificial Intelligence Security Technology' course lecture notes from Shanghai Jiao Tong University (Instructor: Zhang Zhuosheng). It aims to provide introductory programming references related to large models. Through simple practice, it helps students quickly grasp large models, better engage in course design, or academic research. The tutorial covers topics such as fine-tuning and deployment, prompt learning and thought chains, knowledge editing, model watermarking, jailbreak attacks, multimodal models, large model intelligent agents, and security. Disclaimer: The content is based on contributors' personal experiences, internet data, and accumulated research work, provided for reference only.

github

: 568

Topu-ai

TOPU Md is a simple WhatsApp user bot created by Topu Tech. It offers various features such as multi-device support, AI photo enhancement, downloader commands, hidden NSFW commands, logo commands, anime commands, economy menu, various games, and audio/video editor commands. Users can fork the repo, get a session ID by pairing code, and deploy on Heroku. The bot requires Node version 18.x or higher for optimal performance. Contributions to TOPU-MD are welcome, and the tool is safe for use on WhatsApp and Heroku. The tool is licensed under the MIT License and is designed to enhance the WhatsApp experience with diverse features.

github

: 62

IDvs.MoRec

This repository contains the source code for the SIGIR 2023 paper 'Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited'. It provides resources for evaluating foundation, transferable, multi-modal, and LLM recommendation models, along with datasets, pre-trained models, and training strategies for IDRec and MoRec using in-batch debiased cross-entropy loss. The repository also offers large-scale datasets, code for SASRec with in-batch debias cross-entropy loss, and information on joining the lab for research opportunities.

github

: 119

gateway

Gateway is a tool that streamlines requests to 100+ open & closed source models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, load balancing, and can be edge-deployed for minimum latency. It is blazing fast with a tiny footprint, supports load balancing across multiple models, providers, and keys, ensures app resilience with fallbacks, offers automatic retries with exponential fallbacks, allows configurable request timeouts, supports multimodal routing, and can be extended with plug-in middleware. It is battle-tested over 300B tokens and enterprise-ready for enhanced security, scale, and custom deployments.

github

: 6.0k

amber-train

Amber is the first model in the LLM360 family, an initiative for comprehensive and fully open-sourced LLMs. It is a 7B English language model with the LLaMA architecture. The model type is a language model with the same architecture as LLaMA-7B. It is licensed under Apache 2.0. The resources available include training code, data preparation, metrics, and fully processed Amber pretraining data. The model has been trained on various datasets like Arxiv, Book, C4, Refined-Web, StarCoder, StackExchange, and Wikipedia. The hyperparameters include a total of 6.7B parameters, hidden size of 4096, intermediate size of 11008, 32 attention heads, 32 hidden layers, RMSNorm ε of 1e^-6, max sequence length of 2048, and a vocabulary size of 32000.

github

: 136

EVE

EVE is an official PyTorch implementation of Unveiling Encoder-Free Vision-Language Models. The project aims to explore the removal of vision encoders from Vision-Language Models (VLMs) and transfer LLMs to encoder-free VLMs efficiently. It also focuses on bridging the performance gap between encoder-free and encoder-based VLMs. EVE offers a superior capability with arbitrary image aspect ratio, data efficiency by utilizing publicly available data for pre-training, and training efficiency with a transparent and practical strategy for developing a pure decoder-only architecture across modalities.

github

: 155

llm-awq

AWQ (Activation-aware Weight Quantization) is a tool designed for efficient and accurate low-bit weight quantization (INT3/4) for Large Language Models (LLMs). It supports instruction-tuned models and multi-modal LMs, providing features such as AWQ search for accurate quantization, pre-computed AWQ model zoo for various LLMs, memory-efficient 4-bit linear in PyTorch, and efficient CUDA kernel implementation for fast inference. The tool enables users to run large models on resource-constrained edge platforms, delivering more efficient responses with LLM/VLM chatbots through 4-bit inference.

github

: 1.9k

llm-apps-java-spring-ai

The 'LLM Applications with Java and Spring AI' repository provides samples demonstrating how to build Java applications powered by Generative AI and Large Language Models (LLMs) using Spring AI. It includes projects for question answering, chat completion models, prompts, templates, multimodality, output converters, embedding models, document ETL pipeline, function calling, image models, and audio models. The repository also lists prerequisites such as Java 21, Docker/Podman, Mistral AI API Key, OpenAI API Key, and Ollama. Users can explore various use cases and projects to leverage LLMs for text generation, vector transformation, document processing, and more.

github

: 294

InternLM

InternLM is a powerful language model series with features such as 200K context window for long-context tasks, outstanding comprehensive performance in reasoning, math, code, chat experience, instruction following, and creative writing, code interpreter & data analysis capabilities, and stronger tool utilization capabilities. It offers models in sizes of 7B and 20B, suitable for research and complex scenarios. The models are recommended for various applications and exhibit better performance than previous generations. InternLM models may match or surpass other open-source models like ChatGPT. The tool has been evaluated on various datasets and has shown superior performance in multiple tasks. It requires Python >= 3.8, PyTorch >= 1.12.0, and Transformers >= 4.34 for usage. InternLM can be used for tasks like chat, agent applications, fine-tuning, deployment, and long-context inference.

github

: 6.3k

llm-app-stack

LLM App Stack, also known as Emerging Architectures for LLM Applications, is a comprehensive list of available tools, projects, and vendors at each layer of the LLM app stack. It covers various categories such as Data Pipelines, Embedding Models, Vector Databases, Playgrounds, Orchestrators, APIs/Plugins, LLM Caches, Logging/Monitoring/Eval, Validators, LLM APIs (proprietary and open source), App Hosting Platforms, Cloud Providers, and Opinionated Clouds. The repository aims to provide a detailed overview of tools and projects for building, deploying, and maintaining enterprise data solutions, AI models, and applications.

github

: 929

For similar tasks

Efficient-Multimodal-LLMs-Survey

github

: 206

For similar jobs

Perplexica

Perplexica is an open-source AI-powered search engine that utilizes advanced machine learning algorithms to provide clear answers with sources cited. It offers various modes like Copilot Mode, Normal Mode, and Focus Modes for specific types of questions. Perplexica ensures up-to-date information by using SearxNG metasearch engine. It also features image and video search capabilities and upcoming features include finalizing Copilot Mode and adding Discover and History Saving features.

github

: 14.0k

KULLM

KULLM (구름) is a Korean Large Language Model developed by Korea University NLP & AI Lab and HIAI Research Institute. It is based on the upstage/SOLAR-10.7B-v1.0 model and has been fine-tuned for instruction. The model has been trained on 8×A100 GPUs and is capable of generating responses in Korean language. KULLM exhibits hallucination and repetition phenomena due to its decoding strategy. Users should be cautious as the model may produce inaccurate or harmful results. Performance may vary in benchmarks without a fixed system prompt.

github

: 527

MMMU

MMMU is a benchmark designed to evaluate multimodal models on college-level subject knowledge tasks, covering 30 subjects and 183 subfields with 11.5K questions. It focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of various models highlights substantial challenges, with room for improvement to stimulate the community towards expert artificial general intelligence (AGI).

github

: 338

1filellm

1filellm is a command-line data aggregation tool designed for LLM ingestion. It aggregates and preprocesses data from various sources into a single text file, facilitating the creation of information-dense prompts for large language models. The tool supports automatic source type detection, handling of multiple file formats, web crawling functionality, integration with Sci-Hub for research paper downloads, text preprocessing, and token count reporting. Users can input local files, directories, GitHub repositories, pull requests, issues, ArXiv papers, YouTube transcripts, web pages, Sci-Hub papers via DOI or PMID. The tool provides uncompressed and compressed text outputs, with the uncompressed text automatically copied to the clipboard for easy pasting into LLMs.

github

: 292

gpt-researcher

GPT Researcher is an autonomous agent designed for comprehensive online research on a variety of tasks. It can produce detailed, factual, and unbiased research reports with customization options. The tool addresses issues of speed, determinism, and reliability by leveraging parallelized agent work. The main idea involves running 'planner' and 'execution' agents to generate research questions, seek related information, and create research reports. GPT Researcher optimizes costs and completes tasks in around 3 minutes. Features include generating long research reports, aggregating web sources, an easy-to-use web interface, scraping web sources, and exporting reports to various formats.

github

: 14.5k

ChatTTS

ChatTTS is a generative speech model optimized for dialogue scenarios, providing natural and expressive speech synthesis with fine-grained control over prosodic features. It supports multiple speakers and surpasses most open-source TTS models in terms of prosody. The model is trained with 100,000+ hours of Chinese and English audio data, and the open-source version on HuggingFace is a 40,000-hour pre-trained model without SFT. The roadmap includes open-sourcing additional features like VQ encoder, multi-emotion control, and streaming audio generation. The tool is intended for academic and research use only, with precautions taken to limit potential misuse.

github

: 31.5k

HebTTS

HebTTS is a language modeling approach to diacritic-free Hebrew text-to-speech (TTS) system. It addresses the challenge of accurately mapping text to speech in Hebrew by proposing a language model that operates on discrete speech representations and is conditioned on a word-piece tokenizer. The system is optimized using weakly supervised recordings and outperforms diacritic-based Hebrew TTS systems in terms of content preservation and naturalness of generated speech.

github

: 52

do-research-in-AI

This repository is a collection of research lectures and experience sharing posts from frontline researchers in the field of AI. It aims to help individuals upgrade their research skills and knowledge through insightful talks and experiences shared by experts. The content covers various topics such as evaluating research papers, choosing research directions, research methodologies, and tips for writing high-quality scientific papers. The repository also includes discussions on academic career paths, research ethics, and the emotional aspects of research work. Overall, it serves as a valuable resource for individuals interested in advancing their research capabilities in the field of AI.

github

: 61